From 3e64c780f5b066fdffa55593bde65a56f628af71 Mon Sep 17 00:00:00 2001 From: Jon Moss Date: Thu, 19 Jul 2018 15:35:01 -0400 Subject: [PATCH 001/551] doc: lint README.md Fixes grammar, removes extra lines and spaces, etc. Also removes a few references to `node-waf`, which was removed ~6 years ago now. Happy to add back if people still need that information. PR-URL: https://github.com/nodejs/node-gyp/pull/1498 Reviewed-By: Vse Mozhet Byt Reviewed-By: Richard Lau --- README.md | 168 +++++++++++++++++++++++------------------------------- 1 file changed, 72 insertions(+), 96 deletions(-) diff --git a/README.md b/README.md index 0fed03c543..e88bd0b5be 100644 --- a/README.md +++ b/README.md @@ -1,28 +1,21 @@ -node-gyp -========= -## Node.js native addon build tool +# `node-gyp` - Node.js native addon build tool `node-gyp` is a cross-platform command-line tool written in Node.js for compiling -native addon modules for Node.js. It bundles the [gyp](https://gyp.gsrc.io) +native addon modules for Node.js. It bundles the [gyp](https://gyp.gsrc.io) project used by the Chromium team and takes away the pain of dealing with the -various differences in build platforms. It is the replacement to the `node-waf` -program which is removed for node `v0.8`. If you have a native addon for node that -still has a `wscript` file, then you should definitely add a `binding.gyp` file -to support the latest versions of node. +various differences in build platforms. -Multiple target versions of node are supported (i.e. `0.8`, ..., `4`, `5`, `6`, -etc.), regardless of what version of node is actually installed on your system +Multiple target versions of Node.js are supported (i.e. `0.8`, ..., `4`, `5`, `6`, +etc.), regardless of what version of Node.js is actually installed on your system (`node-gyp` downloads the necessary development files or headers for the target version). ## Features * Easy to use, consistent interface * Same commands to build your module on every platform - * Supports multiple target versions of Node + * Supports multiple target versions of Node.js - -Installation ------------- +## Installation You can install with `npm`: @@ -69,7 +62,7 @@ version `node-gyp` uses by setting the '--python' variable: $ node-gyp --python /path/to/python2.7 ``` -If `node-gyp` is called by way of `npm` *and* you have multiple versions of +If `node-gyp` is called by way of `npm`, *and* you have multiple versions of Python installed, then you can set `npm`'s 'python' config key to the appropriate value: @@ -77,8 +70,7 @@ value: $ npm config set python /path/to/executable/python2.7 ``` -How to Use ----------- +## How to Use To compile your native addon, first go to its root directory: @@ -99,33 +91,30 @@ needs to be added (not needed when run by npm as configured above): $ node-gyp configure --msvs_version=2015 ``` -__Note__: The `configure` step looks for the `binding.gyp` file in the current -directory to process. See below for instructions on creating the `binding.gyp` file. +__Note__: The `configure` step looks for a `binding.gyp` file in the current +directory to process. See below for instructions on creating a `binding.gyp` file. Now you will have either a `Makefile` (on Unix platforms) or a `vcxproj` file -(on Windows) in the `build/` directory. Next invoke the `build` command: +(on Windows) in the `build/` directory. Next, invoke the `build` command: ``` bash $ node-gyp build ``` Now you have your compiled `.node` bindings file! The compiled bindings end up -in `build/Debug/` or `build/Release/`, depending on the build mode. At this point -you can require the `.node` file with Node and run your tests! +in `build/Debug/` or `build/Release/`, depending on the build mode. At this point, +you can require the `.node` file with Node.js and run your tests! __Note:__ To create a _Debug_ build of the bindings file, pass the `--debug` (or -`-d`) switch when running either the `configure`, `build` or `rebuild` command. - +`-d`) switch when running either the `configure`, `build` or `rebuild` commands. -The "binding.gyp" file ----------------------- +## The `binding.gyp` file -Previously when node had `node-waf` you had to write a `wscript` file. The -replacement for that is the `binding.gyp` file, which describes the configuration -to build your module in a JSON-like format. This file gets placed in the root of -your package, alongside the `package.json` file. +A `binding.gyp` file describes the configuration to build your module, in a +JSON-like format. This file gets placed in the root of your package, alongside +`package.json`. -A barebones `gyp` file appropriate for building a node addon looks like: +A barebones `gyp` file appropriate for building a Node.js addon could look like: ``` python { @@ -147,8 +136,7 @@ Some additional resources for addons and writing `gyp` files: * [*"binding.gyp" files out in the wild* wiki page](https://github.com/nodejs/node-gyp/wiki/%22binding.gyp%22-files-out-in-the-wild) -Commands --------- +## Commands `node-gyp` responds to the following commands: @@ -159,86 +147,74 @@ Commands | `clean` | Removes the `build` directory if it exists | `configure` | Generates project build files for the current platform | `rebuild` | Runs `clean`, `configure` and `build` all in a row -| `install` | Installs node header files for the given version -| `list` | Lists the currently installed node header versions -| `remove` | Removes the node header files for the given version +| `install` | Installs Node.js header files for the given version +| `list` | Lists the currently installed Node.js header versions +| `remove` | Removes the Node.js header files for the given version -Command Options --------- +## Command Options `node-gyp` accepts the following command options: | **Command** | **Description** |:----------------------------------|:------------------------------------------ -| `-j n`, `--jobs n` | Run make in parallel -| `--target=v6.2.1` | Node version to build for (default=process.version) +| `-j n`, `--jobs n` | Run `make` in parallel +| `--target=v6.2.1` | Node.js version to build for (default is `process.version`) | `--silly`, `--loglevel=silly` | Log all progress to console | `--verbose`, `--loglevel=verbose` | Log most progress to console | `--silent`, `--loglevel=silent` | Don't log anything to console -| `debug`, `--debug` | Make Debug build (default=Release) +| `debug`, `--debug` | Make Debug build (default is `Release`) | `--release`, `--no-debug` | Make Release build | `-C $dir`, `--directory=$dir` | Run command in different directory -| `--make=$make` | Override make command (e.g. gmake) +| `--make=$make` | Override `make` command (e.g. `gmake`) | `--thin=yes` | Enable thin static libraries | `--arch=$arch` | Set target architecture (e.g. ia32) | `--tarball=$path` | Get headers from a local tarball -| `--devdir=$path` | SDK download directory (default=~/.node-gyp) +| `--devdir=$path` | SDK download directory (default is `~/.node-gyp`) | `--ensure` | Don't reinstall headers if already present | `--dist-url=$url` | Download header tarball from custom URL | `--proxy=$url` | Set HTTP proxy for downloading header tarball | `--cafile=$cafile` | Override default CA chain (to download tarball) | `--nodedir=$path` | Set the path to the node source code -| `--python=$path` | Set path to the python (2) binary -| `--msvs_version=$version` | Set Visual Studio version (win) -| `--solution=$solution` | Set Visual Studio Solution version (win) - - -Configuration --------- - -__`node-gyp` responds to environment variables or `npm` configuration__ -1. Environment variables take the form `npm_config_OPTION_NAME` for any of the - options listed above (dashes in option names should be replaced by underscores). - These work also when `node-gyp` is invoked directly: - `$ export npm_config_devdir=/tmp/.gyp` - or on Windows - `> set npm_config_devdir=c:\temp\.gyp` -2. As `npm` configuration, variables take the form `OPTION_NAME`. - This way only works when `node-gyp` is executed by `npm`: - `$ npm config set [--global] devdir /tmp/.gyp` - `$ npm i buffertools` - - - -License -------- - -(The MIT License) - -Copyright (c) 2012 Nathan Rajlich <nathan@tootallnate.net> - -Permission is hereby granted, free of charge, to any person obtaining -a copy of this software and associated documentation files (the -'Software'), to deal in the Software without restriction, including -without limitation the rights to use, copy, modify, merge, publish, -distribute, sublicense, and/or sell copies of the Software, and to -permit persons to whom the Software is furnished to do so, subject to -the following conditions: - -The above copyright notice and this permission notice shall be -included in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, -EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF -MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. -IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY -CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, -TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE -SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - - -[python-v2.7.10]: https://www.python.org/downloads/release/python-2710/ -[msvc2013]: https://www.microsoft.com/en-gb/download/details.aspx?id=44914 -[win7sdk]: https://www.microsoft.com/en-us/download/details.aspx?id=8279 -[compiler update for the Windows SDK 7.1]: https://www.microsoft.com/en-us/download/details.aspx?id=4422 +| `--python=$path` | Set path to the Python 2 binary +| `--msvs_version=$version` | Set Visual Studio version (Windows only) +| `--solution=$solution` | Set Visual Studio Solution version (Windows only) + +## Configuration + +### Environment variables + +Use the form `npm_config_OPTION_NAME` for any of the command options listed +above (dashes in option names should be replaced by underscores). + +For example, to set `devdir` equal to `/tmp/.gyp`, you would: + +Run this on Unix: + +```bash +$ export npm_config_devdir=/tmp/.gyp +``` + +Or this on Windows: + +```console +> set npm_config_devdir=c:\temp\.gyp +``` + +### `npm` configuration + +Use the form `OPTION_NAME` for any of the command options listed above. + +For example, to set `devdir` equal to `/tmp/.gyp`, you would run: + +```bash +$ npm config set [--global] devdir /tmp/.gyp +``` + +**Note:** Configuration set via `npm` will only be used when `node-gyp` +is run via `npm`, not when `node-gyp` is run directly. + +## License + +`node-gyp` is available under the MIT license. See the [LICENSE +file](LICENSE) for details. From 9e46872ea38f4c8a1f02975437bec8812f260e8d Mon Sep 17 00:00:00 2001 From: Jon Moss Date: Fri, 27 Jul 2018 23:36:02 -0400 Subject: [PATCH 002/551] bin,lib: remove extra comments/lines/spaces - Removes "module dependencies" comments and things that, IMHO, don't add too much value. Happy to add back if helps some people when reading through `node-gyp`. - DRY up `lib/process-release.js`. - Removes a bunch of extra blank lines, as well as random spaces. PR-URL: https://github.com/nodejs/node-gyp/pull/1508 Reviewed-By: Richard Lau --- bin/node-gyp.js | 10 +-------- lib/build.js | 14 +------------ lib/clean.js | 7 ------- lib/configure.js | 15 ++----------- lib/find-node-directory.js | 4 +--- lib/find-vs2017.js | 4 +--- lib/install.js | 7 ------- lib/list.js | 6 ------ lib/node-gyp.js | 13 ------------ lib/process-release.js | 43 +++++++++++++------------------------- lib/rebuild.js | 1 - lib/remove.js | 6 ------ 12 files changed, 21 insertions(+), 109 deletions(-) diff --git a/bin/node-gyp.js b/bin/node-gyp.js index 70d7d50262..0cc3517748 100755 --- a/bin/node-gyp.js +++ b/bin/node-gyp.js @@ -1,15 +1,7 @@ #!/usr/bin/env node -/** - * Set the title. - */ - process.title = 'node-gyp' -/** - * Module dependencies. - */ - var gyp = require('../') var log = require('npmlog') var osenv = require('osenv') @@ -126,7 +118,7 @@ process.on('uncaughtException', function (err) { }) function errorMessage () { - // copied from npm's lib/util/error-handler.js + // copied from npm's lib/utils/error-handler.js var os = require('os') log.error('System', os.type() + ' ' + os.release()) log.error('command', process.argv diff --git a/lib/build.js b/lib/build.js index 2f8e14c374..81494d65aa 100644 --- a/lib/build.js +++ b/lib/build.js @@ -1,10 +1,6 @@ module.exports = exports = build -/** - * Module dependencies. - */ - var fs = require('graceful-fs') , rm = require('rimraf') , path = require('path') @@ -32,8 +28,6 @@ function build (gyp, argv, callback) { var release = processRelease(argv, gyp, process.version, process.release) , makeCommand = gyp.opts.make || process.env.MAKE || platformMake , command = win ? 'msbuild' : makeCommand - , buildDir = path.resolve('build') - , configPath = path.resolve(buildDir, 'config.gypi') , jobs = gyp.opts.jobs || process.env.JOBS , buildType , config @@ -47,6 +41,7 @@ function build (gyp, argv, callback) { */ function loadConfigGypi () { + var configPath = path.resolve('build', 'config.gypi') fs.readFile(configPath, 'utf8', function (err, data) { if (err) { if (err.code == 'ENOENT') { @@ -187,13 +182,11 @@ function build (gyp, argv, callback) { }) } - /** * Actually spawn the process and compile the module. */ function doBuild () { - // Enable Verbose build var verbose = log.levels[log.level] <= log.levels.verbose if (!win && verbose) { @@ -253,10 +246,6 @@ function build (gyp, argv, callback) { proc.on('exit', onExit) } - /** - * Invoked after the make/msbuild command exits. - */ - function onExit (code, signal) { if (code !== 0) { return callback(new Error('`' + command + '` failed with exit code: ' + code)) @@ -266,5 +255,4 @@ function build (gyp, argv, callback) { } callback() } - } diff --git a/lib/clean.js b/lib/clean.js index e69164d45a..8172e1c3d3 100644 --- a/lib/clean.js +++ b/lib/clean.js @@ -3,20 +3,13 @@ module.exports = exports = clean exports.usage = 'Removes any generated build files and the "out" dir' -/** - * Module dependencies. - */ - var rm = require('rimraf') var log = require('npmlog') - function clean (gyp, argv, callback) { - // Remove the 'build' dir var buildDir = 'build' log.verbose('clean', 'removing "%s" directory', buildDir) rm(buildDir, callback) - } diff --git a/lib/configure.js b/lib/configure.js index a42abe5972..8d37a7436e 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -5,10 +5,6 @@ module.exports.test = { findPython: findPython, } -/** - * Module dependencies. - */ - var fs = require('graceful-fs') , path = require('path') , log = require('npmlog') @@ -28,7 +24,6 @@ if (win) exports.usage = 'Generates ' + (win ? 'MSVC project files' : 'a Makefile') + ' for the current module' function configure (gyp, argv, callback) { - var python = gyp.opts.python || process.env.PYTHON || 'python2' , buildDir = path.resolve('build') , configNames = [ 'config.gypi', 'common.gypi' ] @@ -46,7 +41,6 @@ function configure (gyp, argv, callback) { }) function getNodeDir () { - // 'python' should be set by now process.env.PYTHON = python @@ -56,7 +50,6 @@ function configure (gyp, argv, callback) { log.verbose('get node dir', 'compiling against specified --nodedir dev files: %s', nodeDir) createBuildDir() - } else { // if no --nodedir specified, ensure node dependencies are installed if ('v' + release.version !== process.version) { @@ -245,12 +238,12 @@ function configure (gyp, argv, callback) { }) // For AIX and z/OS we need to set up the path to the exports file - // which contains the symbols needed for linking. + // which contains the symbols needed for linking. var node_exp_file = undefined if (process.platform === 'aix' || process.platform === 'os390') { var ext = process.platform === 'aix' ? 'exp' : 'x' var node_root_dir = findNodeDirectory() - var candidates = undefined + var candidates = undefined if (process.platform === 'aix') { candidates = ['include/node/node', 'out/Release/node', @@ -336,10 +329,6 @@ function configure (gyp, argv, callback) { }) } - /** - * Called when the `gyp` child process exits. - */ - function onCpExit (code, signal) { if (code !== 0) { callback(new Error('`gyp` failed with exit code: ' + code)) diff --git a/lib/find-node-directory.js b/lib/find-node-directory.js index 3aee8a109a..524909581f 100644 --- a/lib/find-node-directory.js +++ b/lib/find-node-directory.js @@ -1,7 +1,7 @@ var path = require('path') , log = require('npmlog') -function findNodeDirectory(scriptLocation, processObj) { +module.exports = function findNodeDirectory(scriptLocation, processObj) { // set dirname and process if not passed in // this facilitates regression tests if (scriptLocation === undefined) { @@ -57,5 +57,3 @@ function findNodeDirectory(scriptLocation, processObj) { } return node_root_dir } - -module.exports = findNodeDirectory diff --git a/lib/find-vs2017.js b/lib/find-vs2017.js index ad46ceaf88..8655d7c931 100644 --- a/lib/find-vs2017.js +++ b/lib/find-vs2017.js @@ -2,7 +2,7 @@ var log = require('npmlog') , execFile = require('child_process').execFile , path = require('path') -function findVS2017(callback) { +module.exports = function findVS2017(callback) { var ps = path.join(process.env.SystemRoot, 'System32', 'WindowsPowerShell', 'v1.0', 'powershell.exe') var csFile = path.join(__dirname, 'Find-VS2017.cs') @@ -42,5 +42,3 @@ function findVS2017(callback) { child.stdin.end() } - -module.exports = findVS2017 diff --git a/lib/install.js b/lib/install.js index 8a615dfb38..fa0fef6fe0 100644 --- a/lib/install.js +++ b/lib/install.js @@ -10,10 +10,6 @@ module.exports.test = { exports.usage = 'Install node development files for the specified node version.' -/** - * Module dependencies. - */ - var fs = require('graceful-fs') , osenv = require('osenv') , tar = require('tar') @@ -28,7 +24,6 @@ var fs = require('graceful-fs') , win = process.platform == 'win32' function install (fs, gyp, argv, callback) { - var release = processRelease(argv, gyp, process.version, process.release) // ensure no double-callbacks happen @@ -124,7 +119,6 @@ function install (fs, gyp, argv, callback) { } function go () { - log.verbose('ensuring nodedir is created', devDir) // first create the dir for the node dev files @@ -165,7 +159,6 @@ function install (fs, gyp, argv, callback) { } // download the tarball and extract! - if (tarPath) { return tar.extract({ file: tarPath, diff --git a/lib/list.js b/lib/list.js index 9d680a56a4..cda4fd2f2f 100644 --- a/lib/list.js +++ b/lib/list.js @@ -3,20 +3,14 @@ module.exports = exports = list exports.usage = 'Prints a listing of the currently installed node development files' -/** - * Module dependencies. - */ - var fs = require('graceful-fs') , path = require('path') , log = require('npmlog') function list (gyp, args, callback) { - var devDir = gyp.devDir log.verbose('list', 'using node-gyp dir:', devDir) - // readdir() the node-gyp dir fs.readdir(devDir, onreaddir) function onreaddir (err, versions) { diff --git a/lib/node-gyp.js b/lib/node-gyp.js index a841161e32..b35a5ebe44 100644 --- a/lib/node-gyp.js +++ b/lib/node-gyp.js @@ -1,14 +1,6 @@ -/** - * Module exports. - */ - module.exports = exports = gyp -/** - * Module dependencies. - */ - var fs = require('graceful-fs') , path = require('path') , nopt = require('nopt') @@ -35,10 +27,6 @@ var fs = require('graceful-fs') // differentiate node-gyp's logs from npm's log.heading = 'gyp' -/** - * The `gyp` function. - */ - function gyp () { return new Gyp() } @@ -213,4 +201,3 @@ Object.defineProperty(proto, 'version', { } , enumerable: true }) - diff --git a/lib/process-release.js b/lib/process-release.js index 0d177f1c93..4805fcc7d1 100644 --- a/lib/process-release.js +++ b/lib/process-release.js @@ -9,7 +9,7 @@ var semver = require('semver') , bitsreV3 = /\/win-(x86|ia32|x64)\// // io.js v3.x.x shipped with "ia32" but should // have been "x86" -// Captures all the logic required to determine download URLs, local directory and +// Captures all the logic required to determine download URLs, local directory and // file names. Inputs come from command-line switches (--target, --dist-url), // `process.version` and `process.release` where it exists. function processRelease (argv, gyp, defaultVersion, defaultRelease) { @@ -88,35 +88,22 @@ function processRelease (argv, gyp, defaultVersion, defaultRelease) { baseUrl = url.resolve(defaultRelease.headersUrl, './') libUrl32 = resolveLibUrl(name, defaultRelease.libUrl || baseUrl || distBaseUrl, 'x86', versionSemver.major) libUrl64 = resolveLibUrl(name, defaultRelease.libUrl || baseUrl || distBaseUrl, 'x64', versionSemver.major) - - return { - version: version, - semver: versionSemver, - name: name, - baseUrl: baseUrl, - tarballUrl: defaultRelease.headersUrl, - shasumsUrl: url.resolve(baseUrl, 'SHASUMS256.txt'), - versionDir: (name !== 'node' ? name + '-' : '') + version, - libUrl32: libUrl32, - libUrl64: libUrl64, - libPath32: normalizePath(path.relative(url.parse(baseUrl).path, url.parse(libUrl32).path)), - libPath64: normalizePath(path.relative(url.parse(baseUrl).path, url.parse(libUrl64).path)) - } + tarballUrl = defaultRelease.headersUrl + } else { + // older versions without process.release are captured here and we have to make + // a lot of assumptions, additionally if you --target=x.y.z then we can't use the + // current process.release + baseUrl = distBaseUrl + libUrl32 = resolveLibUrl(name, baseUrl, 'x86', versionSemver.major) + libUrl64 = resolveLibUrl(name, baseUrl, 'x64', versionSemver.major) + + // making the bold assumption that anything with a version number >3.0.0 will + // have a *-headers.tar.gz file in its dist location, even some frankenstein + // custom version + canGetHeaders = semver.satisfies(versionSemver, headersTarballRange) + tarballUrl = url.resolve(baseUrl, name + '-v' + version + (canGetHeaders ? '-headers' : '') + '.tar.gz') } - // older versions without process.release are captured here and we have to make - // a lot of assumptions, additionally if you --target=x.y.z then we can't use the - // current process.release - - baseUrl = distBaseUrl - libUrl32 = resolveLibUrl(name, baseUrl, 'x86', versionSemver.major) - libUrl64 = resolveLibUrl(name, baseUrl, 'x64', versionSemver.major) - // making the bold assumption that anything with a version number >3.0.0 will - // have a *-headers.tar.gz file in its dist location, even some frankenstein - // custom version - canGetHeaders = semver.satisfies(versionSemver, headersTarballRange) - tarballUrl = url.resolve(baseUrl, name + '-v' + version + (canGetHeaders ? '-headers' : '') + '.tar.gz') - return { version: version, semver: versionSemver, diff --git a/lib/rebuild.js b/lib/rebuild.js index 4c6f472aa7..dc8025a7b0 100644 --- a/lib/rebuild.js +++ b/lib/rebuild.js @@ -4,7 +4,6 @@ module.exports = exports = rebuild exports.usage = 'Runs "clean", "configure" and "build" all at once' function rebuild (gyp, argv, callback) { - gyp.todo.push( { name: 'clean', args: [] } , { name: 'configure', args: argv } diff --git a/lib/remove.js b/lib/remove.js index eb80981b88..213af2b9d3 100644 --- a/lib/remove.js +++ b/lib/remove.js @@ -3,10 +3,6 @@ module.exports = exports = remove exports.usage = 'Removes the node development files for the specified version' -/** - * Module dependencies. - */ - var fs = require('fs') , rm = require('rimraf') , path = require('path') @@ -14,7 +10,6 @@ var fs = require('fs') , semver = require('semver') function remove (gyp, argv, callback) { - var devDir = gyp.devDir log.verbose('remove', 'using node-gyp dir:', devDir) @@ -48,5 +43,4 @@ function remove (gyp, argv, callback) { // Go ahead and delete the dir rm(versionPath, callback) }) - } From a5b741049748992bc1eabf2e88b395b51583a8dd Mon Sep 17 00:00:00 2001 From: Jon Moss Date: Thu, 2 Aug 2018 18:12:08 -0400 Subject: [PATCH 003/551] Add ESLint no-unused-vars rule MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Uses `.eslintrc.yaml` for configuration - `npm run lint` is part of `npm test` PR-URL: https://github.com/nodejs/node-gyp/pull/1497 Reviewed-By: Gibson Fahnestock Reviewed-By: João Reis --- .eslintrc.yaml | 5 +++++ bin/node-gyp.js | 2 +- lib/build.js | 9 +++------ lib/configure.js | 10 +++++----- lib/install.js | 9 +++------ lib/list.js | 1 - lib/node-gyp.js | 3 +-- lib/remove.js | 2 +- package.json | 9 ++++++--- test/test-configure-python.js | 4 ++-- test/test-find-accessible-sync.js | 2 +- test/test-find-python.js | 12 ++++++------ 12 files changed, 34 insertions(+), 34 deletions(-) create mode 100644 .eslintrc.yaml diff --git a/.eslintrc.yaml b/.eslintrc.yaml new file mode 100644 index 0000000000..659dce8810 --- /dev/null +++ b/.eslintrc.yaml @@ -0,0 +1,5 @@ +--- + env: + node: true + rules: + no-unused-vars: error diff --git a/bin/node-gyp.js b/bin/node-gyp.js index 0cc3517748..a8fd9bc529 100755 --- a/bin/node-gyp.js +++ b/bin/node-gyp.js @@ -34,7 +34,7 @@ if (prog.todo.length === 0) { } else { console.log('%s', prog.usage()) } - return process.exit(0) + process.exit(0) } log.info('it worked if it ends with', 'ok') diff --git a/lib/build.js b/lib/build.js index 81494d65aa..b8389bcd7b 100644 --- a/lib/build.js +++ b/lib/build.js @@ -2,13 +2,11 @@ module.exports = exports = build var fs = require('graceful-fs') - , rm = require('rimraf') , path = require('path') , glob = require('glob') , log = require('npmlog') , which = require('which') , exec = require('child_process').exec - , processRelease = require('./process-release') , win = process.platform === 'win32' exports.usage = 'Invokes `' + (win ? 'msbuild' : 'make') + '` and builds the module' @@ -25,8 +23,7 @@ function build (gyp, argv, callback) { }) } - var release = processRelease(argv, gyp, process.version, process.release) - , makeCommand = gyp.opts.make || process.env.MAKE || platformMake + var makeCommand = gyp.opts.make || process.env.MAKE || platformMake , command = win ? 'msbuild' : makeCommand , jobs = gyp.opts.jobs || process.env.JOBS , buildType @@ -132,7 +129,7 @@ function build (gyp, argv, callback) { var cmd = 'reg query "HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions" /s' if (process.arch !== 'ia32') cmd += ' /reg:32' - exec(cmd, function (err, stdout, stderr) { + exec(cmd, function (err, stdout) { if (err) { return callback(new Error(err.message + '\n' + notfoundErr)) } @@ -162,7 +159,7 @@ function build (gyp, argv, callback) { ;(function verifyMsbuild () { if (!msbuilds.length) return callback(new Error(notfoundErr)) msbuildPath = path.resolve(msbuilds.pop().path, 'msbuild.exe') - fs.stat(msbuildPath, function (err, stat) { + fs.stat(msbuildPath, function (err) { if (err) { if (err.code == 'ENOENT') { if (msbuilds.length) { diff --git a/lib/configure.js b/lib/configure.js index 8d37a7436e..0baf58dabc 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -69,7 +69,7 @@ function configure (gyp, argv, callback) { // into devdir. Otherwise only install if they're not already there. gyp.opts.ensure = gyp.opts.tarball ? false : true - gyp.commands.install([ release.version ], function (err, version) { + gyp.commands.install([ release.version ], function (err) { if (err) return callback(err) log.verbose('get node dir', 'target node version installed:', release.versionDir) nodeDir = path.resolve(gyp.devDir, release.versionDir) @@ -188,7 +188,7 @@ function configure (gyp, argv, callback) { if (!name) return runGyp() var fullPath = path.resolve(name) log.verbose(name, 'checking for gypi file: %s', fullPath) - fs.stat(fullPath, function (err, stat) { + fs.stat(fullPath, function (err) { if (err) { if (err.code == 'ENOENT') { findConfigs() // check next gypi filename @@ -275,7 +275,7 @@ function configure (gyp, argv, callback) { var gyp_script = path.resolve(__dirname, '..', 'gyp', 'gyp_main.py') var addon_gypi = path.resolve(__dirname, '..', 'addon.gypi') var common_gypi = path.resolve(nodeDir, 'include/node/common.gypi') - fs.stat(common_gypi, function (err, stat) { + fs.stat(common_gypi, function (err) { if (err) common_gypi = path.resolve(nodeDir, 'common.gypi') @@ -329,7 +329,7 @@ function configure (gyp, argv, callback) { }) } - function onCpExit (code, signal) { + function onCpExit (code) { if (code !== 0) { callback(new Error('`gyp` failed with exit code: ' + code)) } else { @@ -492,7 +492,7 @@ PythonFinder.prototype = { } var pythonPath = this.resolve(rootDir, 'Python27', 'python.exe') this.log.verbose('ensuring that file exists:', pythonPath) - this.stat(pythonPath, function (err, stat) { + this.stat(pythonPath, function (err) { if (err) { if (err.code == 'ENOENT') { this.failNoPython() diff --git a/lib/install.js b/lib/install.js index fa0fef6fe0..eea7618a01 100644 --- a/lib/install.js +++ b/lib/install.js @@ -13,7 +13,6 @@ exports.usage = 'Install node development files for the specified node version.' var fs = require('graceful-fs') , osenv = require('osenv') , tar = require('tar') - , rm = require('rimraf') , path = require('path') , crypto = require('crypto') , log = require('npmlog') @@ -33,7 +32,7 @@ function install (fs, gyp, argv, callback) { if (err) { log.warn('install', 'got an error, rolling back install') // roll-back the install if anything went wrong - gyp.commands.remove([ release.versionDir ], function (err2) { + gyp.commands.remove([ release.versionDir ], function () { callback(err) }) } else { @@ -75,7 +74,7 @@ function install (fs, gyp, argv, callback) { // check if it is already installed, and only install when needed if (gyp.opts.ensure) { log.verbose('install', '--ensure was passed, so won\'t reinstall if already installed') - fs.stat(devDir, function (err, stat) { + fs.stat(devDir, function (err) { if (err) { if (err.code == 'ENOENT') { log.verbose('install', 'version not already installed, continuing with install', release.version) @@ -146,7 +145,7 @@ function install (fs, gyp, argv, callback) { // checks if a file to be extracted from the tarball is valid. // only .h header files and the gyp files get extracted - function isValid (path, entry) { + function isValid (path) { var isValid = valid(path) if (isValid) { log.verbose('extracted file from tarball', path) @@ -265,8 +264,6 @@ function install (fs, gyp, argv, callback) { function downloadShasums(done) { log.verbose('check download content checksum, need to download `SHASUMS256.txt`...') - var shasumsPath = path.resolve(devDir, 'SHASUMS256.txt') - log.verbose('checksum url', release.shasumsUrl) try { var req = download(gyp, process.env, release.shasumsUrl) diff --git a/lib/list.js b/lib/list.js index cda4fd2f2f..90f6447180 100644 --- a/lib/list.js +++ b/lib/list.js @@ -4,7 +4,6 @@ module.exports = exports = list exports.usage = 'Prints a listing of the currently installed node development files' var fs = require('graceful-fs') - , path = require('path') , log = require('npmlog') function list (gyp, args, callback) { diff --git a/lib/node-gyp.js b/lib/node-gyp.js index b35a5ebe44..8dd551bf3d 100644 --- a/lib/node-gyp.js +++ b/lib/node-gyp.js @@ -1,8 +1,7 @@ module.exports = exports = gyp -var fs = require('graceful-fs') - , path = require('path') +var path = require('path') , nopt = require('nopt') , log = require('npmlog') , child_process = require('child_process') diff --git a/lib/remove.js b/lib/remove.js index 213af2b9d3..46a1a3d4c2 100644 --- a/lib/remove.js +++ b/lib/remove.js @@ -31,7 +31,7 @@ function remove (gyp, argv, callback) { log.verbose('remove', 'removing development files for version:', version) // first check if its even installed - fs.stat(versionPath, function (err, stat) { + fs.stat(versionPath, function (err) { if (err) { if (err.code == 'ENOENT') { callback(null, 'version was already uninstalled: ' + version) diff --git a/package.json b/package.json index d0a299ecf7..accb86b44b 100644 --- a/package.json +++ b/package.json @@ -38,12 +38,15 @@ "node": ">= 4.0.0" }, "devDependencies": { - "tape": "~4.2.0", + "babel-eslint": "^8.2.5", "bindings": "~1.2.1", + "eslint": "^5.0.1", "nan": "^2.0.0", - "require-inject": "~1.3.0" + "require-inject": "~1.3.0", + "tape": "~4.2.0" }, "scripts": { - "test": "tape test/test-*" + "lint": "eslint bin lib test", + "test": "npm run lint && tape test/test-*" } } diff --git a/test/test-configure-python.js b/test/test-configure-python.js index f235bdbba1..07cdce2b17 100644 --- a/test/test-configure-python.js +++ b/test/test-configure-python.js @@ -6,8 +6,8 @@ var gyp = require('../lib/node-gyp') var requireInject = require('require-inject') var configure = requireInject('../lib/configure', { 'graceful-fs': { - 'openSync': function (file, mode) { return 0; }, - 'closeSync': function (fd) { }, + 'openSync': function () { return 0; }, + 'closeSync': function () { }, 'writeFile': function (file, data, cb) { cb() }, 'stat': function (file, cb) { cb(null, {}) } } diff --git a/test/test-find-accessible-sync.js b/test/test-find-accessible-sync.js index d336243dd0..114403e278 100644 --- a/test/test-find-accessible-sync.js +++ b/test/test-find-accessible-sync.js @@ -5,7 +5,7 @@ var path = require('path') var requireInject = require('require-inject') var configure = requireInject('../lib/configure', { 'graceful-fs': { - 'closeSync': function (fd) { return undefined }, + 'closeSync': function () { return undefined }, 'openSync': function (path) { if (readableFiles.some(function (f) { return f === path} )) { return 0 diff --git a/test/test-find-python.js b/test/test-find-python.js index 570eb180de..250e37f4fe 100644 --- a/test/test-find-python.js +++ b/test/test-find-python.js @@ -88,7 +88,7 @@ test('find python - python too old', function (t) { } f.checkPython() - function done(err, python) { + function done(err) { t.ok(/is not supported by gyp/.test(err)) } }) @@ -108,7 +108,7 @@ test('find python - python too new', function (t) { } f.checkPython() - function done(err, python) { + function done(err) { t.ok(/is not supported by gyp/.test(err)) } }) @@ -123,7 +123,7 @@ test('find python - no python', function (t) { } f.checkPython() - function done(err, python) { + function done(err) { t.ok(/Can't find Python executable/.test(err)) } }) @@ -170,7 +170,7 @@ test('find python - no python2, no python, unix', function (t) { } f.checkPython() - function done(err, python) { + function done(err) { t.ok(/Can't find Python executable/.test(err)) } }) @@ -271,7 +271,7 @@ test('find python - python 3, use python launcher, python 2 too old', } f.checkPython() - function done(err, python) { + function done(err) { t.ok(/is not supported by gyp/.test(err)) } }) @@ -333,7 +333,7 @@ test('find python - no python, no python launcher, bad guess', function (t) { } f.checkPython() - function done(err, python) { + function done(err) { t.ok(/Can't find Python executable/.test(err)) } }) From 323cee732358846a7a31d822f2ff967e5f1e946b Mon Sep 17 00:00:00 2001 From: Refael Ackermann Date: Tue, 26 Sep 2017 16:45:11 -0400 Subject: [PATCH 004/551] deps: pin `request` version range Required for "node < 4" compatibility and is congruent with `npm` minimum for passing `npm test` >= 2.9.0 setting < 2.82.0 allows deduping PR-URL: https://github.com/nodejs/node-gyp/pull/1300 Fixes: https://github.com/nodejs/node-gyp/issues/1299 Reviewed-By: Ben Noordhuis Reviewed-By: Anna Henningsen --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index accb86b44b..76da24cca8 100644 --- a/package.json +++ b/package.json @@ -28,7 +28,7 @@ "nopt": "2 || 3", "npmlog": "0 || 1 || 2 || 3 || 4", "osenv": "0", - "request": "^2.87.0", + "request": ">=2.9.0 <2.82.0", "rimraf": "2", "semver": "~5.3.0", "tar": "^4.4.8", From f83b457e0306d61f7a5f31af799438be79a47c2b Mon Sep 17 00:00:00 2001 From: Rohit Hazra Date: Thu, 5 Jul 2018 00:03:15 +0530 Subject: [PATCH 005/551] deps: bump request to 2.8.7, fixes heok/hawk issues PR-URL: https://github.com/nodejs/node-gyp/pull/1492 Reviewed-By: Jeremiah Senkpiel Reviewed-By: Matheus Marchini Reviewed-By: Jon Moss Reviewed-By: Rod Vagg --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index 76da24cca8..accb86b44b 100644 --- a/package.json +++ b/package.json @@ -28,7 +28,7 @@ "nopt": "2 || 3", "npmlog": "0 || 1 || 2 || 3 || 4", "osenv": "0", - "request": ">=2.9.0 <2.82.0", + "request": "^2.87.0", "rimraf": "2", "semver": "~5.3.0", "tar": "^4.4.8", From 788e767179a4819d79923146ff2d6fc51d57c761 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Thu, 11 Oct 2018 02:39:17 +0100 Subject: [PATCH 006/551] test: remove unused variable This fixes linting. https://github.com/nodejs/node-gyp/pull/1561 Reviewed-By: Refael Ackermann --- test/process-exec-sync.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/process-exec-sync.js b/test/process-exec-sync.js index 859cbc1f6f..13daf308a3 100644 --- a/test/process-exec-sync.js +++ b/test/process-exec-sync.js @@ -10,7 +10,7 @@ if (!String.prototype.startsWith) { } function processExecSync(file, args, options) { - var child, error, timeout, tmpdir, command, quote + var child, error, timeout, tmpdir, command command = makeCommand(file, args) /* From abef93ded5fcbab9f69d162af33b68ee117b7f03 Mon Sep 17 00:00:00 2001 From: cclauss Date: Tue, 14 Nov 2017 09:17:20 +0100 Subject: [PATCH 007/551] gyp: get ready for python 3 https://github.com/nodejs/node-gyp/pull/1335 Reviewed-By: Refael Ackermann --- gyp/pylib/gyp/MSVSSettings.py | 21 ++--- gyp/pylib/gyp/MSVSVersion.py | 2 +- gyp/pylib/gyp/__init__.py | 17 ++-- gyp/pylib/gyp/common.py | 8 +- gyp/pylib/gyp/easy_xml.py | 1 + gyp/pylib/gyp/flock_tool.py | 2 +- gyp/pylib/gyp/generator/analyzer.py | 79 ++++++++++--------- gyp/pylib/gyp/generator/android.py | 7 +- gyp/pylib/gyp/generator/cmake.py | 11 +-- .../gyp/generator/dump_dependency_json.py | 3 +- gyp/pylib/gyp/generator/eclipse.py | 2 +- gyp/pylib/gyp/generator/make.py | 9 ++- gyp/pylib/gyp/generator/msvs.py | 13 +-- gyp/pylib/gyp/generator/ninja.py | 9 ++- gyp/pylib/gyp/generator/xcode.py | 13 +-- gyp/pylib/gyp/input.py | 29 +++---- gyp/pylib/gyp/mac_tool.py | 11 +-- gyp/pylib/gyp/win_tool.py | 11 +-- gyp/pylib/gyp/xcode_emulation.py | 15 ++-- gyp/pylib/gyp/xcode_ninja.py | 2 +- gyp/pylib/gyp/xcodeproj_file.py | 2 +- gyp/tools/graphviz.py | 29 +++---- gyp/tools/pretty_gyp.py | 11 +-- gyp/tools/pretty_sln.py | 47 +++++------ gyp/tools/pretty_vcproj.py | 15 ++-- test/fixtures/test-charmap.py | 7 +- 26 files changed, 198 insertions(+), 178 deletions(-) diff --git a/gyp/pylib/gyp/MSVSSettings.py b/gyp/pylib/gyp/MSVSSettings.py index a08cc154d7..94525ae1cc 100644 --- a/gyp/pylib/gyp/MSVSSettings.py +++ b/gyp/pylib/gyp/MSVSSettings.py @@ -13,6 +13,7 @@ The MSBuild schemas were also considered. They are typically found in the MSBuild install directory, e.g. c:\Program Files (x86)\MSBuild """ +from __future__ import print_function import sys import re @@ -400,7 +401,7 @@ def _ValidateExclusionSetting(setting, settings, error_msg, stderr=sys.stderr): if unrecognized: # We don't know this setting. Give a warning. - print >> stderr, error_msg + print(error_msg, file=stderr) def FixVCMacroSlashes(s): @@ -461,9 +462,9 @@ def ConvertToMSBuildSettings(msvs_settings, stderr=sys.stderr): # Invoke the translation function. try: msvs_tool[msvs_setting](msvs_value, msbuild_settings) - except ValueError, e: - print >> stderr, ('Warning: while converting %s/%s to MSBuild, ' - '%s' % (msvs_tool_name, msvs_setting, e)) + except ValueError as e: + print(('Warning: while converting %s/%s to MSBuild, ' + '%s' % (msvs_tool_name, msvs_setting, e)), file=stderr) else: _ValidateExclusionSetting(msvs_setting, msvs_tool, @@ -472,8 +473,8 @@ def ConvertToMSBuildSettings(msvs_settings, stderr=sys.stderr): (msvs_tool_name, msvs_setting)), stderr) else: - print >> stderr, ('Warning: unrecognized tool %s while converting to ' - 'MSBuild.' % msvs_tool_name) + print(('Warning: unrecognized tool %s while converting to ' + 'MSBuild.' % msvs_tool_name), file=stderr) return msbuild_settings @@ -517,9 +518,9 @@ def _ValidateSettings(validators, settings, stderr): if setting in tool_validators: try: tool_validators[setting](value) - except ValueError, e: - print >> stderr, ('Warning: for %s/%s, %s' % - (tool_name, setting, e)) + except ValueError as e: + print(('Warning: for %s/%s, %s' % + (tool_name, setting, e)), file=stderr) else: _ValidateExclusionSetting(setting, tool_validators, @@ -528,7 +529,7 @@ def _ValidateSettings(validators, settings, stderr): stderr) else: - print >> stderr, ('Warning: unrecognized tool %s' % tool_name) + print(('Warning: unrecognized tool %s' % tool_name), file=stderr) # MSVS and MBuild names of the tools. diff --git a/gyp/pylib/gyp/MSVSVersion.py b/gyp/pylib/gyp/MSVSVersion.py index d9bfa684fa..c31ff3e114 100644 --- a/gyp/pylib/gyp/MSVSVersion.py +++ b/gyp/pylib/gyp/MSVSVersion.py @@ -158,7 +158,7 @@ def _RegistryQuery(key, value=None): text = None try: text = _RegistryQueryBase('Sysnative', key, value) - except OSError, e: + except OSError as e: if e.errno == errno.ENOENT: text = _RegistryQueryBase('System32', key, value) else: diff --git a/gyp/pylib/gyp/__init__.py b/gyp/pylib/gyp/__init__.py index 668f38b60d..30fb7093ee 100755 --- a/gyp/pylib/gyp/__init__.py +++ b/gyp/pylib/gyp/__init__.py @@ -4,6 +4,7 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. +from __future__ import print_function import copy import gyp.input import optparse @@ -34,8 +35,8 @@ def DebugOutput(mode, message, *args): pass if args: message %= args - print '%s:%s:%d:%s %s' % (mode.upper(), os.path.basename(ctx[0]), - ctx[1], ctx[2], message) + print('%s:%s:%d:%s %s' % (mode.upper(), os.path.basename(ctx[0]), + ctx[1], ctx[2], message)) def FindBuildFiles(): extension = '.gyp' @@ -226,12 +227,12 @@ def Noop(value): (action == 'store_false' and not value)): flags.append(opt) elif options.use_environment and env_name: - print >>sys.stderr, ('Warning: environment regeneration unimplemented ' + print(('Warning: environment regeneration unimplemented ' 'for %s flag %r env_name %r' % (action, opt, - env_name)) + env_name)), file=sys.stderr) else: - print >>sys.stderr, ('Warning: regeneration unimplemented for action %r ' - 'flag %r' % (action, opt)) + print(('Warning: regeneration unimplemented for action %r ' + 'flag %r' % (action, opt)), file=sys.stderr) return flags @@ -475,7 +476,7 @@ def gyp_main(args): if home_dot_gyp != None: default_include = os.path.join(home_dot_gyp, 'include.gypi') if os.path.exists(default_include): - print 'Using overrides found in ' + default_include + print('Using overrides found in ' + default_include) includes.append(default_include) # Command-line --include files come after the default include. @@ -536,7 +537,7 @@ def gyp_main(args): def main(args): try: return gyp_main(args) - except GypError, e: + except GypError as e: sys.stderr.write("gyp: %s\n" % e) return 1 diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py index 501118796f..b7bae925f4 100644 --- a/gyp/pylib/gyp/common.py +++ b/gyp/pylib/gyp/common.py @@ -363,7 +363,7 @@ def close(self): same = False try: same = filecmp.cmp(self.tmp_path, filename, False) - except OSError, e: + except OSError as e: if e.errno != errno.ENOENT: raise @@ -382,9 +382,9 @@ def close(self): # # No way to get the umask without setting a new one? Set a safe one # and then set it back to the old value. - umask = os.umask(077) + umask = os.umask(0o77) os.umask(umask) - os.chmod(self.tmp_path, 0666 & ~umask) + os.chmod(self.tmp_path, 0o666 & ~umask) if sys.platform == 'win32' and os.path.exists(filename): # NOTE: on windows (but not cygwin) rename will not replace an # existing file, so it must be preceded with a remove. Sadly there @@ -464,7 +464,7 @@ def CopyTool(flavor, out_path): ''.join([source[0], '# Generated by gyp. Do not edit.\n'] + source[1:])) # Make file executable. - os.chmod(tool_path, 0755) + os.chmod(tool_path, 0o755) # From Alex Martelli, diff --git a/gyp/pylib/gyp/easy_xml.py b/gyp/pylib/gyp/easy_xml.py index 841f31f925..fd525a712b 100644 --- a/gyp/pylib/gyp/easy_xml.py +++ b/gyp/pylib/gyp/easy_xml.py @@ -5,6 +5,7 @@ import re import os import locale +from functools import reduce def XmlToString(content, encoding='utf-8', pretty=False): diff --git a/gyp/pylib/gyp/flock_tool.py b/gyp/pylib/gyp/flock_tool.py index b38d8660f7..81fb79d136 100755 --- a/gyp/pylib/gyp/flock_tool.py +++ b/gyp/pylib/gyp/flock_tool.py @@ -39,7 +39,7 @@ def ExecFlock(self, lockfile, *cmd_list): # where fcntl.flock(fd, LOCK_EX) always fails # with EBADF, that's why we use this F_SETLK # hack instead. - fd = os.open(lockfile, os.O_WRONLY|os.O_NOCTTY|os.O_CREAT, 0666) + fd = os.open(lockfile, os.O_WRONLY|os.O_NOCTTY|os.O_CREAT, 0o666) if sys.platform.startswith('aix'): # Python on AIX is compiled with LARGEFILE support, which changes the # struct size. diff --git a/gyp/pylib/gyp/generator/analyzer.py b/gyp/pylib/gyp/generator/analyzer.py index 921c1a6b71..ce7f83c932 100644 --- a/gyp/pylib/gyp/generator/analyzer.py +++ b/gyp/pylib/gyp/generator/analyzer.py @@ -61,6 +61,7 @@ directly supplied to gyp. OTOH if both "a.gyp" and "b.gyp" are supplied to gyp then the "all" target includes "b1" and "b2". """ +from __future__ import print_function import gyp.common import gyp.ninja_syntax as ninja_syntax @@ -155,7 +156,7 @@ def _AddSources(sources, base_path, base_path_components, result): continue result.append(base_path + source) if debug: - print 'AddSource', org_source, result[len(result) - 1] + print('AddSource', org_source, result[len(result) - 1]) def _ExtractSourcesFromAction(action, base_path, base_path_components, @@ -185,7 +186,7 @@ def _ExtractSources(target, target_dict, toplevel_dir): base_path += '/' if debug: - print 'ExtractSources', target, base_path + print('ExtractSources', target, base_path) results = [] if 'sources' in target_dict: @@ -278,7 +279,7 @@ def _WasBuildFileModified(build_file, data, files, toplevel_dir): the root of the source tree.""" if _ToLocalPath(toplevel_dir, _ToGypPath(build_file)) in files: if debug: - print 'gyp file modified', build_file + print('gyp file modified', build_file) return True # First element of included_files is the file itself. @@ -291,8 +292,8 @@ def _WasBuildFileModified(build_file, data, files, toplevel_dir): _ToGypPath(gyp.common.UnrelativePath(include_file, build_file)) if _ToLocalPath(toplevel_dir, rel_include_file) in files: if debug: - print 'included gyp file modified, gyp_file=', build_file, \ - 'included file=', rel_include_file + print('included gyp file modified, gyp_file=', build_file, \ + 'included file=', rel_include_file) return True return False @@ -373,7 +374,7 @@ def _GenerateTargets(data, target_list, target_dicts, toplevel_dir, files, # If a build file (or any of its included files) is modified we assume all # targets in the file are modified. if build_file_in_files[build_file]: - print 'matching target from modified build file', target_name + print('matching target from modified build file', target_name) target.match_status = MATCH_STATUS_MATCHES matching_targets.append(target) else: @@ -381,7 +382,7 @@ def _GenerateTargets(data, target_list, target_dicts, toplevel_dir, files, toplevel_dir) for source in sources: if _ToGypPath(os.path.normpath(source)) in files: - print 'target', target_name, 'matches', source + print('target', target_name, 'matches', source) target.match_status = MATCH_STATUS_MATCHES matching_targets.append(target) break @@ -433,7 +434,7 @@ def _DoesTargetDependOnMatchingTargets(target): for dep in target.deps: if _DoesTargetDependOnMatchingTargets(dep): target.match_status = MATCH_STATUS_MATCHES_BY_DEPENDENCY - print '\t', target.name, 'matches by dep', dep.name + print('\t', target.name, 'matches by dep', dep.name) return True target.match_status = MATCH_STATUS_DOESNT_MATCH return False @@ -445,7 +446,7 @@ def _GetTargetsDependingOnMatchingTargets(possible_targets): supplied as input to analyzer. possible_targets: targets to search from.""" found = [] - print 'Targets that matched by dependency:' + print('Targets that matched by dependency:') for target in possible_targets: if _DoesTargetDependOnMatchingTargets(target): found.append(target) @@ -484,12 +485,12 @@ def _AddCompileTargets(target, roots, add_if_no_ancestor, result): (add_if_no_ancestor or target.requires_build)) or (target.is_static_library and add_if_no_ancestor and not target.is_or_has_linked_ancestor)): - print '\t\tadding to compile targets', target.name, 'executable', \ + print('\t\tadding to compile targets', target.name, 'executable', \ target.is_executable, 'added_to_compile_targets', \ target.added_to_compile_targets, 'add_if_no_ancestor', \ add_if_no_ancestor, 'requires_build', target.requires_build, \ 'is_static_library', target.is_static_library, \ - 'is_or_has_linked_ancestor', target.is_or_has_linked_ancestor + 'is_or_has_linked_ancestor', target.is_or_has_linked_ancestor) result.add(target) target.added_to_compile_targets = True @@ -500,7 +501,7 @@ def _GetCompileTargets(matching_targets, supplied_targets): supplied_targets: set of targets supplied to analyzer to search from.""" result = set() for target in matching_targets: - print 'finding compile targets for match', target.name + print('finding compile targets for match', target.name) _AddCompileTargets(target, supplied_targets, True, result) return result @@ -508,46 +509,46 @@ def _GetCompileTargets(matching_targets, supplied_targets): def _WriteOutput(params, **values): """Writes the output, either to stdout or a file is specified.""" if 'error' in values: - print 'Error:', values['error'] + print('Error:', values['error']) if 'status' in values: - print values['status'] + print(values['status']) if 'targets' in values: values['targets'].sort() - print 'Supplied targets that depend on changed files:' + print('Supplied targets that depend on changed files:') for target in values['targets']: - print '\t', target + print('\t', target) if 'invalid_targets' in values: values['invalid_targets'].sort() - print 'The following targets were not found:' + print('The following targets were not found:') for target in values['invalid_targets']: - print '\t', target + print('\t', target) if 'build_targets' in values: values['build_targets'].sort() - print 'Targets that require a build:' + print('Targets that require a build:') for target in values['build_targets']: - print '\t', target + print('\t', target) if 'compile_targets' in values: values['compile_targets'].sort() - print 'Targets that need to be built:' + print('Targets that need to be built:') for target in values['compile_targets']: - print '\t', target + print('\t', target) if 'test_targets' in values: values['test_targets'].sort() - print 'Test targets:' + print('Test targets:') for target in values['test_targets']: - print '\t', target + print('\t', target) output_path = params.get('generator_flags', {}).get( 'analyzer_output_path', None) if not output_path: - print json.dumps(values) + print(json.dumps(values)) return try: f = open(output_path, 'w') f.write(json.dumps(values) + '\n') f.close() except IOError as e: - print 'Error writing to output file', output_path, str(e) + print('Error writing to output file', output_path, str(e)) def _WasGypIncludeFileModified(params, files): @@ -556,7 +557,7 @@ def _WasGypIncludeFileModified(params, files): if params['options'].includes: for include in params['options'].includes: if _ToGypPath(os.path.normpath(include)) in files: - print 'Include file modified, assuming all changed', include + print('Include file modified, assuming all changed', include) return True return False @@ -638,13 +639,13 @@ def find_matching_test_target_names(self): set(self._root_targets))] else: test_targets = [x for x in test_targets_no_all] - print 'supplied test_targets' + print('supplied test_targets') for target_name in self._test_target_names: - print '\t', target_name - print 'found test_targets' + print('\t', target_name) + print('found test_targets') for target in test_targets: - print '\t', target.name - print 'searching for matching test targets' + print('\t', target.name) + print('searching for matching test targets') matching_test_targets = _GetTargetsDependingOnMatchingTargets(test_targets) matching_test_targets_contains_all = (test_target_names_contains_all and set(matching_test_targets) & @@ -654,14 +655,14 @@ def find_matching_test_target_names(self): # 'all' is subsequentely added to the matching names below. matching_test_targets = [x for x in (set(matching_test_targets) & set(test_targets_no_all))] - print 'matched test_targets' + print('matched test_targets') for target in matching_test_targets: - print '\t', target.name + print('\t', target.name) matching_target_names = [gyp.common.ParseQualifiedTarget(target.name)[1] for target in matching_test_targets] if matching_test_targets_contains_all: matching_target_names.append('all') - print '\tall' + print('\tall') return matching_target_names def find_matching_compile_target_names(self): @@ -677,10 +678,10 @@ def find_matching_compile_target_names(self): if 'all' in self._supplied_target_names(): supplied_targets = [x for x in (set(supplied_targets) | set(self._root_targets))] - print 'Supplied test_targets & compile_targets' + print('Supplied test_targets & compile_targets') for target in supplied_targets: - print '\t', target.name - print 'Finding compile targets' + print('\t', target.name) + print('Finding compile targets') compile_targets = _GetCompileTargets(self._changed_targets, supplied_targets) return [gyp.common.ParseQualifiedTarget(target.name)[1] @@ -699,7 +700,7 @@ def GenerateOutput(target_list, target_dicts, data, params): toplevel_dir = _ToGypPath(os.path.abspath(params['options'].toplevel_dir)) if debug: - print 'toplevel_dir', toplevel_dir + print('toplevel_dir', toplevel_dir) if _WasGypIncludeFileModified(params, config.files): result_dict = { 'status': all_changed_string, diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py index 5b26cc785a..ed507fbd0d 100644 --- a/gyp/pylib/gyp/generator/android.py +++ b/gyp/pylib/gyp/generator/android.py @@ -1,3 +1,4 @@ +from __future__ import print_function # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. @@ -636,8 +637,8 @@ def ComputeOutputParts(self, spec): elif self.type == 'none': target_ext = '.stamp' elif self.type != 'executable': - print ("ERROR: What output file should be generated?", - "type", self.type, "target", target) + print(("ERROR: What output file should be generated?", + "type", self.type, "target", target)) if self.type != 'static_library' and self.type != 'shared_library': target_prefix = spec.get('product_prefix', target_prefix) @@ -956,7 +957,7 @@ def PerformBuild(data, configurations, params): env = dict(os.environ) env['ONE_SHOT_MAKEFILE'] = makefile arguments = ['make', '-C', os.environ['ANDROID_BUILD_TOP'], 'gyp_all_modules'] - print 'Building: %s' % arguments + print('Building: %s' % arguments) subprocess.check_call(arguments, env=env) diff --git a/gyp/pylib/gyp/generator/cmake.py b/gyp/pylib/gyp/generator/cmake.py index 17f5e6396c..6f901656ce 100644 --- a/gyp/pylib/gyp/generator/cmake.py +++ b/gyp/pylib/gyp/generator/cmake.py @@ -27,6 +27,7 @@ not be able to find the header file directories described in the generated CMakeLists.txt file. """ +from __future__ import print_function import multiprocessing import os @@ -863,8 +864,8 @@ def WriteTarget(namer, qualified_target, target_dicts, build_dir, config_to_use, default_product_ext = generator_default_variables['SHARED_LIB_SUFFIX'] elif target_type != 'executable': - print ('ERROR: What output file should be generated?', - 'type', target_type, 'target', target_name) + print(('ERROR: What output file should be generated?', + 'type', target_type, 'target', target_name)) product_prefix = spec.get('product_prefix', default_product_prefix) product_name = spec.get('product_name', default_product_name) @@ -1180,11 +1181,11 @@ def PerformBuild(data, configurations, params): output_dir, config_name)) arguments = ['cmake', '-G', 'Ninja'] - print 'Generating [%s]: %s' % (config_name, arguments) + print('Generating [%s]: %s' % (config_name, arguments)) subprocess.check_call(arguments, cwd=build_dir) arguments = ['ninja', '-C', build_dir] - print 'Building [%s]: %s' % (config_name, arguments) + print('Building [%s]: %s' % (config_name, arguments)) subprocess.check_call(arguments) @@ -1212,7 +1213,7 @@ def GenerateOutput(target_list, target_dicts, data, params): arglists.append((target_list, target_dicts, data, params, config_name)) pool.map(CallGenerateOutputForConfig, arglists) - except KeyboardInterrupt, e: + except KeyboardInterrupt as e: pool.terminate() raise e else: diff --git a/gyp/pylib/gyp/generator/dump_dependency_json.py b/gyp/pylib/gyp/generator/dump_dependency_json.py index 160eafe2ef..8e4f3168f3 100644 --- a/gyp/pylib/gyp/generator/dump_dependency_json.py +++ b/gyp/pylib/gyp/generator/dump_dependency_json.py @@ -1,3 +1,4 @@ +from __future__ import print_function # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. @@ -96,4 +97,4 @@ def GenerateOutput(target_list, target_dicts, data, params): f = open(filename, 'w') json.dump(edges, f) f.close() - print 'Wrote json to %s.' % filename + print('Wrote json to %s.' % filename) diff --git a/gyp/pylib/gyp/generator/eclipse.py b/gyp/pylib/gyp/generator/eclipse.py index 3544347b3b..b7c6aa951f 100644 --- a/gyp/pylib/gyp/generator/eclipse.py +++ b/gyp/pylib/gyp/generator/eclipse.py @@ -141,7 +141,7 @@ def GetAllIncludeDirectories(target_list, target_dicts, compiler_includes_list.append(include_dir) # Find standard gyp include dirs. - if config.has_key('include_dirs'): + if 'include_dirs' in config: include_dirs = config['include_dirs'] for shared_intermediate_dir in shared_intermediate_dirs: for include_dir in include_dirs: diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index fe801b77ce..8f7082ab45 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -1,3 +1,4 @@ +from __future__ import print_function # Copyright (c) 2013 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. @@ -1381,8 +1382,8 @@ def ComputeOutputBasename(self, spec): elif self.type == 'none': target = '%s.stamp' % target elif self.type != 'executable': - print ("ERROR: What output file should be generated?", - "type", self.type, "target", target) + print(("ERROR: What output file should be generated?", + "type", self.type, "target", target)) target_prefix = spec.get('product_prefix', target_prefix) target = spec.get('product_name', target) @@ -1638,7 +1639,7 @@ def WriteTarget(self, spec, configs, deps, link_deps, bundle_deps, self.WriteDoCmd([self.output_binary], deps, 'touch', part_of_all, postbuilds=postbuilds) else: - print "WARNING: no output for", self.type, target + print("WARNING: no output for", self.type, target) # Add an alias for each target (if there are any outputs). # Installable target aliases are created below. @@ -1992,7 +1993,7 @@ def PerformBuild(data, configurations, params): if options.toplevel_dir and options.toplevel_dir != '.': arguments += '-C', options.toplevel_dir arguments.append('BUILDTYPE=' + config) - print 'Building [%s]: %s' % (config, arguments) + print('Building [%s]: %s' % (config, arguments)) subprocess.check_call(arguments) diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index 3901ba9416..344cd1d18c 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -1,3 +1,4 @@ +from __future__ import print_function # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. @@ -755,8 +756,8 @@ def _Replace(match): # the VCProj but cause the same problem on the final command-line. Moving # the item to the end of the list does works, but that's only possible if # there's only one such item. Let's just warn the user. - print >> sys.stderr, ('Warning: MSVS may misinterpret the odd number of ' + - 'quotes in ' + s) + print(('Warning: MSVS may misinterpret the odd number of ' + + 'quotes in ' + s), file=sys.stderr) return s @@ -1949,7 +1950,7 @@ def PerformBuild(data, configurations, params): for config in configurations: arguments = [devenv, sln_path, '/Build', config] - print 'Building [%s]: %s' % (config, arguments) + print('Building [%s]: %s' % (config, arguments)) rtn = subprocess.check_call(arguments) @@ -2031,7 +2032,7 @@ def GenerateOutput(target_list, target_dicts, data, params): if generator_flags.get('msvs_error_on_missing_sources', False): raise GypError(error_message) else: - print >> sys.stdout, "Warning: " + error_message + print("Warning: " + error_message, file=sys.stdout) def _GenerateMSBuildFiltersFile(filters_path, source_files, @@ -2729,7 +2730,7 @@ def _GetMSBuildPropertySheets(configurations): props_specified = False for name, settings in sorted(configurations.iteritems()): configuration = _GetConfigurationCondition(name, settings) - if settings.has_key('msbuild_props'): + if 'msbuild_props' in settings: additional_props[configuration] = _FixPaths(settings['msbuild_props']) props_specified = True else: @@ -2782,7 +2783,7 @@ def _ConvertMSVSBuildAttributes(spec, config, build_file): elif a == 'ConfigurationType': msbuild_attributes[a] = _ConvertMSVSConfigurationType(msvs_attributes[a]) else: - print 'Warning: Do not know how to convert MSVS attribute ' + a + print('Warning: Do not know how to convert MSVS attribute ' + a) return msbuild_attributes diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py index a00573ebf2..1fc00e1f98 100644 --- a/gyp/pylib/gyp/generator/ninja.py +++ b/gyp/pylib/gyp/generator/ninja.py @@ -1,3 +1,4 @@ +from __future__ import print_function # Copyright (c) 2013 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. @@ -472,8 +473,8 @@ def WriteSpec(self, spec, config_name, generator_flags): if self.flavor != 'mac' or len(self.archs) == 1: link_deps += [self.GypPathToNinja(o) for o in obj_outputs] else: - print "Warning: Actions/rules writing object files don't work with " \ - "multiarch targets, dropping. (target %s)" % spec['target_name'] + print("Warning: Actions/rules writing object files don't work with " \ + "multiarch targets, dropping. (target %s)" % spec['target_name']) elif self.flavor == 'mac' and len(self.archs) > 1: link_deps = collections.defaultdict(list) @@ -2376,7 +2377,7 @@ def PerformBuild(data, configurations, params): for config in configurations: builddir = os.path.join(options.toplevel_dir, 'out', config) arguments = ['ninja', '-C', builddir] - print 'Building [%s]: %s' % (config, arguments) + print('Building [%s]: %s' % (config, arguments)) subprocess.check_call(arguments) @@ -2413,7 +2414,7 @@ def GenerateOutput(target_list, target_dicts, data, params): arglists.append( (target_list, target_dicts, data, params, config_name)) pool.map(CallGenerateOutputForConfig, arglists) - except KeyboardInterrupt, e: + except KeyboardInterrupt as e: pool.terminate() raise e else: diff --git a/gyp/pylib/gyp/generator/xcode.py b/gyp/pylib/gyp/generator/xcode.py index 0e3fb9301e..63f80d7657 100644 --- a/gyp/pylib/gyp/generator/xcode.py +++ b/gyp/pylib/gyp/generator/xcode.py @@ -1,3 +1,4 @@ +from __future__ import print_function # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. @@ -128,7 +129,7 @@ def __init__(self, gyp_path, path, build_file_dict): try: os.makedirs(self.path) self.created_dir = True - except OSError, e: + except OSError as e: if e.errno != errno.EEXIST: raise @@ -453,7 +454,7 @@ def Write(self): same = False try: same = filecmp.cmp(pbxproj_path, new_pbxproj_path, False) - except OSError, e: + except OSError as e: if e.errno != errno.ENOENT: raise @@ -472,10 +473,10 @@ def Write(self): # # No way to get the umask without setting a new one? Set a safe one # and then set it back to the old value. - umask = os.umask(077) + umask = os.umask(0o77) os.umask(umask) - os.chmod(new_pbxproj_path, 0666 & ~umask) + os.chmod(new_pbxproj_path, 0o666 & ~umask) os.rename(new_pbxproj_path, pbxproj_path) except Exception: @@ -576,7 +577,7 @@ def PerformBuild(data, configurations, params): for config in configurations: arguments = ['xcodebuild', '-project', xcodeproj_path] arguments += ['-configuration', config] - print "Building [%s]: %s" % (config, arguments) + print("Building [%s]: %s" % (config, arguments)) subprocess.check_call(arguments) @@ -736,7 +737,7 @@ def GenerateOutput(target_list, target_dicts, data, params): xctarget_type = gyp.xcodeproj_file.PBXNativeTarget try: target_properties['productType'] = _types[type_bundle_key] - except KeyError, e: + except KeyError as e: gyp.common.ExceptionAppend(e, "-- unknown product type while " "writing target %s" % target_name) raise diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index 10f6e0dba1..f55619f3b0 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -1,3 +1,4 @@ +from __future__ import print_function # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. @@ -249,10 +250,10 @@ def LoadOneBuildFile(build_file_path, data, aux_data, includes, else: build_file_data = eval(build_file_contents, {'__builtins__': None}, None) - except SyntaxError, e: + except SyntaxError as e: e.filename = build_file_path raise - except Exception, e: + except Exception as e: gyp.common.ExceptionAppend(e, 'while reading ' + build_file_path) raise @@ -272,7 +273,7 @@ def LoadOneBuildFile(build_file_path, data, aux_data, includes, else: LoadBuildFileIncludesIntoDict(build_file_data, build_file_path, data, aux_data, None, check) - except Exception, e: + except Exception as e: gyp.common.ExceptionAppend(e, 'while reading includes of ' + build_file_path) raise @@ -474,7 +475,7 @@ def LoadTargetBuildFile(build_file_path, data, aux_data, variables, includes, try: LoadTargetBuildFile(dependency, data, aux_data, variables, includes, depth, check, load_dependencies) - except Exception, e: + except Exception as e: gyp.common.ExceptionAppend( e, 'while loading dependencies of %s' % build_file_path) raise @@ -517,12 +518,12 @@ def CallLoadTargetBuildFile(global_flags, return (build_file_path, build_file_data, dependencies) - except GypError, e: + except GypError as e: sys.stderr.write("gyp: %s\n" % e) return None - except Exception, e: - print >>sys.stderr, 'Exception:', e - print >>sys.stderr, traceback.format_exc() + except Exception as e: + print('Exception:', e, file=sys.stderr) + print(traceback.format_exc(), file=sys.stderr) return None @@ -612,7 +613,7 @@ def LoadTargetBuildFilesParallel(build_files, data, variables, includes, depth, args = (global_flags, dependency, variables, includes, depth, check, generator_input_info), callback = parallel_state.LoadTargetBuildFileCallback) - except KeyboardInterrupt, e: + except KeyboardInterrupt as e: parallel_state.pool.terminate() raise e @@ -912,7 +913,7 @@ def ExpandVariables(input, phase, variables, build_file): stderr=subprocess.PIPE, stdin=subprocess.PIPE, cwd=build_file_dir) - except Exception, e: + except Exception as e: raise GypError("%s while executing command '%s' in %s" % (e, contents, build_file)) @@ -1097,13 +1098,13 @@ def EvalSingleCondition( if eval(ast_code, {'__builtins__': None}, variables): return true_dict return false_dict - except SyntaxError, e: + except SyntaxError as e: syntax_error = SyntaxError('%s while evaluating condition \'%s\' in %s ' 'at character %d.' % (str(e.args[0]), e.text, build_file, e.offset), e.filename, e.lineno, e.offset, e.text) raise syntax_error - except NameError, e: + except NameError as e: gyp.common.ExceptionAppend(e, 'while evaluating condition \'%s\' in %s' % (cond_expr_expanded, build_file)) raise GypError(e) @@ -1858,7 +1859,7 @@ def VerifyNoGYPFileCircularDependencies(targets): for dependency in target_dependencies: try: dependency_build_file = gyp.common.BuildFile(dependency) - except GypError, e: + except GypError as e: gyp.common.ExceptionAppend( e, 'while computing dependencies of .gyp file %s' % build_file) raise @@ -2781,7 +2782,7 @@ def Load(build_files, variables, includes, depth, generator_input_info, check, try: LoadTargetBuildFile(build_file, data, aux_data, variables, includes, depth, check, True) - except Exception, e: + except Exception as e: gyp.common.ExceptionAppend(e, 'while trying to load %s' % build_file) raise diff --git a/gyp/pylib/gyp/mac_tool.py b/gyp/pylib/gyp/mac_tool.py index eeeaceb0c7..36203801c8 100755 --- a/gyp/pylib/gyp/mac_tool.py +++ b/gyp/pylib/gyp/mac_tool.py @@ -7,6 +7,7 @@ These functions are executed via gyp-mac-tool when using the Makefile generator. """ +from __future__ import print_function import fcntl import fnmatch @@ -243,7 +244,7 @@ def ExecFilterLibtool(self, *cmd_list): _, err = libtoolout.communicate() for line in err.splitlines(): if not libtool_re.match(line) and not libtool_re5.match(line): - print >>sys.stderr, line + print(line, file=sys.stderr) # Unconditionally touch the output .a file on the command line if present # and the command succeeded. A bit hacky. if not libtoolout.returncode: @@ -440,8 +441,8 @@ def _FindProvisioningProfile(self, profile, bundle_identifier): profiles_dir = os.path.join( os.environ['HOME'], 'Library', 'MobileDevice', 'Provisioning Profiles') if not os.path.isdir(profiles_dir): - print >>sys.stderr, ( - 'cannot find mobile provisioning for %s' % bundle_identifier) + print(( + 'cannot find mobile provisioning for %s' % bundle_identifier), file=sys.stderr) sys.exit(1) provisioning_profiles = None if profile: @@ -462,8 +463,8 @@ def _FindProvisioningProfile(self, profile, bundle_identifier): valid_provisioning_profiles[app_id_pattern] = ( profile_path, profile_data, team_identifier) if not valid_provisioning_profiles: - print >>sys.stderr, ( - 'cannot find mobile provisioning for %s' % bundle_identifier) + print(( + 'cannot find mobile provisioning for %s' % bundle_identifier), file=sys.stderr) sys.exit(1) # If the user has multiple provisioning profiles installed that can be # used for ${bundle_identifier}, pick the most specific one (ie. the diff --git a/gyp/pylib/gyp/win_tool.py b/gyp/pylib/gyp/win_tool.py index bb6f1ea436..85a7926095 100755 --- a/gyp/pylib/gyp/win_tool.py +++ b/gyp/pylib/gyp/win_tool.py @@ -8,6 +8,7 @@ These functions are executed via gyp-win-tool when using the ninja generator. """ +from __future__ import print_function import os import re @@ -126,7 +127,7 @@ def ExecLinkWrapper(self, arch, use_separate_mspdbsrv, *args): if (not line.startswith(' Creating library ') and not line.startswith('Generating code') and not line.startswith('Finished generating code')): - print line + print(line) return link.returncode def ExecLinkWithManifests(self, arch, embed_manifest, out, ldcmd, resname, @@ -215,7 +216,7 @@ def ExecManifestWrapper(self, arch, *args): out, _ = popen.communicate() for line in out.splitlines(): if line and 'manifest authoring warning 81010002' not in line: - print line + print(line) return popen.returncode def ExecManifestToRc(self, arch, *args): @@ -255,7 +256,7 @@ def ExecMidlWrapper(self, arch, outdir, tlb, h, dlldata, iid, proxy, idl, for x in lines if x.startswith(prefixes)) for line in lines: if not line.startswith(prefixes) and line not in processing: - print line + print(line) return popen.returncode def ExecAsmWrapper(self, arch, *args): @@ -269,7 +270,7 @@ def ExecAsmWrapper(self, arch, *args): not line.startswith('Microsoft (R) Macro Assembler') and not line.startswith(' Assembling: ') and line): - print line + print(line) return popen.returncode def ExecRcWrapper(self, arch, *args): @@ -283,7 +284,7 @@ def ExecRcWrapper(self, arch, *args): if (not line.startswith('Microsoft (R) Windows (R) Resource Compiler') and not line.startswith('Copyright (C) Microsoft Corporation') and line): - print line + print(line) return popen.returncode def ExecActionWrapper(self, arch, rspfile, *dir): diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py index 69f7d97cfa..cbd589b43b 100644 --- a/gyp/pylib/gyp/xcode_emulation.py +++ b/gyp/pylib/gyp/xcode_emulation.py @@ -6,6 +6,7 @@ This module contains classes that help to emulate xcodebuild behavior on top of other build systems, such as make and ninja. """ +from __future__ import print_function import copy import gyp.common @@ -73,7 +74,7 @@ def _ExpandArchs(self, archs, sdkroot): if arch not in expanded_archs: expanded_archs.append(arch) except KeyError as e: - print 'Warning: Ignoring unsupported variable "%s".' % variable + print('Warning: Ignoring unsupported variable "%s".' % variable) elif arch not in expanded_archs: expanded_archs.append(arch) return expanded_archs @@ -194,8 +195,8 @@ def _ConvertConditionalKeys(self, configname): new_key = key.split("[")[0] settings[new_key] = settings[key] else: - print 'Warning: Conditional keys not implemented, ignoring:', \ - ' '.join(conditional_keys) + print('Warning: Conditional keys not implemented, ignoring:', \ + ' '.join(conditional_keys)) del settings[key] def _Settings(self): @@ -213,7 +214,7 @@ def _Appendf(self, lst, test_key, format_str, default=None): def _WarnUnimplemented(self, test_key): if test_key in self._Settings(): - print 'Warning: Ignoring not yet implemented key "%s".' % test_key + print('Warning: Ignoring not yet implemented key "%s".' % test_key) def IsBinaryOutputFormat(self, configname): default = "binary" if self.isIOS else "xml" @@ -997,8 +998,8 @@ def _GetIOSPostbuilds(self, configname, output_binary): unimpl = ['OTHER_CODE_SIGN_FLAGS'] unimpl = set(unimpl) & set(self.xcode_settings[configname].keys()) if unimpl: - print 'Warning: Some codesign keys not implemented, ignoring: %s' % ( - ', '.join(sorted(unimpl))) + print('Warning: Some codesign keys not implemented, ignoring: %s' % ( + ', '.join(sorted(unimpl)))) return ['%s code-sign-bundle "%s" "%s" "%s" "%s"' % ( os.path.join('${TARGET_BUILD_DIR}', 'gyp-mac-tool'), key, @@ -1597,7 +1598,7 @@ def GetEdges(node): order = gyp.common.TopologicallySorted(env.keys(), GetEdges) order.reverse() return order - except gyp.common.CycleError, e: + except gyp.common.CycleError as e: raise GypError( 'Xcode environment variables are cyclically dependent: ' + str(e.nodes)) diff --git a/gyp/pylib/gyp/xcode_ninja.py b/gyp/pylib/gyp/xcode_ninja.py index 3820d6bf04..d781471005 100644 --- a/gyp/pylib/gyp/xcode_ninja.py +++ b/gyp/pylib/gyp/xcode_ninja.py @@ -28,7 +28,7 @@ def _WriteWorkspace(main_gyp, sources_gyp, params): workspace_path = os.path.join(options.generator_output, workspace_path) try: os.makedirs(workspace_path) - except OSError, e: + except OSError as e: if e.errno != errno.EEXIST: raise output_string = '\n' + \ diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py index d08b7f7770..712eefe3f8 100644 --- a/gyp/pylib/gyp/xcodeproj_file.py +++ b/gyp/pylib/gyp/xcodeproj_file.py @@ -691,7 +691,7 @@ def _XCKVPrint(self, file, tabs, key, value): printable_value[0] == '"' and printable_value[-1] == '"': printable_value = printable_value[1:-1] printable += printable_key + ' = ' + printable_value + ';' + after_kv - except TypeError, e: + except TypeError as e: gyp.common.ExceptionAppend(e, 'while printing key "%s"' % key) raise diff --git a/gyp/tools/graphviz.py b/gyp/tools/graphviz.py index 326ae221cf..dcf7c765e5 100755 --- a/gyp/tools/graphviz.py +++ b/gyp/tools/graphviz.py @@ -7,6 +7,7 @@ """Using the JSON dumped by the dump-dependency-json generator, generate input suitable for graphviz to render a dependency graph of targets.""" +from __future__ import print_function import collections import json @@ -50,9 +51,9 @@ def WriteGraph(edges): build_file, target_name, toolset = ParseTarget(src) files[build_file].append(src) - print 'digraph D {' - print ' fontsize=8' # Used by subgraphs. - print ' node [fontsize=8]' + print('digraph D {') + print(' fontsize=8') # Used by subgraphs. + print(' node [fontsize=8]') # Output nodes by file. We must first write out each node within # its file grouping before writing out any edges that may refer @@ -63,31 +64,31 @@ def WriteGraph(edges): # the display by making it a box without an internal node. target = targets[0] build_file, target_name, toolset = ParseTarget(target) - print ' "%s" [shape=box, label="%s\\n%s"]' % (target, filename, - target_name) + print(' "%s" [shape=box, label="%s\\n%s"]' % (target, filename, + target_name)) else: # Group multiple nodes together in a subgraph. - print ' subgraph "cluster_%s" {' % filename - print ' label = "%s"' % filename + print(' subgraph "cluster_%s" {' % filename) + print(' label = "%s"' % filename) for target in targets: build_file, target_name, toolset = ParseTarget(target) - print ' "%s" [label="%s"]' % (target, target_name) - print ' }' + print(' "%s" [label="%s"]' % (target, target_name)) + print(' }') # Now that we've placed all the nodes within subgraphs, output all # the edges between nodes. for src, dsts in edges.items(): for dst in dsts: - print ' "%s" -> "%s"' % (src, dst) + print(' "%s" -> "%s"' % (src, dst)) - print '}' + print('}') def main(): if len(sys.argv) < 2: - print >>sys.stderr, __doc__ - print >>sys.stderr - print >>sys.stderr, 'usage: %s target1 target2...' % (sys.argv[0]) + print(__doc__, file=sys.stderr) + print(file=sys.stderr) + print('usage: %s target1 target2...' % (sys.argv[0]), file=sys.stderr) return 1 edges = LoadEdges('dump.json', sys.argv[1:]) diff --git a/gyp/tools/pretty_gyp.py b/gyp/tools/pretty_gyp.py index c51d35872c..8111224595 100755 --- a/gyp/tools/pretty_gyp.py +++ b/gyp/tools/pretty_gyp.py @@ -5,6 +5,7 @@ # found in the LICENSE file. """Pretty-prints the contents of a GYP file.""" +from __future__ import print_function import sys import re @@ -119,22 +120,22 @@ def prettyprint_input(lines): last_line = "" for line in lines: if COMMENT_RE.match(line): - print line + print(line) else: line = line.strip('\r\n\t ') # Otherwise doesn't strip \r on Unix. if len(line) > 0: (brace_diff, after) = count_braces(line) if brace_diff != 0: if after: - print " " * (basic_offset * indent) + line + print(" " * (basic_offset * indent) + line) indent += brace_diff else: indent += brace_diff - print " " * (basic_offset * indent) + line + print(" " * (basic_offset * indent) + line) else: - print " " * (basic_offset * indent) + line + print(" " * (basic_offset * indent) + line) else: - print "" + print("") last_line = line diff --git a/gyp/tools/pretty_sln.py b/gyp/tools/pretty_sln.py index ca8cf4ad3f..64a9ec2bf6 100755 --- a/gyp/tools/pretty_sln.py +++ b/gyp/tools/pretty_sln.py @@ -11,6 +11,7 @@ Then it outputs a possible build order. """ +from __future__ import print_function __author__ = 'nsylvain (Nicolas Sylvain)' @@ -26,7 +27,7 @@ def BuildProject(project, built, projects, deps): for dep in deps[project]: if dep not in built: BuildProject(dep, built, projects, deps) - print project + print(project) built.append(project) def ParseSolution(solution_file): @@ -100,44 +101,44 @@ def ParseSolution(solution_file): return (projects, dependencies) def PrintDependencies(projects, deps): - print "---------------------------------------" - print "Dependencies for all projects" - print "---------------------------------------" - print "-- --" + print("---------------------------------------") + print("Dependencies for all projects") + print("---------------------------------------") + print("-- --") for (project, dep_list) in sorted(deps.items()): - print "Project : %s" % project - print "Path : %s" % projects[project][0] + print("Project : %s" % project) + print("Path : %s" % projects[project][0]) if dep_list: for dep in dep_list: - print " - %s" % dep - print "" + print(" - %s" % dep) + print("") - print "-- --" + print("-- --") def PrintBuildOrder(projects, deps): - print "---------------------------------------" - print "Build order " - print "---------------------------------------" - print "-- --" + print("---------------------------------------") + print("Build order ") + print("---------------------------------------") + print("-- --") built = [] for (project, _) in sorted(deps.items()): if project not in built: BuildProject(project, built, projects, deps) - print "-- --" + print("-- --") def PrintVCProj(projects): for project in projects: - print "-------------------------------------" - print "-------------------------------------" - print project - print project - print project - print "-------------------------------------" - print "-------------------------------------" + print("-------------------------------------") + print("-------------------------------------") + print(project) + print(project) + print(project) + print("-------------------------------------") + print("-------------------------------------") project_path = os.path.abspath(os.path.join(os.path.dirname(sys.argv[1]), projects[project][2])) @@ -153,7 +154,7 @@ def PrintVCProj(projects): def main(): # check if we have exactly 1 parameter. if len(sys.argv) < 2: - print 'Usage: %s "c:\\path\\to\\project.sln"' % sys.argv[0] + print('Usage: %s "c:\\path\\to\\project.sln"' % sys.argv[0]) return 1 (projects, deps) = ParseSolution(sys.argv[1]) diff --git a/gyp/tools/pretty_vcproj.py b/gyp/tools/pretty_vcproj.py index 6099bd7cc4..c96ba061ef 100755 --- a/gyp/tools/pretty_vcproj.py +++ b/gyp/tools/pretty_vcproj.py @@ -11,6 +11,7 @@ It outputs the resulting xml to stdout. """ +from __future__ import print_function __author__ = 'nsylvain (Nicolas Sylvain)' @@ -61,7 +62,7 @@ def get_string(node): def PrettyPrintNode(node, indent=0): if node.nodeType == Node.TEXT_NODE: if node.data.strip(): - print '%s%s' % (' '*indent, node.data.strip()) + print('%s%s' % (' '*indent, node.data.strip())) return if node.childNodes: @@ -73,23 +74,23 @@ def PrettyPrintNode(node, indent=0): # Print the main tag if attr_count == 0: - print '%s<%s>' % (' '*indent, node.nodeName) + print('%s<%s>' % (' '*indent, node.nodeName)) else: - print '%s<%s' % (' '*indent, node.nodeName) + print('%s<%s' % (' '*indent, node.nodeName)) all_attributes = [] for (name, value) in node.attributes.items(): all_attributes.append((name, value)) all_attributes.sort(CmpTuple()) for (name, value) in all_attributes: - print '%s %s="%s"' % (' '*indent, name, value) - print '%s>' % (' '*indent) + print('%s %s="%s"' % (' '*indent, name, value)) + print('%s>' % (' '*indent)) if node.nodeValue: - print '%s %s' % (' '*indent, node.nodeValue) + print('%s %s' % (' '*indent, node.nodeValue)) for sub_node in node.childNodes: PrettyPrintNode(sub_node, indent=indent+2) - print '%s' % (' '*indent, node.nodeName) + print('%s' % (' '*indent, node.nodeName)) def FlattenFilter(node): diff --git a/test/fixtures/test-charmap.py b/test/fixtures/test-charmap.py index d9fa6fb25e..b752f0bbbf 100644 --- a/test/fixtures/test-charmap.py +++ b/test/fixtures/test-charmap.py @@ -1,3 +1,4 @@ +from __future__ import print_function import sys import locale @@ -14,9 +15,9 @@ def main(): 'cp1252': u'Lat\u012Bna', 'cp932': u'\u306b\u307b\u3093\u3054' } - if textmap.has_key(encoding): - print textmap[encoding] + if encoding in textmap: + print(textmap[encoding]) return True if __name__ == '__main__': - print main() + print(main()) From 2040cd21cc087bf25156fe39ce3b3e7eb97a205f Mon Sep 17 00:00:00 2001 From: Craig Rodrigues Date: Sun, 19 Mar 2017 13:26:41 -0700 Subject: [PATCH 008/551] gyp: use print as a function, as specified in PEP 3105. https://github.com/nodejs/node-gyp/pull/1150 Reviewed-By: Refael Ackermann --- gyp/pylib/gyp/MSVSSettings.py | 15 ++++++++------- gyp/pylib/gyp/__init__.py | 9 +++++---- gyp/pylib/gyp/generator/analyzer.py | 15 ++++++++------- gyp/pylib/gyp/generator/android.py | 15 ++++++++------- gyp/pylib/gyp/generator/cmake.py | 9 +++++---- gyp/pylib/gyp/generator/make.py | 12 ++++++------ gyp/pylib/gyp/generator/msvs.py | 13 +++++++------ gyp/pylib/gyp/input.py | 7 ++++--- gyp/pylib/gyp/mac_tool.py | 7 +++---- gyp/pylib/gyp/win_tool.py | 1 + gyp/pylib/gyp/xcode_emulation.py | 1 + gyp/tools/graphviz.py | 1 + gyp/tools/pretty_gyp.py | 1 + gyp/tools/pretty_sln.py | 3 ++- gyp/tools/pretty_vcproj.py | 8 ++++---- 15 files changed, 64 insertions(+), 53 deletions(-) diff --git a/gyp/pylib/gyp/MSVSSettings.py b/gyp/pylib/gyp/MSVSSettings.py index 94525ae1cc..455862c636 100644 --- a/gyp/pylib/gyp/MSVSSettings.py +++ b/gyp/pylib/gyp/MSVSSettings.py @@ -13,6 +13,7 @@ The MSBuild schemas were also considered. They are typically found in the MSBuild install directory, e.g. c:\Program Files (x86)\MSBuild """ + from __future__ import print_function import sys @@ -463,8 +464,8 @@ def ConvertToMSBuildSettings(msvs_settings, stderr=sys.stderr): try: msvs_tool[msvs_setting](msvs_value, msbuild_settings) except ValueError as e: - print(('Warning: while converting %s/%s to MSBuild, ' - '%s' % (msvs_tool_name, msvs_setting, e)), file=stderr) + print('Warning: while converting %s/%s to MSBuild, ' + '%s' % (msvs_tool_name, msvs_setting, e), file=stderr) else: _ValidateExclusionSetting(msvs_setting, msvs_tool, @@ -473,8 +474,8 @@ def ConvertToMSBuildSettings(msvs_settings, stderr=sys.stderr): (msvs_tool_name, msvs_setting)), stderr) else: - print(('Warning: unrecognized tool %s while converting to ' - 'MSBuild.' % msvs_tool_name), file=stderr) + print('Warning: unrecognized tool %s while converting to ' + 'MSBuild.' % msvs_tool_name, file=stderr) return msbuild_settings @@ -519,8 +520,8 @@ def _ValidateSettings(validators, settings, stderr): try: tool_validators[setting](value) except ValueError as e: - print(('Warning: for %s/%s, %s' % - (tool_name, setting, e)), file=stderr) + print('Warning: for %s/%s, %s' % + (tool_name, setting, e), file=stderr) else: _ValidateExclusionSetting(setting, tool_validators, @@ -529,7 +530,7 @@ def _ValidateSettings(validators, settings, stderr): stderr) else: - print(('Warning: unrecognized tool %s' % tool_name), file=stderr) + print('Warning: unrecognized tool %s' % (tool_name), file=stderr) # MSVS and MBuild names of the tools. diff --git a/gyp/pylib/gyp/__init__.py b/gyp/pylib/gyp/__init__.py index 30fb7093ee..bd51cde888 100755 --- a/gyp/pylib/gyp/__init__.py +++ b/gyp/pylib/gyp/__init__.py @@ -5,6 +5,7 @@ # found in the LICENSE file. from __future__ import print_function + import copy import gyp.input import optparse @@ -227,12 +228,12 @@ def Noop(value): (action == 'store_false' and not value)): flags.append(opt) elif options.use_environment and env_name: - print(('Warning: environment regeneration unimplemented ' + print('Warning: environment regeneration unimplemented ' 'for %s flag %r env_name %r' % (action, opt, - env_name)), file=sys.stderr) + env_name), file=sys.stderr) else: - print(('Warning: regeneration unimplemented for action %r ' - 'flag %r' % (action, opt)), file=sys.stderr) + print('Warning: regeneration unimplemented for action %r ' + 'flag %r' % (action, opt), file=sys.stderr) return flags diff --git a/gyp/pylib/gyp/generator/analyzer.py b/gyp/pylib/gyp/generator/analyzer.py index ce7f83c932..dc17c96524 100644 --- a/gyp/pylib/gyp/generator/analyzer.py +++ b/gyp/pylib/gyp/generator/analyzer.py @@ -61,6 +61,7 @@ directly supplied to gyp. OTOH if both "a.gyp" and "b.gyp" are supplied to gyp then the "all" target includes "b1" and "b2". """ + from __future__ import print_function import gyp.common @@ -292,8 +293,8 @@ def _WasBuildFileModified(build_file, data, files, toplevel_dir): _ToGypPath(gyp.common.UnrelativePath(include_file, build_file)) if _ToLocalPath(toplevel_dir, rel_include_file) in files: if debug: - print('included gyp file modified, gyp_file=', build_file, \ - 'included file=', rel_include_file) + print('included gyp file modified, gyp_file=', build_file, + 'included file=', rel_include_file) return True return False @@ -485,11 +486,11 @@ def _AddCompileTargets(target, roots, add_if_no_ancestor, result): (add_if_no_ancestor or target.requires_build)) or (target.is_static_library and add_if_no_ancestor and not target.is_or_has_linked_ancestor)): - print('\t\tadding to compile targets', target.name, 'executable', \ - target.is_executable, 'added_to_compile_targets', \ - target.added_to_compile_targets, 'add_if_no_ancestor', \ - add_if_no_ancestor, 'requires_build', target.requires_build, \ - 'is_static_library', target.is_static_library, \ + print('\t\tadding to compile targets', target.name, 'executable', + target.is_executable, 'added_to_compile_targets', + target.added_to_compile_targets, 'add_if_no_ancestor', + add_if_no_ancestor, 'requires_build', target.requires_build, + 'is_static_library', target.is_static_library, 'is_or_has_linked_ancestor', target.is_or_has_linked_ancestor) result.add(target) target.added_to_compile_targets = True diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py index ed507fbd0d..66139892cc 100644 --- a/gyp/pylib/gyp/generator/android.py +++ b/gyp/pylib/gyp/generator/android.py @@ -1,4 +1,3 @@ -from __future__ import print_function # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. @@ -15,6 +14,8 @@ # variables set potentially clash with other Android build system variables. # Try to avoid setting global variables where possible. +from __future__ import print_function + import gyp import gyp.common import gyp.generator.make as make # Reuse global functions from make backend. @@ -251,7 +252,7 @@ def WriteActions(self, actions, extra_sources, extra_outputs): dirs = set() for out in outputs: if not out.startswith('$'): - print ('WARNING: Action for target "%s" writes output to local path ' + print('WARNING: Action for target "%s" writes output to local path ' '"%s".' % (self.target, out)) dir = os.path.split(out)[0] if dir: @@ -356,7 +357,7 @@ def WriteRules(self, rules, extra_sources, extra_outputs): dirs = set() for out in outputs: if not out.startswith('$'): - print ('WARNING: Rule for target %s writes output to local path %s' + print('WARNING: Rule for target %s writes output to local path %s' % (self.target, out)) dir = os.path.dirname(out) if dir: @@ -430,7 +431,7 @@ def WriteCopies(self, copies, extra_outputs): # $(gyp_shared_intermediate_dir). Note that we can't use an assertion # because some of the gyp tests depend on this. if not copy['destination'].startswith('$'): - print ('WARNING: Copy rule for target %s writes output to ' + print('WARNING: Copy rule for target %s writes output to ' 'local path %s' % (self.target, copy['destination'])) # LocalPathify() calls normpath, stripping trailing slashes. @@ -637,8 +638,8 @@ def ComputeOutputParts(self, spec): elif self.type == 'none': target_ext = '.stamp' elif self.type != 'executable': - print(("ERROR: What output file should be generated?", - "type", self.type, "target", target)) + print("ERROR: What output file should be generated?", + "type", self.type, "target", target) if self.type != 'static_library' and self.type != 'shared_library': target_prefix = spec.get('product_prefix', target_prefix) @@ -1066,7 +1067,7 @@ def CalculateMakefilePath(build_file, base_name): write_alias_target=write_alias_targets, sdk_version=sdk_version) if android_module in android_modules: - print ('ERROR: Android module names must be unique. The following ' + print('ERROR: Android module names must be unique. The following ' 'targets both generate Android module name %s.\n %s\n %s' % (android_module, android_modules[android_module], qualified_target)) diff --git a/gyp/pylib/gyp/generator/cmake.py b/gyp/pylib/gyp/generator/cmake.py index 6f901656ce..7aabddb633 100644 --- a/gyp/pylib/gyp/generator/cmake.py +++ b/gyp/pylib/gyp/generator/cmake.py @@ -27,6 +27,7 @@ not be able to find the header file directories described in the generated CMakeLists.txt file. """ + from __future__ import print_function import multiprocessing @@ -640,8 +641,8 @@ def WriteTarget(namer, qualified_target, target_dicts, build_dir, config_to_use, cmake_target_type = cmake_target_type_from_gyp_target_type.get(target_type) if cmake_target_type is None: - print ('Target %s has unknown target type %s, skipping.' % - ( target_name, target_type ) ) + print('Target %s has unknown target type %s, skipping.' % + ( target_name, target_type )) return SetVariable(output, 'TARGET', target_name) @@ -864,8 +865,8 @@ def WriteTarget(namer, qualified_target, target_dicts, build_dir, config_to_use, default_product_ext = generator_default_variables['SHARED_LIB_SUFFIX'] elif target_type != 'executable': - print(('ERROR: What output file should be generated?', - 'type', target_type, 'target', target_name)) + print('ERROR: What output file should be generated?', + 'type', target_type, 'target', target_name) product_prefix = spec.get('product_prefix', default_product_prefix) product_name = spec.get('product_name', default_product_name) diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index 8f7082ab45..96212be1fc 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -1,4 +1,3 @@ -from __future__ import print_function # Copyright (c) 2013 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. @@ -22,6 +21,8 @@ # toplevel Makefile. It may make sense to generate some .mk files on # the side to keep the files readable. +from __future__ import print_function + import os import re import sys @@ -678,9 +679,8 @@ def _ValidateSourcesForOSX(spec, all_sources): error += ' %s: %s\n' % (basename, ' '.join(files)) if error: - print('static library %s has several files with the same basename:\n' % - spec['target_name'] + error + 'libtool on OS X will generate' + - ' warnings for them.') + print(('static library %s has several files with the same basename:\n' % spec['target_name']) + + error + 'libtool on OS X will generate' + ' warnings for them.') raise GypError('Duplicate basenames in sources section, see list above') @@ -1382,8 +1382,8 @@ def ComputeOutputBasename(self, spec): elif self.type == 'none': target = '%s.stamp' % target elif self.type != 'executable': - print(("ERROR: What output file should be generated?", - "type", self.type, "target", target)) + print("ERROR: What output file should be generated?", + "type", self.type, "target", target) target_prefix = spec.get('product_prefix', target_prefix) target = spec.get('product_name', target) diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index 344cd1d18c..128456b181 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -1,8 +1,9 @@ -from __future__ import print_function # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. +from __future__ import print_function + import copy import ntpath import os @@ -756,8 +757,8 @@ def _Replace(match): # the VCProj but cause the same problem on the final command-line. Moving # the item to the end of the list does works, but that's only possible if # there's only one such item. Let's just warn the user. - print(('Warning: MSVS may misinterpret the odd number of ' + - 'quotes in ' + s), file=sys.stderr) + print('Warning: MSVS may misinterpret the odd number of ' + + 'quotes in ' + s, file=sys.stderr) return s @@ -975,8 +976,8 @@ def _ValidateSourcesForMSVSProject(spec, version): error += ' %s: %s\n' % (basename, ' '.join(files)) if error: - print('static library %s has several files with the same basename:\n' % - spec['target_name'] + error + 'MSVC08 cannot handle that.') + print('static library %s has several files with the same basename:\n' % spec['target_name'] + + error + 'MSVC08 cannot handle that.') raise GypError('Duplicate basenames in sources section, see list above') @@ -3028,7 +3029,7 @@ def _FinalizeMSBuildSettings(spec, configuration): for ignored_setting in ignored_settings: value = configuration.get(ignored_setting) if value: - print ('Warning: The automatic conversion to MSBuild does not handle ' + print('Warning: The automatic conversion to MSBuild does not handle ' '%s. Ignoring setting of %s' % (ignored_setting, str(value))) defines = [_EscapeCppDefineForMSBuild(d) for d in defines] diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index f55619f3b0..5afbba03fe 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -1,8 +1,9 @@ -from __future__ import print_function # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. +from __future__ import print_function + from compiler.ast import Const from compiler.ast import Dict from compiler.ast import Discard @@ -2530,8 +2531,8 @@ def ValidateSourcesInTarget(target, target_dict, build_file, error += ' %s: %s\n' % (basename, ' '.join(files)) if error: - print('static library %s has several files with the same basename:\n' % - target + error + 'libtool on Mac cannot handle that. Use ' + print('static library %s has several files with the same basename:\n' % target + + error + 'libtool on Mac cannot handle that. Use ' '--no-duplicate-basename-check to disable this validation.') raise GypError('Duplicate basenames in sources section, see list above') diff --git a/gyp/pylib/gyp/mac_tool.py b/gyp/pylib/gyp/mac_tool.py index 36203801c8..0ab3de18f7 100755 --- a/gyp/pylib/gyp/mac_tool.py +++ b/gyp/pylib/gyp/mac_tool.py @@ -7,6 +7,7 @@ These functions are executed via gyp-mac-tool when using the Makefile generator. """ + from __future__ import print_function import fcntl @@ -441,8 +442,7 @@ def _FindProvisioningProfile(self, profile, bundle_identifier): profiles_dir = os.path.join( os.environ['HOME'], 'Library', 'MobileDevice', 'Provisioning Profiles') if not os.path.isdir(profiles_dir): - print(( - 'cannot find mobile provisioning for %s' % bundle_identifier), file=sys.stderr) + print('cannot find mobile provisioning for %s' % (bundle_identifier), file=sys.stderr) sys.exit(1) provisioning_profiles = None if profile: @@ -463,8 +463,7 @@ def _FindProvisioningProfile(self, profile, bundle_identifier): valid_provisioning_profiles[app_id_pattern] = ( profile_path, profile_data, team_identifier) if not valid_provisioning_profiles: - print(( - 'cannot find mobile provisioning for %s' % bundle_identifier), file=sys.stderr) + print('cannot find mobile provisioning for %s' % (bundle_identifier), file=sys.stderr) sys.exit(1) # If the user has multiple provisioning profiles installed that can be # used for ${bundle_identifier}, pick the most specific one (ie. the diff --git a/gyp/pylib/gyp/win_tool.py b/gyp/pylib/gyp/win_tool.py index 85a7926095..3bade00dbe 100755 --- a/gyp/pylib/gyp/win_tool.py +++ b/gyp/pylib/gyp/win_tool.py @@ -8,6 +8,7 @@ These functions are executed via gyp-win-tool when using the ninja generator. """ + from __future__ import print_function import os diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py index cbd589b43b..fd4427b86b 100644 --- a/gyp/pylib/gyp/xcode_emulation.py +++ b/gyp/pylib/gyp/xcode_emulation.py @@ -6,6 +6,7 @@ This module contains classes that help to emulate xcodebuild behavior on top of other build systems, such as make and ninja. """ + from __future__ import print_function import copy diff --git a/gyp/tools/graphviz.py b/gyp/tools/graphviz.py index dcf7c765e5..538b059da4 100755 --- a/gyp/tools/graphviz.py +++ b/gyp/tools/graphviz.py @@ -7,6 +7,7 @@ """Using the JSON dumped by the dump-dependency-json generator, generate input suitable for graphviz to render a dependency graph of targets.""" + from __future__ import print_function import collections diff --git a/gyp/tools/pretty_gyp.py b/gyp/tools/pretty_gyp.py index 8111224595..d01c692edc 100755 --- a/gyp/tools/pretty_gyp.py +++ b/gyp/tools/pretty_gyp.py @@ -5,6 +5,7 @@ # found in the LICENSE file. """Pretty-prints the contents of a GYP file.""" + from __future__ import print_function import sys diff --git a/gyp/tools/pretty_sln.py b/gyp/tools/pretty_sln.py index 64a9ec2bf6..731844caf7 100755 --- a/gyp/tools/pretty_sln.py +++ b/gyp/tools/pretty_sln.py @@ -11,10 +11,11 @@ Then it outputs a possible build order. """ -from __future__ import print_function __author__ = 'nsylvain (Nicolas Sylvain)' +from __future__ import print_function + import os import re import sys diff --git a/gyp/tools/pretty_vcproj.py b/gyp/tools/pretty_vcproj.py index c96ba061ef..51cfc4e081 100755 --- a/gyp/tools/pretty_vcproj.py +++ b/gyp/tools/pretty_vcproj.py @@ -11,10 +11,10 @@ It outputs the resulting xml to stdout. """ -from __future__ import print_function - __author__ = 'nsylvain (Nicolas Sylvain)' +from __future__ import print_function + import os import sys @@ -284,8 +284,8 @@ def main(argv): # check if we have exactly 1 parameter. if len(argv) < 2: - print ('Usage: %s "c:\\path\\to\\vcproj.vcproj" [key1=value1] ' - '[key2=value2]' % argv[0]) + print(('Usage: %s "c:\\path\\to\\vcproj.vcproj" [key1=value1] ' + '[key2=value2]' % argv[0])) return 1 # Parse the keys From 7535e4478ec916166a44e645173c3821c2575c74 Mon Sep 17 00:00:00 2001 From: Craig Rodrigues Date: Sun, 19 Mar 2017 16:36:13 -0400 Subject: [PATCH 009/551] gyp: replace deprecated functions * assertEquals() with assertEqual() * iteritems() with items() for Python 3. * xrange() with range() for Python 3. https://github.com/nodejs/node-gyp/pull/1150 Reviewed-By: Refael Ackermann --- gyp/PRESUBMIT.py | 3 +- gyp/pylib/gyp/MSVSSettings.py | 8 ++-- gyp/pylib/gyp/MSVSUserFile.py | 4 +- gyp/pylib/gyp/MSVSUtil.py | 2 +- gyp/pylib/gyp/__init__.py | 4 +- gyp/pylib/gyp/easy_xml.py | 2 +- gyp/pylib/gyp/generator/android.py | 6 +-- gyp/pylib/gyp/generator/gypd.py | 2 +- gyp/pylib/gyp/generator/make.py | 4 +- gyp/pylib/gyp/generator/msvs.py | 58 ++++++++++++++-------------- gyp/pylib/gyp/generator/ninja.py | 6 +-- gyp/pylib/gyp/generator/xcode.py | 20 +++++----- gyp/pylib/gyp/input.py | 62 +++++++++++++++--------------- gyp/pylib/gyp/input_test.py | 22 +++++------ gyp/pylib/gyp/mac_tool.py | 6 +-- gyp/pylib/gyp/msvs_emulation.py | 6 +-- gyp/pylib/gyp/ordered_dict.py | 4 +- gyp/pylib/gyp/simple_copy.py | 2 +- gyp/pylib/gyp/win_tool.py | 2 +- gyp/pylib/gyp/xcode_emulation.py | 6 +-- gyp/pylib/gyp/xcode_ninja.py | 4 +- gyp/pylib/gyp/xcodeproj_file.py | 26 ++++++------- 22 files changed, 129 insertions(+), 130 deletions(-) diff --git a/gyp/PRESUBMIT.py b/gyp/PRESUBMIT.py index f6c8a357af..e52f9d2d22 100644 --- a/gyp/PRESUBMIT.py +++ b/gyp/PRESUBMIT.py @@ -76,8 +76,7 @@ def _LicenseHeader(input_api): # Accept any year number from 2009 to the current year. current_year = int(input_api.time.strftime('%Y')) - allowed_years = (str(s) for s in reversed(xrange(2009, current_year + 1))) - + allowed_years = (str(s) for s in reversed(range(2009, current_year + 1))) years_re = '(' + '|'.join(allowed_years) + ')' # The (c) is deprecated, but tolerate it until it's removed from all files. diff --git a/gyp/pylib/gyp/MSVSSettings.py b/gyp/pylib/gyp/MSVSSettings.py index 455862c636..00721eb237 100644 --- a/gyp/pylib/gyp/MSVSSettings.py +++ b/gyp/pylib/gyp/MSVSSettings.py @@ -435,7 +435,7 @@ def ConvertVCMacrosToMSBuild(s): '$(PlatformName)': '$(Platform)', '$(SafeInputName)': '%(Filename)', } - for old, new in replace_map.iteritems(): + for old, new in replace_map.items(): s = s.replace(old, new) s = FixVCMacroSlashes(s) return s @@ -455,10 +455,10 @@ def ConvertToMSBuildSettings(msvs_settings, stderr=sys.stderr): dictionaries of settings and their values. """ msbuild_settings = {} - for msvs_tool_name, msvs_tool_settings in msvs_settings.iteritems(): + for msvs_tool_name, msvs_tool_settings in msvs_settings.items(): if msvs_tool_name in _msvs_to_msbuild_converters: msvs_tool = _msvs_to_msbuild_converters[msvs_tool_name] - for msvs_setting, msvs_value in msvs_tool_settings.iteritems(): + for msvs_setting, msvs_value in msvs_tool_settings.items(): if msvs_setting in msvs_tool: # Invoke the translation function. try: @@ -515,7 +515,7 @@ def _ValidateSettings(validators, settings, stderr): for tool_name in settings: if tool_name in validators: tool_validators = validators[tool_name] - for setting, value in settings[tool_name].iteritems(): + for setting, value in settings[tool_name].items(): if setting in tool_validators: try: tool_validators[setting](value) diff --git a/gyp/pylib/gyp/MSVSUserFile.py b/gyp/pylib/gyp/MSVSUserFile.py index 6c07e9a893..2264d64015 100644 --- a/gyp/pylib/gyp/MSVSUserFile.py +++ b/gyp/pylib/gyp/MSVSUserFile.py @@ -91,7 +91,7 @@ def AddDebugSettings(self, config_name, command, environment = {}, if environment and isinstance(environment, dict): env_list = ['%s="%s"' % (key, val) - for (key,val) in environment.iteritems()] + for (key,val) in environment.items()] environment = ' '.join(env_list) else: environment = '' @@ -135,7 +135,7 @@ def AddDebugSettings(self, config_name, command, environment = {}, def WriteIfChanged(self): """Writes the user file.""" configs = ['Configurations'] - for config, spec in sorted(self.configurations.iteritems()): + for config, spec in sorted(self.configurations.items()): configs.append(spec) content = ['VisualStudioUserFile', diff --git a/gyp/pylib/gyp/MSVSUtil.py b/gyp/pylib/gyp/MSVSUtil.py index 0b32e91180..c8187eb331 100644 --- a/gyp/pylib/gyp/MSVSUtil.py +++ b/gyp/pylib/gyp/MSVSUtil.py @@ -235,7 +235,7 @@ def InsertLargePdbShims(target_list, target_dicts, vars): # Set up the shim to output its PDB to the same location as the final linker # target. - for config_name, config in shim_dict.get('configurations').iteritems(): + for config_name, config in shim_dict.get('configurations').items(): pdb_path = _GetPdbPath(target_dict, config_name, vars) # A few keys that we don't want to propagate. diff --git a/gyp/pylib/gyp/__init__.py b/gyp/pylib/gyp/__init__.py index bd51cde888..5f82f2ef2c 100755 --- a/gyp/pylib/gyp/__init__.py +++ b/gyp/pylib/gyp/__init__.py @@ -209,7 +209,7 @@ def Noop(value): # We always want to ignore the environment when regenerating, to avoid # duplicate or changed flags in the environment at the time of regeneration. flags = ['--ignore-environment'] - for name, metadata in options._regeneration_metadata.iteritems(): + for name, metadata in options._regeneration_metadata.items(): opt = metadata['opt'] value = getattr(options, name) value_predicate = metadata['type'] == 'path' and FixPath or Noop @@ -434,7 +434,7 @@ def gyp_main(args): build_file_dir = os.path.abspath(os.path.dirname(build_file)) build_file_dir_components = build_file_dir.split(os.path.sep) components_len = len(build_file_dir_components) - for index in xrange(components_len - 1, -1, -1): + for index in range(components_len - 1, -1, -1): if build_file_dir_components[index] == 'src': options.depth = os.path.sep.join(build_file_dir_components) break diff --git a/gyp/pylib/gyp/easy_xml.py b/gyp/pylib/gyp/easy_xml.py index fd525a712b..7c3f621f1f 100644 --- a/gyp/pylib/gyp/easy_xml.py +++ b/gyp/pylib/gyp/easy_xml.py @@ -81,7 +81,7 @@ def _ConstructContentList(xml_parts, specification, pretty, level=0): # Optionally in second position is a dictionary of the attributes. rest = specification[1:] if rest and isinstance(rest[0], dict): - for at, val in sorted(rest[0].iteritems()): + for at, val in sorted(rest[0].items()): xml_parts.append(' %s="%s"' % (at, _XmlEscape(val, attr=True))) rest = rest[1:] if rest: diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py index 66139892cc..b7f9842888 100644 --- a/gyp/pylib/gyp/generator/android.py +++ b/gyp/pylib/gyp/generator/android.py @@ -460,7 +460,7 @@ def WriteSourceFlags(self, spec, configs): Args: spec, configs: input from gyp. """ - for configname, config in sorted(configs.iteritems()): + for configname, config in sorted(configs.items()): extracted_includes = [] self.WriteLn('\n# Flags passed to both C and C++ files.') @@ -790,7 +790,7 @@ def WriteTargetFlags(self, spec, configs, link_deps): static_libs, dynamic_libs, ldflags_libs = self.FilterLibraries(libraries) if self.type != 'static_library': - for configname, config in sorted(configs.iteritems()): + for configname, config in sorted(configs.items()): ldflags = list(config.get('ldflags', [])) self.WriteLn('') self.WriteList(ldflags, 'LOCAL_LDFLAGS_%s' % configname) @@ -839,7 +839,7 @@ def WriteTarget(self, spec, configs, deps, link_deps, part_of_all, settings = spec.get('aosp_build_settings', {}) if settings: self.WriteLn('### Set directly by aosp_build_settings.') - for k, v in settings.iteritems(): + for k, v in settings.items(): if isinstance(v, list): self.WriteList(v, k) else: diff --git a/gyp/pylib/gyp/generator/gypd.py b/gyp/pylib/gyp/generator/gypd.py index 3efdb9966a..78eeaa61b2 100644 --- a/gyp/pylib/gyp/generator/gypd.py +++ b/gyp/pylib/gyp/generator/gypd.py @@ -88,7 +88,7 @@ def GenerateOutput(target_list, target_dicts, data, params): if not output_file in output_files: output_files[output_file] = input_file - for output_file, input_file in output_files.iteritems(): + for output_file, input_file in output_files.items(): output = open(output_file, 'w') pprint.pprint(data[input_file], output) output.close() diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index 96212be1fc..cfc7b05433 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -674,7 +674,7 @@ def _ValidateSourcesForOSX(spec, all_sources): basenames.setdefault(basename, []).append(source) error = '' - for basename, files in basenames.iteritems(): + for basename, files in basenames.items(): if len(files) > 1: error += ' %s: %s\n' % (basename, ' '.join(files)) @@ -1547,7 +1547,7 @@ def WriteTarget(self, spec, configs, deps, link_deps, bundle_deps, # Postbuilds expect to be run in the gyp file's directory, so insert an # implicit postbuild to cd to there. postbuilds.insert(0, gyp.common.EncodePOSIXShellList(['cd', self.path])) - for i in xrange(len(postbuilds)): + for i in range(len(postbuilds)): if not postbuilds[i].startswith('$'): postbuilds[i] = EscapeShellArgument(postbuilds[i]) self.WriteLn('%s: builddir := $(abs_builddir)' % QuoteSpaces(self.output)) diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index 128456b181..fb2549d025 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -451,7 +451,7 @@ def _AddCustomBuildToolForMSVS(p, spec, primary_input, 'CommandLine': cmd, }) # Add to the properties of primary input for each config. - for config_name, c_data in spec['configurations'].iteritems(): + for config_name, c_data in spec['configurations'].items(): p.AddFileConfig(_FixPath(primary_input), _ConfigFullName(config_name, c_data), tools=[tool]) @@ -971,7 +971,7 @@ def _ValidateSourcesForMSVSProject(spec, version): basenames.setdefault(basename, []).append(source) error = '' - for basename, files in basenames.iteritems(): + for basename, files in basenames.items(): if len(files) > 1: error += ' %s: %s\n' % (basename, ' '.join(files)) @@ -1003,7 +1003,7 @@ def _GenerateMSVSProject(project, options, version, generator_flags): relative_path_of_gyp_file = gyp.common.RelativePath(gyp_path, project_dir) config_type = _GetMSVSConfigurationType(spec, project.build_file) - for config_name, config in spec['configurations'].iteritems(): + for config_name, config in spec['configurations'].items(): _AddConfigurationToMSVSProject(p, spec, config_type, config_name, config) # MSVC08 and prior version cannot handle duplicate basenames in the same @@ -1369,10 +1369,10 @@ def _ConvertToolsToExpectedForm(tools): A list of Tool objects. """ tool_list = [] - for tool, settings in tools.iteritems(): + for tool, settings in tools.items(): # Collapse settings with lists. settings_fixed = {} - for setting, value in settings.iteritems(): + for setting, value in settings.items(): if type(value) == list: if ((tool == 'VCLinkerTool' and setting == 'AdditionalDependencies') or @@ -1547,7 +1547,7 @@ def _IdlFilesHandledNonNatively(spec, sources): def _GetPrecompileRelatedFiles(spec): # Gather a list of precompiled header related sources. precompiled_related = [] - for _, config in spec['configurations'].iteritems(): + for _, config in spec['configurations'].items(): for k in precomp_keys: f = config.get(k) if f: @@ -1558,7 +1558,7 @@ def _GetPrecompileRelatedFiles(spec): def _ExcludeFilesFromBeingBuilt(p, spec, excluded_sources, excluded_idl, list_excluded): exclusions = _GetExcludedFilesFromBuild(spec, excluded_sources, excluded_idl) - for file_name, excluded_configs in exclusions.iteritems(): + for file_name, excluded_configs in exclusions.items(): if (not list_excluded and len(excluded_configs) == len(spec['configurations'])): # If we're not listing excluded files, then they won't appear in the @@ -1575,7 +1575,7 @@ def _GetExcludedFilesFromBuild(spec, excluded_sources, excluded_idl): # Exclude excluded sources from being built. for f in excluded_sources: excluded_configs = [] - for config_name, config in spec['configurations'].iteritems(): + for config_name, config in spec['configurations'].items(): precomped = [_FixPath(config.get(i, '')) for i in precomp_keys] # Don't do this for ones that are precompiled header related. if f not in precomped: @@ -1585,7 +1585,7 @@ def _GetExcludedFilesFromBuild(spec, excluded_sources, excluded_idl): # Exclude them now. for f in excluded_idl: excluded_configs = [] - for config_name, config in spec['configurations'].iteritems(): + for config_name, config in spec['configurations'].items(): excluded_configs.append((config_name, config)) exclusions[f] = excluded_configs return exclusions @@ -1594,7 +1594,7 @@ def _GetExcludedFilesFromBuild(spec, excluded_sources, excluded_idl): def _AddToolFilesToMSVS(p, spec): # Add in tool files (rules). tool_files = OrderedSet() - for _, config in spec['configurations'].iteritems(): + for _, config in spec['configurations'].items(): for f in config.get('msvs_tool_files', []): tool_files.add(f) for f in tool_files: @@ -1607,7 +1607,7 @@ def _HandlePreCompiledHeaders(p, sources, spec): # kind (i.e. C vs. C++) as the precompiled header source stub needs # to have use of precompiled headers disabled. extensions_excluded_from_precompile = [] - for config_name, config in spec['configurations'].iteritems(): + for config_name, config in spec['configurations'].items(): source = config.get('msvs_precompiled_source') if source: source = _FixPath(source) @@ -1628,7 +1628,7 @@ def DisableForSourceTree(source_tree): else: basename, extension = os.path.splitext(source) if extension in extensions_excluded_from_precompile: - for config_name, config in spec['configurations'].iteritems(): + for config_name, config in spec['configurations'].items(): tool = MSVSProject.Tool('VCCLCompilerTool', {'UsePrecompiledHeader': '0', 'ForcedIncludeFiles': '$(NOINHERIT)'}) @@ -1679,7 +1679,7 @@ def _WriteMSVSUserFile(project_path, version, spec): return # Nothing to add # Write out the user file. user_file = _CreateMSVSUserFile(project_path, version, spec) - for config_name, c_data in spec['configurations'].iteritems(): + for config_name, c_data in spec['configurations'].items(): user_file.AddDebugSettings(_ConfigFullName(config_name, c_data), action, environment, working_directory) user_file.WriteIfChanged() @@ -1730,7 +1730,7 @@ def _GetPathDict(root, path): def _DictsToFolders(base_path, bucket, flat): # Convert to folders recursively. children = [] - for folder, contents in bucket.iteritems(): + for folder, contents in bucket.items(): if type(contents) == dict: folder_children = _DictsToFolders(os.path.join(base_path, folder), contents, flat) @@ -1802,7 +1802,7 @@ def _GetPlatformOverridesOfProject(spec): # Prepare a dict indicating which project configurations are used for which # solution configurations for this target. config_platform_overrides = {} - for config_name, c in spec['configurations'].iteritems(): + for config_name, c in spec['configurations'].items(): config_fullname = _ConfigFullName(config_name, c) platform = c.get('msvs_target_platform', _ConfigPlatform(c)) fixed_config_fullname = '%s|%s' % ( @@ -1941,7 +1941,7 @@ def PerformBuild(data, configurations, params): msvs_version = params['msvs_version'] devenv = os.path.join(msvs_version.path, 'Common7', 'IDE', 'devenv.com') - for build_file, build_file_dict in data.iteritems(): + for build_file, build_file_dict in data.items(): (build_file_root, build_file_ext) = os.path.splitext(build_file) if build_file_ext != '.gyp': continue @@ -1990,7 +1990,7 @@ def GenerateOutput(target_list, target_dicts, data, params): configs = set() for qualified_target in target_list: spec = target_dicts[qualified_target] - for config_name, config in spec['configurations'].iteritems(): + for config_name, config in spec['configurations'].items(): configs.add(_ConfigFullName(config_name, config)) configs = list(configs) @@ -2630,7 +2630,7 @@ def _GetConfigurationCondition(name, settings): def _GetMSBuildProjectConfigurations(configurations): group = ['ItemGroup', {'Label': 'ProjectConfigurations'}] - for (name, settings) in sorted(configurations.iteritems()): + for (name, settings) in sorted(configurations.items()): configuration, platform = _GetConfigurationAndPlatform(name, settings) designation = '%s|%s' % (configuration, platform) group.append( @@ -2700,7 +2700,7 @@ def _GetMSBuildGlobalProperties(spec, guid, gyp_file_name): def _GetMSBuildConfigurationDetails(spec, build_file): properties = {} - for name, settings in spec['configurations'].iteritems(): + for name, settings in spec['configurations'].items(): msbuild_attributes = _GetMSBuildAttributes(spec, settings, build_file) condition = _GetConfigurationCondition(name, settings) character_set = msbuild_attributes.get('CharacterSet') @@ -2729,7 +2729,7 @@ def _GetMSBuildPropertySheets(configurations): user_props = r'$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props' additional_props = {} props_specified = False - for name, settings in sorted(configurations.iteritems()): + for name, settings in sorted(configurations.items()): configuration = _GetConfigurationCondition(name, settings) if 'msbuild_props' in settings: additional_props[configuration] = _FixPaths(settings['msbuild_props']) @@ -2751,7 +2751,7 @@ def _GetMSBuildPropertySheets(configurations): ] else: sheets = [] - for condition, props in additional_props.iteritems(): + for condition, props in additional_props.items(): import_group = [ 'ImportGroup', {'Label': 'PropertySheets', @@ -2878,7 +2878,7 @@ def _GetMSBuildConfigurationGlobalProperties(spec, configurations, build_file): new_paths = '$(ExecutablePath);' + ';'.join(new_paths) properties = {} - for (name, configuration) in sorted(configurations.iteritems()): + for (name, configuration) in sorted(configurations.items()): condition = _GetConfigurationCondition(name, configuration) attributes = _GetMSBuildAttributes(spec, configuration, build_file) msbuild_settings = configuration['finalized_msbuild_settings'] @@ -2903,7 +2903,7 @@ def _GetMSBuildConfigurationGlobalProperties(spec, configurations, build_file): _AddConditionalProperty(properties, condition, 'ExecutablePath', new_paths) tool_settings = msbuild_settings.get('', {}) - for name, value in sorted(tool_settings.iteritems()): + for name, value in sorted(tool_settings.items()): formatted_value = _GetValueFormattedForMSBuild('', name, value) _AddConditionalProperty(properties, condition, name, formatted_value) return _GetMSBuildPropertyGroup(spec, None, properties) @@ -2972,7 +2972,7 @@ def GetEdges(node): # NOTE: reverse(topsort(DAG)) = topsort(reverse_edges(DAG)) for name in reversed(properties_ordered): values = properties[name] - for value, conditions in sorted(values.iteritems()): + for value, conditions in sorted(values.items()): if len(conditions) == num_configurations: # If the value is the same all configurations, # just add one unconditional entry. @@ -2985,18 +2985,18 @@ def GetEdges(node): def _GetMSBuildToolSettingsSections(spec, configurations): groups = [] - for (name, configuration) in sorted(configurations.iteritems()): + for (name, configuration) in sorted(configurations.items()): msbuild_settings = configuration['finalized_msbuild_settings'] group = ['ItemDefinitionGroup', {'Condition': _GetConfigurationCondition(name, configuration)} ] - for tool_name, tool_settings in sorted(msbuild_settings.iteritems()): + for tool_name, tool_settings in sorted(msbuild_settings.items()): # Skip the tool named '' which is a holder of global settings handled # by _GetMSBuildConfigurationGlobalProperties. if tool_name: if tool_settings: tool = [tool_name] - for name, value in sorted(tool_settings.iteritems()): + for name, value in sorted(tool_settings.items()): formatted_value = _GetValueFormattedForMSBuild(tool_name, name, value) tool.append([name, formatted_value]) @@ -3196,7 +3196,7 @@ def _AddSources2(spec, sources, exclusions, grouped_sources, {'Condition': condition}, 'true']) # Add precompile if needed - for config_name, configuration in spec['configurations'].iteritems(): + for config_name, configuration in spec['configurations'].items(): precompiled_source = configuration.get('msvs_precompiled_source', '') if precompiled_source != '': precompiled_source = _FixPath(precompiled_source) @@ -3436,7 +3436,7 @@ def _GenerateActionsForMSBuild(spec, actions_to_add): """ sources_handled_by_action = OrderedSet() actions_spec = [] - for primary_input, actions in actions_to_add.iteritems(): + for primary_input, actions in actions_to_add.items(): inputs = OrderedSet() outputs = OrderedSet() descriptions = [] diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py index 1fc00e1f98..aae5b71310 100644 --- a/gyp/pylib/gyp/generator/ninja.py +++ b/gyp/pylib/gyp/generator/ninja.py @@ -797,7 +797,7 @@ def WriteMacXCassets(self, xcassets, bundle_depends): 'XCASSETS_LAUNCH_IMAGE': 'launch-image', } settings = self.xcode_settings.xcode_settings[self.config_name] - for settings_key, arg_name in settings_to_arg.iteritems(): + for settings_key, arg_name in settings_to_arg.items(): value = settings.get(settings_key) if value: extra_arguments[arg_name] = value @@ -1890,7 +1890,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, wrappers[key[:-len('_wrapper')]] = os.path.join(build_to_root, value) # Support wrappers from environment variables too. - for key, value in os.environ.iteritems(): + for key, value in os.environ.items(): if key.lower().endswith('_wrapper'): key_prefix = key[:-len('_wrapper')] key_prefix = re.sub(r'\.HOST$', '.host', key_prefix) @@ -1906,7 +1906,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, configs, generator_flags) cl_paths = gyp.msvs_emulation.GenerateEnvironmentFiles( toplevel_build, generator_flags, shared_system_includes, OpenOutput) - for arch, path in cl_paths.iteritems(): + for arch, path in cl_paths.items(): if clang_cl: # If we have selected clang-cl, use that instead. path = clang_cl diff --git a/gyp/pylib/gyp/generator/xcode.py b/gyp/pylib/gyp/generator/xcode.py index 63f80d7657..694a28afb1 100644 --- a/gyp/pylib/gyp/generator/xcode.py +++ b/gyp/pylib/gyp/generator/xcode.py @@ -183,7 +183,7 @@ def Finalize1(self, xcode_targets, serialize_all_tests): # the tree tree view for UI display. # Any values set globally are applied to all configurations, then any # per-configuration values are applied. - for xck, xcv in self.build_file_dict.get('xcode_settings', {}).iteritems(): + for xck, xcv in self.build_file_dict.get('xcode_settings', {}).items(): xccl.SetBuildSetting(xck, xcv) if 'xcode_config_file' in self.build_file_dict: config_ref = self.project.AddOrGetFileInRootGroup( @@ -197,7 +197,7 @@ def Finalize1(self, xcode_targets, serialize_all_tests): if build_file_configuration_named: xcc = xccl.ConfigurationNamed(config_name) for xck, xcv in build_file_configuration_named.get('xcode_settings', - {}).iteritems(): + {}).items(): xcc.SetBuildSetting(xck, xcv) if 'xcode_config_file' in build_file_configuration_named: config_ref = self.project.AddOrGetFileInRootGroup( @@ -273,7 +273,7 @@ def Finalize1(self, xcode_targets, serialize_all_tests): script = script + "\n".join( ['export %s="%s"' % (key, gyp.xcodeproj_file.ConvertVariablesToShellSyntax(val)) - for (key, val) in command.get('environment').iteritems()]) + "\n" + for (key, val) in command.get('environment').items()]) + "\n" # Some test end up using sockets, files on disk, etc. and can get # confused if more then one test runs at a time. The generator @@ -566,7 +566,7 @@ def EscapeXcodeDefine(s): def PerformBuild(data, configurations, params): options = params['options'] - for build_file, build_file_dict in data.iteritems(): + for build_file, build_file_dict in data.items(): (build_file_root, build_file_ext) = os.path.splitext(build_file) if build_file_ext != '.gyp': continue @@ -625,7 +625,7 @@ def GenerateOutput(target_list, target_dicts, data, params): skip_excluded_files = \ not generator_flags.get('xcode_list_excluded_files', True) xcode_projects = {} - for build_file, build_file_dict in data.iteritems(): + for build_file, build_file_dict in data.items(): (build_file_root, build_file_ext) = os.path.splitext(build_file) if build_file_ext != '.gyp': continue @@ -1014,7 +1014,7 @@ def GenerateOutput(target_list, target_dicts, data, params): # target. makefile.write('all: \\\n') for concrete_output_index in \ - xrange(0, len(concrete_outputs_by_rule_source)): + range(0, len(concrete_outputs_by_rule_source)): # Only list the first (index [0]) concrete output of each input # in the "all" target. Otherwise, a parallel make (-j > 1) would # attempt to process each input multiple times simultaneously. @@ -1037,7 +1037,7 @@ def GenerateOutput(target_list, target_dicts, data, params): # rule source. Collect the names of the directories that are # required. concrete_output_dirs = [] - for concrete_output_index in xrange(0, len(concrete_outputs)): + for concrete_output_index in range(0, len(concrete_outputs)): concrete_output = concrete_outputs[concrete_output_index] if concrete_output_index == 0: bol = '' @@ -1056,7 +1056,7 @@ def GenerateOutput(target_list, target_dicts, data, params): # the set of additional rule inputs, if any. prerequisites = [rule_source] prerequisites.extend(rule.get('inputs', [])) - for prerequisite_index in xrange(0, len(prerequisites)): + for prerequisite_index in range(0, len(prerequisites)): prerequisite = prerequisites[prerequisite_index] if prerequisite_index == len(prerequisites) - 1: eol = '' @@ -1278,7 +1278,7 @@ def GenerateOutput(target_list, target_dicts, data, params): set_define = EscapeXcodeDefine(define) xcbc.AppendBuildSetting('GCC_PREPROCESSOR_DEFINITIONS', set_define) if 'xcode_settings' in configuration: - for xck, xcv in configuration['xcode_settings'].iteritems(): + for xck, xcv in configuration['xcode_settings'].items(): xcbc.SetBuildSetting(xck, xcv) if 'xcode_config_file' in configuration: config_ref = pbxp.AddOrGetFileInRootGroup( @@ -1286,7 +1286,7 @@ def GenerateOutput(target_list, target_dicts, data, params): xcbc.SetBaseConfiguration(config_ref) build_files = [] - for build_file, build_file_dict in data.iteritems(): + for build_file, build_file_dict in data.items(): if build_file.endswith('.gyp'): build_files.append(build_file) diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index 5afbba03fe..68e51131d8 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -311,7 +311,7 @@ def LoadBuildFileIncludesIntoDict(subdict, subdict_path, data, aux_data, subdict_path, include) # Recurse into subdictionaries. - for k, v in subdict.iteritems(): + for k, v in subdict.items(): if type(v) is dict: LoadBuildFileIncludesIntoDict(v, subdict_path, data, aux_data, None, check) @@ -497,7 +497,7 @@ def CallLoadTargetBuildFile(global_flags, signal.signal(signal.SIGINT, signal.SIG_IGN) # Apply globals so that the worker process behaves the same. - for key, value in global_flags.iteritems(): + for key, value in global_flags.items(): globals()[key] = value SetGeneratorGlobals(generator_input_info) @@ -1028,7 +1028,7 @@ def ExpandVariables(input, phase, variables, build_file): # Convert all strings that are canonically-represented integers into integers. if type(output) is list: - for index in xrange(0, len(output)): + for index in range(0, len(output)): if IsStrCanonicalInt(output[index]): output[index] = int(output[index]) elif IsStrCanonicalInt(output): @@ -1160,7 +1160,7 @@ def ProcessConditionsInDict(the_dict, phase, variables, build_file): def LoadAutomaticVariablesFromDict(variables, the_dict): # Any keys with plain string values in the_dict become automatic variables. # The variable name is the key name with a "_" character prepended. - for key, value in the_dict.iteritems(): + for key, value in the_dict.items(): if type(value) in (str, int, list): variables['_' + key] = value @@ -1173,7 +1173,7 @@ def LoadVariablesFromVariablesDict(variables, the_dict, the_dict_key): # the_dict in the_dict's parent dict. If the_dict's parent is not a dict # (it could be a list or it could be parentless because it is a root dict), # the_dict_key will be None. - for key, value in the_dict.get('variables', {}).iteritems(): + for key, value in the_dict.get('variables', {}).items(): if type(value) not in (str, int, list): continue @@ -1212,7 +1212,7 @@ def ProcessVariablesAndConditionsInDict(the_dict, phase, variables_in, # list before we process them so that you can reference one # variable from another. They will be fully expanded by recursion # in ExpandVariables. - for key, value in the_dict['variables'].iteritems(): + for key, value in the_dict['variables'].items(): variables[key] = value # Handle the associated variables dict first, so that any variable @@ -1225,7 +1225,7 @@ def ProcessVariablesAndConditionsInDict(the_dict, phase, variables_in, LoadVariablesFromVariablesDict(variables, the_dict, the_dict_key) - for key, value in the_dict.iteritems(): + for key, value in the_dict.items(): # Skip "variables", which was already processed if present. if key != 'variables' and type(value) is str: expanded = ExpandVariables(value, phase, variables, build_file) @@ -1283,7 +1283,7 @@ def ProcessVariablesAndConditionsInDict(the_dict, phase, variables_in, # Recurse into child dicts, or process child lists which may result in # further recursion into descendant dicts. - for key, value in the_dict.iteritems(): + for key, value in the_dict.items(): # Skip "variables" and string values, which were already processed if # present. if key == 'variables' or type(value) is str: @@ -1380,12 +1380,12 @@ def QualifyDependencies(targets): for dep in dependency_sections for op in ('', '!', '/')] - for target, target_dict in targets.iteritems(): + for target, target_dict in targets.items(): target_build_file = gyp.common.BuildFile(target) toolset = target_dict['toolset'] for dependency_key in all_dependency_sections: dependencies = target_dict.get(dependency_key, []) - for index in xrange(0, len(dependencies)): + for index in range(0, len(dependencies)): dep_file, dep_target, dep_toolset = gyp.common.ResolveTarget( target_build_file, dependencies[index], toolset) if not multiple_toolsets: @@ -1420,13 +1420,13 @@ def ExpandWildcardDependencies(targets, data): dependency list, must be qualified when this function is called. """ - for target, target_dict in targets.iteritems(): + for target, target_dict in targets.items(): toolset = target_dict['toolset'] target_build_file = gyp.common.BuildFile(target) for dependency_key in dependency_sections: dependencies = target_dict.get(dependency_key, []) - # Loop this way instead of "for dependency in" or "for index in xrange" + # Loop this way instead of "for dependency in" or "for index in range" # because the dependencies list will be modified within the loop body. index = 0 while index < len(dependencies): @@ -1482,7 +1482,7 @@ def Unify(l): def RemoveDuplicateDependencies(targets): """Makes sure every dependency appears only once in all targets's dependency lists.""" - for target_name, target_dict in targets.iteritems(): + for target_name, target_dict in targets.items(): for dependency_key in dependency_sections: dependencies = target_dict.get(dependency_key, []) if dependencies: @@ -1498,7 +1498,7 @@ def Filter(l, item): def RemoveSelfDependencies(targets): """Remove self dependencies from targets that have the prune_self_dependency variable set.""" - for target_name, target_dict in targets.iteritems(): + for target_name, target_dict in targets.items(): for dependency_key in dependency_sections: dependencies = target_dict.get(dependency_key, []) if dependencies: @@ -1511,7 +1511,7 @@ def RemoveSelfDependencies(targets): def RemoveLinkDependenciesFromNoneTargets(targets): """Remove dependencies having the 'link_dependency' attribute from the 'none' targets.""" - for target_name, target_dict in targets.iteritems(): + for target_name, target_dict in targets.items(): for dependency_key in dependency_sections: dependencies = target_dict.get(dependency_key, []) if dependencies: @@ -1797,14 +1797,14 @@ def BuildDependencyList(targets): # Create a DependencyGraphNode for each target. Put it into a dict for easy # access. dependency_nodes = {} - for target, spec in targets.iteritems(): + for target, spec in targets.items(): if target not in dependency_nodes: dependency_nodes[target] = DependencyGraphNode(target) # Set up the dependency links. Targets that have no dependencies are treated # as dependent on root_node. root_node = DependencyGraphNode(None) - for target, spec in targets.iteritems(): + for target, spec in targets.items(): target_node = dependency_nodes[target] target_build_file = gyp.common.BuildFile(target) dependencies = spec.get('dependencies') @@ -1853,7 +1853,7 @@ def VerifyNoGYPFileCircularDependencies(targets): dependency_nodes[build_file] = DependencyGraphNode(build_file) # Set up the dependency links. - for target, spec in targets.iteritems(): + for target, spec in targets.items(): build_file = gyp.common.BuildFile(target) build_file_node = dependency_nodes[build_file] target_dependencies = spec.get('dependencies', []) @@ -2118,7 +2118,7 @@ def is_in_set_or_list(x, s, l): def MergeDicts(to, fro, to_file, fro_file): # I wanted to name the parameter "from" but it's a Python keyword... - for k, v in fro.iteritems(): + for k, v in fro.items(): # It would be nice to do "if not k in to: to[k] = v" but that wouldn't give # copy semantics. Something else may want to merge from the |fro| dict # later, and having the same dict ref pointed to twice in the tree isn't @@ -2253,13 +2253,13 @@ def SetUpConfigurations(target, target_dict): if not 'configurations' in target_dict: target_dict['configurations'] = {'Default': {}} if not 'default_configuration' in target_dict: - concrete = [i for (i, config) in target_dict['configurations'].iteritems() + concrete = [i for (i, config) in target_dict['configurations'].items() if not config.get('abstract')] target_dict['default_configuration'] = sorted(concrete)[0] merged_configurations = {} configs = target_dict['configurations'] - for (configuration, old_configuration_dict) in configs.iteritems(): + for (configuration, old_configuration_dict) in configs.items(): # Skip abstract configurations (saves work only). if old_configuration_dict.get('abstract'): continue @@ -2267,7 +2267,7 @@ def SetUpConfigurations(target, target_dict): # Get the inheritance relationship right by making a copy of the target # dict. new_configuration_dict = {} - for (key, target_val) in target_dict.iteritems(): + for (key, target_val) in target_dict.items(): key_ext = key[-1:] if key_ext in key_suffixes: key_base = key[:-1] @@ -2351,7 +2351,7 @@ def ProcessListFiltersInDict(name, the_dict): lists = [] del_lists = [] - for key, value in the_dict.iteritems(): + for key, value in the_dict.items(): operation = key[-1] if operation != '!' and operation != '/': continue @@ -2399,7 +2399,7 @@ def ProcessListFiltersInDict(name, the_dict): exclude_key = list_key + '!' if exclude_key in the_dict: for exclude_item in the_dict[exclude_key]: - for index in xrange(0, len(the_list)): + for index in range(0, len(the_list)): if exclude_item == the_list[index]: # This item matches the exclude_item, so set its action to 0 # (exclude). @@ -2425,7 +2425,7 @@ def ProcessListFiltersInDict(name, the_dict): raise ValueError('Unrecognized action ' + action + ' in ' + name + \ ' key ' + regex_key) - for index in xrange(0, len(the_list)): + for index in range(0, len(the_list)): list_item = the_list[index] if list_actions[index] == action_value: # Even if the regex matches, nothing will change so continue (regex @@ -2456,7 +2456,7 @@ def ProcessListFiltersInDict(name, the_dict): # the indices of items that haven't been seen yet don't shift. That means # that things need to be prepended to excluded_list to maintain them in the # same order that they existed in the_list. - for index in xrange(len(list_actions) - 1, -1, -1): + for index in range(len(list_actions) - 1, -1, -1): if list_actions[index] == 0: # Dump anything with action 0 (exclude). Keep anything with action 1 # (include) or -1 (no include or exclude seen for the item). @@ -2469,7 +2469,7 @@ def ProcessListFiltersInDict(name, the_dict): the_dict[excluded_key] = excluded_list # Now recurse into subdicts and lists that may contain dicts. - for key, value in the_dict.iteritems(): + for key, value in the_dict.items(): if type(value) is dict: ProcessListFiltersInDict(key, value) elif type(value) is list: @@ -2526,7 +2526,7 @@ def ValidateSourcesInTarget(target, target_dict, build_file, basenames.setdefault(basename, []).append(source) error = '' - for basename, files in basenames.iteritems(): + for basename, files in basenames.items(): if len(files) > 1: error += ' %s: %s\n' % (basename, ' '.join(files)) @@ -2646,7 +2646,7 @@ def ValidateActionsInTarget(target, target_dict, build_file): def TurnIntIntoStrInDict(the_dict): """Given dict the_dict, recursively converts all integers into strings. """ - # Use items instead of iteritems because there's no need to try to look at + # Use items instead of items because there's no need to try to look at # reinserted keys and their associated values. for k, v in the_dict.items(): if type(v) is int: @@ -2665,7 +2665,7 @@ def TurnIntIntoStrInDict(the_dict): def TurnIntIntoStrInList(the_list): """Given list the_list, recursively converts all integers into strings. """ - for index in xrange(0, len(the_list)): + for index in range(0, len(the_list)): item = the_list[index] if type(item) is int: the_list[index] = str(item) @@ -2805,7 +2805,7 @@ def Load(build_files, variables, includes, depth, generator_input_info, check, RemoveLinkDependenciesFromNoneTargets(targets) # Apply exclude (!) and regex (/) list filters only for dependency_sections. - for target_name, target_dict in targets.iteritems(): + for target_name, target_dict in targets.items(): tmp_dict = {} for key_base in dependency_sections: for op in ('', '!', '/'): diff --git a/gyp/pylib/gyp/input_test.py b/gyp/pylib/gyp/input_test.py index 4234fbb830..1bc5e3d308 100755 --- a/gyp/pylib/gyp/input_test.py +++ b/gyp/pylib/gyp/input_test.py @@ -22,38 +22,38 @@ def _create_dependency(self, dependent, dependency): dependency.dependents.append(dependent) def test_no_cycle_empty_graph(self): - for label, node in self.nodes.iteritems(): - self.assertEquals([], node.FindCycles()) + for label, node in self.nodes.items(): + self.assertEqual([], node.FindCycles()) def test_no_cycle_line(self): self._create_dependency(self.nodes['a'], self.nodes['b']) self._create_dependency(self.nodes['b'], self.nodes['c']) self._create_dependency(self.nodes['c'], self.nodes['d']) - for label, node in self.nodes.iteritems(): - self.assertEquals([], node.FindCycles()) + for label, node in self.nodes.items(): + self.assertEqual([], node.FindCycles()) def test_no_cycle_dag(self): self._create_dependency(self.nodes['a'], self.nodes['b']) self._create_dependency(self.nodes['a'], self.nodes['c']) self._create_dependency(self.nodes['b'], self.nodes['c']) - for label, node in self.nodes.iteritems(): - self.assertEquals([], node.FindCycles()) + for label, node in self.nodes.items(): + self.assertEqual([], node.FindCycles()) def test_cycle_self_reference(self): self._create_dependency(self.nodes['a'], self.nodes['a']) - self.assertEquals([[self.nodes['a'], self.nodes['a']]], + self.assertEqual([[self.nodes['a'], self.nodes['a']]], self.nodes['a'].FindCycles()) def test_cycle_two_nodes(self): self._create_dependency(self.nodes['a'], self.nodes['b']) self._create_dependency(self.nodes['b'], self.nodes['a']) - self.assertEquals([[self.nodes['a'], self.nodes['b'], self.nodes['a']]], + self.assertEqual([[self.nodes['a'], self.nodes['b'], self.nodes['a']]], self.nodes['a'].FindCycles()) - self.assertEquals([[self.nodes['b'], self.nodes['a'], self.nodes['b']]], + self.assertEqual([[self.nodes['b'], self.nodes['a'], self.nodes['b']]], self.nodes['b'].FindCycles()) def test_two_cycles(self): @@ -68,7 +68,7 @@ def test_two_cycles(self): [self.nodes['a'], self.nodes['b'], self.nodes['a']] in cycles) self.assertTrue( [self.nodes['b'], self.nodes['c'], self.nodes['b']] in cycles) - self.assertEquals(2, len(cycles)) + self.assertEqual(2, len(cycles)) def test_big_cycle(self): self._create_dependency(self.nodes['a'], self.nodes['b']) @@ -77,7 +77,7 @@ def test_big_cycle(self): self._create_dependency(self.nodes['d'], self.nodes['e']) self._create_dependency(self.nodes['e'], self.nodes['a']) - self.assertEquals([[self.nodes['a'], + self.assertEqual([[self.nodes['a'], self.nodes['b'], self.nodes['c'], self.nodes['d'], diff --git a/gyp/pylib/gyp/mac_tool.py b/gyp/pylib/gyp/mac_tool.py index 0ab3de18f7..b8f38a8fdc 100755 --- a/gyp/pylib/gyp/mac_tool.py +++ b/gyp/pylib/gyp/mac_tool.py @@ -326,7 +326,7 @@ def ExecCompileXcassets(self, keys, *inputs): ]) if keys: keys = json.loads(keys) - for key, value in keys.iteritems(): + for key, value in keys.items(): arg_name = '--' + key if isinstance(value, bool): if value: @@ -487,7 +487,7 @@ def _LoadProvisioningProfile(self, profile_path): def _MergePlist(self, merged_plist, plist): """Merge |plist| into |merged_plist|.""" - for key, value in plist.iteritems(): + for key, value in plist.items(): if isinstance(value, dict): merged_value = merged_plist.get(key, {}) if isinstance(merged_value, dict): @@ -597,7 +597,7 @@ def _ExpandVariables(self, data, substitutions): the key was not found. """ if isinstance(data, str): - for key, value in substitutions.iteritems(): + for key, value in substitutions.items(): data = data.replace('$(%s)' % key, value) return data if isinstance(data, list): diff --git a/gyp/pylib/gyp/msvs_emulation.py b/gyp/pylib/gyp/msvs_emulation.py index ca67b122f0..75da78526c 100644 --- a/gyp/pylib/gyp/msvs_emulation.py +++ b/gyp/pylib/gyp/msvs_emulation.py @@ -205,7 +205,7 @@ def __init__(self, spec, generator_flags): configs = spec['configurations'] for field, default in supported_fields: setattr(self, field, {}) - for configname, config in configs.iteritems(): + for configname, config in configs.items(): getattr(self, field)[configname] = config.get(field, default()) self.msvs_cygwin_dirs = spec.get('msvs_cygwin_dirs', ['.']) @@ -941,7 +941,7 @@ def ExpandMacros(string, expansions): """Expand $(Variable) per expansions dict. See MsvsSettings.GetVSMacroEnv for the canonical way to retrieve a suitable dict.""" if '$' in string: - for old, new in expansions.iteritems(): + for old, new in expansions.items(): assert '$(' not in new, new string = string.replace(old, new) return string @@ -985,7 +985,7 @@ def _FormatAsEnvironmentBlock(envvar_dict): CreateProcess documentation for more details.""" block = '' nul = '\0' - for key, value in envvar_dict.iteritems(): + for key, value in envvar_dict.items(): block += key + '=' + value + nul block += nul return block diff --git a/gyp/pylib/gyp/ordered_dict.py b/gyp/pylib/gyp/ordered_dict.py index a1e89f9199..6fe9c1f6c7 100644 --- a/gyp/pylib/gyp/ordered_dict.py +++ b/gyp/pylib/gyp/ordered_dict.py @@ -161,8 +161,8 @@ def itervalues(self): for k in self: yield self[k] - def iteritems(self): - 'od.iteritems -> an iterator over the (key, value) items in od' + def items(self): + 'od.items -> an iterator over the (key, value) items in od' for k in self: yield (k, self[k]) diff --git a/gyp/pylib/gyp/simple_copy.py b/gyp/pylib/gyp/simple_copy.py index 74c98c5a79..463929386e 100644 --- a/gyp/pylib/gyp/simple_copy.py +++ b/gyp/pylib/gyp/simple_copy.py @@ -38,7 +38,7 @@ def _deepcopy_list(x): def _deepcopy_dict(x): y = {} - for key, value in x.iteritems(): + for key, value in x.items(): y[deepcopy(key)] = deepcopy(value) return y d[dict] = _deepcopy_dict diff --git a/gyp/pylib/gyp/win_tool.py b/gyp/pylib/gyp/win_tool.py index 3bade00dbe..bca0b9e346 100755 --- a/gyp/pylib/gyp/win_tool.py +++ b/gyp/pylib/gyp/win_tool.py @@ -294,7 +294,7 @@ def ExecActionWrapper(self, arch, rspfile, *dir): env = self._GetEnv(arch) # TODO(scottmg): This is a temporary hack to get some specific variables # through to actions that are set after gyp-time. http://crbug.com/333738. - for k, v in os.environ.iteritems(): + for k, v in os.environ.items(): if k not in env: env[k] = v args = open(rspfile).read() diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py index fd4427b86b..66f325932a 100644 --- a/gyp/pylib/gyp/xcode_emulation.py +++ b/gyp/pylib/gyp/xcode_emulation.py @@ -170,7 +170,7 @@ def __init__(self, spec): # the same for all configs are implicitly per-target settings. self.xcode_settings = {} configs = spec['configurations'] - for configname, config in configs.iteritems(): + for configname, config in configs.items(): self.xcode_settings[configname] = config.get('xcode_settings', {}) self._ConvertConditionalKeys(configname) if self.xcode_settings[configname].get('IPHONEOS_DEPLOYMENT_TARGET', @@ -891,7 +891,7 @@ def GetPerTargetSettings(self): result = dict(self.xcode_settings[configname]) first_pass = False else: - for key, value in self.xcode_settings[configname].iteritems(): + for key, value in self.xcode_settings[configname].items(): if key not in result: continue elif result[key] != value: @@ -1639,7 +1639,7 @@ def _AddIOSDeviceConfigurations(targets): for target_dict in targets.itervalues(): toolset = target_dict['toolset'] configs = target_dict['configurations'] - for config_name, config_dict in dict(configs).iteritems(): + for config_name, config_dict in dict(configs).items(): iphoneos_config_dict = copy.deepcopy(config_dict) configs[config_name + '-iphoneos'] = iphoneos_config_dict configs[config_name + '-iphonesimulator'] = config_dict diff --git a/gyp/pylib/gyp/xcode_ninja.py b/gyp/pylib/gyp/xcode_ninja.py index d781471005..5acd82e004 100644 --- a/gyp/pylib/gyp/xcode_ninja.py +++ b/gyp/pylib/gyp/xcode_ninja.py @@ -161,7 +161,7 @@ def CreateWrapper(target_list, target_dicts, data, params): params: Dict of global options for gyp. """ orig_gyp = params['build_files'][0] - for gyp_name, gyp_dict in data.iteritems(): + for gyp_name, gyp_dict in data.items(): if gyp_name == orig_gyp: depth = gyp_dict['_DEPTH'] @@ -228,7 +228,7 @@ def CreateWrapper(target_list, target_dicts, data, params): sources_target['configurations'] = {'Default': { 'include_dirs': [ depth ] } } sources = [] - for target, target_dict in target_dicts.iteritems(): + for target, target_dict in target_dicts.items(): base = os.path.dirname(target) files = target_dict.get('sources', []) + \ target_dict.get('mac_bundle_resources', []) diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py index 712eefe3f8..9bd582e477 100644 --- a/gyp/pylib/gyp/xcodeproj_file.py +++ b/gyp/pylib/gyp/xcodeproj_file.py @@ -314,7 +314,7 @@ def Copy(self): """ that = self.__class__(id=self.id, parent=self.parent) - for key, value in self._properties.iteritems(): + for key, value in self._properties.items(): is_strong = self._schema[key][2] if isinstance(value, XCObject): @@ -452,7 +452,7 @@ def _HashUpdate(hash, data): digest_int_count = hash.digest_size / 4 digest_ints = struct.unpack('>' + 'I' * digest_int_count, hash.digest()) id_ints = [0, 0, 0] - for index in xrange(0, digest_int_count): + for index in range(0, digest_int_count): id_ints[index % 3] ^= digest_ints[index] self.id = '%08X%08X%08X' % tuple(id_ints) @@ -475,7 +475,7 @@ def Children(self): """Returns a list of all of this object's owned (strong) children.""" children = [] - for property, attributes in self._schema.iteritems(): + for property, attributes in self._schema.items(): (is_list, property_type, is_strong) = attributes[0:3] if is_strong and property in self._properties: if not is_list: @@ -622,7 +622,7 @@ def _XCPrintableValue(self, tabs, value, flatten_list=False): printable += end_tabs + ')' elif isinstance(value, dict): printable = '{' + sep - for item_key, item_value in sorted(value.iteritems()): + for item_key, item_value in sorted(value.items()): printable += element_tabs + \ self._XCPrintableValue(tabs + 1, item_key, flatten_list) + ' = ' + \ self._XCPrintableValue(tabs + 1, item_value, flatten_list) + ';' + \ @@ -730,7 +730,7 @@ def Print(self, file=sys.stdout): self._XCKVPrint(file, 3, 'isa', self.__class__.__name__) # The remaining elements of an object dictionary are sorted alphabetically. - for property, value in sorted(self._properties.iteritems()): + for property, value in sorted(self._properties.items()): self._XCKVPrint(file, 3, property, value) # End the object. @@ -752,7 +752,7 @@ def UpdateProperties(self, properties, do_copy=False): if properties is None: return - for property, value in properties.iteritems(): + for property, value in properties.items(): # Make sure the property is in the schema. if not property in self._schema: raise KeyError(property + ' not in ' + self.__class__.__name__) @@ -865,7 +865,7 @@ def VerifyHasRequiredProperties(self): # TODO(mark): A stronger verification mechanism is needed. Some # subclasses need to perform validation beyond what the schema can enforce. - for property, attributes in self._schema.iteritems(): + for property, attributes in self._schema.items(): (is_list, property_type, is_strong, is_required) = attributes[0:4] if is_required and not property in self._properties: raise KeyError(self.__class__.__name__ + ' requires ' + property) @@ -875,7 +875,7 @@ def _SetDefaultsFromSchema(self): overwrite properties that have already been set.""" defaults = {} - for property, attributes in self._schema.iteritems(): + for property, attributes in self._schema.items(): (is_list, property_type, is_strong, is_required) = attributes[0:4] if is_required and len(attributes) >= 5 and \ not property in self._properties: @@ -1426,7 +1426,7 @@ def PathHashables(self): xche = self while xche != None and isinstance(xche, XCHierarchicalElement): xche_hashables = xche.Hashables() - for index in xrange(0, len(xche_hashables)): + for index in range(0, len(xche_hashables)): hashables.insert(index, xche_hashables[index]) xche = xche.parent return hashables @@ -2401,7 +2401,7 @@ def HeadersPhase(self): # The headers phase should come before the resources, sources, and # frameworks phases, if any. insert_at = len(self._properties['buildPhases']) - for index in xrange(0, len(self._properties['buildPhases'])): + for index in range(0, len(self._properties['buildPhases'])): phase = self._properties['buildPhases'][index] if isinstance(phase, PBXResourcesBuildPhase) or \ isinstance(phase, PBXSourcesBuildPhase) or \ @@ -2422,7 +2422,7 @@ def ResourcesPhase(self): # The resources phase should come before the sources and frameworks # phases, if any. insert_at = len(self._properties['buildPhases']) - for index in xrange(0, len(self._properties['buildPhases'])): + for index in range(0, len(self._properties['buildPhases'])): phase = self._properties['buildPhases'][index] if isinstance(phase, PBXSourcesBuildPhase) or \ isinstance(phase, PBXFrameworksBuildPhase): @@ -2844,7 +2844,7 @@ def CompareProducts(x, y, remote_products): # determine the sort order. return cmp(x_index, y_index) - for other_pbxproject, ref_dict in self._other_pbxprojects.iteritems(): + for other_pbxproject, ref_dict in self._other_pbxprojects.items(): # Build up a list of products in the remote project file, ordered the # same as the targets that produce them. remote_products = [] @@ -2889,7 +2889,7 @@ def Print(self, file=sys.stdout): self._XCPrint(file, 0, '{ ') else: self._XCPrint(file, 0, '{\n') - for property, value in sorted(self._properties.iteritems(), + for property, value in sorted(self._properties.items(), cmp=lambda x, y: cmp(x, y)): if property == 'objects': self._PrintObjects(file) From 98226d198c82c7c58eee678f81ec9656f9255a05 Mon Sep 17 00:00:00 2001 From: Craig Rodrigues Date: Sun, 19 Mar 2017 13:59:05 -0700 Subject: [PATCH 010/551] gyp: replace basestring with str, but only on Python 3. On Python 2, basestring is (unicode, str). On Python 3, basestring and unicode are gone, and there is only str. --- gyp/pylib/gyp/MSVSSettings.py | 10 ++++++---- gyp/pylib/gyp/__init__.py | 9 ++++++++- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/gyp/pylib/gyp/MSVSSettings.py b/gyp/pylib/gyp/MSVSSettings.py index 00721eb237..8073f92b8d 100644 --- a/gyp/pylib/gyp/MSVSSettings.py +++ b/gyp/pylib/gyp/MSVSSettings.py @@ -16,6 +16,8 @@ from __future__ import print_function +from gyp import string_types + import sys import re @@ -108,11 +110,11 @@ class _String(_Type): """A setting that's just a string.""" def ValidateMSVS(self, value): - if not isinstance(value, basestring): + if not isinstance(value, string_types): raise ValueError('expected string; got %r' % value) def ValidateMSBuild(self, value): - if not isinstance(value, basestring): + if not isinstance(value, string_types): raise ValueError('expected string; got %r' % value) def ConvertToMSBuild(self, value): @@ -124,11 +126,11 @@ class _StringList(_Type): """A settings that's a list of strings.""" def ValidateMSVS(self, value): - if not isinstance(value, basestring) and not isinstance(value, list): + if not isinstance(value, string_types) and not isinstance(value, list): raise ValueError('expected string list; got %r' % value) def ValidateMSBuild(self, value): - if not isinstance(value, basestring) and not isinstance(value, list): + if not isinstance(value, string_types) and not isinstance(value, list): raise ValueError('expected string list; got %r' % value) def ConvertToMSBuild(self, value): diff --git a/gyp/pylib/gyp/__init__.py b/gyp/pylib/gyp/__init__.py index 5f82f2ef2c..d5fa9ecf50 100755 --- a/gyp/pylib/gyp/__init__.py +++ b/gyp/pylib/gyp/__init__.py @@ -16,6 +16,13 @@ import traceback from gyp.common import GypError +try: + # Python 2 + string_types = basestring +except NameError: + # Python 3 + string_types = str + # Default debug modes for GYP debug = {} @@ -412,7 +419,7 @@ def gyp_main(args): for option, value in sorted(options.__dict__.items()): if option[0] == '_': continue - if isinstance(value, basestring): + if isinstance(value, string_types): DebugOutput(DEBUG_GENERAL, " %s: '%s'", option, value) else: DebugOutput(DEBUG_GENERAL, " %s: %s", option, value) From 588d333c144cf0ba0534e0d40b4f120bbfe4b569 Mon Sep 17 00:00:00 2001 From: Craig Rodrigues Date: Sun, 19 Mar 2017 14:04:36 -0700 Subject: [PATCH 011/551] gyp: _winreg module was renamed to winreg in Python 3. https://github.com/nodejs/node-gyp/pull/1150 Reviewed-By: Refael Ackermann --- gyp/pylib/gyp/MSVSVersion.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/gyp/pylib/gyp/MSVSVersion.py b/gyp/pylib/gyp/MSVSVersion.py index c31ff3e114..293b4145c1 100644 --- a/gyp/pylib/gyp/MSVSVersion.py +++ b/gyp/pylib/gyp/MSVSVersion.py @@ -176,12 +176,18 @@ def _RegistryGetValueUsingWinReg(key, value): contents of the registry key's value, or None on failure. Throws ImportError if _winreg is unavailable. """ - import _winreg + try: + # Python 2 + from _winreg import OpenKey, QueryValueEx + except ImportError: + # Python 3 + from winreg import OpenKey, QueryValueEx + try: root, subkey = key.split('\\', 1) assert root == 'HKLM' # Only need HKLM for now. - with _winreg.OpenKey(_winreg.HKEY_LOCAL_MACHINE, subkey) as hkey: - return _winreg.QueryValueEx(hkey, value)[0] + with OpenKey(_winreg.HKEY_LOCAL_MACHINE, subkey) as hkey: + return QueryValueEx(hkey, value)[0] except WindowsError: return None From febdfa21371e2e11596f1d43fbd4f8232ffe231d Mon Sep 17 00:00:00 2001 From: cclauss Date: Tue, 14 Nov 2017 07:56:07 +0100 Subject: [PATCH 012/551] gyp: fix sntex error PR-URL: https://github.com/nodejs/node-gyp/pull/1333 Reviewed-By: Refael Ackermann --- gyp/pylib/gyp/mac_tool.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gyp/pylib/gyp/mac_tool.py b/gyp/pylib/gyp/mac_tool.py index b8f38a8fdc..b8b7344eff 100755 --- a/gyp/pylib/gyp/mac_tool.py +++ b/gyp/pylib/gyp/mac_tool.py @@ -127,7 +127,7 @@ def _DetectInputEncoding(self, file_name): fp = open(file_name, 'rb') try: header = fp.read(3) - except e: + except Exception: fp.close() return None fp.close() From 92e8b52ceeb801fa044aa4e69b215b800ba5f4a6 Mon Sep 17 00:00:00 2001 From: cclauss Date: Tue, 14 Nov 2017 09:00:29 +0100 Subject: [PATCH 013/551] gyp: fix target --> self.target target is an undefined name in this context. https://github.com/nodejs/node-gyp/pull/1334 Reviewed-By: Refael Ackermann --- gyp/pylib/gyp/generator/make.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index cfc7b05433..d549e899d8 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -1639,7 +1639,7 @@ def WriteTarget(self, spec, configs, deps, link_deps, bundle_deps, self.WriteDoCmd([self.output_binary], deps, 'touch', part_of_all, postbuilds=postbuilds) else: - print("WARNING: no output for", self.type, target) + print("WARNING: no output for", self.type, self.target) # Add an alias for each target (if there are any outputs). # Installable target aliases are created below. From 8098ebdeb4b93f560785b63ecf2a9427101d454f Mon Sep 17 00:00:00 2001 From: Selwyn Date: Sat, 13 Oct 2018 14:58:11 +0200 Subject: [PATCH 014/551] deps: replace `osenv` dependency with native `os` Breaking change: needs Node.js version 6 or higher https://github.com/nodejs/node-gyp/pull/1570 Reviewed-By: Refael Ackermann --- bin/node-gyp.js | 4 ++-- lib/configure.js | 4 ++-- lib/install.js | 6 +++--- package.json | 3 +-- 4 files changed, 8 insertions(+), 9 deletions(-) diff --git a/bin/node-gyp.js b/bin/node-gyp.js index a8fd9bc529..9978c3b35f 100755 --- a/bin/node-gyp.js +++ b/bin/node-gyp.js @@ -4,7 +4,7 @@ process.title = 'node-gyp' var gyp = require('../') var log = require('npmlog') -var osenv = require('osenv') +var os = require('os') var path = require('path') /** @@ -16,7 +16,7 @@ var completed = false prog.parseArgv(process.argv) prog.devDir = prog.opts.devdir -var homeDir = osenv.home() +var homeDir = os.homedir() if (prog.devDir) { prog.devDir = prog.devDir.replace(/^~/, homeDir) } else if (homeDir) { diff --git a/lib/configure.js b/lib/configure.js index 0baf58dabc..33bb7e746a 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -8,7 +8,7 @@ module.exports.test = { var fs = require('graceful-fs') , path = require('path') , log = require('npmlog') - , osenv = require('osenv') + , os = require('os') , which = require('which') , semver = require('semver') , mkdirp = require('mkdirp') @@ -46,7 +46,7 @@ function configure (gyp, argv, callback) { if (gyp.opts.nodedir) { // --nodedir was specified. use that for the dev files - nodeDir = gyp.opts.nodedir.replace(/^~/, osenv.home()) + nodeDir = gyp.opts.nodedir.replace(/^~/, os.homedir()) log.verbose('get node dir', 'compiling against specified --nodedir dev files: %s', nodeDir) createBuildDir() diff --git a/lib/install.js b/lib/install.js index eea7618a01..735c31c0d0 100644 --- a/lib/install.js +++ b/lib/install.js @@ -11,7 +11,7 @@ module.exports.test = { exports.usage = 'Install node development files for the specified node version.' var fs = require('graceful-fs') - , osenv = require('osenv') + , os = require('os') , tar = require('tar') , path = require('path') , crypto = require('crypto') @@ -400,9 +400,9 @@ function install (fs, gyp, argv, callback) { function eaccesFallback (err) { var noretry = '--node_gyp_internal_noretry' if (-1 !== argv.indexOf(noretry)) return cb(err) - var tmpdir = osenv.tmpdir() + var tmpdir = os.tmpdir() gyp.devDir = path.resolve(tmpdir, '.node-gyp') - log.warn('EACCES', 'user "%s" does not have permission to access the dev dir "%s"', osenv.user(), devDir) + log.warn('EACCES', 'user "%s" does not have permission to access the dev dir "%s"', os.userInfo().username, devDir) log.warn('EACCES', 'attempting to reinstall using temporary dev dir "%s"', gyp.devDir) if (process.cwd() == tmpdir) { log.verbose('tmpdir == cwd', 'automatically will remove dev files after to save disk space') diff --git a/package.json b/package.json index accb86b44b..c00363ca21 100644 --- a/package.json +++ b/package.json @@ -27,7 +27,6 @@ "mkdirp": "^0.5.0", "nopt": "2 || 3", "npmlog": "0 || 1 || 2 || 3 || 4", - "osenv": "0", "request": "^2.87.0", "rimraf": "2", "semver": "~5.3.0", @@ -35,7 +34,7 @@ "which": "1" }, "engines": { - "node": ">= 4.0.0" + "node": ">= 6.0.0" }, "devDependencies": { "babel-eslint": "^8.2.5", From 8a8397274353de92ce99547b8c7d6813653ebedf Mon Sep 17 00:00:00 2001 From: Selwyn Date: Sat, 13 Oct 2018 15:12:15 +0200 Subject: [PATCH 015/551] bin: follow XDG OS conventions for storing data PR-URL: https://github.com/nodejs/node-gyp/pull/1570 Fixes: https://github.com/nodejs/node-gyp/pull/175 Fixes: https://github.com/nodejs/node-gyp/pull/1124 Reviewed-By: Refael Ackermann --- README.md | 2 +- bin/node-gyp.js | 4 ++-- package.json | 1 + 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index e88bd0b5be..f378616964 100644 --- a/README.md +++ b/README.md @@ -170,7 +170,7 @@ Some additional resources for addons and writing `gyp` files: | `--thin=yes` | Enable thin static libraries | `--arch=$arch` | Set target architecture (e.g. ia32) | `--tarball=$path` | Get headers from a local tarball -| `--devdir=$path` | SDK download directory (default is `~/.node-gyp`) +| `--devdir=$path` | SDK download directory (default is OS cache directory) | `--ensure` | Don't reinstall headers if already present | `--dist-url=$url` | Download header tarball from custom URL | `--proxy=$url` | Set HTTP proxy for downloading header tarball diff --git a/bin/node-gyp.js b/bin/node-gyp.js index 9978c3b35f..8c9b6169bb 100755 --- a/bin/node-gyp.js +++ b/bin/node-gyp.js @@ -2,10 +2,10 @@ process.title = 'node-gyp' +var envPaths = require('env-paths') var gyp = require('../') var log = require('npmlog') var os = require('os') -var path = require('path') /** * Process and execute the selected commands. @@ -20,7 +20,7 @@ var homeDir = os.homedir() if (prog.devDir) { prog.devDir = prog.devDir.replace(/^~/, homeDir) } else if (homeDir) { - prog.devDir = path.resolve(homeDir, '.node-gyp') + prog.devDir = envPaths('node-gyp', { suffix: '' }).cache } else { throw new Error( "node-gyp requires that the user's home directory is specified " + diff --git a/package.json b/package.json index c00363ca21..cce2e4d55f 100644 --- a/package.json +++ b/package.json @@ -22,6 +22,7 @@ "bin": "./bin/node-gyp.js", "main": "./lib/node-gyp.js", "dependencies": { + "env-paths": "^1.0.0", "glob": "^7.0.3", "graceful-fs": "^4.1.2", "mkdirp": "^0.5.0", From 6f5a408934d4f89b2ee1923a11ff44a93d0d44a1 Mon Sep 17 00:00:00 2001 From: Jens Date: Sat, 20 Oct 2018 23:14:58 +0200 Subject: [PATCH 016/551] tools: fix usage of inherited -fPIC and -fPIE PR-URL: https://github.com/nodejs/node-gyp/pull/1340 Reviewed-By: Refael Ackermann --- addon.gypi | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/addon.gypi b/addon.gypi index 55fb321118..6b4641b693 100644 --- a/addon.gypi +++ b/addon.gypi @@ -55,6 +55,14 @@ 'standalone_static_library': '<(standalone_static_library)' }], + ['_type!="executable"', { + 'conditions': [ + [ 'OS=="android"', { + 'cflags!': [ '-fPIE' ], + }] + ] + }], + ['_win_delay_load_hook=="true"', { # If the addon specifies `'win_delay_load_hook': 'true'` in its # binding.gyp, link a delay-load hook into the DLL. This hook ensures @@ -138,10 +146,10 @@ '_FILE_OFFSET_BITS=64' ], }], - [ 'OS in "freebsd openbsd netbsd solaris" or \ + [ 'OS in "freebsd openbsd netbsd solaris android" or \ (OS=="linux" and target_arch!="ia32")', { 'cflags': [ '-fPIC' ], - }] + }], ] } } From d3b21220a0f720345b8bbcaccef3090966e51cb5 Mon Sep 17 00:00:00 2001 From: Andy Dill Date: Mon, 22 Oct 2018 15:04:13 -0700 Subject: [PATCH 017/551] win: fix delay-load hook for electron 4 PR-URL: nodejs#1566 Reviewed-By: Refael Ackermann --- addon.gypi | 4 +++- src/win_delay_load_hook.cc | 11 +++++------ test/test-addon.js | 35 +++++++++++++++++++++++++++++++++-- 3 files changed, 41 insertions(+), 9 deletions(-) diff --git a/addon.gypi b/addon.gypi index 6b4641b693..e8b95c5501 100644 --- a/addon.gypi +++ b/addon.gypi @@ -1,6 +1,7 @@ { 'variables' : { 'node_engine_include_dir%': 'deps/v8/include', + 'node_host_binary%': 'node' }, 'target_defaults': { 'type': 'loadable_module', @@ -70,12 +71,13 @@ # is named node.exe, iojs.exe, or something else. 'conditions': [ [ 'OS=="win"', { + 'defines': [ 'HOST_BINARY=\"<(node_host_binary)<(EXECUTABLE_SUFFIX)\"', ], 'sources': [ '<(node_gyp_dir)/src/win_delay_load_hook.cc', ], 'msvs_settings': { 'VCLinkerTool': { - 'DelayLoadDLLs': [ 'iojs.exe', 'node.exe' ], + 'DelayLoadDLLs': [ '<(node_host_binary)<(EXECUTABLE_SUFFIX)' ], # Don't print a linker warning when no imports from either .exe # are used. 'AdditionalOptions': [ '/ignore:4199' ], diff --git a/src/win_delay_load_hook.cc b/src/win_delay_load_hook.cc index e75954b605..5e7f14b2b2 100644 --- a/src/win_delay_load_hook.cc +++ b/src/win_delay_load_hook.cc @@ -1,10 +1,10 @@ /* * When this file is linked to a DLL, it sets up a delay-load hook that - * intervenes when the DLL is trying to load 'node.exe' or 'iojs.exe' - * dynamically. Instead of trying to locate the .exe file it'll just return - * a handle to the process image. + * intervenes when the DLL is trying to load the host executable + * dynamically. Instead of trying to locate the .exe file it'll just + * return a handle to the process image. * - * This allows compiled addons to work when node.exe or iojs.exe is renamed. + * This allows compiled addons to work when the host executable is renamed. */ #ifdef _MSC_VER @@ -23,8 +23,7 @@ static FARPROC WINAPI load_exe_hook(unsigned int event, DelayLoadInfo* info) { if (event != dliNotePreLoadLibrary) return NULL; - if (_stricmp(info->szDll, "iojs.exe") != 0 && - _stricmp(info->szDll, "node.exe") != 0) + if (_stricmp(info->szDll, HOST_BINARY) != 0) return NULL; m = GetModuleHandle(NULL); diff --git a/test/test-addon.js b/test/test-addon.js index 89350effc4..900b0b049d 100644 --- a/test/test-addon.js +++ b/test/test-addon.js @@ -4,14 +4,19 @@ var test = require('tape') var path = require('path') var fs = require('graceful-fs') var child_process = require('child_process') +var os = require('os') var addonPath = path.resolve(__dirname, 'node_modules', 'hello_world') var nodeGyp = path.resolve(__dirname, '..', 'bin', 'node-gyp.js') var execFileSync = child_process.execFileSync || require('./process-exec-sync') var execFile = child_process.execFile -function runHello() { +function runHello(hostProcess) { + if (!hostProcess) { + hostProcess = process.execPath + } var testCode = "console.log(require('hello_world').hello())" - return execFileSync(process.execPath, ['-e', testCode], { cwd: __dirname }).toString() + console.log('running ', hostProcess); + return execFileSync(hostProcess, ['-e', testCode], { cwd: __dirname }).toString() } function getEncoding() { @@ -111,3 +116,29 @@ test('build simple addon in path with non-ascii characters', function (t) { proc.stdout.setEncoding('utf-8') proc.stderr.setEncoding('utf-8') }) + +test('addon works with renamed host executable', function (t) { + // No `fs.copyFileSync` before node8. + if (process.version.substr(1).split('.')[0] < 8) { + t.skip("skipping test for old node version"); + t.end(); + return; + } + + t.plan(3) + + var notNodePath = path.join(os.tmpdir(), 'notnode' + path.extname(process.execPath)) + fs.copyFileSync(process.execPath, notNodePath) + + var cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose'] + var proc = execFile(process.execPath, cmd, function (err, stdout, stderr) { + var logLines = stderr.toString().trim().split(/\r?\n/) + var lastLine = logLines[logLines.length-1] + t.strictEqual(err, null) + t.strictEqual(lastLine, 'gyp info ok', 'should end in ok') + t.strictEqual(runHello(notNodePath).trim(), 'world') + fs.unlinkSync(notNodePath) + }) + proc.stdout.setEncoding('utf-8') + proc.stderr.setEncoding('utf-8') +}) From 49ab79d2213d62613aa4494bb516fd3a2897e0b0 Mon Sep 17 00:00:00 2001 From: Refael Ackermann Date: Sat, 19 Aug 2017 23:18:01 -0400 Subject: [PATCH 018/551] python: more informative error MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR-URL: https://github.com/nodejs/node-gyp/pull/1269 Refs: https://github.com/nodejs/node-gyp/pull/1582 Reviewed-By: Rod Vagg Reviewed-By: João Reis --- .eslintrc.yaml | 1 + .jshintrc | 2 +- bin/node-gyp.js | 8 ++++++-- lib/configure.js | 35 ++++++++++++++++++++++++----------- test/test-find-python.js | 21 ++++++++++++--------- 5 files changed, 44 insertions(+), 23 deletions(-) diff --git a/.eslintrc.yaml b/.eslintrc.yaml index 659dce8810..f808ce2d9a 100644 --- a/.eslintrc.yaml +++ b/.eslintrc.yaml @@ -1,5 +1,6 @@ --- env: node: true + es6: true rules: no-unused-vars: error diff --git a/.jshintrc b/.jshintrc index 52475ba2e6..4b8521dac0 100644 --- a/.jshintrc +++ b/.jshintrc @@ -1,7 +1,7 @@ { "asi": true, "laxcomma": true, - "es5": true, + "esversion": 6, "node": true, "strict": false } diff --git a/bin/node-gyp.js b/bin/node-gyp.js index 8c9b6169bb..55ff48e908 100755 --- a/bin/node-gyp.js +++ b/bin/node-gyp.js @@ -79,8 +79,12 @@ function run () { prog.commands[command.name](command.args, function (err) { if (err) { log.error(command.name + ' error') - log.error('stack', err.stack) - errorMessage() + if (err.noPython) { + log.error(err.message) + } else { + log.error('stack', err.stack) + errorMessage() + } log.error('not ok') return process.exit(1) } diff --git a/lib/configure.js b/lib/configure.js index 33bb7e746a..0a7d5ab53d 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -449,7 +449,7 @@ PythonFinder.prototype = { '`%s -c "' + args[1] + '"` returned: %j', this.python, stdout) var version = stdout.trim() - var range = semver.Range('>=2.5.0 <3.0.0') + var range = semver.Range('>=2.6.0 <3.0.0') var valid = false try { valid = range.test(version) @@ -467,19 +467,32 @@ PythonFinder.prototype = { }, failNoPython: function failNoPython () { - var errmsg = - 'Can\'t find Python executable "' + this.python + - '", you can set the PYTHON env variable.' - this.callback(new Error(errmsg)) + const err = new Error( + '\n******************************************************************\n' + + `node-gyp can't use "${this.python}",\n` + + 'It is recommended that you install python 2.7, set the PYTHON env,\n' + + 'or use the --python switch to point to a Python >= v2.6.0 & < 3.0.0.\n' + + 'For more information consult the documentation at:\n' + + 'https://github.com/nodejs/node-gyp#installation\n' + + '***********************************************************************' + ); + err.noPython = true; + this.callback(err) }, failPythonVersion: function failPythonVersion (badVersion) { - var errmsg = - 'Python executable "' + this.python + - '" is v' + badVersion + ', which is not supported by gyp.\n' + - 'You can pass the --python switch to point to ' + - 'Python >= v2.5.0 & < 3.0.0.' - this.callback(new Error(errmsg)) + const err = new Error( + '\n******************************************************************\n' + + `Python executable "${this.python}" is v${badVersion}\n` + + 'this version is not supported by GYP and hence by node-gyp.\n' + + 'It is recommended that you install python 2.7, set the PYTHON env,\n' + + 'or use the --python switch to point to a Python >= v2.6.0 & < 3.0.0.\n' + + 'For more information consult the documentation at:\n' + + 'https://github.com/nodejs/node-gyp#installation\n' + + '***********************************************************************' + ); + err.noPython = true; + this.callback(err) }, // Called on Windows when "python" isn't available in the current $PATH. diff --git a/test/test-find-python.js b/test/test-find-python.js index 250e37f4fe..c3209e13e6 100644 --- a/test/test-find-python.js +++ b/test/test-find-python.js @@ -74,7 +74,7 @@ test('find python - python', function (t) { }) test('find python - python too old', function (t) { - t.plan(4) + t.plan(5) var f = new TestPythonFinder('python', done) f.which = function(program, cb) { @@ -89,12 +89,13 @@ test('find python - python too old', function (t) { f.checkPython() function done(err) { - t.ok(/is not supported by gyp/.test(err)) + t.ok(/this version is not supported by GYP/.test(err)) + t.ok(/For more information consult the documentation/.test(err)) } }) test('find python - python too new', function (t) { - t.plan(4) + t.plan(5) var f = new TestPythonFinder('python', done) f.which = function(program, cb) { @@ -109,7 +110,8 @@ test('find python - python too new', function (t) { f.checkPython() function done(err) { - t.ok(/is not supported by gyp/.test(err)) + t.ok(/this version is not supported by GYP/.test(err)) + t.ok(/For more information consult the documentation/.test(err)) } }) @@ -124,7 +126,7 @@ test('find python - no python', function (t) { f.checkPython() function done(err) { - t.ok(/Can't find Python executable/.test(err)) + t.ok(/For more information consult the documentation/.test(err)) } }) @@ -171,7 +173,7 @@ test('find python - no python2, no python, unix', function (t) { f.checkPython() function done(err) { - t.ok(/Can't find Python executable/.test(err)) + t.ok(/For more information consult the documentation/.test(err)) } }) @@ -242,7 +244,7 @@ test('find python - python 3, use python launcher', function (t) { test('find python - python 3, use python launcher, python 2 too old', function (t) { - t.plan(9) + t.plan(10) var f = new TestPythonFinder('python', done) f.checkedPythonLauncher = false @@ -272,7 +274,8 @@ test('find python - python 3, use python launcher, python 2 too old', f.checkPython() function done(err) { - t.ok(/is not supported by gyp/.test(err)) + t.ok(/this version is not supported by GYP/.test(err)) + t.ok(/For more information consult the documentation/.test(err)) } }) @@ -334,6 +337,6 @@ test('find python - no python, no python launcher, bad guess', function (t) { f.checkPython() function done(err) { - t.ok(/Can't find Python executable/.test(err)) + t.ok(/For more information consult the documentation/.test(err)) } }) From 43031fadcb6589949769febd70d54645b2d1b203 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Wed, 24 Oct 2018 05:33:55 +0100 Subject: [PATCH 019/551] python: clean-up detection Try everything until Python is found. PR-URL: https://github.com/nodejs/node-gyp/pull/1582 Reviewed-By: Rod Vagg --- bin/node-gyp.js | 8 +- lib/configure.js | 370 ++++++++++++++++++++++++++------------- test/test-find-python.js | 358 ++++++++++++++++++------------------- 3 files changed, 434 insertions(+), 302 deletions(-) diff --git a/bin/node-gyp.js b/bin/node-gyp.js index 55ff48e908..8c9b6169bb 100755 --- a/bin/node-gyp.js +++ b/bin/node-gyp.js @@ -79,12 +79,8 @@ function run () { prog.commands[command.name](command.args, function (err) { if (err) { log.error(command.name + ' error') - if (err.noPython) { - log.error(err.message) - } else { - log.error('stack', err.stack) - errorMessage() - } + log.error('stack', err.stack) + errorMessage() log.error('not ok') return process.exit(1) } diff --git a/lib/configure.js b/lib/configure.js index 0a7d5ab53d..32d77523ca 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -9,7 +9,6 @@ var fs = require('graceful-fs') , path = require('path') , log = require('npmlog') , os = require('os') - , which = require('which') , semver = require('semver') , mkdirp = require('mkdirp') , cp = require('child_process') @@ -24,14 +23,14 @@ if (win) exports.usage = 'Generates ' + (win ? 'MSVC project files' : 'a Makefile') + ' for the current module' function configure (gyp, argv, callback) { - var python = gyp.opts.python || process.env.PYTHON || 'python2' + var python , buildDir = path.resolve('build') , configNames = [ 'config.gypi', 'common.gypi' ] , configs = [] , nodeDir , release = processRelease(argv, gyp, process.version, process.release) - findPython(python, function (err, found) { + findPython(gyp.opts.python, function (err, found) { if (err) { callback(err) } else { @@ -363,48 +362,156 @@ function findAccessibleSync (logprefix, dir, candidates) { return undefined } -function PythonFinder(python, callback) { +function PythonFinder(configPython, callback) { this.callback = callback - this.python = python + this.configPython = configPython + this.errorLog = [] } PythonFinder.prototype = { - checkPythonLauncherDepth: 0, - env: process.env, + log: logWithPrefix(log, 'find Python'), + argsExecutable: ['-c', 'import sys; print(sys.executable);'], + argsVersion: ['-c', 'import sys; print("%s.%s.%s" % sys.version_info[:3]);'], + semverRange: '>=2.6.0 <3.0.0', + + // These can be overridden for testing: execFile: cp.execFile, - log: log, - resolve: path.win32 && path.win32.resolve || path.resolve, - stat: fs.stat, - which: which, + env: process.env, win: win, + pyLauncher: 'py.exe', + defaultLocation: path.join(process.env.SystemDrive || 'C:', 'Python27', + 'python.exe'), + + // Logs a message at verbose level, but also saves it to be displayed later + // at error level if an error occurs. This should help diagnose the problem. + addLog: function addLog(message) { + this.log.verbose(message) + this.errorLog.push(message) + }, + + + // Find Python by trying a sequence of possibilities. + // Ignore errors, keep trying until Python is found. + findPython: function findPython() { + const SKIP=0, FAIL=1 + const toCheck = [ + { + before: () => { + if (!this.configPython) { + this.addLog( + 'Python is not set from command line or npm configuration') + return SKIP + } + this.addLog('checking Python explicitly set from command line or ' + + 'npm configuration') + this.addLog('- "--python=" or "npm config get python" is ' + + `"${this.configPython}"`) + }, + check: this.checkCommand, + arg: this.configPython, + }, + { + before: () => { + if (!this.env.PYTHON) { + this.addLog('Python is not set from environment variable PYTHON') + return SKIP + } + this.addLog( + 'checking Python explicitly set from environment variable PYTHON') + this.addLog(`- process.env.PYTHON is "${this.env.PYTHON}"`) + }, + check: this.checkCommand, + arg: this.env.PYTHON, + }, + { + before: () => { this.addLog('checking if "python2" can be used') }, + check: this.checkCommand, + arg: 'python2', + }, + { + before: () => { this.addLog('checking if "python" can be used') }, + check: this.checkCommand, + arg: 'python', + }, + { + before: () => { + if (!this.win) { + // Everything after this is Windows specific + return FAIL + } + this.addLog( + 'checking if the py launcher can be used to find Python 2') + }, + check: this.checkPyLauncher, + }, + { + before: () => { + this.addLog( + 'checking if Python 2 is installed in the default location') + }, + check: this.checkExecPath, + arg: this.defaultLocation, + }, + ] + + function runChecks(err) { + this.log.silly('runChecks: err = %j', err && err.stack || err) + + const check = toCheck.shift() + if (!check) { + return this.fail() + } + + const before = check.before.apply(this) + if (before === SKIP) { + return runChecks.apply(this) + } + if (before === FAIL) { + return this.fail() + } + + const args = [ runChecks.bind(this) ] + if (check.arg) { + args.unshift(check.arg) + } + check.check.apply(this, args) + } + + runChecks.apply(this) + }, + + + // Check if command is a valid Python to use. + // Will exit the Python finder on success. + // If on Windows, run in a CMD shell to support BAT/CMD launchers. + checkCommand: function checkCommand (command, errorCallback) { + var exec = command + var args = this.argsExecutable + var shell = false + if (this.win) { + // Arguments have to be manually quoted + exec = `"${exec}"` + args = args.map(a => `"${a}"`) + shell = true + } - checkPython: function checkPython () { - this.log.verbose('check python', - 'checking for Python executable "%s" in the PATH', - this.python) - this.which(this.python, function (err, execPath) { + this.log.verbose(`- executing "${command}" to get executable path`) + this.run(exec, args, shell, function (err, execPath) { + // Possible outcomes: + // - Error: not in PATH, not executable or execution fails + // - Gibberish: the next command to check version will fail + // - Absolute path to executable if (err) { - this.log.verbose('`which` failed', this.python, err) - if (this.python === 'python2') { - this.python = 'python' - return this.checkPython() - } - if (this.win) { - this.checkPythonLauncher() - } else { - this.failNoPython() - } - } else { - this.log.verbose('`which` succeeded', this.python, execPath) - // Found the `python` executable, and from now on we use it explicitly. - // This solves #667 and #750 (`execFile` won't run batch files - // (*.cmd, and *.bat)) - this.python = execPath - this.checkPythonVersion() + this.addLog(`- "${command}" is not in PATH or produced an error`) + return errorCallback(err) } + this.addLog(`- executable path is "${execPath}"`) + this.checkExecPath(execPath, errorCallback) }.bind(this)) }, + // Check if the py launcher can find a valid Python to use. + // Will exit the Python finder on success. // Distributions of Python on Windows by default install with the "py.exe" // Python launcher which is more likely to exist than the Python executable // being in the $PATH. @@ -413,114 +520,137 @@ PythonFinder.prototype = { // the first command line argument. Since "py.exe -2" would be an invalid // executable for "execFile", we have to use the launcher to figure out // where the actual "python.exe" executable is located. - checkPythonLauncher: function checkPythonLauncher () { - this.checkPythonLauncherDepth += 1 - + checkPyLauncher: function checkPyLauncher (errorCallback) { this.log.verbose( - 'could not find "' + this.python + '". checking python launcher') - var env = extend({}, this.env) - env.TERM = 'dumb' - - var launcherArgs = ['-2', '-c', 'import sys; print sys.executable'] - this.execFile('py.exe', launcherArgs, { env: env }, function (err, stdout) { + `- executing "${this.pyLauncher}" to get Python 2 executable path`) + this.run(this.pyLauncher, ['-2', ...this.argsExecutable], false, + function (err, execPath) { + // Possible outcomes: same as checkCommand if (err) { - this.guessPython() - } else { - this.python = stdout.trim() - this.log.verbose('check python launcher', - 'python executable found: %j', - this.python) - this.checkPythonVersion() + this.addLog( + `- "${this.pyLauncher}" is not in PATH or produced an error`) + return errorCallback(err) } - this.checkPythonLauncherDepth -= 1 + this.addLog(`- executable path is "${execPath}"`) + this.checkExecPath(execPath, errorCallback) }.bind(this)) }, - checkPythonVersion: function checkPythonVersion () { - var args = ['-c', 'import sys; print("%s.%s.%s" % sys.version_info[:3]);'] - var env = extend({}, this.env) - env.TERM = 'dumb' - - this.execFile(this.python, args, { env: env }, function (err, stdout) { + // Check if a Python executable is the correct version to use. + // Will exit the Python finder on success. + checkExecPath: function checkExecPath (execPath, errorCallback) { + this.log.verbose(`- executing "${execPath}" to get version`) + this.run(execPath, this.argsVersion, false, function (err, version) { + // Possible outcomes: + // - Error: executable can not be run (likely meaning the command wasn't + // a Python executable and the previous command produced gibberish) + // - Gibberish: somehow the last command produced an executable path, + // this will fail when verifying the version + // - Version of the Python executable if (err) { - return this.callback(err) + this.addLog(`- "${execPath}" could not be run`) + return errorCallback(err) } - this.log.verbose('check python version', - '`%s -c "' + args[1] + '"` returned: %j', - this.python, stdout) - var version = stdout.trim() - var range = semver.Range('>=2.6.0 <3.0.0') + this.addLog(`- version is "${version}"`) + + const range = semver.Range(this.semverRange) var valid = false try { valid = range.test(version) - } catch (e) { - this.log.silly('range.test() error', e) + } catch (err) { + this.log.silly('range.test() threw:\n%s', err.stack) + this.addLog(`- "${execPath}" does not have a valid version`) + this.addLog('- is it a Python executable?') + return errorCallback(err) } - if (valid) { - this.callback(null, this.python) - } else if (this.win && this.checkPythonLauncherDepth === 0) { - this.checkPythonLauncher() - } else { - this.failPythonVersion(version) + + if (!valid) { + this.addLog(`- version is ${version} - should be ${this.semverRange}`) + this.addLog('- THIS VERSION OF PYTHON IS NOT SUPPORTED') + return errorCallback(new Error( + `Found unsupported Python version ${version}`)) } + this.succeed(execPath, version) }.bind(this)) }, - failNoPython: function failNoPython () { - const err = new Error( - '\n******************************************************************\n' + - `node-gyp can't use "${this.python}",\n` + - 'It is recommended that you install python 2.7, set the PYTHON env,\n' + - 'or use the --python switch to point to a Python >= v2.6.0 & < 3.0.0.\n' + - 'For more information consult the documentation at:\n' + - 'https://github.com/nodejs/node-gyp#installation\n' + - '***********************************************************************' - ); - err.noPython = true; - this.callback(err) + // Run an executable or shell command, trimming the output. + run: function run(exec, args, shell, callback) { + var env = extend({}, this.env) + env.TERM = 'dumb' + const opts = { env: env, shell: shell } + + this.log.silly('execFile: exec = %j', exec) + this.log.silly('execFile: args = %j', args) + this.log.silly('execFile: opts = %j', opts) + try { + this.execFile(exec, args, opts, execFileCallback.bind(this)) + } catch (err) { + this.log.silly('execFile: threw:\n%s', err.stack) + return callback(err) + } + + function execFileCallback(err, stdout, stderr) { + this.log.silly('execFile result: err = %j', err && err.stack || err) + this.log.silly('execFile result: stdout = %j', stdout) + this.log.silly('execFile result: stderr = %j', stderr) + if (err) { + return callback(err) + } + const execPath = stdout.trim() + callback(null, execPath) + } }, - failPythonVersion: function failPythonVersion (badVersion) { - const err = new Error( - '\n******************************************************************\n' + - `Python executable "${this.python}" is v${badVersion}\n` + - 'this version is not supported by GYP and hence by node-gyp.\n' + - 'It is recommended that you install python 2.7, set the PYTHON env,\n' + - 'or use the --python switch to point to a Python >= v2.6.0 & < 3.0.0.\n' + - 'For more information consult the documentation at:\n' + - 'https://github.com/nodejs/node-gyp#installation\n' + - '***********************************************************************' - ); - err.noPython = true; - this.callback(err) + + succeed: function succeed(execPath, version) { + this.log.info(`using Python version ${version} found at "${execPath}"`) + process.nextTick(this.callback.bind(null, null, execPath)) }, - // Called on Windows when "python" isn't available in the current $PATH. - // We are going to check if "%SystemDrive%\python27\python.exe" exists. - guessPython: function guessPython () { - this.log.verbose('could not find "' + this.python + '". guessing location') - var rootDir = this.env.SystemDrive || 'C:\\' - if (rootDir[rootDir.length - 1] !== '\\') { - rootDir += '\\' - } - var pythonPath = this.resolve(rootDir, 'Python27', 'python.exe') - this.log.verbose('ensuring that file exists:', pythonPath) - this.stat(pythonPath, function (err) { - if (err) { - if (err.code == 'ENOENT') { - this.failNoPython() - } else { - this.callback(err) - } - return - } - this.python = pythonPath - this.checkPythonVersion() - }.bind(this)) + fail: function fail() { + const errorLog = this.errorLog.join('\n') + + const pathExample = this.win ? 'C:\\Path\\To\\python.exe' : + '/path/to/pythonexecutable' + // For Windows 80 col console, use up to the column before the one marked + // with X (total 79 chars including logger prefix, 58 chars usable here): + // X + const info = [ + '**********************************************************', + 'You need to install the latest version of Python 2.7.', + 'Node-gyp should be able to find and use Python. If not,', + 'you can try one of the following options:', + `- Use the switch --python="${pathExample}"`, + ' (accepted by both node-gyp and npm)', + '- Set the environment variable PYTHON', + '- Set the npm configuration variable python:', + ` npm config set python "${pathExample}"`, + 'For more information consult the documentation at:', + 'https://github.com/nodejs/node-gyp#installation', + '**********************************************************', + ].join('\n') + + this.log.error(`\n${errorLog}\n\n${info}\n`) + process.nextTick(this.callback.bind(null, new Error ( + 'Could not find any Python 2 installation to use'))) }, } -function findPython (python, callback) { - var finder = new PythonFinder(python, callback) - finder.checkPython() +function findPython (configPython, callback) { + var finder = new PythonFinder(configPython, callback) + finder.findPython() +} + +function logWithPrefix (log, prefix) { + function setPrefix(logFunction) { + return (...args) => logFunction.apply(null, [prefix, ...args]) + } + return { + silly: setPrefix(log.silly), + verbose: setPrefix(log.verbose), + info: setPrefix(log.info), + warn: setPrefix(log.warn), + error: setPrefix(log.error), + } } diff --git a/test/test-find-python.js b/test/test-find-python.js index c3209e13e6..f3c14ce659 100644 --- a/test/test-find-python.js +++ b/test/test-find-python.js @@ -1,7 +1,6 @@ 'use strict' var test = require('tape') -var path = require('path') var configure = require('../lib/configure') var execFile = require('child_process').execFile var PythonFinder = configure.test.PythonFinder @@ -9,7 +8,7 @@ var PythonFinder = configure.test.PythonFinder test('find python', function (t) { t.plan(4) - configure.test.findPython('python', function (err, found) { + configure.test.findPython(null, function (err, found) { t.strictEqual(err, null) var proc = execFile(found, ['-V'], function (err, stdout, stderr) { t.strictEqual(err, null) @@ -23,183 +22,188 @@ test('find python', function (t) { function poison(object, property) { function fail() { - throw new Error('Property ' + property + ' should not have been accessed.') + console.error(Error(`Property ${property} should not have been accessed.`)) + process.abort() } var descriptor = { - configurable: true, + configurable: false, enumerable: false, - writable: true, - getter: fail, - setter: fail, + get: fail, + set: fail, } Object.defineProperty(object, property, descriptor) } -// Work around a v0.10.x CI issue where path.resolve() on UNIX systems prefixes -// Windows paths with the current working directory. v0.12 and up are free of -// this issue because they use path.win32.resolve() which does the right thing. -var resolve = path.win32 && path.win32.resolve || function() { - function rstrip(s) { return s.replace(/\\+$/, '') } - return [].slice.call(arguments).map(rstrip).join('\\') -} - function TestPythonFinder() { PythonFinder.apply(this, arguments) } TestPythonFinder.prototype = Object.create(PythonFinder.prototype) -poison(TestPythonFinder.prototype, 'env') -poison(TestPythonFinder.prototype, 'execFile') -poison(TestPythonFinder.prototype, 'resolve') -poison(TestPythonFinder.prototype, 'stat') -poison(TestPythonFinder.prototype, 'which') -poison(TestPythonFinder.prototype, 'win') +// Silence npmlog - remove for debugging +TestPythonFinder.prototype.log = { + silly: () => {}, + verbose: () => {}, + info: () => {}, + warn: () => {}, + error: () => {}, +} test('find python - python', function (t) { - t.plan(5) + t.plan(6) var f = new TestPythonFinder('python', done) - f.which = function(program, cb) { - t.strictEqual(program, 'python') - cb(null, program) - } f.execFile = function(program, args, opts, cb) { - t.strictEqual(program, 'python') - t.ok(/import sys/.test(args[1])) - cb(null, '2.7.0') + f.execFile = function(program, args, opts, cb) { + poison(f, 'execFile') + t.strictEqual(program, '/path/python') + t.ok(/sys\.version_info/.test(args[1])) + cb(null, '2.7.15') + } + t.strictEqual(program, + process.platform === 'win32' ? '"python"' : 'python') + t.ok(/sys\.executable/.test(args[1])) + cb(null, '/path/python') } - f.checkPython() + f.findPython() function done(err, python) { t.strictEqual(err, null) - t.strictEqual(python, 'python') + t.strictEqual(python, '/path/python') } }) test('find python - python too old', function (t) { - t.plan(5) + t.plan(2) - var f = new TestPythonFinder('python', done) - f.which = function(program, cb) { - t.strictEqual(program, 'python') - cb(null, program) - } + var f = new TestPythonFinder(null, done) f.execFile = function(program, args, opts, cb) { - t.strictEqual(program, 'python') - t.ok(/import sys/.test(args[1])) - cb(null, '2.3.4') + if (/sys\.executable/.test(args[args.length-1])) { + cb(null, '/path/python') + } else if (/sys\.version_info/.test(args[args.length-1])) { + cb(null, '2.3.4') + } else { + t.fail() + } } - f.checkPython() + f.findPython() function done(err) { - t.ok(/this version is not supported by GYP/.test(err)) - t.ok(/For more information consult the documentation/.test(err)) + t.ok(/Could not find any Python/.test(err)) + t.ok(/not supported/i.test(f.errorLog)) } }) test('find python - python too new', function (t) { - t.plan(5) + t.plan(2) - var f = new TestPythonFinder('python', done) - f.which = function(program, cb) { - t.strictEqual(program, 'python') - cb(null, program) - } + var f = new TestPythonFinder(null, done) f.execFile = function(program, args, opts, cb) { - t.strictEqual(program, 'python') - t.ok(/import sys/.test(args[1])) - cb(null, '3.0.0') + if (/sys\.executable/.test(args[args.length-1])) { + cb(null, '/path/python') + } else if (/sys\.version_info/.test(args[args.length-1])) { + cb(null, '3.0.0') + } else { + t.fail() + } } - f.checkPython() + f.findPython() function done(err) { - t.ok(/this version is not supported by GYP/.test(err)) - t.ok(/For more information consult the documentation/.test(err)) + t.ok(/Could not find any Python/.test(err)) + t.ok(/not supported/i.test(f.errorLog)) } }) test('find python - no python', function (t) { t.plan(2) - var f = new TestPythonFinder('python', done) - f.which = function(program, cb) { - t.strictEqual(program, 'python') - cb(new Error('not found')) + var f = new TestPythonFinder(null, done) + f.execFile = function(program, args, opts, cb) { + if (/sys\.executable/.test(args[args.length-1])) { + cb(new Error('not found')) + } else if (/sys\.version_info/.test(args[args.length-1])) { + cb(new Error('not a Python executable')) + } else { + t.fail() + } } - f.checkPython() + f.findPython() function done(err) { - t.ok(/For more information consult the documentation/.test(err)) + t.ok(/Could not find any Python/.test(err)) + t.ok(/not in PATH/.test(f.errorLog)) } }) test('find python - no python2', function (t) { - t.plan(6) + t.plan(2) - var f = new TestPythonFinder('python2', done) - f.which = function(program, cb) { - f.which = function(program, cb) { - t.strictEqual(program, 'python') - cb(null, program) - } - t.strictEqual(program, 'python2') - cb(new Error('not found')) - } + var f = new TestPythonFinder(null, done) f.execFile = function(program, args, opts, cb) { - t.strictEqual(program, 'python') - t.ok(/import sys/.test(args[1])) - cb(null, '2.7.0') + if (/sys\.executable/.test(args[args.length-1])) { + if (program == 'python2') { + cb(new Error('not found')) + } else { + cb(null, '/path/python') + } + } else if (/sys\.version_info/.test(args[args.length-1])) { + cb(null, '2.7.14') + } else { + t.fail() + } } - f.checkPython() + f.findPython() function done(err, python) { t.strictEqual(err, null) - t.strictEqual(python, 'python') + t.strictEqual(python, '/path/python') } }) test('find python - no python2, no python, unix', function (t) { - t.plan(3) + t.plan(2) - var f = new TestPythonFinder('python2', done) - poison(f, 'checkPythonLauncher') + var f = new TestPythonFinder(null, done) + f.checkPyLauncher = t.fail f.win = false - f.which = function(program, cb) { - f.which = function(program, cb) { - t.strictEqual(program, 'python') + f.execFile = function(program, args, opts, cb) { + if (/sys\.executable/.test(args[args.length-1])) { cb(new Error('not found')) + } else { + t.fail() } - t.strictEqual(program, 'python2') - cb(new Error('not found')) } - f.checkPython() + f.findPython() function done(err) { - t.ok(/For more information consult the documentation/.test(err)) + t.ok(/Could not find any Python/.test(err)) + t.ok(/not in PATH/.test(f.errorLog)) } }) test('find python - no python, use python launcher', function (t) { - t.plan(8) + t.plan(4) - var f = new TestPythonFinder('python', done) - f.env = {} + var f = new TestPythonFinder(null, done) f.win = true - f.which = function(program, cb) { - t.strictEqual(program, 'python') - cb(new Error('not found')) - } f.execFile = function(program, args, opts, cb) { - f.execFile = function(program, args, opts, cb) { - t.strictEqual(program, 'Z:\\snake.exe') - t.ok(/import sys/.test(args[1])) - cb(null, '2.7.0') + if (program === 'py.exe') { + t.notEqual(args.indexOf('-2'), -1) + t.notEqual(args.indexOf('-c'), -1) + return cb(null, 'Z:\\snake.exe') + } + if (/sys\.executable/.test(args[args.length-1])) { + cb(new Error('not found')) + } else if (/sys\.version_info/.test(args[args.length-1])) { + if (program === 'Z:\\snake.exe') { + cb(null, '2.7.14') + } else { + t.fail() + } + } else { + t.fail() } - t.strictEqual(program, 'py.exe') - t.notEqual(args.indexOf('-2'), -1) - t.notEqual(args.indexOf('-c'), -1) - cb(null, 'Z:\\snake.exe') } - f.checkPython() + f.findPython() function done(err, python) { t.strictEqual(err, null) @@ -208,33 +212,34 @@ test('find python - no python, use python launcher', function (t) { }) test('find python - python 3, use python launcher', function (t) { - t.plan(10) + t.plan(4) - var f = new TestPythonFinder('python', done) - f.env = {} + var f = new TestPythonFinder(null, done) f.win = true - f.which = function(program, cb) { - t.strictEqual(program, 'python') - cb(null, program) - } f.execFile = function(program, args, opts, cb) { - f.execFile = function(program, args, opts, cb) { + if (program === 'py.exe') { f.execFile = function(program, args, opts, cb) { - t.strictEqual(program, 'Z:\\snake.exe') - t.ok(/import sys/.test(args[1])) - cb(null, '2.7.0') + poison(f, 'execFile') + if (/sys\.version_info/.test(args[args.length-1])) { + cb(null, '2.7.14') + } else { + t.fail() + } } - t.strictEqual(program, 'py.exe') t.notEqual(args.indexOf('-2'), -1) t.notEqual(args.indexOf('-c'), -1) - cb(null, 'Z:\\snake.exe') + return cb(null, 'Z:\\snake.exe') + } + if (/sys\.executable/.test(args[args.length-1])) { + cb(null, '/path/python') + } else if (/sys\.version_info/.test(args[args.length-1])) { + cb(null, '3.0.0') + } else { + t.fail() } - t.strictEqual(program, 'python') - t.ok(/import sys/.test(args[1])) - cb(null, '3.0.0') } - f.checkPython() + f.findPython() function done(err, python) { t.strictEqual(err, null) @@ -244,99 +249,100 @@ test('find python - python 3, use python launcher', function (t) { test('find python - python 3, use python launcher, python 2 too old', function (t) { - t.plan(10) + t.plan(6) - var f = new TestPythonFinder('python', done) - f.checkedPythonLauncher = false - f.env = {} + var f = new TestPythonFinder(null, done) f.win = true - f.which = function(program, cb) { - t.strictEqual(program, 'python') - cb(null, program) - } f.execFile = function(program, args, opts, cb) { - f.execFile = function(program, args, opts, cb) { + if (program === 'py.exe') { f.execFile = function(program, args, opts, cb) { - t.strictEqual(program, 'Z:\\snake.exe') - t.ok(/import sys/.test(args[1])) - cb(null, '2.3.4') + if (/sys\.version_info/.test(args[args.length-1])) { + f.execFile = function(program, args, opts, cb) { + if (/sys\.version_info/.test(args[args.length-1])) { + poison(f, 'execFile') + t.strictEqual(program, f.defaultLocation) + cb(new Error('not found')) + } else { + t.fail() + } + } + t.strictEqual(program, 'Z:\\snake.exe') + cb(null, '2.3.4') + } else { + t.fail() + } } - t.strictEqual(program, 'py.exe') t.notEqual(args.indexOf('-2'), -1) t.notEqual(args.indexOf('-c'), -1) - cb(null, 'Z:\\snake.exe') + return cb(null, 'Z:\\snake.exe') + } + if (/sys\.executable/.test(args[args.length-1])) { + cb(null, '/path/python') + } else if (/sys\.version_info/.test(args[args.length-1])) { + cb(null, '3.0.0') + } else { + t.fail() } - t.strictEqual(program, 'python') - t.ok(/import sys/.test(args[1])) - cb(null, '3.0.0') } - f.checkPython() + f.findPython() function done(err) { - t.ok(/this version is not supported by GYP/.test(err)) - t.ok(/For more information consult the documentation/.test(err)) + t.ok(/Could not find any Python/.test(err)) + t.ok(/not supported/i.test(f.errorLog)) } }) test('find python - no python, no python launcher, good guess', function (t) { - t.plan(6) + t.plan(4) var re = /C:[\\\/]Python27[\\\/]python[.]exe/ - var f = new TestPythonFinder('python', done) - f.env = {} + var f = new TestPythonFinder(null, done) f.win = true - f.which = function(program, cb) { - t.strictEqual(program, 'python') - cb(new Error('not found')) - } f.execFile = function(program, args, opts, cb) { - f.execFile = function(program, args, opts, cb) { - t.ok(re.test(program)) - t.ok(/import sys/.test(args[1])) - cb(null, '2.7.0') + if (program === 'py.exe') { + f.execFile = function(program, args, opts, cb) { + poison(f, 'execFile') + t.ok(re.test(program)) + t.ok(/sys\.version_info/.test(args[args.length-1])) + cb(null, '2.7.14') + } + return cb(new Error('not found')) + } + if (/sys\.executable/.test(args[args.length-1])) { + cb(new Error('not found')) + } else { + t.fail() } - t.strictEqual(program, 'py.exe') - cb(new Error('not found')) - } - f.resolve = resolve - f.stat = function(path, cb) { - t.ok(re.test(path)) - cb(null, {}) } - f.checkPython() + f.findPython() function done(err, python) { + t.strictEqual(err, null) t.ok(re.test(python)) } }) test('find python - no python, no python launcher, bad guess', function (t) { - t.plan(4) + t.plan(2) - var f = new TestPythonFinder('python', done) - f.env = { SystemDrive: 'Z:\\' } + var f = new TestPythonFinder(null, done) f.win = true - f.which = function(program, cb) { - t.strictEqual(program, 'python') - cb(new Error('not found')) - } f.execFile = function(program, args, opts, cb) { - t.strictEqual(program, 'py.exe') - cb(new Error('not found')) - } - f.resolve = resolve - f.stat = function(path, cb) { - t.ok(/Z:[\\\/]Python27[\\\/]python.exe/.test(path)) - var err = new Error('not found') - err.code = 'ENOENT' - cb(err) + if (/sys\.executable/.test(args[args.length-1])) { + cb(new Error('not found')) + } else if (/sys\.version_info/.test(args[args.length-1])) { + cb(new Error('not a Python executable')) + } else { + t.fail() + } } - f.checkPython() + f.findPython() function done(err) { - t.ok(/For more information consult the documentation/.test(err)) + t.ok(/Could not find any Python/.test(err)) + t.ok(/not in PATH/.test(f.errorLog)) } }) From cca2d667275178b7f5756fa3209d3398b170d48f Mon Sep 17 00:00:00 2001 From: "Taylor D. Lee" Date: Mon, 26 Nov 2018 08:40:11 -0600 Subject: [PATCH 020/551] doc: python info needs own header PR-URL: https://github.com/nodejs/node-gyp/pull/1245 Reviewed-By: Gibson Fahnestock Reviewed-By: Richard Lau Reviewed-By: Refael Ackermann --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index f378616964..d9222d0ce1 100644 --- a/README.md +++ b/README.md @@ -55,8 +55,10 @@ Install tools and configuration manually: If the above steps didn't work for you, please visit [Microsoft's Node.js Guidelines for Windows](https://github.com/Microsoft/nodejs-guidelines/blob/master/windows-environment.md#compiling-native-addon-modules) for additional tips. +### Configuring Python Dependency + If you have multiple Python versions installed, you can identify which Python -version `node-gyp` uses by setting the '--python' variable: +version `node-gyp` uses by setting the `--python` variable: ``` bash $ node-gyp --python /path/to/python2.7 From c515912d08b058dc94b25dc3481fc851249d6899 Mon Sep 17 00:00:00 2001 From: Bartosz Sosnowski Date: Wed, 28 Nov 2018 21:26:30 +0100 Subject: [PATCH 021/551] doc: improve issue template MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Suggest using --verbose npm switch when providing logs. Hopefully, better direct users to use backticks correctly. PR-URL: https://github.com/nodejs/node-gyp/pull/1618 Reviewed-By: Richard Lau Reviewed-By: João Reis Reviewed-By: Ben Noordhuis --- .github/ISSUE_TEMPLATE.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md index dbd053a90b..b5bed7fdd1 100644 --- a/.github/ISSUE_TEMPLATE.md +++ b/.github/ISSUE_TEMPLATE.md @@ -14,10 +14,11 @@ that module's issue tracker (`npm issues modulename`).
Verbose output (from npm or node-gyp): - - ``` - +Paste your log here, between the backticks. It can be: + - npm --verbose output, + - or contents of npm-debug.log, + - or output of node-gyp rebuild --verbose. ```
From a6e0a6c7edcf30cbd6348e5d8e4bb05399c7c11b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Paul=20Mar=C3=A9chal?= Date: Wed, 6 Feb 2019 10:27:59 -0500 Subject: [PATCH 022/551] gyp: move compile_commands_json MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit It isn't possible to access `tools/gyp/pylib/gyp/generator/compile_commands_json.py` using the `--format` option. This commit moves the file in a place where it can be accessed. Fixes: https://github.com/nodejs/node-gyp/issues/1526 PR-URL: https://github.com/nodejs/node-gyp/pull/1661 Reviewed-By: Ben Noordhuis Signed-off-by: Paul Maréchal --- {tools/gyp => gyp}/pylib/gyp/generator/compile_commands_json.py | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {tools/gyp => gyp}/pylib/gyp/generator/compile_commands_json.py (100%) diff --git a/tools/gyp/pylib/gyp/generator/compile_commands_json.py b/gyp/pylib/gyp/generator/compile_commands_json.py similarity index 100% rename from tools/gyp/pylib/gyp/generator/compile_commands_json.py rename to gyp/pylib/gyp/generator/compile_commands_json.py From 721dc7d3148ab71ee283d9cb15df84d9b87b7efc Mon Sep 17 00:00:00 2001 From: Jon Kunkee Date: Thu, 24 Jan 2019 16:00:31 -0800 Subject: [PATCH 023/551] Add ARM64 to MSBuild /Platform logic MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR-URL: https://github.com/nodejs/node-gyp/pull/1655 Reviewed-By: Ben Noordhuis Reviewed-By: João Reis --- lib/build.js | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/build.js b/lib/build.js index b8389bcd7b..363bd66a5f 100644 --- a/lib/build.js +++ b/lib/build.js @@ -200,9 +200,13 @@ function build (gyp, argv, callback) { // Specify the build type, Release by default if (win) { + // Convert .gypi config target_arch to MSBuild /Platform + // Since there are many ways to state '32-bit Intel', default to it. + // N.B. msbuild's Condition string equality tests are case-insensitive. var archLower = arch.toLowerCase() var p = archLower === 'x64' ? 'x64' : - (archLower === 'arm' ? 'ARM' : 'Win32') + (archLower === 'arm' ? 'ARM' : + (archLower === 'arm64' ? 'ARM64' : 'Win32')) argv.push('/p:Configuration=' + buildType + ';Platform=' + p) if (jobs) { var j = parseInt(jobs, 10) From 997bc3c748d92723da89b3162ff7cd615a65b8c0 Mon Sep 17 00:00:00 2001 From: Jon Kunkee Date: Thu, 24 Jan 2019 16:00:55 -0800 Subject: [PATCH 024/551] readme: add ARM64 info to MSVC setup instructions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Since ARM64 build dependencies are not included in the "Desktop development with C++" workload of Visual Studio 2017, this change adds a note explaining what additional components are required to build native modules for the new Windows platform. PR-URL: https://github.com/nodejs/node-gyp/pull/1655 Reviewed-By: Ben Noordhuis Reviewed-By: João Reis --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index d9222d0ce1..01d428bc6c 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,8 @@ Install tools and configuration manually: If the above steps didn't work for you, please visit [Microsoft's Node.js Guidelines for Windows](https://github.com/Microsoft/nodejs-guidelines/blob/master/windows-environment.md#compiling-native-addon-modules) for additional tips. + To target native ARM64 Node.js on Windows 10 on ARM, add the components "Visual C++ compilers and libraries for ARM64" and "Visual C++ ATL for ARM64". + ### Configuring Python Dependency If you have multiple Python versions installed, you can identify which Python From 45e3221fd46edf7f14f2c4c640d5f8548d7568f0 Mon Sep 17 00:00:00 2001 From: cclauss Date: Mon, 21 Jan 2019 09:46:23 +0100 Subject: [PATCH 025/551] Remove an outdated workaround for Python 2.4 PR-URL: https://github.com/nodejs/node-gyp/pull/1650 Reviewed-By: Ben Noordhuis Reviewed-By: Sakthipriyan Vairamani --- gyp/pylib/gyp/MSVSNew.py | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-) diff --git a/gyp/pylib/gyp/MSVSNew.py b/gyp/pylib/gyp/MSVSNew.py index 593f0e5b0b..91fcb413a6 100644 --- a/gyp/pylib/gyp/MSVSNew.py +++ b/gyp/pylib/gyp/MSVSNew.py @@ -4,22 +4,12 @@ """New implementation of Visual Studio project generation.""" +import hashlib import os import random import gyp.common -# hashlib is supplied as of Python 2.5 as the replacement interface for md5 -# and other secure hashes. In 2.6, md5 is deprecated. Import hashlib if -# available, avoiding a deprecation warning under 2.6. Import md5 otherwise, -# preserving 2.4 compatibility. -try: - import hashlib - _new_md5 = hashlib.md5 -except ImportError: - import md5 - _new_md5 = md5.new - # Initialize random number generator random.seed() @@ -50,7 +40,7 @@ def MakeGuid(name, seed='msvs_new'): not change when the project for a target is rebuilt. """ # Calculate a MD5 signature for the seed and name. - d = _new_md5(str(seed) + str(name)).hexdigest().upper() + d = hashlib.md5(str(seed) + str(name)).hexdigest().upper() # Convert most of the signature to GUID form (discard the rest) guid = ('{' + d[:8] + '-' + d[8:12] + '-' + d[12:16] + '-' + d[16:20] + '-' + d[20:32] + '}') From 7a71d68bce49cba73ef2a86e4febbcf840872834 Mon Sep 17 00:00:00 2001 From: Bartosz Sosnowski Date: Wed, 13 Feb 2019 21:32:18 +0100 Subject: [PATCH 026/551] win: use msbuild from the configure stage MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit If node-gyp configure has set up MSBuild location use it instead the one that happens to be first on the PATH. PR-URL: https://github.com/nodejs/node-gyp/pull/1654 Fixes: https://github.com/nodejs/node-gyp/issues/1653 Reviewed-By: Richard Lau Reviewed-By: Refael Ackermann Reviewed-By: João Reis --- lib/build.js | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/lib/build.js b/lib/build.js index 363bd66a5f..eeaf680c6b 100644 --- a/lib/build.js +++ b/lib/build.js @@ -95,6 +95,13 @@ function build (gyp, argv, callback) { */ function doWhich () { + // On Windows use msbuild provided by node-gyp configure + if (win && config.variables.msbuild_path) { + command = config.variables.msbuild_path + log.verbose('using MSBuild:', command) + doBuild() + return + } // First make sure we have the build command in the PATH which(command, function (err, execPath) { if (err) { @@ -117,13 +124,6 @@ function build (gyp, argv, callback) { */ function findMsbuild () { - if (config.variables.msbuild_path) { - command = config.variables.msbuild_path - log.verbose('using MSBuild:', command) - doBuild() - return - } - log.verbose('could not find "msbuild.exe" in PATH - finding location in registry') var notfoundErr = 'Can\'t find "msbuild.exe". Do you have Microsoft Visual Studio C++ 2008+ installed?' var cmd = 'reg query "HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions" /s' From 4748f6ab75e3aa661cc67de468c6cf19c4102384 Mon Sep 17 00:00:00 2001 From: Ben Noordhuis Date: Thu, 14 Feb 2019 19:27:42 +0100 Subject: [PATCH 027/551] Remove deprecated compatibility code. * Remove support for the IOJS_ORG_MIRROR, NVM_IOJS_ORG_MIRROR and NVM_NODEJS_ORG_MIRROR enviroment variables. * Remove obsolete support for io.js. It's been out of support for over three years now. PR-URL: https://github.com/nodejs/node-gyp/pull/1670 Reviewed-By: Richard Lau --- lib/process-release.js | 32 +---- test/docker.sh | 52 -------- test/test-process-release.js | 238 ----------------------------------- 3 files changed, 5 insertions(+), 317 deletions(-) diff --git a/lib/process-release.js b/lib/process-release.js index 4805fcc7d1..8b3fda81c6 100644 --- a/lib/process-release.js +++ b/lib/process-release.js @@ -17,7 +17,6 @@ function processRelease (argv, gyp, defaultVersion, defaultRelease) { , versionSemver = semver.parse(version) , overrideDistUrl = gyp.opts['dist-url'] || gyp.opts.disturl , isDefaultVersion - , isIojs , name , distBaseUrl , baseUrl @@ -42,37 +41,16 @@ function processRelease (argv, gyp, defaultVersion, defaultRelease) { if (defaultRelease) { // v3 onward, has process.release - name = defaultRelease.name.replace(/io\.js/, 'iojs') // remove the '.' for directory naming purposes - isIojs = name === 'iojs' + name = defaultRelease.name } else { // old node or alternative --target= // semver.satisfies() doesn't like prerelease tags so test major directly - isIojs = versionSemver.major >= 1 && versionSemver.major < 4 - name = isIojs ? 'iojs' : 'node' + name = 'node' } // check for the nvm.sh standard mirror env variables - if (!overrideDistUrl) { - if (isIojs) { - if (process.env.IOJS_ORG_MIRROR) { - overrideDistUrl = process.env.IOJS_ORG_MIRROR - } else if (process.env.NVM_IOJS_ORG_MIRROR) {// remove on next semver-major - overrideDistUrl = process.env.NVM_IOJS_ORG_MIRROR - log.warn('download', - 'NVM_IOJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, ' + - 'please use IOJS_ORG_MIRROR') - } - } else { - if (process.env.NODEJS_ORG_MIRROR) { - overrideDistUrl = process.env.NODEJS_ORG_MIRROR - } else if (process.env.NVM_NODEJS_ORG_MIRROR) {// remove on next semver-major - overrideDistUrl = process.env.NVM_NODEJS_ORG_MIRROR - log.warn('download', - 'NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, ' + - 'please use NODEJS_ORG_MIRROR') - } - } - } + if (!overrideDistUrl && process.env.NODEJS_ORG_MIRROR) + overrideDistUrl = process.env.NODEJS_ORG_MIRROR if (overrideDistUrl) log.verbose('download', 'using dist-url', overrideDistUrl) @@ -80,7 +58,7 @@ function processRelease (argv, gyp, defaultVersion, defaultRelease) { if (overrideDistUrl) distBaseUrl = overrideDistUrl.replace(/\/+$/, '') else - distBaseUrl = isIojs ? 'https://iojs.org/download/release' : 'https://nodejs.org/dist' + distBaseUrl = 'https://nodejs.org/dist' distBaseUrl += '/v' + version + '/' // new style, based on process.release so we have a lot of the data we need diff --git a/test/docker.sh b/test/docker.sh index ac21aa8d75..8a9f160c0a 100755 --- a/test/docker.sh +++ b/test/docker.sh @@ -3,7 +3,6 @@ #set -e test_node_versions="0.8.28 0.10.40 0.12.7 4.3.0 5.6.0" -test_iojs_versions="1.8.4 2.4.0 3.3.0" myuid=$(id -u) mygid=$(id -g) @@ -77,25 +76,6 @@ for v in $test_node_versions; do " done -# An image for each of the io.js versions we want to test with that version installed and the latest npm -for v in $test_iojs_versions; do - setup_container "node-gyp-test/${v}" "node-gyp-test/clones" " - curl -sL https://iojs.org/dist/v${v}/iojs-v${v}-linux-x64.tar.gz | tar -zxv --strip-components=1 -C /usr/ && - npm install npm@latest -g && - node -v && npm -v - " -done - -# Run the tests for all of the test images we've created, -# we should see node-gyp doing its download, configure and run thing -# _NOTE: bignum doesn't compile on 0.8 currently so it'll fail for that version only_ -for v in $test_node_versions $test_iojs_versions; do - run_tests $v " - cd node-buffertools && npm install --loglevel=info && npm test && cd - " - # removed for now, too noisy: cd node-bignum && npm install --loglevel=info && npm test -done - # Test use of --target=x.y.z to compile against alternate versions test_download_node_version() { local run_with_ver="$1" @@ -112,9 +92,7 @@ test_download_node_version() { } test_download_node_version "0.12.7" "0.10.30/src" "0.10.30" -test_download_node_version "3.3.0" "iojs-1.8.4/src" "1.8.4" # should download the headers file -test_download_node_version "3.3.0" "iojs-3.3.0/include/node" "3.3.0" test_download_node_version "4.3.0" "4.3.0/include/node" "4.3.0" test_download_node_version "5.6.0" "5.6.0/include/node" "5.6.0" @@ -124,36 +102,6 @@ test_download_node_version "5.6.0" "5.6.0/include/node" "5.6.0" # point for tarballs # we can test whether it uses the proxy because after 2 connections the proxy will # die and therefore should not be running at the end of the test, `nc` can tell us this -run_tests "3.3.0" " - (node /node-gyp-src/test/simple-proxy.js 8080 /foobar/ https://iojs.org/dist/ &) && - cd node-buffertools && - /node-gyp-src/bin/node-gyp.js --loglevel=info --dist-url=http://localhost:8080/foobar/ rebuild && - nc -z localhost 8080 && echo -e \"\\n\\n\\033[31mFAILED TO USE LOCAL PROXY\\033[39m\\n\\n\" -" - -# REMOVE after next semver-major -run_tests "3.3.0" " - (node /node-gyp-src/test/simple-proxy.js 8080 /doobar/ https://iojs.org/dist/ &) && - cd node-buffertools && - NVM_IOJS_ORG_MIRROR=http://localhost:8080/doobar/ /node-gyp-src/bin/node-gyp.js --loglevel=info rebuild && - nc -z localhost 8080 && echo -e \"\\n\\n\\033[31mFAILED TO USE LOCAL PROXY\\033[39m\\n\\n\" -" - -# REMOVE after next semver-major -run_tests "0.12.7" " - (node /node-gyp-src/test/simple-proxy.js 8080 /boombar/ https://nodejs.org/dist/ &) && - cd node-buffertools && - NVM_NODEJS_ORG_MIRROR=http://localhost:8080/boombar/ /node-gyp-src/bin/node-gyp.js --loglevel=info rebuild && - nc -z localhost 8080 && echo -e \"\\n\\n\\033[31mFAILED TO USE LOCAL PROXY\\033[39m\\n\\n\" -" - -run_tests "3.3.0" " - (node /node-gyp-src/test/simple-proxy.js 8080 /doobar/ https://iojs.org/dist/ &) && - cd node-buffertools && - IOJS_ORG_MIRROR=http://localhost:8080/doobar/ /node-gyp-src/bin/node-gyp.js --loglevel=info rebuild && - nc -z localhost 8080 && echo -e \"\\n\\n\\033[31mFAILED TO USE LOCAL PROXY\\033[39m\\n\\n\" -" - run_tests "0.12.7" " (node /node-gyp-src/test/simple-proxy.js 8080 /boombar/ https://nodejs.org/dist/ &) && cd node-buffertools && diff --git a/test/test-process-release.js b/test/test-process-release.js index 48411ae0a7..2890e8a61c 100644 --- a/test/test-process-release.js +++ b/test/test-process-release.js @@ -187,131 +187,6 @@ test('test process release - process.release ~ node@4.1.23 / corp build', functi }) }) -test('test process release - process.version = 1.8.4', function (t) { - t.plan(2) - - var release = processRelease([], { opts: {} }, 'v1.8.4', null) - - t.equal(release.semver.version, '1.8.4') - delete release.semver - - t.deepEqual(release, { - version: '1.8.4', - name: 'iojs', - baseUrl: 'https://iojs.org/download/release/v1.8.4/', - tarballUrl: 'https://iojs.org/download/release/v1.8.4/iojs-v1.8.4.tar.gz', - shasumsUrl: 'https://iojs.org/download/release/v1.8.4/SHASUMS256.txt', - versionDir: 'iojs-1.8.4', - libUrl32: 'https://iojs.org/download/release/v1.8.4/win-x86/iojs.lib', - libUrl64: 'https://iojs.org/download/release/v1.8.4/win-x64/iojs.lib', - libPath32: 'win-x86/iojs.lib', - libPath64: 'win-x64/iojs.lib' - }) -}) - -test('test process release - process.release ~ iojs@3.2.24', function (t) { - t.plan(2) - - var release = processRelease([], { opts: {} }, 'v3.2.24', { - name: 'io.js', - headersUrl: 'https://iojs.org/download/release/v3.2.24/iojs-v3.2.24-headers.tar.gz' - }) - - t.equal(release.semver.version, '3.2.24') - delete release.semver - - t.deepEqual(release, { - version: '3.2.24', - name: 'iojs', - baseUrl: 'https://iojs.org/download/release/v3.2.24/', - tarballUrl: 'https://iojs.org/download/release/v3.2.24/iojs-v3.2.24-headers.tar.gz', - shasumsUrl: 'https://iojs.org/download/release/v3.2.24/SHASUMS256.txt', - versionDir: 'iojs-3.2.24', - libUrl32: 'https://iojs.org/download/release/v3.2.24/win-x86/iojs.lib', - libUrl64: 'https://iojs.org/download/release/v3.2.24/win-x64/iojs.lib', - libPath32: 'win-x86/iojs.lib', - libPath64: 'win-x64/iojs.lib' - }) -}) - -test('test process release - process.release ~ iojs@3.2.11 +libUrl32', function (t) { - t.plan(2) - - var release = processRelease([], { opts: {} }, 'v3.2.11', { - name: 'io.js', - headersUrl: 'https://iojs.org/download/release/v3.2.11/iojs-v3.2.11-headers.tar.gz', - libUrl: 'https://iojs.org/download/release/v3.2.11/win-x86/iojs.lib' // custom - }) - - t.equal(release.semver.version, '3.2.11') - delete release.semver - - t.deepEqual(release, { - version: '3.2.11', - name: 'iojs', - baseUrl: 'https://iojs.org/download/release/v3.2.11/', - tarballUrl: 'https://iojs.org/download/release/v3.2.11/iojs-v3.2.11-headers.tar.gz', - shasumsUrl: 'https://iojs.org/download/release/v3.2.11/SHASUMS256.txt', - versionDir: 'iojs-3.2.11', - libUrl32: 'https://iojs.org/download/release/v3.2.11/win-x86/iojs.lib', - libUrl64: 'https://iojs.org/download/release/v3.2.11/win-x64/iojs.lib', - libPath32: 'win-x86/iojs.lib', - libPath64: 'win-x64/iojs.lib' - }) -}) - -test('test process release - process.release ~ iojs@3.2.101 +libUrl64', function (t) { - t.plan(2) - - var release = processRelease([], { opts: {} }, 'v3.2.101', { - name: 'io.js', - headersUrl: 'https://iojs.org/download/release/v3.2.101/iojs-v3.2.101-headers.tar.gz', - libUrl: 'https://iojs.org/download/release/v3.2.101/win-x64/iojs.lib' // custom - }) - - t.equal(release.semver.version, '3.2.101') - delete release.semver - - t.deepEqual(release, { - version: '3.2.101', - name: 'iojs', - baseUrl: 'https://iojs.org/download/release/v3.2.101/', - tarballUrl: 'https://iojs.org/download/release/v3.2.101/iojs-v3.2.101-headers.tar.gz', - shasumsUrl: 'https://iojs.org/download/release/v3.2.101/SHASUMS256.txt', - versionDir: 'iojs-3.2.101', - libUrl32: 'https://iojs.org/download/release/v3.2.101/win-x86/iojs.lib', - libUrl64: 'https://iojs.org/download/release/v3.2.101/win-x64/iojs.lib', - libPath32: 'win-x86/iojs.lib', - libPath64: 'win-x64/iojs.lib' - }) -}) - -test('test process release - process.release ~ iojs@3.3.0 - borked win-ia32', function (t) { - t.plan(2) - - var release = processRelease([], { opts: {} }, 'v3.2.101', { - name: 'io.js', - headersUrl: 'https://iojs.org/download/release/v3.2.101/iojs-v3.2.101-headers.tar.gz', - libUrl: 'https://iojs.org/download/release/v3.2.101/win-ia32/iojs.lib' // custom - }) - - t.equal(release.semver.version, '3.2.101') - delete release.semver - - t.deepEqual(release, { - version: '3.2.101', - name: 'iojs', - baseUrl: 'https://iojs.org/download/release/v3.2.101/', - tarballUrl: 'https://iojs.org/download/release/v3.2.101/iojs-v3.2.101-headers.tar.gz', - shasumsUrl: 'https://iojs.org/download/release/v3.2.101/SHASUMS256.txt', - versionDir: 'iojs-3.2.101', - libUrl32: 'https://iojs.org/download/release/v3.2.101/win-x86/iojs.lib', - libUrl64: 'https://iojs.org/download/release/v3.2.101/win-x64/iojs.lib', - libPath32: 'win-x86/iojs.lib', - libPath64: 'win-x64/iojs.lib' - }) -}) - test('test process release - process.release ~ node@4.1.23 --target=0.10.40', function (t) { t.plan(2) @@ -337,31 +212,6 @@ test('test process release - process.release ~ node@4.1.23 --target=0.10.40', fu }) }) -test('test process release - process.release ~ node@4.1.23 --target=1.8.4', function (t) { - t.plan(2) - - var release = processRelease([], { opts: { target: '1.8.4' } }, 'v4.1.23', { - name: 'node', - headersUrl: 'https://nodejs.org/dist/v4.1.23/node-v4.1.23-headers.tar.gz' - }) - - t.equal(release.semver.version, '1.8.4') - delete release.semver - - t.deepEqual(release, { - version: '1.8.4', - name: 'iojs', - baseUrl: 'https://iojs.org/download/release/v1.8.4/', - tarballUrl: 'https://iojs.org/download/release/v1.8.4/iojs-v1.8.4.tar.gz', - shasumsUrl: 'https://iojs.org/download/release/v1.8.4/SHASUMS256.txt', - versionDir: 'iojs-1.8.4', - libUrl32: 'https://iojs.org/download/release/v1.8.4/win-x86/iojs.lib', - libUrl64: 'https://iojs.org/download/release/v1.8.4/win-x64/iojs.lib', - libPath32: 'win-x86/iojs.lib', - libPath64: 'win-x64/iojs.lib' - }) -}) - test('test process release - process.release ~ node@4.1.23 --dist-url=https://foo.bar/baz', function (t) { t.plan(2) @@ -547,91 +397,3 @@ test('test process release - NODEJS_ORG_MIRROR', function (t) { delete process.env.NODEJS_ORG_MIRROR }) - -test('test process release - NVM_NODEJS_ORG_MIRROR', function (t) { - t.plan(2) - - process.env.NVM_NODEJS_ORG_MIRROR = 'http://foo.bar' - - var release = processRelease([], { opts: {} }, 'v4.1.23', { - name: 'node', - headersUrl: 'https://nodejs.org/dist/v4.1.23/node-v4.1.23-headers.tar.gz' - }) - - t.equal(release.semver.version, '4.1.23') - delete release.semver - - t.deepEqual(release, { - version: '4.1.23', - name: 'node', - baseUrl: 'http://foo.bar/v4.1.23/', - tarballUrl: 'http://foo.bar/v4.1.23/node-v4.1.23-headers.tar.gz', - shasumsUrl: 'http://foo.bar/v4.1.23/SHASUMS256.txt', - versionDir: '4.1.23', - libUrl32: 'http://foo.bar/v4.1.23/win-x86/node.lib', - libUrl64: 'http://foo.bar/v4.1.23/win-x64/node.lib', - libPath32: 'win-x86/node.lib', - libPath64: 'win-x64/node.lib' - }) - - delete process.env.NVM_NODEJS_ORG_MIRROR -}) - -test('test process release - IOJS_ORG_MIRROR', function (t) { - t.plan(2) - - process.env.IOJS_ORG_MIRROR = 'http://foo.bar' - - var release = processRelease([], { opts: {} }, 'v3.2.24', { - name: 'io.js', - headersUrl: 'https://iojs.org/download/release/v3.2.24/iojs-v3.2.24-headers.tar.gz' - }) - - t.equal(release.semver.version, '3.2.24') - delete release.semver - - t.deepEqual(release, { - version: '3.2.24', - name: 'iojs', - baseUrl: 'http://foo.bar/v3.2.24/', - tarballUrl: 'http://foo.bar/v3.2.24/iojs-v3.2.24-headers.tar.gz', - shasumsUrl: 'http://foo.bar/v3.2.24/SHASUMS256.txt', - versionDir: 'iojs-3.2.24', - libUrl32: 'http://foo.bar/v3.2.24/win-x86/iojs.lib', - libUrl64: 'http://foo.bar/v3.2.24/win-x64/iojs.lib', - libPath32: 'win-x86/iojs.lib', - libPath64: 'win-x64/iojs.lib' - }) - - delete process.env.IOJS_ORG_MIRROR -}) - - -test('test process release - NVM_IOJS_ORG_MIRROR', function (t) { - t.plan(2) - - process.env.NVM_IOJS_ORG_MIRROR = 'http://foo.bar' - - var release = processRelease([], { opts: {} }, 'v3.2.24', { - name: 'io.js', - headersUrl: 'https://iojs.org/download/release/v3.2.24/iojs-v3.2.24-headers.tar.gz' - }) - - t.equal(release.semver.version, '3.2.24') - delete release.semver - - t.deepEqual(release, { - version: '3.2.24', - name: 'iojs', - baseUrl: 'http://foo.bar/v3.2.24/', - tarballUrl: 'http://foo.bar/v3.2.24/iojs-v3.2.24-headers.tar.gz', - shasumsUrl: 'http://foo.bar/v3.2.24/SHASUMS256.txt', - versionDir: 'iojs-3.2.24', - libUrl32: 'http://foo.bar/v3.2.24/win-x86/iojs.lib', - libUrl64: 'http://foo.bar/v3.2.24/win-x64/iojs.lib', - libPath32: 'win-x86/iojs.lib', - libPath64: 'win-x64/iojs.lib' - }) - - delete process.env.NVM_IOJS_ORG_MIRROR -}) From 81f3a923386acfedf22d5b8c34372ec41e3b3096 Mon Sep 17 00:00:00 2001 From: Ben Noordhuis Date: Tue, 19 Mar 2019 11:58:46 +0100 Subject: [PATCH 028/551] Update list of Node.js versions to test against. PR-URL: https://github.com/nodejs/node-gyp/pull/1670 Reviewed-By: Richard Lau --- test/docker.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/docker.sh b/test/docker.sh index 8a9f160c0a..67014209d0 100755 --- a/test/docker.sh +++ b/test/docker.sh @@ -2,7 +2,7 @@ #set -e -test_node_versions="0.8.28 0.10.40 0.12.7 4.3.0 5.6.0" +test_node_versions="6.17.0 8.15.1 10.15.3 11.12.0" myuid=$(id -u) mygid=$(id -g) From 91744bfecc67fda7db58e2a1c7aa72f196d6da4f Mon Sep 17 00:00:00 2001 From: Richard Townsend Date: Thu, 2 May 2019 15:58:55 +0100 Subject: [PATCH 029/551] gyp: add support for Windows on Arm MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cherry-pick of https://github.com/refack/GYP/pull/33, supersedes https://github.com/nodejs/node-gyp/pull/1678 until GYP3 is merged. `npm test` passes Change-Id: I2b1e1e03e378b4812d34afa527087793864d1576 PR-URL: https://github.com/nodejs/node-gyp/pull/1739 Reviewed-By: Refael Ackermann Reviewed-By: João Reis --- gyp/pylib/gyp/MSVSSettings.py | 4 +++- gyp/pylib/gyp/generator/msvs.py | 2 ++ gyp/pylib/gyp/msvs_emulation.py | 17 +++++++++++++---- lib/configure.js | 3 +++ 4 files changed, 21 insertions(+), 5 deletions(-) diff --git a/gyp/pylib/gyp/MSVSSettings.py b/gyp/pylib/gyp/MSVSSettings.py index 8073f92b8d..e71fbf01d3 100644 --- a/gyp/pylib/gyp/MSVSSettings.py +++ b/gyp/pylib/gyp/MSVSSettings.py @@ -973,7 +973,9 @@ def _ValidateSettings(validators, settings, stderr): _Enumeration(['NotSet', 'Win32', # /env win32 'Itanium', # /env ia64 - 'X64'])) # /env x64 + 'X64', # /env x64 + 'ARM64', # /env arm64 + ])) _Same(_midl, 'EnableErrorChecks', _Enumeration(['EnableCustom', 'None', # /error none diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index fb2549d025..badf8f5918 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -1887,6 +1887,8 @@ def _InitNinjaFlavor(params, target_list, target_dicts): configuration = '$(Configuration)' if params.get('target_arch') == 'x64': configuration += '_x64' + if params.get('target_arch') == 'arm64': + configuration += '_arm64' spec['msvs_external_builder_out_dir'] = os.path.join( gyp.common.RelativePath(params['options'].toplevel_dir, gyp_dir), ninja_generator.ComputeOutputDir(params), diff --git a/gyp/pylib/gyp/msvs_emulation.py b/gyp/pylib/gyp/msvs_emulation.py index 75da78526c..4a50b1b74c 100644 --- a/gyp/pylib/gyp/msvs_emulation.py +++ b/gyp/pylib/gyp/msvs_emulation.py @@ -237,7 +237,11 @@ def GetExtension(self): def GetVSMacroEnv(self, base_to_build=None, config=None): """Get a dict of variables mapping internal VS macro names to their gyp equivalents.""" - target_platform = 'Win32' if self.GetArch(config) == 'x86' else 'x64' + target_arch = self.GetArch(config) + if target_arch == 'x86': + target_platform = 'Win32' + else: + target_platform = target_arch target_name = self.spec.get('product_prefix', '') + \ self.spec.get('product_name', self.spec['target_name']) target_dir = base_to_build + '\\' if base_to_build else '' @@ -299,7 +303,7 @@ def GetArch(self, config): if not platform: # If no specific override, use the configuration's. platform = configuration_platform # Map from platform to architecture. - return {'Win32': 'x86', 'x64': 'x64'}.get(platform, 'x86') + return {'Win32': 'x86', 'x64': 'x64', 'ARM64': 'arm64'}.get(platform, 'x86') def _TargetConfig(self, config): """Returns the target-specific configuration.""" @@ -563,7 +567,10 @@ def GetLdflags(self, config, gyp_to_build_path, expand_special, 'VCLinkerTool', append=ldflags) self._GetDefFileAsLdflags(ldflags, gyp_to_build_path) ld('GenerateDebugInformation', map={'true': '/DEBUG'}) - ld('TargetMachine', map={'1': 'X86', '17': 'X64', '3': 'ARM'}, + # TODO: These 'map' values come from machineTypeOption enum, + # and does not have an official value for ARM64 in VS2017 (yet). + # It needs to verify the ARM64 value when machineTypeOption is updated. + ld('TargetMachine', map={'1': 'X86', '17': 'X64', '3': 'ARM', '18': 'ARM64'}, prefix='/MACHINE:') ldflags.extend(self._GetAdditionalLibraryDirectories( 'VCLinkerTool', config, gyp_to_build_path)) @@ -860,7 +867,9 @@ def midl(name, default=None): ('iid', iid), ('proxy', proxy)] # TODO(scottmg): Are there configuration settings to set these flags? - target_platform = 'win32' if self.GetArch(config) == 'x86' else 'x64' + target_platform = self.GetArch(config) + if target_platform == 'x86': + target_platform = 'win32' flags = ['/char', 'signed', '/env', target_platform, '/Oicf'] return outdir, output, variables, flags diff --git a/lib/configure.js b/lib/configure.js index 32d77523ca..d8e2f20335 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -134,6 +134,9 @@ function configure (gyp, argv, callback) { // set the target_arch variable variables.target_arch = gyp.opts.arch || process.arch || 'ia32' + if (variables.target_arch == 'arm64') { + defaults['msvs_configuration_platform'] = 'ARM64' + } // set the node development directory variables.nodedir = nodeDir From 721eb691cf15556cc2700eda0558d5bad5f84232 Mon Sep 17 00:00:00 2001 From: Jon Kunkee Date: Mon, 25 Feb 2019 13:39:48 -0800 Subject: [PATCH 030/551] gyp: teach MSVS generator about MARMASM Items MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This change allows MSVS projects generated for ARM64 to include ASM files. PR-URL: https://github.com/nodejs/node-gyp/pull/1679 Reviewed-By: João Reis --- gyp/pylib/gyp/MSVSSettings.py | 2 ++ gyp/pylib/gyp/generator/msvs.py | 31 +++++++++++++++++++++++-------- 2 files changed, 25 insertions(+), 8 deletions(-) diff --git a/gyp/pylib/gyp/MSVSSettings.py b/gyp/pylib/gyp/MSVSSettings.py index e71fbf01d3..065a339a80 100644 --- a/gyp/pylib/gyp/MSVSSettings.py +++ b/gyp/pylib/gyp/MSVSSettings.py @@ -543,6 +543,7 @@ def _ValidateSettings(validators, settings, stderr): _lib = _Tool('VCLibrarianTool', 'Lib') _manifest = _Tool('VCManifestTool', 'Manifest') _masm = _Tool('MASM', 'MASM') +_armasm = _Tool('ARMASM', 'ARMASM') _AddTool(_compile) @@ -552,6 +553,7 @@ def _ValidateSettings(validators, settings, stderr): _AddTool(_lib) _AddTool(_manifest) _AddTool(_masm) +_AddTool(_armasm) # Add sections only found in the MSBuild settings. _msbuild_validators[''] = {} _msbuild_validators['ProjectReference'] = {} diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index badf8f5918..5f622ab506 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -2039,7 +2039,8 @@ def GenerateOutput(target_list, target_dicts, data, params): def _GenerateMSBuildFiltersFile(filters_path, source_files, - rule_dependencies, extension_to_rule_name): + rule_dependencies, extension_to_rule_name, + platforms): """Generate the filters file. This file is used by Visual Studio to organize the presentation of source @@ -2053,7 +2054,8 @@ def _GenerateMSBuildFiltersFile(filters_path, source_files, filter_group = [] source_group = [] _AppendFiltersForMSBuild('', source_files, rule_dependencies, - extension_to_rule_name, filter_group, source_group) + extension_to_rule_name, platforms, + filter_group, source_group) if filter_group: content = ['Project', {'ToolsVersion': '4.0', @@ -2069,7 +2071,7 @@ def _GenerateMSBuildFiltersFile(filters_path, source_files, def _AppendFiltersForMSBuild(parent_filter_name, sources, rule_dependencies, - extension_to_rule_name, + extension_to_rule_name, platforms, filter_group, source_group): """Creates the list of filters and sources to be added in the filter file. @@ -2095,11 +2097,12 @@ def _AppendFiltersForMSBuild(parent_filter_name, sources, rule_dependencies, # Recurse and add its dependents. _AppendFiltersForMSBuild(filter_name, source.contents, rule_dependencies, extension_to_rule_name, - filter_group, source_group) + platforms, filter_group, source_group) else: # It's a source. Create a source entry. _, element = _MapFileToMsBuildSourceType(source, rule_dependencies, - extension_to_rule_name) + extension_to_rule_name, + platforms) source_entry = [element, {'Include': source}] # Specify the filter it is part of, if any. if parent_filter_name: @@ -2108,7 +2111,7 @@ def _AppendFiltersForMSBuild(parent_filter_name, sources, rule_dependencies, def _MapFileToMsBuildSourceType(source, rule_dependencies, - extension_to_rule_name): + extension_to_rule_name, platforms): """Returns the group and element type of the source file. Arguments: @@ -2134,6 +2137,9 @@ def _MapFileToMsBuildSourceType(source, rule_dependencies, elif ext == '.asm': group = 'masm' element = 'MASM' + for platform in platforms: + if platform.lower() in ['arm', 'arm64']: + element = 'MARMASM' elif ext == '.idl': group = 'midl' element = 'Midl' @@ -3227,7 +3233,8 @@ def _AddSources2(spec, sources, exclusions, grouped_sources, detail.append(['ForcedIncludeFiles', '']) group, element = _MapFileToMsBuildSourceType(source, rule_dependencies, - extension_to_rule_name) + extension_to_rule_name, + _GetUniquePlatforms(spec)) grouped_sources[group].append([element, {'Include': source}] + detail) @@ -3310,7 +3317,7 @@ def _GenerateMSBuildProject(project, options, version, generator_flags): _GenerateMSBuildFiltersFile(project.path + '.filters', sources, rule_dependencies, - extension_to_rule_name) + extension_to_rule_name, _GetUniquePlatforms(spec)) missing_sources = _VerifySourcesExist(sources, project_dir) for configuration in configurations.itervalues(): @@ -3330,6 +3337,12 @@ def _GenerateMSBuildProject(project, options, version, generator_flags): import_masm_targets_section = [ ['Import', {'Project': r'$(VCTargetsPath)\BuildCustomizations\masm.targets'}]] + import_marmasm_props_section = [ + ['Import', + {'Project': r'$(VCTargetsPath)\BuildCustomizations\marmasm.props'}]] + import_marmasm_targets_section = [ + ['Import', + {'Project': r'$(VCTargetsPath)\BuildCustomizations\marmasm.targets'}]] macro_section = [['PropertyGroup', {'Label': 'UserMacros'}]] content = [ @@ -3349,6 +3362,7 @@ def _GenerateMSBuildProject(project, options, version, generator_flags): content += _GetMSBuildLocalProperties(project.msbuild_toolset) content += import_cpp_props_section content += import_masm_props_section + content += import_marmasm_props_section content += _GetMSBuildExtensions(props_files_of_rules) content += _GetMSBuildPropertySheets(configurations) content += macro_section @@ -3361,6 +3375,7 @@ def _GenerateMSBuildProject(project, options, version, generator_flags): content += _GetMSBuildProjectReferences(project) content += import_cpp_targets_section content += import_masm_targets_section + content += import_marmasm_targets_section content += _GetMSBuildExtensionTargets(targets_files_of_rules) if spec.get('msvs_external_builder'): From 7fe4095974148d32bd3d696c3d6062d1e689f016 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Fri, 24 May 2019 12:26:31 +0100 Subject: [PATCH 031/551] win: generic Visual Studio 2017 detection PR-URL: https://github.com/nodejs/node-gyp/pull/1762 Reviewed-By: Rod Vagg Reviewed-By: Refael Ackermann --- lib/{Find-VS2017.cs => Find-VisualStudio.cs} | 88 +++----- lib/configure.js | 18 +- lib/find-visualstudio.js | 213 +++++++++++++++++++ lib/find-vs2017.js | 44 ---- lib/util.js | 12 ++ test/fixtures/VS_2017_BuildTools_minimal.txt | 1 + test/fixtures/VS_2017_Community_workload.txt | 1 + test/fixtures/VS_2017_Unusable.txt | 1 + test/test-find-visualstudio.js | 127 +++++++++++ 9 files changed, 387 insertions(+), 118 deletions(-) rename lib/{Find-VS2017.cs => Find-VisualStudio.cs} (64%) create mode 100644 lib/find-visualstudio.js delete mode 100644 lib/find-vs2017.js create mode 100644 lib/util.js create mode 100644 test/fixtures/VS_2017_BuildTools_minimal.txt create mode 100644 test/fixtures/VS_2017_Community_workload.txt create mode 100644 test/fixtures/VS_2017_Unusable.txt create mode 100644 test/test-find-visualstudio.js diff --git a/lib/Find-VS2017.cs b/lib/Find-VisualStudio.cs similarity index 64% rename from lib/Find-VS2017.cs rename to lib/Find-VisualStudio.cs index 6e7429b771..0a533f42d6 100644 --- a/lib/Find-VS2017.cs +++ b/lib/Find-VisualStudio.cs @@ -3,10 +3,13 @@ // See accompanying file LICENSE at https://github.com/node4good/windows-autoconf // Usage: -// powershell -ExecutionPolicy Unrestricted -Version "2.0" -Command "&{Add-Type -Path Find-VS2017.cs; [VisualStudioConfiguration.Main]::Query()}" +// powershell -ExecutionPolicy Unrestricted -Command "Add-Type -Path Find-VisualStudio.cs; [VisualStudioConfiguration.Main]::PrintJson()" +// This script needs to be compatible with PowerShell v2 to run on Windows 2008R2 and Windows 7. + using System; using System.Text; using System.Runtime.InteropServices; +using System.Collections.Generic; namespace VisualStudioConfiguration { @@ -184,7 +187,7 @@ public class SetupConfigurationClass public static class Main { - public static void Query() + public static void PrintJson() { ISetupConfiguration query = new SetupConfiguration(); ISetupConfiguration2 query2 = (ISetupConfiguration2)query; @@ -192,82 +195,49 @@ public static void Query() int pceltFetched; ISetupInstance2[] rgelt = new ISetupInstance2[1]; - StringBuilder log = new StringBuilder(); + List instances = new List(); while (true) { e.Next(1, rgelt, out pceltFetched); if (pceltFetched <= 0) { - Console.WriteLine(String.Format("{{\"log\":\"{0}\"}}", log.ToString())); + Console.WriteLine(String.Format("[{0}]", string.Join(",", instances.ToArray()))); return; } - if (CheckInstance(rgelt[0], ref log)) - return; + + instances.Add(InstanceJson(rgelt[0])); } } - private static bool CheckInstance(ISetupInstance2 setupInstance2, ref StringBuilder log) + private static string JsonString(string s) { - // Visual Studio Community 2017 component directory: - // https://www.visualstudio.com/en-us/productinfo/vs2017-install-product-Community.workloads + return "\"" + s.Replace("\\", "\\\\").Replace("\"", "\\\"") + "\""; + } - string path = setupInstance2.GetInstallationPath().Replace("\\", "\\\\"); - log.Append(String.Format("Found installation at: {0}\\n", path)); + private static string InstanceJson(ISetupInstance2 setupInstance2) + { + // Visual Studio component directory: + // https://docs.microsoft.com/en-us/visualstudio/install/workload-and-component-ids - bool hasMSBuild = false; - bool hasVCTools = false; - uint Win10SDKVer = 0; - bool hasWin8SDK = false; + StringBuilder json = new StringBuilder(); + json.Append("{"); - foreach (ISetupPackageReference package in setupInstance2.GetPackages()) - { - const string Win10SDKPrefix = "Microsoft.VisualStudio.Component.Windows10SDK."; - - string id = package.GetId(); - if (id == "Microsoft.VisualStudio.VC.MSBuild.Base") - hasMSBuild = true; - else if (id == "Microsoft.VisualStudio.Component.VC.Tools.x86.x64") - hasVCTools = true; - else if (id.StartsWith(Win10SDKPrefix)) { - string[] parts = id.Substring(Win10SDKPrefix.Length).Split('.'); - if (parts.Length > 1 && parts[1] != "Desktop") - continue; - uint foundSdkVer; - if (UInt32.TryParse(parts[0], out foundSdkVer)) - Win10SDKVer = Math.Max(Win10SDKVer, foundSdkVer); - } else if (id == "Microsoft.VisualStudio.Component.Windows81SDK") - hasWin8SDK = true; - else - continue; - - log.Append(String.Format(" - Found {0}\\n", id)); - } + string path = JsonString(setupInstance2.GetInstallationPath()); + json.Append(String.Format("\"path\":{0},", path)); - if (!hasMSBuild) - log.Append(" - Missing Visual Studio C++ core features (Microsoft.VisualStudio.VC.MSBuild.Base)\\n"); - if (!hasVCTools) - log.Append(" - Missing VC++ 2017 v141 toolset (x86,x64) (Microsoft.VisualStudio.Component.VC.Tools.x86.x64)\\n"); - if ((Win10SDKVer == 0) && (!hasWin8SDK)) - log.Append(" - Missing a Windows SDK (Microsoft.VisualStudio.Component.Windows10SDK.* or Microsoft.VisualStudio.Component.Windows81SDK)\\n"); + string version = JsonString(setupInstance2.GetInstallationVersion()); + json.Append(String.Format("\"version\":{0},", version)); - if (hasMSBuild && hasVCTools) + List packages = new List(); + foreach (ISetupPackageReference package in setupInstance2.GetPackages()) { - if (Win10SDKVer > 0) - { - log.Append(" - Using this installation with Windows 10 SDK"/*\\n*/); - Console.WriteLine(String.Format("{{\"log\":\"{0}\",\"path\":\"{1}\",\"sdk\":\"10.0.{2}.0\"}}", log.ToString(), path, Win10SDKVer)); - return true; - } - else if (hasWin8SDK) - { - log.Append(" - Using this installation with Windows 8.1 SDK"/*\\n*/); - Console.WriteLine(String.Format("{{\"log\":\"{0}\",\"path\":\"{1}\",\"sdk\":\"8.1\"}}", log.ToString(), path)); - return true; - } + string id = JsonString(package.GetId()); + packages.Add(id); } + json.Append(String.Format("\"packages\":[{0}]", string.Join(",", packages.ToArray()))); - log.Append(" - Some required components are missing, not using this installation\\n"); - return false; + json.Append("}"); + return json.ToString(); } } } diff --git a/lib/configure.js b/lib/configure.js index d8e2f20335..254824f824 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -17,8 +17,9 @@ var fs = require('graceful-fs') , win = process.platform === 'win32' , findNodeDirectory = require('./find-node-directory') , msgFormat = require('util').format + , logWithPrefix = require('./util').logWithPrefix if (win) - var findVS2017 = require('./find-vs2017') + var findVisualStudio = require('./find-visualstudio') exports.usage = 'Generates ' + (win ? 'MSVC project files' : 'a Makefile') + ' for the current module' @@ -83,7 +84,7 @@ function configure (gyp, argv, callback) { if (err) return callback(err) log.verbose('build dir', '"build" dir needed to be created?', isNew) if (win && (!gyp.opts.msvs_version || gyp.opts.msvs_version === '2017')) { - findVS2017(function (err, vsSetup) { + findVisualStudio(function (err, vsSetup) { if (err) { log.verbose('Not using VS2017:', err.message) createConfigFile() @@ -644,16 +645,3 @@ function findPython (configPython, callback) { var finder = new PythonFinder(configPython, callback) finder.findPython() } - -function logWithPrefix (log, prefix) { - function setPrefix(logFunction) { - return (...args) => logFunction.apply(null, [prefix, ...args]) - } - return { - silly: setPrefix(log.silly), - verbose: setPrefix(log.verbose), - info: setPrefix(log.info), - warn: setPrefix(log.warn), - error: setPrefix(log.error), - } -} diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js new file mode 100644 index 0000000000..624567154f --- /dev/null +++ b/lib/find-visualstudio.js @@ -0,0 +1,213 @@ +module.exports = exports = findVisualStudio +module.exports.test = { + VisualStudioFinder: VisualStudioFinder, + findVisualStudio: findVisualStudio +} + +const log = require('npmlog') +const execFile = require('child_process').execFile +const path = require('path').win32 +const logWithPrefix = require('./util').logWithPrefix + +function findVisualStudio (callback) { + const finder = new VisualStudioFinder(callback) + finder.findVisualStudio() +} + +function VisualStudioFinder (callback) { + this.callback = callback + this.errorLog = [] +} + +VisualStudioFinder.prototype = { + log: logWithPrefix(log, 'find VS'), + + // Logs a message at verbose level, but also saves it to be displayed later + // at error level if an error occurs. This should help diagnose the problem. + addLog: function addLog (message) { + this.log.verbose(message) + this.errorLog.push(message) + }, + + findVisualStudio: function findVisualStudio () { + var ps = path.join(process.env.SystemRoot, 'System32', + 'WindowsPowerShell', 'v1.0', 'powershell.exe') + var csFile = path.join(__dirname, 'Find-VisualStudio.cs') + var psArgs = ['-ExecutionPolicy', 'Unrestricted', '-NoProfile', + '-Command', '&{Add-Type -Path \'' + csFile + '\';' + + '[VisualStudioConfiguration.Main]::PrintJson()}'] + + this.log.silly('Running', ps, psArgs) + var child = execFile(ps, psArgs, { encoding: 'utf8' }, + this.parseData.bind(this)) + child.stdin.end() + }, + + parseData: function parseData (err, stdout, stderr) { + this.log.silly('PS stderr = %j', stderr) + + if (err) { + this.log.silly('PS err = %j', err && (err.stack || err)) + return this.failPowershell() + } + + var vsInfo + try { + vsInfo = JSON.parse(stdout) + } catch (e) { + this.log.silly('PS stdout = %j', stdout) + this.log.silly(e) + return this.failPowershell() + } + + if (!Array.isArray(vsInfo)) { + this.log.silly('PS stdout = %j', stdout) + return this.failPowershell() + } + + vsInfo = vsInfo.map((info) => { + this.log.silly(`processing installation: "${info.path}"`) + const versionYear = this.getVersionYear(info) + return { + path: info.path, + version: info.version, + versionYear: versionYear, + hasMSBuild: this.getHasMSBuild(info), + toolset: this.getToolset(info, versionYear), + sdk: this.getSDK(info) + } + }) + this.log.silly('vsInfo:', vsInfo) + + // Remove future versions or errors parsing version number + vsInfo = vsInfo.filter((info) => { + if (info.versionYear) { return true } + this.addLog(`unknown version "${info.version}" found at "${info.path}"`) + return false + }) + + // Sort to place newer versions first + vsInfo.sort((a, b) => b.versionYear - a.versionYear) + + for (var i = 0; i < vsInfo.length; ++i) { + const info = vsInfo[i] + this.addLog(`checking VS${info.versionYear} (${info.version}) found ` + + `at\n"${info.path}"`) + + if (info.hasMSBuild) { + this.addLog('- found "Visual Studio C++ core features"') + } else { + this.addLog('- "Visual Studio C++ core features" missing') + continue + } + + if (info.toolset) { + this.addLog(`- found VC++ toolset: ${info.toolset}`) + } else { + this.addLog('- missing any VC++ toolset') + continue + } + + if (info.sdk) { + this.addLog(`- found Windows SDK: ${info.sdk}`) + } else { + this.addLog('- missing any Windows SDK') + continue + } + + this.succeed(info) + return + } + + this.fail() + }, + + succeed: function succeed (info) { + this.log.info(`using VS${info.versionYear} (${info.version}) found ` + + `at\n"${info.path}"`) + process.nextTick(this.callback.bind(null, null, info)) + }, + + failPowershell: function failPowershell () { + process.nextTick(this.callback.bind(null, new Error( + 'Could not use PowerShell to find Visual Studio'))) + }, + + fail: function fail () { + const errorLog = this.errorLog.join('\n') + + // For Windows 80 col console, use up to the column before the one marked + // with X (total 79 chars including logger prefix, 62 chars usable here): + // X + const infoLog = [ + '**************************************************************', + 'You need to install the latest version of Visual Studio', + 'including the "Desktop development with C++" workload.', + 'For more information consult the documentation at:', + 'https://github.com/nodejs/node-gyp#on-windows', + '**************************************************************' + ].join('\n') + + this.log.error(`\n${errorLog}\n\n${infoLog}\n`) + process.nextTick(this.callback.bind(null, new Error( + 'Could not find any Visual Studio installation to use'))) + }, + + getVersionYear: function getVersionYear (info) { + const version = parseInt(info.version, 10) + if (version === 15) { + return 2017 + } + this.log.silly('- failed to parse version:', info.version) + return null + }, + + getHasMSBuild: function getHasMSBuild (info) { + const pkg = 'Microsoft.VisualStudio.VC.MSBuild.Base' + return info.packages.indexOf(pkg) !== -1 + }, + + getToolset: function getToolset (info, versionYear) { + const pkg = 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64' + if (info.packages.indexOf(pkg) !== -1) { + this.log.silly('- found VC.Tools.x86.x64') + if (versionYear === 2017) { + return 'v141' + } + } + return null + }, + + getSDK: function getSDK (info) { + const win8SDK = 'Microsoft.VisualStudio.Component.Windows81SDK' + const win10SDKPrefix = 'Microsoft.VisualStudio.Component.Windows10SDK.' + + var Win10SDKVer = 0 + info.packages.forEach((pkg) => { + if (!pkg.startsWith(win10SDKPrefix)) { + return + } + const parts = pkg.split('.') + if (parts.length > 5 && parts[5] !== 'Desktop') { + this.log.silly('- ignoring non-Desktop Win10SDK:', pkg) + return + } + const foundSdkVer = parseInt(parts[4], 10) + if (isNaN(foundSdkVer)) { + // Microsoft.VisualStudio.Component.Windows10SDK.IpOverUsb + this.log.silly('- failed to parse Win10SDK number:', pkg) + return + } + this.log.silly('- found Win10SDK:', foundSdkVer) + Win10SDKVer = Math.max(Win10SDKVer, foundSdkVer) + }) + + if (Win10SDKVer !== 0) { + return `10.0.${Win10SDKVer}.0` + } else if (info.packages.indexOf(win8SDK) !== -1) { + this.log.silly('- found Win8SDK') + return '8.1' + } + return null + } +} diff --git a/lib/find-vs2017.js b/lib/find-vs2017.js deleted file mode 100644 index 8655d7c931..0000000000 --- a/lib/find-vs2017.js +++ /dev/null @@ -1,44 +0,0 @@ -var log = require('npmlog') - , execFile = require('child_process').execFile - , path = require('path') - -module.exports = function findVS2017(callback) { - var ps = path.join(process.env.SystemRoot, 'System32', 'WindowsPowerShell', - 'v1.0', 'powershell.exe') - var csFile = path.join(__dirname, 'Find-VS2017.cs') - var psArgs = ['-ExecutionPolicy', 'Unrestricted', '-NoProfile', - '-Command', '&{Add-Type -Path \'' + csFile + '\';' + - '[VisualStudioConfiguration.Main]::Query()}'] - - log.silly('find vs2017', 'Running', ps, psArgs) - var child = execFile(ps, psArgs, { encoding: 'utf8' }, - function (err, stdout, stderr) { - log.silly('find vs2017', 'PS err:', err) - log.silly('find vs2017', 'PS stdout:', stdout) - log.silly('find vs2017', 'PS stderr:', stderr) - - if (err) - return callback(new Error('Could not use PowerShell to find VS2017')) - - var vsSetup - try { - vsSetup = JSON.parse(stdout) - } catch (e) { - log.silly('find vs2017', e) - return callback(new Error('Could not use PowerShell to find VS2017')) - } - log.silly('find vs2017', 'vsSetup:', vsSetup) - - if (vsSetup && vsSetup.log) - log.verbose('find vs2017', vsSetup.log.trimRight()) - - if (!vsSetup || !vsSetup.path || !vsSetup.sdk) { - return callback(new Error('No usable installation of VS2017 found')) - } - - log.verbose('find vs2017', 'using installation:', vsSetup.path) - callback(null, { "path": vsSetup.path, "sdk": vsSetup.sdk }) - }) - - child.stdin.end() -} diff --git a/lib/util.js b/lib/util.js new file mode 100644 index 0000000000..cb12b936ef --- /dev/null +++ b/lib/util.js @@ -0,0 +1,12 @@ +module.exports.logWithPrefix = function logWithPrefix (log, prefix) { + function setPrefix(logFunction) { + return (...args) => logFunction.apply(null, [prefix, ...args]) + } + return { + silly: setPrefix(log.silly), + verbose: setPrefix(log.verbose), + info: setPrefix(log.info), + warn: setPrefix(log.warn), + error: setPrefix(log.error), + } +} diff --git a/test/fixtures/VS_2017_BuildTools_minimal.txt b/test/fixtures/VS_2017_BuildTools_minimal.txt new file mode 100644 index 0000000000..244f6b0798 --- /dev/null +++ b/test/fixtures/VS_2017_BuildTools_minimal.txt @@ -0,0 +1 @@ +[{"path":"C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTools","version":"15.9.28307.665","packages":["Microsoft.VisualStudio.Product.BuildTools","Microsoft.VisualStudio.Component.VC.CoreIde","Microsoft.VisualStudio.VC.Ide.Pro","Microsoft.VisualStudio.VC.Ide.Pro.Resources","Microsoft.VisualStudio.VC.Templates.Pro","Microsoft.VisualStudio.VC.Templates.Pro.Resources","Microsoft.VisualStudio.VC.Items.Pro","Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Reduced","Microsoft.VisualStudio.VC.Ide.MDD","Microsoft.VisualStudio.VC.Ide.x64","Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Express","Microsoft.VisualStudio.PackageGroup.Debugger.Script","Microsoft.VisualStudio.JavaScript.LanguageService","Microsoft.VisualStudio.JavaScript.LanguageService.Resources","Microsoft.VisualStudio.Debugger.Script.Msi","Microsoft.VisualStudio.Debugger.Script","Microsoft.VisualStudio.Debugger.Script","Microsoft.VisualStudio.Debugger.Script.Resources","Microsoft.VisualStudio.Debugger.Script.Resources","Microsoft.VisualStudio.VC.Ide.WinXPlus","Microsoft.VisualStudio.VC.Ide.Dskx","Microsoft.VisualStudio.VC.Ide.Dskx.Resources","Microsoft.VisualStudio.VC.Ide.Core","Microsoft.VisualStudio.VC.Ide.Core.Resources","Microsoft.VisualStudio.VC.Ide.Base","Microsoft.VisualStudio.VC.Ide.LanguageService","Microsoft.VisualStudio.VC.Ide.ResourceEditor","Microsoft.VisualStudio.VC.Ide.ResourceEditor.Resources","Microsoft.VisualStudio.VC.Ide.ProjectSystem","Microsoft.VisualStudio.VC.Ide.ProjectSystem.Resources","Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine","Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine.Resources","Microsoft.VisualStudio.VC.Ide.LanguageService.Resources","Microsoft.VisualStudio.VC.Ide.Base.Resources","Microsoft.VisualStudio.PackageGroup.Core","Microsoft.VisualStudio.TestTools.TeamFoundationClient","Microsoft.VisualStudio.PackageGroup.Debugger.Core","Microsoft.VisualStudio.Debugger.VSCodeDebuggerHost","Microsoft.VisualStudio.VC.Ide.Debugger","Microsoft.VisualStudio.VC.Ide.Debugger.Resources","Microsoft.VisualStudio.VC.Ide.Common","Microsoft.VisualStudio.VC.Ide.Common.Resources","Microsoft.VisualStudio.Debugger.Parallel","Microsoft.VisualStudio.Debugger.Parallel.Resources","Microsoft.VisualStudio.Debugger.CollectionAgents","Microsoft.VisualStudio.Debugger.Managed","Microsoft.CodeAnalysis.VisualStudio.Setup.Resources","Microsoft.CodeAnalysis.VisualStudio.Setup","Microsoft.CodeAnalysis.ExpressionEvaluator.Resources","Microsoft.CodeAnalysis.ExpressionEvaluator","Microsoft.VisualStudio.Debugger.Managed.Resources","Microsoft.VisualStudio.Debugger.Remote","Microsoft.VisualStudio.Debugger.Remote","Microsoft.VisualStudio.Debugger.Remote.Resources","Microsoft.VisualStudio.Debugger.Remote.Resources","Microsoft.VisualStudio.Debugger","Microsoft.VisualStudio.VC.MSVCDis","Microsoft.VisualStudio.ScriptedHost","Microsoft.VisualStudio.ScriptedHost.Targeted","Microsoft.VisualStudio.ScriptedHost.Resources","Microsoft.IntelliTrace.DiagnosticsHub","Microsoft.VisualStudio.Debugger.Resources","Microsoft.PackageGroup.ClientDiagnostics","Microsoft.VisualStudio.AppResponsiveness","Microsoft.VisualStudio.AppResponsiveness.Targeted","Microsoft.VisualStudio.AppResponsiveness.Resources","Microsoft.VisualStudio.ClientDiagnostics","Microsoft.VisualStudio.ClientDiagnostics.Targeted","Microsoft.VisualStudio.ClientDiagnostics.Resources","Microsoft.VisualStudio.PackageGroup.CommunityCore","Microsoft.VisualStudio.ProjectSystem.Full","Microsoft.VisualStudio.ProjectSystem","Microsoft.VisualStudio.Community.x86","Microsoft.VisualStudio.Community.x64","Microsoft.VisualStudio.Community","Microsoft.IntelliTrace.CollectorCab","Microsoft.VisualStudio.Community.Resources","Microsoft.VisualStudio.WebSiteProject.DTE","Microsoft.MSHtml","Microsoft.VisualStudio.Community.Msi.Resources","Microsoft.VisualStudio.Community.Msi","Microsoft.VisualStudio.MinShell.Interop.Msi","Microsoft.VisualStudio.PackageGroup.CoreEditor","PortableFacades","Microsoft.VisualStudio.VirtualTree","Microsoft.VisualStudio.PackageGroup.Progression","Microsoft.VisualStudio.PerformanceProvider","Microsoft.VisualStudio.GraphModel","Microsoft.VisualStudio.GraphProvider","Microsoft.DiaSymReader","Microsoft.VisualStudio.TextMateGrammars","Microsoft.VisualStudio.PackageGroup.TeamExplorer","Microsoft.TeamFoundation.OfficeIntegration","Microsoft.TeamFoundation.OfficeIntegration.Resources","Microsoft.VisualStudio.TeamExplorer","Microsoft.ServiceHub","Microsoft.VisualStudio.ProjectServices","Microsoft.VisualStudio.SLNX.VSIX","Microsoft.VisualStudio.FileHandler.Msi","Microsoft.VisualStudio.FileHandler.Msi","Microsoft.VisualStudio.PackageGroup.MinShell","Microsoft.VisualStudio.MinShell.Msi","Microsoft.VisualStudio.MinShell.Msi.Resources","Microsoft.VisualStudio.MinShell.Interop","Microsoft.VisualStudio.Log","Microsoft.VisualStudio.Log.Targeted","Microsoft.VisualStudio.Log.Resources","Microsoft.VisualStudio.Finalizer","Microsoft.VisualStudio.CoreEditor","Microsoft.VisualStudio.Connected","Microsoft.VisualStudio.Connected.Resources","Microsoft.VisualStudio.MinShell","Microsoft.VisualStudio.MinShell.Platform","Microsoft.VisualStudio.MinShell.Platform.Resources","Microsoft.VisualStudio.MefHosting","Microsoft.VisualStudio.MefHosting.Resources","Microsoft.VisualStudio.Initializer","Microsoft.VisualStudio.ExtensionManager","Microsoft.VisualStudio.Editors","Microsoft.Net.4.TargetingPack","Microsoft.VisualStudio.Component.Windows10SDK.17134","Win10SDK_10.0.17134","Microsoft.VisualStudio.Component.VC.Tools.x86.x64","Microsoft.VisualCpp.CodeAnalysis.Extensions","Microsoft.VisualCpp.CodeAnalysis.Extensions.X86","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X86","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X86.Resources","Microsoft.VisualCpp.CodeAnalysis.Extensions.X64","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X64","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X64.Resources","Microsoft.VisualStudio.Component.Static.Analysis.Tools","Microsoft.VisualStudio.StaticAnalysis","Microsoft.VisualStudio.StaticAnalysis.Resources","Microsoft.VisualCpp.Tools.HostX64.TargetX86","Microsoft.VisualCpp.VCTip.HostX64.TargetX86","Microsoft.VisualCpp.Tools.HostX64.TargetX86.Resources","Microsoft.VisualCpp.Tools.HostX64.TargetX64","Microsoft.VisualCpp.VCTip.HostX64.TargetX64","Microsoft.VisualCpp.Tools.HostX64.TargetX64.Resources","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX64","Microsoft.VisualCpp.Premium.Tools.Hostx86.Targetx64.Resources","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86.Resources","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86.Resources","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64.Resources","Microsoft.VisualCpp.PGO.X86","Microsoft.VisualCpp.PGO.X64","Microsoft.VisualCpp.PGO.Headers","Microsoft.VisualCpp.CRT.x86.Store","Microsoft.VisualCpp.CRT.x86.OneCore.Desktop","Microsoft.VisualCpp.CRT.x64.Store","Microsoft.VisualCpp.CRT.x64.OneCore.Desktop","Microsoft.VisualCpp.CRT.Redist.x86.OneCore.Desktop","Microsoft.VisualCpp.CRT.Redist.x64.OneCore.Desktop","Microsoft.VisualCpp.CRT.ClickOnce.Msi","Microsoft.VisualStudio.PackageGroup.VC.Tools.x86","Microsoft.VisualCpp.Tools.HostX86.TargetX64","Microsoft.VisualCpp.VCTip.hostX86.targetX64","Microsoft.VisualCpp.Tools.Hostx86.Targetx64.Resources","Microsoft.VisualCpp.Tools.HostX86.TargetX86","Microsoft.VisualCpp.VCTip.hostX86.targetX86","Microsoft.VisualCpp.Tools.HostX86.TargetX86.Resources","Microsoft.VisualCpp.Tools.Core.Resources","Microsoft.VisualCpp.Tools.Core.x86","Microsoft.VisualCpp.Tools.Common.Utils","Microsoft.VisualCpp.Tools.Common.Utils.Resources","Microsoft.VisualCpp.DIA.SDK","Microsoft.VisualCpp.CRT.x86.Desktop","Microsoft.VisualCpp.CRT.x64.Desktop","Microsoft.VisualCpp.CRT.Source","Microsoft.VisualCpp.CRT.Redist.X86","Microsoft.VisualCpp.CRT.Redist.X64","Microsoft.VisualCpp.CRT.Redist.Resources","Microsoft.VisualCpp.RuntimeDebug.14","Microsoft.VisualCpp.RuntimeDebug.14","Microsoft.VisualCpp.Redist.14","Microsoft.VisualCpp.Redist.14","Microsoft.VisualCpp.CRT.Headers","Microsoft.VisualStudio.VC.MSBuild.X86","Microsoft.VisualStudio.VC.MSBuild.X64","Microsoft.VS.VC.MSBuild.X64.Resources","Microsoft.VisualStudio.VC.MSBuild.Base","Microsoft.VisualStudio.VC.MSBuild.Base.Resources","Microsoft.VisualStudio.VC.MSBuild.ARM","Microsoft.VisualStudio.Workload.MSBuildTools","Microsoft.VisualStudio.Component.CoreBuildTools","Microsoft.VisualStudio.Setup.Configuration","Microsoft.VisualStudio.PackageGroup.VsDevCmd","Microsoft.VisualStudio.VsDevCmd.Ext.NetFxSdk","Microsoft.VisualStudio.VsDevCmd.Core.WinSdk","Microsoft.VisualStudio.VsDevCmd.Core.DotNet","Microsoft.VisualStudio.VC.DevCmd","Microsoft.VisualStudio.VC.DevCmd.Resources","Microsoft.VisualStudio.BuildTools.Resources","Microsoft.VisualStudio.Net.Eula.Resources","Microsoft.Build.Dependencies","Microsoft.Build.FileTracker.Msi","Microsoft.Component.MSBuild","Microsoft.PythonTools.BuildCore.Vsix","Microsoft.NuGet.Build.Tasks","Microsoft.VisualStudio.Component.Roslyn.Compiler","Microsoft.CodeAnalysis.Compilers.Resources","Microsoft.CodeAnalysis.Compilers","Microsoft.Net.PackageGroup.4.6.1.Redist","Microsoft.VisualStudio.NativeImageSupport","Microsoft.Build"]}] diff --git a/test/fixtures/VS_2017_Community_workload.txt b/test/fixtures/VS_2017_Community_workload.txt new file mode 100644 index 0000000000..dd5e77dafb --- /dev/null +++ b/test/fixtures/VS_2017_Community_workload.txt @@ -0,0 +1 @@ +[{"path":"C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community","version":"15.9.28307.665","packages":["Microsoft.VisualStudio.Component.Windows10SDK.IpOverUsb","Win10SDK_IpOverUsb","Microsoft.VisualStudio.Component.VC.ATL.ARM64","Microsoft.VisualCpp.ATL.ARM64","Microsoft.VisualStudio.Component.VC.ATL.ARM","Microsoft.VisualCpp.ATL.ARM","Microsoft.VisualStudio.Component.VC.Tools.ARM","Microsoft.VisualCpp.Tools.HostX64.TargetX86.Resources","Microsoft.VisualStudio.Graphics.Analyzer.Resources","Microsoft.Icecap.Analysis","Microsoft.VisualCpp.CRT.Redist.arm.OneCore.Desktop","Microsoft.VisualCpp.CRT.arm.Store","Microsoft.VisualCpp.CRT.arm.Desktop","Microsoft.VisualStudio.PackageGroup.VC.Tools.x64.ARM","Microsoft.VisualCpp.Premium.Tools.Hostx86.Targetarm","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetARM.Resources","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetARM","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetARM.Resources","Microsoft.VisualCpp.Premium.Tools.ARM.Base","Microsoft.VisualCpp.Premium.Tools.ARM.Base.Resources","Microsoft.VisualCpp.PGO.ARM","Microsoft.VisualCpp.Tools.HostX64.TargetX64","Microsoft.VisualStudio.Product.Community","Microsoft.VisualCpp.Tools.Hostx86.Targetarm","Microsoft.VisualStudio.Component.VC.Tools.ARM64","Microsoft.VisualStudio.VC.MSBuild.Arm64","Microsoft.VisualCpp.CRT.Redist.ARM64.OneCore.Desktop","Microsoft.VisualCpp.VCTip.HostX64.TargetX64","Microsoft.VisualCpp.CRT.ARM64.OneCore.Desktop","Microsoft.VisualCpp.CRT.ARM64.Store","Microsoft.VisualCpp.CRT.ARM64.Desktop","Microsoft.VisualCpp.Tools.HostX64.TargetX64.Resources","Microsoft.Icecap.Analysis.Resources","Microsoft.VisualCpp.VCTip.hostX86.targetARM","Microsoft.VisualStudio.PackageGroup.VC.Tools.x64.ARM64","Microsoft.VisualCpp.Tools.Core","Microsoft.VisualCpp.PGO.ARM64","Microsoft.VisualCpp.Premium.Tools.Hostx86.Targetarm64","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetARM64.Resources","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetARM64","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetARM64.Resources","Microsoft.VisualCpp.Premium.Tools.ARM64.Base","Microsoft.VisualCpp.Tools.HostX86.TargetX64","Microsoft.VisualCpp.Tools.HostX86.TargetARM.Resources","Microsoft.VisualCpp.CRT.Redist.ARM64","Microsoft.VisualCpp.CRT.arm.OneCore.Desktop","Microsoft.VisualCpp.CodeAnalysis.Extensions.X86","Microsoft.VisualCpp.CodeAnalysis.Extensions.X64","Microsoft.VisualCpp.VCTip.HostX64.TargetX86","Component.WixToolset.VisualStudioExtension.Dev15","WixToolset.VisualStudioExtension.Dev15","Microsoft.VisualCpp.MFC.X64","Microsoft.VisualCpp.ATL.Headers","Microsoft.VisualStudio.Component.VC.CMake.Project","Microsoft.VisualStudio.VC.CMake","Microsoft.VisualStudio.VC.CMake.Project","Microsoft.VisualStudio.Component.Windows10SDK.17763","Microsoft.VisualStudio.VC.MSBuild.Base.Resources","MLGen","Microsoft.VisualStudio.Graphics.Analyzer","Microsoft.VisualStudio.Component.TestTools.Core","Microsoft.VisualCpp.Tools.Core.x86","Microsoft.VisualCpp.CRT.x86.OneCore.Desktop","Microsoft.VisualCpp.DIA.SDK","Microsoft.VisualCpp.CRT.x64.OneCore.Desktop","Microsoft.VisualCpp.CRT.ClickOnce.Msi","Microsoft.VisualStudio.NuGet.Licenses","SQLCommon","Microsoft.VisualStudio.VC.MSBuild.X86","Microsoft.VisualCpp.Tools.HostX64.TargetARM","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X64","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X64.Resources","Microsoft.VisualCpp.HTMLHelpWorkshop.Msi","Microsoft.Icecap.Collection.Msi.Resources","Microsoft.VisualCpp.Tools.Hostx86.Targetx64.Resources","Microsoft.VisualCpp.VCTip.hostX64.targetARM","Microsoft.VisualStudio.VC.Ide.Dskx.Resources","Microsoft.VisualStudio.VC.Templates.UnitTest","Microsoft.VisualStudio.TestTools.TestPlatform.V1.CPP","Microsoft.VisualStudio.VC.Ide.Core","Microsoft.VisualStudio.Graphics.Appid","Microsoft.VisualCpp.ATL.Source","Microsoft.VisualStudio.VC.Ide.Core.Resources","Microsoft.VisualStudio.Debugger.ImmersiveActivateHelper.Msi","Microsoft.VisualStudio.Debugger.JustInTime","Microsoft.DiagnosticsHub.CpuSampling","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Common","Microsoft.VisualStudio.TestTools.TP.Legacy.Common.Res","Microsoft.VisualStudio.ProTools.Resources","Microsoft.VisualStudio.Community.Msi","Microsoft.VisualCpp.Tools.HostX64.TargetARM.Resources","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Agent","Microsoft.Component.MSBuild","Microsoft.VisualStudio.Graphics.Msi","Microsoft.VisualStudio.WebToolsExtensions","Microsoft.VisualCpp.Tools.Hostx86.Targetarm64","Microsoft.VisualStudio.TextTemplating.MSBuild","Microsoft.VisualCpp.VCTip.hostX86.targetARM64","Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine","Microsoft.VisualCpp.Tools.HostX86.TargetARM64.Resources","Microsoft.VisualStudio.RazorExtension","Microsoft.VisualCpp.CRT.x86.Store","Microsoft.VisualCpp.Tools.Core.Resources","Microsoft.VisualStudio.Debugger.Script","Microsoft.VisualStudio.Debugger.Script","Microsoft.VisualCpp.MFC.Source","Microsoft.VisualCpp.CRT.x86.Desktop","Microsoft.VisualStudio.VC.MSBuild.X64","Microsoft.VisualStudio.VC.Items.Pro","Microsoft.VisualStudio.Graphics.Viewers","Microsoft.VisualCpp.CRT.x64.Desktop","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X86.Resources","Microsoft.VisualCpp.MFC.Redist.X86","Microsoft.VisualStudio.WebToolsExtensions.Chip","Microsoft.DiagnosticsHub.Runtime.Resources","Microsoft.DiagnosticsHub.CpuSampling.Targeted","Microsoft.VisualStudio.VC.Ide.LanguageService.Resources","Microsoft.VisualStudio.Component.VC.DiagnosticTools","Microsoft.VisualCpp.MFC.Redist.X64","Microsoft.VisualStudio.PackageGroup.TestTools.Native","Microsoft.VisualStudio.Graphics.Viewers.Resources","Microsoft.VisualCpp.MFC.MBCS","Microsoft.VisualStudio.Debugger.Remote.Resources","Microsoft.VisualStudio.Component.TextTemplating","Win10SDK_10.0.17763","Microsoft.VisualStudio.VC.Ide.Base.Resources","Microsoft.VisualCpp.MFC.MBCS.X64","Microsoft.VisualStudio.PackageGroup.TestTools.CodeCoverage","Microsoft.VisualStudio.Graphics.EnableTools","Microsoft.VisualStudio.Graphics.Appid.Resources","Microsoft.VisualStudio.VC.MSBuild.Base","Microsoft.VisualStudio.VC.MSBuild.ARM","Microsoft.VisualCpp.MFC.Headers","Microsoft.VisualCpp.CRT.Redist.x86.OneCore.Desktop","Microsoft.VisualCpp.Tools.HostX86.TargetX86","Microsoft.VisualStudio.VC.Ide.Base","Microsoft.VisualStudio.Graphics.Analyzer.Targeted","Microsoft.VisualCpp.CRT.Headers","Microsoft.DiagnosticsHub.Runtime.Targeted","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86","Microsoft.VisualCpp.Tools.HostX64.TargetARM64","Microsoft.VisualCpp.VCTip.hostX64.targetARM64","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86.Resources","Microsoft.Icecap.Collection.Msi","Microsoft.VisualCpp.ATL.X86","Microsoft.VisualCpp.Tools.HostX64.TargetARM64.Resources","Microsoft.VisualStudio.Component.VC.ATLMFC","Microsoft.VisualCpp.VCTip.hostX86.targetX86","Microsoft.Icecap.Collection.Msi.Resources.Targeted","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86","Microsoft.VisualStudio.Component.Graphics.Tools","Microsoft.VisualStudio.WebTools.Resources","Microsoft.VisualCpp.ATL.X64","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86.Resources","Microsoft.VisualStudio.Component.Graphics.Win81","Microsoft.VisualStudio.VC.Ide.MDD","Microsoft.VisualStudio.VC.Ide.ResourceEditor","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64","Microsoft.Icecap.Analysis.Resources.Targeted","Microsoft.VisualStudio.Debugger.Script.Msi","Microsoft.VisualStudio.Component.VC.CoreIde","Microsoft.VisualStudio.VC.Ide.MFC.Resources","Microsoft.VisualStudio.Debugger.Script.Resources","Microsoft.VisualStudio.PackageGroup.VC.Tools.x86","Microsoft.VisualStudio.TextTemplating.Core","Microsoft.VisualStudio.JavaScript.LanguageService","Microsoft.VisualStudio.VC.Ide.ResourceEditor.Resources","Microsoft.VisualStudio.VC.Ide.ProjectSystem.Resources","Microsoft.VisualStudio.Component.VC.TestAdapterForBoostTest","Microsoft.VisualStudio.VC.Ide.ProjectSystem","Microsoft.VisualStudio.VC.Ide.Dskx","Microsoft.VisualCpp.Tools.HostX86.TargetX86.Resources","Microsoft.CredentialProvider","Microsoft.VisualStudio.VC.Templates.Desktop","Microsoft.VisualStudio.VC.Ide.Pro.Resources","Microsoft.VisualStudio.ComponentGroup.WebToolsExtensions","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Core","Microsoft.VisualStudio.TextTemplating.Integration","Microsoft.VisualStudio.Component.NuGet","Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Reduced","Microsoft.VisualCpp.PGO.Headers","Microsoft.DiagnosticsHub.Collection","Microsoft.Icecap.Collection.Msi.Targeted","Microsoft.VisualStudio.VC.Ide.LanguageService","Microsoft.VisualStudio.WebTools.WSP.FSA","Microsoft.VisualStudio.Graphics.Msi","Microsoft.VisualCpp.CRT.Redist.X86","Microsoft.VisualStudio.Branding.Community","Microsoft.VisualStudio.VC.Ide.x64","Microsoft.VisualStudio.WebToolsExtensions.Common","Microsoft.VisualStudio.WebTools.MSBuild","Microsoft.VisualStudio.NuGet.Core","Microsoft.DiagnosticsHub.Collection.Service","Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine.Resources","Microsoft.CodeAnalysis.ExpressionEvaluator","Microsoft.VisualCpp.CRT.Redist.X64","Microsoft.VisualCpp.RuntimeDebug.14","Microsoft.VS.VC.MSBuild.X64.Resources","Microsoft.VisualCpp.CRT.Source","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Tips.Resources","Microsoft.VisualStudio.VC.Ide.WinXPlus","Microsoft.VisualCpp.RuntimeDebug.14","Microsoft.VisualCpp.Redist.14.Latest","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX64","Microsoft.VisualCpp.CRT.Redist.Resources","Microsoft.VisualCpp.Redist.14.Latest","Microsoft.Net.4.TargetingPack","Microsoft.VisualStudio.Debugger.Script.Resources","Microsoft.VisualCpp.CRT.x64.Store","Microsoft.VisualStudio.VC.Ide.Debugger.Resources","Microsoft.DiaSymReader.Native","Microsoft.VisualCpp.Redist.14","Microsoft.VisualCpp.Redist.14","Microsoft.VisualStudio.StaticAnalysis","Microsoft.VisualStudio.TestTools.TeamFoundationClient","Microsoft.VisualStudio.TestTools.TestPlatform.V1.CLI","Microsoft.VisualStudio.VC.Ide.Common","Microsoft.VisualStudio.Community.Extra.Resources","Microsoft.VisualStudio.Component.Roslyn.LanguageServices","Microsoft.DiagnosticsHub.Collection.StopService.Install","Microsoft.VisualStudio.InteractiveWindow","Microsoft.PackageGroup.DiagnosticsHub.Platform","Microsoft.VisualStudio.StaticAnalysis.Resources","Microsoft.VisualStudio.Debugger.Remote","Microsoft.VisualStudio.VC.Ide.Common.Resources","Microsoft.VisualCpp.Tools.HostX64.TargetX86","Microsoft.VisualStudio.VC.DevCmd","Microsoft.VisualStudio.Community.Extra","Microsoft.VisualStudio.TestTools.TP.Legacy.Tips.Msi","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.TestTools","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Core.Resources","Microsoft.VisualStudio.PackageGroup.TestTools.Core","Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.V2.CLI","Microsoft.VisualStudio.Debugger.Remote","Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.V1.CLI","Microsoft.VisualStudio.Component.VC.Tools.x86.x64","Microsoft.VisualStudio.TestTools.Pex.Common","Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.Legacy","Microsoft.VisualStudio.PackageGroup.MinShell.Interop","Microsoft.CodeAnalysis.ExpressionEvaluator.Resources","Microsoft.VisualCpp.CodeAnalysis.Extensions","Microsoft.VisualStudio.PackageGroup.CoreEditor","Microsoft.VisualStudio.Component.Roslyn.Compiler","Microsoft.VisualStudio.ScriptedHost.Targeted","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Professional","Microsoft.VisualStudio.Debugger.Resources","Microsoft.VisualStudio.Debugger.Parallel","Microsoft.VisualStudio.Debugger.Parallel.Resources","Microsoft.VisualCpp.PGO.X64","Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.IDE","Microsoft.VisualStudio.GraphModel","Microsoft.VisualStudio.PackageGroup.TestTools.DataCollectors","sqlsysclrtypes","Microsoft.VisualStudio.ProTools","Component.Microsoft.VisualStudio.RazorExtension","Microsoft.VisualStudio.TestTools.TestPlatform.V2.CLI","Microsoft.Build.Dependencies","Microsoft.VisualStudio.WebTools.WSP.FSA.Resources","Microsoft.VisualStudio.Component.Static.Analysis.Tools","Microsoft.VisualStudio.VC.Ide.ATL.Resources","Microsoft.VisualStudio.VC.Templates.UnitTest.Resources","Microsoft.VisualStudio.Debugger.Managed","Microsoft.VisualStudio.Workload.NativeDesktop","Microsoft.VisualStudio.Component.VC.TestAdapterForGoogleTest","Microsoft.VisualStudio.Debugger.JustInTime.Msi","Microsoft.Net.PackageGroup.4.6.1.Redist","Microsoft.VisualStudio.Debugger.VSCodeDebuggerHost","sqlsysclrtypes","Microsoft.VisualStudio.Debugger.Managed.Resources","Microsoft.VisualStudio.TestTools.TP.Legacy.Tips.Common","Microsoft.VisualStudio.VC.Ide.Debugger","Microsoft.VisualStudio.AppResponsiveness","Microsoft.VisualStudio.Debugger.Remote.Resources","Microsoft.VisualStudio.TestTools.TestWIExtension","Microsoft.VisualStudio.VC.Ide.Pro","Microsoft.VisualStudio.PackageGroup.Debugger.Core","Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Express","Microsoft.VisualStudio.WebTools","Microsoft.VisualStudio.Component.VC.Redist.14.Latest","Microsoft.VisualStudio.VsDevCmd.Core.WinSdk","Microsoft.VisualStudio.TestTools.TestPlatform.IDE","Microsoft.VisualStudio.TextTemplating.Integration.Resources","Microsoft.VisualStudio.Debugger.CollectionAgents","Microsoft.VisualStudio.Debugger","Microsoft.VisualStudio.PackageGroup.Debugger.Script","Microsoft.VisualStudio.VC.MSVCDis","Microsoft.VisualStudio.ScriptedHost","Microsoft.VisualStudio.ClientDiagnostics.Targeted","Microsoft.VisualStudio.ScriptedHost.Resources","Microsoft.TeamFoundation.OfficeIntegration.Resources","Microsoft.IntelliTrace.DiagnosticsHub","Microsoft.VisualStudio.JavaScript.LanguageService.Resources","Microsoft.VisualStudio.VC.Ide.TestAdapterForGoogleTest","Microsoft.VisualStudio.PackageGroup.Community","Microsoft.VisualStudio.ClientDiagnostics","Microsoft.VisualStudio.Component.Windows10SDK.17134","Microsoft.VisualStudio.PackageGroup.Core","PortableFacades","Microsoft.DiaSymReader","Microsoft.DiagnosticsHub.Runtime","Microsoft.VisualStudio.Component.CoreEditor","Microsoft.VisualStudio.AppResponsiveness.Targeted","Microsoft.VisualStudio.AppResponsiveness.Resources","Microsoft.VisualStudio.Community","Microsoft.TeamFoundation.OfficeIntegration","Microsoft.VisualStudio.WebSiteProject.DTE","Microsoft.VisualStudio.ClientDiagnostics.Resources","Microsoft.VisualStudio.ProjectSystem.Full","Microsoft.VisualStudio.ProjectSystem","Microsoft.VisualCpp.Tools.Common.UtilsPrereq","Microsoft.IntelliTrace.CollectorCab","Microsoft.VisualStudio.Community.Resources","Microsoft.VisualCpp.Tools.Common.Utils","Microsoft.ServiceHub","Microsoft.VisualStudio.Editors","Microsoft.VisualStudio.TeamExplorer","Microsoft.CodeAnalysis.VisualStudio.InteractiveComponents.Resources","Microsoft.VisualStudio.MinShell.Interop.Msi","Microsoft.VisualStudio.GraphProvider","Microsoft.CodeAnalysis.VisualStudio.InteractiveComponents","Microsoft.CodeAnalysis.VisualStudio.Setup.Interactive.Resources","Microsoft.CodeAnalysis.VisualStudio.Setup.Resources","Microsoft.VisualStudio.Community.x86","Microsoft.VisualStudio.Community.x64","Microsoft.CodeAnalysis.VisualStudio.Setup","Microsoft.NuGet.Build.Tasks","Microsoft.PackageGroup.ClientDiagnostics","Microsoft.CodeAnalysis.Compilers.Resources","Microsoft.CodeAnalysis.Compilers","Microsoft.VisualCpp.Tools.Common.Utils.Resources","Microsoft.VisualStudio.Net.Eula.Resources","Microsoft.VisualStudio.PackageGroup.CommunityCore","Microsoft.Build","Microsoft.VisualStudio.VC.Ide.TestAdapterForBoostTest","Microsoft.VisualStudio.VC.Ide.ATL","Microsoft.VisualStudio.TextMateGrammars","Microsoft.VisualStudio.Workload.CoreEditor","Microsoft.VisualStudio.MinShell.Interop","Microsoft.Build.FileTracker.Msi","Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core","Microsoft.MSHtml","Microsoft.VisualStudio.Community.Msi.Resources","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Tips","Microsoft.VisualStudio.Devenv.Msi","Microsoft.VisualStudio.Component.VC.ATL","Microsoft.VisualStudio.VC.Templates.Pro","Microsoft.VisualCpp.CRT.Redist.x64.OneCore.Desktop","Microsoft.VisualStudio.SLNX.VSIX","Microsoft.VisualStudio.CoreEditor","Win10SDK_10.0.17134","Microsoft.VisualStudio.VsDevCmd.Ext.NetFxSdk","Microsoft.VisualStudio.Component.Debugger.JustInTime","Microsoft.VisualStudio.VC.Ide.MFC","Microsoft.VisualStudio.VsDevCmd.Core.DotNet","Microsoft.VisualStudio.PackageGroup.VsDevCmd","Microsoft.VisualStudio.Finalizer","Microsoft.VisualStudio.VirtualTree","Microsoft.VisualStudio.FileHandler.Msi","Microsoft.VisualStudio.ProjectServices","Microsoft.VisualStudio.VC.DevCmd.Resources","Microsoft.VisualStudio.MinShell","Microsoft.VisualStudio.PackageGroup.Progression","Microsoft.VisualStudio.PerformanceProvider","Microsoft.VisualStudio.Connected.Resources","Microsoft.VisualStudio.Log","Microsoft.VisualStudio.PackageGroup.TeamExplorer","Microsoft.VisualStudio.Log.Targeted","Microsoft.VisualStudio.MinShell.Platform","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X86","Microsoft.VisualStudio.FileHandler.Msi","Microsoft.VisualStudio.VC.Templates.Pro.Resources","Microsoft.VisualStudio.Devenv","Microsoft.VisualCpp.VCTip.hostX86.targetX64","Microsoft.VisualStudio.Devenv.Resources","Microsoft.VisualStudio.MinShell.Platform.Resources","Microsoft.VisualStudio.Connected","Microsoft.VisualStudio.MefHosting","Microsoft.DiagnosticsHub.Collection.StopService.Uninstall","Microsoft.VisualStudio.PackageGroup.MinShell","Microsoft.VisualStudio.MefHosting.Resources","Microsoft.VisualCpp.MFC.X86","Microsoft.VisualStudio.Log.Resources","Microsoft.Icecap.Analysis.Targeted","Microsoft.VisualCpp.Premium.Tools.Hostx86.Targetx64.Resources","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64.Resources","Microsoft.VisualCpp.PGO.X86","Microsoft.VisualStudio.ExtensionManager","Microsoft.VisualStudio.MinShell.x86","Microsoft.VisualStudio.MinShell.Msi","Microsoft.VisualStudio.Setup.Configuration","Microsoft.VisualStudio.LanguageServer","Microsoft.VisualStudio.NativeImageSupport","Microsoft.VisualStudio.MinShell.Msi.Resources","Microsoft.VisualStudio.Devenv.Config","Microsoft.VisualStudio.MinShell.Resources","Microsoft.VisualStudio.Initializer","Microsoft.Net.PackageGroup.4.6.Redist"]}] diff --git a/test/fixtures/VS_2017_Unusable.txt b/test/fixtures/VS_2017_Unusable.txt new file mode 100644 index 0000000000..c037565327 --- /dev/null +++ b/test/fixtures/VS_2017_Unusable.txt @@ -0,0 +1 @@ +[{"path":"C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTools","version":"15.9.28307.665","packages":["Microsoft.VisualStudio.Product.BuildTools","Microsoft.VisualStudio.Component.Windows10SDK.17134","Win10SDK_10.0.17134","Microsoft.VisualStudio.Component.VC.Tools.x86.x64","Microsoft.VisualCpp.CodeAnalysis.Extensions","Microsoft.VisualCpp.CodeAnalysis.Extensions.X86","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X86","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X86.Resources","Microsoft.VisualCpp.CodeAnalysis.Extensions.X64","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X64","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X64.Resources","Microsoft.VisualStudio.Component.Static.Analysis.Tools","Microsoft.VisualStudio.StaticAnalysis","Microsoft.VisualStudio.StaticAnalysis.Resources","Microsoft.VisualCpp.Tools.HostX64.TargetX86","Microsoft.VisualCpp.VCTip.HostX64.TargetX86","Microsoft.VisualCpp.Tools.HostX64.TargetX86.Resources","Microsoft.VisualCpp.Tools.HostX64.TargetX64","Microsoft.VisualCpp.VCTip.HostX64.TargetX64","Microsoft.VisualCpp.Tools.HostX64.TargetX64.Resources","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX64","Microsoft.VisualCpp.Premium.Tools.Hostx86.Targetx64.Resources","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86.Resources","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86.Resources","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64.Resources","Microsoft.VisualCpp.PGO.X86","Microsoft.VisualCpp.PGO.X64","Microsoft.VisualCpp.PGO.Headers","Microsoft.VisualCpp.CRT.x86.Store","Microsoft.VisualCpp.CRT.x86.OneCore.Desktop","Microsoft.VisualCpp.CRT.x64.Store","Microsoft.VisualCpp.CRT.x64.OneCore.Desktop","Microsoft.VisualCpp.CRT.Redist.x86.OneCore.Desktop","Microsoft.VisualCpp.CRT.Redist.x64.OneCore.Desktop","Microsoft.VisualCpp.CRT.ClickOnce.Msi","Microsoft.VisualStudio.PackageGroup.VC.Tools.x86","Microsoft.VisualCpp.Tools.HostX86.TargetX64","Microsoft.VisualCpp.VCTip.hostX86.targetX64","Microsoft.VisualCpp.Tools.Hostx86.Targetx64.Resources","Microsoft.VisualCpp.Tools.HostX86.TargetX86","Microsoft.VisualCpp.VCTip.hostX86.targetX86","Microsoft.VisualCpp.Tools.HostX86.TargetX86.Resources","Microsoft.VisualCpp.Tools.Core.Resources","Microsoft.VisualCpp.Tools.Core.x86","Microsoft.VisualCpp.Tools.Common.Utils","Microsoft.VisualCpp.Tools.Common.Utils.Resources","Microsoft.VisualCpp.DIA.SDK","Microsoft.VisualCpp.CRT.x86.Desktop","Microsoft.VisualCpp.CRT.x64.Desktop","Microsoft.VisualCpp.CRT.Source","Microsoft.VisualCpp.CRT.Redist.X86","Microsoft.VisualCpp.CRT.Redist.X64","Microsoft.VisualCpp.CRT.Redist.Resources","Microsoft.VisualCpp.RuntimeDebug.14","Microsoft.VisualCpp.RuntimeDebug.14","Microsoft.VisualCpp.Redist.14","Microsoft.VisualCpp.Redist.14","Microsoft.VisualCpp.CRT.Headers","Microsoft.VisualStudio.Workload.MSBuildTools","Microsoft.VisualStudio.Component.CoreBuildTools","Microsoft.VisualStudio.Setup.Configuration","Microsoft.VisualStudio.PackageGroup.VsDevCmd","Microsoft.VisualStudio.VsDevCmd.Ext.NetFxSdk","Microsoft.VisualStudio.VsDevCmd.Core.WinSdk","Microsoft.VisualStudio.VsDevCmd.Core.DotNet","Microsoft.VisualStudio.VC.DevCmd","Microsoft.VisualStudio.VC.DevCmd.Resources","Microsoft.VisualStudio.BuildTools.Resources","Microsoft.VisualStudio.Net.Eula.Resources","Microsoft.Build.Dependencies","Microsoft.Build.FileTracker.Msi","Microsoft.Component.MSBuild","Microsoft.PythonTools.BuildCore.Vsix","Microsoft.NuGet.Build.Tasks","Microsoft.VisualStudio.Component.Roslyn.Compiler","Microsoft.CodeAnalysis.Compilers.Resources","Microsoft.CodeAnalysis.Compilers","Microsoft.Net.PackageGroup.4.6.1.Redist","Microsoft.Net.4.6.1.FullRedist.NonThreshold","Microsoft.Windows.UniversalCRT.Msu.81","Microsoft.VisualStudio.NativeImageSupport","Microsoft.Build"]}] diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js new file mode 100644 index 0000000000..d7bb1d85a7 --- /dev/null +++ b/test/test-find-visualstudio.js @@ -0,0 +1,127 @@ +'use strict' + +const test = require('tape') +const fs = require('fs') +const path = require('path') +const findVisualStudio = require('../lib/find-visualstudio') +const VisualStudioFinder = findVisualStudio.test.VisualStudioFinder + +test('empty output', function (t) { + t.plan(2) + + const finder = new VisualStudioFinder(function (err, info) { + t.ok(/se PowerShell/i.test(err), 'expect error') + t.false(info, 'no data') + }) + + finder.parseData(null, '', '') +}) + +test('output not JSON', function (t) { + t.plan(2) + + const finder = new VisualStudioFinder(function (err, info) { + t.ok(/use PowerShell/i.test(err), 'expect error') + t.false(info, 'no data') + }) + + finder.parseData(null, 'AAAABBBB', '') +}) + +test('wrong JSON', function (t) { + t.plan(2) + + const finder = new VisualStudioFinder(function (err, info) { + t.ok(/use PowerShell/i.test(err), 'expect error') + t.false(info, 'no data') + }) + + finder.parseData(null, '{}', '') +}) + +test('empty JSON', function (t) { + t.plan(2) + + const finder = new VisualStudioFinder(function (err, info) { + t.ok(/find any Visual Studio/i.test(err), 'expect error') + t.false(info, 'no data') + }) + + finder.parseData(null, '[]', '') +}) + +test('future version', function (t) { + t.plan(2) + + const finder = new VisualStudioFinder(function (err, info) { + t.ok(/find any Visual Studio/i.test(err), 'expect error') + t.false(info, 'no data') + }) + + finder.parseData(null, JSON.stringify([{ + packages: [ + 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64', + 'Microsoft.VisualStudio.Component.Windows10SDK.17763', + 'Microsoft.VisualStudio.VC.MSBuild.Base' + ], + path: 'C:\\VS', + version: '9999.9999.9999.9999' + }]), '') +}) + +test('single unusable VS2017', function (t) { + t.plan(2) + + const finder = new VisualStudioFinder(function (err, info) { + t.ok(/find any Visual Studio/i.test(err), 'expect error') + t.false(info, 'no data') + }) + + const file = path.join(__dirname, 'fixtures', 'VS_2017_Unusable.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '') +}) + +test('minimal VS2017 Build Tools', function (t) { + t.plan(2) + + const finder = new VisualStudioFinder(function (err, info) { + t.strictEqual(err, null) + t.deepEqual(info, { + hasMSBuild: true, + path: + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTools', + sdk: '10.0.17134.0', + toolset: 'v141', + version: '15.9.28307.665', + versionYear: 2017 + }) + }) + + const file = path.join(__dirname, 'fixtures', + 'VS_2017_BuildTools_minimal.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '') +}) + +test('VS2017 Community with C++ workload', function (t) { + t.plan(2) + + const finder = new VisualStudioFinder(function (err, info) { + t.strictEqual(err, null) + t.deepEqual(info, { + hasMSBuild: true, + path: + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community', + sdk: '10.0.17763.0', + toolset: 'v141', + version: '15.9.28307.665', + versionYear: 2017 + }) + }) + + const file = path.join(__dirname, 'fixtures', + 'VS_2017_Community_workload.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '') +}) From 8f43f6827583b9d0168f6c30bd6729c00a9ed070 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Mon, 27 May 2019 17:05:49 +0100 Subject: [PATCH 032/551] win: detect all VS versions in node-gyp PR-URL: https://github.com/nodejs/node-gyp/pull/1762 Reviewed-By: Rod Vagg Reviewed-By: Refael Ackermann --- lib/build.js | 76 +------ lib/configure.js | 47 ++--- lib/find-visualstudio.js | 298 ++++++++++++++++++++------ lib/util.js | 52 ++++- test/test-find-visualstudio.js | 369 +++++++++++++++++++++++++++++---- 5 files changed, 636 insertions(+), 206 deletions(-) diff --git a/lib/build.js b/lib/build.js index eeaf680c6b..783ab9c595 100644 --- a/lib/build.js +++ b/lib/build.js @@ -6,7 +6,6 @@ var fs = require('graceful-fs') , glob = require('glob') , log = require('npmlog') , which = require('which') - , exec = require('child_process').exec , win = process.platform === 'win32' exports.usage = 'Invokes `' + (win ? 'msbuild' : 'make') + '` and builds the module' @@ -96,7 +95,11 @@ function build (gyp, argv, callback) { function doWhich () { // On Windows use msbuild provided by node-gyp configure - if (win && config.variables.msbuild_path) { + if (win) { + if (!config.variables.msbuild_path) { + return callback(new Error( + 'MSBuild is not set, please run `node-gyp configure`.')) + } command = config.variables.msbuild_path log.verbose('using MSBuild:', command) doBuild() @@ -105,13 +108,8 @@ function build (gyp, argv, callback) { // First make sure we have the build command in the PATH which(command, function (err, execPath) { if (err) { - if (win && /not found/.test(err.message)) { - // On windows and no 'msbuild' found. Let's guess where it is - findMsbuild() - } else { - // Some other error or 'make' not found on Unix, report that to the user - callback(err) - } + // Some other error or 'make' not found on Unix, report that to the user + callback(err) return } log.verbose('`which` succeeded for `' + command + '`', execPath) @@ -119,66 +117,6 @@ function build (gyp, argv, callback) { }) } - /** - * Search for the location of "msbuild.exe" file on Windows. - */ - - function findMsbuild () { - log.verbose('could not find "msbuild.exe" in PATH - finding location in registry') - var notfoundErr = 'Can\'t find "msbuild.exe". Do you have Microsoft Visual Studio C++ 2008+ installed?' - var cmd = 'reg query "HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions" /s' - if (process.arch !== 'ia32') - cmd += ' /reg:32' - exec(cmd, function (err, stdout) { - if (err) { - return callback(new Error(err.message + '\n' + notfoundErr)) - } - var reVers = /ToolsVersions\\([^\\]+)$/i - , rePath = /\r\n[ \t]+MSBuildToolsPath[ \t]+REG_SZ[ \t]+([^\r]+)/i - , msbuilds = [] - , r - , msbuildPath - stdout.split('\r\n\r\n').forEach(function(l) { - if (!l) return - l = l.trim() - if (r = reVers.exec(l.substring(0, l.indexOf('\r\n')))) { - var ver = parseFloat(r[1], 10) - if (ver >= 3.5) { - if (r = rePath.exec(l)) { - msbuilds.push({ - version: ver, - path: r[1] - }) - } - } - } - }) - msbuilds.sort(function (x, y) { - return (x.version < y.version ? -1 : 1) - }) - ;(function verifyMsbuild () { - if (!msbuilds.length) return callback(new Error(notfoundErr)) - msbuildPath = path.resolve(msbuilds.pop().path, 'msbuild.exe') - fs.stat(msbuildPath, function (err) { - if (err) { - if (err.code == 'ENOENT') { - if (msbuilds.length) { - return verifyMsbuild() - } else { - callback(new Error(notfoundErr)) - } - } else { - callback(err) - } - return - } - command = msbuildPath - doBuild() - }) - })() - }) - } - /** * Actually spawn the process and compile the module. */ diff --git a/lib/configure.js b/lib/configure.js index 254824f824..6e49c23d91 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -83,22 +83,16 @@ function configure (gyp, argv, callback) { mkdirp(buildDir, function (err, isNew) { if (err) return callback(err) log.verbose('build dir', '"build" dir needed to be created?', isNew) - if (win && (!gyp.opts.msvs_version || gyp.opts.msvs_version === '2017')) { - findVisualStudio(function (err, vsSetup) { - if (err) { - log.verbose('Not using VS2017:', err.message) - createConfigFile() - } else { - createConfigFile(null, vsSetup) - } - }) + if (win) { + findVisualStudio(release.semver, gyp.opts.msvs_version, + createConfigFile) } else { createConfigFile() } }) } - function createConfigFile (err, vsSetup) { + function createConfigFile (err, vsInfo) { if (err) return callback(err) var configFilename = 'config.gypi' @@ -145,17 +139,14 @@ function configure (gyp, argv, callback) { // disable -T "thin" static archives by default variables.standalone_static_library = gyp.opts.thin ? 0 : 1 - if (vsSetup) { - // GYP doesn't (yet) have support for VS2017, so we force it to VS2015 - // to avoid pulling a floating patch that has not landed upstream. - // Ref: https://chromium-review.googlesource.com/#/c/433540/ - gyp.opts.msvs_version = '2015' - process.env['GYP_MSVS_VERSION'] = 2015 - process.env['GYP_MSVS_OVERRIDE_PATH'] = vsSetup.path - defaults['msbuild_toolset'] = 'v141' - defaults['msvs_windows_target_platform_version'] = vsSetup.sdk - variables['msbuild_path'] = path.join(vsSetup.path, 'MSBuild', '15.0', - 'Bin', 'MSBuild.exe') + if (win) { + process.env['GYP_MSVS_VERSION'] = Math.min(vsInfo.versionYear, 2015) + process.env['GYP_MSVS_OVERRIDE_PATH'] = vsInfo.path + defaults['msbuild_toolset'] = vsInfo.toolset + if (vsInfo.sdk) { + defaults['msvs_windows_target_platform_version'] = vsInfo.sdk + } + variables['msbuild_path'] = vsInfo.msBuild } // loop through the rest of the opts and add the unknown ones as variables. @@ -221,20 +212,6 @@ function configure (gyp, argv, callback) { } } - function hasMsvsVersion () { - return argv.some(function (arg) { - return arg.indexOf('msvs_version') === 0 - }) - } - - if (win && !hasMsvsVersion()) { - if ('msvs_version' in gyp.opts) { - argv.push('-G', 'msvs_version=' + gyp.opts.msvs_version) - } else { - argv.push('-G', 'msvs_version=auto') - } - } - // include all the ".gypi" files that were found configs.forEach(function (config) { argv.push('-I', config) diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js index 624567154f..94cc994d79 100644 --- a/lib/find-visualstudio.js +++ b/lib/find-visualstudio.js @@ -8,20 +8,27 @@ const log = require('npmlog') const execFile = require('child_process').execFile const path = require('path').win32 const logWithPrefix = require('./util').logWithPrefix +const regSearchKeys = require('./util').regSearchKeys -function findVisualStudio (callback) { - const finder = new VisualStudioFinder(callback) +function findVisualStudio (nodeSemver, configMsvsVersion, callback) { + const finder = new VisualStudioFinder(nodeSemver, configMsvsVersion, + callback) finder.findVisualStudio() } -function VisualStudioFinder (callback) { +function VisualStudioFinder (nodeSemver, configMsvsVersion, callback) { + this.nodeSemver = nodeSemver + this.configMsvsVersion = configMsvsVersion this.callback = callback this.errorLog = [] + this.validVersions = [] } VisualStudioFinder.prototype = { log: logWithPrefix(log, 'find VS'), + regSearchKeys: regSearchKeys, + // Logs a message at verbose level, but also saves it to be displayed later // at error level if an error occurs. This should help diagnose the problem. addLog: function addLog (message) { @@ -30,6 +37,85 @@ VisualStudioFinder.prototype = { }, findVisualStudio: function findVisualStudio () { + this.configVersionYear = null + this.configPath = null + if (this.configMsvsVersion) { + this.addLog('msvs_version was set from command line or npm config') + if (this.configMsvsVersion.match(/^\d{4}$/)) { + this.configVersionYear = parseInt(this.configMsvsVersion, 10) + this.addLog( + `- looking for Visual Studio version ${this.configVersionYear}`) + } else { + this.configPath = path.resolve(this.configMsvsVersion) + this.addLog( + `- looking for Visual Studio installed in "${this.configPath}"`) + } + } else { + this.addLog('msvs_version not set from command line or npm config') + } + + this.findVisualStudio2017OrNewer((info) => { + if (info) { + return this.succeed(info) + } + this.findVisualStudio2015((info) => { + if (info) { + return this.succeed(info) + } + this.findVisualStudio2013((info) => { + if (info) { + return this.succeed(info) + } + this.fail() + }) + }) + }) + }, + + succeed: function succeed (info) { + this.log.info(`using VS${info.versionYear} (${info.version}) found at:` + + `\n"${info.path}"` + + '\nrun with --verbose for detailed information') + process.nextTick(this.callback.bind(null, null, info)) + }, + + fail: function fail () { + // If msvs_version was specified but finding VS failed, print what would + // have been accepted + if (this.configMsvsVersion) { + this.errorLog.push('') + if (this.validVersions) { + this.errorLog.push('valid versions for msvs_version:') + this.validVersions.forEach((version) => { + this.errorLog.push(`- "${version}"`) + }) + } else { + this.errorLog.push('no valid versions for msvs_version were found') + } + } + + const errorLog = this.errorLog.join('\n') + + // For Windows 80 col console, use up to the column before the one marked + // with X (total 79 chars including logger prefix, 62 chars usable here): + // X + const infoLog = [ + '**************************************************************', + 'You need to install the latest version of Visual Studio', + 'including the "Desktop development with C++" workload.', + 'For more information consult the documentation at:', + 'https://github.com/nodejs/node-gyp#on-windows', + '**************************************************************' + ].join('\n') + + this.log.error(`\n${errorLog}\n\n${infoLog}\n`) + process.nextTick(this.callback.bind(null, new Error( + 'Could not find any Visual Studio installation to use'))) + }, + + // Invoke the PowerShell script to get information about Visual Studio 2017 + // or newer installations + findVisualStudio2017OrNewer: function findVisualStudio2017OrNewer (cb) { var ps = path.join(process.env.SystemRoot, 'System32', 'WindowsPowerShell', 'v1.0', 'powershell.exe') var csFile = path.join(__dirname, 'Find-VisualStudio.cs') @@ -39,16 +125,26 @@ VisualStudioFinder.prototype = { this.log.silly('Running', ps, psArgs) var child = execFile(ps, psArgs, { encoding: 'utf8' }, - this.parseData.bind(this)) + (err, stdout, stderr) => { + this.parseData(err, stdout, stderr, cb) + }) child.stdin.end() }, - parseData: function parseData (err, stdout, stderr) { + // Parse the output of the PowerShell script and look for an installation + // of Visual Studio 2017 or newer to use + parseData: function parseData (err, stdout, stderr, cb) { this.log.silly('PS stderr = %j', stderr) + const failPowershell = () => { + this.addLog( + 'could not use PowerShell to find Visual Studio 2017 or newer') + cb(null) + } + if (err) { this.log.silly('PS err = %j', err && (err.stack || err)) - return this.failPowershell() + return failPowershell() } var vsInfo @@ -57,25 +153,22 @@ VisualStudioFinder.prototype = { } catch (e) { this.log.silly('PS stdout = %j', stdout) this.log.silly(e) - return this.failPowershell() + return failPowershell() } if (!Array.isArray(vsInfo)) { this.log.silly('PS stdout = %j', stdout) - return this.failPowershell() + return failPowershell() } vsInfo = vsInfo.map((info) => { this.log.silly(`processing installation: "${info.path}"`) - const versionYear = this.getVersionYear(info) - return { - path: info.path, - version: info.version, - versionYear: versionYear, - hasMSBuild: this.getHasMSBuild(info), - toolset: this.getToolset(info, versionYear), - sdk: this.getSDK(info) - } + var ret = this.getVersionInfo(info) + ret.path = info.path + ret.msBuild = this.getMSBuild(info, ret.versionYear) + ret.toolset = this.getToolset(info, ret.versionYear) + ret.sdk = this.getSDK(info) + return ret }) this.log.silly('vsInfo:', vsInfo) @@ -92,9 +185,9 @@ VisualStudioFinder.prototype = { for (var i = 0; i < vsInfo.length; ++i) { const info = vsInfo[i] this.addLog(`checking VS${info.versionYear} (${info.version}) found ` + - `at\n"${info.path}"`) + `at:\n"${info.path}"`) - if (info.hasMSBuild) { + if (info.msBuild) { this.addLog('- found "Visual Studio C++ core features"') } else { this.addLog('- "Visual Studio C++ core features" missing') @@ -115,58 +208,52 @@ VisualStudioFinder.prototype = { continue } - this.succeed(info) - return - } - - this.fail() - }, - - succeed: function succeed (info) { - this.log.info(`using VS${info.versionYear} (${info.version}) found ` + - `at\n"${info.path}"`) - process.nextTick(this.callback.bind(null, null, info)) - }, - - failPowershell: function failPowershell () { - process.nextTick(this.callback.bind(null, new Error( - 'Could not use PowerShell to find Visual Studio'))) - }, - - fail: function fail () { - const errorLog = this.errorLog.join('\n') + if (!this.checkConfigVersion(info.versionYear, info.path)) { + continue + } - // For Windows 80 col console, use up to the column before the one marked - // with X (total 79 chars including logger prefix, 62 chars usable here): - // X - const infoLog = [ - '**************************************************************', - 'You need to install the latest version of Visual Studio', - 'including the "Desktop development with C++" workload.', - 'For more information consult the documentation at:', - 'https://github.com/nodejs/node-gyp#on-windows', - '**************************************************************' - ].join('\n') + return cb(info) + } - this.log.error(`\n${errorLog}\n\n${infoLog}\n`) - process.nextTick(this.callback.bind(null, new Error( - 'Could not find any Visual Studio installation to use'))) + this.addLog( + 'could not find a version of Visual Studio 2017 or newer to use') + cb(null) }, - getVersionYear: function getVersionYear (info) { - const version = parseInt(info.version, 10) - if (version === 15) { - return 2017 + // Helper - process version information + getVersionInfo: function getVersionInfo (info) { + const match = /^(\d+)\.(\d+)\..*/.exec(info.version) + if (!match) { + this.log.silly('- failed to parse version:', info.version) + return {} } - this.log.silly('- failed to parse version:', info.version) - return null + this.log.silly('- version match = %j', match) + var ret = { + version: info.version, + versionMajor: parseInt(match[1], 10), + versionMinor: parseInt(match[2], 10) + } + if (ret.versionMajor === 15) { + ret.versionYear = 2017 + return ret + } + this.log.silly('- unsupported version:', ret.versionMajor) + return {} }, - getHasMSBuild: function getHasMSBuild (info) { + // Helper - process MSBuild information + getMSBuild: function getMSBuild (info, versionYear) { const pkg = 'Microsoft.VisualStudio.VC.MSBuild.Base' - return info.packages.indexOf(pkg) !== -1 + if (info.packages.indexOf(pkg) !== -1) { + this.log.silly('- found VC.MSBuild.Base') + if (versionYear === 2017) { + return path.join(info.path, 'MSBuild', '15.0', 'Bin', 'MSBuild.exe') + } + } + return null }, + // Helper - process toolset information getToolset: function getToolset (info, versionYear) { const pkg = 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64' if (info.packages.indexOf(pkg) !== -1) { @@ -178,6 +265,7 @@ VisualStudioFinder.prototype = { return null }, + // Helper - process Windows SDK information getSDK: function getSDK (info) { const win8SDK = 'Microsoft.VisualStudio.Component.Windows81SDK' const win10SDKPrefix = 'Microsoft.VisualStudio.Component.Windows10SDK.' @@ -209,5 +297,93 @@ VisualStudioFinder.prototype = { return '8.1' } return null + }, + + // Find an installation of Visual Studio 2015 to use + findVisualStudio2015: function findVisualStudio2015 (cb) { + return this.findOldVS({ + version: '14.0', + versionMajor: 14, + versionMinor: 0, + versionYear: 2015, + toolset: 'v140' + }, cb) + }, + + // Find an installation of Visual Studio 2013 to use + findVisualStudio2013: function findVisualStudio2013 (cb) { + if (this.nodeSemver.major >= 9) { + this.addLog( + 'not looking for VS2013 as it is only supported up to Node.js 8') + return cb(null) + } + return this.findOldVS({ + version: '12.0', + versionMajor: 12, + versionMinor: 0, + versionYear: 2013, + toolset: 'v120' + }, cb) + }, + + // Helper - common code for VS2013 and VS2015 + findOldVS: function findOldVS (info, cb) { + const regVC7 = ['HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7', + 'HKLM\\Software\\Wow6432Node\\Microsoft\\VisualStudio\\SxS\\VC7'] + const regMSBuild = 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions' + + this.addLog(`looking for Visual Studio ${info.versionYear}`) + this.regSearchKeys(regVC7, info.version, [], (err, res) => { + if (err) { + this.addLog('- not found') + return cb(null) + } + + const vsPath = path.resolve(res, '..') + this.addLog(`- found in "${vsPath}"`) + + const msBuildRegOpts = process.arch === 'ia32' ? [] : ['/reg:32'] + this.regSearchKeys([`${regMSBuild}\\${info.version}`], + 'MSBuildToolsPath', msBuildRegOpts, (err, res) => { + if (err) { + this.addLog( + '- could not find MSBuild in registry for this version') + return cb(null) + } + + const msBuild = path.join(res, 'MSBuild.exe') + this.addLog(`- MSBuild in "${msBuild}"`) + + if (!this.checkConfigVersion(info.versionYear, vsPath)) { + return cb(null) + } + + info.path = vsPath + info.msBuild = msBuild + info.sdk = null + cb(info) + }) + }) + }, + + // After finding a usable version of Visual Stuido: + // - add it to validVersions to be displayed at the end if a specific + // version was requested and not found; + // - check if this is the version that was requested. + checkConfigVersion: function checkConfigVersion (versionYear, vsPath) { + this.validVersions.push(versionYear) + this.validVersions.push(vsPath) + + if (this.configVersionYear && + this.configVersionYear !== versionYear) { + this.addLog('- not looking for this version') + return false + } + if (this.configPath && this.configPath !== vsPath) { + this.addLog('- not looking for this installation') + return false + } + + return true } } diff --git a/lib/util.js b/lib/util.js index cb12b936ef..ac6a875354 100644 --- a/lib/util.js +++ b/lib/util.js @@ -1,4 +1,14 @@ -module.exports.logWithPrefix = function logWithPrefix (log, prefix) { +module.exports = { + logWithPrefix: logWithPrefix, + regGetValue: regGetValue, + regSearchKeys: regSearchKeys +} + +const log = require('npmlog') +const execFile = require('child_process').execFile +const path = require('path') + +function logWithPrefix (log, prefix) { function setPrefix(logFunction) { return (...args) => logFunction.apply(null, [prefix, ...args]) } @@ -10,3 +20,43 @@ module.exports.logWithPrefix = function logWithPrefix (log, prefix) { error: setPrefix(log.error), } } + +function regGetValue (key, value, addOpts, cb) { + const outReValue = value.replace(/\W/g, '.') + const outRe = new RegExp(`^\\s+${outReValue}\\s+REG_\\w+\\s+(\\S.*)$`, 'im') + const reg = path.join(process.env.SystemRoot, 'System32', 'reg.exe') + const regArgs = ['query', key, '/v', value].concat(addOpts) + + log.silly('reg', 'running', reg, regArgs) + const child = execFile(reg, regArgs, { encoding: 'utf8' }, + function (err, stdout, stderr) { + log.silly('reg', 'reg.exe stdout = %j', stdout) + if (err || stderr.trim() !== '') { + log.silly('reg', 'reg.exe err = %j', err && (err.stack || err)) + log.silly('reg', 'reg.exe stderr = %j', stderr) + return cb(err, stderr) + } + + const result = outRe.exec(stdout) + if (!result) { + log.silly('reg', 'error parsing stdout') + return cb(new Error('Could not parse output of reg.exe')) + } + log.silly('reg', 'found: %j', result[1]) + cb(null, result[1]) + }) + child.stdin.end() +} + +function regSearchKeys (keys, value, addOpts, cb) { + var i = 0 + const search = () => { + log.silly('reg-search', 'looking for %j in %j', value, keys[i]) + regGetValue(keys[i], value, addOpts, (err, res) => { + ++i + if (err && i < keys.length) { return search() } + cb(err, res) + }) + } + search() +} diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js index d7bb1d85a7..507db96792 100644 --- a/test/test-find-visualstudio.js +++ b/test/test-find-visualstudio.js @@ -6,57 +6,206 @@ const path = require('path') const findVisualStudio = require('../lib/find-visualstudio') const VisualStudioFinder = findVisualStudio.test.VisualStudioFinder -test('empty output', function (t) { - t.plan(2) +const semverV1 = { major: 1, minor: 0, patch: 0 } - const finder = new VisualStudioFinder(function (err, info) { - t.ok(/se PowerShell/i.test(err), 'expect error') - t.false(info, 'no data') +function poison (object, property) { + function fail () { + console.error(Error(`Property ${property} should not have been accessed.`)) + process.abort() + } + var descriptor = { + configurable: false, + enumerable: false, + get: fail, + set: fail + } + Object.defineProperty(object, property, descriptor) +} + +function TestVisualStudioFinder () { VisualStudioFinder.apply(this, arguments) } +TestVisualStudioFinder.prototype = Object.create(VisualStudioFinder.prototype) +// Silence npmlog - remove for debugging +TestVisualStudioFinder.prototype.log = { + silly: () => {}, + verbose: () => {}, + info: () => {}, + warn: () => {}, + error: () => {} +} + +test('VS2013', function (t) { + t.plan(4) + + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info, { + msBuild: 'C:\\MSBuild12\\MSBuild.exe', + path: 'C:\\VS2013', + sdk: null, + toolset: 'v120', + version: '12.0', + versionMajor: 12, + versionMinor: 0, + versionYear: 2013 + }) }) - finder.parseData(null, '', '') + finder.findVisualStudio2017OrNewer = (cb) => { + finder.parseData(new Error(), '', '', cb) + } + finder.regSearchKeys = (keys, value, addOpts, cb) => { + for (var i = 0; i < keys.length; ++i) { + const fullName = `${keys[i]}\\${value}` + switch (fullName) { + case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': + case 'HKLM\\Software\\Wow6432Node\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': + continue + case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\12.0': + t.pass(`expected search for registry value ${fullName}`) + return cb(null, 'C:\\VS2013\\VC\\') + case 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions\\12.0\\MSBuildToolsPath': + t.pass(`expected search for registry value ${fullName}`) + return cb(null, 'C:\\MSBuild12\\') + default: + t.fail(`unexpected search for registry value ${fullName}`) + } + } + return cb(new Error()) + } + finder.findVisualStudio() }) -test('output not JSON', function (t) { +test('VS2013 should not be found on new node versions', function (t) { t.plan(2) - const finder = new VisualStudioFinder(function (err, info) { - t.ok(/use PowerShell/i.test(err), 'expect error') + const finder = new TestVisualStudioFinder({ + major: 10, + minor: 0, + patch: 0 + }, null, (err, info) => { + t.ok(/find .* Visual Studio/i.test(err), 'expect error') t.false(info, 'no data') }) - finder.parseData(null, 'AAAABBBB', '') + finder.findVisualStudio2017OrNewer = (cb) => { + const file = path.join(__dirname, 'fixtures', 'VS_2017_Unusable.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '', cb) + } + finder.regSearchKeys = (keys, value, addOpts, cb) => { + for (var i = 0; i < keys.length; ++i) { + const fullName = `${keys[i]}\\${value}` + switch (fullName) { + case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': + case 'HKLM\\Software\\Wow6432Node\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': + continue + default: + t.fail(`unexpected search for registry value ${fullName}`) + } + } + return cb(new Error()) + } + finder.findVisualStudio() }) -test('wrong JSON', function (t) { +test('VS2015', function (t) { + t.plan(4) + + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info, { + msBuild: 'C:\\MSBuild14\\MSBuild.exe', + path: 'C:\\VS2015', + sdk: null, + toolset: 'v140', + version: '14.0', + versionMajor: 14, + versionMinor: 0, + versionYear: 2015 + }) + }) + + finder.findVisualStudio2017OrNewer = (cb) => { + finder.parseData(new Error(), '', '', cb) + } + finder.regSearchKeys = (keys, value, addOpts, cb) => { + for (var i = 0; i < keys.length; ++i) { + const fullName = `${keys[i]}\\${value}` + switch (fullName) { + case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': + t.pass(`expected search for registry value ${fullName}`) + return cb(null, 'C:\\VS2015\\VC\\') + case 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions\\14.0\\MSBuildToolsPath': + t.pass(`expected search for registry value ${fullName}`) + return cb(null, 'C:\\MSBuild14\\') + default: + t.fail(`unexpected search for registry value ${fullName}`) + } + } + return cb(new Error()) + } + finder.findVisualStudio() +}) + +test('error from PowerShell', function (t) { t.plan(2) - const finder = new VisualStudioFinder(function (err, info) { - t.ok(/use PowerShell/i.test(err), 'expect error') + const finder = new TestVisualStudioFinder(semverV1, null, null) + + finder.parseData(new Error(), '', '', (info) => { + t.ok(/use PowerShell/i.test(finder.errorLog[0]), 'expect error') t.false(info, 'no data') }) +}) - finder.parseData(null, '{}', '') +test('empty output from PowerShell', function (t) { + t.plan(2) + + const finder = new TestVisualStudioFinder(semverV1, null, null) + + finder.parseData(null, '', '', (info) => { + t.ok(/use PowerShell/i.test(finder.errorLog[0]), 'expect error') + t.false(info, 'no data') + }) }) -test('empty JSON', function (t) { +test('output from PowerShell not JSON', function (t) { t.plan(2) - const finder = new VisualStudioFinder(function (err, info) { - t.ok(/find any Visual Studio/i.test(err), 'expect error') + const finder = new TestVisualStudioFinder(semverV1, null, null) + + finder.parseData(null, 'AAAABBBB', '', (info) => { + t.ok(/use PowerShell/i.test(finder.errorLog[0]), 'expect error') t.false(info, 'no data') }) +}) + +test('wrong JSON from PowerShell', function (t) { + t.plan(2) + + const finder = new TestVisualStudioFinder(semverV1, null, null) - finder.parseData(null, '[]', '') + finder.parseData(null, '{}', '', (info) => { + t.ok(/use PowerShell/i.test(finder.errorLog[0]), 'expect error') + t.false(info, 'no data') + }) }) -test('future version', function (t) { +test('empty JSON from PowerShell', function (t) { t.plan(2) - const finder = new VisualStudioFinder(function (err, info) { - t.ok(/find any Visual Studio/i.test(err), 'expect error') + const finder = new TestVisualStudioFinder(semverV1, null, null) + + finder.parseData(null, '[]', '', (info) => { + t.ok(/find .* Visual Studio/i.test(finder.errorLog[0]), 'expect error') t.false(info, 'no data') }) +}) + +test('future version', function (t) { + t.plan(3) + + const finder = new TestVisualStudioFinder(semverV1, null, null) finder.parseData(null, JSON.stringify([{ packages: [ @@ -66,62 +215,202 @@ test('future version', function (t) { ], path: 'C:\\VS', version: '9999.9999.9999.9999' - }]), '') + }]), '', (info) => { + t.ok(/unknown version/i.test(finder.errorLog[0]), 'expect error') + t.ok(/find .* Visual Studio/i.test(finder.errorLog[1]), 'expect error') + t.false(info, 'no data') + }) }) test('single unusable VS2017', function (t) { - t.plan(2) + t.plan(3) - const finder = new VisualStudioFinder(function (err, info) { - t.ok(/find any Visual Studio/i.test(err), 'expect error') - t.false(info, 'no data') - }) + const finder = new TestVisualStudioFinder(semverV1, null, null) const file = path.join(__dirname, 'fixtures', 'VS_2017_Unusable.txt') const data = fs.readFileSync(file) - finder.parseData(null, data, '') + finder.parseData(null, data, '', (info) => { + t.ok(/checking/i.test(finder.errorLog[0]), 'expect error') + t.ok(/find .* Visual Studio/i.test(finder.errorLog[2]), 'expect error') + t.false(info, 'no data') + }) }) test('minimal VS2017 Build Tools', function (t) { t.plan(2) - const finder = new VisualStudioFinder(function (err, info) { + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { t.strictEqual(err, null) t.deepEqual(info, { - hasMSBuild: true, + msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\' + + 'BuildTools\\MSBuild\\15.0\\Bin\\MSBuild.exe', path: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTools', sdk: '10.0.17134.0', toolset: 'v141', version: '15.9.28307.665', + versionMajor: 15, + versionMinor: 9, versionYear: 2017 }) }) - const file = path.join(__dirname, 'fixtures', - 'VS_2017_BuildTools_minimal.txt') - const data = fs.readFileSync(file) - finder.parseData(null, data, '') + poison(finder, 'regSearchKeys') + finder.findVisualStudio2017OrNewer = (cb) => { + const file = path.join(__dirname, 'fixtures', + 'VS_2017_BuildTools_minimal.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '', cb) + } + finder.findVisualStudio() }) test('VS2017 Community with C++ workload', function (t) { t.plan(2) - const finder = new VisualStudioFinder(function (err, info) { + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { t.strictEqual(err, null) t.deepEqual(info, { - hasMSBuild: true, + msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\' + + 'Community\\MSBuild\\15.0\\Bin\\MSBuild.exe', path: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community', sdk: '10.0.17763.0', toolset: 'v141', version: '15.9.28307.665', + versionMajor: 15, + versionMinor: 9, versionYear: 2017 }) }) - const file = path.join(__dirname, 'fixtures', - 'VS_2017_Community_workload.txt') - const data = fs.readFileSync(file) - finder.parseData(null, data, '') + poison(finder, 'regSearchKeys') + finder.findVisualStudio2017OrNewer = (cb) => { + const file = path.join(__dirname, 'fixtures', + 'VS_2017_Community_workload.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '', cb) + } + finder.findVisualStudio() +}) + +function allVsVersions (t, finder) { + finder.findVisualStudio2017OrNewer = (cb) => { + const file1 = path.join(__dirname, 'fixtures', 'VS_2017_BuildTools_minimal.txt') + const data1 = JSON.parse(fs.readFileSync(file1)) + const file2 = path.join(__dirname, 'fixtures', 'VS_2017_Community_workload.txt') + const data2 = JSON.parse(fs.readFileSync(file2)) + const data = JSON.stringify(data1.concat(data2)) + finder.parseData(null, data, '', cb) + } + finder.regSearchKeys = (keys, value, addOpts, cb) => { + for (var i = 0; i < keys.length; ++i) { + const fullName = `${keys[i]}\\${value}` + switch (fullName) { + case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': + case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\12.0': + continue + case 'HKLM\\Software\\Wow6432Node\\Microsoft\\VisualStudio\\SxS\\VC7\\12.0': + return cb(null, 'C:\\VS2013\\VC\\') + case 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions\\12.0\\MSBuildToolsPath': + return cb(null, 'C:\\MSBuild12\\') + case 'HKLM\\Software\\Wow6432Node\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': + return cb(null, 'C:\\VS2015\\VC\\') + case 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions\\14.0\\MSBuildToolsPath': + return cb(null, 'C:\\MSBuild14\\') + default: + t.fail(`unexpected search for registry value ${fullName}`) + } + } + return cb(new Error()) + } +} + +test('fail when looking for invalid path', function (t) { + t.plan(2) + + const finder = new TestVisualStudioFinder(semverV1, 'AABB', (err, info) => { + t.ok(/find .* Visual Studio/i.test(err), 'expect error') + t.false(info, 'no data') + }) + + allVsVersions(t, finder) + finder.findVisualStudio() +}) + +test('look for VS2013 by version number', function (t) { + t.plan(2) + + const finder = new TestVisualStudioFinder(semverV1, '2013', (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info.versionYear, 2013) + }) + + allVsVersions(t, finder) + finder.findVisualStudio() +}) + +test('look for VS2013 by installation path', function (t) { + t.plan(2) + + const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2013', + (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info.path, 'C:\\VS2013') + }) + + allVsVersions(t, finder) + finder.findVisualStudio() +}) + +test('look for VS2015 by version number', function (t) { + t.plan(2) + + const finder = new TestVisualStudioFinder(semverV1, '2015', (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info.versionYear, 2015) + }) + + allVsVersions(t, finder) + finder.findVisualStudio() +}) + +test('look for VS2015 by installation path', function (t) { + t.plan(2) + + const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2015', + (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info.path, 'C:\\VS2015') + }) + + allVsVersions(t, finder) + finder.findVisualStudio() +}) + +test('look for VS2017 by version number', function (t) { + t.plan(2) + + const finder = new TestVisualStudioFinder(semverV1, '2017', (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info.versionYear, 2017) + }) + + allVsVersions(t, finder) + finder.findVisualStudio() +}) + +test('look for VS2017 by installation path', function (t) { + t.plan(2) + + const finder = new TestVisualStudioFinder(semverV1, + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community', + (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info.path, + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community') + }) + + allVsVersions(t, finder) + finder.findVisualStudio() }) From 360ddbdf3a70ae5fc679b8267dfda46969c0d615 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Wed, 29 May 2019 00:20:52 +0100 Subject: [PATCH 033/551] win: add support for Visual Studio 2019 PR-URL: https://github.com/nodejs/node-gyp/pull/1762 Reviewed-By: Rod Vagg Reviewed-By: Refael Ackermann --- lib/find-visualstudio.js | 10 ++ test/fixtures/VS_2019_BuildTools_minimal.txt | 1 + test/fixtures/VS_2019_Community_workload.txt | 1 + test/fixtures/VS_2019_Preview.txt | 1 + test/test-find-visualstudio.js | 145 ++++++++++++++++++- 5 files changed, 153 insertions(+), 5 deletions(-) create mode 100644 test/fixtures/VS_2019_BuildTools_minimal.txt create mode 100644 test/fixtures/VS_2019_Community_workload.txt create mode 100644 test/fixtures/VS_2019_Preview.txt diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js index 94cc994d79..ea35ca9db7 100644 --- a/lib/find-visualstudio.js +++ b/lib/find-visualstudio.js @@ -237,6 +237,10 @@ VisualStudioFinder.prototype = { ret.versionYear = 2017 return ret } + if (ret.versionMajor === 16) { + ret.versionYear = 2019 + return ret + } this.log.silly('- unsupported version:', ret.versionMajor) return {} }, @@ -249,6 +253,9 @@ VisualStudioFinder.prototype = { if (versionYear === 2017) { return path.join(info.path, 'MSBuild', '15.0', 'Bin', 'MSBuild.exe') } + if (versionYear === 2019) { + return path.join(info.path, 'MSBuild', 'Current', 'Bin', 'MSBuild.exe') + } } return null }, @@ -261,6 +268,9 @@ VisualStudioFinder.prototype = { if (versionYear === 2017) { return 'v141' } + if (versionYear === 2019) { + return 'v142' + } } return null }, diff --git a/test/fixtures/VS_2019_BuildTools_minimal.txt b/test/fixtures/VS_2019_BuildTools_minimal.txt new file mode 100644 index 0000000000..f07d254164 --- /dev/null +++ b/test/fixtures/VS_2019_BuildTools_minimal.txt @@ -0,0 +1 @@ +[{"path":"C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools","version":"16.1.28922.388","packages":["Microsoft.VisualStudio.Product.BuildTools","Microsoft.VisualStudio.Component.VC.CoreIde","Microsoft.VisualStudio.VC.Ide.Pro","Microsoft.VisualStudio.VC.Ide.Pro.Resources","Microsoft.VisualStudio.VC.Templates.Pro","Microsoft.VisualStudio.VC.Templates.Pro.Resources","Microsoft.VisualStudio.VC.Items.Pro","Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Reduced","Microsoft.VisualStudio.VC.Ide.MDD","Microsoft.VisualStudio.PackageGroup.Core","Microsoft.VisualStudio.CodeSense.Community","Microsoft.VisualStudio.TestTools.TeamFoundationClient","Microsoft.PackageGroup.ClientDiagnostics","Microsoft.VisualStudio.AppResponsiveness","Microsoft.VisualStudio.AppResponsiveness.Targeted","Microsoft.VisualStudio.AppResponsiveness.Resources","Microsoft.VisualStudio.ClientDiagnostics","Microsoft.VisualStudio.ClientDiagnostics.Targeted","Microsoft.VisualStudio.ClientDiagnostics.Resources","Microsoft.VisualStudio.PackageGroup.CommunityCore","Microsoft.VisualStudio.ProjectSystem.Full","Microsoft.VisualStudio.ProjectSystem","Microsoft.VisualStudio.Community.x86","Microsoft.VisualStudio.Community.x64","Microsoft.VisualStudio.Community","Microsoft.IntelliTrace.CollectorCab","Microsoft.VisualStudio.Community.Resources","Microsoft.VisualStudio.WebSiteProject.DTE","Microsoft.MSHtml","Microsoft.VisualStudio.Platform.CallHierarchy","Microsoft.VisualStudio.Community.Msi.Resources","Microsoft.VisualStudio.Community.Msi","Microsoft.VisualStudio.MinShell.Interop.Msi","Microsoft.VisualStudio.PackageGroup.CoreEditor","Microsoft.VisualStudio.VirtualTree","Microsoft.VisualStudio.PackageGroup.Progression","Microsoft.VisualStudio.PerformanceProvider","Microsoft.VisualStudio.GraphModel","Microsoft.VisualStudio.GraphProvider","Microsoft.VisualStudio.TextMateGrammars","Microsoft.VisualStudio.PackageGroup.TeamExplorer.Common","Microsoft.VisualStudio.TeamExplorer","Microsoft.ServiceHub","Microsoft.VisualStudio.ProjectServices","Microsoft.VisualStudio.OpenFolder.VSIX","Microsoft.VisualStudio.FileHandler.Msi","Microsoft.VisualStudio.FileHandler.Msi","Microsoft.VisualStudio.PackageGroup.MinShell","Microsoft.VisualStudio.MinShell.Msi","Microsoft.VisualStudio.MinShell.Msi.Resources","Microsoft.VisualStudio.MinShell.Interop","Microsoft.VisualStudio.Log","Microsoft.VisualStudio.Log.Targeted","Microsoft.VisualStudio.Log.Resources","Microsoft.VisualStudio.Finalizer","Microsoft.VisualStudio.CoreEditor","Microsoft.VisualStudio.Platform.NavigateTo","Microsoft.VisualStudio.Connected","Microsoft.VisualStudio.Connected.Resources","Microsoft.VisualStudio.VC.Ide.x64","Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Express","Microsoft.VisualStudio.PackageGroup.Debugger.Script","Microsoft.VisualStudio.Debugger.Script.Msi","Microsoft.VisualStudio.Debugger.Script","Microsoft.VisualStudio.Debugger.Script","Microsoft.VisualStudio.Debugger.Script.Resources","Microsoft.VisualStudio.Debugger.Script.Resources","Microsoft.VisualStudio.VC.Ide.WinXPlus","Microsoft.VisualStudio.VC.Ide.Dskx","Microsoft.VisualStudio.VC.Ide.Dskx.Resources","Microsoft.VisualStudio.VC.Ide.Base","Microsoft.VisualStudio.VC.Ide.LanguageService","Microsoft.VisualStudio.VC.Ide.Core","Microsoft.VisualStudio.VisualC.Logging","Microsoft.VisualStudio.VC.Ide.Core.Resources","Microsoft.VisualStudio.VC.Ide.VCPkgDatabase","Microsoft.VisualStudio.VC.Ide.ResourceEditor","Microsoft.VisualStudio.VC.Ide.ResourceEditor.Resources","Microsoft.VisualStudio.VC.Ide.ProjectSystem","Microsoft.VisualStudio.VC.Ide.ProjectSystem.Resources","Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine","Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine.Resources","Microsoft.VisualStudio.VC.Ide.LanguageService.Resources","Microsoft.VisualStudio.VC.Ide.Base.Resources","Microsoft.Net.4.TargetingPack","Microsoft.VisualStudio.PackageGroup.Debugger.Core","Microsoft.VisualStudio.PackageGroup.Debugger.TimeTravel.Record","Microsoft.VisualStudio.Debugger.TimeTravel.Runtime","Microsoft.VisualStudio.Debugger.TimeTravel.Runtime","Microsoft.VisualStudio.Debugger.TimeTravel.Agent","Microsoft.VisualStudio.Debugger.TimeTravel.Record","Microsoft.VisualStudio.Debugger.VSCodeDebuggerHost","Microsoft.VisualStudio.VC.Ide.Debugger","Microsoft.VisualStudio.VC.Ide.Debugger.Concord","Microsoft.VisualStudio.VC.Ide.Debugger.Concord.Resources","Microsoft.VisualStudio.VC.Ide.Debugger.Resources","Microsoft.VisualStudio.VC.Ide.Common","Microsoft.VisualStudio.VC.Ide.Common.Resources","Microsoft.VisualStudio.Debugger.Parallel","Microsoft.VisualStudio.Debugger.Parallel.Resources","Microsoft.VisualStudio.Debugger.CollectionAgents","Microsoft.VisualStudio.Debugger.Managed","Microsoft.DiaSymReader","Microsoft.CodeAnalysis.ExpressionEvaluator","Microsoft.VisualStudio.Debugger.Concord.Managed","Microsoft.VisualStudio.Debugger.Concord.Managed.Resources","Microsoft.VisualStudio.Debugger.Managed.Resources","Microsoft.VisualStudio.Debugger.Remote","Microsoft.VisualStudio.Debugger.Concord.Remote","Microsoft.VisualStudio.Debugger.Concord.Remote.Resources","Microsoft.VisualStudio.Debugger.Remote","Microsoft.VisualStudio.Debugger.Concord.Remote","Microsoft.VisualStudio.Debugger.Concord.Remote.Resources","Microsoft.VisualStudio.Debugger.Remote.Resources","Microsoft.VisualStudio.Debugger.Remote.Resources","Microsoft.VisualStudio.Debugger","Microsoft.VisualStudio.PerfLib","Microsoft.VisualStudio.Debugger.Package.DiagHub.Client.VSx86","Microsoft.VisualStudio.Debugger.Remote.DiagHub.Client","Microsoft.VisualStudio.Debugger.Remote.DiagHub.Client","Microsoft.VisualStudio.VC.MSVCDis","Microsoft.VisualStudio.ScriptedHost","Microsoft.VisualStudio.ScriptedHost.Targeted","Microsoft.VisualStudio.ScriptedHost.Resources","Microsoft.VisualStudio.Editors","Microsoft.IntelliTrace.DiagnosticsHub","Microsoft.VisualStudio.MinShell","Microsoft.VisualStudio.MinShell.Platform","Microsoft.VisualStudio.MinShell.Platform.Resources","Microsoft.VisualStudio.MefHosting","Microsoft.VisualStudio.MefHosting.Resources","Microsoft.VisualStudio.Initializer","Microsoft.VisualStudio.ExtensionManager","Microsoft.VisualStudio.Platform.Editor","Microsoft.VisualStudio.Debugger.Concord","Microsoft.VisualStudio.Debugger.Concord.Resources","Microsoft.VisualStudio.Debugger.Resources","Microsoft.CodeAnalysis.VisualStudio.Setup","Microsoft.VisualStudio.Component.Windows10SDK.17134","Win10SDK_10.0.17134","Microsoft.VisualStudio.Component.VC.Tools.x86.x64","Microsoft.VisualCpp.CodeAnalysis.Extensions","Microsoft.VisualCpp.CodeAnalysis.Extensions.X86","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X86","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X86.Resources","Microsoft.VisualCpp.CodeAnalysis.Extensions.X64","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X64","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X64.Resources","Microsoft.VisualStudio.StaticAnalysis","Microsoft.VisualStudio.StaticAnalysis.Resources","Microsoft.VisualCpp.Tools.HostX64.TargetX86","Microsoft.VisualCpp.VCTip.HostX64.TargetX86","Microsoft.VisualCpp.Tools.HostX64.TargetX86.Resources","Microsoft.VisualCpp.Tools.HostX64.TargetX64","Microsoft.VisualCpp.VCTip.HostX64.TargetX64","Microsoft.VisualCpp.Tools.HostX64.TargetX64.Resources","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX64","Microsoft.VisualCpp.Premium.Tools.Hostx86.Targetx64.Resources","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86.Resources","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86.Resources","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64.Resources","Microsoft.VisualCpp.PGO.X86","Microsoft.VisualCpp.PGO.X64","Microsoft.VisualCpp.PGO.Headers","Microsoft.VisualCpp.CRT.x86.Store","Microsoft.VisualCpp.CRT.x86.OneCore.Desktop","Microsoft.VisualCpp.CRT.x64.Store","Microsoft.VisualCpp.CRT.x64.OneCore.Desktop","Microsoft.VisualCpp.CRT.Redist.x86.OneCore.Desktop","Microsoft.VisualCpp.CRT.Redist.x64.OneCore.Desktop","Microsoft.VisualCpp.CRT.ClickOnce.Msi","Microsoft.VisualStudio.PackageGroup.VC.Tools.x86","Microsoft.VisualCpp.Tools.HostX86.TargetX64","Microsoft.VisualCpp.VCTip.hostX86.targetX64","Microsoft.VisualCpp.Tools.Hostx86.Targetx64.Resources","Microsoft.VisualCpp.Tools.HostX86.TargetX86","Microsoft.VisualCpp.VCTip.hostX86.targetX86","Microsoft.VisualCpp.Tools.HostX86.TargetX86.Resources","Microsoft.VisualCpp.Tools.Core.Resources","Microsoft.VisualCpp.Tools.Core.x86","Microsoft.VisualCpp.Tools.Common.Utils","Microsoft.VisualCpp.Tools.Common.Utils.Resources","Microsoft.VisualCpp.DIA.SDK","Microsoft.VisualCpp.CRT.x86.Desktop","Microsoft.VisualCpp.CRT.x64.Desktop","Microsoft.VisualCpp.CRT.Source","Microsoft.VisualCpp.CRT.Redist.X86","Microsoft.VisualCpp.CRT.Redist.X64","Microsoft.VisualCpp.CRT.Redist.Resources","Microsoft.VisualCpp.RuntimeDebug.14","Microsoft.VisualCpp.RuntimeDebug.14","Microsoft.VisualCpp.Redist.14.Latest","Microsoft.VisualCpp.Redist.14.Latest","Microsoft.VisualCpp.Redist.14","Microsoft.VisualCpp.Redist.14","Microsoft.VisualCpp.CRT.Headers","Microsoft.VisualStudio.VC.MSBuild.x86.v142","Microsoft.VisualStudio.VC.MSBuild.X86","Microsoft.VisualStudio.VC.MSBuild.X64.v142","Microsoft.VisualStudio.VC.MSBuild.X64","Microsoft.VS.VC.MSBuild.X64.Resources","Microsoft.VisualStudio.VC.MSBuild.ARM.v142","Microsoft.VisualStudio.VC.MSBuild.ARM","Microsoft.VisualStudio.VC.MSBuild.Base","Microsoft.VisualStudio.VC.MSBuild.Base.Resources","Microsoft.VisualStudio.Workload.MSBuildTools","Microsoft.VisualStudio.Component.CoreBuildTools","Microsoft.VisualStudio.Setup.Configuration","Microsoft.VisualStudio.PackageGroup.VsDevCmd","Microsoft.VisualStudio.VsDevCmd.Ext.NetFxSdk","Microsoft.VisualStudio.VsDevCmd.Core.WinSdk","Microsoft.VisualStudio.VsDevCmd.Core.DotNet","Microsoft.VisualStudio.VC.DevCmd","Microsoft.VisualStudio.VC.DevCmd.Resources","Microsoft.VisualStudio.BuildTools.Resources","Microsoft.VisualStudio.Net.Eula.Resources","Microsoft.Build.Dependencies","Microsoft.Build.FileTracker.Msi","Microsoft.Component.MSBuild","Microsoft.PythonTools.BuildCore.Vsix","Microsoft.NuGet.Build.Tasks","Microsoft.VisualStudio.Component.Roslyn.Compiler","Microsoft.CodeAnalysis.Compilers","Microsoft.Net.PackageGroup.4.7.2.Redist","Microsoft.VisualStudio.NativeImageSupport","Microsoft.Build","Microsoft.VisualStudio.PackageGroup.NuGet","Microsoft.VisualStudio.NuGet.BuildTools"]}] diff --git a/test/fixtures/VS_2019_Community_workload.txt b/test/fixtures/VS_2019_Community_workload.txt new file mode 100644 index 0000000000..50071c25f1 --- /dev/null +++ b/test/fixtures/VS_2019_Community_workload.txt @@ -0,0 +1 @@ +[{"path":"C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community","version":"16.1.28922.388","packages":["Microsoft.VisualStudio.Workload.NativeDesktop","Microsoft.VisualStudio.Component.VC.TestAdapterForGoogleTest","Microsoft.VisualStudio.VC.Ide.TestAdapterForGoogleTest","Microsoft.VisualStudio.Component.VC.TestAdapterForBoostTest","Microsoft.VisualStudio.VC.Ide.TestAdapterForBoostTest","Microsoft.VisualStudio.Component.VC.ATL","Microsoft.VisualStudio.VC.Ide.ATL","Microsoft.VisualStudio.VC.Ide.ATL.Resources","Microsoft.VisualCpp.ATL.X86","Microsoft.VisualCpp.ATL.X64","Microsoft.VisualCpp.ATL.Source","Microsoft.VisualCpp.ATL.Headers","Microsoft.VisualStudio.Component.VC.CMake.Project","Microsoft.VisualStudio.VC.CMake","Microsoft.VisualStudio.VC.CMake.Project","Microsoft.VisualStudio.VC.ExternalBuildFramework","Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core","Microsoft.VisualStudio.PackageGroup.TestTools.Native","Microsoft.VisualStudio.Component.VC.Redist.14.Latest","Microsoft.VisualStudio.VC.Templates.UnitTest","Microsoft.VisualStudio.VC.UnitTest.Desktop.Build.Core","Microsoft.VisualStudio.TestTools.TestPlatform.V1.CPP","Microsoft.VisualStudio.VC.Templates.UnitTest.Resources","Microsoft.VisualStudio.VC.Templates.Desktop","Microsoft.VisualStudio.Component.Debugger.JustInTime","Microsoft.VisualStudio.Debugger.ImmersiveActivateHelper.Msi","Microsoft.VisualStudio.Debugger.JustInTime","Microsoft.VisualStudio.Debugger.JustInTime.Msi","Microsoft.VisualStudio.Component.Windows10SDK.17763","Win10SDK_10.0.17763","Microsoft.VisualStudio.Component.VC.DiagnosticTools","Microsoft.VisualStudio.Component.Graphics.Tools","Microsoft.VisualStudio.Component.VC.Tools.x86.x64","Microsoft.VisualCpp.CodeAnalysis.Extensions","Microsoft.VisualCpp.CodeAnalysis.Extensions.X86","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X86","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X86.Resources","Microsoft.VisualCpp.CodeAnalysis.Extensions.X64","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X64","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X64.Resources","Microsoft.VisualCpp.Tools.HostX64.TargetX86","Microsoft.VisualCpp.VCTip.HostX64.TargetX86","Microsoft.VisualCpp.Tools.HostX64.TargetX86.Resources","Microsoft.VisualCpp.Tools.HostX64.TargetX64","Microsoft.VisualCpp.VCTip.HostX64.TargetX64","Microsoft.VisualCpp.Tools.HostX64.TargetX64.Resources","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX64","Microsoft.VisualCpp.Premium.Tools.Hostx86.Targetx64.Resources","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86.Resources","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86.Resources","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64.Resources","Microsoft.VisualCpp.PGO.X86","Microsoft.VisualCpp.PGO.X64","Microsoft.VisualCpp.PGO.Headers","Microsoft.VisualCpp.CRT.x86.Store","Microsoft.VisualCpp.CRT.x86.OneCore.Desktop","Microsoft.VisualCpp.CRT.x64.Store","Microsoft.VisualCpp.CRT.x64.OneCore.Desktop","Microsoft.VisualCpp.CRT.Redist.x86.OneCore.Desktop","Microsoft.VisualCpp.CRT.Redist.x64.OneCore.Desktop","Microsoft.VisualStudio.PackageGroup.VC.Tools.x86","Microsoft.VisualCpp.Tools.HostX86.TargetX64","Microsoft.VisualCpp.VCTip.hostX86.targetX64","Microsoft.VisualCpp.Tools.Hostx86.Targetx64.Resources","Microsoft.VisualCpp.Tools.HostX86.TargetX86","Microsoft.VisualCpp.VCTip.hostX86.targetX86","Microsoft.VisualCpp.Tools.HostX86.TargetX86.Resources","Microsoft.VisualCpp.Tools.Core.Resources","Microsoft.VisualCpp.Tools.Core.x86","Microsoft.VisualCpp.DIA.SDK","Microsoft.VisualCpp.CRT.x86.Desktop","Microsoft.VisualCpp.CRT.x64.Desktop","Microsoft.VisualCpp.CRT.Source","Microsoft.VisualCpp.CRT.Redist.X86","Microsoft.VisualCpp.CRT.Redist.X64","Microsoft.VisualCpp.CRT.Redist.Resources","Microsoft.VisualCpp.RuntimeDebug.14","Microsoft.VisualCpp.RuntimeDebug.14","Microsoft.VisualCpp.Redist.14.Latest","Microsoft.VisualCpp.Redist.14.Latest","Microsoft.VisualCpp.CRT.Headers","Microsoft.VisualStudio.Graphics.Viewers","Microsoft.VisualStudio.Graphics.Viewers.Resources","Microsoft.VisualStudio.Graphics.Msi","Microsoft.VisualStudio.Graphics.Msi","Microsoft.VisualStudio.Graphics.Analyzer","Microsoft.VisualStudio.Graphics.Analyzer.Targeted","Microsoft.VisualStudio.Graphics.Analyzer.Resources","Microsoft.VisualStudio.Graphics.Appid","Microsoft.VisualStudio.Graphics.Appid.Resources","Microsoft.VisualStudio.Component.VC.CoreIde","Microsoft.VisualStudio.VC.Ide.Pro","Microsoft.VisualStudio.VC.Ide.Pro.Resources","Microsoft.VisualStudio.VC.Templates.Pro","Microsoft.VisualStudio.VC.Templates.Pro.Resources","Microsoft.VisualStudio.VC.Items.Pro","Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Reduced","Microsoft.VisualStudio.VC.Ide.x64","Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Express","Microsoft.VisualStudio.VC.MSBuild.X64.v142","Microsoft.VisualStudio.VC.MSBuild.X64","Microsoft.VS.VC.MSBuild.X64.Resources","Microsoft.VisualStudio.VC.MSBuild.ARM.v142","Microsoft.VisualStudio.VC.MSBuild.ARM","Microsoft.VisualStudio.VC.MSBuild.x86.v142","Microsoft.VisualStudio.VC.MSBuild.X86","Microsoft.VisualStudio.VC.MSBuild.Base","Microsoft.VisualStudio.VC.MSBuild.Base.Resources","Microsoft.VisualStudio.VC.Ide.WinXPlus","Microsoft.VisualStudio.VC.Ide.Dskx","Microsoft.VisualStudio.VC.Ide.Dskx.Resources","Microsoft.VisualStudio.VC.Ide.Base","Microsoft.VisualStudio.VC.Ide.LanguageService","Microsoft.VisualStudio.VC.Ide.Core","Microsoft.VisualStudio.VC.Ide.Core.Resources","Microsoft.VisualStudio.VC.Ide.VCPkgDatabase","Microsoft.VisualStudio.VC.Ide.ProjectSystem","Microsoft.VisualStudio.VC.Ide.ProjectSystem.Resources","Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine","Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine.Resources","Microsoft.VisualStudio.VC.Ide.LanguageService.Resources","Microsoft.VisualStudio.VC.Ide.Base.Resources","Component.Microsoft.VisualStudio.LiveShare","Microsoft.VisualStudio.LiveShare","Microsoft.Icecap.Analysis","Microsoft.Icecap.Analysis.Targeted","Microsoft.Icecap.Analysis.Resources","Microsoft.Icecap.Analysis.Resources.Targeted","Microsoft.Icecap.Collection.Msi","Microsoft.Icecap.Collection.Msi.Targeted","Microsoft.Icecap.Collection.Msi.Resources","Microsoft.Icecap.Collection.Msi.Resources.Targeted","Microsoft.DiagnosticsHub.Instrumentation","Microsoft.DiagnosticsHub.CpuSampling.ExternalDependencies","Microsoft.DiagnosticsHub.CpuSampling","Microsoft.DiagnosticsHub.CpuSampling.Targeted","Microsoft.PackageGroup.DiagnosticsHub.Platform","Microsoft.DiagnosticsHub.Runtime.ExternalDependencies","Microsoft.DiagnosticsHub.Runtime.ExternalDependencies.Targeted","Microsoft.DiagnosticsHub.Collection.ExternalDependencies.x64","Microsoft.DiagnosticsHub.Collection.StopService.Uninstall","Microsoft.DiagnosticsHub.Runtime","Microsoft.DiagnosticsHub.Runtime.Targeted","Microsoft.DiagnosticsHub.Collection","Microsoft.DiagnosticsHub.Collection.Service","Microsoft.DiagnosticsHub.Collection.StopService.Install","Microsoft.VisualStudio.Component.IntelliCode","Microsoft.VisualStudio.IntelliCode","Microsoft.Net.4.TargetingPack","Microsoft.VisualStudio.VC.Ide.ResourceEditor","Microsoft.VisualStudio.VC.Ide.ResourceEditor.Resources","Microsoft.VisualStudio.PackageGroup.TestTools.CodeCoverage","Microsoft.VisualStudio.PackageGroup.TestTools.Core","Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.V2.CLI","Microsoft.VisualStudio.TestTools.TestPlatform.V2.CLI","Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.V1.CLI","Microsoft.VisualStudio.TestTools.TestPlatform.V1.CLI","Microsoft.VisualStudio.TestTools.Pex.Common","Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.Legacy","Microsoft.VisualStudio.PackageGroup.MinShell.Interop","Microsoft.VisualStudio.TestTools.TP.Legacy.Tips.Msi","Microsoft.VisualStudio.TestTools.TP.Legacy.Tips.Common","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Tips","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Tips.Resources","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.TestTools","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Professional","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Common","Microsoft.VisualStudio.TestTools.TP.Legacy.Common.Res","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Core","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Core.Resources","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Agent","Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.IDE","Microsoft.VisualStudio.TestTools.TestWIExtension","Microsoft.VisualStudio.TestTools.TestPlatform.IDE","Microsoft.VisualStudio.PackageGroup.TestTools.DataCollectors","Microsoft.VisualStudio.LiveShareApi","Microsoft.VisualStudio.Component.TextTemplating","Microsoft.VisualStudio.TextTemplating.MSBuild","Microsoft.VisualStudio.TextTemplating.Integration","Microsoft.VisualStudio.TextTemplating.Core","Microsoft.VisualStudio.TextTemplating.Integration.Resources","Microsoft.VisualCpp.CRT.ClickOnce.Msi","Microsoft.Component.MSBuild","Microsoft.NuGet.Build.Tasks","Microsoft.DiagnosticsHub.KB2882822.Win7","Microsoft.VisualStudio.PackageGroup.Debugger.Script","Microsoft.VisualStudio.Debugger.Script.Msi","Microsoft.VisualStudio.Debugger.Script","Microsoft.VisualStudio.Debugger.Script","Microsoft.VisualStudio.Debugger.Script.Resources","Microsoft.VisualStudio.Debugger.Script.Resources","Microsoft.VisualStudio.ComponentGroup.WebToolsExtensions","Microsoft.VisualStudio.ProTools","sqlsysclrtypes","sqlsysclrtypes","SQLCommon","Microsoft.VisualStudio.ProTools.Resources","Microsoft.VisualStudio.WebToolsExtensions","Microsoft.VisualStudio.WebTools","Microsoft.VisualStudio.WebTools.Resources","Microsoft.VisualStudio.WebTools.MSBuild","Microsoft.VisualStudio.WebTools.WSP.FSA","Microsoft.VisualStudio.WebTools.WSP.FSA.Resources","Microsoft.VisualStudio.VC.Ide.MDD","Microsoft.VisualStudio.VisualC.Logging","Microsoft.WebTools.Shared","Microsoft.WebTools.DotNet.Core.ItemTemplates","Microsoft.VisualCpp.Redist.14","Microsoft.VisualCpp.Redist.14","Microsoft.Windows.UniversalCRT.Msu.7","Microsoft.VisualStudio.StaticAnalysis","Microsoft.VisualStudio.StaticAnalysis.Resources","Microsoft.VisualStudio.Component.Roslyn.Compiler","Microsoft.CodeAnalysis.Compilers","Microsoft.VisualStudio.Component.NuGet","Microsoft.CredentialProvider","Microsoft.VisualStudio.NuGet.PowershellBindingRedirect","Microsoft.VisualStudio.NuGet.Licenses","Microsoft.VisualStudio.PackageGroup.Community","Microsoft.VisualStudio.Community.Extra.Resources","Microsoft.VisualStudio.Community.Extra","Microsoft.VisualStudio.PackageGroup.Core","Microsoft.VisualStudio.CodeSense.Community","Microsoft.VisualStudio.TestTools.TeamFoundationClient","Microsoft.VisualStudio.PackageGroup.Debugger.Core","Microsoft.VisualStudio.PackageGroup.Debugger.TimeTravel.Record","Microsoft.VisualStudio.Debugger.TimeTravel.Runtime","Microsoft.VisualStudio.Debugger.TimeTravel.Runtime","Microsoft.VisualStudio.Debugger.TimeTravel.Agent","Microsoft.VisualStudio.Debugger.TimeTravel.Record","Microsoft.VisualStudio.Debugger.VSCodeDebuggerHost","Microsoft.VisualStudio.VC.Ide.Debugger","Microsoft.VisualStudio.VC.Ide.Debugger.Concord","Microsoft.VisualStudio.VC.Ide.Debugger.Concord.Resources","Microsoft.VisualStudio.VC.Ide.Debugger.Resources","Microsoft.VisualStudio.VC.Ide.Common","Microsoft.VisualStudio.VC.Ide.Common.Resources","Microsoft.VisualStudio.Debugger.Parallel","Microsoft.VisualStudio.Debugger.Parallel.Resources","Microsoft.VisualStudio.Debugger.CollectionAgents","Microsoft.VisualStudio.Debugger.Managed","Microsoft.CodeAnalysis.VisualStudio.Setup","Microsoft.CodeAnalysis.ExpressionEvaluator","Microsoft.VisualStudio.Debugger.Concord.Managed","Microsoft.VisualStudio.Debugger.Concord.Managed.Resources","Microsoft.VisualStudio.Debugger.Managed.Resources","Microsoft.VisualStudio.Debugger.Remote","Microsoft.VisualStudio.Debugger.Remote.DbgHelp.Win8","Microsoft.VisualStudio.Debugger.Concord.Remote","Microsoft.VisualStudio.Debugger.Concord.Remote.Resources","Microsoft.VisualStudio.Debugger.Remote","Microsoft.VisualStudio.Debugger.Remote.DbgHelp.Win8","Microsoft.VisualStudio.Debugger.Concord.Remote","Microsoft.VisualStudio.Debugger.Concord.Remote.Resources","Microsoft.VisualStudio.Debugger.Remote.Resources","Microsoft.VisualStudio.Debugger.Remote.Resources","Microsoft.VisualStudio.Debugger","Microsoft.VisualStudio.Debugger.Package.DiagHub.Client.VSx86","Microsoft.VisualStudio.Debugger.Remote.DiagHub.Client","Microsoft.VisualStudio.Debugger.Remote.DiagHub.Client","Microsoft.VisualStudio.Debugger.DbgHelp.Win8","Microsoft.VisualStudio.VC.MSVCDis","Microsoft.VisualStudio.ScriptedHost","Microsoft.VisualStudio.ScriptedHost.Targeted","Microsoft.VisualStudio.ScriptedHost.Resources","Microsoft.IntelliTrace.DiagnosticsHub","Microsoft.VisualStudio.Debugger.Concord","Microsoft.VisualStudio.Debugger.Concord.Resources","Microsoft.VisualStudio.Debugger.Resources","Microsoft.PackageGroup.ClientDiagnostics","Microsoft.VisualStudio.AppResponsiveness","Microsoft.VisualStudio.AppResponsiveness.Targeted","Microsoft.VisualStudio.AppResponsiveness.Resources","Microsoft.VisualStudio.ClientDiagnostics","Microsoft.VisualStudio.ClientDiagnostics.Targeted","Microsoft.VisualStudio.ClientDiagnostics.Resources","Microsoft.VisualStudio.PackageGroup.CommunityCore","Microsoft.VisualStudio.ProjectSystem.Full","Microsoft.VisualStudio.ProjectSystem","Microsoft.VisualStudio.Community.x86","Microsoft.VisualStudio.Community.x64","Microsoft.VisualStudio.Community","Microsoft.IntelliTrace.CollectorCab","Microsoft.VisualStudio.Community.Resources","Microsoft.VisualStudio.Net.Eula.Resources","Microsoft.VisualStudio.WebSiteProject.DTE","Microsoft.MSHtml","Microsoft.VisualStudio.Platform.CallHierarchy","Microsoft.VisualStudio.Community.Msi.Resources","Microsoft.VisualStudio.Community.Msi","Microsoft.VisualStudio.Devenv.Msi","Microsoft.VisualStudio.MinShell.Interop.Msi","Microsoft.VisualStudio.Editors","Microsoft.VisualStudio.Product.Community","Microsoft.VisualStudio.Workload.CoreEditor","Microsoft.VisualStudio.Component.CoreEditor","Microsoft.VisualStudio.PackageGroup.CoreEditor","Microsoft.VisualCpp.Tools.Common.UtilsPrereq","Microsoft.VisualCpp.Tools.Common.Utils","Microsoft.VisualCpp.Tools.Common.Utils.Resources","Microsoft.VisualStudio.PackageGroup.VsDevCmd","Microsoft.VisualStudio.VsDevCmd.Ext.NetFxSdk","Microsoft.VisualStudio.VsDevCmd.Core.WinSdk","Microsoft.VisualStudio.VsDevCmd.Core.DotNet","Microsoft.VisualStudio.VC.DevCmd","Microsoft.VisualStudio.VC.DevCmd.Resources","Microsoft.VisualStudio.VirtualTree","Microsoft.VisualStudio.PackageGroup.Progression","Microsoft.VisualStudio.PerformanceProvider","Microsoft.VisualStudio.GraphModel","Microsoft.VisualStudio.GraphProvider","Microsoft.DiaSymReader","Microsoft.Build.Dependencies","Microsoft.Build.FileTracker.Msi","Microsoft.Build","Microsoft.VisualStudio.PackageGroup.NuGet","Microsoft.VisualStudio.NuGet.Core","Microsoft.VisualStudio.TextMateGrammars","Microsoft.VisualStudio.PackageGroup.TeamExplorer.Common","Microsoft.VisualStudio.TeamExplorer","Microsoft.ServiceHub","Microsoft.VisualStudio.ProjectServices","Microsoft.VisualStudio.OpenFolder.VSIX","Microsoft.VisualStudio.FileHandler.Msi","Microsoft.VisualStudio.FileHandler.Msi","Microsoft.VisualStudio.PackageGroup.MinShell","Microsoft.VisualStudio.MinShell.Interop","Microsoft.VisualStudio.Log","Microsoft.VisualStudio.Log.Targeted","Microsoft.VisualStudio.Log.Resources","Microsoft.VisualStudio.Finalizer","Microsoft.VisualStudio.Devenv","Microsoft.VisualStudio.Devenv.Resources","Microsoft.VisualStudio.CoreEditor","Microsoft.VisualStudio.Platform.NavigateTo","Microsoft.VisualStudio.Connected","Microsoft.VisualStudio.PerfLib","Microsoft.VisualStudio.Connected.Resources","Microsoft.VisualStudio.MinShell","Microsoft.VisualStudio.Setup.Configuration","Microsoft.VisualStudio.MinShell.Platform","Microsoft.VisualStudio.MinShell.Platform.Resources","Microsoft.VisualStudio.MefHosting","Microsoft.VisualStudio.MefHosting.Resources","Microsoft.VisualStudio.Initializer","Microsoft.VisualStudio.ExtensionManager","Microsoft.VisualStudio.Platform.Editor","Microsoft.VisualStudio.MinShell.x86","Microsoft.VisualStudio.NativeImageSupport","Microsoft.VisualStudio.MinShell.Msi","Microsoft.VisualStudio.MinShell.Msi.Resources","Microsoft.VisualStudio.LanguageServer","Microsoft.VisualStudio.Devenv.Config","Microsoft.VisualStudio.MinShell.Resources","Microsoft.Net.PackageGroup.4.7.2.Redist","Microsoft.Net.4.7.2.FullRedist","Microsoft.VisualStudio.Branding.Community"]}] diff --git a/test/fixtures/VS_2019_Preview.txt b/test/fixtures/VS_2019_Preview.txt new file mode 100644 index 0000000000..806509e7ce --- /dev/null +++ b/test/fixtures/VS_2019_Preview.txt @@ -0,0 +1 @@ +[{"path":"C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Preview","version":"16.0.28608.199","packages":["Microsoft.VisualStudio.Product.Enterprise","Microsoft.VisualStudio.Workload.NativeDesktop","Microsoft.VisualStudio.Component.VC.TestAdapterForGoogleTest","Microsoft.VisualStudio.VC.Ide.TestAdapterForGoogleTest","Microsoft.VisualStudio.Component.VC.TestAdapterForBoostTest","Microsoft.VisualStudio.VC.Ide.TestAdapterForBoostTest","Microsoft.VisualStudio.Component.VC.ATL","Microsoft.VisualStudio.VC.Ide.ATL","Microsoft.VisualStudio.VC.Ide.ATL.Resources","Microsoft.VisualCpp.ATL.X86","Microsoft.VisualCpp.ATL.X64","Microsoft.VisualCpp.ATL.Source","Microsoft.VisualCpp.ATL.Headers","Microsoft.VisualStudio.Component.VC.CMake.Project","Microsoft.VisualStudio.VC.CMake","Microsoft.VisualStudio.VC.CMake.Project","Microsoft.VisualStudio.VC.ExternalBuildFramework","Microsoft.VisualStudio.Component.VC.DiagnosticTools","Microsoft.VisualStudio.Component.Graphics.Tools","Microsoft.VisualStudio.Graphics.Viewers","Microsoft.VisualStudio.Graphics.Viewers.Resources","Microsoft.VisualStudio.Graphics.EnableTools","Microsoft.VisualStudio.Graphics.Msi","Microsoft.VisualStudio.Graphics.Msi","Microsoft.VisualStudio.Graphics.Analyzer","Microsoft.VisualStudio.Graphics.Analyzer.Targeted","Microsoft.VisualStudio.Graphics.Analyzer.Resources","Microsoft.VisualStudio.Graphics.Appid","Microsoft.VisualStudio.Graphics.Appid.Resources","Microsoft.VisualStudio.Component.Windows10SDK.17763","Win10SDK_10.0.17763","Microsoft.VisualStudio.Component.VC.Tools.x86.x64","Microsoft.VisualCpp.CodeAnalysis.Extensions","Microsoft.VisualCpp.CodeAnalysis.Extensions.X86","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X86","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X86.Resources","Microsoft.VisualCpp.CodeAnalysis.Extensions.X64","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X64","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X64.Resources","Microsoft.VisualCpp.Tools.HostX64.TargetX86","Microsoft.VisualCpp.VCTip.HostX64.TargetX86","Microsoft.VisualCpp.Tools.HostX64.TargetX86.Resources","Microsoft.VisualCpp.Tools.HostX64.TargetX64","Microsoft.VisualCpp.VCTip.HostX64.TargetX64","Microsoft.VisualCpp.Tools.HostX64.TargetX64.Resources","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX64","Microsoft.VisualCpp.Premium.Tools.Hostx86.Targetx64.Resources","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86.Resources","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86.Resources","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64.Resources","Microsoft.VisualCpp.PGO.X86","Microsoft.VisualCpp.PGO.X64","Microsoft.VisualCpp.PGO.Headers","Microsoft.VisualCpp.CRT.x86.Store","Microsoft.VisualCpp.CRT.x86.OneCore.Desktop","Microsoft.VisualCpp.CRT.x64.Store","Microsoft.VisualCpp.CRT.x64.OneCore.Desktop","Microsoft.VisualCpp.CRT.Redist.x86.OneCore.Desktop","Microsoft.VisualCpp.CRT.Redist.x64.OneCore.Desktop","Microsoft.VisualStudio.PackageGroup.VC.Tools.x86","Microsoft.VisualCpp.Tools.HostX86.TargetX64","Microsoft.VisualCpp.VCTip.hostX86.targetX64","Microsoft.VisualCpp.Tools.Hostx86.Targetx64.Resources","Microsoft.VisualCpp.Tools.HostX86.TargetX86","Microsoft.VisualCpp.VCTip.hostX86.targetX86","Microsoft.VisualCpp.Tools.HostX86.TargetX86.Resources","Microsoft.VisualCpp.Tools.Core.Resources","Microsoft.VisualCpp.Tools.Core.x86","Microsoft.VisualCpp.DIA.SDK","Microsoft.VisualCpp.CRT.x86.Desktop","Microsoft.VisualCpp.CRT.x64.Desktop","Microsoft.VisualCpp.CRT.Source","Microsoft.VisualCpp.CRT.Redist.X86","Microsoft.VisualCpp.CRT.Redist.X64","Microsoft.VisualCpp.CRT.Redist.Resources","Microsoft.VisualCpp.RuntimeDebug.14","Microsoft.VisualCpp.RuntimeDebug.14","Microsoft.VisualCpp.CRT.Headers","Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core","Microsoft.VisualStudio.PackageGroup.TestTools.Native","Microsoft.VisualStudio.ComponentGroup.ArchitectureTools.Native","Microsoft.VisualStudio.Component.ClassDesigner","Microsoft.VisualStudio.ClassDesigner","Microsoft.VisualStudio.ClassDesigner.Resources","Microsoft.VisualStudio.Component.VC.Redist.14.Latest","Microsoft.VisualCpp.Redist.14.Latest","Microsoft.VisualCpp.Redist.14.Latest","Microsoft.VisualStudio.VC.Templates.UnitTest","Microsoft.VisualStudio.VC.UnitTest.Desktop.Build.Core","Microsoft.VisualStudio.TestTools.TestPlatform.V1.CPP","Microsoft.VisualStudio.VC.Templates.UnitTest.Resources","Microsoft.VisualStudio.VC.Templates.Desktop","Microsoft.VisualStudio.Component.VC.CoreIde","Microsoft.VisualStudio.VC.Ide.Pro","Microsoft.VisualStudio.VC.Ide.Pro.Resources","Microsoft.VisualStudio.VC.Templates.Pro","Microsoft.VisualStudio.VC.Templates.Pro.Resources","Microsoft.VisualStudio.VC.Items.Pro","Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Reduced","Microsoft.VisualStudio.VC.Ide.x64","Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Express","Microsoft.VisualStudio.VC.MSBuild.X64.v142","Microsoft.VisualStudio.VC.MSBuild.X64","Microsoft.VS.VC.MSBuild.X64.Resources","Microsoft.VisualStudio.VC.MSBuild.ARM.v142","Microsoft.VisualStudio.VC.MSBuild.ARM","Microsoft.VisualStudio.VC.MSBuild.x86.v142","Microsoft.VisualStudio.VC.MSBuild.X86","Microsoft.VisualStudio.VC.MSBuild.Base","Microsoft.VisualStudio.VC.MSBuild.Base.Resources","Microsoft.VisualStudio.VC.Ide.WinXPlus","Microsoft.VisualStudio.VC.Ide.Dskx","Microsoft.VisualStudio.VC.Ide.Dskx.Resources","Microsoft.VisualStudio.VC.Ide.Base","Microsoft.VisualStudio.VC.Ide.LanguageService","Microsoft.VisualStudio.VC.Ide.Core","Microsoft.VisualStudio.VC.Ide.Core.Resources","Microsoft.VisualStudio.VC.Ide.VCPkgDatabase","Microsoft.VisualStudio.VC.Ide.Progression.Enterprise","Microsoft.VisualStudio.VC.Ide.ProjectSystem","Microsoft.VisualStudio.VC.Ide.ProjectSystem.Resources","Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine","Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine.Resources","Microsoft.VisualStudio.VC.Ide.LanguageService.Resources","Microsoft.VisualStudio.VC.Ide.Base.Resources","Microsoft.VisualStudio.Component.CodeMap","Microsoft.VisualStudio.Component.GraphDocument","Microsoft.VisualStudio.Vmp","Microsoft.VisualStudio.GraphDocument","Microsoft.VisualStudio.GraphDocument.Resources","Microsoft.VisualStudio.CodeMap","Microsoft.VisualStudio.Component.SQL.LocalDB.Runtime","Microsoft.VisualStudio.Component.SQL.NCLI","sqllocaldb","sqlncli","Microsoft.VisualStudio.ComponentGroup.WebToolsExtensions","Microsoft.VisualStudio.WebToolsExtensions","Microsoft.VisualStudio.WebTools","Microsoft.VisualStudio.WebTools.Resources","Microsoft.VisualStudio.WebTools.MSBuild","Microsoft.VisualStudio.PackageGroup.Debugger.Script","Microsoft.VisualStudio.Debugger.Script.Msi","Microsoft.VisualStudio.Debugger.Script","Microsoft.VisualStudio.Debugger.Script","Microsoft.VisualStudio.Debugger.Script.Resources","Microsoft.VisualStudio.Debugger.Script.Resources","Microsoft.VisualStudio.VC.Ide.MDD","Microsoft.VisualStudio.Component.NuGet","Microsoft.CredentialProvider","Component.Microsoft.VisualStudio.LiveShare","Microsoft.VisualStudio.LiveShare","Microsoft.VisualStudio.Component.Debugger.JustInTime","Microsoft.VisualStudio.Debugger.ImmersiveActivateHelper.Msi","Microsoft.VisualStudio.Debugger.JustInTime","Microsoft.VisualStudio.Debugger.JustInTime.Msi","Microsoft.VisualStudio.Component.IntelliTrace.FrontEnd","Microsoft.IntelliTrace.DiagnosticsHubAgent.Targeted","Microsoft.IntelliTrace.Debugger","Microsoft.IntelliTrace.Debugger.Targeted","Microsoft.IntelliTrace.FrontEnd","Microsoft.DiagnosticsHub.Instrumentation","Microsoft.DiagnosticsHub.CpuSampling","Microsoft.DiagnosticsHub.CpuSampling.Targeted","Microsoft.PackageGroup.DiagnosticsHub.Platform","Microsoft.DiagnosticsHub.Collection.StopService.Uninstall","Microsoft.DiagnosticsHub.Runtime","Microsoft.DiagnosticsHub.Runtime.Targeted","Microsoft.DiagnosticsHub.Runtime.Resources","Microsoft.DiagnosticsHub.Collection","Microsoft.DiagnosticsHub.Collection.Service","Microsoft.DiagnosticsHub.Collection.StopService.Install","Microsoft.VisualStudio.Dsl.GraphObject","Microsoft.Net.4.TargetingPack","Microsoft.VisualStudio.VC.Ide.ResourceEditor","Microsoft.VisualStudio.VC.Ide.ResourceEditor.Resources","Microsoft.VisualStudio.NuGet.Licenses","Microsoft.WebTools.Shared","Microsoft.VisualStudio.WebToolsExtensions.DotNet.Core.ItemTemplates","Microsoft.VisualStudio.Component.TextTemplating","Microsoft.VisualStudio.TextTemplating.MSBuild","Microsoft.VisualStudio.TextTemplating.Integration","Microsoft.VisualStudio.TextTemplating.Core","Microsoft.VisualStudio.TextTemplating.Integration.Resources","Microsoft.VisualStudio.ProTools","sqlsysclrtypes","sqlsysclrtypes","SQLCommon","Microsoft.VisualStudio.ProTools.Resources","Microsoft.VisualStudio.NuGet.Core","Microsoft.VisualStudio.PackageGroup.TestTools.CodeCoverage","Microsoft.VisualStudio.PackageGroup.IntelliTrace.Core","Microsoft.IntelliTrace.Core","Microsoft.IntelliTrace.Core.Concord","Microsoft.IntelliTrace.Core.Targeted","Microsoft.IntelliTrace.ProfilerProxy.Msi.x64","Microsoft.IntelliTrace.ProfilerProxy.Msi","Microsoft.VisualStudio.TestTools.DynamicCodeCoverage","Microsoft.VisualStudio.TestTools.CodeCoverage.Msi","Microsoft.VisualStudio.TestTools.CodeCoverage","Microsoft.Icecap.Analysis","Microsoft.Icecap.Analysis.Targeted","Microsoft.Icecap.Analysis.Resources","Microsoft.Icecap.Analysis.Resources.Targeted","Microsoft.Icecap.Collection.Msi","Microsoft.Icecap.Collection.Msi.Targeted","Microsoft.Icecap.Collection.Msi.Resources","Microsoft.Icecap.Collection.Msi.Resources.Targeted","Microsoft.VisualStudio.PackageGroup.TestTools.Core","Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.V2.CLI","Microsoft.VisualStudio.TestTools.TestPlatform.V2.CLI","Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.V1.CLI","Microsoft.VisualStudio.TestTools.TestPlatform.V1.CLI","Microsoft.VisualStudio.TestTools.Pex.Common","Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.Legacy","Microsoft.VisualStudio.PackageGroup.MinShell.Interop","Microsoft.VisualStudio.TestTools.TP.Legacy.Tips.Msi","Microsoft.VisualStudio.TestTools.TP.Legacy.Tips.Common","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Tips","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Tips.Resources","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.TestTools","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Remote","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Professional","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Core.Premium","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Common","Microsoft.VisualStudio.TestTools.TP.Legacy.Common.Res","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Core","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Core.Resources","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Agent","Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.IDE","Microsoft.VisualStudio.TestTools.TestWIExtension","Microsoft.VisualStudio.TestTools.TestPlatform.IDE","Microsoft.VisualStudio.PackageGroup.TestTools.DataCollectors","Microsoft.VisualStudio.TestTools.NE.Msi.Targeted","Microsoft.VisualStudio.TestTools.NetworkEmulation","Microsoft.VisualStudio.TestTools.DataCollectors","Microsoft.VisualCpp.CRT.ClickOnce.Msi","Microsoft.VisualStudio.WebTools.WSP.FSA","Microsoft.VisualStudio.WebTools.WSP.FSA.Resources","Microsoft.VisualStudio.Component.Static.Analysis.Tools","Microsoft.VisualCpp.Redist.14","Microsoft.VisualCpp.Redist.14","Microsoft.VisualStudio.StaticAnalysis","Microsoft.VisualStudio.StaticAnalysis.Resources","Microsoft.VisualStudio.PackageGroup.Community","Microsoft.VisualStudio.Community.Extra.Resources","Microsoft.VisualStudio.Community.Extra","Microsoft.VisualStudio.PackageGroup.Core","Microsoft.VisualStudio.CodeSense","Microsoft.VisualStudio.CodeSense.Community","Microsoft.VisualStudio.TestTools.TeamFoundationClient","Microsoft.VisualStudio.PackageGroup.Debugger.Core","Microsoft.VisualStudio.PackageGroup.Debugger.TimeTravel.Record","Microsoft.VisualStudio.Debugger.TimeTravel.Runtime","Microsoft.VisualStudio.Debugger.TimeTravel.Runtime","Microsoft.VisualStudio.Debugger.TimeTravel.Agent","Microsoft.VisualStudio.Debugger.TimeTravel.Record","Microsoft.VisualStudio.Debugger.VSCodeDebuggerHost","Microsoft.VisualStudio.VC.Ide.Debugger","Microsoft.VisualStudio.VC.Ide.Debugger.Concord","Microsoft.VisualStudio.VC.Ide.Debugger.Concord.Resources","Microsoft.VisualStudio.VC.Ide.Debugger.Resources","Microsoft.VisualStudio.VC.Ide.Common","Microsoft.VisualStudio.VC.Ide.Common.Resources","Microsoft.VisualStudio.Debugger.Parallel","Microsoft.VisualStudio.Debugger.Parallel.Resources","Microsoft.VisualStudio.Debugger.CollectionAgents","Microsoft.VisualStudio.Debugger.Managed","Microsoft.VisualStudio.Debugger.Concord.Managed","Microsoft.VisualStudio.Debugger.Concord.Managed.Resources","Microsoft.VisualStudio.Debugger.Managed.Resources","Microsoft.VisualStudio.Debugger.Remote","Microsoft.VisualStudio.Debugger.Concord.Remote","Microsoft.VisualStudio.Debugger.Concord.Remote.Resources","Microsoft.VisualStudio.Debugger.Remote","Microsoft.VisualStudio.Debugger.Concord.Remote","Microsoft.VisualStudio.Debugger.Concord.Remote.Resources","Microsoft.VisualStudio.Debugger.Remote.Resources","Microsoft.VisualStudio.Debugger.Remote.Resources","Microsoft.VisualStudio.Debugger","Microsoft.VisualStudio.Debugger.Package.DiagHub.Client.VSx86","Microsoft.VisualStudio.Debugger.Remote.DiagHub.Client","Microsoft.VisualStudio.Debugger.Remote.DiagHub.Client","Microsoft.VisualStudio.VC.MSVCDis","Microsoft.VisualStudio.ScriptedHost","Microsoft.VisualStudio.ScriptedHost.Targeted","Microsoft.VisualStudio.ScriptedHost.Resources","Microsoft.IntelliTrace.DiagnosticsHub","Microsoft.VisualStudio.Debugger.Concord","Microsoft.VisualStudio.Debugger.Concord.Resources","Microsoft.VisualStudio.Debugger.Resources","Microsoft.PackageGroup.ClientDiagnostics","Microsoft.VisualStudio.AppResponsiveness","Microsoft.VisualStudio.AppResponsiveness.Targeted","Microsoft.VisualStudio.AppResponsiveness.Resources","Microsoft.VisualStudio.ClientDiagnostics","Microsoft.VisualStudio.ClientDiagnostics.Targeted","Microsoft.VisualStudio.ClientDiagnostics.Resources","Microsoft.VisualStudio.PackageGroup.ProfessionalCore","Microsoft.VisualStudio.Professional","Microsoft.VisualStudio.Professional.Msi","Microsoft.VisualStudio.PackageGroup.EnterpriseCore","Microsoft.VisualStudio.Enterprise.Msi","Microsoft.VisualStudio.Enterprise","Microsoft.ShDocVw","Microsoft.VisualStudio.PackageGroup.CommunityCore","Microsoft.VisualStudio.ProjectSystem.Full","Microsoft.VisualStudio.ProjectSystem","Microsoft.VisualStudio.Community.x86","Microsoft.VisualStudio.Community.x64","Microsoft.VisualStudio.Community","Microsoft.IntelliTrace.CollectorCab","Microsoft.VisualStudio.Community.Resources","Microsoft.VisualStudio.WebSiteProject.DTE","Microsoft.MSHtml","Microsoft.VisualStudio.Platform.CallHierarchy","Microsoft.VisualStudio.Community.Msi.Resources","Microsoft.VisualStudio.Community.Msi","Microsoft.VisualStudio.Devenv.Msi","Microsoft.VisualStudio.MinShell.Interop.Msi","Microsoft.VisualStudio.Editors","Microsoft.VisualStudio.Net.Eula.Resources","Microsoft.CodeAnalysis.ExpressionEvaluator","Microsoft.CodeAnalysis.VisualStudio.Setup","Microsoft.Component.MSBuild","Microsoft.NuGet.Build.Tasks","Microsoft.VisualStudio.Component.Roslyn.Compiler","Microsoft.CodeAnalysis.Compilers","Microsoft.VisualStudio.Workload.CoreEditor","Microsoft.VisualStudio.Component.CoreEditor","Microsoft.VisualStudio.PackageGroup.CoreEditor","Microsoft.VisualCpp.Tools.Common.UtilsPrereq","Microsoft.VisualCpp.Tools.Common.Utils","Microsoft.VisualCpp.Tools.Common.Utils.Resources","Microsoft.VisualStudio.PackageGroup.VsDevCmd","Microsoft.VisualStudio.VsDevCmd.Ext.NetFxSdk","Microsoft.VisualStudio.VsDevCmd.Core.WinSdk","Microsoft.VisualStudio.VsDevCmd.Core.DotNet","Microsoft.VisualStudio.VC.DevCmd","Microsoft.VisualStudio.VC.DevCmd.Resources","Microsoft.VisualStudio.VirtualTree","Microsoft.VisualStudio.PackageGroup.Progression","Microsoft.VisualStudio.PerformanceProvider","Microsoft.VisualStudio.GraphModel","Microsoft.VisualStudio.GraphProvider","Microsoft.DiaSymReader","Microsoft.Build.Dependencies","Microsoft.Build.FileTracker.Msi","Microsoft.Build","Microsoft.VisualStudio.TextMateGrammars","Microsoft.VisualStudio.PackageGroup.TeamExplorer.Common","Microsoft.VisualStudio.TeamExplorer","Microsoft.ServiceHub","Microsoft.VisualStudio.ProjectServices","Microsoft.VisualStudio.SLNX.VSIX","Microsoft.VisualStudio.FileHandler.Msi","Microsoft.VisualStudio.FileHandler.Msi","Microsoft.VisualStudio.PackageGroup.MinShell","Microsoft.VisualStudio.MinShell.Interop","Microsoft.VisualStudio.Log","Microsoft.VisualStudio.Log.Targeted","Microsoft.VisualStudio.Log.Resources","Microsoft.VisualStudio.Finalizer","Microsoft.VisualStudio.Devenv","Microsoft.VisualStudio.Devenv.Resources","Microsoft.VisualStudio.CoreEditor","Microsoft.VisualStudio.Platform.NavigateTo","Microsoft.VisualStudio.Connected","Microsoft.VisualStudio.Connected.Resources","Microsoft.VisualStudio.MinShell","Microsoft.VisualStudio.Setup.Configuration","Microsoft.VisualStudio.Platform.Search","Microsoft.VisualStudio.MinShell.Platform","Microsoft.VisualStudio.MinShell.Platform.Resources","Microsoft.VisualStudio.MefHosting","Microsoft.VisualStudio.MefHosting.Resources","Microsoft.VisualStudio.Initializer","Microsoft.VisualStudio.ExtensionManager","Microsoft.VisualStudio.Platform.Editor","Microsoft.VisualStudio.MinShell.x86","Microsoft.VisualStudio.NativeImageSupport","Microsoft.VisualStudio.MinShell.Msi","Microsoft.VisualStudio.MinShell.Msi.Resources","Microsoft.VisualStudio.LanguageServer","Microsoft.VisualStudio.Devenv.Config","Microsoft.VisualStudio.MinShell.Resources","Microsoft.Net.PackageGroup.4.7.2.Redist","Microsoft.VisualStudio.Branding.Enterprise"]}] diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js index 507db96792..60b59f60b9 100644 --- a/test/test-find-visualstudio.js +++ b/test/test-find-visualstudio.js @@ -294,13 +294,109 @@ test('VS2017 Community with C++ workload', function (t) { finder.findVisualStudio() }) +test('VS2019 Preview with C++ workload', function (t) { + t.plan(2) + + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info, { + msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' + + 'Preview\\MSBuild\\Current\\Bin\\MSBuild.exe', + path: + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Preview', + sdk: '10.0.17763.0', + toolset: 'v142', + version: '16.0.28608.199', + versionMajor: 16, + versionMinor: 0, + versionYear: 2019 + }) + }) + + poison(finder, 'regSearchKeys') + finder.findVisualStudio2017OrNewer = (cb) => { + const file = path.join(__dirname, 'fixtures', + 'VS_2019_Preview.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '', cb) + } + finder.findVisualStudio() +}) + +test('minimal VS2019 Build Tools', function (t) { + t.plan(2) + + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info, { + msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' + + 'BuildTools\\MSBuild\\Current\\Bin\\MSBuild.exe', + path: + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools', + sdk: '10.0.17134.0', + toolset: 'v142', + version: '16.1.28922.388', + versionMajor: 16, + versionMinor: 1, + versionYear: 2019 + }) + }) + + poison(finder, 'regSearchKeys') + finder.findVisualStudio2017OrNewer = (cb) => { + const file = path.join(__dirname, 'fixtures', + 'VS_2019_BuildTools_minimal.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '', cb) + } + finder.findVisualStudio() +}) + +test('VS2019 Community with C++ workload', function (t) { + t.plan(2) + + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info, { + msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' + + 'Community\\MSBuild\\Current\\Bin\\MSBuild.exe', + path: + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community', + sdk: '10.0.17763.0', + toolset: 'v142', + version: '16.1.28922.388', + versionMajor: 16, + versionMinor: 1, + versionYear: 2019 + }) + }) + + poison(finder, 'regSearchKeys') + finder.findVisualStudio2017OrNewer = (cb) => { + const file = path.join(__dirname, 'fixtures', + 'VS_2019_Community_workload.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '', cb) + } + finder.findVisualStudio() +}) + function allVsVersions (t, finder) { finder.findVisualStudio2017OrNewer = (cb) => { - const file1 = path.join(__dirname, 'fixtures', 'VS_2017_BuildTools_minimal.txt') - const data1 = JSON.parse(fs.readFileSync(file1)) - const file2 = path.join(__dirname, 'fixtures', 'VS_2017_Community_workload.txt') - const data2 = JSON.parse(fs.readFileSync(file2)) - const data = JSON.stringify(data1.concat(data2)) + const data0 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', + 'VS_2017_Unusable.txt'))) + const data1 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', + 'VS_2017_BuildTools_minimal.txt'))) + const data2 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', + 'VS_2017_Community_workload.txt'))) + const data3 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', + 'VS_2019_Preview.txt'))) + const data4 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', + 'VS_2019_BuildTools_minimal.txt'))) + const data5 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', + 'VS_2019_Community_workload.txt'))) + const data = JSON.stringify(data0.concat(data1, data2, data3, data4, + data5)) finder.parseData(null, data, '', cb) } finder.regSearchKeys = (keys, value, addOpts, cb) => { @@ -414,3 +510,42 @@ test('look for VS2017 by installation path', function (t) { allVsVersions(t, finder) finder.findVisualStudio() }) + +test('look for VS2019 by version number', function (t) { + t.plan(2) + + const finder = new TestVisualStudioFinder(semverV1, '2019', (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info.versionYear, 2019) + }) + + allVsVersions(t, finder) + finder.findVisualStudio() +}) + +test('look for VS2017 by installation path', function (t) { + t.plan(2) + + const finder = new TestVisualStudioFinder(semverV1, + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools', + (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info.path, + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools') + }) + + allVsVersions(t, finder) + finder.findVisualStudio() +}) + +test('latest version should be found by default', function (t) { + t.plan(2) + + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info.versionYear, 2019) + }) + + allVsVersions(t, finder) + finder.findVisualStudio() +}) From 0efb8fb34bacb0a040574241352b7103ee42dea8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Fri, 31 May 2019 20:46:15 +0100 Subject: [PATCH 034/551] win: support running in VS Command Prompt PR-URL: https://github.com/nodejs/node-gyp/pull/1762 Reviewed-By: Refael Ackermann --- lib/find-visualstudio.js | 32 +++++++++++---- test/fixtures/VS_2017_Unusable.txt | 2 +- test/test-find-visualstudio.js | 64 ++++++++++++++++++++++++++++++ 3 files changed, 90 insertions(+), 8 deletions(-) diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js index ea35ca9db7..bf47f4ce0a 100644 --- a/lib/find-visualstudio.js +++ b/lib/find-visualstudio.js @@ -54,6 +54,15 @@ VisualStudioFinder.prototype = { this.addLog('msvs_version not set from command line or npm config') } + if (process.env.VCINSTALLDIR) { + this.envVcInstallDir = + path.resolve(process.env.VCINSTALLDIR, '..') + this.addLog('running in VS Command Prompt, installation path is:\n' + + `"${this.envVcInstallDir}"\n- will only use this version`) + } else { + this.addLog('VCINSTALLDIR not set, not running in VS Command Prompt') + } + this.findVisualStudio2017OrNewer((info) => { if (info) { return this.succeed(info) @@ -80,9 +89,13 @@ VisualStudioFinder.prototype = { }, fail: function fail () { - // If msvs_version was specified but finding VS failed, print what would - // have been accepted - if (this.configMsvsVersion) { + if (this.configMsvsVersion && this.envVcInstallDir) { + this.errorLog.push( + 'msvs_version does not match this VS Command Prompt or the', + 'installation cannot be used.') + } else if (this.configMsvsVersion) { + // If msvs_version was specified but finding VS failed, print what would + // have been accepted this.errorLog.push('') if (this.validVersions) { this.errorLog.push('valid versions for msvs_version:') @@ -163,6 +176,7 @@ VisualStudioFinder.prototype = { vsInfo = vsInfo.map((info) => { this.log.silly(`processing installation: "${info.path}"`) + info.path = path.resolve(info.path) var ret = this.getVersionInfo(info) ret.path = info.path ret.msBuild = this.getMSBuild(info, ret.versionYear) @@ -380,17 +394,21 @@ VisualStudioFinder.prototype = { // - add it to validVersions to be displayed at the end if a specific // version was requested and not found; // - check if this is the version that was requested. + // - check if this matches the Visual Studio Command Prompt checkConfigVersion: function checkConfigVersion (versionYear, vsPath) { this.validVersions.push(versionYear) this.validVersions.push(vsPath) - if (this.configVersionYear && - this.configVersionYear !== versionYear) { - this.addLog('- not looking for this version') + if (this.configVersionYear && this.configVersionYear !== versionYear) { + this.addLog('- msvs_version does not match this version') return false } if (this.configPath && this.configPath !== vsPath) { - this.addLog('- not looking for this installation') + this.addLog('- msvs_version does not point to this installation') + return false + } + if (this.envVcInstallDir && this.envVcInstallDir !== vsPath) { + this.addLog('- does not match this Visual Studio Command Prompt') return false } diff --git a/test/fixtures/VS_2017_Unusable.txt b/test/fixtures/VS_2017_Unusable.txt index c037565327..fc0a257f44 100644 --- a/test/fixtures/VS_2017_Unusable.txt +++ b/test/fixtures/VS_2017_Unusable.txt @@ -1 +1 @@ -[{"path":"C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTools","version":"15.9.28307.665","packages":["Microsoft.VisualStudio.Product.BuildTools","Microsoft.VisualStudio.Component.Windows10SDK.17134","Win10SDK_10.0.17134","Microsoft.VisualStudio.Component.VC.Tools.x86.x64","Microsoft.VisualCpp.CodeAnalysis.Extensions","Microsoft.VisualCpp.CodeAnalysis.Extensions.X86","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X86","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X86.Resources","Microsoft.VisualCpp.CodeAnalysis.Extensions.X64","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X64","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X64.Resources","Microsoft.VisualStudio.Component.Static.Analysis.Tools","Microsoft.VisualStudio.StaticAnalysis","Microsoft.VisualStudio.StaticAnalysis.Resources","Microsoft.VisualCpp.Tools.HostX64.TargetX86","Microsoft.VisualCpp.VCTip.HostX64.TargetX86","Microsoft.VisualCpp.Tools.HostX64.TargetX86.Resources","Microsoft.VisualCpp.Tools.HostX64.TargetX64","Microsoft.VisualCpp.VCTip.HostX64.TargetX64","Microsoft.VisualCpp.Tools.HostX64.TargetX64.Resources","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX64","Microsoft.VisualCpp.Premium.Tools.Hostx86.Targetx64.Resources","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86.Resources","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86.Resources","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64.Resources","Microsoft.VisualCpp.PGO.X86","Microsoft.VisualCpp.PGO.X64","Microsoft.VisualCpp.PGO.Headers","Microsoft.VisualCpp.CRT.x86.Store","Microsoft.VisualCpp.CRT.x86.OneCore.Desktop","Microsoft.VisualCpp.CRT.x64.Store","Microsoft.VisualCpp.CRT.x64.OneCore.Desktop","Microsoft.VisualCpp.CRT.Redist.x86.OneCore.Desktop","Microsoft.VisualCpp.CRT.Redist.x64.OneCore.Desktop","Microsoft.VisualCpp.CRT.ClickOnce.Msi","Microsoft.VisualStudio.PackageGroup.VC.Tools.x86","Microsoft.VisualCpp.Tools.HostX86.TargetX64","Microsoft.VisualCpp.VCTip.hostX86.targetX64","Microsoft.VisualCpp.Tools.Hostx86.Targetx64.Resources","Microsoft.VisualCpp.Tools.HostX86.TargetX86","Microsoft.VisualCpp.VCTip.hostX86.targetX86","Microsoft.VisualCpp.Tools.HostX86.TargetX86.Resources","Microsoft.VisualCpp.Tools.Core.Resources","Microsoft.VisualCpp.Tools.Core.x86","Microsoft.VisualCpp.Tools.Common.Utils","Microsoft.VisualCpp.Tools.Common.Utils.Resources","Microsoft.VisualCpp.DIA.SDK","Microsoft.VisualCpp.CRT.x86.Desktop","Microsoft.VisualCpp.CRT.x64.Desktop","Microsoft.VisualCpp.CRT.Source","Microsoft.VisualCpp.CRT.Redist.X86","Microsoft.VisualCpp.CRT.Redist.X64","Microsoft.VisualCpp.CRT.Redist.Resources","Microsoft.VisualCpp.RuntimeDebug.14","Microsoft.VisualCpp.RuntimeDebug.14","Microsoft.VisualCpp.Redist.14","Microsoft.VisualCpp.Redist.14","Microsoft.VisualCpp.CRT.Headers","Microsoft.VisualStudio.Workload.MSBuildTools","Microsoft.VisualStudio.Component.CoreBuildTools","Microsoft.VisualStudio.Setup.Configuration","Microsoft.VisualStudio.PackageGroup.VsDevCmd","Microsoft.VisualStudio.VsDevCmd.Ext.NetFxSdk","Microsoft.VisualStudio.VsDevCmd.Core.WinSdk","Microsoft.VisualStudio.VsDevCmd.Core.DotNet","Microsoft.VisualStudio.VC.DevCmd","Microsoft.VisualStudio.VC.DevCmd.Resources","Microsoft.VisualStudio.BuildTools.Resources","Microsoft.VisualStudio.Net.Eula.Resources","Microsoft.Build.Dependencies","Microsoft.Build.FileTracker.Msi","Microsoft.Component.MSBuild","Microsoft.PythonTools.BuildCore.Vsix","Microsoft.NuGet.Build.Tasks","Microsoft.VisualStudio.Component.Roslyn.Compiler","Microsoft.CodeAnalysis.Compilers.Resources","Microsoft.CodeAnalysis.Compilers","Microsoft.Net.PackageGroup.4.6.1.Redist","Microsoft.Net.4.6.1.FullRedist.NonThreshold","Microsoft.Windows.UniversalCRT.Msu.81","Microsoft.VisualStudio.NativeImageSupport","Microsoft.Build"]}] +[{"path":"C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildToolsUnusable","version":"15.9.28307.665","packages":["Microsoft.VisualStudio.Product.BuildTools","Microsoft.VisualStudio.Component.Windows10SDK.17134","Win10SDK_10.0.17134","Microsoft.VisualStudio.Component.VC.Tools.x86.x64","Microsoft.VisualCpp.CodeAnalysis.Extensions","Microsoft.VisualCpp.CodeAnalysis.Extensions.X86","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X86","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X86.Resources","Microsoft.VisualCpp.CodeAnalysis.Extensions.X64","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X64","Microsoft.VisualCpp.CodeAnalysis.ConcurrencyCheck.X64.Resources","Microsoft.VisualStudio.Component.Static.Analysis.Tools","Microsoft.VisualStudio.StaticAnalysis","Microsoft.VisualStudio.StaticAnalysis.Resources","Microsoft.VisualCpp.Tools.HostX64.TargetX86","Microsoft.VisualCpp.VCTip.HostX64.TargetX86","Microsoft.VisualCpp.Tools.HostX64.TargetX86.Resources","Microsoft.VisualCpp.Tools.HostX64.TargetX64","Microsoft.VisualCpp.VCTip.HostX64.TargetX64","Microsoft.VisualCpp.Tools.HostX64.TargetX64.Resources","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX64","Microsoft.VisualCpp.Premium.Tools.Hostx86.Targetx64.Resources","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86","Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86.Resources","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86.Resources","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64","Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64.Resources","Microsoft.VisualCpp.PGO.X86","Microsoft.VisualCpp.PGO.X64","Microsoft.VisualCpp.PGO.Headers","Microsoft.VisualCpp.CRT.x86.Store","Microsoft.VisualCpp.CRT.x86.OneCore.Desktop","Microsoft.VisualCpp.CRT.x64.Store","Microsoft.VisualCpp.CRT.x64.OneCore.Desktop","Microsoft.VisualCpp.CRT.Redist.x86.OneCore.Desktop","Microsoft.VisualCpp.CRT.Redist.x64.OneCore.Desktop","Microsoft.VisualCpp.CRT.ClickOnce.Msi","Microsoft.VisualStudio.PackageGroup.VC.Tools.x86","Microsoft.VisualCpp.Tools.HostX86.TargetX64","Microsoft.VisualCpp.VCTip.hostX86.targetX64","Microsoft.VisualCpp.Tools.Hostx86.Targetx64.Resources","Microsoft.VisualCpp.Tools.HostX86.TargetX86","Microsoft.VisualCpp.VCTip.hostX86.targetX86","Microsoft.VisualCpp.Tools.HostX86.TargetX86.Resources","Microsoft.VisualCpp.Tools.Core.Resources","Microsoft.VisualCpp.Tools.Core.x86","Microsoft.VisualCpp.Tools.Common.Utils","Microsoft.VisualCpp.Tools.Common.Utils.Resources","Microsoft.VisualCpp.DIA.SDK","Microsoft.VisualCpp.CRT.x86.Desktop","Microsoft.VisualCpp.CRT.x64.Desktop","Microsoft.VisualCpp.CRT.Source","Microsoft.VisualCpp.CRT.Redist.X86","Microsoft.VisualCpp.CRT.Redist.X64","Microsoft.VisualCpp.CRT.Redist.Resources","Microsoft.VisualCpp.RuntimeDebug.14","Microsoft.VisualCpp.RuntimeDebug.14","Microsoft.VisualCpp.Redist.14","Microsoft.VisualCpp.Redist.14","Microsoft.VisualCpp.CRT.Headers","Microsoft.VisualStudio.Workload.MSBuildTools","Microsoft.VisualStudio.Component.CoreBuildTools","Microsoft.VisualStudio.Setup.Configuration","Microsoft.VisualStudio.PackageGroup.VsDevCmd","Microsoft.VisualStudio.VsDevCmd.Ext.NetFxSdk","Microsoft.VisualStudio.VsDevCmd.Core.WinSdk","Microsoft.VisualStudio.VsDevCmd.Core.DotNet","Microsoft.VisualStudio.VC.DevCmd","Microsoft.VisualStudio.VC.DevCmd.Resources","Microsoft.VisualStudio.BuildTools.Resources","Microsoft.VisualStudio.Net.Eula.Resources","Microsoft.Build.Dependencies","Microsoft.Build.FileTracker.Msi","Microsoft.Component.MSBuild","Microsoft.PythonTools.BuildCore.Vsix","Microsoft.NuGet.Build.Tasks","Microsoft.VisualStudio.Component.Roslyn.Compiler","Microsoft.CodeAnalysis.Compilers.Resources","Microsoft.CodeAnalysis.Compilers","Microsoft.Net.PackageGroup.4.6.1.Redist","Microsoft.Net.4.6.1.FullRedist.NonThreshold","Microsoft.Windows.UniversalCRT.Msu.81","Microsoft.VisualStudio.NativeImageSupport","Microsoft.Build"]}] diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js index 60b59f60b9..e2bcc9954f 100644 --- a/test/test-find-visualstudio.js +++ b/test/test-find-visualstudio.js @@ -8,6 +8,8 @@ const VisualStudioFinder = findVisualStudio.test.VisualStudioFinder const semverV1 = { major: 1, minor: 0, patch: 0 } +delete process.env.VCINSTALLDIR + function poison (object, property) { function fail () { console.error(Error(`Property ${property} should not have been accessed.`)) @@ -549,3 +551,65 @@ test('latest version should be found by default', function (t) { allVsVersions(t, finder) finder.findVisualStudio() }) + +test('run on a usable VS Command Prompt', function (t) { + t.plan(2) + + process.env.VCINSTALLDIR = 'C:\\VS2015\\VC' + // VSINSTALLDIR is not defined on Visual C++ Build Tools 2015 + delete process.env.VSINSTALLDIR + + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info.path, 'C:\\VS2015') + }) + + allVsVersions(t, finder) + finder.findVisualStudio() +}) + +test('run on a unusable VS Command Prompt', function (t) { + t.plan(2) + + process.env.VCINSTALLDIR = + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildToolsUnusable\\VC' + + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + t.ok(/find .* Visual Studio/i.test(err), 'expect error') + t.false(info, 'no data') + }) + + allVsVersions(t, finder) + finder.findVisualStudio() +}) + +test('run on a VS Command Prompt with matching msvs_version', function (t) { + t.plan(2) + + process.env.VCINSTALLDIR = 'C:\\VS2015\\VC' + + const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2015', + (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info.path, 'C:\\VS2015') + }) + + allVsVersions(t, finder) + finder.findVisualStudio() +}) + +test('run on a VS Command Prompt with mismatched msvs_version', function (t) { + t.plan(2) + + process.env.VCINSTALLDIR = + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC' + + const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2015', + (err, info) => { + t.ok(/find .* Visual Studio/i.test(err), 'expect error') + t.false(info, 'no data') + }) + + allVsVersions(t, finder) + finder.findVisualStudio() +}) From a20faedc9152ff5ad0a2e48318af6f77bedb44fb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Tue, 28 May 2019 23:02:32 +0100 Subject: [PATCH 035/551] gyp: enable MARMASM items only on new VS versions PR-URL: https://github.com/nodejs/node-gyp/pull/1762 Reviewed-By: Rod Vagg Reviewed-By: Refael Ackermann --- gyp/pylib/gyp/generator/msvs.py | 7 +++++-- lib/configure.js | 9 +++++++++ 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index 5f622ab506..aacbe60836 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -88,6 +88,7 @@ def _import_OrderedDict(): 'msvs_enable_winrt', 'msvs_requires_importlibrary', 'msvs_enable_winphone', + 'msvs_enable_marmasm', 'msvs_application_type_revision', 'msvs_target_platform_version', 'msvs_target_platform_minversion', @@ -3362,7 +3363,8 @@ def _GenerateMSBuildProject(project, options, version, generator_flags): content += _GetMSBuildLocalProperties(project.msbuild_toolset) content += import_cpp_props_section content += import_masm_props_section - content += import_marmasm_props_section + if spec.get('msvs_enable_marmasm'): + content += import_marmasm_props_section content += _GetMSBuildExtensions(props_files_of_rules) content += _GetMSBuildPropertySheets(configurations) content += macro_section @@ -3375,7 +3377,8 @@ def _GenerateMSBuildProject(project, options, version, generator_flags): content += _GetMSBuildProjectReferences(project) content += import_cpp_targets_section content += import_masm_targets_section - content += import_marmasm_targets_section + if spec.get('msvs_enable_marmasm'): + content += import_marmasm_targets_section content += _GetMSBuildExtensionTargets(targets_files_of_rules) if spec.get('msvs_external_builder'): diff --git a/lib/configure.js b/lib/configure.js index 6e49c23d91..c3a07d472d 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -146,6 +146,15 @@ function configure (gyp, argv, callback) { if (vsInfo.sdk) { defaults['msvs_windows_target_platform_version'] = vsInfo.sdk } + if (variables.target_arch == 'arm64') { + if (vsInfo.versionMajor > 15 || + (vsInfo.versionMajor === 15 && vsInfo.versionMajor >= 9)) { + defaults['msvs_enable_marmasm'] = 1 + } else { + log.warn('Compiling ARM64 assembly is only available in\n' + + 'Visual Studio 2017 version 15.9 and above') + } + } variables['msbuild_path'] = vsInfo.msBuild } From 182e846b2a9af2540b37ddf2aac0bd873679d1dc Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Wed, 24 Apr 2019 10:38:37 +1000 Subject: [PATCH 036/551] v5.0.0: bump version and update changelog --- CHANGELOG.md | 39 +++++++++++++++++++++++++++++++++++++++ package.json | 2 +- 2 files changed, 40 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 05167c92dc..e9c1b9718f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,42 @@ +v5.0.0 2019-06-13 +================= + +* [[`8a83972743`](https://github.com/nodejs/node-gyp/commit/8a83972743)] - **(SEMVER-MAJOR)** **bin**: follow XDG OS conventions for storing data (Selwyn) [#1570](https://github.com/nodejs/node-gyp/pull/1570) +* [[`9e46872ea3`](https://github.com/nodejs/node-gyp/commit/9e46872ea3)] - **bin,lib**: remove extra comments/lines/spaces (Jon Moss) [#1508](https://github.com/nodejs/node-gyp/pull/1508) +* [[`8098ebdeb4`](https://github.com/nodejs/node-gyp/commit/8098ebdeb4)] - **deps**: replace `osenv` dependency with native `os` (Selwyn) +* [[`f83b457e03`](https://github.com/nodejs/node-gyp/commit/f83b457e03)] - **deps**: bump request to 2.8.7, fixes heok/hawk issues (Rohit Hazra) [#1492](https://github.com/nodejs/node-gyp/pull/1492) +* [[`323cee7323`](https://github.com/nodejs/node-gyp/commit/323cee7323)] - **deps**: pin `request` version range (Refael Ackermann) [#1300](https://github.com/nodejs/node-gyp/pull/1300) +* [[`c515912d08`](https://github.com/nodejs/node-gyp/commit/c515912d08)] - **doc**: improve issue template (Bartosz Sosnowski) [#1618](https://github.com/nodejs/node-gyp/pull/1618) +* [[`cca2d66727`](https://github.com/nodejs/node-gyp/commit/cca2d66727)] - **doc**: python info needs own header (Taylor D. Lee) [#1245](https://github.com/nodejs/node-gyp/pull/1245) +* [[`3e64c780f5`](https://github.com/nodejs/node-gyp/commit/3e64c780f5)] - **doc**: lint README.md (Jon Moss) [#1498](https://github.com/nodejs/node-gyp/pull/1498) +* [[`a20faedc91`](https://github.com/nodejs/node-gyp/commit/a20faedc91)] - **(SEMVER-MAJOR)** **gyp**: enable MARMASM items only on new VS versions (João Reis) [#1762](https://github.com/nodejs/node-gyp/pull/1762) +* [[`721eb691cf`](https://github.com/nodejs/node-gyp/commit/721eb691cf)] - **gyp**: teach MSVS generator about MARMASM Items (Jon Kunkee) [#1679](https://github.com/nodejs/node-gyp/pull/1679) +* [[`91744bfecc`](https://github.com/nodejs/node-gyp/commit/91744bfecc)] - **gyp**: add support for Windows on Arm (Richard Townsend) [#1739](https://github.com/nodejs/node-gyp/pull/1739) +* [[`a6e0a6c7ed`](https://github.com/nodejs/node-gyp/commit/a6e0a6c7ed)] - **gyp**: move compile\_commands\_json (Paul Maréchal) [#1661](https://github.com/nodejs/node-gyp/pull/1661) +* [[`92e8b52cee`](https://github.com/nodejs/node-gyp/commit/92e8b52cee)] - **gyp**: fix target --\> self.target (cclauss) +* [[`febdfa2137`](https://github.com/nodejs/node-gyp/commit/febdfa2137)] - **gyp**: fix sntex error (cclauss) [#1333](https://github.com/nodejs/node-gyp/pull/1333) +* [[`588d333c14`](https://github.com/nodejs/node-gyp/commit/588d333c14)] - **gyp**: \_winreg module was renamed to winreg in Python 3. (Craig Rodrigues) +* [[`98226d198c`](https://github.com/nodejs/node-gyp/commit/98226d198c)] - **gyp**: replace basestring with str, but only on Python 3. (Craig Rodrigues) +* [[`7535e4478e`](https://github.com/nodejs/node-gyp/commit/7535e4478e)] - **gyp**: replace deprecated functions (Craig Rodrigues) +* [[`2040cd21cc`](https://github.com/nodejs/node-gyp/commit/2040cd21cc)] - **gyp**: use print as a function, as specified in PEP 3105. (Craig Rodrigues) +* [[`abef93ded5`](https://github.com/nodejs/node-gyp/commit/abef93ded5)] - **gyp**: get ready for python 3 (cclauss) +* [[`43031fadcb`](https://github.com/nodejs/node-gyp/commit/43031fadcb)] - **python**: clean-up detection (João Reis) [#1582](https://github.com/nodejs/node-gyp/pull/1582) +* [[`49ab79d221`](https://github.com/nodejs/node-gyp/commit/49ab79d221)] - **python**: more informative error (Refael Ackermann) [#1269](https://github.com/nodejs/node-gyp/pull/1269) +* [[`997bc3c748`](https://github.com/nodejs/node-gyp/commit/997bc3c748)] - **readme**: add ARM64 info to MSVC setup instructions (Jon Kunkee) [#1655](https://github.com/nodejs/node-gyp/pull/1655) +* [[`788e767179`](https://github.com/nodejs/node-gyp/commit/788e767179)] - **test**: remove unused variable (João Reis) +* [[`6f5a408934`](https://github.com/nodejs/node-gyp/commit/6f5a408934)] - **tools**: fix usage of inherited -fPIC and -fPIE (Jens) [#1340](https://github.com/nodejs/node-gyp/pull/1340) +* [[`0efb8fb34b`](https://github.com/nodejs/node-gyp/commit/0efb8fb34b)] - **(SEMVER-MAJOR)** **win**: support running in VS Command Prompt (João Reis) [#1762](https://github.com/nodejs/node-gyp/pull/1762) +* [[`360ddbdf3a`](https://github.com/nodejs/node-gyp/commit/360ddbdf3a)] - **(SEMVER-MAJOR)** **win**: add support for Visual Studio 2019 (João Reis) [#1762](https://github.com/nodejs/node-gyp/pull/1762) +* [[`8f43f68275`](https://github.com/nodejs/node-gyp/commit/8f43f68275)] - **(SEMVER-MAJOR)** **win**: detect all VS versions in node-gyp (João Reis) [#1762](https://github.com/nodejs/node-gyp/pull/1762) +* [[`7fe4095974`](https://github.com/nodejs/node-gyp/commit/7fe4095974)] - **(SEMVER-MAJOR)** **win**: generic Visual Studio 2017 detection (João Reis) [#1762](https://github.com/nodejs/node-gyp/pull/1762) +* [[`7a71d68bce`](https://github.com/nodejs/node-gyp/commit/7a71d68bce)] - **win**: use msbuild from the configure stage (Bartosz Sosnowski) [#1654](https://github.com/nodejs/node-gyp/pull/1654) +* [[`d3b21220a0`](https://github.com/nodejs/node-gyp/commit/d3b21220a0)] - **win**: fix delay-load hook for electron 4 (Andy Dill) +* [[`81f3a92338`](https://github.com/nodejs/node-gyp/commit/81f3a92338)] - Update list of Node.js versions to test against. (Ben Noordhuis) [#1670](https://github.com/nodejs/node-gyp/pull/1670) +* [[`4748f6ab75`](https://github.com/nodejs/node-gyp/commit/4748f6ab75)] - Remove deprecated compatibility code. (Ben Noordhuis) [#1670](https://github.com/nodejs/node-gyp/pull/1670) +* [[`45e3221fd4`](https://github.com/nodejs/node-gyp/commit/45e3221fd4)] - Remove an outdated workaround for Python 2.4 (cclauss) [#1650](https://github.com/nodejs/node-gyp/pull/1650) +* [[`721dc7d314`](https://github.com/nodejs/node-gyp/commit/721dc7d314)] - Add ARM64 to MSBuild /Platform logic (Jon Kunkee) [#1655](https://github.com/nodejs/node-gyp/pull/1655) +* [[`a5b7410497`](https://github.com/nodejs/node-gyp/commit/a5b7410497)] - Add ESLint no-unused-vars rule (Jon Moss) [#1497](https://github.com/nodejs/node-gyp/pull/1497) + v4.0.0 2019-04-24 ================= diff --git a/package.json b/package.json index cce2e4d55f..e19bc0d98a 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "4.0.0", + "version": "5.0.0", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From e3861722edef5b47e4e0d3b84ebb179ece1a44fd Mon Sep 17 00:00:00 2001 From: David Sanders Date: Wed, 19 Jun 2019 18:20:00 -0700 Subject: [PATCH 037/551] doc: document --jobs max PR-URL: https://github.com/nodejs/node-gyp/pull/1770 Reviewed-By: Richard Lau Reviewed-By: Rod Vagg --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 01d428bc6c..b0900028d7 100644 --- a/README.md +++ b/README.md @@ -162,7 +162,7 @@ Some additional resources for addons and writing `gyp` files: | **Command** | **Description** |:----------------------------------|:------------------------------------------ -| `-j n`, `--jobs n` | Run `make` in parallel +| `-j n`, `--jobs n` | Run `make` in parallel. The value `max` will use all available CPU cores | `--target=v6.2.1` | Node.js version to build for (default is `process.version`) | `--silly`, `--loglevel=silly` | Log all progress to console | `--verbose`, `--loglevel=verbose` | Log most progress to console From 1cfdb2888623673979c7eef1d2407ac3cf374a70 Mon Sep 17 00:00:00 2001 From: Samuel Attard Date: Wed, 19 Jun 2019 12:02:03 -0700 Subject: [PATCH 038/551] lib: reintroduce support for iojs file naming for releases >= 1 && < 4 For Electron 3 PR-URL: https://github.com/nodejs/node-gyp/pull/1777 Reviewed-By: Richard Lau Reviewed-By: Rod Vagg --- lib/process-release.js | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/lib/process-release.js b/lib/process-release.js index 8b3fda81c6..f0c640b2d7 100644 --- a/lib/process-release.js +++ b/lib/process-release.js @@ -17,6 +17,7 @@ function processRelease (argv, gyp, defaultVersion, defaultRelease) { , versionSemver = semver.parse(version) , overrideDistUrl = gyp.opts['dist-url'] || gyp.opts.disturl , isDefaultVersion + , isNamedForLegacyIojs , name , distBaseUrl , baseUrl @@ -41,11 +42,17 @@ function processRelease (argv, gyp, defaultVersion, defaultRelease) { if (defaultRelease) { // v3 onward, has process.release - name = defaultRelease.name + name = defaultRelease.name.replace(/io\.js/, 'iojs') // remove the '.' for directory naming purposes } else { // old node or alternative --target= // semver.satisfies() doesn't like prerelease tags so test major directly - name = 'node' + isNamedForLegacyIojs = versionSemver.major >= 1 && versionSemver.major < 4 + // isNamedForLegacyIojs is required to support Electron < 4 (in particular Electron 3) + // as previously this logic was used to ensure "iojs" was used to download iojs releases + // and "node" for node releases. Unfortunately the logic was broad enough that electron@3 + // published release assets as "iojs" so that the node-gyp logic worked. Once Electron@3 has + // been EOL for a while (late 2019) we should remove this hack. + name = isNamedForLegacyIojs ? 'iojs' : 'node' } // check for the nvm.sh standard mirror env variables From a75723985eb75b02b882959b0edf6dbe274bd0eb Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Thu, 20 Jun 2019 11:40:54 +1000 Subject: [PATCH 039/551] v5.0.1: bump version and update changelog --- CHANGELOG.md | 6 ++++++ package.json | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e9c1b9718f..6a687ab49e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,9 @@ +v5.0.1 2019-06-20 +================= + +* [[`e3861722ed`](https://github.com/nodejs/node-gyp/commit/e3861722ed)] - **doc**: document --jobs max (David Sanders) [#1770](https://github.com/nodejs/node-gyp/pull/1770) +* [[`1cfdb28886`](https://github.com/nodejs/node-gyp/commit/1cfdb28886)] - **lib**: reintroduce support for iojs file naming for releases \>= 1 && \< 4 (Samuel Attard) [#1777](https://github.com/nodejs/node-gyp/pull/1777) + v5.0.0 2019-06-13 ================= diff --git a/package.json b/package.json index e19bc0d98a..393e7cac7e 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "5.0.0", + "version": "5.0.1", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From 2761afbf730f7001675bbdbfa4a9e4821243adcb Mon Sep 17 00:00:00 2001 From: Gabriel Schulhof Date: Mon, 11 Mar 2019 22:14:58 -0700 Subject: [PATCH 040/551] build,test: add duplicate symbol test On OSX symbols are exported by default, and they overlap if two different copies of the same symbol appear in a process. This change ensures that, on OSX, symbols are hidden by default. Re: https://github.com/nodejs/node-addon-api/pull/456 Fixes: https://github.com/nodejs/node/issues/26765 PR-URL: https://github.com/nodejs/node-gyp/pull/1689 Reviewed-By: Ben Noordhuis Reviewed-By: Refael Ackermann --- addon.gypi | 4 ++ .../node_modules/duplicate_symbols/binding.cc | 10 +++++ .../duplicate_symbols/binding.gyp | 19 ++++++++++ test/node_modules/duplicate_symbols/common.h | 37 +++++++++++++++++++ test/node_modules/duplicate_symbols/extra.cc | 6 +++ test/node_modules/duplicate_symbols/index.js | 5 +++ .../duplicate_symbols/package.json | 14 +++++++ test/test-addon.js | 25 +++++++++++++ 8 files changed, 120 insertions(+) create mode 100644 test/node_modules/duplicate_symbols/binding.cc create mode 100644 test/node_modules/duplicate_symbols/binding.gyp create mode 100644 test/node_modules/duplicate_symbols/common.h create mode 100644 test/node_modules/duplicate_symbols/extra.cc create mode 100644 test/node_modules/duplicate_symbols/index.js create mode 100644 test/node_modules/duplicate_symbols/package.json diff --git a/addon.gypi b/addon.gypi index e8b95c5501..e1c4b24af2 100644 --- a/addon.gypi +++ b/addon.gypi @@ -90,10 +90,14 @@ 'conditions': [ [ 'OS=="mac"', { + 'cflags': [ + '-fvisibility=hidden' + ], 'defines': [ '_DARWIN_USE_64_BIT_INODE=1' ], 'xcode_settings': { + 'GCC_SYMBOLS_PRIVATE_EXTERN': 'YES', # -fvisibility=hidden 'DYLIB_INSTALL_NAME_BASE': '@rpath' }, }], diff --git a/test/node_modules/duplicate_symbols/binding.cc b/test/node_modules/duplicate_symbols/binding.cc new file mode 100644 index 0000000000..bc5f6f5134 --- /dev/null +++ b/test/node_modules/duplicate_symbols/binding.cc @@ -0,0 +1,10 @@ +#include +#include "common.h" + +void Init(v8::Local exports) { + exports->Set(Nan::New("pointerCheck").ToLocalChecked(), + Nan::New(Something::PointerCheck) + ->GetFunction()); +} + +NODE_MODULE(NODE_GYP_MODULE_NAME, Init) diff --git a/test/node_modules/duplicate_symbols/binding.gyp b/test/node_modules/duplicate_symbols/binding.gyp new file mode 100644 index 0000000000..46f1c8cba4 --- /dev/null +++ b/test/node_modules/duplicate_symbols/binding.gyp @@ -0,0 +1,19 @@ +{ + "target_defaults": { + "include_dirs": [ + " + +class Something { + public: + static void PointerCheck(const Nan::FunctionCallbackInfo& info); +}; + +// Removing the inline keyword below will result in the addon failing to link +// on OSX because of a duplicate symbol. +inline void +Something::PointerCheck(const Nan::FunctionCallbackInfo& info) { + v8::Local v8result; + + if (info.Length() > 0) { + // If an argument was passed in, it is a pointer to the `PointerCheck` + // method from the other addon. So, we compare it to the value of the + // pointer to the `PointerCheck` method in this addon, and return + // `"equal"` if they are equal, and `"not equal"` otherwise". + + const char* result = + (reinterpret_cast(Something::PointerCheck) == + info[0].As()->Value()) ? + "equal" : "not equal"; + v8result = Nan::New(result).ToLocalChecked(); + } else { + // If no argument was passed in, we wrap the pointer to the `PointerCheck` + // method in this addon into a `v8::External` and pass it into JavaScript. + v8result = Nan::New( + reinterpret_cast(Something::PointerCheck)); + } + info.GetReturnValue().Set(v8result); +} + +#endif // DUPLICATE_SYMBOLS_COMMON_H_ diff --git a/test/node_modules/duplicate_symbols/extra.cc b/test/node_modules/duplicate_symbols/extra.cc new file mode 100644 index 0000000000..0c9d5dbebd --- /dev/null +++ b/test/node_modules/duplicate_symbols/extra.cc @@ -0,0 +1,6 @@ +#include "common.h" + +// It is important that common.h be included from two different translation +// units, because doing so can create duplicate symbols in some instances. If +// it does so and fails to build because of it, that is considered a test +// failure. diff --git a/test/node_modules/duplicate_symbols/index.js b/test/node_modules/duplicate_symbols/index.js new file mode 100644 index 0000000000..09dbed81b7 --- /dev/null +++ b/test/node_modules/duplicate_symbols/index.js @@ -0,0 +1,5 @@ +'use strict' +module.exports = { + pointerCheck1: require('bindings')('binding1').pointerCheck, + pointerCheck2: require('bindings')('binding2').pointerCheck +}; diff --git a/test/node_modules/duplicate_symbols/package.json b/test/node_modules/duplicate_symbols/package.json new file mode 100644 index 0000000000..ee21161cf1 --- /dev/null +++ b/test/node_modules/duplicate_symbols/package.json @@ -0,0 +1,14 @@ +{ + "name": "duplicate_symbols", + "version": "0.0.0", + "description": "Duplicate Symbols Test", + "main": "index.js", + "private": true, + "dependencies": { + "bindings": "~1.2.1", + "nan": "^2.0.0" + }, + "scripts": { + "test": "node index.js" + } +} diff --git a/test/test-addon.js b/test/test-addon.js index 900b0b049d..12687de2ac 100644 --- a/test/test-addon.js +++ b/test/test-addon.js @@ -19,6 +19,16 @@ function runHello(hostProcess) { return execFileSync(hostProcess, ['-e', testCode], { cwd: __dirname }).toString() } +function runDuplicateBindings() { + const hostProcess = process.execPath; + var testCode = + "console.log((function(bindings) {" + + "return bindings.pointerCheck1(bindings.pointerCheck2());" + + "})(require('duplicate_symbols')))" + console.log('running ', hostProcess); + return execFileSync(hostProcess, ['-e', testCode], { cwd: __dirname }).toString() +} + function getEncoding() { var code = 'import locale;print locale.getdefaultlocale()[1]' return execFileSync('python', ['-c', code]).toString().trim() @@ -52,6 +62,21 @@ test('build simple addon', function (t) { proc.stderr.setEncoding('utf-8') }) +test('make sure addon symbols do not overlap', function (t) { + t.plan(3) + + var addonPath = path.resolve(__dirname, 'node_modules', 'duplicate_symbols') + // Set the loglevel otherwise the output disappears when run via 'npm test' + var cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose'] + execFile(process.execPath, cmd, function (err, stdout, stderr) { + var logLines = stderr.trim().split(/\r?\n/) + var lastLine = logLines[logLines.length-1] + t.strictEqual(err, null) + t.strictEqual(lastLine, 'gyp info ok', 'should end in ok') + t.strictEqual(runDuplicateBindings().trim(), 'not equal') + }) +}) + test('build simple addon in path with non-ascii characters', function (t) { t.plan(1) From 1597c84aadc619e828c30001e6e9e39906aace73 Mon Sep 17 00:00:00 2001 From: cclauss Date: Tue, 14 May 2019 19:39:08 +0200 Subject: [PATCH 041/551] test: use Travis CI to run tests on every pull request This is a second attempt at #1336 which got into a bad git-state... Use flake8 to find Python syntax errors and undefined names. There are Python 3 syntax errors and many undefined names which may raise NameError at runtime. This PR runs flake8 runs in two passes: The first looks at critical issues in stop-the-build mode and the second looks at style violations in everything-is-a-warning mode. PR-URL: https://github.com/nodejs/node-gyp/pull/1752 Reviewed-By: Rod Vagg --- .travis.yml | 24 ++++++++++++++++++++++++ gyp/pylib/gyp/MSVSVersion.py | 6 +++--- gyp/pylib/gyp/input.py | 2 +- gyp/pylib/gyp/xcode_emulation.py | 2 +- 4 files changed, 29 insertions(+), 5 deletions(-) create mode 100644 .travis.yml diff --git a/.travis.yml b/.travis.yml new file mode 100644 index 0000000000..7607d306fc --- /dev/null +++ b/.travis.yml @@ -0,0 +1,24 @@ +dist: xenial +language: python +cache: pip +python: + - 2.7 + - 3.7 +matrix: + allow_failures: + - python: 3.7 +install: + #- pip install -r requirements.txt + - pip install flake8 # pytest # add another testing frameworks later +before_script: + # stop the build if there are Python syntax errors or undefined names + - flake8 . --count --select=E9,F63,F72,F82 --show-source --statistics + # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide + - flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics + - npm install +script: + - npm test + #- pytest --capture=sys # add other tests here +notifications: + on_success: change + on_failure: change # `always` will be the setting once code changes slow down diff --git a/gyp/pylib/gyp/MSVSVersion.py b/gyp/pylib/gyp/MSVSVersion.py index 293b4145c1..13d9777f0e 100644 --- a/gyp/pylib/gyp/MSVSVersion.py +++ b/gyp/pylib/gyp/MSVSVersion.py @@ -178,15 +178,15 @@ def _RegistryGetValueUsingWinReg(key, value): """ try: # Python 2 - from _winreg import OpenKey, QueryValueEx + from _winreg import HKEY_LOCAL_MACHINE, OpenKey, QueryValueEx except ImportError: # Python 3 - from winreg import OpenKey, QueryValueEx + from winreg import HKEY_LOCAL_MACHINE, OpenKey, QueryValueEx try: root, subkey = key.split('\\', 1) assert root == 'HKLM' # Only need HKLM for now. - with OpenKey(_winreg.HKEY_LOCAL_MACHINE, subkey) as hkey: + with OpenKey(HKEY_LOCAL_MACHINE, subkey) as hkey: return QueryValueEx(hkey, value)[0] except WindowsError: return None diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index 68e51131d8..eb9858f0c8 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -1182,7 +1182,7 @@ def LoadVariablesFromVariablesDict(variables, the_dict, the_dict_key): if variable_name in variables: # If the variable is already set, don't set it. continue - if the_dict_key is 'variables' and variable_name in the_dict: + if the_dict_key == 'variables' and variable_name in the_dict: # If the variable is set without a % in the_dict, and the_dict is a # variables dict (making |variables| a varaibles sub-dict of a # variables dict), use the_dict's definition. diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py index 66f325932a..6ae41e293a 100644 --- a/gyp/pylib/gyp/xcode_emulation.py +++ b/gyp/pylib/gyp/xcode_emulation.py @@ -844,7 +844,7 @@ def GetLdflags(self, configname, product_dir, gyp_to_build_path, arch=None): if self._IsXCTest(): platform_root = self._XcodePlatformPath(configname) if platform_root: - cflags.append('-F' + platform_root + '/Developer/Library/Frameworks/') + cflags.append('-F' + platform_root + '/Developer/Library/Frameworks/') # noqa TODO @cclauss is_extension = self._IsIosAppExtension() or self._IsIosWatchKitExtension() if sdk_root and is_extension: From dd9bf929ac9b39fa5e3f4a307cf518d5a3e0edaf Mon Sep 17 00:00:00 2001 From: "Shuowang (Wayne) Zhang" Date: Mon, 3 Jun 2019 11:44:24 -0400 Subject: [PATCH 042/551] zos: update compiler options PR-URL: https://github.com/nodejs/node-gyp/pull/1768 Reviewed-By: Richard Lau --- addon.gypi | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/addon.gypi b/addon.gypi index e1c4b24af2..3cadaab78b 100644 --- a/addon.gypi +++ b/addon.gypi @@ -110,7 +110,14 @@ 'cflags': [ '-q64', '-Wc,DLL', - '-qlonglong' + '-qlonglong', + '-qenum=int', + '-qxclang=-fexec-charset=ISO8859-1' + ], + 'defines': [ + '_ALL_SOURCE=1', + 'MAP_FAILED=-1', + '_UNIX03_SOURCE=1' ], 'ldflags': [ '-q64', From 4f4a677dfa96a5ba968f2a4bb4255388c73d31a7 Mon Sep 17 00:00:00 2001 From: "Shuowang (Wayne) Zhang" Date: Mon, 3 Jun 2019 11:44:43 -0400 Subject: [PATCH 043/551] gyp: use different default compiler for z/OS PR-URL: https://github.com/nodejs/node-gyp/pull/1768 Reviewed-By: Richard Lau --- gyp/pylib/gyp/generator/make.py | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index d549e899d8..37ac255bfa 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -2059,6 +2059,14 @@ def CalculateMakefilePath(build_file, base_name): 'srcdir': srcdir, 'copy_archive_args': copy_archive_arguments, 'makedep_args': makedep_arguments, + 'CC.target': GetEnvironFallback(('CC_target', 'CC'), '$(CC)'), + 'AR.target': GetEnvironFallback(('AR_target', 'AR'), '$(AR)'), + 'CXX.target': GetEnvironFallback(('CXX_target', 'CXX'), '$(CXX)'), + 'LINK.target': GetEnvironFallback(('LINK_target', 'LINK'), '$(LINK)'), + 'CC.host': GetEnvironFallback(('CC_host', 'CC'), 'gcc'), + 'AR.host': GetEnvironFallback(('AR_host', 'AR'), 'ar'), + 'CXX.host': GetEnvironFallback(('CXX_host', 'CXX'), 'g++'), + 'LINK.host': GetEnvironFallback(('LINK_host', 'LINK'), '$(CXX.host)'), } if flavor == 'mac': flock_command = './gyp-mac-tool flock' @@ -2079,6 +2087,10 @@ def CalculateMakefilePath(build_file, base_name): 'copy_archive_args': copy_archive_arguments, 'makedep_args': makedep_arguments, 'link_commands': LINK_COMMANDS_OS390, + 'CC.target': GetEnvironFallback(('CC_target', 'CC'), 'njsc'), + 'CXX.target': GetEnvironFallback(('CXX_target', 'CXX'), 'njsc++'), + 'CC.host': GetEnvironFallback(('CC_host', 'CC'), 'njsc'), + 'CXX.host': GetEnvironFallback(('CXX_host', 'CXX'), 'njsc++'), }) elif flavor == 'solaris': header_params.update({ @@ -2104,17 +2116,6 @@ def CalculateMakefilePath(build_file, base_name): 'flock_index': 2, }) - header_params.update({ - 'CC.target': GetEnvironFallback(('CC_target', 'CC'), '$(CC)'), - 'AR.target': GetEnvironFallback(('AR_target', 'AR'), '$(AR)'), - 'CXX.target': GetEnvironFallback(('CXX_target', 'CXX'), '$(CXX)'), - 'LINK.target': GetEnvironFallback(('LINK_target', 'LINK'), '$(LINK)'), - 'CC.host': GetEnvironFallback(('CC_host', 'CC'), 'gcc'), - 'AR.host': GetEnvironFallback(('AR_host', 'AR'), 'ar'), - 'CXX.host': GetEnvironFallback(('CXX_host', 'CXX'), 'g++'), - 'LINK.host': GetEnvironFallback(('LINK_host', 'LINK'), '$(CXX.host)'), - }) - build_file, _, _ = gyp.common.ParseQualifiedTarget(target_list[0]) make_global_settings_array = data[build_file].get('make_global_settings', []) wrappers = {} From f952b08f84505c0a283a2d0df957f81c8e39d373 Mon Sep 17 00:00:00 2001 From: cclauss Date: Thu, 20 Jun 2019 12:58:32 +0200 Subject: [PATCH 044/551] gyp: move from __future__ import to the top of the file Fixes #1774 PR-URL: https://github.com/nodejs/node-gyp/pull/1789 Reviewed-By: Rod Vagg --- gyp/tools/pretty_sln.py | 4 ++-- gyp/tools/pretty_vcproj.py | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/gyp/tools/pretty_sln.py b/gyp/tools/pretty_sln.py index 731844caf7..196566fb9e 100755 --- a/gyp/tools/pretty_sln.py +++ b/gyp/tools/pretty_sln.py @@ -12,8 +12,6 @@ Then it outputs a possible build order. """ -__author__ = 'nsylvain (Nicolas Sylvain)' - from __future__ import print_function import os @@ -21,6 +19,8 @@ import sys import pretty_vcproj +__author__ = 'nsylvain (Nicolas Sylvain)' + def BuildProject(project, built, projects, deps): # if all dependencies are done, we can build it, otherwise we try to build the # dependency. diff --git a/gyp/tools/pretty_vcproj.py b/gyp/tools/pretty_vcproj.py index 51cfc4e081..da7eb254cc 100755 --- a/gyp/tools/pretty_vcproj.py +++ b/gyp/tools/pretty_vcproj.py @@ -11,7 +11,6 @@ It outputs the resulting xml to stdout. """ -__author__ = 'nsylvain (Nicolas Sylvain)' from __future__ import print_function @@ -21,6 +20,8 @@ from xml.dom.minidom import parse from xml.dom.minidom import Node +__author__ = 'nsylvain (Nicolas Sylvain)' + REPLACEMENTS = dict() ARGUMENTS = None From ec2eb44a30a9a817b281786b928d52abb7ae4cfc Mon Sep 17 00:00:00 2001 From: Gabriel Schulhof Date: Thu, 20 Jun 2019 09:32:40 -0700 Subject: [PATCH 045/551] test: use Nan in duplicate_symbols Use `Nan::Set()` and `Nan::GetFunction()` instead of their V8 equivalents to avoid OSX test failures with Node.js v12.x. Thanks @rvagg! Re: https://github.com/nodejs/node-addon-api/pull/456 Fixes: https://github.com/nodejs/node/issues/26765 PR-URL: https://github.com/nodejs/node-gyp/pull/1689 Reviewed-By: Rod Vagg --- test/node_modules/duplicate_symbols/binding.cc | 6 +++--- test/node_modules/duplicate_symbols/package.json | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/test/node_modules/duplicate_symbols/binding.cc b/test/node_modules/duplicate_symbols/binding.cc index bc5f6f5134..a0999b76dc 100644 --- a/test/node_modules/duplicate_symbols/binding.cc +++ b/test/node_modules/duplicate_symbols/binding.cc @@ -2,9 +2,9 @@ #include "common.h" void Init(v8::Local exports) { - exports->Set(Nan::New("pointerCheck").ToLocalChecked(), - Nan::New(Something::PointerCheck) - ->GetFunction()); + Nan::Set(exports, Nan::New("pointerCheck").ToLocalChecked(), + Nan::GetFunction( + Nan::New(Something::PointerCheck)).ToLocalChecked()); } NODE_MODULE(NODE_GYP_MODULE_NAME, Init) diff --git a/test/node_modules/duplicate_symbols/package.json b/test/node_modules/duplicate_symbols/package.json index ee21161cf1..2a193f230f 100644 --- a/test/node_modules/duplicate_symbols/package.json +++ b/test/node_modules/duplicate_symbols/package.json @@ -6,7 +6,7 @@ "private": true, "dependencies": { "bindings": "~1.2.1", - "nan": "^2.0.0" + "nan": "^2.14.0" }, "scripts": { "test": "node index.js" From 611bc3c89f70f247f66af04a3206044c6fd1721d Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Thu, 20 Jun 2019 17:08:53 +1000 Subject: [PATCH 046/551] lib: add .json suffix for explicit require Multiple reports of this failing, seems to be due to node wrapping tools that mess with the require extensions. So let's be explicit about it. PR-URL: https://github.com/nodejs/node-gyp/pull/1787 Reviewed-By: Reviewed-By: Richard Lau --- lib/node-gyp.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/node-gyp.js b/lib/node-gyp.js index 8dd551bf3d..21ed80f9ff 100644 --- a/lib/node-gyp.js +++ b/lib/node-gyp.js @@ -51,7 +51,7 @@ var proto = Gyp.prototype * Export the contents of the package.json. */ -proto.package = require('../package') +proto.package = require('../package.json') /** * nopt configuration definitions From d3478d7b0b137694483af805e3b60ff4691f0c28 Mon Sep 17 00:00:00 2001 From: Refael Ackermann Date: Sun, 14 Oct 2018 18:17:49 -0400 Subject: [PATCH 047/551] meta: add to .gitignore PR-URL: https://github.com/nodejs/node-gyp/pull/1573 Reviewed-By: Rod Vagg --- .gitignore | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitignore b/.gitignore index 6748492014..4d6b4d55b7 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,5 @@ gyp/test node_modules test/.node-gyp +.ncu +package-lock.json From 03683f09d6d153c2b5f27764aaa1bc0c046ab380 Mon Sep 17 00:00:00 2001 From: Pavel Medvedev Date: Mon, 20 Jun 2016 16:55:59 +0300 Subject: [PATCH 048/551] lib: code de-duplication Modified the result of processRelease() to return uniform names `libUrl`, `libPath` for Windows in objects named as target platforms (`ia32` and `x64`) This allows to loop over the target platforms in downloadNodeLib() and to use the universal names instead of two almost identical code parts with `libUrl32`, `libPath32` and `libUrl64`, `libPath64` names. PR-URL: https://github.com/nodejs/node-gyp/pull/965 Reviewed-By: Rod Vagg --- lib/install.js | 97 ++++++++++++------------------------ lib/process-release.js | 6 +-- test/test-process-release.js | 96 ++++++++++++----------------------- 3 files changed, 67 insertions(+), 132 deletions(-) diff --git a/lib/install.js b/lib/install.js index 735c31c0d0..06353663db 100644 --- a/lib/install.js +++ b/lib/install.js @@ -301,75 +301,44 @@ function install (fs, gyp, argv, callback) { function downloadNodeLib (done) { log.verbose('on Windows; need to download `' + release.name + '.lib`...') - var dir32 = path.resolve(devDir, 'ia32') - , dir64 = path.resolve(devDir, 'x64') - , libPath32 = path.resolve(dir32, release.name + '.lib') - , libPath64 = path.resolve(dir64, release.name + '.lib') - - log.verbose('32-bit ' + release.name + '.lib dir', dir32) - log.verbose('64-bit ' + release.name + '.lib dir', dir64) - log.verbose('`' + release.name + '.lib` 32-bit url', release.libUrl32) - log.verbose('`' + release.name + '.lib` 64-bit url', release.libUrl64) - - var async = 2 - mkdir(dir32, function (err) { - if (err) return done(err) - log.verbose('streaming 32-bit ' + release.name + '.lib to:', libPath32) - - try { - var req = download(gyp, process.env, release.libUrl32, cb) - } catch (e) { - return cb(e) - } - - req.on('error', done) - req.on('response', function (res) { - if (res.statusCode !== 200) { - done(new Error(res.statusCode + ' status code downloading 32-bit ' + release.name + '.lib')) - return + var archs = [ 'ia32', 'x64' ] + var async = archs.length + archs.forEach(function (arch) { + var dir = path.resolve(devDir, arch) + var targetLibPath = path.resolve(dir, release.name + '.lib') + var libUrl = release[arch].libUrl + var libPath = release[arch].libPath + var name = arch + ' ' + release.name + '.lib' + log.verbose(name, 'dir', dir) + log.verbose(name, 'url', libUrl) + + mkdir(dir, function (err) { + if (err) return done(err) + log.verbose('streaming', name, 'to:', targetLibPath) + + try { + var req = download(gyp, process.env, libUrl, cb) + } catch (e) { + return cb(e) } - getContentSha(res, function (_, checksum) { - contentShasums[release.libPath32] = checksum - log.verbose('content checksum', release.libPath32, checksum) - }) - - var ws = fs.createWriteStream(libPath32) - ws.on('error', cb) - req.pipe(ws) - }) - req.on('end', function () { - --async || done() - }) - }) - mkdir(dir64, function (err) { - if (err) return done(err) - log.verbose('streaming 64-bit ' + release.name + '.lib to:', libPath64) - - try { - var req = download(gyp, process.env, release.libUrl64, cb) - } catch (e) { - return cb(e) - } + req.on('error', done) + req.on('response', function (res) { + if (res.statusCode !== 200) { + done(new Error(res.statusCode + ' status code downloading ' + name)) + return + } - req.on('error', done) - req.on('response', function (res) { - if (res.statusCode !== 200) { - done(new Error(res.statusCode + ' status code downloading 64-bit ' + release.name + '.lib')) - return - } + getContentSha(res, function (_, checksum) { + contentShasums[libPath] = checksum + log.verbose('content checksum', libPath, checksum) + }) - getContentSha(res, function (_, checksum) { - contentShasums[release.libPath64] = checksum - log.verbose('content checksum', release.libPath64, checksum) + var ws = fs.createWriteStream(targetLibPath) + ws.on('error', cb) + req.pipe(ws) }) - - var ws = fs.createWriteStream(libPath64) - ws.on('error', cb) - req.pipe(ws) - }) - req.on('end', function () { - --async || done() + req.on('end', function () { --async || done() }) }) }) } // downloadNodeLib() diff --git a/lib/process-release.js b/lib/process-release.js index f0c640b2d7..1be5be97ab 100644 --- a/lib/process-release.js +++ b/lib/process-release.js @@ -97,10 +97,8 @@ function processRelease (argv, gyp, defaultVersion, defaultRelease) { tarballUrl: tarballUrl, shasumsUrl: url.resolve(baseUrl, 'SHASUMS256.txt'), versionDir: (name !== 'node' ? name + '-' : '') + version, - libUrl32: libUrl32, - libUrl64: libUrl64, - libPath32: normalizePath(path.relative(url.parse(baseUrl).path, url.parse(libUrl32).path)), - libPath64: normalizePath(path.relative(url.parse(baseUrl).path, url.parse(libUrl64).path)) + ia32: { libUrl: libUrl32, libPath: normalizePath(path.relative(url.parse(baseUrl).path, url.parse(libUrl32).path)) }, + x64: { libUrl: libUrl64, libPath: normalizePath(path.relative(url.parse(baseUrl).path, url.parse(libUrl64).path)) }, } } diff --git a/test/test-process-release.js b/test/test-process-release.js index 2890e8a61c..5e4fb81042 100644 --- a/test/test-process-release.js +++ b/test/test-process-release.js @@ -16,10 +16,8 @@ test('test process release - process.version = 0.8.20', function (t) { tarballUrl: 'https://nodejs.org/dist/v0.8.20/node-v0.8.20.tar.gz', shasumsUrl: 'https://nodejs.org/dist/v0.8.20/SHASUMS256.txt', versionDir: '0.8.20', - libUrl32: 'https://nodejs.org/dist/v0.8.20/node.lib', - libUrl64: 'https://nodejs.org/dist/v0.8.20/x64/node.lib', - libPath32: 'node.lib', - libPath64: 'x64/node.lib' + ia32: { libUrl: 'https://nodejs.org/dist/v0.8.20/node.lib', libPath: 'node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.8.20/x64/node.lib', libPath: 'x64/node.lib' }, }) }) @@ -38,10 +36,8 @@ test('test process release - process.version = 0.10.21', function (t) { tarballUrl: 'https://nodejs.org/dist/v0.10.21/node-v0.10.21.tar.gz', shasumsUrl: 'https://nodejs.org/dist/v0.10.21/SHASUMS256.txt', versionDir: '0.10.21', - libUrl32: 'https://nodejs.org/dist/v0.10.21/node.lib', - libUrl64: 'https://nodejs.org/dist/v0.10.21/x64/node.lib', - libPath32: 'node.lib', - libPath64: 'x64/node.lib' + ia32: { libUrl: 'https://nodejs.org/dist/v0.10.21/node.lib', libPath: 'node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.10.21/x64/node.lib', libPath: 'x64/node.lib' }, }) }) @@ -61,10 +57,8 @@ test('test process release - process.version = 0.12.9', function (t) { tarballUrl: 'https://nodejs.org/dist/v0.12.9/node-v0.12.9.tar.gz', shasumsUrl: 'https://nodejs.org/dist/v0.12.9/SHASUMS256.txt', versionDir: '0.12.9', - libUrl32: 'https://nodejs.org/dist/v0.12.9/node.lib', - libUrl64: 'https://nodejs.org/dist/v0.12.9/x64/node.lib', - libPath32: 'node.lib', - libPath64: 'x64/node.lib' + ia32: { libUrl: 'https://nodejs.org/dist/v0.12.9/node.lib', libPath: 'node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.12.9/x64/node.lib', libPath: 'x64/node.lib' }, }) }) @@ -84,10 +78,8 @@ test('test process release - process.version = 0.10.41', function (t) { tarballUrl: 'https://nodejs.org/dist/v0.10.41/node-v0.10.41.tar.gz', shasumsUrl: 'https://nodejs.org/dist/v0.10.41/SHASUMS256.txt', versionDir: '0.10.41', - libUrl32: 'https://nodejs.org/dist/v0.10.41/node.lib', - libUrl64: 'https://nodejs.org/dist/v0.10.41/x64/node.lib', - libPath32: 'node.lib', - libPath64: 'x64/node.lib' + ia32: { libUrl: 'https://nodejs.org/dist/v0.10.41/node.lib', libPath: 'node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.10.41/x64/node.lib', libPath: 'x64/node.lib' }, }) }) @@ -107,10 +99,8 @@ test('test process release - process.release ~ node@0.10.42', function (t) { tarballUrl: 'https://nodejs.org/dist/v0.10.42/node-v0.10.42-headers.tar.gz', shasumsUrl: 'https://nodejs.org/dist/v0.10.42/SHASUMS256.txt', versionDir: '0.10.42', - libUrl32: 'https://nodejs.org/dist/v0.10.42/node.lib', - libUrl64: 'https://nodejs.org/dist/v0.10.42/x64/node.lib', - libPath32: 'node.lib', - libPath64: 'x64/node.lib' + ia32: { libUrl: 'https://nodejs.org/dist/v0.10.42/node.lib', libPath: 'node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.10.42/x64/node.lib', libPath: 'x64/node.lib' }, }) }) @@ -130,10 +120,8 @@ test('test process release - process.release ~ node@0.12.10', function (t) { tarballUrl: 'https://nodejs.org/dist/v0.12.10/node-v0.12.10-headers.tar.gz', shasumsUrl: 'https://nodejs.org/dist/v0.12.10/SHASUMS256.txt', versionDir: '0.12.10', - libUrl32: 'https://nodejs.org/dist/v0.12.10/node.lib', - libUrl64: 'https://nodejs.org/dist/v0.12.10/x64/node.lib', - libPath32: 'node.lib', - libPath64: 'x64/node.lib' + ia32: { libUrl: 'https://nodejs.org/dist/v0.12.10/node.lib', libPath: 'node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.12.10/x64/node.lib', libPath: 'x64/node.lib' }, }) }) @@ -155,10 +143,8 @@ test('test process release - process.release ~ node@4.1.23', function (t) { tarballUrl: 'https://nodejs.org/dist/v4.1.23/node-v4.1.23-headers.tar.gz', shasumsUrl: 'https://nodejs.org/dist/v4.1.23/SHASUMS256.txt', versionDir: '4.1.23', - libUrl32: 'https://nodejs.org/dist/v4.1.23/win-x86/node.lib', - libUrl64: 'https://nodejs.org/dist/v4.1.23/win-x64/node.lib', - libPath32: 'win-x86/node.lib', - libPath64: 'win-x64/node.lib' + ia32: { libUrl: 'https://nodejs.org/dist/v4.1.23/win-x86/node.lib', libPath: 'win-x86/node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' }, }) }) @@ -180,10 +166,8 @@ test('test process release - process.release ~ node@4.1.23 / corp build', functi tarballUrl: 'https://some.custom.location/node-v4.1.23-headers.tar.gz', shasumsUrl: 'https://some.custom.location/SHASUMS256.txt', versionDir: '4.1.23', - libUrl32: 'https://some.custom.location/win-x86/node.lib', - libUrl64: 'https://some.custom.location/win-x64/node.lib', - libPath32: 'win-x86/node.lib', - libPath64: 'win-x64/node.lib' + ia32: { libUrl: 'https://some.custom.location/win-x86/node.lib', libPath: 'win-x86/node.lib' }, + x64: { libUrl: 'https://some.custom.location/win-x64/node.lib', libPath: 'win-x64/node.lib' }, }) }) @@ -205,10 +189,8 @@ test('test process release - process.release ~ node@4.1.23 --target=0.10.40', fu tarballUrl: 'https://nodejs.org/dist/v0.10.40/node-v0.10.40.tar.gz', shasumsUrl: 'https://nodejs.org/dist/v0.10.40/SHASUMS256.txt', versionDir: '0.10.40', - libUrl32: 'https://nodejs.org/dist/v0.10.40/node.lib', - libUrl64: 'https://nodejs.org/dist/v0.10.40/x64/node.lib', - libPath32: 'node.lib', - libPath64: 'x64/node.lib' + ia32: { libUrl: 'https://nodejs.org/dist/v0.10.40/node.lib', libPath: 'node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.10.40/x64/node.lib', libPath: 'x64/node.lib' }, }) }) @@ -230,10 +212,8 @@ test('test process release - process.release ~ node@4.1.23 --dist-url=https://fo tarballUrl: 'https://foo.bar/baz/v4.1.23/node-v4.1.23-headers.tar.gz', shasumsUrl: 'https://foo.bar/baz/v4.1.23/SHASUMS256.txt', versionDir: '4.1.23', - libUrl32: 'https://foo.bar/baz/v4.1.23/win-x86/node.lib', - libUrl64: 'https://foo.bar/baz/v4.1.23/win-x64/node.lib', - libPath32: 'win-x86/node.lib', - libPath64: 'win-x64/node.lib' + ia32: { libUrl: 'https://foo.bar/baz/v4.1.23/win-x86/node.lib', libPath: 'win-x86/node.lib' }, + x64: { libUrl: 'https://foo.bar/baz/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' }, }) }) @@ -255,10 +235,8 @@ test('test process release - process.release ~ frankenstein@4.1.23', function (t tarballUrl: 'https://frankensteinjs.org/dist/v4.1.23/frankenstein-v4.1.23-headers.tar.gz', shasumsUrl: 'https://frankensteinjs.org/dist/v4.1.23/SHASUMS256.txt', versionDir: 'frankenstein-4.1.23', - libUrl32: 'https://frankensteinjs.org/dist/v4.1.23/win-x86/frankenstein.lib', - libUrl64: 'https://frankensteinjs.org/dist/v4.1.23/win-x64/frankenstein.lib', - libPath32: 'win-x86/frankenstein.lib', - libPath64: 'win-x64/frankenstein.lib' + ia32: { libUrl: 'https://frankensteinjs.org/dist/v4.1.23/win-x86/frankenstein.lib', libPath: 'win-x86/frankenstein.lib' }, + x64: { libUrl: 'https://frankensteinjs.org/dist/v4.1.23/win-x64/frankenstein.lib', libPath: 'win-x64/frankenstein.lib' }, }) }) @@ -281,10 +259,8 @@ test('test process release - process.release ~ frankenstein@4.1.23 --dist-url=ht tarballUrl: 'http://foo.bar/baz/v4.1.23/frankenstein-v4.1.23-headers.tar.gz', shasumsUrl: 'http://foo.bar/baz/v4.1.23/SHASUMS256.txt', versionDir: 'frankenstein-4.1.23', - libUrl32: 'http://foo.bar/baz/v4.1.23/win-x86/frankenstein.lib', - libUrl64: 'http://foo.bar/baz/v4.1.23/win-x64/frankenstein.lib', - libPath32: 'win-x86/frankenstein.lib', - libPath64: 'win-x64/frankenstein.lib' + ia32: { libUrl: 'http://foo.bar/baz/v4.1.23/win-x86/frankenstein.lib', libPath: 'win-x86/frankenstein.lib' }, + x64: { libUrl: 'http://foo.bar/baz/v4.1.23/win-x64/frankenstein.lib', libPath: 'win-x64/frankenstein.lib' }, }) }) @@ -306,10 +282,8 @@ test('test process release - process.release ~ node@4.0.0-rc.4', function (t) { tarballUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz', shasumsUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/SHASUMS256.txt', versionDir: '4.0.0-rc.4', - libUrl32: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x86/node.lib', - libUrl64: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', - libPath32: 'win-x86/node.lib', - libPath64: 'win-x64/node.lib' + ia32: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x86/node.lib', libPath: 'win-x86/node.lib' }, + x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' }, }) }) @@ -334,10 +308,8 @@ test('test process release - process.release ~ node@4.0.0-rc.4 passed as argv[0] tarballUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz', shasumsUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/SHASUMS256.txt', versionDir: '4.0.0-rc.4', - libUrl32: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x86/node.lib', - libUrl64: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', - libPath32: 'win-x86/node.lib', - libPath64: 'win-x64/node.lib' + ia32: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x86/node.lib', libPath: 'win-x86/node.lib' }, + x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' }, }) }) @@ -362,10 +334,8 @@ test('test process release - process.release ~ node@4.0.0-rc.4 - bogus string pa tarballUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz', shasumsUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/SHASUMS256.txt', versionDir: '4.0.0-rc.4', - libUrl32: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x86/node.lib', - libUrl64: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', - libPath32: 'win-x86/node.lib', - libPath64: 'win-x64/node.lib' + ia32: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x86/node.lib', libPath: 'win-x86/node.lib' }, + x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' }, }) }) @@ -389,10 +359,8 @@ test('test process release - NODEJS_ORG_MIRROR', function (t) { tarballUrl: 'http://foo.bar/v4.1.23/node-v4.1.23-headers.tar.gz', shasumsUrl: 'http://foo.bar/v4.1.23/SHASUMS256.txt', versionDir: '4.1.23', - libUrl32: 'http://foo.bar/v4.1.23/win-x86/node.lib', - libUrl64: 'http://foo.bar/v4.1.23/win-x64/node.lib', - libPath32: 'win-x86/node.lib', - libPath64: 'win-x64/node.lib' + ia32: { libUrl: 'http://foo.bar/v4.1.23/win-x86/node.lib', libPath: 'win-x86/node.lib' }, + x64: { libUrl: 'http://foo.bar/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' }, }) delete process.env.NODEJS_ORG_MIRROR From a52c6eb9e8f8efebcbcebd2dbfa5eb63488c7cfb Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Sat, 22 Jun 2019 13:35:45 +1000 Subject: [PATCH 049/551] test: migrate from tape to tap PR-URL: https://github.com/nodejs/node-gyp/pull/1795 Reviewed-By: Richard Lau --- .gitignore | 1 + .travis.yml | 2 +- package.json | 5 +++-- test/simple-proxy.js | 1 - test/test-addon.js | 4 +--- test/test-configure-python.js | 4 +++- test/test-download.js | 4 +++- test/test-find-accessible-sync.js | 2 +- test/test-find-node-directory.js | 2 +- test/test-find-python.js | 4 +++- test/test-find-visualstudio.js | 2 +- test/test-install.js | 5 +++-- test/test-options.js | 2 +- test/test-process-release.js | 2 +- 14 files changed, 23 insertions(+), 17 deletions(-) diff --git a/.gitignore b/.gitignore index 4d6b4d55b7..776066e34d 100644 --- a/.gitignore +++ b/.gitignore @@ -2,4 +2,5 @@ gyp/test node_modules test/.node-gyp .ncu +.nyc_output package-lock.json diff --git a/.travis.yml b/.travis.yml index 7607d306fc..f76e561464 100644 --- a/.travis.yml +++ b/.travis.yml @@ -17,7 +17,7 @@ before_script: - flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics - npm install script: - - npm test + - npm run test #- pytest --capture=sys # add other tests here notifications: on_success: change diff --git a/package.json b/package.json index 393e7cac7e..36a00a48ae 100644 --- a/package.json +++ b/package.json @@ -43,10 +43,11 @@ "eslint": "^5.0.1", "nan": "^2.0.0", "require-inject": "~1.3.0", - "tape": "~4.2.0" + "tap": "~14.2.4" }, "scripts": { "lint": "eslint bin lib test", - "test": "npm run lint && tape test/test-*" + "test": "npm run lint && tap test/test-*", + "test-ci": "npm run lint && tap -Rtap test/test-*" } } diff --git a/test/simple-proxy.js b/test/simple-proxy.js index e55330c445..8ac737a652 100644 --- a/test/simple-proxy.js +++ b/test/simple-proxy.js @@ -13,7 +13,6 @@ function handler (req, res) { throw new Error('request url [' + req.url + '] does not start with [' + prefix + ']') var upstreamUrl = upstream + req.url.substring(prefix.length) - console.log(req.url + ' -> ' + upstreamUrl) https.get(upstreamUrl, function (ures) { ures.on('end', function () { if (++calls == 2) diff --git a/test/test-addon.js b/test/test-addon.js index 12687de2ac..5334ab0b18 100644 --- a/test/test-addon.js +++ b/test/test-addon.js @@ -1,6 +1,6 @@ 'use strict' -var test = require('tape') +var test = require('tap').test var path = require('path') var fs = require('graceful-fs') var child_process = require('child_process') @@ -15,7 +15,6 @@ function runHello(hostProcess) { hostProcess = process.execPath } var testCode = "console.log(require('hello_world').hello())" - console.log('running ', hostProcess); return execFileSync(hostProcess, ['-e', testCode], { cwd: __dirname }).toString() } @@ -25,7 +24,6 @@ function runDuplicateBindings() { "console.log((function(bindings) {" + "return bindings.pointerCheck1(bindings.pointerCheck2());" + "})(require('duplicate_symbols')))" - console.log('running ', hostProcess); return execFileSync(hostProcess, ['-e', testCode], { cwd: __dirname }).toString() } diff --git a/test/test-configure-python.js b/test/test-configure-python.js index 07cdce2b17..895e488ba8 100644 --- a/test/test-configure-python.js +++ b/test/test-configure-python.js @@ -1,6 +1,6 @@ 'use strict' -var test = require('tape') +var test = require('tap').test var path = require('path') var gyp = require('../lib/node-gyp') var requireInject = require('require-inject') @@ -17,6 +17,8 @@ var EXPECTED_PYPATH = path.join(__dirname, '..', 'gyp', 'pylib') var SEPARATOR = process.platform == 'win32' ? ';' : ':' var SPAWN_RESULT = { on: function () { } } +require('npmlog').level = 'warn' + test('configure PYTHONPATH with no existing env', function (t) { t.plan(1) diff --git a/test/test-download.js b/test/test-download.js index 6e6f64f058..1615b55fb8 100644 --- a/test/test-download.js +++ b/test/test-download.js @@ -1,11 +1,13 @@ 'use strict' +var test = require('tap').test var fs = require('fs') var http = require('http') var https = require('https') -var test = require('tape') var install = require('../lib/install') +require('npmlog').level = 'warn' + test('download over http', function (t) { t.plan(2) diff --git a/test/test-find-accessible-sync.js b/test/test-find-accessible-sync.js index 114403e278..15adec5379 100644 --- a/test/test-find-accessible-sync.js +++ b/test/test-find-accessible-sync.js @@ -1,6 +1,6 @@ 'use strict' -var test = require('tape') +var test = require('tap').test var path = require('path') var requireInject = require('require-inject') var configure = requireInject('../lib/configure', { diff --git a/test/test-find-node-directory.js b/test/test-find-node-directory.js index 46659d0cfe..7cc680d1fa 100644 --- a/test/test-find-node-directory.js +++ b/test/test-find-node-directory.js @@ -1,4 +1,4 @@ -var test = require('tape') +var test = require('tap').test var path = require('path') var findNodeDirectory = require('../lib/find-node-directory') diff --git a/test/test-find-python.js b/test/test-find-python.js index f3c14ce659..fbb35c3d36 100644 --- a/test/test-find-python.js +++ b/test/test-find-python.js @@ -1,10 +1,12 @@ 'use strict' -var test = require('tape') +var test = require('tap').test var configure = require('../lib/configure') var execFile = require('child_process').execFile var PythonFinder = configure.test.PythonFinder +require('npmlog').level = 'warn' + test('find python', function (t) { t.plan(4) diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js index e2bcc9954f..cf6349c808 100644 --- a/test/test-find-visualstudio.js +++ b/test/test-find-visualstudio.js @@ -1,6 +1,6 @@ 'use strict' -const test = require('tape') +var test = require('tap').test const fs = require('fs') const path = require('path') const findVisualStudio = require('../lib/find-visualstudio') diff --git a/test/test-install.js b/test/test-install.js index f647326a7f..9e44c39bc1 100644 --- a/test/test-install.js +++ b/test/test-install.js @@ -1,8 +1,10 @@ 'use strict' -var test = require('tape') +var test = require('tap').test var install = require('../lib/install').test.install +require('npmlog').level = 'error' // we expect a warning + test('EACCES retry once', function (t) { t.plan(3) @@ -14,7 +16,6 @@ test('EACCES retry once', function (t) { t.ok(true); } - var gyp = {} gyp.devDir = __dirname gyp.opts = {} diff --git a/test/test-options.js b/test/test-options.js index d097f81be6..51d473308c 100644 --- a/test/test-options.js +++ b/test/test-options.js @@ -1,6 +1,6 @@ 'use strict'; -var test = require('tape') +var test = require('tap').test var gyp = require('../lib/node-gyp') test('options in environment', function (t) { diff --git a/test/test-process-release.js b/test/test-process-release.js index 5e4fb81042..83998af041 100644 --- a/test/test-process-release.js +++ b/test/test-process-release.js @@ -1,4 +1,4 @@ -var test = require('tape') +var test = require('tap').test var processRelease = require('../lib/process-release') test('test process release - process.version = 0.8.20', function (t) { From 395f843de08b4a1041de0a02ff6128edfccd127b Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Sat, 22 Jun 2019 16:28:22 +1000 Subject: [PATCH 050/551] test: replace self-signed cert with 'localhost' PR-URL: https://github.com/nodejs/node-gyp/pull/1795 Reviewed-By: Richard Lau --- .travis.yml | 2 +- test/fixtures/ca.crt | 38 +++++++++++++-------------- test/fixtures/server.crt | 36 +++++++++++++------------- test/fixtures/server.key | 55 ++++++++++++++++++++-------------------- test/test-download.js | 4 +-- 5 files changed, 68 insertions(+), 67 deletions(-) diff --git a/.travis.yml b/.travis.yml index f76e561464..7607d306fc 100644 --- a/.travis.yml +++ b/.travis.yml @@ -17,7 +17,7 @@ before_script: - flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics - npm install script: - - npm run test + - npm test #- pytest --capture=sys # add other tests here notifications: on_success: change diff --git a/test/fixtures/ca.crt b/test/fixtures/ca.crt index 9d2755a74f..aaf97575b1 100644 --- a/test/fixtures/ca.crt +++ b/test/fixtures/ca.crt @@ -1,21 +1,21 @@ -----BEGIN CERTIFICATE----- -MIIDbzCCAlcCAmm6MA0GCSqGSIb3DQEBCwUAMH0xCzAJBgNVBAYTAlVTMQswCQYD -VQQIDAJDQTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzEZMBcGA1UECgwQU3Ryb25n -TG9vcCwgSW5jLjESMBAGA1UECwwJU3Ryb25nT3BzMRowGAYDVQQDDBFjYS5zdHJv -bmdsb29wLmNvbTAeFw0xNTEyMDgyMzM1MzNaFw00MzA0MjQyMzM1MzNaMH0xCzAJ -BgNVBAYTAlVTMQswCQYDVQQIDAJDQTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzEZ -MBcGA1UECgwQU3Ryb25nTG9vcCwgSW5jLjESMBAGA1UECwwJU3Ryb25nT3BzMRow -GAYDVQQDDBFjYS5zdHJvbmdsb29wLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEP -ADCCAQoCggEBANfj86jkvvYDjHBgiqWhk9Cj+bqiMq3MqnV0CBO4iuK33Fo6XssE -H+yVdXlIBFbFe6t655MdBVOR2Sfj7WqNh96vhu6PyDHiwcQlTaiLU6nhIed1J4Wv -lvnJHFmp8Wbtx5AgLT4UYu03ftvXEl2DLi3vhSL2tRM1ebXHB/KPbRWkb25DPX0P -foOHot3f2dgNe2x6kponf7E/QDmAu3s7Nlkfh+ryDhgGU7wocXEhXbprNqRqOGNo -xbXgUI+/9XDxYT/7Gn5LF/fPjtN+aB0SKMnTsDhprVlZie83mlqJ46fOOrR+vrsQ -mi/1m/TadrARtZoIExC/cQRdVM05EK4tUa8CAwEAATANBgkqhkiG9w0BAQsFAAOC -AQEAQ7k5WhyhDTIGYCNzRnrMHWSzGqa1y4tJMW06wafJNRqTm1cthq1ibc6Hfq5a -K10K0qMcgauRTfQ1MWrVCTW/KnJ1vkhiTOH+RvxapGn84gSaRmV6KZen0+gMsgae -KEGe/3Hn+PmDVV+PTamHgPACfpTww38WHIe/7Ce9gHfG7MZ8cKHNZhDy0IAYPln+ -YRwMLd7JNQffHAbWb2CE1mcea4H/12U8JZW5tHCF6y9V+7IuDzqwIrLKcW3lG17n -VUG6ODF/Ryqn3V5X+TL91YyXi6c34y34IpC7MQDV/67U7+5Bp5CfeDPWW2wVSrW+ -uGZtfEvhbNm6m2i4UNmpCXxUZQ== +MIIDZDCCAkwCCQCAzfCLqrJvuTANBgkqhkiG9w0BAQsFADB0MQswCQYDVQQGEwJV +UzELMAkGA1UECAwCQ0ExEDAOBgNVBAoMB05vZGUuanMxETAPBgNVBAsMCG5vZGUt +Z3lwMRIwEAYDVQQDDAlsb2NhbGhvc3QxHzAdBgkqhkiG9w0BCQEWEGJ1aWxkQG5v +ZGVqcy5vcmcwHhcNMTkwNjIyMDYyMjMzWhcNMjIwNDExMDYyMjMzWjB0MQswCQYD +VQQGEwJVUzELMAkGA1UECAwCQ0ExEDAOBgNVBAoMB05vZGUuanMxETAPBgNVBAsM +CG5vZGUtZ3lwMRIwEAYDVQQDDAlsb2NhbGhvc3QxHzAdBgkqhkiG9w0BCQEWEGJ1 +aWxkQG5vZGVqcy5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDS +CHjvtVW4HdbbUwZ/ZV9s6U4x0KSoyNQrsCZjB8kRpFPe50DS5mfmu2SNBGYKRgzk +4QEEwFB9N2o8YTWsCefSRl6ti4ToPZqulU4hhRKYrEGtMJcRzi3IN7s200JaO3UH +01Su8ruO0NESb5zEU1Ykfh8Lub8TGEAINmgI61d/5d5Aq3kDjUHQJt1Ekw03Ylnu +juQyCGZxLxnngu0mIvwzyL/UeeUgsfQLzvppUk6In7tC1zzMjSPWo0c8qu6KvrW4 +bKYnkZkzdQifzbpO5ERMEsh5HWq0uHa6+dgcVHFvlhdqF4Uat87ygNplVf0txsZB +MNVqbz1k6xkZYMnzDoydAgMBAAEwDQYJKoZIhvcNAQELBQADggEBADspZGtKpWxy +J1W3FA1aeQhMvequQTcMRz4avkm4K4HfTdV1iVD4CbvdezBphouBlyLVLDFJP7RZ +m7dBJVgBwnxufoFLne8cR2MGqDRoySbFT1AtDJdxabE6Fg+QGUpgOQfeBJ6ANlSB ++qJ+HG4QA+Ouh5hxz9mgYwkIsMUABHiwENdZ/kT8Edw4xKgd3uH0YP4iiePMD66c +rzW3uXH5J1jnKgBlpxtog4P6dHCcoq+PZJ17W5bdXNyqC1LPzQqniZ2BNcEZ4ix3 +slAZAOWD1zLLGJhBPMV1fa0sHNBWc6oicr3YK/IDb0cp9kiLvnUu1pHy+LWQGqtC +rceJuGsnJEQ= -----END CERTIFICATE----- diff --git a/test/fixtures/server.crt b/test/fixtures/server.crt index fe13bb96c5..5d0c440e07 100644 --- a/test/fixtures/server.crt +++ b/test/fixtures/server.crt @@ -1,19 +1,21 @@ -----BEGIN CERTIFICATE----- -MIIDJjCCAg4CAhnOMA0GCSqGSIb3DQEBBQUAMH0xCzAJBgNVBAYTAlVTMQswCQYD -VQQIDAJDQTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzEZMBcGA1UECgwQU3Ryb25n -TG9vcCwgSW5jLjESMBAGA1UECwwJU3Ryb25nT3BzMRowGAYDVQQDDBFjYS5zdHJv -bmdsb29wLmNvbTAeFw0xNTEyMDgyMzM1MzNaFw00MzA0MjQyMzM1MzNaMBkxFzAV -BgNVBAMMDnN0cm9uZ2xvb3AuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB -CgKCAQEAwOYI7OZ2FX/YjRgLZoDQlbPc5UZXU/j0e1wwiJNPtPEax9Y5Uoza0Pnt -Ikzkc2SfvQ+IJrhXo385tI0W5juuqbHnE7UrjUuPjUX6NHevkxcs/flmjan5wnZM -cPsGhH71WDuUEEflvZihf2Se2x+xgZtMhc5XGmVmRuZFYKvkgUhA2/w8/QrK+jPT -n9QRJxZjWNh2RBdC1B7u4jffSmOSUljYFH1I2eTeY+Rdi6YUIYSU9gEoZxsv3Tia -SomfMF5jt2Mouo6MzA+IhLvvFjcrcph1Qxgi9RkfdCMMd+Ipm9YWELkyG1bDRpQy -0iyHD4gvVsAqz1Y2KdRSdc3Kt+nTqwIDAQABoxkwFzAVBgNVHREEDjAMhwQAAAAA -hwR/AAABMA0GCSqGSIb3DQEBBQUAA4IBAQAhy4J0hML3NgmDRHdL5/iTucBe22Mf -jJjg2aifD1S187dHm+Il4qZNO2plWwAhN0h704f+8wpsaALxUvBIu6nvlvcMP5PH -jGN5JLe2Km3UaPvYOQU2SgacLilu+uBcIo2JSHLV6O7ziqUj5Gior6YxDLCtEZie -Ea8aX5/YjuACtEMJ1JjRqjgkM66XAoUe0E8onOK3FgTIO3tGoTJwRp0zS50pFuP0 -PsZtT04ck6mmXEXXknNoAyBCvPypfms9OHqcUIW9fiQnrGbS/Ri4QSQYj0DtFk/1 -na4fY1gf3zTHxH8259b/TOOaPfTnCEsOQtjUrWNR4xhmVZ+HJy4yytUW +MIIDYjCCAkoCCQCSlmGR7KzZGTANBgkqhkiG9w0BAQsFADB0MQswCQYDVQQGEwJV +UzELMAkGA1UECAwCQ0ExEDAOBgNVBAoMB05vZGUuanMxETAPBgNVBAsMCG5vZGUt +Z3lwMRIwEAYDVQQDDAlsb2NhbGhvc3QxHzAdBgkqhkiG9w0BCQEWEGJ1aWxkQG5v +ZGVqcy5vcmcwHhcNMTkwNjIyMDYyNTU1WhcNMjkwNjE5MDYyNTU1WjByMQswCQYD +VQQGEwJVUzELMAkGA1UECAwCQ0ExEDAOBgNVBAoMB05vZGUuanMxETAPBgNVBAsM +CG5vZGUtZ3lwMRIwEAYDVQQDDAlsb2NhbGhvc3QxHTAbBgkqhkiG9w0BCQEWDmJ1 +aWxkQGlvanMub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6S1E +2WchgmbJYqCnpN7310ZgHjIOqeJe6MpSue2u6z6mTNd5izgvQNaANmn3xLFCS5zs +uZaTvdPYPkcmSQzb1YcZSUYnAxZifjYARc6kb5GSBl3q+O70ELyFrimXfZ4JI+bd +IG9KiHY17DlvZZZj/csGYVWWg0mkeH3O5LPX6/DXQVh/9+gZ02/cdIBCAtZHQwqx +7tF/qZA/kD4GZNFpU1DYHzf9H6g9htoCqmNHQWrV2T9yFybt6mbZp9kglBmyKYCc +7hmQnb7N/mHn1yIuwhBsirCJTfKH86gN81u8M3+SVHA2VUHDllcNhpDWlmInXA+I +tHdGZHCp95ohqpCPgQIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQCdvYj6CD0ZLwT2 +3t1r+deC3TJuHlNVSeKeT7wIfFnh2FW5riGV0q/w6eXPLTHjuiS6YmpAAbdNUgX/ +sq64FqI2RLpX6pgY5yB0SKopMcJxMLKqmF4zHpIHxtYN5EmN3PR0vehneBR/nZ2T +3ikvWD5JeXlm7Dfw+tjijdxM/sEoDWErGup4mMKMd1s5s830p+ITJUa50d0DLFdH +mqPSbUZF8mMPwGJd+nu1Ht3gTLtK7+gYJgGtXMJmGC0Qg77EJHDB2NbotgDGNmSU +1H9BpAeFHHIcbh2Rr7kkTvnh/c03vFe+CsDZmezcmRpRzW1fKj3YbfqBxU4XwJrL +a5T/N9xU -----END CERTIFICATE----- diff --git a/test/fixtures/server.key b/test/fixtures/server.key index f8227f4c0c..a8447391e0 100644 --- a/test/fixtures/server.key +++ b/test/fixtures/server.key @@ -1,28 +1,27 @@ ------BEGIN PRIVATE KEY----- -MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDA5gjs5nYVf9iN -GAtmgNCVs9zlRldT+PR7XDCIk0+08RrH1jlSjNrQ+e0iTORzZJ+9D4gmuFejfzm0 -jRbmO66psecTtSuNS4+NRfo0d6+TFyz9+WaNqfnCdkxw+waEfvVYO5QQR+W9mKF/ -ZJ7bH7GBm0yFzlcaZWZG5kVgq+SBSEDb/Dz9Csr6M9Of1BEnFmNY2HZEF0LUHu7i -N99KY5JSWNgUfUjZ5N5j5F2LphQhhJT2AShnGy/dOJpKiZ8wXmO3Yyi6jozMD4iE -u+8WNytymHVDGCL1GR90Iwx34imb1hYQuTIbVsNGlDLSLIcPiC9WwCrPVjYp1FJ1 -zcq36dOrAgMBAAECggEACg60Xm2xsHNG/ixHw+NpfLSxCr89JGKxlJD88tIDcOK1 -S8AOoxA3BHhTddteeenALmJV7fbkkuC6SICmtgBcnfppmuxyRd6vsGT6o6ut2tR1 -gxRy1WYMYKg8WhOshlH8RspscODeyKDhorvDUJd5cNGBDuTwQ68PwxiUe3La6iac -EVQoKohg9EmRIhMF1i8I00zXE8p3XENrlTc491ipc+gLPIP5vtqHyQztEUkZHkWd -dXbs+n1hGCr+4FxrphGYEW80HINzmume7dGChr8nvF4ZZcuWW13DJuNim6pQno1i -hM8VdXm8XphLh0XEGI5OCfu/CetkBILZRXKltZk6AQKBgQDoBqJzRlp7regYNU4q -usfS+43tPNaJ0o4DIzcLawqpmK/B/cZStzHl14Sm62BVkKV6cnWAJPeLkENPMFoV -7Q7wLZBJxpPzqXkpeiDkKN4Wovca891Rffne5Sz6IDB5mOxMjfKIEPd5RkmB5Lkp -qQLwm3YJ2AJcLagG/Gi1DFDRAQKBgQDU1G9T43Mjke6TXG0u7gCSb+VwyDRsrvJA -u2vy6+MANRc1EEF31YLmTKOU5XxUmhtIu7TUbgPoNi0HuRFXx4Zul3BPlAosLMJv -kNQbA/9d0YQAfSgTsploN5CX65dLZ4ejIzVgDZREzpIBWTze6YZTA2DT5iOIet84 -DD5DujY4qwKBgG0PuUo/9oYOD3tZiv1wwD5+uY6auykbTF9TLStzzBY9y9d+hrsY -mx6zOAoRtz1g+TdeF7b9KVJzo//T9XQ68nuYnyreaWrt7SK+4jj8sK+pOEd1+0Cz -20CXLpX/jWmKpP+y9R5aA0kA7cpdjV90rwoTuN8Vpr5XQ5TNDhaTzGUBAoGABYig -fGXlkH8y3NICZL37ddNC+/O4qTrDQbudyusnM9ItkEuj6CG9DY/gkPaGjQyUuQdo -ZD2YDGmcMh81vDqL3ERDv03yFcP0KkJxwWIRObdA32JhsGFsa7FGKS0O+f7vH+bC -dITl3gQg97gCRSl9PJtR4TCSq/HF7Acld01YK5ECgYEAwLFB5JIuxrowJe74cCMP -n5Rwuc8vWdOsg+ytvQTv0/hVCdzcaLet6YvagnWTWaU7PUwTFxZs/mLQ9CAWVutK -IRzs/GWxGFjH5xotDaJdDDzSdQye4tUqvUVxv7zzzsVycCPBYFkyRQ8Tmr5FLtUJ -Cl48TZ6J8Rx5avjdtOw3QC8= ------END PRIVATE KEY----- +-----BEGIN RSA PRIVATE KEY----- +MIIEowIBAAKCAQEA6S1E2WchgmbJYqCnpN7310ZgHjIOqeJe6MpSue2u6z6mTNd5 +izgvQNaANmn3xLFCS5zsuZaTvdPYPkcmSQzb1YcZSUYnAxZifjYARc6kb5GSBl3q ++O70ELyFrimXfZ4JI+bdIG9KiHY17DlvZZZj/csGYVWWg0mkeH3O5LPX6/DXQVh/ +9+gZ02/cdIBCAtZHQwqx7tF/qZA/kD4GZNFpU1DYHzf9H6g9htoCqmNHQWrV2T9y +Fybt6mbZp9kglBmyKYCc7hmQnb7N/mHn1yIuwhBsirCJTfKH86gN81u8M3+SVHA2 +VUHDllcNhpDWlmInXA+ItHdGZHCp95ohqpCPgQIDAQABAoIBABW8R4ewalo6dJlB ++n6O3jFt+PW3mtBRLqGqgm2cb0q0a1IMX+MPWLBFjmwEErl+AH0F4rcmBx2Ryr17 +amEy1qcf0caXyHksNAApznqzWXag7iizxnxv4cZRnHBwphNqkNWM5p3oYd04j6w2 +amDg1O9KZozaKo6QZclpiMiezwjKG+PVZLT8p7afswjv+yDWPDByhlcGiye9QD1T +VuZ0QCoXp6N/8JxW0gdkLp9NqFvGeGFzJ5h6L+d7A6BWw8akXrBRHHcKkyvVYBfd +myhSzSK4FPFMaxaEY/65FlVSyAO6ezGm3Umx4g7mkFjLdwKWaIOjkBkPeFgl3Pp4 +7Lo5X3UCgYEA/FrrIwmEU5ayulBVScEMKeavu5eNY4r0Sqbpov2oyTdYe8G49Pzy +ryMXfunY43moLKpajGwgTKRGvdqFtK08AAkaCssiAPkP3rZuZvMTF4sLo/vlWrjP +3er+tUqj22BzXi5XV0BAvH8Y3TL8KQ3he/8JxDvkC811/DQ9Y/Da3U8CgYEA7Itw +UM37URma08Bj9VTMoL9ZCyURewX+ZLDb2+O8sXGXJs28i1RkE6PTBlnRmedn+Jjk +byzQ5Cs5wA5uMbhYTA7kgXOs1bvgQqmlLmyL6FfHkucoMhr2Di7VeGf4OxE26JZ8 +JdY4+1MOyI3A2rR8WU+GmHxy0ay4K2xe6W0vsi8CgYBoGLEKIPDe8jkDtgOYivOT +jT9MaLXALB+dc8DIpU4swpHTaxP6qyUIrbcReTEolJSU6Ci16BxiwRkVU8D3yMYJ +VbfSX/zE3fh37FUaToa/nXHN0SjJBZdpeXhcHE//PIgaf48zxKNvnhYJmPB/luQ+ +m/PRaMsnOzfCM2JniYEe7QKBgGwjnxhB4tgDtaWCue/pcZc3gzS2IJS2e8N6mzie +l6Ajhu+FdOHZldrotUuc+la61OxwsVYmDeWR4VftAPGYDj3PPSX1RRl9R5wSRGLB +2wBASQvew6CMdNqtDIh8N56BUzHnwh/mHKzBHuwO6hDSHFsUITtLAY7bwGKRq55Z +fUmfAoGBANOYFyoJoDLcl+Jd750lyqfCifcGtkRdmZMtrPXaYnD8ZGme9vz1vsK/ +4iUkV3mi7Z9s1LXMa/tPPfKdVhCM1PXost3/si0+u1Bz5yKqEPXlyy2ltpIVyGu8 +yiy7y75asp8Iii/1cgtwyp9+VeSif8wJ+MHQoGdGxvAQP80R3EjF +-----END RSA PRIVATE KEY----- diff --git a/test/test-download.js b/test/test-download.js index 1615b55fb8..52b881271a 100644 --- a/test/test-download.js +++ b/test/test-download.js @@ -18,7 +18,7 @@ test('download over http', function (t) { server.close() }) - var host = '127.0.0.1' + var host = 'localhost' server.listen(0, host, function () { var port = this.address().port var gyp = { @@ -62,7 +62,7 @@ test('download over https with custom ca', function (t) { throw err }) - var host = '127.0.0.1' + var host = 'localhost' server.listen(8000, host, function () { var port = this.address().port var gyp = { From a991f633d6ecd45085fbaae61068a95c11303c3d Mon Sep 17 00:00:00 2001 From: cclauss Date: Fri, 21 Jun 2019 17:45:44 +0200 Subject: [PATCH 051/551] gyp: fix the remaining Python 3 issues PR-URL: https://github.com/nodejs/node-gyp/pull/1793 Reviewed-By: Rod Vagg Reviewed-By: Ben Noordhuis --- .travis.yml | 3 --- gyp/pylib/gyp/MSVSNew.py | 5 +++++ gyp/pylib/gyp/simple_copy.py | 8 ++++++-- gyp/pylib/gyp/xcodeproj_file.py | 23 +++++++++-------------- gyp/tools/pretty_vcproj.py | 6 ++++++ test/fixtures/test-charmap.py | 11 +++++++++-- 6 files changed, 35 insertions(+), 21 deletions(-) diff --git a/.travis.yml b/.travis.yml index 7607d306fc..30142cf1de 100644 --- a/.travis.yml +++ b/.travis.yml @@ -4,9 +4,6 @@ cache: pip python: - 2.7 - 3.7 -matrix: - allow_failures: - - python: 3.7 install: #- pip install -r requirements.txt - pip install flake8 # pytest # add another testing frameworks later diff --git a/gyp/pylib/gyp/MSVSNew.py b/gyp/pylib/gyp/MSVSNew.py index 91fcb413a6..0ec628cc1f 100644 --- a/gyp/pylib/gyp/MSVSNew.py +++ b/gyp/pylib/gyp/MSVSNew.py @@ -10,6 +10,11 @@ import gyp.common +try: + cmp +except NameError: + def cmp(x, y): + return (x > y) - (x < y) # Initialize random number generator random.seed() diff --git a/gyp/pylib/gyp/simple_copy.py b/gyp/pylib/gyp/simple_copy.py index 463929386e..94a6f17dab 100644 --- a/gyp/pylib/gyp/simple_copy.py +++ b/gyp/pylib/gyp/simple_copy.py @@ -28,8 +28,12 @@ def deepcopy(x): def _deepcopy_atomic(x): return x -for x in (type(None), int, long, float, - bool, str, unicode, type): +try: + types = bool, float, int, str, type, type(None), long, unicode +except NameError: # Python 3 + types = bool, float, int, str, type, type(None) + +for x in types: d[x] = _deepcopy_atomic def _deepcopy_list(x): diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py index 9bd582e477..b0385468c5 100644 --- a/gyp/pylib/gyp/xcodeproj_file.py +++ b/gyp/pylib/gyp/xcodeproj_file.py @@ -138,21 +138,18 @@ """ import gyp.common +import hashlib import posixpath import re import struct import sys -# hashlib is supplied as of Python 2.5 as the replacement interface for sha -# and other secure hashes. In 2.6, sha is deprecated. Import hashlib if -# available, avoiding a deprecation warning under 2.6. Import sha otherwise, -# preserving 2.4 compatibility. try: - import hashlib - _new_sha1 = hashlib.sha1 -except ImportError: - import sha - _new_sha1 = sha.new + basestring, cmp, unicode +except NameError: # Python 3 + basestring = unicode = str + def cmp(x, y): + return (x > y) - (x < y) # See XCObject._EncodeString. This pattern is used to determine when a string @@ -324,8 +321,7 @@ def Copy(self): that._properties[key] = new_value else: that._properties[key] = value - elif isinstance(value, str) or isinstance(value, unicode) or \ - isinstance(value, int): + elif isinstance(value, (basestring, int)): that._properties[key] = value elif isinstance(value, list): if is_strong: @@ -422,7 +418,7 @@ def _HashUpdate(hash, data): hash.update(data) if seed_hash is None: - seed_hash = _new_sha1() + seed_hash = hashlib.sha1() hash = seed_hash.copy() @@ -788,8 +784,7 @@ def UpdateProperties(self, properties, do_copy=False): self._properties[property] = value.Copy() else: self._properties[property] = value - elif isinstance(value, str) or isinstance(value, unicode) or \ - isinstance(value, int): + elif isinstance(value, (basestring, int)): self._properties[property] = value elif isinstance(value, list): if is_strong: diff --git a/gyp/tools/pretty_vcproj.py b/gyp/tools/pretty_vcproj.py index da7eb254cc..e1ec7fee76 100755 --- a/gyp/tools/pretty_vcproj.py +++ b/gyp/tools/pretty_vcproj.py @@ -22,6 +22,12 @@ __author__ = 'nsylvain (Nicolas Sylvain)' +try: + cmp +except NameError: + def cmp(x, y): + return (x > y) - (x < y) + REPLACEMENTS = dict() ARGUMENTS = None diff --git a/test/fixtures/test-charmap.py b/test/fixtures/test-charmap.py index b752f0bbbf..b338f915bc 100644 --- a/test/fixtures/test-charmap.py +++ b/test/fixtures/test-charmap.py @@ -2,14 +2,21 @@ import sys import locale -reload(sys) +try: + reload(sys) +except NameError: # Python 3 + pass def main(): encoding = locale.getdefaultlocale()[1] if not encoding: return False - sys.setdefaultencoding(encoding) + try: + sys.setdefaultencoding(encoding) + except AttributeError: # Python 3 + pass + textmap = { 'cp936': u'\u4e2d\u6587', 'cp1252': u'Lat\u012Bna', From 7dd7f2b2a24a1c748e5bd228b60c7f7500ff4796 Mon Sep 17 00:00:00 2001 From: cclauss Date: Fri, 21 Jun 2019 18:27:43 +0200 Subject: [PATCH 052/551] test: fix Python syntax error in test-adding.js PR-URL: https://github.com/nodejs/node-gyp/pull/1793 Reviewed-By: Rod Vagg Reviewed-By: Ben Noordhuis --- test/test-addon.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/test-addon.js b/test/test-addon.js index 5334ab0b18..ce447bf365 100644 --- a/test/test-addon.js +++ b/test/test-addon.js @@ -28,7 +28,7 @@ function runDuplicateBindings() { } function getEncoding() { - var code = 'import locale;print locale.getdefaultlocale()[1]' + var code = 'import locale;print(locale.getdefaultlocale()[1])' return execFileSync('python', ['-c', code]).toString().trim() } From afaaa29c613d8f521893081afaeacdb3c73b4fd3 Mon Sep 17 00:00:00 2001 From: cclauss Date: Tue, 25 Jun 2019 10:23:38 +0200 Subject: [PATCH 053/551] gyp: remove from __future__ import with_statement This \_\_future\_\_ import is no longer required in Python >= 2.6. https://docs.python.org/2/library/__future__.html PR-URL: https://github.com/nodejs/node-gyp/pull/1799 Reviewed-By: Rod Vagg --- gyp/pylib/gyp/common.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py index b7bae925f4..834a8e6c95 100644 --- a/gyp/pylib/gyp/common.py +++ b/gyp/pylib/gyp/common.py @@ -2,8 +2,6 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. -from __future__ import with_statement - import collections import errno import filecmp From 82f129d6de6a1d94d4c86f7f5d4913ee5a808fe0 Mon Sep 17 00:00:00 2001 From: KiYugadgeter Date: Thu, 1 Nov 2018 00:31:10 +0900 Subject: [PATCH 054/551] gyp: replace optparse to argparse * add_option with add_argument for argparse * change methods methods of RegeneratableOptionParser to match argparse function name PR-URL: https://github.com/nodejs/node-gyp/pull/1591 Reviewed-By: cclauss --- gyp/pylib/gyp/__init__.py | 58 +++++++++++++++++++-------------------- 1 file changed, 29 insertions(+), 29 deletions(-) diff --git a/gyp/pylib/gyp/__init__.py b/gyp/pylib/gyp/__init__.py index d5fa9ecf50..dee834013f 100755 --- a/gyp/pylib/gyp/__init__.py +++ b/gyp/pylib/gyp/__init__.py @@ -8,7 +8,7 @@ import copy import gyp.input -import optparse +import argparse import os.path import re import shlex @@ -244,15 +244,15 @@ def Noop(value): return flags -class RegeneratableOptionParser(optparse.OptionParser): - def __init__(self): +class RegeneratableOptionParser(argparse.ArgumentParser): + def __init__(self, usage): self.__regeneratable_options = {} - optparse.OptionParser.__init__(self) + argparse.ArgumentParser.__init__(self, usage=usage) - def add_option(self, *args, **kw): + def add_argument(self, *args, **kw): """Add an option to the parser. - This accepts the same arguments as OptionParser.add_option, plus the + This accepts the same arguments as ArgumentParser.add_argument, plus the following: regenerate: can be set to False to prevent this option from being included in regeneration. @@ -269,7 +269,7 @@ def add_option(self, *args, **kw): # it as a string. type = kw.get('type') if type == 'path': - kw['type'] = 'string' + kw['type'] = str self.__regeneratable_options[dest] = { 'action': kw.get('action'), @@ -278,50 +278,50 @@ def add_option(self, *args, **kw): 'opt': args[0], } - optparse.OptionParser.add_option(self, *args, **kw) + argparse.ArgumentParser.add_argument(self, *args, **kw) def parse_args(self, *args): - values, args = optparse.OptionParser.parse_args(self, *args) + values, args = argparse.ArgumentParser.parse_known_args(self, *args) values._regeneration_metadata = self.__regeneratable_options return values, args def gyp_main(args): my_name = os.path.basename(sys.argv[0]) + usage = 'usage: %(prog)s [options ...] [build_file ...]' - parser = RegeneratableOptionParser() - usage = 'usage: %s [options ...] [build_file ...]' - parser.set_usage(usage.replace('%s', '%prog')) - parser.add_option('--build', dest='configs', action='append', + + parser = RegeneratableOptionParser(usage=usage.replace('%s', '%(prog)s')) + parser.add_argument('--build', dest='configs', action='append', help='configuration for build after project generation') - parser.add_option('--check', dest='check', action='store_true', + parser.add_argument('--check', dest='check', action='store_true', help='check format of gyp files') - parser.add_option('--config-dir', dest='config_dir', action='store', + parser.add_argument('--config-dir', dest='config_dir', action='store', env_name='GYP_CONFIG_DIR', default=None, help='The location for configuration files like ' 'include.gypi.') - parser.add_option('-d', '--debug', dest='debug', metavar='DEBUGMODE', + parser.add_argument('-d', '--debug', dest='debug', metavar='DEBUGMODE', action='append', default=[], help='turn on a debugging ' 'mode for debugging GYP. Supported modes are "variables", ' '"includes" and "general" or "all" for all of them.') - parser.add_option('-D', dest='defines', action='append', metavar='VAR=VAL', + parser.add_argument('-D', dest='defines', action='append', metavar='VAR=VAL', env_name='GYP_DEFINES', help='sets variable VAR to value VAL') - parser.add_option('--depth', dest='depth', metavar='PATH', type='path', + parser.add_argument('--depth', dest='depth', metavar='PATH', type='path', help='set DEPTH gyp variable to a relative path to PATH') - parser.add_option('-f', '--format', dest='formats', action='append', + parser.add_argument('-f', '--format', dest='formats', action='append', env_name='GYP_GENERATORS', regenerate=False, help='output formats to generate') - parser.add_option('-G', dest='generator_flags', action='append', default=[], + parser.add_argument('-G', dest='generator_flags', action='append', default=[], metavar='FLAG=VAL', env_name='GYP_GENERATOR_FLAGS', help='sets generator flag FLAG to VAL') - parser.add_option('--generator-output', dest='generator_output', + parser.add_argument('--generator-output', dest='generator_output', action='store', default=None, metavar='DIR', type='path', env_name='GYP_GENERATOR_OUTPUT', help='puts generated build files under DIR') - parser.add_option('--ignore-environment', dest='use_environment', + parser.add_argument('--ignore-environment', dest='use_environment', action='store_false', default=True, regenerate=False, help='do not read options from environment variables') - parser.add_option('-I', '--include', dest='includes', action='append', + parser.add_argument('-I', '--include', dest='includes', action='append', metavar='INCLUDE', type='path', help='files to include in all loaded .gyp files') # --no-circular-check disables the check for circular relationships between @@ -331,7 +331,7 @@ def gyp_main(args): # option allows the strict behavior to be used on Macs and the lenient # behavior to be used elsewhere. # TODO(mark): Remove this option when http://crbug.com/35878 is fixed. - parser.add_option('--no-circular-check', dest='circular_check', + parser.add_argument('--no-circular-check', dest='circular_check', action='store_false', default=True, regenerate=False, help="don't check for circular relationships between files") # --no-duplicate-basename-check disables the check for duplicate basenames @@ -340,18 +340,18 @@ def gyp_main(args): # when duplicate basenames are passed into Make generator on Mac. # TODO(yukawa): Remove this option when these legacy generators are # deprecated. - parser.add_option('--no-duplicate-basename-check', + parser.add_argument('--no-duplicate-basename-check', dest='duplicate_basename_check', action='store_false', default=True, regenerate=False, help="don't check for duplicate basenames") - parser.add_option('--no-parallel', action='store_true', default=False, + parser.add_argument('--no-parallel', action='store_true', default=False, help='Disable multiprocessing') - parser.add_option('-S', '--suffix', dest='suffix', default='', + parser.add_argument('-S', '--suffix', dest='suffix', default='', help='suffix to add to generated files') - parser.add_option('--toplevel-dir', dest='toplevel_dir', action='store', + parser.add_argument('--toplevel-dir', dest='toplevel_dir', action='store', default=None, metavar='DIR', type='path', help='directory to use as the root of the source tree') - parser.add_option('-R', '--root-target', dest='root_targets', + parser.add_argument('-R', '--root-target', dest='root_targets', action='append', metavar='TARGET', help='include only TARGET and its deep dependencies') From 7a9a038e9e9639201dbe48829862531a1cbee9bc Mon Sep 17 00:00:00 2001 From: cclauss Date: Tue, 25 Jun 2019 11:20:22 +0200 Subject: [PATCH 055/551] test: add parallel test runs on macOS and Windows Note that the Windows run is in __allow_failures__ mode but that should be fixed in a separate PR. PR-URL: https://github.com/nodejs/node-gyp/pull/1800 Reviewed-By: Rod Vagg --- .travis.yml | 45 +++++++++++++++++++++++++++++++-------------- 1 file changed, 31 insertions(+), 14 deletions(-) diff --git a/.travis.yml b/.travis.yml index 30142cf1de..77c778ad22 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,21 +1,38 @@ dist: xenial language: python cache: pip -python: - - 2.7 - - 3.7 +matrix: + include: + - name: "Python 2.7 on Linux" + python: 2.7 + - name: "Python 3.7 on Linux" + python: 3.7 + - name: "Python 2.7 on macOS" + os: osx + osx_image: xcode10.2 + language: shell # 'language: python' is not yet supported on macOS + before_install: HOMEBREW_NO_AUTO_UPDATE=1 brew install npm + - name: "Python 2.7 on Windows" + os: windows + language: node_js + node_js: 12 # node + env: PATH=/c/Python27:/c/Python27/Scripts:$PATH + before_install: choco install python2 + allow_failures: + - os: windows install: - #- pip install -r requirements.txt - - pip install flake8 # pytest # add another testing frameworks later + #- pip install -r requirements.txt + - pip install flake8 # pytest # add another testing frameworks later before_script: - # stop the build if there are Python syntax errors or undefined names - - flake8 . --count --select=E9,F63,F72,F82 --show-source --statistics - # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide - - flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics - - npm install + - flake8 --version + # stop the build if there are Python syntax errors or undefined names + - flake8 . --count --select=E9,F63,F72,F82 --show-source --statistics + # exit-zero treats all errors as warnings. Two space indentation is OK. The GitHub editor is 127 chars wide + - flake8 . --count --exit-zero --ignore=E111,E114,W503 --max-complexity=10 --max-line-length=127 --statistics + - npm install script: - - npm test - #- pytest --capture=sys # add other tests here + - npm test + #- pytest --capture=sys # add other tests here notifications: - on_success: change - on_failure: change # `always` will be the setting once code changes slow down + on_success: change + on_failure: change # `always` will be the setting once code changes slow down From 49c7f99a74a0e7bc6027b6228ffd1844b4731ba7 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Fri, 21 Jun 2019 13:36:15 +1000 Subject: [PATCH 056/551] v5.0.2: bump version and update changelog --- CHANGELOG.md | 20 ++++++++++++++++++++ package.json | 2 +- 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a687ab49e..8aee70eb5f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,23 @@ +v5.0.2 2019-06-27 +================= + +* [[`2761afbf73`](https://github.com/nodejs/node-gyp/commit/2761afbf73)] - **build,test**: add duplicate symbol test (Gabriel Schulhof) [#1689](https://github.com/nodejs/node-gyp/pull/1689) +* [[`82f129d6de`](https://github.com/nodejs/node-gyp/commit/82f129d6de)] - **gyp**: replace optparse to argparse (KiYugadgeter) [#1591](https://github.com/nodejs/node-gyp/pull/1591) +* [[`afaaa29c61`](https://github.com/nodejs/node-gyp/commit/afaaa29c61)] - **gyp**: remove from \_\_future\_\_ import with\_statement (cclauss) [#1799](https://github.com/nodejs/node-gyp/pull/1799) +* [[`a991f633d6`](https://github.com/nodejs/node-gyp/commit/a991f633d6)] - **gyp**: fix the remaining Python 3 issues (cclauss) [#1793](https://github.com/nodejs/node-gyp/pull/1793) +* [[`f952b08f84`](https://github.com/nodejs/node-gyp/commit/f952b08f84)] - **gyp**: move from \_\_future\_\_ import to the top of the file (cclauss) [#1789](https://github.com/nodejs/node-gyp/pull/1789) +* [[`4f4a677dfa`](https://github.com/nodejs/node-gyp/commit/4f4a677dfa)] - **gyp**: use different default compiler for z/OS (Shuowang (Wayne) Zhang) [#1768](https://github.com/nodejs/node-gyp/pull/1768) +* [[`03683f09d6`](https://github.com/nodejs/node-gyp/commit/03683f09d6)] - **lib**: code de-duplication (Pavel Medvedev) [#965](https://github.com/nodejs/node-gyp/pull/965) +* [[`611bc3c89f`](https://github.com/nodejs/node-gyp/commit/611bc3c89f)] - **lib**: add .json suffix for explicit require (Rod Vagg) [#1787](https://github.com/nodejs/node-gyp/pull/1787) +* [[`d3478d7b0b`](https://github.com/nodejs/node-gyp/commit/d3478d7b0b)] - **meta**: add to .gitignore (Refael Ackermann) [#1573](https://github.com/nodejs/node-gyp/pull/1573) +* [[`7a9a038e9e`](https://github.com/nodejs/node-gyp/commit/7a9a038e9e)] - **test**: add parallel test runs on macOS and Windows (cclauss) [#1800](https://github.com/nodejs/node-gyp/pull/1800) +* [[`7dd7f2b2a2`](https://github.com/nodejs/node-gyp/commit/7dd7f2b2a2)] - **test**: fix Python syntax error in test-adding.js (cclauss) [#1793](https://github.com/nodejs/node-gyp/pull/1793) +* [[`395f843de0`](https://github.com/nodejs/node-gyp/commit/395f843de0)] - **test**: replace self-signed cert with 'localhost' (Rod Vagg) [#1795](https://github.com/nodejs/node-gyp/pull/1795) +* [[`a52c6eb9e8`](https://github.com/nodejs/node-gyp/commit/a52c6eb9e8)] - **test**: migrate from tape to tap (Rod Vagg) [#1795](https://github.com/nodejs/node-gyp/pull/1795) +* [[`ec2eb44a30`](https://github.com/nodejs/node-gyp/commit/ec2eb44a30)] - **test**: use Nan in duplicate\_symbols (Gabriel Schulhof) [#1689](https://github.com/nodejs/node-gyp/pull/1689) +* [[`1597c84aad`](https://github.com/nodejs/node-gyp/commit/1597c84aad)] - **test**: use Travis CI to run tests on every pull request (cclauss) [#1752](https://github.com/nodejs/node-gyp/pull/1752) +* [[`dd9bf929ac`](https://github.com/nodejs/node-gyp/commit/dd9bf929ac)] - **zos**: update compiler options (Shuowang (Wayne) Zhang) [#1768](https://github.com/nodejs/node-gyp/pull/1768) + v5.0.1 2019-06-20 ================= diff --git a/package.json b/package.json index 36a00a48ae..a72018c344 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "5.0.1", + "version": "5.0.2", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From 24109148df76358fc425e3daf438fef7f54713fb Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Fri, 5 Jul 2019 16:36:44 +1000 Subject: [PATCH 057/551] test: downgrade to tap@^12 for continued Node 6 support MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR-URL: https://github.com/nodejs/node-gyp/pull/1808 Reviewed-By: João Reis Reviewed-By: Richard Lau Reviewed-By: cclauss --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index a72018c344..dc416c331d 100644 --- a/package.json +++ b/package.json @@ -43,7 +43,7 @@ "eslint": "^5.0.1", "nan": "^2.0.0", "require-inject": "~1.3.0", - "tap": "~14.2.4" + "tap": "~12.7.0" }, "scripts": { "lint": "eslint bin lib test", From 7e8127068fa6b178970f9b11c9a58681d130be18 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Fri, 5 Jul 2019 17:01:09 +1000 Subject: [PATCH 058/551] test: cover supported node versions with travis PR-URL: https://github.com/nodejs/node-gyp/pull/1809 Reviewed-By: Richard Lau Reviewed-By: cclauss --- .travis.yml | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/.travis.yml b/.travis.yml index 77c778ad22..5703bfa01d 100644 --- a/.travis.yml +++ b/.travis.yml @@ -5,8 +5,6 @@ matrix: include: - name: "Python 2.7 on Linux" python: 2.7 - - name: "Python 3.7 on Linux" - python: 3.7 - name: "Python 2.7 on macOS" os: osx osx_image: xcode10.2 @@ -18,6 +16,18 @@ matrix: node_js: 12 # node env: PATH=/c/Python27:/c/Python27/Scripts:$PATH before_install: choco install python2 + - name: "Node.js 6 & Python 3.7 on Linux" + python: 3.7 + before_install: nvm install 6 + - name: "Node.js 8 & Python 3.7 on Linux" + python: 3.7 + before_install: nvm install 8 + - name: "Node.js 10 & Python 3.7 on Linux" + python: 3.7 + before_install: nvm install 10 + - name: "Node.js 12 & Python 3.7 on Linux" + python: 3.7 + before_install: nvm install 12 allow_failures: - os: windows install: @@ -26,7 +36,7 @@ install: before_script: - flake8 --version # stop the build if there are Python syntax errors or undefined names - - flake8 . --count --select=E9,F63,F72,F82 --show-source --statistics + - flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics # exit-zero treats all errors as warnings. Two space indentation is OK. The GitHub editor is 127 chars wide - flake8 . --count --exit-zero --ignore=E111,E114,W503 --max-complexity=10 --max-line-length=127 --statistics - npm install From e40c99e2830f51c47db7230b5d97da07ea40d6b1 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Sat, 22 Jun 2019 13:10:59 +1000 Subject: [PATCH 059/551] src: implement standard.js linting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In addition: * moved module.exports to the bottom * no single-line if statements * no if statements without a { * const for requires * array declarations get spaces inside [ ] * 'use strict' in all .js files PR-URL: https://github.com/nodejs/node-gyp/pull/1794 Reviewed-By: Richard Lau Reviewed-By: Ben Noordhuis Reviewed-By: João Reis --- .eslintrc.yaml | 6 - .jshintrc | 7 - bin/node-gyp.js | 29 +-- lib/build.js | 56 +++--- lib/clean.js | 12 +- lib/configure.js | 294 ++++++++++++++++-------------- lib/find-node-directory.js | 52 +++--- lib/find-visualstudio.js | 26 ++- lib/install.js | 110 +++++------ lib/list.js | 17 +- lib/node-gyp.js | 147 ++++++++------- lib/process-release.js | 73 +++++--- lib/rebuild.js | 10 +- lib/remove.js | 20 +- lib/util.js | 18 +- package.json | 8 +- test/process-exec-sync.js | 52 +++--- test/simple-proxy.js | 22 ++- test/test-addon.js | 77 ++++---- test/test-configure-python.js | 30 ++- test/test-download.js | 37 ++-- test/test-find-accessible-sync.js | 22 +-- test/test-find-node-directory.js | 42 +++-- test/test-find-python.js | 184 ++++++++++--------- test/test-find-visualstudio.js | 2 +- test/test-install.js | 6 +- test/test-options.js | 14 +- test/test-process-release.js | 41 ++--- 28 files changed, 746 insertions(+), 668 deletions(-) delete mode 100644 .eslintrc.yaml delete mode 100644 .jshintrc diff --git a/.eslintrc.yaml b/.eslintrc.yaml deleted file mode 100644 index f808ce2d9a..0000000000 --- a/.eslintrc.yaml +++ /dev/null @@ -1,6 +0,0 @@ ---- - env: - node: true - es6: true - rules: - no-unused-vars: error diff --git a/.jshintrc b/.jshintrc deleted file mode 100644 index 4b8521dac0..0000000000 --- a/.jshintrc +++ /dev/null @@ -1,7 +0,0 @@ -{ - "asi": true, - "laxcomma": true, - "esversion": 6, - "node": true, - "strict": false -} diff --git a/bin/node-gyp.js b/bin/node-gyp.js index 8c9b6169bb..13f2d399f7 100755 --- a/bin/node-gyp.js +++ b/bin/node-gyp.js @@ -1,17 +1,19 @@ #!/usr/bin/env node +'use strict' + process.title = 'node-gyp' -var envPaths = require('env-paths') -var gyp = require('../') -var log = require('npmlog') -var os = require('os') +const envPaths = require('env-paths') +const gyp = require('../') +const log = require('npmlog') +const os = require('os') /** * Process and execute the selected commands. */ -var prog = gyp() +const prog = gyp() var completed = false prog.parseArgv(process.argv) prog.devDir = prog.opts.devdir @@ -24,8 +26,8 @@ if (prog.devDir) { } else { throw new Error( "node-gyp requires that the user's home directory is specified " + - "in either of the environmental variables HOME or USERPROFILE. " + - "Overide with: --devdir /path/to/.node-gyp") + 'in either of the environmental variables HOME or USERPROFILE. ' + + 'Overide with: --devdir /path/to/.node-gyp') } if (prog.todo.length === 0) { @@ -42,7 +44,6 @@ log.verbose('cli', process.argv) log.info('using', 'node-gyp@%s', prog.version) log.info('using', 'node@%s | %s | %s', process.versions.node, process.platform, process.arch) - /** * Change dir if -C/--directory was passed. */ @@ -84,7 +85,7 @@ function run () { log.error('not ok') return process.exit(1) } - if (command.name == 'list') { + if (command.name === 'list') { var versions = arguments[1] if (versions.length > 0) { versions.forEach(function (version) { @@ -122,7 +123,7 @@ function errorMessage () { var os = require('os') log.error('System', os.type() + ' ' + os.release()) log.error('command', process.argv - .map(JSON.stringify).join(' ')) + .map(JSON.stringify).join(' ')) log.error('cwd', process.cwd()) log.error('node -v', process.version) log.error('node-gyp -v', 'v' + prog.package.version) @@ -130,10 +131,10 @@ function errorMessage () { function issueMessage () { errorMessage() - log.error('', [ 'This is a bug in `node-gyp`.' - , 'Try to update node-gyp and file an Issue if it does not help:' - , ' ' - ].join('\n')) + log.error('', [ 'This is a bug in `node-gyp`.', + 'Try to update node-gyp and file an Issue if it does not help:', + ' ' + ].join('\n')) } // start running the given commands! diff --git a/lib/build.js b/lib/build.js index 783ab9c595..24cfe38de2 100644 --- a/lib/build.js +++ b/lib/build.js @@ -1,12 +1,11 @@ +'use strict' -module.exports = exports = build - -var fs = require('graceful-fs') - , path = require('path') - , glob = require('glob') - , log = require('npmlog') - , which = require('which') - , win = process.platform === 'win32' +const fs = require('graceful-fs') +const path = require('path') +const glob = require('glob') +const log = require('npmlog') +const which = require('which') +const win = process.platform === 'win32' exports.usage = 'Invokes `' + (win ? 'msbuild' : 'make') + '` and builds the module' @@ -17,18 +16,19 @@ function build (gyp, argv, callback) { } else if (process.platform.indexOf('bsd') !== -1) { platformMake = 'gmake' } else if (win && argv.length > 0) { - argv = argv.map(function(target) { + argv = argv.map(function (target) { return '/t:' + target }) } var makeCommand = gyp.opts.make || process.env.MAKE || platformMake - , command = win ? 'msbuild' : makeCommand - , jobs = gyp.opts.jobs || process.env.JOBS - , buildType - , config - , arch - , nodeDir + var command = win ? 'msbuild' : makeCommand + var jobs = gyp.opts.jobs || process.env.JOBS + var buildType + var config + var arch + var nodeDir + var guessedSolution loadConfigGypi() @@ -38,16 +38,17 @@ function build (gyp, argv, callback) { function loadConfigGypi () { var configPath = path.resolve('build', 'config.gypi') + fs.readFile(configPath, 'utf8', function (err, data) { if (err) { - if (err.code == 'ENOENT') { + if (err.code === 'ENOENT') { callback(new Error('You must run `node-gyp configure` first!')) } else { callback(err) } return } - config = JSON.parse(data.replace(/\#.+\n/, '')) + config = JSON.parse(data.replace(/#.+\n/, '')) // get the 'arch', 'buildType', and 'nodeDir' vars from the config buildType = config.target_defaults.default_configuration @@ -79,7 +80,9 @@ function build (gyp, argv, callback) { function findSolutionFile () { glob('build/*.sln', function (err, files) { - if (err) return callback(err) + if (err) { + return callback(err) + } if (files.length === 0) { return callback(new Error('Could not find *.sln file. Did you run "configure"?')) } @@ -124,9 +127,12 @@ function build (gyp, argv, callback) { function doBuild () { // Enable Verbose build var verbose = log.levels[log.level] <= log.levels.verbose + var j + if (!win && verbose) { argv.push('V=1') } + if (win && !verbose) { argv.push('/clp:Verbosity=minimal') } @@ -142,12 +148,12 @@ function build (gyp, argv, callback) { // Since there are many ways to state '32-bit Intel', default to it. // N.B. msbuild's Condition string equality tests are case-insensitive. var archLower = arch.toLowerCase() - var p = archLower === 'x64' ? 'x64' : - (archLower === 'arm' ? 'ARM' : - (archLower === 'arm64' ? 'ARM64' : 'Win32')) + var p = archLower === 'x64' ? 'x64' + : (archLower === 'arm' ? 'ARM' + : (archLower === 'arm64' ? 'ARM64' : 'Win32')) argv.push('/p:Configuration=' + buildType + ';Platform=' + p) if (jobs) { - var j = parseInt(jobs, 10) + j = parseInt(jobs, 10) if (!isNaN(j) && j > 0) { argv.push('/m:' + j) } else if (jobs.toUpperCase() === 'MAX') { @@ -160,7 +166,7 @@ function build (gyp, argv, callback) { argv.push('-C') argv.push('build') if (jobs) { - var j = parseInt(jobs, 10) + j = parseInt(jobs, 10) if (!isNaN(j) && j > 0) { argv.push('--jobs') argv.push(j) @@ -174,7 +180,7 @@ function build (gyp, argv, callback) { if (win) { // did the user specify their own .sln file? var hasSln = argv.some(function (arg) { - return path.extname(arg) == '.sln' + return path.extname(arg) === '.sln' }) if (!hasSln) { argv.unshift(gyp.opts.solution || guessedSolution) @@ -195,3 +201,5 @@ function build (gyp, argv, callback) { callback() } } + +module.exports = build diff --git a/lib/clean.js b/lib/clean.js index 8172e1c3d3..dbfa4dbb99 100644 --- a/lib/clean.js +++ b/lib/clean.js @@ -1,10 +1,7 @@ +'use strict' -module.exports = exports = clean - -exports.usage = 'Removes any generated build files and the "out" dir' - -var rm = require('rimraf') -var log = require('npmlog') +const rm = require('rimraf') +const log = require('npmlog') function clean (gyp, argv, callback) { // Remove the 'build' dir @@ -13,3 +10,6 @@ function clean (gyp, argv, callback) { log.verbose('clean', 'removing "%s" directory', buildDir) rm(buildDir, callback) } + +module.exports = clean +module.exports.usage = 'Removes any generated build files and the "out" dir' diff --git a/lib/configure.js b/lib/configure.js index c3a07d472d..530b9b6357 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -1,35 +1,29 @@ -module.exports = exports = configure -module.exports.test = { - PythonFinder: PythonFinder, - findAccessibleSync: findAccessibleSync, - findPython: findPython, -} - -var fs = require('graceful-fs') - , path = require('path') - , log = require('npmlog') - , os = require('os') - , semver = require('semver') - , mkdirp = require('mkdirp') - , cp = require('child_process') - , extend = require('util')._extend - , processRelease = require('./process-release') - , win = process.platform === 'win32' - , findNodeDirectory = require('./find-node-directory') - , msgFormat = require('util').format - , logWithPrefix = require('./util').logWithPrefix -if (win) +'use strict' + +const fs = require('graceful-fs') +const path = require('path') +const log = require('npmlog') +const os = require('os') +const semver = require('semver') +const mkdirp = require('mkdirp') +const cp = require('child_process') +const extend = require('util')._extend // eslint-disable-line +const processRelease = require('./process-release') +const win = process.platform === 'win32' +const findNodeDirectory = require('./find-node-directory') +const msgFormat = require('util').format +const logWithPrefix = require('./util').logWithPrefix +if (win) { var findVisualStudio = require('./find-visualstudio') - -exports.usage = 'Generates ' + (win ? 'MSVC project files' : 'a Makefile') + ' for the current module' +} function configure (gyp, argv, callback) { var python - , buildDir = path.resolve('build') - , configNames = [ 'config.gypi', 'common.gypi' ] - , configs = [] - , nodeDir - , release = processRelease(argv, gyp, process.version, process.release) + var buildDir = path.resolve('build') + var configNames = [ 'config.gypi', 'common.gypi' ] + var configs = [] + var nodeDir + var release = processRelease(argv, gyp, process.version, process.release) findPython(gyp.opts.python, function (err, found) { if (err) { @@ -67,10 +61,12 @@ function configure (gyp, argv, callback) { // If the tarball option is set, always remove and reinstall the headers // into devdir. Otherwise only install if they're not already there. - gyp.opts.ensure = gyp.opts.tarball ? false : true + gyp.opts.ensure = !gyp.opts.tarball gyp.commands.install([ release.version ], function (err) { - if (err) return callback(err) + if (err) { + return callback(err) + } log.verbose('get node dir', 'target node version installed:', release.versionDir) nodeDir = path.resolve(gyp.devDir, release.versionDir) createBuildDir() @@ -81,7 +77,9 @@ function configure (gyp, argv, callback) { function createBuildDir () { log.verbose('build dir', 'attempting to create "build" dir: %s', buildDir) mkdirp(buildDir, function (err, isNew) { - if (err) return callback(err) + if (err) { + return callback(err) + } log.verbose('build dir', '"build" dir needed to be created?', isNew) if (win) { findVisualStudio(release.semver, gyp.opts.msvs_version, @@ -93,7 +91,9 @@ function configure (gyp, argv, callback) { } function createConfigFile (err, vsInfo) { - if (err) return callback(err) + if (err) { + return callback(err) + } var configFilename = 'config.gypi' var configPath = path.resolve(buildDir, configFilename) @@ -101,14 +101,18 @@ function configure (gyp, argv, callback) { log.verbose('build/' + configFilename, 'creating config file') var config = process.config || {} - , defaults = config.target_defaults - , variables = config.variables + var defaults = config.target_defaults + var variables = config.variables // default "config.variables" - if (!variables) variables = config.variables = {} + if (!variables) { + variables = config.variables = {} + } // default "config.defaults" - if (!defaults) defaults = config.target_defaults = {} + if (!defaults) { + defaults = config.target_defaults = {} + } // don't inherit the "defaults" from node's `process.config` object. // doing so could cause problems in cases where the `node` executable was @@ -123,13 +127,14 @@ function configure (gyp, argv, callback) { if ('debug' in gyp.opts) { defaults.default_configuration = gyp.opts.debug ? 'Debug' : 'Release' } + if (!defaults.default_configuration) { defaults.default_configuration = 'Release' } // set the target_arch variable variables.target_arch = gyp.opts.arch || process.arch || 'ia32' - if (variables.target_arch == 'arm64') { + if (variables.target_arch === 'arm64') { defaults['msvs_configuration_platform'] = 'ARM64' } @@ -146,7 +151,7 @@ function configure (gyp, argv, callback) { if (vsInfo.sdk) { defaults['msvs_windows_target_platform_version'] = vsInfo.sdk } - if (variables.target_arch == 'arm64') { + if (variables.target_arch === 'arm64') { if (vsInfo.versionMajor > 15 || (vsInfo.versionMajor === 15 && vsInfo.versionMajor >= 9)) { defaults['msvs_enable_marmasm'] = 1 @@ -163,15 +168,20 @@ function configure (gyp, argv, callback) { // // $ node-gyp configure --shared-libxml2 Object.keys(gyp.opts).forEach(function (opt) { - if (opt === 'argv') return - if (opt in gyp.configDefs) return + if (opt === 'argv') { + return + } + if (opt in gyp.configDefs) { + return + } variables[opt.replace(/-/g, '_')] = gyp.opts[opt] }) // ensures that any boolean values from `process.config` get stringified function boolsToString (k, v) { - if (typeof v === 'boolean') + if (typeof v === 'boolean') { return String(v) + } return v } @@ -179,21 +189,28 @@ function configure (gyp, argv, callback) { // now write out the config.gypi file to the build/ dir var prefix = '# Do not edit. File was generated by node-gyp\'s "configure" step' - , json = JSON.stringify(config, boolsToString, 2) + + var json = JSON.stringify(config, boolsToString, 2) log.verbose('build/' + configFilename, 'writing out config file: %s', configPath) configs.push(configPath) - fs.writeFile(configPath, [prefix, json, ''].join('\n'), findConfigs) + fs.writeFile(configPath, [ prefix, json, '' ].join('\n'), findConfigs) } function findConfigs (err) { - if (err) return callback(err) + if (err) { + return callback(err) + } + var name = configNames.shift() - if (!name) return runGyp() + if (!name) { + return runGyp() + } var fullPath = path.resolve(name) + log.verbose(name, 'checking for gypi file: %s', fullPath) fs.stat(fullPath, function (err) { if (err) { - if (err.code == 'ENOENT') { + if (err.code === 'ENOENT') { findConfigs() // check next gypi filename } else { callback(err) @@ -207,7 +224,9 @@ function configure (gyp, argv, callback) { } function runGyp (err) { - if (err) return callback(err) + if (err) { + return callback(err) + } if (!~argv.indexOf('-f') && !~argv.indexOf('--format')) { if (win) { @@ -228,63 +247,68 @@ function configure (gyp, argv, callback) { // For AIX and z/OS we need to set up the path to the exports file // which contains the symbols needed for linking. - var node_exp_file = undefined + var nodeExpFile if (process.platform === 'aix' || process.platform === 'os390') { var ext = process.platform === 'aix' ? 'exp' : 'x' - var node_root_dir = findNodeDirectory() - var candidates = undefined + var nodeRootDir = findNodeDirectory() + var candidates + if (process.platform === 'aix') { - candidates = ['include/node/node', - 'out/Release/node', - 'out/Debug/node', - 'node' - ].map(function(file) { - return file + '.' + ext - }) + candidates = [ + 'include/node/node', + 'out/Release/node', + 'out/Debug/node', + 'node' + ].map(function (file) { + return file + '.' + ext + }) } else { - candidates = ['out/Release/obj.target/libnode', - 'out/Debug/obj.target/libnode', - 'lib/libnode' - ].map(function(file) { - return file + '.' + ext - }) + candidates = [ + 'out/Release/obj.target/libnode', + 'out/Debug/obj.target/libnode', + 'lib/libnode' + ].map(function (file) { + return file + '.' + ext + }) } + var logprefix = 'find exports file' - node_exp_file = findAccessibleSync(logprefix, node_root_dir, candidates) - if (node_exp_file !== undefined) { - log.verbose(logprefix, 'Found exports file: %s', node_exp_file) + nodeExpFile = findAccessibleSync(logprefix, nodeRootDir, candidates) + if (nodeExpFile !== undefined) { + log.verbose(logprefix, 'Found exports file: %s', nodeExpFile) } else { - var msg = msgFormat('Could not find node.%s file in %s', ext, node_root_dir) + var msg = msgFormat('Could not find node.%s file in %s', ext, nodeRootDir) log.error(logprefix, 'Could not find exports file') return callback(new Error(msg)) } } // this logic ported from the old `gyp_addon` python file - var gyp_script = path.resolve(__dirname, '..', 'gyp', 'gyp_main.py') - var addon_gypi = path.resolve(__dirname, '..', 'addon.gypi') - var common_gypi = path.resolve(nodeDir, 'include/node/common.gypi') - fs.stat(common_gypi, function (err) { - if (err) - common_gypi = path.resolve(nodeDir, 'common.gypi') - - var output_dir = 'build' + var gypScript = path.resolve(__dirname, '..', 'gyp', 'gyp_main.py') + var addonGypi = path.resolve(__dirname, '..', 'addon.gypi') + var commonGypi = path.resolve(nodeDir, 'include/node/common.gypi') + fs.stat(commonGypi, function (err) { + if (err) { + commonGypi = path.resolve(nodeDir, 'common.gypi') + } + + var outputDir = 'build' if (win) { // Windows expects an absolute path - output_dir = buildDir + outputDir = buildDir } var nodeGypDir = path.resolve(__dirname, '..') var nodeLibFile = path.join(nodeDir, !gyp.opts.nodedir ? '<(target_arch)' : '$(Configuration)', release.name + '.lib') - argv.push('-I', addon_gypi) - argv.push('-I', common_gypi) + argv.push('-I', addonGypi) + argv.push('-I', commonGypi) argv.push('-Dlibrary=shared_library') argv.push('-Dvisibility=default') argv.push('-Dnode_root_dir=' + nodeDir) if (process.platform === 'aix' || process.platform === 'os390') { - argv.push('-Dnode_exp_file=' + node_exp_file) + argv.push('-Dnode_exp_file=' + nodeExpFile) } argv.push('-Dnode_gyp_dir=' + nodeGypDir) argv.push('-Dnode_lib_file=' + nodeLibFile) @@ -295,7 +319,7 @@ function configure (gyp, argv, callback) { argv.push('--no-parallel') // tell gyp to write the Makefile/Solution files into output_dir - argv.push('--generator-output', output_dir) + argv.push('--generator-output', outputDir) // tell make to write its output into the same dir argv.push('-Goutput_dir=.') @@ -304,10 +328,10 @@ function configure (gyp, argv, callback) { argv.unshift('binding.gyp') // execute `gyp` from the current target nodedir - argv.unshift(gyp_script) + argv.unshift(gypScript) // make sure python uses files that came with this particular node package - var pypath = [path.join(__dirname, '..', 'gyp', 'pylib')] + var pypath = [ path.join(__dirname, '..', 'gyp', 'pylib') ] if (process.env.PYTHONPATH) { pypath.push(process.env.PYTHONPATH) } @@ -326,7 +350,6 @@ function configure (gyp, argv, callback) { callback() } } - } /** @@ -336,23 +359,23 @@ function configure (gyp, argv, callback) { */ function findAccessibleSync (logprefix, dir, candidates) { for (var next = 0; next < candidates.length; next++) { - var candidate = path.resolve(dir, candidates[next]) - try { - var fd = fs.openSync(candidate, 'r') - } catch (e) { - // this candidate was not found or not readable, do nothing - log.silly(logprefix, 'Could not open %s: %s', candidate, e.message) - continue - } - fs.closeSync(fd) - log.silly(logprefix, 'Found readable %s', candidate) - return candidate + var candidate = path.resolve(dir, candidates[next]) + try { + var fd = fs.openSync(candidate, 'r') + } catch (e) { + // this candidate was not found or not readable, do nothing + log.silly(logprefix, 'Could not open %s: %s', candidate, e.message) + continue + } + fs.closeSync(fd) + log.silly(logprefix, 'Found readable %s', candidate) + return candidate } return undefined } -function PythonFinder(configPython, callback) { +function PythonFinder (configPython, callback) { this.callback = callback this.configPython = configPython this.errorLog = [] @@ -360,8 +383,8 @@ function PythonFinder(configPython, callback) { PythonFinder.prototype = { log: logWithPrefix(log, 'find Python'), - argsExecutable: ['-c', 'import sys; print(sys.executable);'], - argsVersion: ['-c', 'import sys; print("%s.%s.%s" % sys.version_info[:3]);'], + argsExecutable: [ '-c', 'import sys; print(sys.executable);' ], + argsVersion: [ '-c', 'import sys; print("%s.%s.%s" % sys.version_info[:3]);' ], semverRange: '>=2.6.0 <3.0.0', // These can be overridden for testing: @@ -370,20 +393,19 @@ PythonFinder.prototype = { win: win, pyLauncher: 'py.exe', defaultLocation: path.join(process.env.SystemDrive || 'C:', 'Python27', - 'python.exe'), + 'python.exe'), // Logs a message at verbose level, but also saves it to be displayed later // at error level if an error occurs. This should help diagnose the problem. - addLog: function addLog(message) { + addLog: function addLog (message) { this.log.verbose(message) this.errorLog.push(message) }, - // Find Python by trying a sequence of possibilities. // Ignore errors, keep trying until Python is found. - findPython: function findPython() { - const SKIP=0, FAIL=1 + findPython: function findPython () { + const SKIP = 0; const FAIL = 1 const toCheck = [ { before: () => { @@ -398,7 +420,7 @@ PythonFinder.prototype = { `"${this.configPython}"`) }, check: this.checkCommand, - arg: this.configPython, + arg: this.configPython }, { before: () => { @@ -411,17 +433,17 @@ PythonFinder.prototype = { this.addLog(`- process.env.PYTHON is "${this.env.PYTHON}"`) }, check: this.checkCommand, - arg: this.env.PYTHON, + arg: this.env.PYTHON }, { before: () => { this.addLog('checking if "python2" can be used') }, check: this.checkCommand, - arg: 'python2', + arg: 'python2' }, { before: () => { this.addLog('checking if "python" can be used') }, check: this.checkCommand, - arg: 'python', + arg: 'python' }, { before: () => { @@ -432,7 +454,7 @@ PythonFinder.prototype = { this.addLog( 'checking if the py launcher can be used to find Python 2') }, - check: this.checkPyLauncher, + check: this.checkPyLauncher }, { before: () => { @@ -440,12 +462,12 @@ PythonFinder.prototype = { 'checking if Python 2 is installed in the default location') }, check: this.checkExecPath, - arg: this.defaultLocation, - }, + arg: this.defaultLocation + } ] - function runChecks(err) { - this.log.silly('runChecks: err = %j', err && err.stack || err) + function runChecks (err) { + this.log.silly('runChecks: err = %j', (err && err.stack) || err) const check = toCheck.shift() if (!check) { @@ -470,7 +492,6 @@ PythonFinder.prototype = { runChecks.apply(this) }, - // Check if command is a valid Python to use. // Will exit the Python finder on success. // If on Windows, run in a CMD shell to support BAT/CMD launchers. @@ -513,17 +534,17 @@ PythonFinder.prototype = { checkPyLauncher: function checkPyLauncher (errorCallback) { this.log.verbose( `- executing "${this.pyLauncher}" to get Python 2 executable path`) - this.run(this.pyLauncher, ['-2', ...this.argsExecutable], false, - function (err, execPath) { + this.run(this.pyLauncher, [ '-2', ...this.argsExecutable ], false, + function (err, execPath) { // Possible outcomes: same as checkCommand - if (err) { - this.addLog( - `- "${this.pyLauncher}" is not in PATH or produced an error`) - return errorCallback(err) - } - this.addLog(`- executable path is "${execPath}"`) - this.checkExecPath(execPath, errorCallback) - }.bind(this)) + if (err) { + this.addLog( + `- "${this.pyLauncher}" is not in PATH or produced an error`) + return errorCallback(err) + } + this.addLog(`- executable path is "${execPath}"`) + this.checkExecPath(execPath, errorCallback) + }.bind(this)) }, // Check if a Python executable is the correct version to use. @@ -565,7 +586,7 @@ PythonFinder.prototype = { }, // Run an executable or shell command, trimming the output. - run: function run(exec, args, shell, callback) { + run: function run (exec, args, shell, callback) { var env = extend({}, this.env) env.TERM = 'dumb' const opts = { env: env, shell: shell } @@ -580,8 +601,8 @@ PythonFinder.prototype = { return callback(err) } - function execFileCallback(err, stdout, stderr) { - this.log.silly('execFile result: err = %j', err && err.stack || err) + function execFileCallback (err, stdout, stderr) { + this.log.silly('execFile result: err = %j', (err && err.stack) || err) this.log.silly('execFile result: stdout = %j', stdout) this.log.silly('execFile result: stderr = %j', stderr) if (err) { @@ -592,17 +613,16 @@ PythonFinder.prototype = { } }, - - succeed: function succeed(execPath, version) { + succeed: function succeed (execPath, version) { this.log.info(`using Python version ${version} found at "${execPath}"`) process.nextTick(this.callback.bind(null, null, execPath)) }, - fail: function fail() { + fail: function fail () { const errorLog = this.errorLog.join('\n') - const pathExample = this.win ? 'C:\\Path\\To\\python.exe' : - '/path/to/pythonexecutable' + const pathExample = this.win ? 'C:\\Path\\To\\python.exe' + : '/path/to/pythonexecutable' // For Windows 80 col console, use up to the column before the one marked // with X (total 79 chars including logger prefix, 58 chars usable here): // X @@ -618,16 +638,24 @@ PythonFinder.prototype = { ` npm config set python "${pathExample}"`, 'For more information consult the documentation at:', 'https://github.com/nodejs/node-gyp#installation', - '**********************************************************', + '**********************************************************' ].join('\n') this.log.error(`\n${errorLog}\n\n${info}\n`) - process.nextTick(this.callback.bind(null, new Error ( + process.nextTick(this.callback.bind(null, new Error( 'Could not find any Python 2 installation to use'))) - }, + } } function findPython (configPython, callback) { var finder = new PythonFinder(configPython, callback) finder.findPython() } + +module.exports = configure +module.exports.test = { + PythonFinder: PythonFinder, + findAccessibleSync: findAccessibleSync, + findPython: findPython +} +module.exports.usage = 'Generates ' + (win ? 'MSVC project files' : 'a Makefile') + ' for the current module' diff --git a/lib/find-node-directory.js b/lib/find-node-directory.js index 524909581f..0dd781a6cf 100644 --- a/lib/find-node-directory.js +++ b/lib/find-node-directory.js @@ -1,7 +1,9 @@ -var path = require('path') - , log = require('npmlog') +'use strict' -module.exports = function findNodeDirectory(scriptLocation, processObj) { +const path = require('path') +const log = require('npmlog') + +function findNodeDirectory (scriptLocation, processObj) { // set dirname and process if not passed in // this facilitates regression tests if (scriptLocation === undefined) { @@ -12,48 +14,50 @@ module.exports = function findNodeDirectory(scriptLocation, processObj) { } // Have a look to see what is above us, to try and work out where we are - npm_parent_directory = path.join(scriptLocation, '../../../..') - log.verbose('node-gyp root', 'npm_parent_directory is ' - + path.basename(npm_parent_directory)) - node_root_dir = "" + var npmParentDirectory = path.join(scriptLocation, '../../../..') + log.verbose('node-gyp root', 'npm_parent_directory is ' + + path.basename(npmParentDirectory)) + var nodeRootDir = '' log.verbose('node-gyp root', 'Finding node root directory') - if (path.basename(npm_parent_directory) === 'deps') { + if (path.basename(npmParentDirectory) === 'deps') { // We are in a build directory where this script lives in // deps/npm/node_modules/node-gyp/lib - node_root_dir = path.join(npm_parent_directory, '..') - log.verbose('node-gyp root', 'in build directory, root = ' - + node_root_dir) - } else if (path.basename(npm_parent_directory) === 'node_modules') { + nodeRootDir = path.join(npmParentDirectory, '..') + log.verbose('node-gyp root', 'in build directory, root = ' + + nodeRootDir) + } else if (path.basename(npmParentDirectory) === 'node_modules') { // We are in a node install directory where this script lives in // lib/node_modules/npm/node_modules/node-gyp/lib or // node_modules/npm/node_modules/node-gyp/lib depending on the // platform if (processObj.platform === 'win32') { - node_root_dir = path.join(npm_parent_directory, '..') + nodeRootDir = path.join(npmParentDirectory, '..') } else { - node_root_dir = path.join(npm_parent_directory, '../..') + nodeRootDir = path.join(npmParentDirectory, '../..') } - log.verbose('node-gyp root', 'in install directory, root = ' - + node_root_dir) + log.verbose('node-gyp root', 'in install directory, root = ' + + nodeRootDir) } else { // We don't know where we are, try working it out from the location // of the node binary - var node_dir = path.dirname(processObj.execPath) - var directory_up = path.basename(node_dir) - if (directory_up === 'bin') { - node_root_dir = path.join(node_dir, '..') - } else if (directory_up === 'Release' || directory_up === 'Debug') { + var nodeDir = path.dirname(processObj.execPath) + var directoryUp = path.basename(nodeDir) + if (directoryUp === 'bin') { + nodeRootDir = path.join(nodeDir, '..') + } else if (directoryUp === 'Release' || directoryUp === 'Debug') { // If we are a recently built node, and the directory structure // is that of a repository. If we are on Windows then we only need // to go one level up, everything else, two if (processObj.platform === 'win32') { - node_root_dir = path.join(node_dir, '..') + nodeRootDir = path.join(nodeDir, '..') } else { - node_root_dir = path.join(node_dir, '../..') + nodeRootDir = path.join(nodeDir, '../..') } } // Else return the default blank, "". } - return node_root_dir + return nodeRootDir } + +module.exports = findNodeDirectory diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js index bf47f4ce0a..6799b66ce0 100644 --- a/lib/find-visualstudio.js +++ b/lib/find-visualstudio.js @@ -1,8 +1,4 @@ -module.exports = exports = findVisualStudio -module.exports.test = { - VisualStudioFinder: VisualStudioFinder, - findVisualStudio: findVisualStudio -} +'use strict' const log = require('npmlog') const execFile = require('child_process').execFile @@ -132,9 +128,13 @@ VisualStudioFinder.prototype = { var ps = path.join(process.env.SystemRoot, 'System32', 'WindowsPowerShell', 'v1.0', 'powershell.exe') var csFile = path.join(__dirname, 'Find-VisualStudio.cs') - var psArgs = ['-ExecutionPolicy', 'Unrestricted', '-NoProfile', - '-Command', '&{Add-Type -Path \'' + csFile + '\';' + - '[VisualStudioConfiguration.Main]::PrintJson()}'] + var psArgs = [ + '-ExecutionPolicy', + 'Unrestricted', + '-NoProfile', + '-Command', + '&{Add-Type -Path \'' + csFile + '\';' + '[VisualStudioConfiguration.Main]::PrintJson()}' + ] this.log.silly('Running', ps, psArgs) var child = execFile(ps, psArgs, { encoding: 'utf8' }, @@ -188,7 +188,9 @@ VisualStudioFinder.prototype = { // Remove future versions or errors parsing version number vsInfo = vsInfo.filter((info) => { - if (info.versionYear) { return true } + if (info.versionYear) { + return true + } this.addLog(`unknown version "${info.version}" found at "${info.path}"`) return false }) @@ -415,3 +417,9 @@ VisualStudioFinder.prototype = { return true } } + +module.exports = findVisualStudio +module.exports.test = { + VisualStudioFinder: VisualStudioFinder, + findVisualStudio: findVisualStudio +} diff --git a/lib/install.js b/lib/install.js index 06353663db..6ac8519ef5 100644 --- a/lib/install.js +++ b/lib/install.js @@ -1,33 +1,25 @@ -module.exports = exports = function (gyp, argv, callback) { - return install(fs, gyp, argv, callback) -} - -module.exports.test = { - download: download, - install: install, - readCAFile: readCAFile, -} - -exports.usage = 'Install node development files for the specified node version.' - -var fs = require('graceful-fs') - , os = require('os') - , tar = require('tar') - , path = require('path') - , crypto = require('crypto') - , log = require('npmlog') - , semver = require('semver') - , request = require('request') - , mkdir = require('mkdirp') - , processRelease = require('./process-release') - , win = process.platform == 'win32' +'use strict' + +const fs = require('graceful-fs') +const os = require('os') +const tar = require('tar') +const path = require('path') +const crypto = require('crypto') +const log = require('npmlog') +const semver = require('semver') +const request = require('request') +const mkdir = require('mkdirp') +const processRelease = require('./process-release') +const win = process.platform === 'win32' function install (fs, gyp, argv, callback) { var release = processRelease(argv, gyp, process.version, process.release) // ensure no double-callbacks happen function cb (err) { - if (cb.done) return + if (cb.done) { + return + } cb.done = true if (err) { log.warn('install', 'got an error, rolling back install') @@ -76,10 +68,10 @@ function install (fs, gyp, argv, callback) { log.verbose('install', '--ensure was passed, so won\'t reinstall if already installed') fs.stat(devDir, function (err) { if (err) { - if (err.code == 'ENOENT') { + if (err.code === 'ENOENT') { log.verbose('install', 'version not already installed, continuing with install', release.version) go() - } else if (err.code == 'EACCES') { + } else if (err.code === 'EACCES') { eaccesFallback(err) } else { cb(err) @@ -89,7 +81,7 @@ function install (fs, gyp, argv, callback) { log.verbose('install', 'version is already installed, need to check "installVersion"') var installVersionFile = path.resolve(devDir, 'installVersion') fs.readFile(installVersionFile, 'ascii', function (err, ver) { - if (err && err.code != 'ENOENT') { + if (err && err.code !== 'ENOENT') { return cb(err) } var installVersion = parseInt(ver, 10) || 0 @@ -108,7 +100,7 @@ function install (fs, gyp, argv, callback) { go() } - function getContentSha(res, callback) { + function getContentSha (res, callback) { var shasum = crypto.createHash('sha256') res.on('data', function (chunk) { shasum.update(chunk) @@ -123,7 +115,7 @@ function install (fs, gyp, argv, callback) { // first create the dir for the node dev files mkdir(devDir, function (err, created) { if (err) { - if (err.code == 'EACCES') { + if (err.code === 'EACCES') { eaccesFallback(err) } else { cb(err) @@ -138,8 +130,7 @@ function install (fs, gyp, argv, callback) { // now download the node tarball var tarPath = gyp.opts.tarball var badDownload = false - , extractCount = 0 - + var extractCount = 0 var contentShasums = {} var expectShasums = {} @@ -213,7 +204,9 @@ function install (fs, gyp, argv, callback) { // invoked after the tarball has finished being extracted function afterTarball () { - if (badDownload) return + if (badDownload) { + return + } if (extractCount === 0) { return cb(new Error('There was a fatal problem while downloading/extracting the tarball')) } @@ -244,7 +237,9 @@ function install (fs, gyp, argv, callback) { } function deref (err) { - if (err) return cb(err) + if (err) { + return cb(err) + } async-- if (!async) { @@ -262,7 +257,7 @@ function install (fs, gyp, argv, callback) { } } - function downloadShasums(done) { + function downloadShasums (done) { log.verbose('check download content checksum, need to download `SHASUMS256.txt`...') log.verbose('checksum url', release.shasumsUrl) try { @@ -286,7 +281,9 @@ function install (fs, gyp, argv, callback) { var lines = Buffer.concat(chunks).toString().trim().split('\n') lines.forEach(function (line) { var items = line.trim().split(/\s+/) - if (items.length !== 2) return + if (items.length !== 2) { + return + } // 0035d18e2dcf9aad669b1c7c07319e17abfe3762 ./node-v0.11.4.tar.gz var name = items[1].replace(/^\.\//, '') @@ -313,7 +310,9 @@ function install (fs, gyp, argv, callback) { log.verbose(name, 'url', libUrl) mkdir(dir, function (err) { - if (err) return done(err) + if (err) { + return done(err) + } log.verbose('streaming', name, 'to:', targetLibPath) try { @@ -342,9 +341,7 @@ function install (fs, gyp, argv, callback) { }) }) } // downloadNodeLib() - }) // mkdir() - } // go() /** @@ -353,8 +350,8 @@ function install (fs, gyp, argv, callback) { function valid (file) { // header files - var extname = path.extname(file); - return extname === '.h' || extname === '.gypi'; + var extname = path.extname(file) + return extname === '.h' || extname === '.gypi' } /** @@ -368,28 +365,29 @@ function install (fs, gyp, argv, callback) { function eaccesFallback (err) { var noretry = '--node_gyp_internal_noretry' - if (-1 !== argv.indexOf(noretry)) return cb(err) + if (argv.indexOf(noretry) !== -1) { + return cb(err) + } var tmpdir = os.tmpdir() gyp.devDir = path.resolve(tmpdir, '.node-gyp') log.warn('EACCES', 'user "%s" does not have permission to access the dev dir "%s"', os.userInfo().username, devDir) log.warn('EACCES', 'attempting to reinstall using temporary dev dir "%s"', gyp.devDir) - if (process.cwd() == tmpdir) { + if (process.cwd() === tmpdir) { log.verbose('tmpdir == cwd', 'automatically will remove dev files after to save disk space') gyp.todo.push({ name: 'remove', args: argv }) } gyp.commands.install([noretry].concat(argv), cb) } - } function download (gyp, env, url) { log.http('GET', url) var requestOpts = { - uri: url - , headers: { - 'User-Agent': 'node-gyp v' + gyp.version + ' (node ' + process.version + ')' - } + uri: url, + headers: { + 'User-Agent': 'node-gyp v' + gyp.version + ' (node ' + process.version + ')' + } } var cafile = gyp.opts.cafile @@ -398,10 +396,10 @@ function download (gyp, env, url) { } // basic support for a proxy server - var proxyUrl = gyp.opts.proxy - || env.http_proxy - || env.HTTP_PROXY - || env.npm_config_proxy + var proxyUrl = gyp.opts.proxy || + env.http_proxy || + env.HTTP_PROXY || + env.npm_config_proxy if (proxyUrl) { if (/^https?:\/\//i.test(proxyUrl)) { log.verbose('download', 'using proxy url: "%s"', proxyUrl) @@ -426,3 +424,13 @@ function readCAFile (filename) { var re = /(-----BEGIN CERTIFICATE-----[\S\s]*?-----END CERTIFICATE-----)/g return ca.match(re) } + +module.exports = function (gyp, argv, callback) { + return install(fs, gyp, argv, callback) +} +module.exports.test = { + download: download, + install: install, + readCAFile: readCAFile +} +module.exports.usage = 'Install node development files for the specified node version.' diff --git a/lib/list.js b/lib/list.js index 90f6447180..3e5de28851 100644 --- a/lib/list.js +++ b/lib/list.js @@ -1,10 +1,7 @@ +'use strict' -module.exports = exports = list - -exports.usage = 'Prints a listing of the currently installed node development files' - -var fs = require('graceful-fs') - , log = require('npmlog') +const fs = require('graceful-fs') +const log = require('npmlog') function list (gyp, args, callback) { var devDir = gyp.devDir @@ -13,14 +10,18 @@ function list (gyp, args, callback) { fs.readdir(devDir, onreaddir) function onreaddir (err, versions) { - if (err && err.code != 'ENOENT') { + if (err && err.code !== 'ENOENT') { return callback(err) } + if (Array.isArray(versions)) { - versions = versions.filter(function (v) { return v != 'current' }) + versions = versions.filter(function (v) { return v !== 'current' }) } else { versions = [] } callback(null, versions) } } + +module.exports = list +exports.usage = 'Prints a listing of the currently installed node development files' diff --git a/lib/node-gyp.js b/lib/node-gyp.js index 21ed80f9ff..91880f9433 100644 --- a/lib/node-gyp.js +++ b/lib/node-gyp.js @@ -1,27 +1,26 @@ - -module.exports = exports = gyp - -var path = require('path') - , nopt = require('nopt') - , log = require('npmlog') - , child_process = require('child_process') - , EE = require('events').EventEmitter - , inherits = require('util').inherits - , commands = [ - // Module build commands - 'build' - , 'clean' - , 'configure' - , 'rebuild' - // Development Header File management commands - , 'install' - , 'list' - , 'remove' - ] - , aliases = { - 'ls': 'list' - , 'rm': 'remove' - } +'use strict' + +const path = require('path') +const nopt = require('nopt') +const log = require('npmlog') +const childProcess = require('child_process') +const EE = require('events').EventEmitter +const inherits = require('util').inherits +const commands = [ + // Module build commands + 'build', + 'clean', + 'configure', + 'rebuild', + // Development Header File management commands + 'install', + 'list', + 'remove' +] +const aliases = { + 'ls': 'list', + 'rm': 'remove' +} // differentiate node-gyp's logs from npm's log.heading = 'gyp' @@ -58,24 +57,24 @@ proto.package = require('../package.json') */ proto.configDefs = { - help: Boolean // everywhere - , arch: String // 'configure' - , cafile: String // 'install' - , debug: Boolean // 'build' - , directory: String // bin - , make: String // 'build' - , msvs_version: String // 'configure' - , ensure: Boolean // 'install' - , solution: String // 'build' (windows only) - , proxy: String // 'install' - , devdir: String // everywhere - , nodedir: String // 'configure' - , loglevel: String // everywhere - , python: String // 'configure' - , 'dist-url': String // 'install' - , 'tarball': String // 'install' - , jobs: String // 'build' - , thin: String // 'configure' + help: Boolean, // everywhere + arch: String, // 'configure' + cafile: String, // 'install' + debug: Boolean, // 'build' + directory: String, // bin + make: String, // 'build' + msvs_version: String, // 'configure' + ensure: Boolean, // 'install' + solution: String, // 'build' (windows only) + proxy: String, // 'install' + devdir: String, // everywhere + nodedir: String, // 'configure' + loglevel: String, // everywhere + python: String, // 'configure' + 'dist-url': String, // 'install' + 'tarball': String, // 'install' + jobs: String, // 'build' + thin: String // 'configure' } /** @@ -83,13 +82,13 @@ proto.configDefs = { */ proto.shorthands = { - release: '--no-debug' - , C: '--directory' - , debug: '--debug' - , j: '--jobs' - , silly: '--loglevel=silly' - , verbose: '--loglevel=verbose' - , silent: '--loglevel=silent' + release: '--no-debug', + C: '--directory', + debug: '--debug', + j: '--jobs', + silly: '--loglevel=silly', + verbose: '--loglevel=verbose', + silent: '--loglevel=silent' } /** @@ -134,18 +133,22 @@ proto.parseArgv = function parseOpts (argv) { } // support for inheriting config env variables from npm - var npm_config_prefix = 'npm_config_' + var npmConfigPrefix = 'npm_config_' Object.keys(process.env).forEach(function (name) { - if (name.indexOf(npm_config_prefix) !== 0) return + if (name.indexOf(npmConfigPrefix) !== 0) { + return + } var val = process.env[name] - if (name === npm_config_prefix + 'loglevel') { + if (name === npmConfigPrefix + 'loglevel') { log.level = val } else { // add the user-defined options to the config - name = name.substring(npm_config_prefix.length) + name = name.substring(npmConfigPrefix.length) // gyp@741b7f1 enters an infinite loop when it encounters // zero-length options so ensure those don't get through. - if (name) this.opts[name] = val + if (name) { + this.opts[name] = val + } } }, this) @@ -160,11 +163,13 @@ proto.parseArgv = function parseOpts (argv) { */ proto.spawn = function spawn (command, args, opts) { - if (!opts) opts = {} + if (!opts) { + opts = {} + } if (!opts.silent && !opts.stdio) { opts.stdio = [ 0, 1, 2 ] } - var cp = child_process.spawn(command, args, opts) + var cp = childProcess.spawn(command, args, opts) log.info('spawn', command) log.info('spawn args', args) return cp @@ -176,16 +181,16 @@ proto.spawn = function spawn (command, args, opts) { proto.usage = function usage () { var str = [ - '' - , ' Usage: node-gyp [options]' - , '' - , ' where is one of:' - , commands.map(function (c) { - return ' - ' + c + ' - ' + require('./' + c).usage - }).join('\n') - , '' - , 'node-gyp@' + this.version + ' ' + path.resolve(__dirname, '..') - , 'node@' + process.versions.node + '', + ' Usage: node-gyp [options]', + '', + ' where is one of:', + commands.map(function (c) { + return ' - ' + c + ' - ' + require('./' + c).usage + }).join('\n'), + '', + 'node-gyp@' + this.version + ' ' + path.resolve(__dirname, '..'), + 'node@' + process.versions.node ].join('\n') return str } @@ -195,8 +200,10 @@ proto.usage = function usage () { */ Object.defineProperty(proto, 'version', { - get: function () { - return this.package.version - } - , enumerable: true + get: function () { + return this.package.version + }, + enumerable: true }) + +module.exports = exports = gyp diff --git a/lib/process-release.js b/lib/process-release.js index 1be5be97ab..0acab061bd 100644 --- a/lib/process-release.js +++ b/lib/process-release.js @@ -1,30 +1,32 @@ -var semver = require('semver') - , url = require('url') - , path = require('path') - , log = require('npmlog') +'use strict' - // versions where -headers.tar.gz started shipping - , headersTarballRange = '>= 3.0.0 || ~0.12.10 || ~0.10.42' - , bitsre = /\/win-(x86|x64)\// - , bitsreV3 = /\/win-(x86|ia32|x64)\// // io.js v3.x.x shipped with "ia32" but should - // have been "x86" +const semver = require('semver') +const url = require('url') +const path = require('path') +const log = require('npmlog') + +// versions where -headers.tar.gz started shipping +const headersTarballRange = '>= 3.0.0 || ~0.12.10 || ~0.10.42' +const bitsre = /\/win-(x86|x64)\// +const bitsreV3 = /\/win-(x86|ia32|x64)\// // io.js v3.x.x shipped with "ia32" but should +// have been "x86" // Captures all the logic required to determine download URLs, local directory and // file names. Inputs come from command-line switches (--target, --dist-url), // `process.version` and `process.release` where it exists. function processRelease (argv, gyp, defaultVersion, defaultRelease) { var version = (semver.valid(argv[0]) && argv[0]) || gyp.opts.target || defaultVersion - , versionSemver = semver.parse(version) - , overrideDistUrl = gyp.opts['dist-url'] || gyp.opts.disturl - , isDefaultVersion - , isNamedForLegacyIojs - , name - , distBaseUrl - , baseUrl - , libUrl32 - , libUrl64 - , tarballUrl - , canGetHeaders + var versionSemver = semver.parse(version) + var overrideDistUrl = gyp.opts['dist-url'] || gyp.opts.disturl + var isDefaultVersion + var isNamedForLegacyIojs + var name + var distBaseUrl + var baseUrl + var libUrl32 + var libUrl64 + var tarballUrl + var canGetHeaders if (!versionSemver) { // not a valid semver string, nothing we can do @@ -37,8 +39,9 @@ function processRelease (argv, gyp, defaultVersion, defaultRelease) { isDefaultVersion = version === semver.parse(defaultVersion).version // can't use process.release if we're using --target=x.y.z - if (!isDefaultVersion) + if (!isDefaultVersion) { defaultRelease = null + } if (defaultRelease) { // v3 onward, has process.release @@ -56,16 +59,19 @@ function processRelease (argv, gyp, defaultVersion, defaultRelease) { } // check for the nvm.sh standard mirror env variables - if (!overrideDistUrl && process.env.NODEJS_ORG_MIRROR) + if (!overrideDistUrl && process.env.NODEJS_ORG_MIRROR) { overrideDistUrl = process.env.NODEJS_ORG_MIRROR + } - if (overrideDistUrl) + if (overrideDistUrl) { log.verbose('download', 'using dist-url', overrideDistUrl) + } - if (overrideDistUrl) + if (overrideDistUrl) { distBaseUrl = overrideDistUrl.replace(/\/+$/, '') - else + } else { distBaseUrl = 'https://nodejs.org/dist' + } distBaseUrl += '/v' + version + '/' // new style, based on process.release so we have a lot of the data we need @@ -97,8 +103,14 @@ function processRelease (argv, gyp, defaultVersion, defaultRelease) { tarballUrl: tarballUrl, shasumsUrl: url.resolve(baseUrl, 'SHASUMS256.txt'), versionDir: (name !== 'node' ? name + '-' : '') + version, - ia32: { libUrl: libUrl32, libPath: normalizePath(path.relative(url.parse(baseUrl).path, url.parse(libUrl32).path)) }, - x64: { libUrl: libUrl64, libPath: normalizePath(path.relative(url.parse(baseUrl).path, url.parse(libUrl64).path)) }, + ia32: { + libUrl: libUrl32, + libPath: normalizePath(path.relative(url.parse(baseUrl).path, url.parse(libUrl32).path)) + }, + x64: { + libUrl: libUrl64, + libPath: normalizePath(path.relative(url.parse(baseUrl).path, url.parse(libUrl64).path)) + } } } @@ -108,12 +120,13 @@ function normalizePath (p) { function resolveLibUrl (name, defaultUrl, arch, versionMajor) { var base = url.resolve(defaultUrl, './') - , hasLibUrl = bitsre.test(defaultUrl) || (versionMajor === 3 && bitsreV3.test(defaultUrl)) + var hasLibUrl = bitsre.test(defaultUrl) || (versionMajor === 3 && bitsreV3.test(defaultUrl)) if (!hasLibUrl) { // let's assume it's a baseUrl then - if (versionMajor >= 1) - return url.resolve(base, 'win-' + arch +'/' + name + '.lib') + if (versionMajor >= 1) { + return url.resolve(base, 'win-' + arch + '/' + name + '.lib') + } // prior to io.js@1.0.0 32-bit node.lib lives in /, 64-bit lives in /x64/ return url.resolve(base, (arch === 'x64' ? 'x64/' : '') + name + '.lib') } diff --git a/lib/rebuild.js b/lib/rebuild.js index dc8025a7b0..a1c5b27cbe 100644 --- a/lib/rebuild.js +++ b/lib/rebuild.js @@ -1,13 +1,13 @@ - -module.exports = exports = rebuild - -exports.usage = 'Runs "clean", "configure" and "build" all at once' +'use strict' function rebuild (gyp, argv, callback) { gyp.todo.push( - { name: 'clean', args: [] } + { name: 'clean', args: [] } , { name: 'configure', args: argv } , { name: 'build', args: [] } ) process.nextTick(callback) } + +module.exports = rebuild +module.exports.usage = 'Runs "clean", "configure" and "build" all at once' diff --git a/lib/remove.js b/lib/remove.js index 46a1a3d4c2..8c945e5659 100644 --- a/lib/remove.js +++ b/lib/remove.js @@ -1,13 +1,10 @@ +'use strict' -module.exports = exports = remove - -exports.usage = 'Removes the node development files for the specified version' - -var fs = require('fs') - , rm = require('rimraf') - , path = require('path') - , log = require('npmlog') - , semver = require('semver') +const fs = require('fs') +const rm = require('rimraf') +const path = require('path') +const log = require('npmlog') +const semver = require('semver') function remove (gyp, argv, callback) { var devDir = gyp.devDir @@ -33,7 +30,7 @@ function remove (gyp, argv, callback) { // first check if its even installed fs.stat(versionPath, function (err) { if (err) { - if (err.code == 'ENOENT') { + if (err.code === 'ENOENT') { callback(null, 'version was already uninstalled: ' + version) } else { callback(err) @@ -44,3 +41,6 @@ function remove (gyp, argv, callback) { rm(versionPath, callback) }) } + +module.exports = exports = remove +module.exports.usage = 'Removes the node development files for the specified version' diff --git a/lib/util.js b/lib/util.js index ac6a875354..3e23c628e6 100644 --- a/lib/util.js +++ b/lib/util.js @@ -1,23 +1,19 @@ -module.exports = { - logWithPrefix: logWithPrefix, - regGetValue: regGetValue, - regSearchKeys: regSearchKeys -} +'use strict' const log = require('npmlog') const execFile = require('child_process').execFile const path = require('path') function logWithPrefix (log, prefix) { - function setPrefix(logFunction) { - return (...args) => logFunction.apply(null, [prefix, ...args]) + function setPrefix (logFunction) { + return (...args) => logFunction.apply(null, [ prefix, ...args ]) // eslint-disable-line } return { silly: setPrefix(log.silly), verbose: setPrefix(log.verbose), info: setPrefix(log.info), warn: setPrefix(log.warn), - error: setPrefix(log.error), + error: setPrefix(log.error) } } @@ -60,3 +56,9 @@ function regSearchKeys (keys, value, addOpts, cb) { } search() } + +module.exports = { + logWithPrefix: logWithPrefix, + regGetValue: regGetValue, + regSearchKeys: regSearchKeys +} diff --git a/package.json b/package.json index dc416c331d..260ac9d959 100644 --- a/package.json +++ b/package.json @@ -38,16 +38,14 @@ "node": ">= 6.0.0" }, "devDependencies": { - "babel-eslint": "^8.2.5", "bindings": "~1.2.1", - "eslint": "^5.0.1", "nan": "^2.0.0", "require-inject": "~1.3.0", + "standard": "~12.0.1", "tap": "~12.7.0" }, "scripts": { - "lint": "eslint bin lib test", - "test": "npm run lint && tap test/test-*", - "test-ci": "npm run lint && tap -Rtap test/test-*" + "lint": "standard */*.js test/**/*.js", + "test": "npm run lint && tap test/test-*" } } diff --git a/test/process-exec-sync.js b/test/process-exec-sync.js index 13daf308a3..f786484027 100644 --- a/test/process-exec-sync.js +++ b/test/process-exec-sync.js @@ -1,15 +1,17 @@ 'use strict' -var fs = require('graceful-fs') -var child_process = require('child_process') +const fs = require('graceful-fs') +const childProcess = require('child_process') -if (!String.prototype.startsWith) { - String.prototype.startsWith = function(search, pos) { - return this.substr(!pos || pos < 0 ? 0 : +pos, search.length) === search +function startsWith (str, search, pos) { + if (String.prototype.startsWith) { + return str.startsWith(search, pos) } + + return str.substr(!pos || pos < 0 ? 0 : +pos, search.length) === search } -function processExecSync(file, args, options) { +function processExecSync (file, args, options) { var child, error, timeout, tmpdir, command command = makeCommand(file, args) @@ -22,19 +24,19 @@ function processExecSync(file, args, options) { // init timeout timeout = Date.now() + options.timeout // init tmpdir - var os_temp_base = '/tmp' - var os = determine_os() - os_temp_base = '/tmp' + var osTempBase = '/tmp' + var os = determineOS() + osTempBase = '/tmp' if (process.env.TMP) { - os_temp_base = process.env.TMP + osTempBase = process.env.TMP } - if (os_temp_base[os_temp_base.length - 1] !== '/') { - os_temp_base += '/' + if (osTempBase[osTempBase.length - 1] !== '/') { + osTempBase += '/' } - tmpdir = os_temp_base + 'processExecSync.' + Date.now() + Math.random() + tmpdir = osTempBase + 'processExecSync.' + Date.now() + Math.random() fs.mkdirSync(tmpdir) // init command @@ -47,7 +49,7 @@ function processExecSync(file, args, options) { } // init child - child = child_process.exec(command, options) + child = childProcess.exec(command, options) var maxTry = 100000 // increases the test time by 6 seconds on win-2016-node-0.10 var tryCount = 0 @@ -65,7 +67,7 @@ function processExecSync(file, args, options) { } } - ['stdout', 'stderr', 'status'].forEach(function (file) { + [ 'stdout', 'stderr', 'status' ].forEach(function (file) { child[file] = fs.readFileSync(tmpdir + '/' + file, options.encoding) setTimeout(unlinkFile, 500, tmpdir + '/' + file) }) @@ -84,23 +86,23 @@ function processExecSync(file, args, options) { return child.stdout } -function makeCommand(file, args) { +function makeCommand (file, args) { var command, quote command = file if (args.length > 0) { - for(var i in args) { + for (var i in args) { command = command + ' ' if (args[i][0] === '-') { command = command + args[i] } else { if (!quote) { - command = command + '\"' + command = command + '"' quote = true } command = command + args[i] if (quote) { if (args.length === (parseInt(i) + 1)) { - command = command + '\"' + command = command + '"' } } } @@ -109,29 +111,29 @@ function makeCommand(file, args) { return command } -function determine_os() { +function determineOS () { var os = '' var tmpVar = '' if (process.env.OSTYPE) { tmpVar = process.env.OSTYPE - } else if (process.env.OS) { + } else if (process.env.OS) { tmpVar = process.env.OS } else { - //default is linux + // default is linux tmpVar = 'linux' } - if (tmpVar.startsWith('linux')) { + if (startsWith(tmpVar, 'linux')) { os = 'linux' } - if (tmpVar.startsWith('win')) { + if (startsWith(tmpVar, 'win')) { os = 'win' } return os } -function unlinkFile(file) { +function unlinkFile (file) { fs.unlinkSync(file) } diff --git a/test/simple-proxy.js b/test/simple-proxy.js index 8ac737a652..cb0dfcfec7 100644 --- a/test/simple-proxy.js +++ b/test/simple-proxy.js @@ -1,22 +1,26 @@ -var http = require('http') - , https = require('https') - , server = http.createServer(handler) - , port = +process.argv[2] - , prefix = process.argv[3] - , upstream = process.argv[4] - , calls = 0 +'use strict' + +const http = require('http') +const https = require('https') +const server = http.createServer(handler) +const port = +process.argv[2] +const prefix = process.argv[3] +const upstream = process.argv[4] +var calls = 0 server.listen(port) function handler (req, res) { - if (req.url.indexOf(prefix) != 0) + if (req.url.indexOf(prefix) !== 0) { throw new Error('request url [' + req.url + '] does not start with [' + prefix + ']') + } var upstreamUrl = upstream + req.url.substring(prefix.length) https.get(upstreamUrl, function (ures) { ures.on('end', function () { - if (++calls == 2) + if (++calls === 2) { server.close() + } }) ures.pipe(res) }) diff --git a/test/test-addon.js b/test/test-addon.js index ce447bf365..49e30164c3 100644 --- a/test/test-addon.js +++ b/test/test-addon.js @@ -1,42 +1,42 @@ 'use strict' -var test = require('tap').test -var path = require('path') -var fs = require('graceful-fs') -var child_process = require('child_process') -var os = require('os') -var addonPath = path.resolve(__dirname, 'node_modules', 'hello_world') -var nodeGyp = path.resolve(__dirname, '..', 'bin', 'node-gyp.js') -var execFileSync = child_process.execFileSync || require('./process-exec-sync') -var execFile = child_process.execFile - -function runHello(hostProcess) { +const test = require('tap').test +const path = require('path') +const fs = require('graceful-fs') +const childProcess = require('child_process') +const os = require('os') +const addonPath = path.resolve(__dirname, 'node_modules', 'hello_world') +const nodeGyp = path.resolve(__dirname, '..', 'bin', 'node-gyp.js') +const execFileSync = childProcess.execFileSync || require('./process-exec-sync') +const execFile = childProcess.execFile + +function runHello (hostProcess) { if (!hostProcess) { hostProcess = process.execPath } var testCode = "console.log(require('hello_world').hello())" - return execFileSync(hostProcess, ['-e', testCode], { cwd: __dirname }).toString() + return execFileSync(hostProcess, [ '-e', testCode ], { cwd: __dirname }).toString() } -function runDuplicateBindings() { - const hostProcess = process.execPath; +function runDuplicateBindings () { + const hostProcess = process.execPath var testCode = - "console.log((function(bindings) {" + - "return bindings.pointerCheck1(bindings.pointerCheck2());" + + 'console.log((function(bindings) {' + + 'return bindings.pointerCheck1(bindings.pointerCheck2());' + "})(require('duplicate_symbols')))" - return execFileSync(hostProcess, ['-e', testCode], { cwd: __dirname }).toString() + return execFileSync(hostProcess, [ '-e', testCode ], { cwd: __dirname }).toString() } -function getEncoding() { +function getEncoding () { var code = 'import locale;print(locale.getdefaultlocale()[1])' - return execFileSync('python', ['-c', code]).toString().trim() + return execFileSync('python', [ '-c', code ]).toString().trim() } -function checkCharmapValid() { +function checkCharmapValid () { var data try { - data = execFileSync('python', ['fixtures/test-charmap.py'], - { cwd: __dirname }) + data = execFileSync('python', [ 'fixtures/test-charmap.py' ], + { cwd: __dirname }) } catch (err) { return false } @@ -48,10 +48,10 @@ test('build simple addon', function (t) { t.plan(3) // Set the loglevel otherwise the output disappears when run via 'npm test' - var cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose'] + var cmd = [ nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose' ] var proc = execFile(process.execPath, cmd, function (err, stdout, stderr) { var logLines = stderr.toString().trim().split(/\r?\n/) - var lastLine = logLines[logLines.length-1] + var lastLine = logLines[logLines.length - 1] t.strictEqual(err, null) t.strictEqual(lastLine, 'gyp info ok', 'should end in ok') t.strictEqual(runHello().trim(), 'world') @@ -68,7 +68,7 @@ test('make sure addon symbols do not overlap', function (t) { var cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose'] execFile(process.execPath, cmd, function (err, stdout, stderr) { var logLines = stderr.trim().split(/\r?\n/) - var lastLine = logLines[logLines.length-1] + var lastLine = logLines[logLines.length - 1] t.strictEqual(err, null) t.strictEqual(lastLine, 'gyp info ok', 'should end in ok') t.strictEqual(runDuplicateBindings().trim(), 'not equal') @@ -96,14 +96,15 @@ test('build simple addon in path with non-ascii characters', function (t) { t.plan(3) - var data, configPath = path.join(addonPath, 'build', 'config.gypi') + var data + var configPath = path.join(addonPath, 'build', 'config.gypi') try { data = fs.readFileSync(configPath, 'utf8') } catch (err) { t.error(err) return } - var config = JSON.parse(data.replace(/\#.+\n/, '')) + var config = JSON.parse(data.replace(/#.+\n/, '')) var nodeDir = config.variables.nodedir var testNodeDir = path.join(addonPath, testDirName) // Create symbol link to path with non-ascii characters @@ -121,8 +122,14 @@ test('build simple addon in path with non-ascii characters', function (t) { } } - var cmd = [nodeGyp, 'rebuild', '-C', addonPath, - '--loglevel=verbose', '-nodedir=' + testNodeDir] + var cmd = [ + nodeGyp, + 'rebuild', + '-C', + addonPath, + '--loglevel=verbose', + '-nodedir=' + testNodeDir + ] var proc = execFile(process.execPath, cmd, function (err, stdout, stderr) { try { fs.unlink(testNodeDir) @@ -131,7 +138,7 @@ test('build simple addon in path with non-ascii characters', function (t) { } var logLines = stderr.toString().trim().split(/\r?\n/) - var lastLine = logLines[logLines.length-1] + var lastLine = logLines[logLines.length - 1] t.strictEqual(err, null) t.strictEqual(lastLine, 'gyp info ok', 'should end in ok') t.strictEqual(runHello().trim(), 'world') @@ -143,9 +150,9 @@ test('build simple addon in path with non-ascii characters', function (t) { test('addon works with renamed host executable', function (t) { // No `fs.copyFileSync` before node8. if (process.version.substr(1).split('.')[0] < 8) { - t.skip("skipping test for old node version"); - t.end(); - return; + t.skip('skipping test for old node version') + t.end() + return } t.plan(3) @@ -153,10 +160,10 @@ test('addon works with renamed host executable', function (t) { var notNodePath = path.join(os.tmpdir(), 'notnode' + path.extname(process.execPath)) fs.copyFileSync(process.execPath, notNodePath) - var cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose'] + var cmd = [ nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose' ] var proc = execFile(process.execPath, cmd, function (err, stdout, stderr) { var logLines = stderr.toString().trim().split(/\r?\n/) - var lastLine = logLines[logLines.length-1] + var lastLine = logLines[logLines.length - 1] t.strictEqual(err, null) t.strictEqual(lastLine, 'gyp info ok', 'should end in ok') t.strictEqual(runHello(notNodePath).trim(), 'world') diff --git a/test/test-configure-python.js b/test/test-configure-python.js index 895e488ba8..41b8c70427 100644 --- a/test/test-configure-python.js +++ b/test/test-configure-python.js @@ -1,21 +1,21 @@ 'use strict' -var test = require('tap').test -var path = require('path') -var gyp = require('../lib/node-gyp') -var requireInject = require('require-inject') -var configure = requireInject('../lib/configure', { +const test = require('tap').test +const path = require('path') +const gyp = require('../lib/node-gyp') +const requireInject = require('require-inject') +const configure = requireInject('../lib/configure', { 'graceful-fs': { - 'openSync': function () { return 0; }, + 'openSync': function () { return 0 }, 'closeSync': function () { }, 'writeFile': function (file, data, cb) { cb() }, 'stat': function (file, cb) { cb(null, {}) } } }) -var EXPECTED_PYPATH = path.join(__dirname, '..', 'gyp', 'pylib') -var SEPARATOR = process.platform == 'win32' ? ';' : ':' -var SPAWN_RESULT = { on: function () { } } +const EXPECTED_PYPATH = path.join(__dirname, '..', 'gyp', 'pylib') +const SEPARATOR = process.platform === 'win32' ? ';' : ':' +const SPAWN_RESULT = { on: function () { } } require('npmlog').level = 'warn' @@ -42,11 +42,10 @@ test('configure PYTHONPATH with existing env of one dir', function (t) { var prog = gyp() prog.parseArgv([]) prog.spawn = function () { - - t.equal(process.env.PYTHONPATH, [EXPECTED_PYPATH, existingPath].join(SEPARATOR)) + t.equal(process.env.PYTHONPATH, [ EXPECTED_PYPATH, existingPath ].join(SEPARATOR)) var dirs = process.env.PYTHONPATH.split(SEPARATOR) - t.deepEqual(dirs, [EXPECTED_PYPATH, existingPath]) + t.deepEqual(dirs, [ EXPECTED_PYPATH, existingPath ]) return SPAWN_RESULT } @@ -58,17 +57,16 @@ test('configure PYTHONPATH with existing env of multiple dirs', function (t) { var pythonDir1 = path.join('a', 'b') var pythonDir2 = path.join('b', 'c') - var existingPath = [pythonDir1, pythonDir2].join(SEPARATOR) + var existingPath = [ pythonDir1, pythonDir2 ].join(SEPARATOR) process.env.PYTHONPATH = existingPath var prog = gyp() prog.parseArgv([]) prog.spawn = function () { - - t.equal(process.env.PYTHONPATH, [EXPECTED_PYPATH, existingPath].join(SEPARATOR)) + t.equal(process.env.PYTHONPATH, [ EXPECTED_PYPATH, existingPath ].join(SEPARATOR)) var dirs = process.env.PYTHONPATH.split(SEPARATOR) - t.deepEqual(dirs, [EXPECTED_PYPATH, pythonDir1, pythonDir2]) + t.deepEqual(dirs, [ EXPECTED_PYPATH, pythonDir1, pythonDir2 ]) return SPAWN_RESULT } diff --git a/test/test-download.js b/test/test-download.js index 52b881271a..dbffb20246 100644 --- a/test/test-download.js +++ b/test/test-download.js @@ -1,10 +1,11 @@ 'use strict' -var test = require('tap').test -var fs = require('fs') -var http = require('http') -var https = require('https') -var install = require('../lib/install') +const test = require('tap').test +const fs = require('fs') +const path = require('path') +const http = require('http') +const https = require('https') +const install = require('../lib/install') require('npmlog').level = 'warn' @@ -13,7 +14,7 @@ test('download over http', function (t) { var server = http.createServer(function (req, res) { t.strictEqual(req.headers['user-agent'], - 'node-gyp v42 (node ' + process.version + ')') + 'node-gyp v42 (node ' + process.version + ')') res.end('ok') server.close() }) @@ -23,17 +24,17 @@ test('download over http', function (t) { var port = this.address().port var gyp = { opts: {}, - version: '42', + version: '42' } var url = 'http://' + host + ':' + port var req = install.test.download(gyp, {}, url) req.on('response', function (res) { var body = '' res.setEncoding('utf8') - res.on('data', function(data) { + res.on('data', function (data) { body += data }) - res.on('end', function() { + res.on('end', function () { t.strictEqual(body, 'ok') }) }) @@ -43,17 +44,17 @@ test('download over http', function (t) { test('download over https with custom ca', function (t) { t.plan(3) - var cert = fs.readFileSync(__dirname + '/fixtures/server.crt', 'utf8') - var key = fs.readFileSync(__dirname + '/fixtures/server.key', 'utf8') + var cert = fs.readFileSync(path.join(__dirname, 'fixtures/server.crt'), 'utf8') + var key = fs.readFileSync(path.join(__dirname, 'fixtures/server.key'), 'utf8') - var cafile = __dirname + '/fixtures/ca.crt' + var cafile = path.join(__dirname, '/fixtures/ca.crt') var ca = install.test.readCAFile(cafile) t.strictEqual(ca.length, 1) var options = { ca: ca, cert: cert, key: key } var server = https.createServer(options, function (req, res) { t.strictEqual(req.headers['user-agent'], - 'node-gyp v42 (node ' + process.version + ')') + 'node-gyp v42 (node ' + process.version + ')') res.end('ok') server.close() }) @@ -67,17 +68,17 @@ test('download over https with custom ca', function (t) { var port = this.address().port var gyp = { opts: { cafile: cafile }, - version: '42', + version: '42' } var url = 'https://' + host + ':' + port var req = install.test.download(gyp, {}, url) req.on('response', function (res) { var body = '' res.setEncoding('utf8') - res.on('data', function(data) { + res.on('data', function (data) { body += data }) - res.on('end', function() { + res.on('end', function () { t.strictEqual(body, 'ok') }) }) @@ -87,7 +88,7 @@ test('download over https with custom ca', function (t) { test('download with missing cafile', function (t) { t.plan(1) var gyp = { - opts: { cafile: 'no.such.file' }, + opts: { cafile: 'no.such.file' } } try { install.test.download(gyp, {}, 'http://bad/') @@ -97,7 +98,7 @@ test('download with missing cafile', function (t) { }) test('check certificate splitting', function (t) { - var cas = install.test.readCAFile(__dirname + '/fixtures/ca-bundle.crt') + var cas = install.test.readCAFile(path.join(__dirname, 'fixtures/ca-bundle.crt')) t.plan(2) t.strictEqual(cas.length, 2) t.notStrictEqual(cas[0], cas[1]) diff --git a/test/test-find-accessible-sync.js b/test/test-find-accessible-sync.js index 15adec5379..32234f5389 100644 --- a/test/test-find-accessible-sync.js +++ b/test/test-find-accessible-sync.js @@ -1,13 +1,13 @@ 'use strict' -var test = require('tap').test -var path = require('path') -var requireInject = require('require-inject') -var configure = requireInject('../lib/configure', { +const test = require('tap').test +const path = require('path') +const requireInject = require('require-inject') +const configure = requireInject('../lib/configure', { 'graceful-fs': { 'closeSync': function () { return undefined }, 'openSync': function (path) { - if (readableFiles.some(function (f) { return f === path} )) { + if (readableFiles.some(function (f) { return f === path })) { return 0 } else { var error = new Error('ENOENT - not found') @@ -17,11 +17,11 @@ var configure = requireInject('../lib/configure', { } }) -var dir = path.sep + 'testdir' -var readableFile = 'readable_file' -var anotherReadableFile = 'another_readable_file' -var readableFileInDir = 'somedir' + path.sep + readableFile -var readableFiles = [ +const dir = path.sep + 'testdir' +const readableFile = 'readable_file' +const anotherReadableFile = 'another_readable_file' +const readableFileInDir = 'somedir' + path.sep + readableFile +const readableFiles = [ path.resolve(dir, readableFile), path.resolve(dir, anotherReadableFile), path.resolve(dir, readableFileInDir) @@ -59,7 +59,6 @@ test('find accessible - single item array, unreadable', function (t) { t.strictEqual(found, undefined) }) - test('find accessible - multi item array, no matches', function (t) { t.plan(1) @@ -68,7 +67,6 @@ test('find accessible - multi item array, no matches', function (t) { t.strictEqual(found, undefined) }) - test('find accessible - multi item array, single match', function (t) { t.plan(1) diff --git a/test/test-find-node-directory.js b/test/test-find-node-directory.js index 7cc680d1fa..767b6f6b37 100644 --- a/test/test-find-node-directory.js +++ b/test/test-find-node-directory.js @@ -1,8 +1,10 @@ -var test = require('tap').test -var path = require('path') -var findNodeDirectory = require('../lib/find-node-directory') +'use strict' -var platforms = ['darwin', 'freebsd', 'linux', 'sunos', 'win32', 'aix'] +const test = require('tap').test +const path = require('path') +const findNodeDirectory = require('../lib/find-node-directory') + +const platforms = [ 'darwin', 'freebsd', 'linux', 'sunos', 'win32', 'aix' ] // we should find the directory based on the directory // the script is running in and it should match the layout @@ -11,10 +13,10 @@ var platforms = ['darwin', 'freebsd', 'linux', 'sunos', 'win32', 'aix'] test('test find-node-directory - node install', function (t) { t.plan(platforms.length) for (var next = 0; next < platforms.length; next++) { - var processObj = {execPath: '/x/y/bin/node', platform: platforms[next]} + var processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } t.equal( findNodeDirectory('/x/deps/npm/node_modules/node-gyp/lib', processObj), - path.join('/x')) + path.join('/x')) } }) @@ -26,15 +28,15 @@ test('test find-node-directory - node install', function (t) { test('test find-node-directory - node build', function (t) { t.plan(platforms.length) for (var next = 0; next < platforms.length; next++) { - var processObj = {execPath: '/x/y/bin/node', platform: platforms[next]} + var processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } if (platforms[next] === 'win32') { t.equal( findNodeDirectory('/y/node_modules/npm/node_modules/node-gyp/lib', - processObj), path.join('/y')) + processObj), path.join('/y')) } else { t.equal( findNodeDirectory('/y/lib/node_modules/npm/node_modules/node-gyp/lib', - processObj), path.join('/y')) + processObj), path.join('/y')) } } }) @@ -44,7 +46,7 @@ test('test find-node-directory - node build', function (t) { test('test find-node-directory - node in bin directory', function (t) { t.plan(platforms.length) for (var next = 0; next < platforms.length; next++) { - var processObj = {execPath: '/x/y/bin/node', platform: platforms[next]} + var processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } t.equal( findNodeDirectory('/nothere/npm/node_modules/node-gyp/lib', processObj), path.join('/x/y')) @@ -58,15 +60,15 @@ test('test find-node-directory - node in build release dir', function (t) { for (var next = 0; next < platforms.length; next++) { var processObj if (platforms[next] === 'win32') { - processObj = {execPath: '/x/y/Release/node', platform: platforms[next]} + processObj = { execPath: '/x/y/Release/node', platform: platforms[next] } } else { - processObj = {execPath: '/x/y/out/Release/node', - platform: platforms[next]} + processObj = { execPath: '/x/y/out/Release/node', + platform: platforms[next] } } t.equal( findNodeDirectory('/nothere/npm/node_modules/node-gyp/lib', processObj), - path.join('/x/y')) + path.join('/x/y')) } }) @@ -77,14 +79,14 @@ test('test find-node-directory - node in Debug release dir', function (t) { for (var next = 0; next < platforms.length; next++) { var processObj if (platforms[next] === 'win32') { - processObj = {execPath: '/a/b/Debug/node', platform: platforms[next]} + processObj = { execPath: '/a/b/Debug/node', platform: platforms[next] } } else { - processObj = {execPath: '/a/b/out/Debug/node', platform: platforms[next]} + processObj = { execPath: '/a/b/out/Debug/node', platform: platforms[next] } } t.equal( findNodeDirectory('/nothere/npm/node_modules/node-gyp/lib', processObj), - path.join('/a/b')) + path.join('/a/b')) } }) @@ -93,7 +95,7 @@ test('test find-node-directory - node in Debug release dir', function (t) { test('test find-node-directory - not found', function (t) { t.plan(platforms.length) for (var next = 0; next < platforms.length; next++) { - var processObj = {execPath: '/x/y/z/y', platform:next} + var processObj = { execPath: '/x/y/z/y', platform: next } t.equal(findNodeDirectory('/a/b/c/d', processObj), '') } }) @@ -107,9 +109,9 @@ test('test find-node-directory - not found', function (t) { test('test find-node-directory - node install', function (t) { t.plan(platforms.length) for (var next = 0; next < platforms.length; next++) { - var processObj = {execPath: '/x/y/bin/node', platform: platforms[next]} + var processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } t.equal( findNodeDirectory('/x/y/z/a/b/c/deps/npm/node_modules/node-gyp/lib', - processObj), path.join('/x/y/z/a/b/c')) + processObj), path.join('/x/y/z/a/b/c')) } }) diff --git a/test/test-find-python.js b/test/test-find-python.js index fbb35c3d36..23dddc7bfb 100644 --- a/test/test-find-python.js +++ b/test/test-find-python.js @@ -1,9 +1,9 @@ 'use strict' -var test = require('tap').test -var configure = require('../lib/configure') -var execFile = require('child_process').execFile -var PythonFinder = configure.test.PythonFinder +const test = require('tap').test +const configure = require('../lib/configure') +const execFile = require('child_process').execFile +const PythonFinder = configure.test.PythonFinder require('npmlog').level = 'warn' @@ -22,8 +22,8 @@ test('find python', function (t) { }) }) -function poison(object, property) { - function fail() { +function poison (object, property) { + function fail () { console.error(Error(`Property ${property} should not have been accessed.`)) process.abort() } @@ -31,12 +31,14 @@ function poison(object, property) { configurable: false, enumerable: false, get: fail, - set: fail, + set: fail } Object.defineProperty(object, property, descriptor) } -function TestPythonFinder() { PythonFinder.apply(this, arguments) } +function TestPythonFinder () { + PythonFinder.apply(this, arguments) +} TestPythonFinder.prototype = Object.create(PythonFinder.prototype) // Silence npmlog - remove for debugging TestPythonFinder.prototype.log = { @@ -44,28 +46,28 @@ TestPythonFinder.prototype.log = { verbose: () => {}, info: () => {}, warn: () => {}, - error: () => {}, + error: () => {} } test('find python - python', function (t) { t.plan(6) var f = new TestPythonFinder('python', done) - f.execFile = function(program, args, opts, cb) { - f.execFile = function(program, args, opts, cb) { + f.execFile = function (program, args, opts, cb) { + f.execFile = function (program, args, opts, cb) { poison(f, 'execFile') t.strictEqual(program, '/path/python') t.ok(/sys\.version_info/.test(args[1])) cb(null, '2.7.15') } t.strictEqual(program, - process.platform === 'win32' ? '"python"' : 'python') + process.platform === 'win32' ? '"python"' : 'python') t.ok(/sys\.executable/.test(args[1])) cb(null, '/path/python') } f.findPython() - function done(err, python) { + function done (err, python) { t.strictEqual(err, null) t.strictEqual(python, '/path/python') } @@ -75,10 +77,10 @@ test('find python - python too old', function (t) { t.plan(2) var f = new TestPythonFinder(null, done) - f.execFile = function(program, args, opts, cb) { - if (/sys\.executable/.test(args[args.length-1])) { + f.execFile = function (program, args, opts, cb) { + if (/sys\.executable/.test(args[args.length - 1])) { cb(null, '/path/python') - } else if (/sys\.version_info/.test(args[args.length-1])) { + } else if (/sys\.version_info/.test(args[args.length - 1])) { cb(null, '2.3.4') } else { t.fail() @@ -86,7 +88,7 @@ test('find python - python too old', function (t) { } f.findPython() - function done(err) { + function done (err) { t.ok(/Could not find any Python/.test(err)) t.ok(/not supported/i.test(f.errorLog)) } @@ -96,10 +98,10 @@ test('find python - python too new', function (t) { t.plan(2) var f = new TestPythonFinder(null, done) - f.execFile = function(program, args, opts, cb) { - if (/sys\.executable/.test(args[args.length-1])) { + f.execFile = function (program, args, opts, cb) { + if (/sys\.executable/.test(args[args.length - 1])) { cb(null, '/path/python') - } else if (/sys\.version_info/.test(args[args.length-1])) { + } else if (/sys\.version_info/.test(args[args.length - 1])) { cb(null, '3.0.0') } else { t.fail() @@ -107,7 +109,7 @@ test('find python - python too new', function (t) { } f.findPython() - function done(err) { + function done (err) { t.ok(/Could not find any Python/.test(err)) t.ok(/not supported/i.test(f.errorLog)) } @@ -117,10 +119,10 @@ test('find python - no python', function (t) { t.plan(2) var f = new TestPythonFinder(null, done) - f.execFile = function(program, args, opts, cb) { - if (/sys\.executable/.test(args[args.length-1])) { + f.execFile = function (program, args, opts, cb) { + if (/sys\.executable/.test(args[args.length - 1])) { cb(new Error('not found')) - } else if (/sys\.version_info/.test(args[args.length-1])) { + } else if (/sys\.version_info/.test(args[args.length - 1])) { cb(new Error('not a Python executable')) } else { t.fail() @@ -128,7 +130,7 @@ test('find python - no python', function (t) { } f.findPython() - function done(err) { + function done (err) { t.ok(/Could not find any Python/.test(err)) t.ok(/not in PATH/.test(f.errorLog)) } @@ -138,14 +140,14 @@ test('find python - no python2', function (t) { t.plan(2) var f = new TestPythonFinder(null, done) - f.execFile = function(program, args, opts, cb) { - if (/sys\.executable/.test(args[args.length-1])) { - if (program == 'python2') { + f.execFile = function (program, args, opts, cb) { + if (/sys\.executable/.test(args[args.length - 1])) { + if (program === 'python2') { cb(new Error('not found')) } else { cb(null, '/path/python') } - } else if (/sys\.version_info/.test(args[args.length-1])) { + } else if (/sys\.version_info/.test(args[args.length - 1])) { cb(null, '2.7.14') } else { t.fail() @@ -153,7 +155,7 @@ test('find python - no python2', function (t) { } f.findPython() - function done(err, python) { + function done (err, python) { t.strictEqual(err, null) t.strictEqual(python, '/path/python') } @@ -166,8 +168,8 @@ test('find python - no python2, no python, unix', function (t) { f.checkPyLauncher = t.fail f.win = false - f.execFile = function(program, args, opts, cb) { - if (/sys\.executable/.test(args[args.length-1])) { + f.execFile = function (program, args, opts, cb) { + if (/sys\.executable/.test(args[args.length - 1])) { cb(new Error('not found')) } else { t.fail() @@ -175,7 +177,7 @@ test('find python - no python2, no python, unix', function (t) { } f.findPython() - function done(err) { + function done (err) { t.ok(/Could not find any Python/.test(err)) t.ok(/not in PATH/.test(f.errorLog)) } @@ -187,15 +189,15 @@ test('find python - no python, use python launcher', function (t) { var f = new TestPythonFinder(null, done) f.win = true - f.execFile = function(program, args, opts, cb) { + f.execFile = function (program, args, opts, cb) { if (program === 'py.exe') { t.notEqual(args.indexOf('-2'), -1) t.notEqual(args.indexOf('-c'), -1) return cb(null, 'Z:\\snake.exe') } - if (/sys\.executable/.test(args[args.length-1])) { + if (/sys\.executable/.test(args[args.length - 1])) { cb(new Error('not found')) - } else if (/sys\.version_info/.test(args[args.length-1])) { + } else if (/sys\.version_info/.test(args[args.length - 1])) { if (program === 'Z:\\snake.exe') { cb(null, '2.7.14') } else { @@ -207,7 +209,7 @@ test('find python - no python, use python launcher', function (t) { } f.findPython() - function done(err, python) { + function done (err, python) { t.strictEqual(err, null) t.strictEqual(python, 'Z:\\snake.exe') } @@ -219,11 +221,11 @@ test('find python - python 3, use python launcher', function (t) { var f = new TestPythonFinder(null, done) f.win = true - f.execFile = function(program, args, opts, cb) { + f.execFile = function (program, args, opts, cb) { if (program === 'py.exe') { - f.execFile = function(program, args, opts, cb) { + f.execFile = function (program, args, opts, cb) { poison(f, 'execFile') - if (/sys\.version_info/.test(args[args.length-1])) { + if (/sys\.version_info/.test(args[args.length - 1])) { cb(null, '2.7.14') } else { t.fail() @@ -233,9 +235,9 @@ test('find python - python 3, use python launcher', function (t) { t.notEqual(args.indexOf('-c'), -1) return cb(null, 'Z:\\snake.exe') } - if (/sys\.executable/.test(args[args.length-1])) { + if (/sys\.executable/.test(args[args.length - 1])) { cb(null, '/path/python') - } else if (/sys\.version_info/.test(args[args.length-1])) { + } else if (/sys\.version_info/.test(args[args.length - 1])) { cb(null, '3.0.0') } else { t.fail() @@ -243,76 +245,76 @@ test('find python - python 3, use python launcher', function (t) { } f.findPython() - function done(err, python) { + function done (err, python) { t.strictEqual(err, null) t.strictEqual(python, 'Z:\\snake.exe') } }) test('find python - python 3, use python launcher, python 2 too old', - function (t) { - t.plan(6) - - var f = new TestPythonFinder(null, done) - f.win = true - - f.execFile = function(program, args, opts, cb) { - if (program === 'py.exe') { - f.execFile = function(program, args, opts, cb) { - if (/sys\.version_info/.test(args[args.length-1])) { - f.execFile = function(program, args, opts, cb) { - if (/sys\.version_info/.test(args[args.length-1])) { - poison(f, 'execFile') - t.strictEqual(program, f.defaultLocation) - cb(new Error('not found')) - } else { - t.fail() + function (t) { + t.plan(6) + + var f = new TestPythonFinder(null, done) + f.win = true + + f.execFile = function (program, args, opts, cb) { + if (program === 'py.exe') { + f.execFile = function (program, args, opts, cb) { + if (/sys\.version_info/.test(args[args.length - 1])) { + f.execFile = function (program, args, opts, cb) { + if (/sys\.version_info/.test(args[args.length - 1])) { + poison(f, 'execFile') + t.strictEqual(program, f.defaultLocation) + cb(new Error('not found')) + } else { + t.fail() + } } + t.strictEqual(program, 'Z:\\snake.exe') + cb(null, '2.3.4') + } else { + t.fail() } - t.strictEqual(program, 'Z:\\snake.exe') - cb(null, '2.3.4') - } else { - t.fail() } + t.notEqual(args.indexOf('-2'), -1) + t.notEqual(args.indexOf('-c'), -1) + return cb(null, 'Z:\\snake.exe') + } + if (/sys\.executable/.test(args[args.length - 1])) { + cb(null, '/path/python') + } else if (/sys\.version_info/.test(args[args.length - 1])) { + cb(null, '3.0.0') + } else { + t.fail() } - t.notEqual(args.indexOf('-2'), -1) - t.notEqual(args.indexOf('-c'), -1) - return cb(null, 'Z:\\snake.exe') - } - if (/sys\.executable/.test(args[args.length-1])) { - cb(null, '/path/python') - } else if (/sys\.version_info/.test(args[args.length-1])) { - cb(null, '3.0.0') - } else { - t.fail() } - } - f.findPython() + f.findPython() - function done(err) { - t.ok(/Could not find any Python/.test(err)) - t.ok(/not supported/i.test(f.errorLog)) - } -}) + function done (err) { + t.ok(/Could not find any Python/.test(err)) + t.ok(/not supported/i.test(f.errorLog)) + } + }) test('find python - no python, no python launcher, good guess', function (t) { t.plan(4) - var re = /C:[\\\/]Python27[\\\/]python[.]exe/ + var re = /C:[\\/]Python27[\\/]python[.]exe/ var f = new TestPythonFinder(null, done) f.win = true - f.execFile = function(program, args, opts, cb) { + f.execFile = function (program, args, opts, cb) { if (program === 'py.exe') { - f.execFile = function(program, args, opts, cb) { + f.execFile = function (program, args, opts, cb) { poison(f, 'execFile') t.ok(re.test(program)) - t.ok(/sys\.version_info/.test(args[args.length-1])) + t.ok(/sys\.version_info/.test(args[args.length - 1])) cb(null, '2.7.14') } return cb(new Error('not found')) } - if (/sys\.executable/.test(args[args.length-1])) { + if (/sys\.executable/.test(args[args.length - 1])) { cb(new Error('not found')) } else { t.fail() @@ -320,7 +322,7 @@ test('find python - no python, no python launcher, good guess', function (t) { } f.findPython() - function done(err, python) { + function done (err, python) { t.strictEqual(err, null) t.ok(re.test(python)) } @@ -332,10 +334,10 @@ test('find python - no python, no python launcher, bad guess', function (t) { var f = new TestPythonFinder(null, done) f.win = true - f.execFile = function(program, args, opts, cb) { - if (/sys\.executable/.test(args[args.length-1])) { + f.execFile = function (program, args, opts, cb) { + if (/sys\.executable/.test(args[args.length - 1])) { cb(new Error('not found')) - } else if (/sys\.version_info/.test(args[args.length-1])) { + } else if (/sys\.version_info/.test(args[args.length - 1])) { cb(new Error('not a Python executable')) } else { t.fail() @@ -343,7 +345,7 @@ test('find python - no python, no python launcher, bad guess', function (t) { } f.findPython() - function done(err) { + function done (err) { t.ok(/Could not find any Python/.test(err)) t.ok(/not in PATH/.test(f.errorLog)) } diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js index cf6349c808..acbe7000da 100644 --- a/test/test-find-visualstudio.js +++ b/test/test-find-visualstudio.js @@ -1,6 +1,6 @@ 'use strict' -var test = require('tap').test +const test = require('tap').test const fs = require('fs') const path = require('path') const findVisualStudio = require('../lib/find-visualstudio') diff --git a/test/test-install.js b/test/test-install.js index 9e44c39bc1..c3317155e0 100644 --- a/test/test-install.js +++ b/test/test-install.js @@ -1,7 +1,7 @@ 'use strict' -var test = require('tap').test -var install = require('../lib/install').test.install +const test = require('tap').test +const install = require('../lib/install').test.install require('npmlog').level = 'error' // we expect a warning @@ -13,7 +13,7 @@ test('EACCES retry once', function (t) { var err = new Error() err.code = 'EACCES' cb(err) - t.ok(true); + t.ok(true) } var gyp = {} diff --git a/test/test-options.js b/test/test-options.js index 51d473308c..252baa2035 100644 --- a/test/test-options.js +++ b/test/test-options.js @@ -1,15 +1,15 @@ -'use strict'; +'use strict' -var test = require('tap').test -var gyp = require('../lib/node-gyp') +const test = require('tap').test +const gyp = require('../lib/node-gyp') test('options in environment', function (t) { t.plan(1) // `npm test` dumps a ton of npm_config_* variables in the environment. Object.keys(process.env) - .filter(function(key) { return /^npm_config_/.test(key) }) - .forEach(function(key) { delete process.env[key] }) + .filter(function (key) { return /^npm_config_/.test(key) }) + .forEach(function (key) { delete process.env[key] }) // Zero-length keys should get filtered out. process.env.npm_config_ = '42' @@ -18,8 +18,8 @@ test('options in environment', function (t) { // Except loglevel. process.env.npm_config_loglevel = 'debug' - var g = gyp(); - g.parseArgv(['rebuild']) // Also sets opts.argv. + var g = gyp() + g.parseArgv(['rebuild']) // Also sets opts.argv. t.deepEqual(Object.keys(g.opts).sort(), ['argv', 'x']) }) diff --git a/test/test-process-release.js b/test/test-process-release.js index 83998af041..e4370e59ee 100644 --- a/test/test-process-release.js +++ b/test/test-process-release.js @@ -1,5 +1,7 @@ -var test = require('tap').test -var processRelease = require('../lib/process-release') +'use strict' + +const test = require('tap').test +const processRelease = require('../lib/process-release') test('test process release - process.version = 0.8.20', function (t) { t.plan(2) @@ -17,7 +19,7 @@ test('test process release - process.version = 0.8.20', function (t) { shasumsUrl: 'https://nodejs.org/dist/v0.8.20/SHASUMS256.txt', versionDir: '0.8.20', ia32: { libUrl: 'https://nodejs.org/dist/v0.8.20/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.8.20/x64/node.lib', libPath: 'x64/node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.8.20/x64/node.lib', libPath: 'x64/node.lib' } }) }) @@ -37,7 +39,7 @@ test('test process release - process.version = 0.10.21', function (t) { shasumsUrl: 'https://nodejs.org/dist/v0.10.21/SHASUMS256.txt', versionDir: '0.10.21', ia32: { libUrl: 'https://nodejs.org/dist/v0.10.21/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.10.21/x64/node.lib', libPath: 'x64/node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.10.21/x64/node.lib', libPath: 'x64/node.lib' } }) }) @@ -58,7 +60,7 @@ test('test process release - process.version = 0.12.9', function (t) { shasumsUrl: 'https://nodejs.org/dist/v0.12.9/SHASUMS256.txt', versionDir: '0.12.9', ia32: { libUrl: 'https://nodejs.org/dist/v0.12.9/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.12.9/x64/node.lib', libPath: 'x64/node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.12.9/x64/node.lib', libPath: 'x64/node.lib' } }) }) @@ -79,7 +81,7 @@ test('test process release - process.version = 0.10.41', function (t) { shasumsUrl: 'https://nodejs.org/dist/v0.10.41/SHASUMS256.txt', versionDir: '0.10.41', ia32: { libUrl: 'https://nodejs.org/dist/v0.10.41/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.10.41/x64/node.lib', libPath: 'x64/node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.10.41/x64/node.lib', libPath: 'x64/node.lib' } }) }) @@ -100,7 +102,7 @@ test('test process release - process.release ~ node@0.10.42', function (t) { shasumsUrl: 'https://nodejs.org/dist/v0.10.42/SHASUMS256.txt', versionDir: '0.10.42', ia32: { libUrl: 'https://nodejs.org/dist/v0.10.42/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.10.42/x64/node.lib', libPath: 'x64/node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.10.42/x64/node.lib', libPath: 'x64/node.lib' } }) }) @@ -121,7 +123,7 @@ test('test process release - process.release ~ node@0.12.10', function (t) { shasumsUrl: 'https://nodejs.org/dist/v0.12.10/SHASUMS256.txt', versionDir: '0.12.10', ia32: { libUrl: 'https://nodejs.org/dist/v0.12.10/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.12.10/x64/node.lib', libPath: 'x64/node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.12.10/x64/node.lib', libPath: 'x64/node.lib' } }) }) @@ -144,7 +146,7 @@ test('test process release - process.release ~ node@4.1.23', function (t) { shasumsUrl: 'https://nodejs.org/dist/v4.1.23/SHASUMS256.txt', versionDir: '4.1.23', ia32: { libUrl: 'https://nodejs.org/dist/v4.1.23/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' } }) }) @@ -167,7 +169,7 @@ test('test process release - process.release ~ node@4.1.23 / corp build', functi shasumsUrl: 'https://some.custom.location/SHASUMS256.txt', versionDir: '4.1.23', ia32: { libUrl: 'https://some.custom.location/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://some.custom.location/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + x64: { libUrl: 'https://some.custom.location/win-x64/node.lib', libPath: 'win-x64/node.lib' } }) }) @@ -190,7 +192,7 @@ test('test process release - process.release ~ node@4.1.23 --target=0.10.40', fu shasumsUrl: 'https://nodejs.org/dist/v0.10.40/SHASUMS256.txt', versionDir: '0.10.40', ia32: { libUrl: 'https://nodejs.org/dist/v0.10.40/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.10.40/x64/node.lib', libPath: 'x64/node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.10.40/x64/node.lib', libPath: 'x64/node.lib' } }) }) @@ -213,7 +215,7 @@ test('test process release - process.release ~ node@4.1.23 --dist-url=https://fo shasumsUrl: 'https://foo.bar/baz/v4.1.23/SHASUMS256.txt', versionDir: '4.1.23', ia32: { libUrl: 'https://foo.bar/baz/v4.1.23/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://foo.bar/baz/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + x64: { libUrl: 'https://foo.bar/baz/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' } }) }) @@ -236,11 +238,10 @@ test('test process release - process.release ~ frankenstein@4.1.23', function (t shasumsUrl: 'https://frankensteinjs.org/dist/v4.1.23/SHASUMS256.txt', versionDir: 'frankenstein-4.1.23', ia32: { libUrl: 'https://frankensteinjs.org/dist/v4.1.23/win-x86/frankenstein.lib', libPath: 'win-x86/frankenstein.lib' }, - x64: { libUrl: 'https://frankensteinjs.org/dist/v4.1.23/win-x64/frankenstein.lib', libPath: 'win-x64/frankenstein.lib' }, + x64: { libUrl: 'https://frankensteinjs.org/dist/v4.1.23/win-x64/frankenstein.lib', libPath: 'win-x64/frankenstein.lib' } }) }) - test('test process release - process.release ~ frankenstein@4.1.23 --dist-url=http://foo.bar/baz/', function (t) { t.plan(2) @@ -260,7 +261,7 @@ test('test process release - process.release ~ frankenstein@4.1.23 --dist-url=ht shasumsUrl: 'http://foo.bar/baz/v4.1.23/SHASUMS256.txt', versionDir: 'frankenstein-4.1.23', ia32: { libUrl: 'http://foo.bar/baz/v4.1.23/win-x86/frankenstein.lib', libPath: 'win-x86/frankenstein.lib' }, - x64: { libUrl: 'http://foo.bar/baz/v4.1.23/win-x64/frankenstein.lib', libPath: 'win-x64/frankenstein.lib' }, + x64: { libUrl: 'http://foo.bar/baz/v4.1.23/win-x64/frankenstein.lib', libPath: 'win-x64/frankenstein.lib' } }) }) @@ -283,11 +284,10 @@ test('test process release - process.release ~ node@4.0.0-rc.4', function (t) { shasumsUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/SHASUMS256.txt', versionDir: '4.0.0-rc.4', ia32: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' } }) }) - test('test process release - process.release ~ node@4.0.0-rc.4 passed as argv[0]', function (t) { t.plan(2) @@ -309,11 +309,10 @@ test('test process release - process.release ~ node@4.0.0-rc.4 passed as argv[0] shasumsUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/SHASUMS256.txt', versionDir: '4.0.0-rc.4', ia32: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' } }) }) - test('test process release - process.release ~ node@4.0.0-rc.4 - bogus string passed as argv[0]', function (t) { t.plan(2) @@ -335,7 +334,7 @@ test('test process release - process.release ~ node@4.0.0-rc.4 - bogus string pa shasumsUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/SHASUMS256.txt', versionDir: '4.0.0-rc.4', ia32: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' } }) }) @@ -360,7 +359,7 @@ test('test process release - NODEJS_ORG_MIRROR', function (t) { shasumsUrl: 'http://foo.bar/v4.1.23/SHASUMS256.txt', versionDir: '4.1.23', ia32: { libUrl: 'http://foo.bar/v4.1.23/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'http://foo.bar/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + x64: { libUrl: 'http://foo.bar/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' } }) delete process.env.NODEJS_ORG_MIRROR From 656117cc4a5c8f3342641413e2baa024f7d4e819 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Thu, 4 Jul 2019 12:07:24 +0100 Subject: [PATCH 060/551] win: make VS path match case-insensitive Fixes: https://github.com/nodejs/node-gyp/issues/1805 PR-URL: https://github.com/nodejs/node-gyp/pull/1806 Reviewed-By: Richard Lau Reviewed-By: Rod Vagg Reviewed-By: Bartosz Sosnowski --- lib/find-visualstudio.js | 6 ++++-- test/test-find-visualstudio.js | 33 ++++++++++++++++++++++++++++++++- 2 files changed, 36 insertions(+), 3 deletions(-) diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js index 6799b66ce0..b2c00e6835 100644 --- a/lib/find-visualstudio.js +++ b/lib/find-visualstudio.js @@ -405,11 +405,13 @@ VisualStudioFinder.prototype = { this.addLog('- msvs_version does not match this version') return false } - if (this.configPath && this.configPath !== vsPath) { + if (this.configPath && + path.relative(this.configPath, vsPath) !== '') { this.addLog('- msvs_version does not point to this installation') return false } - if (this.envVcInstallDir && this.envVcInstallDir !== vsPath) { + if (this.envVcInstallDir && + path.relative(this.envVcInstallDir, vsPath) !== '') { this.addLog('- does not match this Visual Studio Command Prompt') return false } diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js index acbe7000da..b00adf0227 100644 --- a/test/test-find-visualstudio.js +++ b/test/test-find-visualstudio.js @@ -525,7 +525,7 @@ test('look for VS2019 by version number', function (t) { finder.findVisualStudio() }) -test('look for VS2017 by installation path', function (t) { +test('look for VS2019 by installation path', function (t) { t.plan(2) const finder = new TestVisualStudioFinder(semverV1, @@ -540,6 +540,21 @@ test('look for VS2017 by installation path', function (t) { finder.findVisualStudio() }) +test('msvs_version match should be case insensitive', function (t) { + t.plan(2) + + const finder = new TestVisualStudioFinder(semverV1, + 'c:\\program files (x86)\\microsoft visual studio\\2019\\BUILDTOOLS', + (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info.path, + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools') + }) + + allVsVersions(t, finder) + finder.findVisualStudio() +}) + test('latest version should be found by default', function (t) { t.plan(2) @@ -568,6 +583,22 @@ test('run on a usable VS Command Prompt', function (t) { finder.findVisualStudio() }) +test('VCINSTALLDIR match should be case insensitive', function (t) { + t.plan(2) + + process.env.VCINSTALLDIR = + 'c:\\program files (x86)\\microsoft visual studio\\2019\\BUILDTOOLS\\VC' + + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info.path, + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools') + }) + + allVsVersions(t, finder) + finder.findVisualStudio() +}) + test('run on a unusable VS Command Prompt', function (t) { t.plan(2) From 7fd924079f57fa690abae92feb0081f1907eab41 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Sun, 7 Jul 2019 20:29:10 +0100 Subject: [PATCH 061/551] test: increase tap timeout test-addon.js includes compiling code, making the default 30 second timeout not suitable. This increases the timeout for all platforms, which is a potential problem everywhere, fixing the timeout that happens on Windows. Fixes: https://github.com/nodejs/node-gyp/issues/1801 PR-URL: https://github.com/nodejs/node-gyp/pull/1812 Reviewed-By: Christian Clauss Reviewed-By: Bartosz Sosnowski Reviewed-By: Richard Lau --- .travis.yml | 2 -- package.json | 2 +- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/.travis.yml b/.travis.yml index 5703bfa01d..895690dc8d 100644 --- a/.travis.yml +++ b/.travis.yml @@ -28,8 +28,6 @@ matrix: - name: "Node.js 12 & Python 3.7 on Linux" python: 3.7 before_install: nvm install 12 - allow_failures: - - os: windows install: #- pip install -r requirements.txt - pip install flake8 # pytest # add another testing frameworks later diff --git a/package.json b/package.json index 260ac9d959..15f07459eb 100644 --- a/package.json +++ b/package.json @@ -46,6 +46,6 @@ }, "scripts": { "lint": "standard */*.js test/**/*.js", - "test": "npm run lint && tap test/test-*" + "test": "npm run lint && tap --timeout=120 test/test-*" } } From bb92c761a98e30c29b973b8d6e48e98763656fc6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Sun, 7 Jul 2019 20:57:24 +0100 Subject: [PATCH 062/551] test: add Node.js 6 on Windows to Travis CI Test the oldest supported Node version on Windows. PR-URL: https://github.com/nodejs/node-gyp/pull/1812 Reviewed-By: Christian Clauss Reviewed-By: Bartosz Sosnowski Reviewed-By: Richard Lau --- .travis.yml | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 895690dc8d..54a13a9382 100644 --- a/.travis.yml +++ b/.travis.yml @@ -10,7 +10,13 @@ matrix: osx_image: xcode10.2 language: shell # 'language: python' is not yet supported on macOS before_install: HOMEBREW_NO_AUTO_UPDATE=1 brew install npm - - name: "Python 2.7 on Windows" + - name: "Node.js 6 & Python 2.7 on Windows" + os: windows + language: node_js + node_js: 6 # node + env: PATH=/c/Python27:/c/Python27/Scripts:$PATH + before_install: choco install python2 + - name: "Node.js 12 & Python 2.7 on Windows" os: windows language: node_js node_js: 12 # node From 7e7fce3fedaa18e5df025e3e63de2d5e61a40b31 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Mon, 8 Jul 2019 16:37:29 +0100 Subject: [PATCH 063/551] python: move Python detection to its own file PR-URL: https://github.com/nodejs/node-gyp/pull/1815 Reviewed-By: Christian Clauss Reviewed-By: Richard Lau --- lib/configure.js | 286 +------------------------------------- lib/find-python.js | 292 +++++++++++++++++++++++++++++++++++++++ test/test-find-python.js | 6 +- 3 files changed, 297 insertions(+), 287 deletions(-) create mode 100644 lib/find-python.js diff --git a/lib/configure.js b/lib/configure.js index 530b9b6357..267e7587f5 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -4,15 +4,12 @@ const fs = require('graceful-fs') const path = require('path') const log = require('npmlog') const os = require('os') -const semver = require('semver') const mkdirp = require('mkdirp') -const cp = require('child_process') -const extend = require('util')._extend // eslint-disable-line const processRelease = require('./process-release') const win = process.platform === 'win32' const findNodeDirectory = require('./find-node-directory') const msgFormat = require('util').format -const logWithPrefix = require('./util').logWithPrefix +var findPython = require('./find-python') if (win) { var findVisualStudio = require('./find-visualstudio') } @@ -375,287 +372,8 @@ function findAccessibleSync (logprefix, dir, candidates) { return undefined } -function PythonFinder (configPython, callback) { - this.callback = callback - this.configPython = configPython - this.errorLog = [] -} - -PythonFinder.prototype = { - log: logWithPrefix(log, 'find Python'), - argsExecutable: [ '-c', 'import sys; print(sys.executable);' ], - argsVersion: [ '-c', 'import sys; print("%s.%s.%s" % sys.version_info[:3]);' ], - semverRange: '>=2.6.0 <3.0.0', - - // These can be overridden for testing: - execFile: cp.execFile, - env: process.env, - win: win, - pyLauncher: 'py.exe', - defaultLocation: path.join(process.env.SystemDrive || 'C:', 'Python27', - 'python.exe'), - - // Logs a message at verbose level, but also saves it to be displayed later - // at error level if an error occurs. This should help diagnose the problem. - addLog: function addLog (message) { - this.log.verbose(message) - this.errorLog.push(message) - }, - - // Find Python by trying a sequence of possibilities. - // Ignore errors, keep trying until Python is found. - findPython: function findPython () { - const SKIP = 0; const FAIL = 1 - const toCheck = [ - { - before: () => { - if (!this.configPython) { - this.addLog( - 'Python is not set from command line or npm configuration') - return SKIP - } - this.addLog('checking Python explicitly set from command line or ' + - 'npm configuration') - this.addLog('- "--python=" or "npm config get python" is ' + - `"${this.configPython}"`) - }, - check: this.checkCommand, - arg: this.configPython - }, - { - before: () => { - if (!this.env.PYTHON) { - this.addLog('Python is not set from environment variable PYTHON') - return SKIP - } - this.addLog( - 'checking Python explicitly set from environment variable PYTHON') - this.addLog(`- process.env.PYTHON is "${this.env.PYTHON}"`) - }, - check: this.checkCommand, - arg: this.env.PYTHON - }, - { - before: () => { this.addLog('checking if "python2" can be used') }, - check: this.checkCommand, - arg: 'python2' - }, - { - before: () => { this.addLog('checking if "python" can be used') }, - check: this.checkCommand, - arg: 'python' - }, - { - before: () => { - if (!this.win) { - // Everything after this is Windows specific - return FAIL - } - this.addLog( - 'checking if the py launcher can be used to find Python 2') - }, - check: this.checkPyLauncher - }, - { - before: () => { - this.addLog( - 'checking if Python 2 is installed in the default location') - }, - check: this.checkExecPath, - arg: this.defaultLocation - } - ] - - function runChecks (err) { - this.log.silly('runChecks: err = %j', (err && err.stack) || err) - - const check = toCheck.shift() - if (!check) { - return this.fail() - } - - const before = check.before.apply(this) - if (before === SKIP) { - return runChecks.apply(this) - } - if (before === FAIL) { - return this.fail() - } - - const args = [ runChecks.bind(this) ] - if (check.arg) { - args.unshift(check.arg) - } - check.check.apply(this, args) - } - - runChecks.apply(this) - }, - - // Check if command is a valid Python to use. - // Will exit the Python finder on success. - // If on Windows, run in a CMD shell to support BAT/CMD launchers. - checkCommand: function checkCommand (command, errorCallback) { - var exec = command - var args = this.argsExecutable - var shell = false - if (this.win) { - // Arguments have to be manually quoted - exec = `"${exec}"` - args = args.map(a => `"${a}"`) - shell = true - } - - this.log.verbose(`- executing "${command}" to get executable path`) - this.run(exec, args, shell, function (err, execPath) { - // Possible outcomes: - // - Error: not in PATH, not executable or execution fails - // - Gibberish: the next command to check version will fail - // - Absolute path to executable - if (err) { - this.addLog(`- "${command}" is not in PATH or produced an error`) - return errorCallback(err) - } - this.addLog(`- executable path is "${execPath}"`) - this.checkExecPath(execPath, errorCallback) - }.bind(this)) - }, - - // Check if the py launcher can find a valid Python to use. - // Will exit the Python finder on success. - // Distributions of Python on Windows by default install with the "py.exe" - // Python launcher which is more likely to exist than the Python executable - // being in the $PATH. - // Because the Python launcher supports all versions of Python, we have to - // explicitly request a Python 2 version. This is done by supplying "-2" as - // the first command line argument. Since "py.exe -2" would be an invalid - // executable for "execFile", we have to use the launcher to figure out - // where the actual "python.exe" executable is located. - checkPyLauncher: function checkPyLauncher (errorCallback) { - this.log.verbose( - `- executing "${this.pyLauncher}" to get Python 2 executable path`) - this.run(this.pyLauncher, [ '-2', ...this.argsExecutable ], false, - function (err, execPath) { - // Possible outcomes: same as checkCommand - if (err) { - this.addLog( - `- "${this.pyLauncher}" is not in PATH or produced an error`) - return errorCallback(err) - } - this.addLog(`- executable path is "${execPath}"`) - this.checkExecPath(execPath, errorCallback) - }.bind(this)) - }, - - // Check if a Python executable is the correct version to use. - // Will exit the Python finder on success. - checkExecPath: function checkExecPath (execPath, errorCallback) { - this.log.verbose(`- executing "${execPath}" to get version`) - this.run(execPath, this.argsVersion, false, function (err, version) { - // Possible outcomes: - // - Error: executable can not be run (likely meaning the command wasn't - // a Python executable and the previous command produced gibberish) - // - Gibberish: somehow the last command produced an executable path, - // this will fail when verifying the version - // - Version of the Python executable - if (err) { - this.addLog(`- "${execPath}" could not be run`) - return errorCallback(err) - } - this.addLog(`- version is "${version}"`) - - const range = semver.Range(this.semverRange) - var valid = false - try { - valid = range.test(version) - } catch (err) { - this.log.silly('range.test() threw:\n%s', err.stack) - this.addLog(`- "${execPath}" does not have a valid version`) - this.addLog('- is it a Python executable?') - return errorCallback(err) - } - - if (!valid) { - this.addLog(`- version is ${version} - should be ${this.semverRange}`) - this.addLog('- THIS VERSION OF PYTHON IS NOT SUPPORTED') - return errorCallback(new Error( - `Found unsupported Python version ${version}`)) - } - this.succeed(execPath, version) - }.bind(this)) - }, - - // Run an executable or shell command, trimming the output. - run: function run (exec, args, shell, callback) { - var env = extend({}, this.env) - env.TERM = 'dumb' - const opts = { env: env, shell: shell } - - this.log.silly('execFile: exec = %j', exec) - this.log.silly('execFile: args = %j', args) - this.log.silly('execFile: opts = %j', opts) - try { - this.execFile(exec, args, opts, execFileCallback.bind(this)) - } catch (err) { - this.log.silly('execFile: threw:\n%s', err.stack) - return callback(err) - } - - function execFileCallback (err, stdout, stderr) { - this.log.silly('execFile result: err = %j', (err && err.stack) || err) - this.log.silly('execFile result: stdout = %j', stdout) - this.log.silly('execFile result: stderr = %j', stderr) - if (err) { - return callback(err) - } - const execPath = stdout.trim() - callback(null, execPath) - } - }, - - succeed: function succeed (execPath, version) { - this.log.info(`using Python version ${version} found at "${execPath}"`) - process.nextTick(this.callback.bind(null, null, execPath)) - }, - - fail: function fail () { - const errorLog = this.errorLog.join('\n') - - const pathExample = this.win ? 'C:\\Path\\To\\python.exe' - : '/path/to/pythonexecutable' - // For Windows 80 col console, use up to the column before the one marked - // with X (total 79 chars including logger prefix, 58 chars usable here): - // X - const info = [ - '**********************************************************', - 'You need to install the latest version of Python 2.7.', - 'Node-gyp should be able to find and use Python. If not,', - 'you can try one of the following options:', - `- Use the switch --python="${pathExample}"`, - ' (accepted by both node-gyp and npm)', - '- Set the environment variable PYTHON', - '- Set the npm configuration variable python:', - ` npm config set python "${pathExample}"`, - 'For more information consult the documentation at:', - 'https://github.com/nodejs/node-gyp#installation', - '**********************************************************' - ].join('\n') - - this.log.error(`\n${errorLog}\n\n${info}\n`) - process.nextTick(this.callback.bind(null, new Error( - 'Could not find any Python 2 installation to use'))) - } -} - -function findPython (configPython, callback) { - var finder = new PythonFinder(configPython, callback) - finder.findPython() -} - module.exports = configure module.exports.test = { - PythonFinder: PythonFinder, - findAccessibleSync: findAccessibleSync, - findPython: findPython + findAccessibleSync: findAccessibleSync } module.exports.usage = 'Generates ' + (win ? 'MSVC project files' : 'a Makefile') + ' for the current module' diff --git a/lib/find-python.js b/lib/find-python.js new file mode 100644 index 0000000000..ac16d477b1 --- /dev/null +++ b/lib/find-python.js @@ -0,0 +1,292 @@ +'use strict' + +const path = require('path') +const log = require('npmlog') +const semver = require('semver') +const cp = require('child_process') +const extend = require('util')._extend // eslint-disable-line +const win = process.platform === 'win32' +const logWithPrefix = require('./util').logWithPrefix + +function PythonFinder (configPython, callback) { + this.callback = callback + this.configPython = configPython + this.errorLog = [] +} + +PythonFinder.prototype = { + log: logWithPrefix(log, 'find Python'), + argsExecutable: [ '-c', 'import sys; print(sys.executable);' ], + argsVersion: [ '-c', 'import sys; print("%s.%s.%s" % sys.version_info[:3]);' ], + semverRange: '>=2.6.0 <3.0.0', + + // These can be overridden for testing: + execFile: cp.execFile, + env: process.env, + win: win, + pyLauncher: 'py.exe', + defaultLocation: path.join(process.env.SystemDrive || 'C:', 'Python27', + 'python.exe'), + + // Logs a message at verbose level, but also saves it to be displayed later + // at error level if an error occurs. This should help diagnose the problem. + addLog: function addLog (message) { + this.log.verbose(message) + this.errorLog.push(message) + }, + + // Find Python by trying a sequence of possibilities. + // Ignore errors, keep trying until Python is found. + findPython: function findPython () { + const SKIP = 0; const FAIL = 1 + const toCheck = [ + { + before: () => { + if (!this.configPython) { + this.addLog( + 'Python is not set from command line or npm configuration') + return SKIP + } + this.addLog('checking Python explicitly set from command line or ' + + 'npm configuration') + this.addLog('- "--python=" or "npm config get python" is ' + + `"${this.configPython}"`) + }, + check: this.checkCommand, + arg: this.configPython + }, + { + before: () => { + if (!this.env.PYTHON) { + this.addLog('Python is not set from environment variable PYTHON') + return SKIP + } + this.addLog( + 'checking Python explicitly set from environment variable PYTHON') + this.addLog(`- process.env.PYTHON is "${this.env.PYTHON}"`) + }, + check: this.checkCommand, + arg: this.env.PYTHON + }, + { + before: () => { this.addLog('checking if "python2" can be used') }, + check: this.checkCommand, + arg: 'python2' + }, + { + before: () => { this.addLog('checking if "python" can be used') }, + check: this.checkCommand, + arg: 'python' + }, + { + before: () => { + if (!this.win) { + // Everything after this is Windows specific + return FAIL + } + this.addLog( + 'checking if the py launcher can be used to find Python 2') + }, + check: this.checkPyLauncher + }, + { + before: () => { + this.addLog( + 'checking if Python 2 is installed in the default location') + }, + check: this.checkExecPath, + arg: this.defaultLocation + } + ] + + function runChecks (err) { + this.log.silly('runChecks: err = %j', (err && err.stack) || err) + + const check = toCheck.shift() + if (!check) { + return this.fail() + } + + const before = check.before.apply(this) + if (before === SKIP) { + return runChecks.apply(this) + } + if (before === FAIL) { + return this.fail() + } + + const args = [ runChecks.bind(this) ] + if (check.arg) { + args.unshift(check.arg) + } + check.check.apply(this, args) + } + + runChecks.apply(this) + }, + + // Check if command is a valid Python to use. + // Will exit the Python finder on success. + // If on Windows, run in a CMD shell to support BAT/CMD launchers. + checkCommand: function checkCommand (command, errorCallback) { + var exec = command + var args = this.argsExecutable + var shell = false + if (this.win) { + // Arguments have to be manually quoted + exec = `"${exec}"` + args = args.map(a => `"${a}"`) + shell = true + } + + this.log.verbose(`- executing "${command}" to get executable path`) + this.run(exec, args, shell, function (err, execPath) { + // Possible outcomes: + // - Error: not in PATH, not executable or execution fails + // - Gibberish: the next command to check version will fail + // - Absolute path to executable + if (err) { + this.addLog(`- "${command}" is not in PATH or produced an error`) + return errorCallback(err) + } + this.addLog(`- executable path is "${execPath}"`) + this.checkExecPath(execPath, errorCallback) + }.bind(this)) + }, + + // Check if the py launcher can find a valid Python to use. + // Will exit the Python finder on success. + // Distributions of Python on Windows by default install with the "py.exe" + // Python launcher which is more likely to exist than the Python executable + // being in the $PATH. + // Because the Python launcher supports all versions of Python, we have to + // explicitly request a Python 2 version. This is done by supplying "-2" as + // the first command line argument. Since "py.exe -2" would be an invalid + // executable for "execFile", we have to use the launcher to figure out + // where the actual "python.exe" executable is located. + checkPyLauncher: function checkPyLauncher (errorCallback) { + this.log.verbose( + `- executing "${this.pyLauncher}" to get Python 2 executable path`) + this.run(this.pyLauncher, [ '-2', ...this.argsExecutable ], false, + function (err, execPath) { + // Possible outcomes: same as checkCommand + if (err) { + this.addLog( + `- "${this.pyLauncher}" is not in PATH or produced an error`) + return errorCallback(err) + } + this.addLog(`- executable path is "${execPath}"`) + this.checkExecPath(execPath, errorCallback) + }.bind(this)) + }, + + // Check if a Python executable is the correct version to use. + // Will exit the Python finder on success. + checkExecPath: function checkExecPath (execPath, errorCallback) { + this.log.verbose(`- executing "${execPath}" to get version`) + this.run(execPath, this.argsVersion, false, function (err, version) { + // Possible outcomes: + // - Error: executable can not be run (likely meaning the command wasn't + // a Python executable and the previous command produced gibberish) + // - Gibberish: somehow the last command produced an executable path, + // this will fail when verifying the version + // - Version of the Python executable + if (err) { + this.addLog(`- "${execPath}" could not be run`) + return errorCallback(err) + } + this.addLog(`- version is "${version}"`) + + const range = semver.Range(this.semverRange) + var valid = false + try { + valid = range.test(version) + } catch (err) { + this.log.silly('range.test() threw:\n%s', err.stack) + this.addLog(`- "${execPath}" does not have a valid version`) + this.addLog('- is it a Python executable?') + return errorCallback(err) + } + + if (!valid) { + this.addLog(`- version is ${version} - should be ${this.semverRange}`) + this.addLog('- THIS VERSION OF PYTHON IS NOT SUPPORTED') + return errorCallback(new Error( + `Found unsupported Python version ${version}`)) + } + this.succeed(execPath, version) + }.bind(this)) + }, + + // Run an executable or shell command, trimming the output. + run: function run (exec, args, shell, callback) { + var env = extend({}, this.env) + env.TERM = 'dumb' + const opts = { env: env, shell: shell } + + this.log.silly('execFile: exec = %j', exec) + this.log.silly('execFile: args = %j', args) + this.log.silly('execFile: opts = %j', opts) + try { + this.execFile(exec, args, opts, execFileCallback.bind(this)) + } catch (err) { + this.log.silly('execFile: threw:\n%s', err.stack) + return callback(err) + } + + function execFileCallback (err, stdout, stderr) { + this.log.silly('execFile result: err = %j', (err && err.stack) || err) + this.log.silly('execFile result: stdout = %j', stdout) + this.log.silly('execFile result: stderr = %j', stderr) + if (err) { + return callback(err) + } + const execPath = stdout.trim() + callback(null, execPath) + } + }, + + succeed: function succeed (execPath, version) { + this.log.info(`using Python version ${version} found at "${execPath}"`) + process.nextTick(this.callback.bind(null, null, execPath)) + }, + + fail: function fail () { + const errorLog = this.errorLog.join('\n') + + const pathExample = this.win ? 'C:\\Path\\To\\python.exe' + : '/path/to/pythonexecutable' + // For Windows 80 col console, use up to the column before the one marked + // with X (total 79 chars including logger prefix, 58 chars usable here): + // X + const info = [ + '**********************************************************', + 'You need to install the latest version of Python 2.7.', + 'Node-gyp should be able to find and use Python. If not,', + 'you can try one of the following options:', + `- Use the switch --python="${pathExample}"`, + ' (accepted by both node-gyp and npm)', + '- Set the environment variable PYTHON', + '- Set the npm configuration variable python:', + ` npm config set python "${pathExample}"`, + 'For more information consult the documentation at:', + 'https://github.com/nodejs/node-gyp#installation', + '**********************************************************' + ].join('\n') + + this.log.error(`\n${errorLog}\n\n${info}\n`) + process.nextTick(this.callback.bind(null, new Error( + 'Could not find any Python 2 installation to use'))) + } +} + +function findPython (configPython, callback) { + var finder = new PythonFinder(configPython, callback) + finder.findPython() +} + +module.exports = findPython +module.exports.test = { + PythonFinder: PythonFinder, + findPython: findPython +} diff --git a/test/test-find-python.js b/test/test-find-python.js index 23dddc7bfb..cf8800f49a 100644 --- a/test/test-find-python.js +++ b/test/test-find-python.js @@ -1,16 +1,16 @@ 'use strict' const test = require('tap').test -const configure = require('../lib/configure') +const findPython = require('../lib/find-python') const execFile = require('child_process').execFile -const PythonFinder = configure.test.PythonFinder +const PythonFinder = findPython.test.PythonFinder require('npmlog').level = 'warn' test('find python', function (t) { t.plan(4) - configure.test.findPython(null, function (err, found) { + findPython.test.findPython(null, function (err, found) { t.strictEqual(err, null) var proc = execFile(found, ['-V'], function (err, stdout, stderr) { t.strictEqual(err, null) From 66ad30577594a37b08939948db1d4f1018d85335 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Mon, 8 Jul 2019 22:09:45 +0100 Subject: [PATCH 064/551] python: accept Python 3 conditionally This allows us to start testing Python 3 without breaking node-gyp for users. This also adds support for NODE_GYP_FORCE_PYTHON, which will be the only Python binary acceptable when defined. PR-URL: https://github.com/nodejs/node-gyp/pull/1815 Reviewed-By: Christian Clauss Reviewed-By: Richard Lau --- .travis.yml | 30 ++++++++- lib/find-python.js | 141 +++++++++++++++++++++++---------------- test/test-find-python.js | 131 +----------------------------------- 3 files changed, 114 insertions(+), 188 deletions(-) diff --git a/.travis.yml b/.travis.yml index 54a13a9382..a6dd2dcf28 100644 --- a/.travis.yml +++ b/.travis.yml @@ -4,36 +4,61 @@ cache: pip matrix: include: - name: "Python 2.7 on Linux" + env: NODE_GYP_FORCE_PYTHON=python2 python: 2.7 - name: "Python 2.7 on macOS" os: osx osx_image: xcode10.2 language: shell # 'language: python' is not yet supported on macOS + env: NODE_GYP_FORCE_PYTHON=python2 before_install: HOMEBREW_NO_AUTO_UPDATE=1 brew install npm - name: "Node.js 6 & Python 2.7 on Windows" os: windows language: node_js node_js: 6 # node - env: PATH=/c/Python27:/c/Python27/Scripts:$PATH + env: >- + PATH=/c/Python27:/c/Python27/Scripts:$PATH + NODE_GYP_FORCE_PYTHON=/c/Python27/python.exe before_install: choco install python2 - name: "Node.js 12 & Python 2.7 on Windows" os: windows language: node_js node_js: 12 # node - env: PATH=/c/Python27:/c/Python27/Scripts:$PATH + env: >- + PATH=/c/Python27:/c/Python27/Scripts:$PATH + NODE_GYP_FORCE_PYTHON=/c/Python27/python.exe before_install: choco install python2 - name: "Node.js 6 & Python 3.7 on Linux" python: 3.7 + env: NODE_GYP_FORCE_PYTHON=python3 EXPERIMENTAL_NODE_GYP_PYTHON3=1 before_install: nvm install 6 - name: "Node.js 8 & Python 3.7 on Linux" python: 3.7 + env: NODE_GYP_FORCE_PYTHON=python3 EXPERIMENTAL_NODE_GYP_PYTHON3=1 before_install: nvm install 8 - name: "Node.js 10 & Python 3.7 on Linux" python: 3.7 + env: NODE_GYP_FORCE_PYTHON=python3 EXPERIMENTAL_NODE_GYP_PYTHON3=1 before_install: nvm install 10 - name: "Node.js 12 & Python 3.7 on Linux" python: 3.7 + env: NODE_GYP_FORCE_PYTHON=python3 EXPERIMENTAL_NODE_GYP_PYTHON3=1 before_install: nvm install 12 + - name: "Node.js 12 & Python 3.7 on Windows" + os: windows + language: node_js + node_js: 12 # node + env: >- + PATH=/c/Python37:/c/Python37/Scripts:$PATH + NODE_GYP_FORCE_PYTHON=/c/Python37/python.exe + EXPERIMENTAL_NODE_GYP_PYTHON3=1 + before_install: choco install python + allow_failures: + - env: NODE_GYP_FORCE_PYTHON=python3 EXPERIMENTAL_NODE_GYP_PYTHON3=1 + - env: >- + PATH=/c/Python37:/c/Python37/Scripts:$PATH + NODE_GYP_FORCE_PYTHON=/c/Python37/python.exe + EXPERIMENTAL_NODE_GYP_PYTHON3=1 install: #- pip install -r requirements.txt - pip install flake8 # pytest # add another testing frameworks later @@ -45,6 +70,7 @@ before_script: - flake8 . --count --exit-zero --ignore=E111,E114,W503 --max-complexity=10 --max-line-length=127 --statistics - npm install script: + - node -e 'require("npmlog").level="verbose"; require("./lib/find-python")(null,()=>{})' - npm test #- pytest --capture=sys # add other tests here notifications: diff --git a/lib/find-python.js b/lib/find-python.js index ac16d477b1..1a4390dbd2 100644 --- a/lib/find-python.js +++ b/lib/find-python.js @@ -18,15 +18,18 @@ PythonFinder.prototype = { log: logWithPrefix(log, 'find Python'), argsExecutable: [ '-c', 'import sys; print(sys.executable);' ], argsVersion: [ '-c', 'import sys; print("%s.%s.%s" % sys.version_info[:3]);' ], - semverRange: '>=2.6.0 <3.0.0', + semverRange: process.env.EXPERIMENTAL_NODE_GYP_PYTHON3 ? '2.7.x || >=3.5.0' + : '>=2.6.0 <3.0.0', // These can be overridden for testing: execFile: cp.execFile, env: process.env, win: win, pyLauncher: 'py.exe', - defaultLocation: path.join(process.env.SystemDrive || 'C:', 'Python27', - 'python.exe'), + winDefaultLocations: [ + path.join(process.env.SystemDrive || 'C:', 'Python27', 'python.exe'), + path.join(process.env.SystemDrive || 'C:', 'Python37', 'python.exe') + ], // Logs a message at verbose level, but also saves it to be displayed later // at error level if an error occurs. This should help diagnose the problem. @@ -39,65 +42,87 @@ PythonFinder.prototype = { // Ignore errors, keep trying until Python is found. findPython: function findPython () { const SKIP = 0; const FAIL = 1 - const toCheck = [ - { - before: () => { - if (!this.configPython) { + var toCheck = getChecks.apply(this) + + function getChecks () { + if (this.env.NODE_GYP_FORCE_PYTHON) { + return [ { + before: () => { this.addLog( - 'Python is not set from command line or npm configuration') - return SKIP - } - this.addLog('checking Python explicitly set from command line or ' + - 'npm configuration') - this.addLog('- "--python=" or "npm config get python" is ' + - `"${this.configPython}"`) - }, - check: this.checkCommand, - arg: this.configPython - }, - { - before: () => { - if (!this.env.PYTHON) { - this.addLog('Python is not set from environment variable PYTHON') - return SKIP - } - this.addLog( - 'checking Python explicitly set from environment variable PYTHON') - this.addLog(`- process.env.PYTHON is "${this.env.PYTHON}"`) + 'checking Python explicitly set from NODE_GYP_FORCE_PYTHON') + this.addLog('- process.env.NODE_GYP_FORCE_PYTHON is ' + + `"${this.env.NODE_GYP_FORCE_PYTHON}"`) + }, + check: this.checkCommand, + arg: this.env.NODE_GYP_FORCE_PYTHON + } ] + } + + var checks = [ + { + before: () => { + if (!this.configPython) { + this.addLog( + 'Python is not set from command line or npm configuration') + return SKIP + } + this.addLog('checking Python explicitly set from command line or ' + + 'npm configuration') + this.addLog('- "--python=" or "npm config get python" is ' + + `"${this.configPython}"`) + }, + check: this.checkCommand, + arg: this.configPython }, - check: this.checkCommand, - arg: this.env.PYTHON - }, - { - before: () => { this.addLog('checking if "python2" can be used') }, - check: this.checkCommand, - arg: 'python2' - }, - { - before: () => { this.addLog('checking if "python" can be used') }, - check: this.checkCommand, - arg: 'python' - }, - { - before: () => { - if (!this.win) { - // Everything after this is Windows specific - return FAIL - } - this.addLog( - 'checking if the py launcher can be used to find Python 2') + { + before: () => { + if (!this.env.PYTHON) { + this.addLog('Python is not set from environment variable ' + + 'PYTHON') + return SKIP + } + this.addLog('checking Python explicitly set from environment ' + + 'variable PYTHON') + this.addLog(`- process.env.PYTHON is "${this.env.PYTHON}"`) + }, + check: this.checkCommand, + arg: this.env.PYTHON }, - check: this.checkPyLauncher - }, - { - before: () => { - this.addLog( - 'checking if Python 2 is installed in the default location') + { + before: () => { this.addLog('checking if "python" can be used') }, + check: this.checkCommand, + arg: 'python' }, - check: this.checkExecPath, - arg: this.defaultLocation + { + before: () => { this.addLog('checking if "python2" can be used') }, + check: this.checkCommand, + arg: 'python2' + } + ] + + if (this.win) { + checks.push({ + before: () => { + this.addLog( + 'checking if the py launcher can be used to find Python 2') + }, + check: this.checkPyLauncher + }) + for (var i = 0; i < this.winDefaultLocations.length; ++i) { + const location = this.winDefaultLocations[i] + checks.push({ + before: () => { + this.addLog('checking if Python is ' + + `${location}`) + }, + check: this.checkExecPath, + arg: location + }) + } } - ] + + return checks + } function runChecks (err) { this.log.silly('runChecks: err = %j', (err && err.stack) || err) @@ -276,7 +301,7 @@ PythonFinder.prototype = { this.log.error(`\n${errorLog}\n\n${info}\n`) process.nextTick(this.callback.bind(null, new Error( - 'Could not find any Python 2 installation to use'))) + 'Could not find any Python installation to use'))) } } diff --git a/test/test-find-python.js b/test/test-find-python.js index cf8800f49a..c52a579666 100644 --- a/test/test-find-python.js +++ b/test/test-find-python.js @@ -5,6 +5,9 @@ const findPython = require('../lib/find-python') const execFile = require('child_process').execFile const PythonFinder = findPython.test.PythonFinder +delete process.env.PYTHON +delete process.env.NODE_GYP_FORCE_PYTHON + require('npmlog').level = 'warn' test('find python', function (t) { @@ -94,27 +97,6 @@ test('find python - python too old', function (t) { } }) -test('find python - python too new', function (t) { - t.plan(2) - - var f = new TestPythonFinder(null, done) - f.execFile = function (program, args, opts, cb) { - if (/sys\.executable/.test(args[args.length - 1])) { - cb(null, '/path/python') - } else if (/sys\.version_info/.test(args[args.length - 1])) { - cb(null, '3.0.0') - } else { - t.fail() - } - } - f.findPython() - - function done (err) { - t.ok(/Could not find any Python/.test(err)) - t.ok(/not supported/i.test(f.errorLog)) - } -}) - test('find python - no python', function (t) { t.plan(2) @@ -136,31 +118,6 @@ test('find python - no python', function (t) { } }) -test('find python - no python2', function (t) { - t.plan(2) - - var f = new TestPythonFinder(null, done) - f.execFile = function (program, args, opts, cb) { - if (/sys\.executable/.test(args[args.length - 1])) { - if (program === 'python2') { - cb(new Error('not found')) - } else { - cb(null, '/path/python') - } - } else if (/sys\.version_info/.test(args[args.length - 1])) { - cb(null, '2.7.14') - } else { - t.fail() - } - } - f.findPython() - - function done (err, python) { - t.strictEqual(err, null) - t.strictEqual(python, '/path/python') - } -}) - test('find python - no python2, no python, unix', function (t) { t.plan(2) @@ -215,88 +172,6 @@ test('find python - no python, use python launcher', function (t) { } }) -test('find python - python 3, use python launcher', function (t) { - t.plan(4) - - var f = new TestPythonFinder(null, done) - f.win = true - - f.execFile = function (program, args, opts, cb) { - if (program === 'py.exe') { - f.execFile = function (program, args, opts, cb) { - poison(f, 'execFile') - if (/sys\.version_info/.test(args[args.length - 1])) { - cb(null, '2.7.14') - } else { - t.fail() - } - } - t.notEqual(args.indexOf('-2'), -1) - t.notEqual(args.indexOf('-c'), -1) - return cb(null, 'Z:\\snake.exe') - } - if (/sys\.executable/.test(args[args.length - 1])) { - cb(null, '/path/python') - } else if (/sys\.version_info/.test(args[args.length - 1])) { - cb(null, '3.0.0') - } else { - t.fail() - } - } - f.findPython() - - function done (err, python) { - t.strictEqual(err, null) - t.strictEqual(python, 'Z:\\snake.exe') - } -}) - -test('find python - python 3, use python launcher, python 2 too old', - function (t) { - t.plan(6) - - var f = new TestPythonFinder(null, done) - f.win = true - - f.execFile = function (program, args, opts, cb) { - if (program === 'py.exe') { - f.execFile = function (program, args, opts, cb) { - if (/sys\.version_info/.test(args[args.length - 1])) { - f.execFile = function (program, args, opts, cb) { - if (/sys\.version_info/.test(args[args.length - 1])) { - poison(f, 'execFile') - t.strictEqual(program, f.defaultLocation) - cb(new Error('not found')) - } else { - t.fail() - } - } - t.strictEqual(program, 'Z:\\snake.exe') - cb(null, '2.3.4') - } else { - t.fail() - } - } - t.notEqual(args.indexOf('-2'), -1) - t.notEqual(args.indexOf('-c'), -1) - return cb(null, 'Z:\\snake.exe') - } - if (/sys\.executable/.test(args[args.length - 1])) { - cb(null, '/path/python') - } else if (/sys\.version_info/.test(args[args.length - 1])) { - cb(null, '3.0.0') - } else { - t.fail() - } - } - f.findPython() - - function done (err) { - t.ok(/Could not find any Python/.test(err)) - t.ok(/not supported/i.test(f.errorLog)) - } - }) - test('find python - no python, no python launcher, good guess', function (t) { t.plan(4) From 0878db3b81a79ead9c62f0562c9db143726d607d Mon Sep 17 00:00:00 2001 From: Ben Noordhuis Date: Tue, 16 Jul 2019 10:22:38 +0200 Subject: [PATCH 065/551] Revert "build,test: add duplicate symbol test" This reverts commit 2761afbf730f7001675bbdbfa4a9e4821243adcb. Building with `-fvisibility=hidden` breaks some of Node's add-on tests and therefore likely also affects third-party add-ons. This change was landed in a patch release so I'm opting to revert it until the next major release. PR-URL: https://github.com/nodejs/node-gyp/pull/1828 Refs: https://github.com/nodejs/node/pull/28647#issuecomment-511715968 Reviewed-By: Richard Lau Reviewed-By: Rod Vagg --- addon.gypi | 4 -- .../node_modules/duplicate_symbols/binding.cc | 10 ----- .../duplicate_symbols/binding.gyp | 19 ---------- test/node_modules/duplicate_symbols/common.h | 37 ------------------- test/node_modules/duplicate_symbols/extra.cc | 6 --- test/node_modules/duplicate_symbols/index.js | 5 --- .../duplicate_symbols/package.json | 14 ------- test/test-addon.js | 24 ------------ 8 files changed, 119 deletions(-) delete mode 100644 test/node_modules/duplicate_symbols/binding.cc delete mode 100644 test/node_modules/duplicate_symbols/binding.gyp delete mode 100644 test/node_modules/duplicate_symbols/common.h delete mode 100644 test/node_modules/duplicate_symbols/extra.cc delete mode 100644 test/node_modules/duplicate_symbols/index.js delete mode 100644 test/node_modules/duplicate_symbols/package.json diff --git a/addon.gypi b/addon.gypi index 3cadaab78b..6462f539ff 100644 --- a/addon.gypi +++ b/addon.gypi @@ -90,14 +90,10 @@ 'conditions': [ [ 'OS=="mac"', { - 'cflags': [ - '-fvisibility=hidden' - ], 'defines': [ '_DARWIN_USE_64_BIT_INODE=1' ], 'xcode_settings': { - 'GCC_SYMBOLS_PRIVATE_EXTERN': 'YES', # -fvisibility=hidden 'DYLIB_INSTALL_NAME_BASE': '@rpath' }, }], diff --git a/test/node_modules/duplicate_symbols/binding.cc b/test/node_modules/duplicate_symbols/binding.cc deleted file mode 100644 index a0999b76dc..0000000000 --- a/test/node_modules/duplicate_symbols/binding.cc +++ /dev/null @@ -1,10 +0,0 @@ -#include -#include "common.h" - -void Init(v8::Local exports) { - Nan::Set(exports, Nan::New("pointerCheck").ToLocalChecked(), - Nan::GetFunction( - Nan::New(Something::PointerCheck)).ToLocalChecked()); -} - -NODE_MODULE(NODE_GYP_MODULE_NAME, Init) diff --git a/test/node_modules/duplicate_symbols/binding.gyp b/test/node_modules/duplicate_symbols/binding.gyp deleted file mode 100644 index 46f1c8cba4..0000000000 --- a/test/node_modules/duplicate_symbols/binding.gyp +++ /dev/null @@ -1,19 +0,0 @@ -{ - "target_defaults": { - "include_dirs": [ - " - -class Something { - public: - static void PointerCheck(const Nan::FunctionCallbackInfo& info); -}; - -// Removing the inline keyword below will result in the addon failing to link -// on OSX because of a duplicate symbol. -inline void -Something::PointerCheck(const Nan::FunctionCallbackInfo& info) { - v8::Local v8result; - - if (info.Length() > 0) { - // If an argument was passed in, it is a pointer to the `PointerCheck` - // method from the other addon. So, we compare it to the value of the - // pointer to the `PointerCheck` method in this addon, and return - // `"equal"` if they are equal, and `"not equal"` otherwise". - - const char* result = - (reinterpret_cast(Something::PointerCheck) == - info[0].As()->Value()) ? - "equal" : "not equal"; - v8result = Nan::New(result).ToLocalChecked(); - } else { - // If no argument was passed in, we wrap the pointer to the `PointerCheck` - // method in this addon into a `v8::External` and pass it into JavaScript. - v8result = Nan::New( - reinterpret_cast(Something::PointerCheck)); - } - info.GetReturnValue().Set(v8result); -} - -#endif // DUPLICATE_SYMBOLS_COMMON_H_ diff --git a/test/node_modules/duplicate_symbols/extra.cc b/test/node_modules/duplicate_symbols/extra.cc deleted file mode 100644 index 0c9d5dbebd..0000000000 --- a/test/node_modules/duplicate_symbols/extra.cc +++ /dev/null @@ -1,6 +0,0 @@ -#include "common.h" - -// It is important that common.h be included from two different translation -// units, because doing so can create duplicate symbols in some instances. If -// it does so and fails to build because of it, that is considered a test -// failure. diff --git a/test/node_modules/duplicate_symbols/index.js b/test/node_modules/duplicate_symbols/index.js deleted file mode 100644 index 09dbed81b7..0000000000 --- a/test/node_modules/duplicate_symbols/index.js +++ /dev/null @@ -1,5 +0,0 @@ -'use strict' -module.exports = { - pointerCheck1: require('bindings')('binding1').pointerCheck, - pointerCheck2: require('bindings')('binding2').pointerCheck -}; diff --git a/test/node_modules/duplicate_symbols/package.json b/test/node_modules/duplicate_symbols/package.json deleted file mode 100644 index 2a193f230f..0000000000 --- a/test/node_modules/duplicate_symbols/package.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "name": "duplicate_symbols", - "version": "0.0.0", - "description": "Duplicate Symbols Test", - "main": "index.js", - "private": true, - "dependencies": { - "bindings": "~1.2.1", - "nan": "^2.14.0" - }, - "scripts": { - "test": "node index.js" - } -} diff --git a/test/test-addon.js b/test/test-addon.js index 49e30164c3..f97215c0a2 100644 --- a/test/test-addon.js +++ b/test/test-addon.js @@ -18,15 +18,6 @@ function runHello (hostProcess) { return execFileSync(hostProcess, [ '-e', testCode ], { cwd: __dirname }).toString() } -function runDuplicateBindings () { - const hostProcess = process.execPath - var testCode = - 'console.log((function(bindings) {' + - 'return bindings.pointerCheck1(bindings.pointerCheck2());' + - "})(require('duplicate_symbols')))" - return execFileSync(hostProcess, [ '-e', testCode ], { cwd: __dirname }).toString() -} - function getEncoding () { var code = 'import locale;print(locale.getdefaultlocale()[1])' return execFileSync('python', [ '-c', code ]).toString().trim() @@ -60,21 +51,6 @@ test('build simple addon', function (t) { proc.stderr.setEncoding('utf-8') }) -test('make sure addon symbols do not overlap', function (t) { - t.plan(3) - - var addonPath = path.resolve(__dirname, 'node_modules', 'duplicate_symbols') - // Set the loglevel otherwise the output disappears when run via 'npm test' - var cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose'] - execFile(process.execPath, cmd, function (err, stdout, stderr) { - var logLines = stderr.trim().split(/\r?\n/) - var lastLine = logLines[logLines.length - 1] - t.strictEqual(err, null) - t.strictEqual(lastLine, 'gyp info ok', 'should end in ok') - t.strictEqual(runDuplicateBindings().trim(), 'not equal') - }) -}) - test('build simple addon in path with non-ascii characters', function (t) { t.plan(1) From 64bb407c14149c216885a48e78df178cedaec8fd Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Tue, 16 Jul 2019 20:16:31 +1000 Subject: [PATCH 066/551] v5.0.3: bump version and update changelog --- CHANGELOG.md | 12 ++++++++++++ package.json | 2 +- 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8aee70eb5f..9a99ef2d0a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,15 @@ +v5.0.3 2019-07-17 +================= + +* [[`66ad305775`](https://github.com/nodejs/node-gyp/commit/66ad305775)] - **python**: accept Python 3 conditionally (João Reis) [#1815](https://github.com/nodejs/node-gyp/pull/1815) +* [[`7e7fce3fed`](https://github.com/nodejs/node-gyp/commit/7e7fce3fed)] - **python**: move Python detection to its own file (João Reis) [#1815](https://github.com/nodejs/node-gyp/pull/1815) +* [[`e40c99e283`](https://github.com/nodejs/node-gyp/commit/e40c99e283)] - **src**: implement standard.js linting (Rod Vagg) [#1794](https://github.com/nodejs/node-gyp/pull/1794) +* [[`bb92c761a9`](https://github.com/nodejs/node-gyp/commit/bb92c761a9)] - **test**: add Node.js 6 on Windows to Travis CI (João Reis) [#1812](https://github.com/nodejs/node-gyp/pull/1812) +* [[`7fd924079f`](https://github.com/nodejs/node-gyp/commit/7fd924079f)] - **test**: increase tap timeout (João Reis) [#1812](https://github.com/nodejs/node-gyp/pull/1812) +* [[`7e8127068f`](https://github.com/nodejs/node-gyp/commit/7e8127068f)] - **test**: cover supported node versions with travis (Rod Vagg) [#1809](https://github.com/nodejs/node-gyp/pull/1809) +* [[`24109148df`](https://github.com/nodejs/node-gyp/commit/24109148df)] - **test**: downgrade to tap@^12 for continued Node 6 support (Rod Vagg) [#1808](https://github.com/nodejs/node-gyp/pull/1808) +* [[`656117cc4a`](https://github.com/nodejs/node-gyp/commit/656117cc4a)] - **win**: make VS path match case-insensitive (João Reis) [#1806](https://github.com/nodejs/node-gyp/pull/1806) + v5.0.2 2019-06-27 ================= diff --git a/package.json b/package.json index 15f07459eb..4eba267b07 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "5.0.2", + "version": "5.0.3", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From c6e3b65a23948aa0cc7d909ec580a3b6dd52776e Mon Sep 17 00:00:00 2001 From: cclauss Date: Thu, 11 Jul 2019 07:44:33 +0200 Subject: [PATCH 067/551] lib: raise the minimum Python version from 2.6 to 2.7 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit As discussed in #1811 PR-URL: https://github.com/nodejs/node-gyp/pull/1818 Reviewed-By: Richard Lau Reviewed-By: João Reis --- lib/find-python.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/find-python.js b/lib/find-python.js index 1a4390dbd2..30bb25fd36 100644 --- a/lib/find-python.js +++ b/lib/find-python.js @@ -19,7 +19,7 @@ PythonFinder.prototype = { argsExecutable: [ '-c', 'import sys; print(sys.executable);' ], argsVersion: [ '-c', 'import sys; print("%s.%s.%s" % sys.version_info[:3]);' ], semverRange: process.env.EXPERIMENTAL_NODE_GYP_PYTHON3 ? '2.7.x || >=3.5.0' - : '>=2.6.0 <3.0.0', + : '>=2.7.0 <3.0.0', // These can be overridden for testing: execFile: cp.execFile, From 573607981e70dc5a28635676acc83afabe43317d Mon Sep 17 00:00:00 2001 From: Ivan Petrovic Date: Fri, 12 Jul 2019 09:43:52 +0200 Subject: [PATCH 068/551] win,src: update win_delay_load_hook.cc to work with /clr MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Added "#pragma unmanaged" in win_delay_load_hook.cc to support clr PR-URL: https://github.com/nodejs/node-gyp/pull/1819 Reviewed-By: João Reis --- src/win_delay_load_hook.cc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/win_delay_load_hook.cc b/src/win_delay_load_hook.cc index 5e7f14b2b2..567c4cc8d7 100644 --- a/src/win_delay_load_hook.cc +++ b/src/win_delay_load_hook.cc @@ -7,6 +7,8 @@ * This allows compiled addons to work when the host executable is renamed. */ +#pragma unmanaged + #ifdef _MSC_VER #ifndef WIN32_LEAN_AND_MEAN From 4ef83eddd08c8b8801a98a18a1eab474249fba2d Mon Sep 17 00:00:00 2001 From: cclauss Date: Fri, 12 Jul 2019 10:33:33 +0200 Subject: [PATCH 069/551] build: more Python 3 compat, replace compile with ast Make Python 3 compatiblity changes so the code works in both Python 2 and Python 3. Especially, make changes required because the compiler module was removed in Python 3 in favor of the ast module that exists in both Python 2 and Python 3. PR-URL: https://github.com/nodejs/node-gyp/pull/1820 Reviewed-By: Ben Noordhuis Reviewed-By: Rod Vagg Reviewed-By: Richard Lau --- gyp/pylib/gyp/generator/analyzer.py | 2 +- gyp/pylib/gyp/generator/eclipse.py | 2 +- gyp/pylib/gyp/generator/make.py | 38 ++++++++++---------- gyp/pylib/gyp/generator/msvs.py | 6 ++-- gyp/pylib/gyp/input.py | 54 +++++++++++++---------------- gyp/pylib/gyp/xcode_emulation.py | 2 +- gyp/pylib/gyp/xcode_ninja.py | 2 +- 7 files changed, 50 insertions(+), 56 deletions(-) diff --git a/gyp/pylib/gyp/generator/analyzer.py b/gyp/pylib/gyp/generator/analyzer.py index dc17c96524..0416b5d9be 100644 --- a/gyp/pylib/gyp/generator/analyzer.py +++ b/gyp/pylib/gyp/generator/analyzer.py @@ -671,7 +671,7 @@ def find_matching_compile_target_names(self): assert self.is_build_impacted(); # Compile targets are found by searching up from changed targets. # Reset the visited status for _GetBuildTargets. - for target in self._name_to_target.itervalues(): + for target in self._name_to_target.values(): target.visited = False supplied_targets = _LookupTargets(self._supplied_target_names_no_all(), diff --git a/gyp/pylib/gyp/generator/eclipse.py b/gyp/pylib/gyp/generator/eclipse.py index b7c6aa951f..372ceec246 100644 --- a/gyp/pylib/gyp/generator/eclipse.py +++ b/gyp/pylib/gyp/generator/eclipse.py @@ -272,7 +272,7 @@ def WriteMacros(out, eclipse_langs, defines): out.write(' \n') for lang in eclipse_langs: out.write(' \n' % lang) - for key in sorted(defines.iterkeys()): + for key in sorted(defines): out.write(' %s%s\n' % (escape(key), escape(defines[key]))) out.write(' \n') diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index 37ac255bfa..385a0f7a08 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -821,7 +821,7 @@ def Write(self, qualified_target, base_path, output_filename, spec, configs, gyp.xcode_emulation.MacPrefixHeader( self.xcode_settings, lambda p: Sourceify(self.Absolutify(p)), self.Pchify)) - sources = filter(Compilable, all_sources) + sources = list(filter(Compilable, all_sources)) if sources: self.WriteLn(SHARED_HEADER_SUFFIX_RULES_COMMENT1) extensions = set([os.path.splitext(s)[1] for s in sources]) @@ -950,7 +950,7 @@ def WriteActions(self, actions, extra_sources, extra_outputs, '%s%s' % (name, cd_action, command)) self.WriteLn() - outputs = map(self.Absolutify, outputs) + outputs = [self.Absolutify(output) for output in outputs] # The makefile rules are all relative to the top dir, but the gyp actions # are defined relative to their containing dir. This replaces the obj # variable for the action rule with an absolute version so that the output @@ -974,7 +974,7 @@ def WriteActions(self, actions, extra_sources, extra_outputs, outputs = [gyp.xcode_emulation.ExpandEnvVars(o, env) for o in outputs] inputs = [gyp.xcode_emulation.ExpandEnvVars(i, env) for i in inputs] - self.WriteDoCmd(outputs, map(Sourceify, map(self.Absolutify, inputs)), + self.WriteDoCmd(outputs, [Sourceify(self.Absolutify(i)) for i in inputs], part_of_all=part_of_all, command=name) # Stuff the outputs in a variable so we can refer to them later. @@ -1023,8 +1023,8 @@ def WriteRules(self, rules, extra_sources, extra_outputs, extra_sources += outputs if int(rule.get('process_outputs_as_mac_bundle_resources', False)): extra_mac_bundle_resources += outputs - inputs = map(Sourceify, map(self.Absolutify, [rule_source] + - rule.get('inputs', []))) + inputs = [Sourceify(self.Absolutify(i)) for i + in [rule_source] + rule.get('inputs', [])] actions = ['$(call do_cmd,%s_%d)' % (name, count)] if name == 'resources_grit': @@ -1040,7 +1040,7 @@ def WriteRules(self, rules, extra_sources, extra_outputs, outputs = [gyp.xcode_emulation.ExpandEnvVars(o, env) for o in outputs] inputs = [gyp.xcode_emulation.ExpandEnvVars(i, env) for i in inputs] - outputs = map(self.Absolutify, outputs) + outputs = [self.Absolutify(output) for output in outputs] all_outputs += outputs # Only write the 'obj' and 'builddir' rules for the "primary" output # (:1); it's superfluous for the "extra outputs", and this avoids @@ -1147,7 +1147,7 @@ def WriteCopies(self, copies, extra_outputs, part_of_all): path = gyp.xcode_emulation.ExpandEnvVars(path, env) self.WriteDoCmd([output], [path], 'copy', part_of_all) outputs.append(output) - self.WriteLn('%s = %s' % (variable, ' '.join(map(QuoteSpaces, outputs)))) + self.WriteLn('%s = %s' % (variable, ' '.join(QuoteSpaces(o) for o in outputs))) extra_outputs.append('$(%s)' % variable) self.WriteLn() @@ -1158,7 +1158,7 @@ def WriteMacBundleResources(self, resources, bundle_deps): for output, res in gyp.xcode_emulation.GetMacBundleResources( generator_default_variables['PRODUCT_DIR'], self.xcode_settings, - map(Sourceify, map(self.Absolutify, resources))): + [Sourceify(self.Absolutify(r)) for r in resources]): _, ext = os.path.splitext(output) if ext != '.xcassets': # Make does not supports '.xcassets' emulation. @@ -1238,11 +1238,11 @@ def WriteSources(self, configs, deps, sources, self.WriteList(cflags_objcc, 'CFLAGS_OBJCC_%s' % configname) includes = config.get('include_dirs') if includes: - includes = map(Sourceify, map(self.Absolutify, includes)) + includes = [Sourceify(self.Absolutify(i)) for i in includes] self.WriteList(includes, 'INCS_%s' % configname, prefix='-I') - compilable = filter(Compilable, sources) - objs = map(self.Objectify, map(self.Absolutify, map(Target, compilable))) + compilable = list(filter(Compilable, sources)) + objs = [self.Objectify(self.Absolutify(Target(c))) for c in compilable] self.WriteList(objs, 'OBJS') for obj in objs: @@ -1314,7 +1314,7 @@ def WriteSources(self, configs, deps, sources, # If there are any object files in our input file list, link them into our # output. - extra_link_deps += filter(Linkable, sources) + extra_link_deps += list(filter(Linkable, sources)) self.WriteLn() @@ -1564,7 +1564,7 @@ def WriteTarget(self, spec, configs, deps, link_deps, bundle_deps, # Bundle dependencies. Note that the code below adds actions to this # target, so if you move these two lines, move the lines below as well. - self.WriteList(map(QuoteSpaces, bundle_deps), 'BUNDLE_DEPS') + self.WriteList([QuoteSpaces(dep) for dep in bundle_deps], 'BUNDLE_DEPS') self.WriteLn('%s: $(BUNDLE_DEPS)' % QuoteSpaces(self.output)) # After the framework is built, package it. Needs to happen before @@ -1598,7 +1598,7 @@ def WriteTarget(self, spec, configs, deps, link_deps, bundle_deps, if self.type == 'executable': self.WriteLn('%s: LD_INPUTS := %s' % ( QuoteSpaces(self.output_binary), - ' '.join(map(QuoteSpaces, link_deps)))) + ' '.join(QuoteSpaces(dep) for dep in link_deps))) if self.toolset == 'host' and self.flavor == 'android': self.WriteDoCmd([self.output_binary], link_deps, 'link_host', part_of_all, postbuilds=postbuilds) @@ -1620,7 +1620,7 @@ def WriteTarget(self, spec, configs, deps, link_deps, bundle_deps, elif self.type == 'shared_library': self.WriteLn('%s: LD_INPUTS := %s' % ( QuoteSpaces(self.output_binary), - ' '.join(map(QuoteSpaces, link_deps)))) + ' '.join(QuoteSpaces(dep) for dep in link_deps))) self.WriteDoCmd([self.output_binary], link_deps, 'solink', part_of_all, postbuilds=postbuilds) elif self.type == 'loadable_module': @@ -1746,8 +1746,8 @@ def WriteMakeRule(self, outputs, inputs, actions=None, comment=None, output is just a name to run the rule command: (optional) command name to generate unambiguous labels """ - outputs = map(QuoteSpaces, outputs) - inputs = map(QuoteSpaces, inputs) + outputs = [QuoteSpaces(o) for o in outputs] + inputs = [QuoteSpaces(i) for i in inputs] if comment: self.WriteLn('# ' + comment) @@ -1836,7 +1836,7 @@ def WriteAndroidNdkModuleRule(self, module_name, all_sources, link_deps): default_cpp_ext = ext self.WriteLn('LOCAL_CPP_EXTENSION := ' + default_cpp_ext) - self.WriteList(map(self.Absolutify, filter(Compilable, all_sources)), + self.WriteList(list(map(self.Absolutify, filter(Compilable, all_sources))), 'LOCAL_SRC_FILES') # Filter out those which do not match prefix and suffix and produce @@ -1979,7 +1979,7 @@ def WriteAutoRegenerationRule(params, root_makefile, makefile_name, "%(makefile_name)s: %(deps)s\n" "\t$(call do_cmd,regen_makefile)\n\n" % { 'makefile_name': makefile_name, - 'deps': ' '.join(map(SourceifyAndQuoteSpaces, build_files)), + 'deps': ' '.join(SourceifyAndQuoteSpaces(bf) for bf in build_files), 'cmd': gyp.common.EncodePOSIXShellList( [gyp_binary, '-fmake'] + gyp.RegenerateFlags(options) + diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index aacbe60836..2dc967d709 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -2691,7 +2691,7 @@ def _GetMSBuildGlobalProperties(spec, guid, gyp_file_name): platform_name = None msvs_windows_target_platform_version = None - for configuration in spec['configurations'].itervalues(): + for configuration in spec['configurations'].values(): platform_name = platform_name or _ConfigPlatform(configuration) msvs_windows_target_platform_version = \ msvs_windows_target_platform_version or \ @@ -3252,7 +3252,7 @@ def _GetMSBuildProjectReferences(project): ['Project', guid], ['ReferenceOutputAssembly', 'false'] ] - for config in dependency.spec.get('configurations', {}).itervalues(): + for config in dependency.spec.get('configurations', {}).values(): if config.get('msvs_use_library_dependency_inputs', 0): project_ref.append(['UseLibraryDependencyInputs', 'true']) break @@ -3321,7 +3321,7 @@ def _GenerateMSBuildProject(project, options, version, generator_flags): extension_to_rule_name, _GetUniquePlatforms(spec)) missing_sources = _VerifySourcesExist(sources, project_dir) - for configuration in configurations.itervalues(): + for configuration in configurations.values(): _FinalizeMSBuildSettings(spec, configuration) # Add attributes to root element diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index eb9858f0c8..dde2823a9f 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -4,14 +4,8 @@ from __future__ import print_function -from compiler.ast import Const -from compiler.ast import Dict -from compiler.ast import Discard -from compiler.ast import List -from compiler.ast import Module -from compiler.ast import Node -from compiler.ast import Stmt -import compiler +import ast + import gyp.common import gyp.simple_copy import multiprocessing @@ -184,43 +178,39 @@ def CheckedEval(file_contents): Note that this is slower than eval() is. """ - ast = compiler.parse(file_contents) - assert isinstance(ast, Module) - c1 = ast.getChildren() - assert c1[0] is None - assert isinstance(c1[1], Stmt) - c2 = c1[1].getChildren() - assert isinstance(c2[0], Discard) - c3 = c2[0].getChildren() - assert len(c3) == 1 - return CheckNode(c3[0], []) + syntax_tree = ast.parse(file_contents) + assert isinstance(syntax_tree, ast.Module) + c1 = syntax_tree.body + assert len(c1) == 1 + c2 = c1[0] + assert isinstance(c2, ast.Expr) + return CheckNode(c2.value, []) def CheckNode(node, keypath): - if isinstance(node, Dict): + if isinstance(node, ast.Dict): c = node.getChildren() dict = {} - for n in range(0, len(c), 2): - assert isinstance(c[n], Const) - key = c[n].getChildren()[0] + for key, value in zip(node.keys, node.values): + assert isinstance(key, ast.Str) + key = key.s if key in dict: raise GypError("Key '" + key + "' repeated at level " + repr(len(keypath) + 1) + " with key path '" + '.'.join(keypath) + "'") kp = list(keypath) # Make a copy of the list for descending this node. kp.append(key) - dict[key] = CheckNode(c[n + 1], kp) + dict[key] = CheckNode(value, kp) return dict - elif isinstance(node, List): - c = node.getChildren() + elif isinstance(node, ast.List): children = [] - for index, child in enumerate(c): + for index, child in enumerate(node.elts): kp = list(keypath) # Copy list. kp.append(repr(index)) children.append(CheckNode(child, kp)) return children - elif isinstance(node, Const): - return node.getChildren()[0] + elif isinstance(node, ast.Str): + return node.s else: raise TypeError("Unknown AST node at key path '" + '.'.join(keypath) + "': " + repr(node)) @@ -954,8 +944,12 @@ def ExpandVariables(input, phase, variables, build_file): else: replacement = variables[contents] + if isinstance(replacement, bytes) and not isinstance(replacement, str): + replacement = replacement.decode("utf-8") # done on Python 3 only if type(replacement) is list: for item in replacement: + if isinstance(item, bytes) and not isinstance(item, str): + item = item.decode("utf-8") # done on Python 3 only if not contents[-1] == '/' and type(item) not in (str, int): raise GypError('Variable ' + contents + ' must expand to a string or list of strings; ' + @@ -1847,7 +1841,7 @@ def VerifyNoGYPFileCircularDependencies(targets): # Create a DependencyGraphNode for each gyp file containing a target. Put # it into a dict for easy access. dependency_nodes = {} - for target in targets.iterkeys(): + for target in targets: build_file = gyp.common.BuildFile(target) if not build_file in dependency_nodes: dependency_nodes[build_file] = DependencyGraphNode(build_file) @@ -1878,7 +1872,7 @@ def VerifyNoGYPFileCircularDependencies(targets): # Files that have no dependencies are treated as dependent on root_node. root_node = DependencyGraphNode(None) - for build_file_node in dependency_nodes.itervalues(): + for build_file_node in dependency_nodes.values(): if len(build_file_node.dependencies) == 0: build_file_node.dependencies.append(root_node) root_node.dependents.append(build_file_node) diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py index 6ae41e293a..0162d978bf 100644 --- a/gyp/pylib/gyp/xcode_emulation.py +++ b/gyp/pylib/gyp/xcode_emulation.py @@ -1636,7 +1636,7 @@ def _HasIOSTarget(targets): def _AddIOSDeviceConfigurations(targets): """Clone all targets and append -iphoneos to the name. Configure these targets to build for iOS devices and use correct architectures for those builds.""" - for target_dict in targets.itervalues(): + for target_dict in targets.values(): toolset = target_dict['toolset'] configs = target_dict['configurations'] for config_name, config_dict in dict(configs).items(): diff --git a/gyp/pylib/gyp/xcode_ninja.py b/gyp/pylib/gyp/xcode_ninja.py index 5acd82e004..d70eddc90a 100644 --- a/gyp/pylib/gyp/xcode_ninja.py +++ b/gyp/pylib/gyp/xcode_ninja.py @@ -85,7 +85,7 @@ def _TargetFromSpec(old_spec, params): "%s/$(CONFIGURATION)$(EFFECTIVE_PLATFORM_NAME)" % ninja_toplevel if 'configurations' in old_spec: - for config in old_spec['configurations'].iterkeys(): + for config in old_spec['configurations']: old_xcode_settings = \ old_spec['configurations'][config].get('xcode_settings', {}) if 'IPHONEOS_DEPLOYMENT_TARGET' in old_xcode_settings: From 5459ecaf3f09bce3b67c1632b8cb0142fdbaf66c Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Sun, 21 Jul 2019 09:31:13 +0200 Subject: [PATCH 070/551] build: import StringIO on Python 2 and Python 3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This change supports access to StringIO on both Python 2 and Python 3 PR-URL: https://github.com/nodejs/node-gyp/pull/1836 Reviewed-By: Rod Vagg Reviewed-By: João Reis --- gyp/pylib/gyp/MSVSSettings_test.py | 8 ++++++-- gyp/pylib/gyp/easy_xml_test.py | 7 +++++-- gyp/pylib/gyp/generator/msvs_test.py | 7 +++++-- gyp/pylib/gyp/generator/ninja.py | 5 ++++- gyp/pylib/gyp/generator/ninja_test.py | 1 - 5 files changed, 20 insertions(+), 8 deletions(-) diff --git a/gyp/pylib/gyp/MSVSSettings_test.py b/gyp/pylib/gyp/MSVSSettings_test.py index bf6ea6b802..c082bbea9b 100755 --- a/gyp/pylib/gyp/MSVSSettings_test.py +++ b/gyp/pylib/gyp/MSVSSettings_test.py @@ -6,7 +6,11 @@ """Unit tests for the MSVSSettings.py file.""" -import StringIO +try: + from cStringIO import StringIO +except ImportError: + from io import StringIO + import unittest import gyp.MSVSSettings as MSVSSettings @@ -14,7 +18,7 @@ class TestSequenceFunctions(unittest.TestCase): def setUp(self): - self.stderr = StringIO.StringIO() + self.stderr = StringIO() def _ExpectedWarnings(self, expected): """Compares recorded lines to expected warnings.""" diff --git a/gyp/pylib/gyp/easy_xml_test.py b/gyp/pylib/gyp/easy_xml_test.py index df64354982..2a80b8a456 100755 --- a/gyp/pylib/gyp/easy_xml_test.py +++ b/gyp/pylib/gyp/easy_xml_test.py @@ -8,13 +8,16 @@ import gyp.easy_xml as easy_xml import unittest -import StringIO +try: + from cStringIO import StringIO +except ImportError: + from io import StringIO class TestSequenceFunctions(unittest.TestCase): def setUp(self): - self.stderr = StringIO.StringIO() + self.stderr = StringIO() def test_EasyXml_simple(self): self.assertEqual( diff --git a/gyp/pylib/gyp/generator/msvs_test.py b/gyp/pylib/gyp/generator/msvs_test.py index c0b021df50..daf4f411bc 100755 --- a/gyp/pylib/gyp/generator/msvs_test.py +++ b/gyp/pylib/gyp/generator/msvs_test.py @@ -7,13 +7,16 @@ import gyp.generator.msvs as msvs import unittest -import StringIO +try: + from cStringIO import StringIO +except ImportError: + from io import StringIO class TestSequenceFunctions(unittest.TestCase): def setUp(self): - self.stderr = StringIO.StringIO() + self.stderr = StringIO() def test_GetLibraries(self): self.assertEqual( diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py index aae5b71310..4b5122af1d 100644 --- a/gyp/pylib/gyp/generator/ninja.py +++ b/gyp/pylib/gyp/generator/ninja.py @@ -19,7 +19,10 @@ import gyp.msvs_emulation import gyp.MSVSUtil as MSVSUtil import gyp.xcode_emulation -from cStringIO import StringIO +try: + from cStringIO import StringIO +except ImportError: + from io import StringIO from gyp.common import GetEnvironFallback import gyp.ninja_syntax as ninja_syntax diff --git a/gyp/pylib/gyp/generator/ninja_test.py b/gyp/pylib/gyp/generator/ninja_test.py index 1767b2f45a..1ad68e4fc9 100644 --- a/gyp/pylib/gyp/generator/ninja_test.py +++ b/gyp/pylib/gyp/generator/ninja_test.py @@ -8,7 +8,6 @@ import gyp.generator.ninja as ninja import unittest -import StringIO import sys import TestCommon From a2bca072f98767da56801b587315f41ccbe55904 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Wed, 24 Jul 2019 16:47:20 +0200 Subject: [PATCH 071/551] build: add test run Python 3.7 on macOS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Refs: https://github.com/nodejs/node-gyp/pull/1846 PR-URL: https://github.com/nodejs/node-gyp/pull/1843 Reviewed-By: João Reis --- .travis.yml | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index a6dd2dcf28..18b00d11a4 100644 --- a/.travis.yml +++ b/.travis.yml @@ -8,7 +8,7 @@ matrix: python: 2.7 - name: "Python 2.7 on macOS" os: osx - osx_image: xcode10.2 + osx_image: xcode11 language: shell # 'language: python' is not yet supported on macOS env: NODE_GYP_FORCE_PYTHON=python2 before_install: HOMEBREW_NO_AUTO_UPDATE=1 brew install npm @@ -28,6 +28,7 @@ matrix: PATH=/c/Python27:/c/Python27/Scripts:$PATH NODE_GYP_FORCE_PYTHON=/c/Python27/python.exe before_install: choco install python2 + - name: "Node.js 6 & Python 3.7 on Linux" python: 3.7 env: NODE_GYP_FORCE_PYTHON=python3 EXPERIMENTAL_NODE_GYP_PYTHON3=1 @@ -44,6 +45,12 @@ matrix: python: 3.7 env: NODE_GYP_FORCE_PYTHON=python3 EXPERIMENTAL_NODE_GYP_PYTHON3=1 before_install: nvm install 12 + - name: "Python 3.7 on macOS" + os: osx + #osx_image: xcode11 + language: shell # 'language: python' is not yet supported on macOS + env: NODE_GYP_FORCE_PYTHON=python3 EXPERIMENTAL_NODE_GYP_PYTHON3=1 + before_install: HOMEBREW_NO_AUTO_UPDATE=1 brew install npm - name: "Node.js 12 & Python 3.7 on Windows" os: windows language: node_js @@ -53,6 +60,7 @@ matrix: NODE_GYP_FORCE_PYTHON=/c/Python37/python.exe EXPERIMENTAL_NODE_GYP_PYTHON3=1 before_install: choco install python + allow_failures: - env: NODE_GYP_FORCE_PYTHON=python3 EXPERIMENTAL_NODE_GYP_PYTHON3=1 - env: >- From a2a862f6ba24d0f1f53fda059aa5bef8b52a5a6f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Tue, 23 Jul 2019 14:58:10 +0100 Subject: [PATCH 072/551] test: accept Python 3 in test-find-python.js Fixes: https://github.com/nodejs/node-gyp/issues/1826 PR-URL: https://github.com/nodejs/node-gyp/pull/1843 Reviewed-By: Christian Clauss Reviewed-By: Rod Vagg Reviewed-By: Matt Cowley Reviewed-By: Richard Lau --- .travis.yml | 3 ++- test/test-find-python.js | 15 ++++++++++----- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/.travis.yml b/.travis.yml index 18b00d11a4..4940dec0df 100644 --- a/.travis.yml +++ b/.travis.yml @@ -62,7 +62,8 @@ matrix: before_install: choco install python allow_failures: - - env: NODE_GYP_FORCE_PYTHON=python3 EXPERIMENTAL_NODE_GYP_PYTHON3=1 + - os: osx + env: NODE_GYP_FORCE_PYTHON=python3 EXPERIMENTAL_NODE_GYP_PYTHON3=1 - env: >- PATH=/c/Python37:/c/Python37/Scripts:$PATH NODE_GYP_FORCE_PYTHON=/c/Python37/python.exe diff --git a/test/test-find-python.js b/test/test-find-python.js index c52a579666..1c86f45b73 100644 --- a/test/test-find-python.js +++ b/test/test-find-python.js @@ -1,13 +1,12 @@ 'use strict' +delete process.env.PYTHON + const test = require('tap').test const findPython = require('../lib/find-python') const execFile = require('child_process').execFile const PythonFinder = findPython.test.PythonFinder -delete process.env.PYTHON -delete process.env.NODE_GYP_FORCE_PYTHON - require('npmlog').level = 'warn' test('find python', function (t) { @@ -17,8 +16,13 @@ test('find python', function (t) { t.strictEqual(err, null) var proc = execFile(found, ['-V'], function (err, stdout, stderr) { t.strictEqual(err, null) - t.strictEqual(stdout, '') - t.ok(/Python 2/.test(stderr)) + if (/Python 2/.test(stderr)) { + t.strictEqual(stdout, '') + t.ok(/Python 2/.test(stderr)) + } else { + t.ok(/Python 3/.test(stdout)) + t.strictEqual(stderr, '') + } }) proc.stdout.setEncoding('utf-8') proc.stderr.setEncoding('utf-8') @@ -51,6 +55,7 @@ TestPythonFinder.prototype.log = { warn: () => {}, error: () => {} } +delete TestPythonFinder.prototype.env.NODE_GYP_FORCE_PYTHON test('find python - python', function (t) { t.plan(6) From 2592036261ec588689e07072e1bef31993a9d7ac Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Tue, 23 Jul 2019 16:02:09 +0100 Subject: [PATCH 073/551] gyp: Python 3 Windows fixes PR-URL: https://github.com/nodejs/node-gyp/pull/1843 Reviewed-By: Christian Clauss Reviewed-By: Rod Vagg Reviewed-By: Matt Cowley Reviewed-By: Richard Lau --- .travis.yml | 4 ---- gyp/pylib/gyp/MSVSNew.py | 2 +- gyp/pylib/gyp/common.py | 3 +++ gyp/pylib/gyp/easy_xml.py | 4 ++-- gyp/pylib/gyp/generator/msvs.py | 8 ++++---- 5 files changed, 10 insertions(+), 11 deletions(-) diff --git a/.travis.yml b/.travis.yml index 4940dec0df..ab77f250af 100644 --- a/.travis.yml +++ b/.travis.yml @@ -64,10 +64,6 @@ matrix: allow_failures: - os: osx env: NODE_GYP_FORCE_PYTHON=python3 EXPERIMENTAL_NODE_GYP_PYTHON3=1 - - env: >- - PATH=/c/Python37:/c/Python37/Scripts:$PATH - NODE_GYP_FORCE_PYTHON=/c/Python37/python.exe - EXPERIMENTAL_NODE_GYP_PYTHON3=1 install: #- pip install -r requirements.txt - pip install flake8 # pytest # add another testing frameworks later diff --git a/gyp/pylib/gyp/MSVSNew.py b/gyp/pylib/gyp/MSVSNew.py index 0ec628cc1f..9b64e2c1c8 100644 --- a/gyp/pylib/gyp/MSVSNew.py +++ b/gyp/pylib/gyp/MSVSNew.py @@ -45,7 +45,7 @@ def MakeGuid(name, seed='msvs_new'): not change when the project for a target is rebuilt. """ # Calculate a MD5 signature for the seed and name. - d = hashlib.md5(str(seed) + str(name)).hexdigest().upper() + d = hashlib.md5((str(seed) + str(name)).encode('utf-8')).hexdigest().upper() # Convert most of the signature to GUID form (discard the rest) guid = ('{' + d[:8] + '-' + d[8:12] + '-' + d[12:16] + '-' + d[16:20] + '-' + d[20:32] + '}') diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py index 834a8e6c95..657134a78f 100644 --- a/gyp/pylib/gyp/common.py +++ b/gyp/pylib/gyp/common.py @@ -394,6 +394,9 @@ def close(self): os.unlink(self.tmp_path) raise + def write(self, s): + self.tmp_file.write(s.encode('utf-8')) + return Writer() diff --git a/gyp/pylib/gyp/easy_xml.py b/gyp/pylib/gyp/easy_xml.py index 7c3f621f1f..1ddd909175 100644 --- a/gyp/pylib/gyp/easy_xml.py +++ b/gyp/pylib/gyp/easy_xml.py @@ -120,7 +120,7 @@ def WriteXmlIfChanged(content, path, encoding='utf-8', pretty=False, default_encoding = locale.getdefaultlocale()[1] if default_encoding.upper() != encoding.upper(): - xml_string = xml_string.decode(default_encoding).encode(encoding) + xml_string = xml_string.encode(encoding) # Get the old content try: @@ -132,7 +132,7 @@ def WriteXmlIfChanged(content, path, encoding='utf-8', pretty=False, # It has changed, write it if existing != xml_string: - f = open(path, 'w') + f = open(path, 'wb') f.write(xml_string) f.close() diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index 2dc967d709..5e5b2ee52a 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -1753,8 +1753,8 @@ def _CollapseSingles(parent, node): # such projects up one level. if (type(node) == dict and len(node) == 1 and - node.keys()[0] == parent + '.vcproj'): - return node[node.keys()[0]] + list(node)[0] == parent + '.vcproj'): + return node[list(node)[0]] if type(node) != dict: return node for child in node: @@ -1773,8 +1773,8 @@ def _GatherSolutionFolders(sln_projects, project_objects, flat): # Walk down from the top until we hit a folder that has more than one entry. # In practice, this strips the top-level "src/" dir from the hierarchy in # the solution. - while len(root) == 1 and type(root[root.keys()[0]]) == dict: - root = root[root.keys()[0]] + while len(root) == 1 and type(root[list(root)[0]]) == dict: + root = root[list(root)[0]] # Collapse singles. root = _CollapseSingles('', root) # Merge buckets until everything is a root entry. From c7f1bcaff5dddd6f137f28576ff793f9afe26626 Mon Sep 17 00:00:00 2001 From: Jose Quijada Date: Wed, 10 Jul 2019 07:47:22 -0400 Subject: [PATCH 074/551] gyp: improve Windows+Cygwin compatibility Fixes: #1782 PR-URL: https://github.com/nodejs/node-gyp/pull/1817 Reviewed-By: Christian Clauss Reviewed-By: Rod Vagg --- gyp/gyp_main.py | 32 ++++++++++++++++++++++++++++++-- gyp/pylib/gyp/common.py | 19 ++++++++++++++++++- gyp/pylib/gyp/generator/msvs.py | 12 +++++++++++- lib/configure.js | 7 +++++++ 4 files changed, 66 insertions(+), 4 deletions(-) diff --git a/gyp/gyp_main.py b/gyp/gyp_main.py index 25a6eba94a..8dfc552e81 100755 --- a/gyp/gyp_main.py +++ b/gyp/gyp_main.py @@ -6,10 +6,38 @@ import os import sys +import subprocess + +# Below IsCygwin() function copied from pylib/gyp/common.py +def IsCygwin(): + try: + out = subprocess.Popen("uname", + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT) + stdout,stderr = out.communicate() + return "CYGWIN" in str(stdout) + except Exception: + return False + + +def UnixifyPath(path): + try: + if not IsCygwin(): + return path + out = subprocess.Popen(["cygpath", "-u", path], + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT) + stdout,stderr = out.communicate() + return str(stdout) + except Exception: + return path + # Make sure we're using the version of pylib in this repo, not one installed -# elsewhere on the system. -sys.path.insert(0, os.path.join(os.path.dirname(sys.argv[0]), 'pylib')) +# elsewhere on the system. Also convert to Unix style path on Cygwin systems, +# else the 'gyp' library will not be found +path = UnixifyPath(sys.argv[0]) +sys.path.insert(0, os.path.join(os.path.dirname(path), 'pylib')) import gyp if __name__ == '__main__': diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py index 657134a78f..8296d11fc3 100644 --- a/gyp/pylib/gyp/common.py +++ b/gyp/pylib/gyp/common.py @@ -9,6 +9,7 @@ import re import tempfile import sys +import subprocess # A minimal memoizing decorator. It'll blow up if the args aren't immutable, @@ -337,11 +338,16 @@ def WriteOnDiff(filename): class Writer(object): """Wrapper around file which only covers the target if it differs.""" def __init__(self): + # On Cygwin remove the "dir" argument because `C:` prefixed paths are treated as relative, + # consequently ending up with current dir "/cygdrive/c/..." being prefixed to those, which was + # obviously a non-existent path, for example: "/cygdrive/c//C:\". + # See https://docs.python.org/2/library/tempfile.html#tempfile.mkstemp for more details + base_temp_dir = "" if IsCygwin() else os.path.dirname(filename) # Pick temporary file. tmp_fd, self.tmp_path = tempfile.mkstemp( suffix='.tmp', prefix=os.path.split(filename)[1] + '.gyp.', - dir=os.path.split(filename)[0]) + dir=base_temp_dir) try: self.tmp_file = os.fdopen(tmp_fd, 'wb') except Exception: @@ -611,3 +617,14 @@ def CrossCompileRequested(): os.environ.get('AR_target') or os.environ.get('CC_target') or os.environ.get('CXX_target')) + +def IsCygwin(): + try: + out = subprocess.Popen("uname", + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT) + stdout,stderr = out.communicate() + return "CYGWIN" in str(stdout) + except Exception: + return False + diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index 5e5b2ee52a..eb9730895f 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -165,7 +165,7 @@ def _FixPath(path): Returns: The path with all slashes made into backslashes. """ - if fixpath_prefix and path and not os.path.isabs(path) and not path[0] == '$': + if fixpath_prefix and path and not os.path.isabs(path) and not path[0] == '$' and not _IsWindowsAbsPath(path): path = os.path.join(fixpath_prefix, path) path = path.replace('/', '\\') path = _NormalizedSource(path) @@ -174,6 +174,16 @@ def _FixPath(path): return path +def _IsWindowsAbsPath(path): + """ + On Cygwin systems Python needs a little help determining if a path is an absolute Windows path or not, so that + it does not treat those as relative, which results in bad paths like: + + '..\C:\\some_source_code_file.cc' + """ + return path.startswith('c:') or path.startswith('C:') + + def _FixPaths(paths): """Fix each of the paths of the list.""" return [_FixPath(i) for i in paths] diff --git a/lib/configure.js b/lib/configure.js index 267e7587f5..20b955cabc 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -295,6 +295,7 @@ function configure (gyp, argv, callback) { outputDir = buildDir } var nodeGypDir = path.resolve(__dirname, '..') + var nodeLibFile = path.join(nodeDir, !gyp.opts.nodedir ? '<(target_arch)' : '$(Configuration)', release.name + '.lib') @@ -308,6 +309,12 @@ function configure (gyp, argv, callback) { argv.push('-Dnode_exp_file=' + nodeExpFile) } argv.push('-Dnode_gyp_dir=' + nodeGypDir) + + // Do this to keep Cygwin environments happy, else the unescaped '\' gets eaten up, + // resulting in bad paths, Ex c:parentFolderfolderanotherFolder instead of c:\parentFolder\folder\anotherFolder + if (win) { + nodeLibFile = nodeLibFile.replace(/\\/g, '\\\\') + } argv.push('-Dnode_lib_file=' + nodeLibFile) argv.push('-Dmodule_root_dir=' + process.cwd()) argv.push('-Dnode_engine=' + From af876e10f01ea8e3fdfeee20dbee3f7138ccffd5 Mon Sep 17 00:00:00 2001 From: Peter Sabath <31621424+peter-sabath@users.noreply.github.com> Date: Thu, 8 Aug 2019 14:37:34 +0200 Subject: [PATCH 075/551] src,win: improve unmanaged handling MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Used scoped disabling of managed code handling to ensure no other files get affected. PR-URL: https://github.com/nodejs/node-gyp/pull/1852 Reviewed-By: João Reis --- src/win_delay_load_hook.cc | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/win_delay_load_hook.cc b/src/win_delay_load_hook.cc index 567c4cc8d7..169f8029f1 100644 --- a/src/win_delay_load_hook.cc +++ b/src/win_delay_load_hook.cc @@ -7,10 +7,10 @@ * This allows compiled addons to work when the host executable is renamed. */ -#pragma unmanaged - #ifdef _MSC_VER +#pragma managed(push, off) + #ifndef WIN32_LEAN_AND_MEAN #define WIN32_LEAN_AND_MEAN #endif @@ -34,4 +34,6 @@ static FARPROC WINAPI load_exe_hook(unsigned int event, DelayLoadInfo* info) { decltype(__pfnDliNotifyHook2) __pfnDliNotifyHook2 = load_exe_hook; +#pragma managed(pop) + #endif From a301abcde1fcfce1ffc2bac3a0c0f6fadc7a1de4 Mon Sep 17 00:00:00 2001 From: Vladyslav Burzakovskyy Date: Mon, 19 Aug 2019 12:16:16 +0200 Subject: [PATCH 076/551] gyp: use "is" when comparing to None PR-URL: https://github.com/nodejs/node-gyp/pull/1860 Reviewed-By: Christian Clauss Reviewed-By: Rod Vagg --- gyp/pylib/gyp/generator/ninja.py | 2 +- gyp/pylib/gyp/generator/xcode.py | 2 +- gyp/pylib/gyp/input.py | 8 ++++---- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py index 4b5122af1d..aa94c9c203 100644 --- a/gyp/pylib/gyp/generator/ninja.py +++ b/gyp/pylib/gyp/generator/ninja.py @@ -726,7 +726,7 @@ def cygwin_munge(path): elif var == 'name': extra_bindings.append(('name', cygwin_munge(basename))) else: - assert var == None, repr(var) + assert var is None, repr(var) outputs = [self.GypPathToNinja(o, env) for o in outputs] if self.flavor == 'win': diff --git a/gyp/pylib/gyp/generator/xcode.py b/gyp/pylib/gyp/generator/xcode.py index 694a28afb1..14ce7473c2 100644 --- a/gyp/pylib/gyp/generator/xcode.py +++ b/gyp/pylib/gyp/generator/xcode.py @@ -539,7 +539,7 @@ def ExpandXcodeVariables(string, expansions): """ matches = _xcode_variable_re.findall(string) - if matches == None: + if matches is None: return string matches.reverse() diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index dde2823a9f..b2eb7d3443 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -155,7 +155,7 @@ def GetIncludedBuildFiles(build_file_path, aux_data, included=None): in the list will be relative to the current directory. """ - if included == None: + if included is None: included = [] if build_file_path in included: @@ -1062,7 +1062,7 @@ def EvalCondition(condition, conditions_key, phase, variables, build_file): else: false_dict = None i = i + 2 - if result == None: + if result is None: result = EvalSingleCondition( cond_expr, true_dict, false_dict, phase, variables, build_file) @@ -1603,7 +1603,7 @@ def Visit(node, path): def DirectDependencies(self, dependencies=None): """Returns a list of just direct dependencies.""" - if dependencies == None: + if dependencies is None: dependencies = [] for dependency in self.dependencies: @@ -1631,7 +1631,7 @@ def _AddImportedDependencies(self, targets, dependencies=None): public entry point. """ - if dependencies == None: + if dependencies is None: dependencies = [] index = 0 From cdb47bd54d4487270235319bfba0551b20546368 Mon Sep 17 00:00:00 2001 From: XhmikosR Date: Sat, 10 Aug 2019 09:03:45 +0300 Subject: [PATCH 077/551] gyp: assorted typo fixes PR-URL: https://github.com/nodejs/node-gyp/pull/1853 Reviewed-By: Christian Clauss Reviewed-By: Richard Lau Reviewed-By: Rich Trott --- gyp/pylib/gyp/generator/android.py | 2 +- gyp/pylib/gyp/generator/cmake.py | 2 +- gyp/pylib/gyp/generator/eclipse.py | 2 +- gyp/pylib/gyp/generator/make.py | 2 +- gyp/pylib/gyp/generator/msvs.py | 2 +- gyp/pylib/gyp/generator/ninja.py | 2 +- gyp/pylib/gyp/generator/xcode.py | 2 +- gyp/pylib/gyp/input.py | 6 +++--- gyp/pylib/gyp/msvs_emulation.py | 2 +- gyp/pylib/gyp/xcode_emulation.py | 16 ++++++++-------- gyp/pylib/gyp/xcodeproj_file.py | 2 +- gyp/tools/pretty_gyp.py | 2 +- gyp/tools/pretty_vcproj.py | 2 +- 13 files changed, 22 insertions(+), 22 deletions(-) diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py index b7f9842888..cecb28c366 100644 --- a/gyp/pylib/gyp/generator/android.py +++ b/gyp/pylib/gyp/generator/android.py @@ -550,7 +550,7 @@ def WriteSources(self, spec, configs, extra_sources): # to work properly. If the file has the wrong C++ extension, then we add # a rule to copy that to intermediates and use the new version. final_generated_sources = [] - # If a source file gets copied, we still need to add the orginal source + # If a source file gets copied, we still need to add the original source # directory as header search path, for GCC searches headers in the # directory that contains the source file by default. origin_src_dirs = [] diff --git a/gyp/pylib/gyp/generator/cmake.py b/gyp/pylib/gyp/generator/cmake.py index 7aabddb633..60cce6228b 100644 --- a/gyp/pylib/gyp/generator/cmake.py +++ b/gyp/pylib/gyp/generator/cmake.py @@ -574,7 +574,7 @@ class CMakeNamer(object): """Converts Gyp target names into CMake target names. CMake requires that target names be globally unique. One way to ensure - this is to fully qualify the names of the targets. Unfortunatly, this + this is to fully qualify the names of the targets. Unfortunately, this ends up with all targets looking like "chrome_chrome_gyp_chrome" instead of just "chrome". If this generator were only interested in building, it would be possible to fully qualify all target names, then create diff --git a/gyp/pylib/gyp/generator/eclipse.py b/gyp/pylib/gyp/generator/eclipse.py index 372ceec246..6b49ad6760 100644 --- a/gyp/pylib/gyp/generator/eclipse.py +++ b/gyp/pylib/gyp/generator/eclipse.py @@ -195,7 +195,7 @@ def GetAllDefines(target_list, target_dicts, data, config_name, params, """Calculate the defines for a project. Returns: - A dict that includes explict defines declared in gyp files along with all of + A dict that includes explicit defines declared in gyp files along with all of the default defines that the compiler uses. """ diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index 385a0f7a08..95f0b433c5 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -12,7 +12,7 @@ # all are sourced by the top-level Makefile. This means that all # variables in .mk-files clobber one another. Be careful to use := # where appropriate for immediate evaluation, and similarly to watch -# that you're not relying on a variable value to last beween different +# that you're not relying on a variable value to last between different # .mk files. # # TODOs: diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index eb9730895f..b3b4bc1053 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -3069,7 +3069,7 @@ def _FinalizeMSBuildSettings(spec, configuration): _ToolAppend(msbuild_settings, 'ResourceCompile', 'AdditionalIncludeDirectories', resource_include_dirs) # Add in libraries, note that even for empty libraries, we want this - # set, to prevent inheriting default libraries from the enviroment. + # set, to prevent inheriting default libraries from the environment. _ToolSetOrAppend(msbuild_settings, 'Link', 'AdditionalDependencies', libraries) _ToolAppend(msbuild_settings, 'Link', 'AdditionalLibraryDirectories', diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py index aa94c9c203..33cc253aba 100644 --- a/gyp/pylib/gyp/generator/ninja.py +++ b/gyp/pylib/gyp/generator/ninja.py @@ -1821,7 +1821,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, # - The priority from low to high is gcc/g++, the 'make_global_settings' in # gyp, the environment variable. # - If there is no 'make_global_settings' for CC.host/CXX.host or - # 'CC_host'/'CXX_host' enviroment variable, cc_host/cxx_host should be set + # 'CC_host'/'CXX_host' environment variable, cc_host/cxx_host should be set # to cc/cxx. if flavor == 'win': ar = 'lib.exe' diff --git a/gyp/pylib/gyp/generator/xcode.py b/gyp/pylib/gyp/generator/xcode.py index 14ce7473c2..f9aeb45f68 100644 --- a/gyp/pylib/gyp/generator/xcode.py +++ b/gyp/pylib/gyp/generator/xcode.py @@ -998,7 +998,7 @@ def GenerateOutput(target_list, target_dicts, data, params): actions.append(action) if len(concrete_outputs_all) > 0: - # TODO(mark): There's a possibilty for collision here. Consider + # TODO(mark): There's a possibility for collision here. Consider # target "t" rule "A_r" and target "t_A" rule "r". makefile_name = '%s.make' % re.sub( '[^a-zA-Z0-9_]', '_' , '%s_%s' % (target_name, rule['rule_name'])) diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index b2eb7d3443..aba81cf654 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -1073,7 +1073,7 @@ def EvalSingleCondition( cond_expr, true_dict, false_dict, phase, variables, build_file): """Returns true_dict if cond_expr evaluates to true, and false_dict otherwise.""" - # Do expansions on the condition itself. Since the conditon can naturally + # Do expansions on the condition itself. Since the condition can naturally # contain variable references without needing to resort to GYP expansion # syntax, this is of dubious value for variables, but someone might want to # use a command expansion directly inside a condition. @@ -1178,7 +1178,7 @@ def LoadVariablesFromVariablesDict(variables, the_dict, the_dict_key): continue if the_dict_key == 'variables' and variable_name in the_dict: # If the variable is set without a % in the_dict, and the_dict is a - # variables dict (making |variables| a varaibles sub-dict of a + # variables dict (making |variables| a variables sub-dict of a # variables dict), use the_dict's definition. value = the_dict[variable_name] else: @@ -1864,7 +1864,7 @@ def VerifyNoGYPFileCircularDependencies(targets): continue dependency_node = dependency_nodes.get(dependency_build_file) if not dependency_node: - raise GypError("Dependancy '%s' not found" % dependency_build_file) + raise GypError("Dependency '%s' not found" % dependency_build_file) if dependency_node not in build_file_node.dependencies: build_file_node.dependencies.append(dependency_node) dependency_node.dependents.append(build_file_node) diff --git a/gyp/pylib/gyp/msvs_emulation.py b/gyp/pylib/gyp/msvs_emulation.py index 4a50b1b74c..9d50bad1f5 100644 --- a/gyp/pylib/gyp/msvs_emulation.py +++ b/gyp/pylib/gyp/msvs_emulation.py @@ -375,7 +375,7 @@ def GetCompilerPdbName(self, config, expand_special): return pdbname def GetMapFileName(self, config, expand_special): - """Gets the explicitly overriden map file name for a target or returns None + """Gets the explicitly overridden map file name for a target or returns None if it's not set.""" config = self._TargetConfig(config) map_file = self._Setting(('VCLinkerTool', 'MapFileName'), config) diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py index 0162d978bf..faf00a82a5 100644 --- a/gyp/pylib/gyp/xcode_emulation.py +++ b/gyp/pylib/gyp/xcode_emulation.py @@ -778,7 +778,7 @@ def GetLdflags(self, configname, product_dir, gyp_to_build_path, arch=None): product_dir: The directory where products such static and dynamic libraries are placed. This is added to the library search path. gyp_to_build_path: A function that converts paths relative to the - current gyp file to paths relative to the build direcotry. + current gyp file to paths relative to the build directory. """ self.configname = configname ldflags = [] @@ -923,7 +923,7 @@ def GetPerTargetSetting(self, setting, default=None): def _GetStripPostbuilds(self, configname, output_binary, quiet): """Returns a list of shell commands that contain the shell commands - neccessary to strip this target's binary. These should be run as postbuilds + necessary to strip this target's binary. These should be run as postbuilds before the actual postbuilds run.""" self.configname = configname @@ -957,7 +957,7 @@ def _GetStripPostbuilds(self, configname, output_binary, quiet): def _GetDebugInfoPostbuilds(self, configname, output, output_binary, quiet): """Returns a list of shell commands that contain the shell commands - neccessary to massage this target's debug information. These should be run + necessary to massage this target's debug information. These should be run as postbuilds before the actual postbuilds run.""" self.configname = configname @@ -1052,7 +1052,7 @@ def _AdjustLibrary(self, library, config_name=None): # "/usr/lib" libraries, is do "-L/usr/lib -lname" which is dependent on the # library order and cause collision when building Chrome. # - # Instead substitude ".tbd" to ".dylib" in the generated project when the + # Instead substitute ".tbd" to ".dylib" in the generated project when the # following conditions are both true: # - library is referenced in the gyp file as "$(SDKROOT)/**/*.dylib", # - the ".dylib" file does not exists but a ".tbd" file do. @@ -1341,7 +1341,7 @@ def GetStdout(cmdlist): def MergeGlobalXcodeSettingsToSpec(global_dict, spec): """Merges the global xcode_settings dictionary into each configuration of the target represented by spec. For keys that are both in the global and the local - xcode_settings dict, the local key gets precendence. + xcode_settings dict, the local key gets precedence. """ # The xcode generator special-cases global xcode_settings and does something # that amounts to merging in the global xcode_settings into each local @@ -1384,7 +1384,7 @@ def GetMacBundleResources(product_dir, xcode_settings, resources): output = dest # The make generator doesn't support it, so forbid it everywhere - # to keep the generators more interchangable. + # to keep the generators more interchangeable. assert ' ' not in res, ( "Spaces in resource filenames not supported (%s)" % res) @@ -1426,14 +1426,14 @@ def GetMacInfoPlist(product_dir, xcode_settings, gyp_path_to_build_path): relative to the build directory. xcode_settings: The XcodeSettings of the current target. gyp_to_build_path: A function that converts paths relative to the - current gyp file to paths relative to the build direcotry. + current gyp file to paths relative to the build directory. """ info_plist = xcode_settings.GetPerTargetSetting('INFOPLIST_FILE') if not info_plist: return None, None, [], {} # The make generator doesn't support it, so forbid it everywhere - # to keep the generators more interchangable. + # to keep the generators more interchangeable. assert ' ' not in info_plist, ( "Spaces in Info.plist filenames not supported (%s)" % info_plist) diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py index b0385468c5..93ffca7c90 100644 --- a/gyp/pylib/gyp/xcodeproj_file.py +++ b/gyp/pylib/gyp/xcodeproj_file.py @@ -220,7 +220,7 @@ class XCObject(object): an empty string ("", in the case of property_type str) or list ([], in the case of is_list True) from being set for the property. - default: Optional. If is_requried is True, default may be set + default: Optional. If is_required is True, default may be set to provide a default value for objects that do not supply their own value. If is_required is True and default is not provided, users of the class must supply their own diff --git a/gyp/tools/pretty_gyp.py b/gyp/tools/pretty_gyp.py index d01c692edc..633048a59a 100755 --- a/gyp/tools/pretty_gyp.py +++ b/gyp/tools/pretty_gyp.py @@ -18,7 +18,7 @@ # Regex to remove quoted strings when we're counting braces. # It takes into account quoted quotes, and makes sure that the quotes match. # NOTE: It does not handle quotes that span more than one line, or -# cases where an escaped quote is preceeded by an escaped backslash. +# cases where an escaped quote is preceded by an escaped backslash. QUOTE_RE_STR = r'(?P[\'"])(.*?)(? Date: Thu, 15 Aug 2019 18:24:52 +0100 Subject: [PATCH 078/551] gyp: rm semicolons (Python != JavaScript) PR-URL: https://github.com/nodejs/node-gyp/pull/1858 Reviewed-By: Christian Clauss Reviewed-By: Rod Vagg --- gyp/pylib/gyp/common_test.py | 8 ++++---- gyp/pylib/gyp/generator/analyzer.py | 4 ++-- gyp/pylib/gyp/generator/cmake.py | 2 +- gyp/pylib/gyp/generator/make.py | 12 ++++++------ gyp/pylib/gyp/generator/xcode.py | 8 ++++---- 5 files changed, 17 insertions(+), 17 deletions(-) diff --git a/gyp/pylib/gyp/common_test.py b/gyp/pylib/gyp/common_test.py index ad6f9a1438..b75bbb8412 100755 --- a/gyp/pylib/gyp/common_test.py +++ b/gyp/pylib/gyp/common_test.py @@ -59,10 +59,10 @@ def test_platform_default(self): self.assertFlavor('freebsd', 'freebsd9' , {}) self.assertFlavor('freebsd', 'freebsd10', {}) self.assertFlavor('openbsd', 'openbsd5' , {}) - self.assertFlavor('solaris', 'sunos5' , {}); - self.assertFlavor('solaris', 'sunos' , {}); - self.assertFlavor('linux' , 'linux2' , {}); - self.assertFlavor('linux' , 'linux3' , {}); + self.assertFlavor('solaris', 'sunos5' , {}) + self.assertFlavor('solaris', 'sunos' , {}) + self.assertFlavor('linux' , 'linux2' , {}) + self.assertFlavor('linux' , 'linux3' , {}) def test_param(self): self.assertFlavor('foobar', 'linux2' , {'flavor': 'foobar'}) diff --git a/gyp/pylib/gyp/generator/analyzer.py b/gyp/pylib/gyp/generator/analyzer.py index 0416b5d9be..59d73dbedb 100644 --- a/gyp/pylib/gyp/generator/analyzer.py +++ b/gyp/pylib/gyp/generator/analyzer.py @@ -615,7 +615,7 @@ def _supplied_target_names(self): def _supplied_target_names_no_all(self): """Returns the supplied test targets without 'all'.""" - result = self._supplied_target_names(); + result = self._supplied_target_names() result.discard('all') return result @@ -668,7 +668,7 @@ def find_matching_test_target_names(self): def find_matching_compile_target_names(self): """Returns the set of output compile targets.""" - assert self.is_build_impacted(); + assert self.is_build_impacted() # Compile targets are found by searching up from changed targets. # Reset the visited status for _GetBuildTargets. for target in self._name_to_target.values(): diff --git a/gyp/pylib/gyp/generator/cmake.py b/gyp/pylib/gyp/generator/cmake.py index 60cce6228b..996b6f25fd 100644 --- a/gyp/pylib/gyp/generator/cmake.py +++ b/gyp/pylib/gyp/generator/cmake.py @@ -681,7 +681,7 @@ def WriteTarget(namer, qualified_target, target_dicts, build_dir, config_to_use, for src in srcs: _, ext = os.path.splitext(src) src_type = COMPILABLE_EXTENSIONS.get(ext, None) - src_norm_path = NormjoinPath(path_from_cmakelists_to_gyp, src); + src_norm_path = NormjoinPath(path_from_cmakelists_to_gyp, src) if src_type == 's': s_sources.append(src_norm_path) diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index 95f0b433c5..bdf7134537 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -1225,16 +1225,16 @@ def WriteSources(self, configs, deps, sources, cflags_c = config.get('cflags_c') cflags_cc = config.get('cflags_cc') - self.WriteLn("# Flags passed to all source files."); + self.WriteLn("# Flags passed to all source files.") self.WriteList(cflags, 'CFLAGS_%s' % configname) - self.WriteLn("# Flags passed to only C files."); + self.WriteLn("# Flags passed to only C files.") self.WriteList(cflags_c, 'CFLAGS_C_%s' % configname) - self.WriteLn("# Flags passed to only C++ files."); + self.WriteLn("# Flags passed to only C++ files.") self.WriteList(cflags_cc, 'CFLAGS_CC_%s' % configname) if self.flavor == 'mac': - self.WriteLn("# Flags passed to only ObjC files."); + self.WriteLn("# Flags passed to only ObjC files.") self.WriteList(cflags_objc, 'CFLAGS_OBJC_%s' % configname) - self.WriteLn("# Flags passed to only ObjC++ files."); + self.WriteLn("# Flags passed to only ObjC++ files.") self.WriteList(cflags_objcc, 'CFLAGS_OBJCC_%s' % configname) includes = config.get('include_dirs') if includes: @@ -1779,7 +1779,7 @@ def WriteMakeRule(self, outputs, inputs, actions=None, comment=None, cmddigest = hashlib.sha1(command if command else self.target).hexdigest() intermediate = "%s.intermediate" % cmddigest self.WriteLn('%s: %s' % (' '.join(outputs), intermediate)) - self.WriteLn('\t%s' % '@:'); + self.WriteLn('\t%s' % '@:') self.WriteLn('%s: %s' % ('.INTERMEDIATE', intermediate)) self.WriteLn('%s: %s%s' % (intermediate, ' '.join(inputs), force_append)) diff --git a/gyp/pylib/gyp/generator/xcode.py b/gyp/pylib/gyp/generator/xcode.py index f9aeb45f68..6317d04c70 100644 --- a/gyp/pylib/gyp/generator/xcode.py +++ b/gyp/pylib/gyp/generator/xcode.py @@ -246,7 +246,7 @@ def Finalize1(self, xcode_targets, serialize_all_tests): targets_for_all.append(xcode_target) if target_name.lower() == 'all': - has_custom_all = True; + has_custom_all = True # If this target has a 'run_as' attribute, add its target to the # targets, and add it to the test targets. @@ -637,7 +637,7 @@ def GenerateOutput(target_list, target_dicts, data, params): pbxp = xcp.project # Set project-level attributes from multiple options - project_attributes = {}; + project_attributes = {} if parallel_builds: project_attributes['BuildIndependentTargetsInParallel'] = 'YES' if upgrade_check_project_version: @@ -776,7 +776,7 @@ def GenerateOutput(target_list, target_dicts, data, params): # logic all happens in ninja. Don't bother creating the extra targets in # that case. if type != 'none' and (spec_actions or spec_rules) and not ninja_wrapper: - support_xccl = CreateXCConfigurationList(configuration_names); + support_xccl = CreateXCConfigurationList(configuration_names) support_target_suffix = generator_flags.get( 'support_target_suffix', ' Support') support_target_properties = { @@ -1171,7 +1171,7 @@ def GenerateOutput(target_list, target_dicts, data, params): dest = '$(SRCROOT)/' + dest code_sign = int(copy_group.get('xcode_code_sign', 0)) - settings = (None, '{ATTRIBUTES = (CodeSignOnCopy, ); }')[code_sign]; + settings = (None, '{ATTRIBUTES = (CodeSignOnCopy, ); }')[code_sign] # Coalesce multiple "copies" sections in the same target with the same # "destination" property into the same PBXCopyFilesBuildPhase, otherwise From ca990a12921a447e859419132ed479e0236381dd Mon Sep 17 00:00:00 2001 From: lagorsse Date: Fri, 28 Jun 2019 18:44:20 +0200 Subject: [PATCH 079/551] doc: fix missing argument for setting python path (accepted with minor modifications by rvagg) PR-URL: https://github.com/nodejs/node-gyp/pull/1802 Reviewed-By: Rod Vagg --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b0900028d7..61a0ed062b 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,7 @@ If you have multiple Python versions installed, you can identify which Python version `node-gyp` uses by setting the `--python` variable: ``` bash -$ node-gyp --python /path/to/python2.7 +$ node-gyp --python /path/to/executable/python2.7 ``` If `node-gyp` is called by way of `npm`, *and* you have multiple versions of From c4002dee1ab9636e8278bc2f0759d19da221ed43 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Sat, 20 Jul 2019 15:03:56 +1000 Subject: [PATCH 080/551] lib: ignore non-critical os.userInfo() failures Fixes: https://github.com/nodejs/node-gyp/issues/1834 PR-URL: https://github.com/nodejs/node-gyp/pull/1835 --- lib/install.js | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/install.js b/lib/install.js index 6ac8519ef5..83a322debc 100644 --- a/lib/install.js +++ b/lib/install.js @@ -370,7 +370,12 @@ function install (fs, gyp, argv, callback) { } var tmpdir = os.tmpdir() gyp.devDir = path.resolve(tmpdir, '.node-gyp') - log.warn('EACCES', 'user "%s" does not have permission to access the dev dir "%s"', os.userInfo().username, devDir) + var userString = '' + try { + // os.userInfo can fail on some systems, it's not critical here + userString = ` ("${os.userInfo().username}")` + } catch (e) {} + log.warn('EACCES', 'current user%s does not have permission to access the dev dir "%s"', userString, devDir) log.warn('EACCES', 'attempting to reinstall using temporary dev dir "%s"', gyp.devDir) if (process.cwd() === tmpdir) { log.verbose('tmpdir == cwd', 'automatically will remove dev files after to save disk space') From 038468388cb32d029ccd8c1fe36c7ab7839fba23 Mon Sep 17 00:00:00 2001 From: Milad Farazmand Date: Fri, 23 Aug 2019 11:20:24 -0400 Subject: [PATCH 081/551] lib: adding keep-alive header to download requests PR-URL: https://github.com/nodejs/node-gyp/pull/1863 Reviewed-By: Rod Vagg Reviewed-By: Richard Lau --- lib/install.js | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/install.js b/lib/install.js index 83a322debc..3730c0965f 100644 --- a/lib/install.js +++ b/lib/install.js @@ -391,7 +391,8 @@ function download (gyp, env, url) { var requestOpts = { uri: url, headers: { - 'User-Agent': 'node-gyp v' + gyp.version + ' (node ' + process.version + ')' + 'User-Agent': 'node-gyp v' + gyp.version + ' (node ' + process.version + ')', + 'Connection': 'keep-alive' } } From cdc49ee3f61b81a51fd5a768aad1ceb02bceac2c Mon Sep 17 00:00:00 2001 From: Halit Ogunc Date: Tue, 24 Sep 2019 18:32:34 +0200 Subject: [PATCH 082/551] bin: fix the usage instructions PR-URL: https://github.com/nodejs/node-gyp/pull/1888 Reviewed-By: Rod Vagg --- lib/build.js | 3 +-- lib/list.js | 2 +- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/build.js b/lib/build.js index 24cfe38de2..c2388fb348 100644 --- a/lib/build.js +++ b/lib/build.js @@ -7,8 +7,6 @@ const log = require('npmlog') const which = require('which') const win = process.platform === 'win32' -exports.usage = 'Invokes `' + (win ? 'msbuild' : 'make') + '` and builds the module' - function build (gyp, argv, callback) { var platformMake = 'make' if (process.platform === 'aix') { @@ -203,3 +201,4 @@ function build (gyp, argv, callback) { } module.exports = build +module.exports.usage = 'Invokes `' + (win ? 'msbuild' : 'make') + '` and builds the module' diff --git a/lib/list.js b/lib/list.js index 3e5de28851..405ebc0d88 100644 --- a/lib/list.js +++ b/lib/list.js @@ -24,4 +24,4 @@ function list (gyp, args, callback) { } module.exports = list -exports.usage = 'Prints a listing of the currently installed node development files' +module.exports.usage = 'Prints a listing of the currently installed node development files' From 5d76938e55b17d5633bb9d4f71b7f73034848505 Mon Sep 17 00:00:00 2001 From: Matheus Marchini Date: Tue, 24 Sep 2019 10:47:29 -0700 Subject: [PATCH 083/551] deps: update tar to 4.4.12 Fixes: https://github.com/nodejs/node-gyp/issues/1887 PR-URL: https://github.com/nodejs/node-gyp/pull/1889 Reviewed-By: Rod Vagg Reviewed-By: Richard Lau --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index 4eba267b07..fe68798faf 100644 --- a/package.json +++ b/package.json @@ -31,7 +31,7 @@ "request": "^2.87.0", "rimraf": "2", "semver": "~5.3.0", - "tar": "^4.4.8", + "tar": "^4.4.12", "which": "1" }, "engines": { From 8d4ea7f13e96419ad4cab86ea18063593a7abd67 Mon Sep 17 00:00:00 2001 From: Nhan Khong <49646896+ktrongnhan@users.noreply.github.com> Date: Sat, 21 Sep 2019 07:22:53 -0700 Subject: [PATCH 084/551] doc: update xcode install instructions to match Node's BUILDING PR-URL: https://github.com/nodejs/node-gyp/pull/1884 Reviewed-By: Rod Vagg --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index 61a0ed062b..0c76ffc09b 100644 --- a/README.md +++ b/README.md @@ -35,8 +35,7 @@ You will also need to install: * `python` (`v2.7` recommended, `v3.x.x` is __*not*__ supported) (already installed on macOS) * [Xcode](https://developer.apple.com/xcode/download/) - * You also need to install the `Command Line Tools` via Xcode. You can find this under the menu `Xcode -> Preferences -> Locations` (or by running `xcode-select --install` in your Terminal) - * This step will install `gcc` and the related toolchain containing `make` + * You also need to install the `XCode Command Line Tools` by running `xcode-select --install`. Alternatively, if you already have the full Xcode installed, you can find them under the menu `Xcode -> Open Developer Tool -> More Developer Tools...`. This step will install `clang`, `clang++`, and `make`. ### On Windows From d90d9c54269e1d65b82f2fcd6967e3016ba86e85 Mon Sep 17 00:00:00 2001 From: cclauss Date: Wed, 25 Sep 2019 19:52:35 +0200 Subject: [PATCH 085/551] gyp: decode stdout on Python 3 PR-URL: https://github.com/nodejs/node-gyp/pull/1890 Reviewed-By: Ben Noordhuis Reviewed-By: Richard Lau --- .travis.yml | 3 --- gyp/pylib/gyp/xcode_emulation.py | 8 +++++++- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/.travis.yml b/.travis.yml index ab77f250af..f27f024f2d 100644 --- a/.travis.yml +++ b/.travis.yml @@ -61,9 +61,6 @@ matrix: EXPERIMENTAL_NODE_GYP_PYTHON3=1 before_install: choco install python - allow_failures: - - os: osx - env: NODE_GYP_FORCE_PYTHON=python3 EXPERIMENTAL_NODE_GYP_PYTHON3=1 install: #- pip install -r requirements.txt - pip install flake8 # pytest # add another testing frameworks later diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py index faf00a82a5..de376edb07 100644 --- a/gyp/pylib/gyp/xcode_emulation.py +++ b/gyp/pylib/gyp/xcode_emulation.py @@ -20,6 +20,8 @@ import tempfile from gyp.common import GypError +PY3 = bytes != str + # Populated lazily by XcodeVersion, for efficiency, and to fix an issue when # "xcodebuild" is called too quickly (it has been found to return incorrect # version number). @@ -1277,7 +1279,7 @@ def XcodeVersion(): except: version = CLTVersion() if version: - version = re.match(r'(\d+\.\d+\.?\d*)', version).groups()[0] + version = ".".join(version.split(".")[:3]) else: raise GypError("No Xcode or CLT version detected!") # The CLT has no build information, so we return an empty string. @@ -1322,6 +1324,8 @@ def GetStdoutQuiet(cmdlist): Raises |GypError| if the command return with a non-zero return code.""" job = subprocess.Popen(cmdlist, stdout=subprocess.PIPE, stderr=subprocess.PIPE) out = job.communicate()[0] + if PY3: + out = out.decode("utf-8") if job.returncode != 0: raise GypError('Error %d running %s' % (job.returncode, cmdlist[0])) return out.rstrip('\n') @@ -1332,6 +1336,8 @@ def GetStdout(cmdlist): Raises |GypError| if the command return with a non-zero return code.""" job = subprocess.Popen(cmdlist, stdout=subprocess.PIPE) out = job.communicate()[0] + if PY3: + out = out.decode("utf-8") if job.returncode != 0: sys.stderr.write(out + '\n') raise GypError('Error %d running %s' % (job.returncode, cmdlist[0])) From 67dec1496a93715fecb503c4d7b27d694661a751 Mon Sep 17 00:00:00 2001 From: cclauss Date: Thu, 26 Sep 2019 11:29:03 +0200 Subject: [PATCH 086/551] gyp: more decode stdout on Python 3 PR-URL: https://github.com/nodejs/node-gyp/pull/1894 Reviewed-By: Ben Noordhuis Reviewed-By: Sam Roberts --- gyp/gyp_main.py | 10 ++++++++-- gyp/pylib/gyp/common.py | 6 +++++- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/gyp/gyp_main.py b/gyp/gyp_main.py index 8dfc552e81..aece53c8eb 100755 --- a/gyp/gyp_main.py +++ b/gyp/gyp_main.py @@ -8,13 +8,17 @@ import sys import subprocess +PY3 = bytes != str + # Below IsCygwin() function copied from pylib/gyp/common.py def IsCygwin(): try: out = subprocess.Popen("uname", stdout=subprocess.PIPE, stderr=subprocess.STDOUT) - stdout,stderr = out.communicate() + stdout, stderr = out.communicate() + if PY3: + stdout = stdout.decode("utf-8") return "CYGWIN" in str(stdout) except Exception: return False @@ -27,7 +31,9 @@ def UnixifyPath(path): out = subprocess.Popen(["cygpath", "-u", path], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) - stdout,stderr = out.communicate() + stdout, stderr = out.communicate() + if PY3: + stdout = stdout.decode("utf-8") return str(stdout) except Exception: return path diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py index 8296d11fc3..071ad7e242 100644 --- a/gyp/pylib/gyp/common.py +++ b/gyp/pylib/gyp/common.py @@ -11,6 +11,8 @@ import sys import subprocess +PY3 = bytes != str + # A minimal memoizing decorator. It'll blow up if the args aren't immutable, # among other "problems". @@ -623,7 +625,9 @@ def IsCygwin(): out = subprocess.Popen("uname", stdout=subprocess.PIPE, stderr=subprocess.STDOUT) - stdout,stderr = out.communicate() + stdout, stderr = out.communicate() + if PY3: + stdout = stdout.decode("utf-8") return "CYGWIN" in str(stdout) except Exception: return False From 8e9ec3b024a23bab9dbbfe60e012d5da50ed29d0 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Fri, 27 Sep 2019 00:29:44 +0200 Subject: [PATCH 087/551] gyp: modify XcodeVersion() to convert "4.2" to "0420" and "10.0" to "1000" Ref: https://github.com/nodejs/node-addon-api/issues/445#issuecomment-535361389 PR-URL: https://github.com/nodejs/node-gyp/pull/1895 Reviewed-By: Rod Vagg --- gyp/pylib/gyp/xcode_emulation.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py index de376edb07..70f41e6bb4 100644 --- a/gyp/pylib/gyp/xcode_emulation.py +++ b/gyp/pylib/gyp/xcode_emulation.py @@ -1286,9 +1286,9 @@ def XcodeVersion(): version_list = [version, ''] version = version_list[0] build = version_list[-1] - # Be careful to convert "4.2" to "0420": - version = version.split()[-1].replace('.', '') - version = (version + '0' * (3 - len(version))).zfill(4) + # Be careful to convert "4.2" to "0420" and "10.0" to "1000": + version = format(''.join((version.split()[-1].split('.') + ['0', '0'])[:3]), + '>04s') if build: build = build.split()[-1] XCODE_VERSION_CACHE = (version, build) From 3d1c60ab8183956cec40cf41ebc3b96a41bdcdcf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Tue, 23 Jul 2019 16:29:17 +0100 Subject: [PATCH 088/551] lib: accept Python 3 by default PR-URL: https://github.com/nodejs/node-gyp/pull/1844 Reviewed-By: Christian Clauss --- .travis.yml | 12 ++++++------ lib/find-python.js | 10 +++++++--- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/.travis.yml b/.travis.yml index f27f024f2d..2d4b010df8 100644 --- a/.travis.yml +++ b/.travis.yml @@ -31,25 +31,25 @@ matrix: - name: "Node.js 6 & Python 3.7 on Linux" python: 3.7 - env: NODE_GYP_FORCE_PYTHON=python3 EXPERIMENTAL_NODE_GYP_PYTHON3=1 + env: NODE_GYP_FORCE_PYTHON=python3 before_install: nvm install 6 - name: "Node.js 8 & Python 3.7 on Linux" python: 3.7 - env: NODE_GYP_FORCE_PYTHON=python3 EXPERIMENTAL_NODE_GYP_PYTHON3=1 + env: NODE_GYP_FORCE_PYTHON=python3 before_install: nvm install 8 - name: "Node.js 10 & Python 3.7 on Linux" python: 3.7 - env: NODE_GYP_FORCE_PYTHON=python3 EXPERIMENTAL_NODE_GYP_PYTHON3=1 + env: NODE_GYP_FORCE_PYTHON=python3 before_install: nvm install 10 - name: "Node.js 12 & Python 3.7 on Linux" python: 3.7 - env: NODE_GYP_FORCE_PYTHON=python3 EXPERIMENTAL_NODE_GYP_PYTHON3=1 + env: NODE_GYP_FORCE_PYTHON=python3 before_install: nvm install 12 - name: "Python 3.7 on macOS" os: osx #osx_image: xcode11 language: shell # 'language: python' is not yet supported on macOS - env: NODE_GYP_FORCE_PYTHON=python3 EXPERIMENTAL_NODE_GYP_PYTHON3=1 + env: NODE_GYP_FORCE_PYTHON=python3 before_install: HOMEBREW_NO_AUTO_UPDATE=1 brew install npm - name: "Node.js 12 & Python 3.7 on Windows" os: windows @@ -58,7 +58,6 @@ matrix: env: >- PATH=/c/Python37:/c/Python37/Scripts:$PATH NODE_GYP_FORCE_PYTHON=/c/Python37/python.exe - EXPERIMENTAL_NODE_GYP_PYTHON3=1 before_install: choco install python install: @@ -71,6 +70,7 @@ before_script: # exit-zero treats all errors as warnings. Two space indentation is OK. The GitHub editor is 127 chars wide - flake8 . --count --exit-zero --ignore=E111,E114,W503 --max-complexity=10 --max-line-length=127 --statistics - npm install + - npm list script: - node -e 'require("npmlog").level="verbose"; require("./lib/find-python")(null,()=>{})' - npm test diff --git a/lib/find-python.js b/lib/find-python.js index 30bb25fd36..bd7cb850ee 100644 --- a/lib/find-python.js +++ b/lib/find-python.js @@ -18,8 +18,7 @@ PythonFinder.prototype = { log: logWithPrefix(log, 'find Python'), argsExecutable: [ '-c', 'import sys; print(sys.executable);' ], argsVersion: [ '-c', 'import sys; print("%s.%s.%s" % sys.version_info[:3]);' ], - semverRange: process.env.EXPERIMENTAL_NODE_GYP_PYTHON3 ? '2.7.x || >=3.5.0' - : '>=2.7.0 <3.0.0', + semverRange: '2.7.x || >=3.5.0', // These can be overridden for testing: execFile: cp.execFile, @@ -93,6 +92,11 @@ PythonFinder.prototype = { check: this.checkCommand, arg: 'python' }, + { + before: () => { this.addLog('checking if "python3" can be used') }, + check: this.checkCommand, + arg: 'python3' + }, { before: () => { this.addLog('checking if "python2" can be used') }, check: this.checkCommand, @@ -286,7 +290,7 @@ PythonFinder.prototype = { // X const info = [ '**********************************************************', - 'You need to install the latest version of Python 2.7.', + 'You need to install the latest version of Python.', 'Node-gyp should be able to find and use Python. If not,', 'you can try one of the following options:', `- Use the switch --python="${pathExample}"`, From c763ca1838b7e56e7ab92965c22fe482d7ca06c2 Mon Sep 17 00:00:00 2001 From: cclauss Date: Sun, 7 Jul 2019 12:47:54 +0200 Subject: [PATCH 089/551] doc: Declare that node-gyp is Python 3 compatible MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit NOTE: node-gyp is compatible with both Python 2.7 and 3.7 but Node.js itself is not yet compatible with Python 3. PR-URL: https://github.com/nodejs/node-gyp/pull/1811 Reviewed-By: João Reis --- README.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 0c76ffc09b..d54098773b 100644 --- a/README.md +++ b/README.md @@ -25,20 +25,24 @@ $ npm install -g node-gyp You will also need to install: +NOTE: node-gyp is compatible with Python 2.7 and 3.7 but node itself is not yet compatible with Python 3. + ### On Unix - * `python` (`v2.7` recommended, `v3.x.x` is __*not*__ supported) + * `Python v2.7 or v3.7` (v3.5 and 3.6 may work but are not currently tested) * `make` * A proper C/C++ compiler toolchain, like [GCC](https://gcc.gnu.org) ### On macOS - * `python` (`v2.7` recommended, `v3.x.x` is __*not*__ supported) (already installed on macOS) + * `Python v2.7 or v3.7` (v3.5 and 3.6 may work but are not currently tested) * [Xcode](https://developer.apple.com/xcode/download/) * You also need to install the `XCode Command Line Tools` by running `xcode-select --install`. Alternatively, if you already have the full Xcode installed, you can find them under the menu `Xcode -> Open Developer Tool -> More Developer Tools...`. This step will install `clang`, `clang++`, and `make`. ### On Windows +Install the current version of Python from the [Microsoft Store package](https://docs.python.org/3.7/using/windows.html#the-microsoft-store-package). + #### Option 1 Install all the required tools and configurations using Microsoft's [windows-build-tools](https://github.com/felixrieseberg/windows-build-tools) using `npm install --global --production windows-build-tools` from an elevated PowerShell or CMD.exe (run as Administrator). @@ -49,7 +53,6 @@ Install tools and configuration manually: * Install Visual C++ Build Environment: [Visual Studio Build Tools](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) (using "Visual C++ build tools" workload) or [Visual Studio 2017 Community](https://visualstudio.microsoft.com/pl/thank-you-downloading-visual-studio/?sku=Community) (using the "Desktop development with C++" workload) - * Install [Python 2.7](https://www.python.org/downloads/) (`v3.x.x` is not supported), and run `npm config set python python2.7` (or see below for further instructions on specifying the proper Python version and path.) * Launch cmd, `npm config set msvs_version 2017` If the above steps didn't work for you, please visit [Microsoft's Node.js Guidelines for Windows](https://github.com/Microsoft/nodejs-guidelines/blob/master/windows-environment.md#compiling-native-addon-modules) for additional tips. @@ -62,7 +65,7 @@ If you have multiple Python versions installed, you can identify which Python version `node-gyp` uses by setting the `--python` variable: ``` bash -$ node-gyp --python /path/to/executable/python2.7 +$ node-gyp --python /path/to/executable/python ``` If `node-gyp` is called by way of `npm`, *and* you have multiple versions of @@ -70,7 +73,7 @@ Python installed, then you can set `npm`'s 'python' config key to the appropriat value: ``` bash -$ npm config set python /path/to/executable/python2.7 +$ npm config set python /path/to/executable/python ``` ## How to Use @@ -119,7 +122,7 @@ JSON-like format. This file gets placed in the root of your package, alongside A barebones `gyp` file appropriate for building a Node.js addon could look like: -``` python +```python { "targets": [ { @@ -179,7 +182,7 @@ Some additional resources for addons and writing `gyp` files: | `--proxy=$url` | Set HTTP proxy for downloading header tarball | `--cafile=$cafile` | Override default CA chain (to download tarball) | `--nodedir=$path` | Set the path to the node source code -| `--python=$path` | Set path to the Python 2 binary +| `--python=$path` | Set path to the Python binary | `--msvs_version=$version` | Set Visual Studio version (Windows only) | `--solution=$solution` | Set Visual Studio Solution version (Windows only) From 2441932dc490b2c0f1b7f962a52810d35a7e3d30 Mon Sep 17 00:00:00 2001 From: Richard Townsend Date: Thu, 12 Sep 2019 15:41:06 +0100 Subject: [PATCH 090/551] src,win: add support for fetching arm64 node.lib MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Windows on Arm support is available in some versions of Node.js v12 and Electron v6. This update allows node-gyp to fetch the appropriate node.lib to build native modules. If an arm64 binary is not available for the target node version, it's logged but ignored. arm64 is not expected to work in very old node.lib distribution formats, the test URLs in these cases are added to be consistent with x64. PR-URL: https://github.com/nodejs/node-gyp/pull/1875 Reviewed-By: João Reis --- lib/install.js | 12 ++++- lib/process-release.js | 11 +++- test/test-process-release.js | 100 +++++++++++++++++++++++++++++------ 3 files changed, 103 insertions(+), 20 deletions(-) diff --git a/lib/install.js b/lib/install.js index 3730c0965f..c1f0693cc7 100644 --- a/lib/install.js +++ b/lib/install.js @@ -298,7 +298,7 @@ function install (fs, gyp, argv, callback) { function downloadNodeLib (done) { log.verbose('on Windows; need to download `' + release.name + '.lib`...') - var archs = [ 'ia32', 'x64' ] + var archs = [ 'ia32', 'x64', 'arm64' ] var async = archs.length archs.forEach(function (arch) { var dir = path.resolve(devDir, arch) @@ -323,7 +323,15 @@ function install (fs, gyp, argv, callback) { req.on('error', done) req.on('response', function (res) { - if (res.statusCode !== 200) { + if (res.statusCode === 404) { + if (arch === 'arm64') { + // Arm64 is a newer platform on Windows and not all node distributions provide it. + log.verbose(`${name} was not found in ${libUrl}`) + } else { + log.warn(`${name} was not found in ${libUrl}`) + } + return + } else if (res.statusCode !== 200) { done(new Error(res.statusCode + ' status code downloading ' + name)) return } diff --git a/lib/process-release.js b/lib/process-release.js index 0acab061bd..f20dc893cd 100644 --- a/lib/process-release.js +++ b/lib/process-release.js @@ -7,7 +7,7 @@ const log = require('npmlog') // versions where -headers.tar.gz started shipping const headersTarballRange = '>= 3.0.0 || ~0.12.10 || ~0.10.42' -const bitsre = /\/win-(x86|x64)\// +const bitsre = /\/win-(x86|x64|arm64)\// const bitsreV3 = /\/win-(x86|ia32|x64)\// // io.js v3.x.x shipped with "ia32" but should // have been "x86" @@ -25,6 +25,7 @@ function processRelease (argv, gyp, defaultVersion, defaultRelease) { var baseUrl var libUrl32 var libUrl64 + var libUrlArm64 var tarballUrl var canGetHeaders @@ -79,6 +80,7 @@ function processRelease (argv, gyp, defaultVersion, defaultRelease) { baseUrl = url.resolve(defaultRelease.headersUrl, './') libUrl32 = resolveLibUrl(name, defaultRelease.libUrl || baseUrl || distBaseUrl, 'x86', versionSemver.major) libUrl64 = resolveLibUrl(name, defaultRelease.libUrl || baseUrl || distBaseUrl, 'x64', versionSemver.major) + libUrlArm64 = resolveLibUrl(name, defaultRelease.libUrl || baseUrl || distBaseUrl, 'arm64', versionSemver.major) tarballUrl = defaultRelease.headersUrl } else { // older versions without process.release are captured here and we have to make @@ -87,6 +89,7 @@ function processRelease (argv, gyp, defaultVersion, defaultRelease) { baseUrl = distBaseUrl libUrl32 = resolveLibUrl(name, baseUrl, 'x86', versionSemver.major) libUrl64 = resolveLibUrl(name, baseUrl, 'x64', versionSemver.major) + libUrlArm64 = resolveLibUrl(name, baseUrl, 'arm64', versionSemver.major) // making the bold assumption that anything with a version number >3.0.0 will // have a *-headers.tar.gz file in its dist location, even some frankenstein @@ -110,6 +113,10 @@ function processRelease (argv, gyp, defaultVersion, defaultRelease) { x64: { libUrl: libUrl64, libPath: normalizePath(path.relative(url.parse(baseUrl).path, url.parse(libUrl64).path)) + }, + arm64: { + libUrl: libUrlArm64, + libPath: normalizePath(path.relative(url.parse(baseUrl).path, url.parse(libUrlArm64).path)) } } } @@ -128,7 +135,7 @@ function resolveLibUrl (name, defaultUrl, arch, versionMajor) { return url.resolve(base, 'win-' + arch + '/' + name + '.lib') } // prior to io.js@1.0.0 32-bit node.lib lives in /, 64-bit lives in /x64/ - return url.resolve(base, (arch === 'x64' ? 'x64/' : '') + name + '.lib') + return url.resolve(base, (arch === 'x86' ? '' : arch + '/') + name + '.lib') } // else we have a proper url to a .lib, just make sure it's the right arch diff --git a/test/test-process-release.js b/test/test-process-release.js index e4370e59ee..42ddbd2cda 100644 --- a/test/test-process-release.js +++ b/test/test-process-release.js @@ -19,7 +19,8 @@ test('test process release - process.version = 0.8.20', function (t) { shasumsUrl: 'https://nodejs.org/dist/v0.8.20/SHASUMS256.txt', versionDir: '0.8.20', ia32: { libUrl: 'https://nodejs.org/dist/v0.8.20/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.8.20/x64/node.lib', libPath: 'x64/node.lib' } + x64: { libUrl: 'https://nodejs.org/dist/v0.8.20/x64/node.lib', libPath: 'x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/dist/v0.8.20/arm64/node.lib', libPath: 'arm64/node.lib' } }) }) @@ -39,7 +40,8 @@ test('test process release - process.version = 0.10.21', function (t) { shasumsUrl: 'https://nodejs.org/dist/v0.10.21/SHASUMS256.txt', versionDir: '0.10.21', ia32: { libUrl: 'https://nodejs.org/dist/v0.10.21/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.10.21/x64/node.lib', libPath: 'x64/node.lib' } + x64: { libUrl: 'https://nodejs.org/dist/v0.10.21/x64/node.lib', libPath: 'x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/dist/v0.10.21/arm64/node.lib', libPath: 'arm64/node.lib' } }) }) @@ -60,7 +62,8 @@ test('test process release - process.version = 0.12.9', function (t) { shasumsUrl: 'https://nodejs.org/dist/v0.12.9/SHASUMS256.txt', versionDir: '0.12.9', ia32: { libUrl: 'https://nodejs.org/dist/v0.12.9/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.12.9/x64/node.lib', libPath: 'x64/node.lib' } + x64: { libUrl: 'https://nodejs.org/dist/v0.12.9/x64/node.lib', libPath: 'x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/dist/v0.12.9/arm64/node.lib', libPath: 'arm64/node.lib' } }) }) @@ -81,7 +84,8 @@ test('test process release - process.version = 0.10.41', function (t) { shasumsUrl: 'https://nodejs.org/dist/v0.10.41/SHASUMS256.txt', versionDir: '0.10.41', ia32: { libUrl: 'https://nodejs.org/dist/v0.10.41/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.10.41/x64/node.lib', libPath: 'x64/node.lib' } + x64: { libUrl: 'https://nodejs.org/dist/v0.10.41/x64/node.lib', libPath: 'x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/dist/v0.10.41/arm64/node.lib', libPath: 'arm64/node.lib' } }) }) @@ -102,7 +106,8 @@ test('test process release - process.release ~ node@0.10.42', function (t) { shasumsUrl: 'https://nodejs.org/dist/v0.10.42/SHASUMS256.txt', versionDir: '0.10.42', ia32: { libUrl: 'https://nodejs.org/dist/v0.10.42/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.10.42/x64/node.lib', libPath: 'x64/node.lib' } + x64: { libUrl: 'https://nodejs.org/dist/v0.10.42/x64/node.lib', libPath: 'x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/dist/v0.10.42/arm64/node.lib', libPath: 'arm64/node.lib' } }) }) @@ -123,7 +128,8 @@ test('test process release - process.release ~ node@0.12.10', function (t) { shasumsUrl: 'https://nodejs.org/dist/v0.12.10/SHASUMS256.txt', versionDir: '0.12.10', ia32: { libUrl: 'https://nodejs.org/dist/v0.12.10/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.12.10/x64/node.lib', libPath: 'x64/node.lib' } + x64: { libUrl: 'https://nodejs.org/dist/v0.12.10/x64/node.lib', libPath: 'x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/dist/v0.12.10/arm64/node.lib', libPath: 'arm64/node.lib' } }) }) @@ -146,7 +152,8 @@ test('test process release - process.release ~ node@4.1.23', function (t) { shasumsUrl: 'https://nodejs.org/dist/v4.1.23/SHASUMS256.txt', versionDir: '4.1.23', ia32: { libUrl: 'https://nodejs.org/dist/v4.1.23/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' } + x64: { libUrl: 'https://nodejs.org/dist/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/dist/v4.1.23/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } }) }) @@ -169,7 +176,60 @@ test('test process release - process.release ~ node@4.1.23 / corp build', functi shasumsUrl: 'https://some.custom.location/SHASUMS256.txt', versionDir: '4.1.23', ia32: { libUrl: 'https://some.custom.location/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://some.custom.location/win-x64/node.lib', libPath: 'win-x64/node.lib' } + x64: { libUrl: 'https://some.custom.location/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + arm64: { libUrl: 'https://some.custom.location/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } + }) +}) + +test('test process release - process.release ~ node@12.8.0 Windows', function (t) { + t.plan(2) + + var release = processRelease([], { opts: {} }, 'v12.8.0', { + name: 'node', + sourceUrl: 'https://nodejs.org/download/release/v12.8.0/node-v12.8.0.tar.gz', + headersUrl: 'https://nodejs.org/download/release/v12.8.0/node-v12.8.0-headers.tar.gz', + libUrl: 'https://nodejs.org/download/release/v12.8.0/win-x64/node.lib' + }) + + t.equal(release.semver.version, '12.8.0') + delete release.semver + + t.deepEqual(release, { + version: '12.8.0', + name: 'node', + baseUrl: 'https://nodejs.org/download/release/v12.8.0/', + tarballUrl: 'https://nodejs.org/download/release/v12.8.0/node-v12.8.0-headers.tar.gz', + shasumsUrl: 'https://nodejs.org/download/release/v12.8.0/SHASUMS256.txt', + versionDir: '12.8.0', + ia32: { libUrl: 'https://nodejs.org/download/release/v12.8.0/win-x86/node.lib', libPath: 'win-x86/node.lib' }, + x64: { libUrl: 'https://nodejs.org/download/release/v12.8.0/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/download/release/v12.8.0/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } + }) +}) + +test('test process release - process.release ~ node@12.8.0 Windows ARM64', function (t) { + t.plan(2) + + var release = processRelease([], { opts: {} }, 'v12.8.0', { + name: 'node', + sourceUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/node-v12.8.0.tar.gz', + headersUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/node-v12.8.0-headers.tar.gz', + libUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/win-arm64/node.lib' + }) + + t.equal(release.semver.version, '12.8.0') + delete release.semver + + t.deepEqual(release, { + version: '12.8.0', + name: 'node', + baseUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/', + tarballUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/node-v12.8.0-headers.tar.gz', + shasumsUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/SHASUMS256.txt', + versionDir: '12.8.0', + ia32: { libUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/win-x86/node.lib', libPath: 'win-x86/node.lib' }, + x64: { libUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + arm64: { libUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } }) }) @@ -192,7 +252,8 @@ test('test process release - process.release ~ node@4.1.23 --target=0.10.40', fu shasumsUrl: 'https://nodejs.org/dist/v0.10.40/SHASUMS256.txt', versionDir: '0.10.40', ia32: { libUrl: 'https://nodejs.org/dist/v0.10.40/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.10.40/x64/node.lib', libPath: 'x64/node.lib' } + x64: { libUrl: 'https://nodejs.org/dist/v0.10.40/x64/node.lib', libPath: 'x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/dist/v0.10.40/arm64/node.lib', libPath: 'arm64/node.lib' } }) }) @@ -215,7 +276,8 @@ test('test process release - process.release ~ node@4.1.23 --dist-url=https://fo shasumsUrl: 'https://foo.bar/baz/v4.1.23/SHASUMS256.txt', versionDir: '4.1.23', ia32: { libUrl: 'https://foo.bar/baz/v4.1.23/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://foo.bar/baz/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' } + x64: { libUrl: 'https://foo.bar/baz/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + arm64: { libUrl: 'https://foo.bar/baz/v4.1.23/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } }) }) @@ -238,7 +300,8 @@ test('test process release - process.release ~ frankenstein@4.1.23', function (t shasumsUrl: 'https://frankensteinjs.org/dist/v4.1.23/SHASUMS256.txt', versionDir: 'frankenstein-4.1.23', ia32: { libUrl: 'https://frankensteinjs.org/dist/v4.1.23/win-x86/frankenstein.lib', libPath: 'win-x86/frankenstein.lib' }, - x64: { libUrl: 'https://frankensteinjs.org/dist/v4.1.23/win-x64/frankenstein.lib', libPath: 'win-x64/frankenstein.lib' } + x64: { libUrl: 'https://frankensteinjs.org/dist/v4.1.23/win-x64/frankenstein.lib', libPath: 'win-x64/frankenstein.lib' }, + arm64: { libUrl: 'https://frankensteinjs.org/dist/v4.1.23/win-arm64/frankenstein.lib', libPath: 'win-arm64/frankenstein.lib' } }) }) @@ -261,7 +324,8 @@ test('test process release - process.release ~ frankenstein@4.1.23 --dist-url=ht shasumsUrl: 'http://foo.bar/baz/v4.1.23/SHASUMS256.txt', versionDir: 'frankenstein-4.1.23', ia32: { libUrl: 'http://foo.bar/baz/v4.1.23/win-x86/frankenstein.lib', libPath: 'win-x86/frankenstein.lib' }, - x64: { libUrl: 'http://foo.bar/baz/v4.1.23/win-x64/frankenstein.lib', libPath: 'win-x64/frankenstein.lib' } + x64: { libUrl: 'http://foo.bar/baz/v4.1.23/win-x64/frankenstein.lib', libPath: 'win-x64/frankenstein.lib' }, + arm64: { libUrl: 'http://foo.bar/baz/v4.1.23/win-arm64/frankenstein.lib', libPath: 'win-arm64/frankenstein.lib' } }) }) @@ -284,7 +348,8 @@ test('test process release - process.release ~ node@4.0.0-rc.4', function (t) { shasumsUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/SHASUMS256.txt', versionDir: '4.0.0-rc.4', ia32: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' } + x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } }) }) @@ -309,7 +374,8 @@ test('test process release - process.release ~ node@4.0.0-rc.4 passed as argv[0] shasumsUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/SHASUMS256.txt', versionDir: '4.0.0-rc.4', ia32: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' } + x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } }) }) @@ -334,7 +400,8 @@ test('test process release - process.release ~ node@4.0.0-rc.4 - bogus string pa shasumsUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/SHASUMS256.txt', versionDir: '4.0.0-rc.4', ia32: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' } + x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } }) }) @@ -359,7 +426,8 @@ test('test process release - NODEJS_ORG_MIRROR', function (t) { shasumsUrl: 'http://foo.bar/v4.1.23/SHASUMS256.txt', versionDir: '4.1.23', ia32: { libUrl: 'http://foo.bar/v4.1.23/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'http://foo.bar/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' } + x64: { libUrl: 'http://foo.bar/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + arm64: { libUrl: 'http://foo.bar/v4.1.23/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } }) delete process.env.NODEJS_ORG_MIRROR From a84b88575629f9190bceb7da6787895f8acf662d Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Tue, 1 Oct 2019 07:54:49 +0100 Subject: [PATCH 091/551] gyp: fix undefined name: cflags --> ldflags The current code would raise NameError at runtime. PR-URL: https://github.com/nodejs/node-gyp/pull/1901 Reviewed-By: Rod Vagg --- gyp/pylib/gyp/xcode_emulation.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py index 70f41e6bb4..74735c033a 100644 --- a/gyp/pylib/gyp/xcode_emulation.py +++ b/gyp/pylib/gyp/xcode_emulation.py @@ -846,7 +846,7 @@ def GetLdflags(self, configname, product_dir, gyp_to_build_path, arch=None): if self._IsXCTest(): platform_root = self._XcodePlatformPath(configname) if platform_root: - cflags.append('-F' + platform_root + '/Developer/Library/Frameworks/') # noqa TODO @cclauss + ldflags.append('-F' + platform_root + '/Developer/Library/Frameworks/') is_extension = self._IsIosAppExtension() or self._IsIosWatchKitExtension() if sdk_root and is_extension: From 6f39fd45221fa59cc45c0b400caf6fbc71daca77 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Thu, 26 Sep 2019 13:40:58 +1000 Subject: [PATCH 092/551] v5.0.4: bump version and update changelog PR-URL: https://github.com/nodejs/node-gyp/pull/1893 --- CHANGELOG.md | 24 ++++++++++++++++++++++++ package.json | 2 +- 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9a99ef2d0a..3dbc5d9244 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,27 @@ +v5.0.4 2019-09-27 +================= + +* [[`1236869ffc`](https://github.com/nodejs/node-gyp/commit/1236869ffc)] - **gyp**: modify XcodeVersion() to convert "4.2" to "0420" and "10.0" to "1000" (Christian Clauss) [#1895](https://github.com/nodejs/node-gyp/pull/1895) +* [[`36638afe48`](https://github.com/nodejs/node-gyp/commit/36638afe48)] - **gyp**: more decode stdout on Python 3 (cclauss) [#1894](https://github.com/nodejs/node-gyp/pull/1894) +* [[`f753c167c5`](https://github.com/nodejs/node-gyp/commit/f753c167c5)] - **gyp**: decode stdout on Python 3 (cclauss) [#1890](https://github.com/nodejs/node-gyp/pull/1890) +* [[`60a4083523`](https://github.com/nodejs/node-gyp/commit/60a4083523)] - **doc**: update xcode install instructions to match Node's BUILDING (Nhan Khong) [#1884](https://github.com/nodejs/node-gyp/pull/1884) +* [[`19dbc9ac32`](https://github.com/nodejs/node-gyp/commit/19dbc9ac32)] - **deps**: update tar to 4.4.12 (Matheus Marchini) [#1889](https://github.com/nodejs/node-gyp/pull/1889) +* [[`5f3ed92181`](https://github.com/nodejs/node-gyp/commit/5f3ed92181)] - **bin**: fix the usage instructions (Halit Ogunc) [#1888](https://github.com/nodejs/node-gyp/pull/1888) +* [[`aab118edf1`](https://github.com/nodejs/node-gyp/commit/aab118edf1)] - **lib**: adding keep-alive header to download requests (Milad Farazmand) [#1863](https://github.com/nodejs/node-gyp/pull/1863) +* [[`1186e89326`](https://github.com/nodejs/node-gyp/commit/1186e89326)] - **lib**: ignore non-critical os.userInfo() failures (Rod Vagg) [#1835](https://github.com/nodejs/node-gyp/pull/1835) +* [[`785e527c3d`](https://github.com/nodejs/node-gyp/commit/785e527c3d)] - **doc**: fix missing argument for setting python path (lagorsse) [#1802](https://github.com/nodejs/node-gyp/pull/1802) +* [[`a97615196c`](https://github.com/nodejs/node-gyp/commit/a97615196c)] - **gyp**: rm semicolons (Python != JavaScript) (MattIPv4) [#1858](https://github.com/nodejs/node-gyp/pull/1858) +* [[`06019bac24`](https://github.com/nodejs/node-gyp/commit/06019bac24)] - **gyp**: assorted typo fixes (XhmikosR) [#1853](https://github.com/nodejs/node-gyp/pull/1853) +* [[`3f4972c1ca`](https://github.com/nodejs/node-gyp/commit/3f4972c1ca)] - **gyp**: use "is" when comparing to None (Vladyslav Burzakovskyy) [#1860](https://github.com/nodejs/node-gyp/pull/1860) +* [[`1cb4708073`](https://github.com/nodejs/node-gyp/commit/1cb4708073)] - **src,win**: improve unmanaged handling (Peter Sabath) [#1852](https://github.com/nodejs/node-gyp/pull/1852) +* [[`5553cd910e`](https://github.com/nodejs/node-gyp/commit/5553cd910e)] - **gyp**: improve Windows+Cygwin compatibility (Jose Quijada) [#1817](https://github.com/nodejs/node-gyp/pull/1817) +* [[`8bcb1fbb43`](https://github.com/nodejs/node-gyp/commit/8bcb1fbb43)] - **gyp**: Python 3 Windows fixes (João Reis) [#1843](https://github.com/nodejs/node-gyp/pull/1843) +* [[`2e24d0a326`](https://github.com/nodejs/node-gyp/commit/2e24d0a326)] - **test**: accept Python 3 in test-find-python.js (João Reis) [#1843](https://github.com/nodejs/node-gyp/pull/1843) +* [[`1267b4dc1c`](https://github.com/nodejs/node-gyp/commit/1267b4dc1c)] - **build**: add test run Python 3.7 on macOS (Christian Clauss) [#1843](https://github.com/nodejs/node-gyp/pull/1843) +* [[`da1b031aa3`](https://github.com/nodejs/node-gyp/commit/da1b031aa3)] - **build**: import StringIO on Python 2 and Python 3 (Christian Clauss) [#1836](https://github.com/nodejs/node-gyp/pull/1836) +* [[`fa0ed4aa42`](https://github.com/nodejs/node-gyp/commit/fa0ed4aa42)] - **build**: more Python 3 compat, replace compile with ast (cclauss) [#1820](https://github.com/nodejs/node-gyp/pull/1820) +* [[`18d5c7c9d0`](https://github.com/nodejs/node-gyp/commit/18d5c7c9d0)] - **win,src**: update win\_delay\_load\_hook.cc to work with /clr (Ivan Petrovic) [#1819](https://github.com/nodejs/node-gyp/pull/1819) + v5.0.3 2019-07-17 ================= diff --git a/package.json b/package.json index fe68798faf..7cecf6eb01 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "5.0.3", + "version": "5.0.4", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From ab2a4cc209f9b03d67cbfcddd5ea22088692062f Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Mon, 30 Sep 2019 15:52:41 +1000 Subject: [PATCH 093/551] src: update to standard@14 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR-URL: https://github.com/nodejs/node-gyp/pull/1899 Reviewed-By: João Reis Reviewed-By: Richard Lau --- bin/node-gyp.js | 2 +- lib/configure.js | 22 +++++++++++----------- lib/find-python.js | 12 ++++++------ lib/install.js | 6 +++--- lib/node-gyp.js | 8 ++++---- lib/process-release.js | 2 ++ package.json | 2 +- test/process-exec-sync.js | 2 +- test/test-addon.js | 16 ++++++++-------- test/test-configure-python.js | 18 +++++++++--------- test/test-find-accessible-sync.js | 16 ++++++++-------- test/test-find-node-directory.js | 8 +++++--- test/test-process-release.js | 4 ++-- 13 files changed, 61 insertions(+), 57 deletions(-) diff --git a/bin/node-gyp.js b/bin/node-gyp.js index 13f2d399f7..49b5721d02 100755 --- a/bin/node-gyp.js +++ b/bin/node-gyp.js @@ -131,7 +131,7 @@ function errorMessage () { function issueMessage () { errorMessage() - log.error('', [ 'This is a bug in `node-gyp`.', + log.error('', ['This is a bug in `node-gyp`.', 'Try to update node-gyp and file an Issue if it does not help:', ' ' ].join('\n')) diff --git a/lib/configure.js b/lib/configure.js index 20b955cabc..564564eea4 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -17,7 +17,7 @@ if (win) { function configure (gyp, argv, callback) { var python var buildDir = path.resolve('build') - var configNames = [ 'config.gypi', 'common.gypi' ] + var configNames = ['config.gypi', 'common.gypi'] var configs = [] var nodeDir var release = processRelease(argv, gyp, process.version, process.release) @@ -60,7 +60,7 @@ function configure (gyp, argv, callback) { // into devdir. Otherwise only install if they're not already there. gyp.opts.ensure = !gyp.opts.tarball - gyp.commands.install([ release.version ], function (err) { + gyp.commands.install([release.version], function (err) { if (err) { return callback(err) } @@ -132,7 +132,7 @@ function configure (gyp, argv, callback) { // set the target_arch variable variables.target_arch = gyp.opts.arch || process.arch || 'ia32' if (variables.target_arch === 'arm64') { - defaults['msvs_configuration_platform'] = 'ARM64' + defaults.msvs_configuration_platform = 'ARM64' } // set the node development directory @@ -142,22 +142,22 @@ function configure (gyp, argv, callback) { variables.standalone_static_library = gyp.opts.thin ? 0 : 1 if (win) { - process.env['GYP_MSVS_VERSION'] = Math.min(vsInfo.versionYear, 2015) - process.env['GYP_MSVS_OVERRIDE_PATH'] = vsInfo.path - defaults['msbuild_toolset'] = vsInfo.toolset + process.env.GYP_MSVS_VERSION = Math.min(vsInfo.versionYear, 2015) + process.env.GYP_MSVS_OVERRIDE_PATH = vsInfo.path + defaults.msbuild_toolset = vsInfo.toolset if (vsInfo.sdk) { - defaults['msvs_windows_target_platform_version'] = vsInfo.sdk + defaults.msvs_windows_target_platform_version = vsInfo.sdk } if (variables.target_arch === 'arm64') { if (vsInfo.versionMajor > 15 || (vsInfo.versionMajor === 15 && vsInfo.versionMajor >= 9)) { - defaults['msvs_enable_marmasm'] = 1 + defaults.msvs_enable_marmasm = 1 } else { log.warn('Compiling ARM64 assembly is only available in\n' + 'Visual Studio 2017 version 15.9 and above') } } - variables['msbuild_path'] = vsInfo.msBuild + variables.msbuild_path = vsInfo.msBuild } // loop through the rest of the opts and add the unknown ones as variables. @@ -190,7 +190,7 @@ function configure (gyp, argv, callback) { var json = JSON.stringify(config, boolsToString, 2) log.verbose('build/' + configFilename, 'writing out config file: %s', configPath) configs.push(configPath) - fs.writeFile(configPath, [ prefix, json, '' ].join('\n'), findConfigs) + fs.writeFile(configPath, [prefix, json, ''].join('\n'), findConfigs) } function findConfigs (err) { @@ -335,7 +335,7 @@ function configure (gyp, argv, callback) { argv.unshift(gypScript) // make sure python uses files that came with this particular node package - var pypath = [ path.join(__dirname, '..', 'gyp', 'pylib') ] + var pypath = [path.join(__dirname, '..', 'gyp', 'pylib')] if (process.env.PYTHONPATH) { pypath.push(process.env.PYTHONPATH) } diff --git a/lib/find-python.js b/lib/find-python.js index bd7cb850ee..c12ccc3bb2 100644 --- a/lib/find-python.js +++ b/lib/find-python.js @@ -16,8 +16,8 @@ function PythonFinder (configPython, callback) { PythonFinder.prototype = { log: logWithPrefix(log, 'find Python'), - argsExecutable: [ '-c', 'import sys; print(sys.executable);' ], - argsVersion: [ '-c', 'import sys; print("%s.%s.%s" % sys.version_info[:3]);' ], + argsExecutable: ['-c', 'import sys; print(sys.executable);'], + argsVersion: ['-c', 'import sys; print("%s.%s.%s" % sys.version_info[:3]);'], semverRange: '2.7.x || >=3.5.0', // These can be overridden for testing: @@ -45,7 +45,7 @@ PythonFinder.prototype = { function getChecks () { if (this.env.NODE_GYP_FORCE_PYTHON) { - return [ { + return [{ before: () => { this.addLog( 'checking Python explicitly set from NODE_GYP_FORCE_PYTHON') @@ -54,7 +54,7 @@ PythonFinder.prototype = { }, check: this.checkCommand, arg: this.env.NODE_GYP_FORCE_PYTHON - } ] + }] } var checks = [ @@ -144,7 +144,7 @@ PythonFinder.prototype = { return this.fail() } - const args = [ runChecks.bind(this) ] + const args = [runChecks.bind(this)] if (check.arg) { args.unshift(check.arg) } @@ -196,7 +196,7 @@ PythonFinder.prototype = { checkPyLauncher: function checkPyLauncher (errorCallback) { this.log.verbose( `- executing "${this.pyLauncher}" to get Python 2 executable path`) - this.run(this.pyLauncher, [ '-2', ...this.argsExecutable ], false, + this.run(this.pyLauncher, ['-2', ...this.argsExecutable], false, function (err, execPath) { // Possible outcomes: same as checkCommand if (err) { diff --git a/lib/install.js b/lib/install.js index c1f0693cc7..b870532880 100644 --- a/lib/install.js +++ b/lib/install.js @@ -24,7 +24,7 @@ function install (fs, gyp, argv, callback) { if (err) { log.warn('install', 'got an error, rolling back install') // roll-back the install if anything went wrong - gyp.commands.remove([ release.versionDir ], function () { + gyp.commands.remove([release.versionDir], function () { callback(err) }) } else { @@ -298,7 +298,7 @@ function install (fs, gyp, argv, callback) { function downloadNodeLib (done) { log.verbose('on Windows; need to download `' + release.name + '.lib`...') - var archs = [ 'ia32', 'x64', 'arm64' ] + var archs = ['ia32', 'x64', 'arm64'] var async = archs.length archs.forEach(function (arch) { var dir = path.resolve(devDir, arch) @@ -400,7 +400,7 @@ function download (gyp, env, url) { uri: url, headers: { 'User-Agent': 'node-gyp v' + gyp.version + ' (node ' + process.version + ')', - 'Connection': 'keep-alive' + Connection: 'keep-alive' } } diff --git a/lib/node-gyp.js b/lib/node-gyp.js index 91880f9433..9d24103900 100644 --- a/lib/node-gyp.js +++ b/lib/node-gyp.js @@ -18,8 +18,8 @@ const commands = [ 'remove' ] const aliases = { - 'ls': 'list', - 'rm': 'remove' + ls: 'list', + rm: 'remove' } // differentiate node-gyp's logs from npm's @@ -72,7 +72,7 @@ proto.configDefs = { loglevel: String, // everywhere python: String, // 'configure' 'dist-url': String, // 'install' - 'tarball': String, // 'install' + tarball: String, // 'install' jobs: String, // 'build' thin: String // 'configure' } @@ -167,7 +167,7 @@ proto.spawn = function spawn (command, args, opts) { opts = {} } if (!opts.silent && !opts.stdio) { - opts.stdio = [ 0, 1, 2 ] + opts.stdio = [0, 1, 2] } var cp = childProcess.spawn(command, args, opts) log.info('spawn', command) diff --git a/lib/process-release.js b/lib/process-release.js index f20dc893cd..95b55e4426 100644 --- a/lib/process-release.js +++ b/lib/process-release.js @@ -1,3 +1,5 @@ +/* eslint-disable node/no-deprecated-api */ + 'use strict' const semver = require('semver') diff --git a/package.json b/package.json index 7cecf6eb01..634febf186 100644 --- a/package.json +++ b/package.json @@ -41,7 +41,7 @@ "bindings": "~1.2.1", "nan": "^2.0.0", "require-inject": "~1.3.0", - "standard": "~12.0.1", + "standard": "~14.3.1", "tap": "~12.7.0" }, "scripts": { diff --git a/test/process-exec-sync.js b/test/process-exec-sync.js index f786484027..21763bc26d 100644 --- a/test/process-exec-sync.js +++ b/test/process-exec-sync.js @@ -67,7 +67,7 @@ function processExecSync (file, args, options) { } } - [ 'stdout', 'stderr', 'status' ].forEach(function (file) { + ['stdout', 'stderr', 'status'].forEach(function (file) { child[file] = fs.readFileSync(tmpdir + '/' + file, options.encoding) setTimeout(unlinkFile, 500, tmpdir + '/' + file) }) diff --git a/test/test-addon.js b/test/test-addon.js index f97215c0a2..f79eff73c1 100644 --- a/test/test-addon.js +++ b/test/test-addon.js @@ -15,18 +15,18 @@ function runHello (hostProcess) { hostProcess = process.execPath } var testCode = "console.log(require('hello_world').hello())" - return execFileSync(hostProcess, [ '-e', testCode ], { cwd: __dirname }).toString() + return execFileSync(hostProcess, ['-e', testCode], { cwd: __dirname }).toString() } function getEncoding () { var code = 'import locale;print(locale.getdefaultlocale()[1])' - return execFileSync('python', [ '-c', code ]).toString().trim() + return execFileSync('python', ['-c', code]).toString().trim() } function checkCharmapValid () { var data try { - data = execFileSync('python', [ 'fixtures/test-charmap.py' ], + data = execFileSync('python', ['fixtures/test-charmap.py'], { cwd: __dirname }) } catch (err) { return false @@ -39,7 +39,7 @@ test('build simple addon', function (t) { t.plan(3) // Set the loglevel otherwise the output disappears when run via 'npm test' - var cmd = [ nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose' ] + var cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose'] var proc = execFile(process.execPath, cmd, function (err, stdout, stderr) { var logLines = stderr.toString().trim().split(/\r?\n/) var lastLine = logLines[logLines.length - 1] @@ -59,9 +59,9 @@ test('build simple addon in path with non-ascii characters', function (t) { } var testDirNames = { - 'cp936': '文件夹', - 'cp1252': 'Latīna', - 'cp932': 'フォルダ' + cp936: '文件夹', + cp1252: 'Latīna', + cp932: 'フォルダ' } // Select non-ascii characters by current encoding var testDirName = testDirNames[getEncoding()] @@ -136,7 +136,7 @@ test('addon works with renamed host executable', function (t) { var notNodePath = path.join(os.tmpdir(), 'notnode' + path.extname(process.execPath)) fs.copyFileSync(process.execPath, notNodePath) - var cmd = [ nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose' ] + var cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose'] var proc = execFile(process.execPath, cmd, function (err, stdout, stderr) { var logLines = stderr.toString().trim().split(/\r?\n/) var lastLine = logLines[logLines.length - 1] diff --git a/test/test-configure-python.js b/test/test-configure-python.js index 41b8c70427..518bf036d0 100644 --- a/test/test-configure-python.js +++ b/test/test-configure-python.js @@ -6,10 +6,10 @@ const gyp = require('../lib/node-gyp') const requireInject = require('require-inject') const configure = requireInject('../lib/configure', { 'graceful-fs': { - 'openSync': function () { return 0 }, - 'closeSync': function () { }, - 'writeFile': function (file, data, cb) { cb() }, - 'stat': function (file, cb) { cb(null, {}) } + openSync: function () { return 0 }, + closeSync: function () { }, + writeFile: function (file, data, cb) { cb() }, + stat: function (file, cb) { cb(null, {}) } } }) @@ -42,10 +42,10 @@ test('configure PYTHONPATH with existing env of one dir', function (t) { var prog = gyp() prog.parseArgv([]) prog.spawn = function () { - t.equal(process.env.PYTHONPATH, [ EXPECTED_PYPATH, existingPath ].join(SEPARATOR)) + t.equal(process.env.PYTHONPATH, [EXPECTED_PYPATH, existingPath].join(SEPARATOR)) var dirs = process.env.PYTHONPATH.split(SEPARATOR) - t.deepEqual(dirs, [ EXPECTED_PYPATH, existingPath ]) + t.deepEqual(dirs, [EXPECTED_PYPATH, existingPath]) return SPAWN_RESULT } @@ -57,16 +57,16 @@ test('configure PYTHONPATH with existing env of multiple dirs', function (t) { var pythonDir1 = path.join('a', 'b') var pythonDir2 = path.join('b', 'c') - var existingPath = [ pythonDir1, pythonDir2 ].join(SEPARATOR) + var existingPath = [pythonDir1, pythonDir2].join(SEPARATOR) process.env.PYTHONPATH = existingPath var prog = gyp() prog.parseArgv([]) prog.spawn = function () { - t.equal(process.env.PYTHONPATH, [ EXPECTED_PYPATH, existingPath ].join(SEPARATOR)) + t.equal(process.env.PYTHONPATH, [EXPECTED_PYPATH, existingPath].join(SEPARATOR)) var dirs = process.env.PYTHONPATH.split(SEPARATOR) - t.deepEqual(dirs, [ EXPECTED_PYPATH, pythonDir1, pythonDir2 ]) + t.deepEqual(dirs, [EXPECTED_PYPATH, pythonDir1, pythonDir2]) return SPAWN_RESULT } diff --git a/test/test-find-accessible-sync.js b/test/test-find-accessible-sync.js index 32234f5389..0a2e584c4f 100644 --- a/test/test-find-accessible-sync.js +++ b/test/test-find-accessible-sync.js @@ -5,8 +5,8 @@ const path = require('path') const requireInject = require('require-inject') const configure = requireInject('../lib/configure', { 'graceful-fs': { - 'closeSync': function () { return undefined }, - 'openSync': function (path) { + closeSync: function () { return undefined }, + openSync: function (path) { if (readableFiles.some(function (f) { return f === path })) { return 0 } else { @@ -38,7 +38,7 @@ test('find accessible - empty array', function (t) { test('find accessible - single item array, readable', function (t) { t.plan(1) - var candidates = [ readableFile ] + var candidates = [readableFile] var found = configure.test.findAccessibleSync('test', dir, candidates) t.strictEqual(found, path.resolve(dir, readableFile)) }) @@ -46,7 +46,7 @@ test('find accessible - single item array, readable', function (t) { test('find accessible - single item array, readable in subdir', function (t) { t.plan(1) - var candidates = [ readableFileInDir ] + var candidates = [readableFileInDir] var found = configure.test.findAccessibleSync('test', dir, candidates) t.strictEqual(found, path.resolve(dir, readableFileInDir)) }) @@ -54,7 +54,7 @@ test('find accessible - single item array, readable in subdir', function (t) { test('find accessible - single item array, unreadable', function (t) { t.plan(1) - var candidates = [ 'unreadable_file' ] + var candidates = ['unreadable_file'] var found = configure.test.findAccessibleSync('test', dir, candidates) t.strictEqual(found, undefined) }) @@ -62,7 +62,7 @@ test('find accessible - single item array, unreadable', function (t) { test('find accessible - multi item array, no matches', function (t) { t.plan(1) - var candidates = [ 'non_existent_file', 'unreadable_file' ] + var candidates = ['non_existent_file', 'unreadable_file'] var found = configure.test.findAccessibleSync('test', dir, candidates) t.strictEqual(found, undefined) }) @@ -70,7 +70,7 @@ test('find accessible - multi item array, no matches', function (t) { test('find accessible - multi item array, single match', function (t) { t.plan(1) - var candidates = [ 'non_existent_file', readableFile ] + var candidates = ['non_existent_file', readableFile] var found = configure.test.findAccessibleSync('test', dir, candidates) t.strictEqual(found, path.resolve(dir, readableFile)) }) @@ -78,7 +78,7 @@ test('find accessible - multi item array, single match', function (t) { test('find accessible - multi item array, return first match', function (t) { t.plan(1) - var candidates = [ 'non_existent_file', anotherReadableFile, readableFile ] + var candidates = ['non_existent_file', anotherReadableFile, readableFile] var found = configure.test.findAccessibleSync('test', dir, candidates) t.strictEqual(found, path.resolve(dir, anotherReadableFile)) }) diff --git a/test/test-find-node-directory.js b/test/test-find-node-directory.js index 767b6f6b37..f1380d162a 100644 --- a/test/test-find-node-directory.js +++ b/test/test-find-node-directory.js @@ -4,7 +4,7 @@ const test = require('tap').test const path = require('path') const findNodeDirectory = require('../lib/find-node-directory') -const platforms = [ 'darwin', 'freebsd', 'linux', 'sunos', 'win32', 'aix' ] +const platforms = ['darwin', 'freebsd', 'linux', 'sunos', 'win32', 'aix'] // we should find the directory based on the directory // the script is running in and it should match the layout @@ -62,8 +62,10 @@ test('test find-node-directory - node in build release dir', function (t) { if (platforms[next] === 'win32') { processObj = { execPath: '/x/y/Release/node', platform: platforms[next] } } else { - processObj = { execPath: '/x/y/out/Release/node', - platform: platforms[next] } + processObj = { + execPath: '/x/y/out/Release/node', + platform: platforms[next] + } } t.equal( diff --git a/test/test-process-release.js b/test/test-process-release.js index 42ddbd2cda..c3ee0703c5 100644 --- a/test/test-process-release.js +++ b/test/test-process-release.js @@ -358,7 +358,7 @@ test('test process release - process.release ~ node@4.0.0-rc.4 passed as argv[0] // note the missing 'v' on the arg, it should normalise when checking // whether we're on the default or not - var release = processRelease([ '4.0.0-rc.4' ], { opts: {} }, 'v4.0.0-rc.4', { + var release = processRelease(['4.0.0-rc.4'], { opts: {} }, 'v4.0.0-rc.4', { name: 'node', headersUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz' }) @@ -384,7 +384,7 @@ test('test process release - process.release ~ node@4.0.0-rc.4 - bogus string pa // additional arguments can be passed in on the commandline that should be ignored if they // are not specifying a valid version @ position 0 - var release = processRelease([ 'this is no version!' ], { opts: {} }, 'v4.0.0-rc.4', { + var release = processRelease(['this is no version!'], { opts: {} }, 'v4.0.0-rc.4', { name: 'node', headersUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz' }) From 968c9067d7a25a80925b9f7a342b67a04c3f76d8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Wed, 2 Oct 2019 01:50:59 +0100 Subject: [PATCH 094/551] win: support VS 2017 Desktop Express Fixes: https://github.com/nodejs/node-gyp/issues/1881 PR-URL: https://github.com/nodejs/node-gyp/pull/1902 Reviewed-By: Rod Vagg --- lib/find-visualstudio.js | 19 ++++++++++------ test/fixtures/VS_2017_Express.txt | 1 + test/test-find-visualstudio.js | 36 ++++++++++++++++++++++++++++--- 3 files changed, 47 insertions(+), 9 deletions(-) create mode 100644 test/fixtures/VS_2017_Express.txt diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js index b2c00e6835..c5d26f9a20 100644 --- a/lib/find-visualstudio.js +++ b/lib/find-visualstudio.js @@ -279,15 +279,22 @@ VisualStudioFinder.prototype = { // Helper - process toolset information getToolset: function getToolset (info, versionYear) { const pkg = 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64' + const express = 'Microsoft.VisualStudio.WDExpress' + if (info.packages.indexOf(pkg) !== -1) { this.log.silly('- found VC.Tools.x86.x64') - if (versionYear === 2017) { - return 'v141' - } - if (versionYear === 2019) { - return 'v142' - } + } else if (info.packages.indexOf(express) !== -1) { + this.log.silly('- found Visual Studio Express (looking for toolset)') + } else { + return null + } + + if (versionYear === 2017) { + return 'v141' + } else if (versionYear === 2019) { + return 'v142' } + this.log.silly('- invalid versionYear:', versionYear) return null }, diff --git a/test/fixtures/VS_2017_Express.txt b/test/fixtures/VS_2017_Express.txt new file mode 100644 index 0000000000..c4b3b5f2b0 --- /dev/null +++ b/test/fixtures/VS_2017_Express.txt @@ -0,0 +1 @@ +[{"path":"C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\WDExpress","version":"15.9.28307.858","packages":["Microsoft.VisualStudio.Product.WDExpress","Microsoft.VisualStudio.Workload.WDExpress","Microsoft.VisualStudio.Component.Windows10SDK.17763","MLGen","Win10SDK_10.0.17763","Microsoft.VisualStudio.Component.Windows10SDK.14393","Win10SDK_10.0.14393.795","Microsoft.VisualStudio.VC.Items.Pro","Microsoft.VisualStudio.VC.Ide.Pro","Microsoft.VisualStudio.VC.Ide.Pro.Resources","Microsoft.VisualStudio.Component.VC.Tools.ARM64","Microsoft.VisualStudio.VC.MSBuild.Arm64","Microsoft.VisualCpp.CRT.Redist.ARM64.OneCore.Desktop","Microsoft.VisualCpp.CRT.Redist.ARM64","Microsoft.VisualCpp.CRT.ARM64.OneCore.Desktop","Microsoft.VisualCpp.CRT.ARM64.Store","Microsoft.VisualCpp.CRT.ARM64.Desktop","Microsoft.VisualCpp.Tools.Hostx86.Targetarm64","Microsoft.VisualCpp.VCTip.hostX86.targetARM64","Microsoft.VisualCpp.Tools.HostX86.TargetARM64.Resources","Microsoft.VisualStudio.Component.VC.Tools.ARM","Microsoft.VisualCpp.Tools.Hostx86.Targetarm","Microsoft.VisualCpp.VCTip.hostX86.targetARM","Microsoft.VisualCpp.Tools.HostX86.TargetARM.Resources","Microsoft.VisualCpp.CRT.x86.Store","Microsoft.VisualCpp.CRT.x86.OneCore.Desktop","Microsoft.VisualCpp.CRT.x64.Store","Microsoft.VisualCpp.CRT.x64.OneCore.Desktop","Microsoft.VisualCpp.CRT.Redist.arm.OneCore.Desktop","Microsoft.VisualCpp.CRT.arm.OneCore.Desktop","Microsoft.VisualCpp.CRT.arm.Store","Microsoft.VisualCpp.CRT.arm.Desktop","Microsoft.VisualStudio.VC.Templates.UnitTest","Microsoft.VisualStudio.TestTools.TestPlatform.V1.CPP","Microsoft.VisualStudio.VC.Templates.UnitTest.Resources","Microsoft.VisualStudio.VC.Templates.Desktop","Microsoft.VisualStudio.VC.Templates.Pro","Microsoft.VisualStudio.VC.Templates.Pro.Resources","Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Express","Microsoft.VisualStudio.PackageGroup.Debugger.Script","Microsoft.VisualStudio.JavaScript.LanguageService","Microsoft.VisualStudio.JavaScript.LanguageService.Resources","Microsoft.VisualStudio.Debugger.Script.Msi","Microsoft.VisualStudio.Debugger.Script","Microsoft.VisualStudio.Debugger.Script","Microsoft.VisualStudio.Debugger.Script.Resources","Microsoft.VisualStudio.Debugger.Script.Resources","Microsoft.VisualStudio.VC.MSBuild.X64","Microsoft.VS.VC.MSBuild.X64.Resources","Microsoft.VisualStudio.VC.MSBuild.ARM","Microsoft.VisualStudio.VC.MSBuild.X86","Microsoft.VisualStudio.VC.MSBuild.Base","Microsoft.VisualStudio.VC.MSBuild.Base.Resources","Microsoft.VisualStudio.VC.Ide.WinXPlus","Microsoft.VisualStudio.VC.Ide.Dskx","Microsoft.VisualStudio.VC.Ide.Dskx.Resources","Microsoft.VisualStudio.VC.Ide.Core","Microsoft.VisualStudio.VC.Ide.Core.Resources","Microsoft.VisualStudio.VC.Ide.Base","Microsoft.VisualStudio.VC.Ide.Base.Resources","Microsoft.VisualStudio.Component.VC.CLI.Support","Microsoft.VisualCpp.CLI.X86","Microsoft.VisualCpp.CLI.X64","Microsoft.VisualCpp.CLI.Source","Microsoft.VisualCpp.CLI.ARM64","Microsoft.VisualCpp.CLI.ARM","Microsoft.VisualStudio.VC.Templates.CLR","Microsoft.VisualStudio.VC.Ide.LanguageService","Microsoft.VisualStudio.VC.Ide.ResourceEditor","Microsoft.VisualStudio.VC.Ide.ResourceEditor.Resources","Microsoft.VisualStudio.VC.Ide.ProjectSystem","Microsoft.VisualStudio.VC.Ide.ProjectSystem.Resources","Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine","Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine.Resources","Microsoft.VisualStudio.VC.Ide.LanguageService.Resources","Microsoft.VisualStudio.VC.Templates.CLR.Resources","Microsoft.Component.VC.Runtime.OSSupport","Microsoft.Windows.UniversalCRT.Tools.Msi","Microsoft.Windows.UniversalCRT.Tools.Msi","Microsoft.Windows.UniversalCRT.ExtensionSDK.Msi","Microsoft.Windows.UniversalCRT.HeadersLibsSources.Msi","Microsoft.VisualStudio.PackageGroup.VC.Tools.x86","Microsoft.VisualCpp.Tools.HostX86.TargetX64","Microsoft.VisualCpp.VCTip.hostX86.targetX64","Microsoft.VisualCpp.Tools.Hostx86.Targetx64.Resources","Microsoft.VisualCpp.Tools.HostX86.TargetX86","Microsoft.VisualCpp.VCTip.hostX86.targetX86","Microsoft.VisualCpp.Tools.HostX86.TargetX86.Resources","Microsoft.VisualCpp.Tools.Core.Resources","Microsoft.VisualCpp.Tools.Core.x86","Microsoft.VisualCpp.Tools.Common.Utils","Microsoft.VisualCpp.Tools.Common.Utils.Resources","Microsoft.VisualCpp.DIA.SDK","Microsoft.VisualCpp.CRT.x86.Desktop","Microsoft.VisualCpp.CRT.x64.Desktop","Microsoft.VisualCpp.CRT.Source","Microsoft.VisualCpp.CRT.Redist.X86","Microsoft.VisualCpp.CRT.Redist.X64","Microsoft.VisualCpp.CRT.Redist.Resources","Microsoft.VisualCpp.RuntimeDebug.14","Microsoft.VisualCpp.RuntimeDebug.14","Microsoft.VisualCpp.CRT.Headers","Microsoft.Component.HelpViewer","Microsoft.HelpViewer","Microsoft.VisualStudio.Help.Configuration.Msi","Microsoft.VisualStudio.Component.SQL.DataSources","Microsoft.VisualStudio.Component.SQL.SSDT","Microsoft.VisualStudio.Component.SQL.CMDUtils","sqlcmdlnutils","Microsoft.VisualStudio.Component.Common.Azure.Tools","Microsoft.VisualStudio.Azure.CommonAzureTools","SSDT","Microsoft.VisualStudio.Component.SQL.ADAL","sql_adalsql","Microsoft.VisualStudio.Component.NuGet","Microsoft.CredentialProvider","Microsoft.VisualStudio.NuGet.Licenses","Microsoft.VisualStudio.Component.SQL.LocalDB.Runtime","Microsoft.VisualStudio.Component.SQL.NCLI","sqllocaldb","sqlncli","Microsoft.VisualStudio.Component.EntityFramework","Microsoft.VisualStudio.PackageGroup.DslRuntime","Microsoft.VisualStudio.Dsl.Core","Microsoft.VisualStudio.Dsl.GraphObject","Microsoft.VisualStudio.Dsl.Core.Resources","Microsoft.VisualStudio.EntityFrameworkTools","Microsoft.VisualStudio.EntityFrameworkTools.Msi","Microsoft.VisualStudio.Component.Roslyn.LanguageServices","Microsoft.VisualStudio.InteractiveWindow","Microsoft.DiaSymReader.Native","Microsoft.VisualStudio.Component.Static.Analysis.Tools","Microsoft.VisualCpp.Redist.14","Microsoft.VisualCpp.Redist.14","Microsoft.VisualStudio.StaticAnalysis","Microsoft.VisualStudio.StaticAnalysis.Resources","Microsoft.CodeAnalysis.VisualStudio.InteractiveComponents.Resources","Microsoft.CodeAnalysis.VisualStudio.InteractiveComponents","Microsoft.CodeAnalysis.VisualStudio.Setup.Interactive.Resources","Microsoft.Net.ComponentGroup.TargetingPacks.Common","Microsoft.Net.Component.4.6.TargetingPack","Microsoft.Net.4.6.TargetingPack","Microsoft.Net.Component.4.5.2.TargetingPack","Microsoft.Net.4.5.2.TargetingPack","Microsoft.Net.Component.4.5.1.TargetingPack","Microsoft.Net.4.5.1.TargetingPack","Microsoft.Net.Component.4.5.TargetingPack","Microsoft.Net.4.5.TargetingPack","Microsoft.Net.Component.4.TargetingPack","Microsoft.Net.4.TargetingPack","Microsoft.Net.ComponentGroup.DevelopmentPrerequisites","Microsoft.Net.Component.4.6.1.TargetingPack","Microsoft.Net.4.6.1.TargetingPack","Microsoft.Net.Cumulative.TargetingPack.Resources","Microsoft.Net.Component.4.6.1.SDK","Microsoft.Net.4.6.1.SDK","Microsoft.VisualStudio.Component.TextTemplating","Microsoft.VisualStudio.TextTemplating.MSBuild","Microsoft.VisualStudio.TextTemplating.Integration","Microsoft.VisualStudio.TextTemplating.Core","Microsoft.VisualStudio.TextTemplating.Integration.Resources","Microsoft.VisualStudio.Component.VisualStudioData","Microsoft.VisualStudio.Component.SQL.CLR","Microsoft.VisualStudio.ProTools","sqlsysclrtypes","sqlsysclrtypes","SQLCommon","Microsoft.VisualStudio.ProTools.Resources","Microsoft.VisualStudio.XamlDiagnostics","Microsoft.VisualStudio.XamlDiagnostics.Resources","Microsoft.VisualStudio.XamlDesigner","Microsoft.VisualStudio.XamlDesigner.Resources","Microsoft.VisualStudio.XamlDesigner.Executables","Microsoft.VisualStudio.XamlShared","Microsoft.VisualStudio.XamlShared.Resources","Microsoft.VisualStudio.PackageGroup.TestTools.Managed","Microsoft.VisualStudio.PackageGroup.IntelliTrace.Core","Microsoft.IntelliTrace.Core","Microsoft.IntelliTrace.Core.Targeted","Microsoft.IntelliTrace.ProfilerProxy.Msi.x64","Microsoft.IntelliTrace.ProfilerProxy.Msi","Microsoft.VisualStudio.NuGet.Core","Microsoft.VisualStudio.TestWindow.SourceBasedTestDiscovery","Microsoft.VisualStudio.TestWindow.Dotnet","Microsoft.VisualStudio.TestTools.TestGeneration","Microsoft.VisualStudio.PackageGroup.TestTools.CodeCoverage","Microsoft.VisualStudio.PackageGroup.TestTools.Enterprise","Microsoft.VisualStudio.PackageGroup.TestTools.MSTestV2.Managed","Microsoft.VisualStudio.TestTools.MSTestV2.WizardExtension.UnitTest","Microsoft.VisualStudio.PackageGroup.TestTools.Core","Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.V2.CLI","Microsoft.VisualStudio.TestTools.TestPlatform.V2.CLI","Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.V1.CLI","Microsoft.VisualStudio.TestTools.TestPlatform.V1.CLI","Microsoft.VisualStudio.TestTools.Pex.Common","Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.Legacy","Microsoft.VisualStudio.PackageGroup.MinShell.Interop","Microsoft.VisualStudio.TestTools.TP.Legacy.Tips.Msi","Microsoft.VisualStudio.TestTools.TP.Legacy.Tips.Common","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Tips","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Tips.Resources","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.TestTools","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Professional","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Common","Microsoft.VisualStudio.TestTools.TP.Legacy.Common.Res","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Core","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Core.Resources","Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Agent","Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.IDE","Microsoft.VisualStudio.TestTools.TestWIExtension","Microsoft.VisualStudio.TestTools.TestPlatform.IDE","Microsoft.VisualStudio.PackageGroup.TestTools.DataCollectors","Microsoft.Component.ClickOnce","Microsoft.VisualStudio.PackageGroup.ClickOnce.MSBuild","Microsoft.VisualCpp.CRT.ClickOnce.Msi","Microsoft.ClickOnce.SignTool.Msi","Microsoft.SQL.ClickOnceBootstrapper.Msi","Microsoft.Net.ClickOnceBootstrapper","Microsoft.ClickOnce.BootStrapper.Msi.Resources","Microsoft.ClickOnce.BootStrapper.Msi","Microsoft.VisualStudio.WebTools.WSP.FSA","Microsoft.VisualStudio.WebTools.WSP.FSA.Resources","Microsoft.VisualStudio.PackageGroup.Community","Microsoft.VisualStudio.Community.Extra.Resources","Microsoft.VisualStudio.Community.Extra","Microsoft.VisualStudio.PackageGroup.Core","Microsoft.VisualStudio.TestTools.TeamFoundationClient","Microsoft.VisualStudio.PackageGroup.Debugger.Core","Microsoft.VisualStudio.Debugger.VSCodeDebuggerHost","Microsoft.VisualStudio.VC.Ide.Debugger","Microsoft.VisualStudio.VC.Ide.Debugger.Resources","Microsoft.VisualStudio.VC.Ide.Common","Microsoft.VisualStudio.VC.Ide.Common.Resources","Microsoft.VisualStudio.Debugger.Parallel","Microsoft.VisualStudio.Debugger.Parallel.Resources","Microsoft.VisualStudio.Debugger.CollectionAgents","Microsoft.VisualStudio.Debugger.Managed","Microsoft.CodeAnalysis.VisualStudio.Setup.Resources","Microsoft.CodeAnalysis.VisualStudio.Setup","Microsoft.CodeAnalysis.ExpressionEvaluator.Resources","Microsoft.CodeAnalysis.ExpressionEvaluator","Microsoft.VisualStudio.Debugger.Managed.Resources","Microsoft.VisualStudio.Debugger.Remote","Microsoft.VisualStudio.Debugger.Remote","Microsoft.VisualStudio.Debugger.Remote.Resources","Microsoft.VisualStudio.Debugger.Remote.Resources","Microsoft.VisualStudio.Debugger","Microsoft.VisualStudio.VC.MSVCDis","Microsoft.VisualStudio.ScriptedHost","Microsoft.VisualStudio.ScriptedHost.Targeted","Microsoft.VisualStudio.ScriptedHost.Resources","Microsoft.IntelliTrace.DiagnosticsHub","Microsoft.VisualStudio.Debugger.Resources","Microsoft.PackageGroup.ClientDiagnostics","Microsoft.VisualStudio.AppResponsiveness","Microsoft.VisualStudio.AppResponsiveness.Targeted","Microsoft.VisualStudio.AppResponsiveness.Resources","Microsoft.VisualStudio.ClientDiagnostics","Microsoft.VisualStudio.ClientDiagnostics.Targeted","Microsoft.VisualStudio.ClientDiagnostics.Resources","Microsoft.VisualStudio.PackageGroup.CommunityCore","Microsoft.VisualStudio.ProjectSystem.Full","Microsoft.VisualStudio.ProjectSystem","Microsoft.VisualStudio.Community.x86","Microsoft.VisualStudio.Community.x64","Microsoft.VisualStudio.Community","Microsoft.IntelliTrace.CollectorCab","Microsoft.VisualStudio.Community.Resources","Microsoft.VisualStudio.Net.Eula.Resources","Microsoft.VisualStudio.WebSiteProject.DTE","Microsoft.MSHtml","Microsoft.VisualStudio.Community.Msi.Resources","Microsoft.VisualStudio.Community.Msi","Microsoft.VisualStudio.MinShell.Interop.Msi","Microsoft.VisualStudio.Editors","Microsoft.VisualStudio.ClickOnce.Resources","Microsoft.VisualStudio.ClickOnce","Microsoft.Component.MSBuild","Microsoft.NuGet.Build.Tasks","Microsoft.VisualStudio.Component.Roslyn.Compiler","Microsoft.CodeAnalysis.Compilers.Resources","Microsoft.CodeAnalysis.Compilers","Microsoft.Net.PackageGroup.4.6.1.Redist","Microsoft.VisualStudio.TemplateEngine","Microsoft.VisualStudio.WebToolsExtensions.Common","Microsoft.NET.Sdk","Microsoft.VisualStudio.PackageGroup.TestTools.Templates.Managed","Microsoft.VisualStudio.TestTools.Templates.Managed","Microsoft.VisualStudio.TestTools.Templates.Managed.Resources","Microsoft.VisualStudio.Templates.VB.MSTestv2.Desktop.UnitTest","Microsoft.VisualStudio.Templates.CS.MSTestv2.Desktop.UnitTest","Microsoft.VisualStudio.Templates.VB.Wpf","Microsoft.VisualStudio.Templates.VB.Wpf.Resources","Microsoft.VisualStudio.Templates.VB.Winforms","Microsoft.VisualStudio.Templates.VB.ManagedCore","Microsoft.VisualStudio.Templates.VB.Shared","Microsoft.VisualStudio.Templates.VB.Shared.Resources","Microsoft.VisualStudio.Templates.VB.ManagedCore.Resources","Microsoft.VisualStudio.Templates.CS.GettingStarted.Desktop.Package","Microsoft.VisualStudio.Templates.GetStarted.Desktop.Setup","Microsoft.VisualStudio.Templates.CS.GettingStarted.Console.Package","Microsoft.VisualStudio.Templates.GetStarted.Resources","Microsoft.VisualStudio.Templates.GetStarted.Common.Setup","Microsoft.VisualStudio.Templates.GetStarted.Console.Setup","Microsoft.VisualStudio.Templates.CS.Wpf","Microsoft.VisualStudio.Templates.CS.Wpf.Resources","Microsoft.VisualStudio.Templates.CS.Winforms","Microsoft.VisualStudio.Templates.CS.ManagedCore","Microsoft.VisualStudio.Templates.CS.Shared","Microsoft.VisualStudio.Templates.Editorconfig.Wizard.Setup","Templates.Editorconfig.SolutionFile.Setup","Microsoft.VisualStudio.Templates.CS.Shared.Resources","Microsoft.VisualStudio.Templates.CS.ManagedCore.Resources","Microsoft.VisualStudio.Component.CoreEditor","Microsoft.VisualStudio.PackageGroup.CoreEditor","PortableFacades","Microsoft.VisualStudio.PackageGroup.VsDevCmd","Microsoft.VisualStudio.VsDevCmd.Ext.NetFxSdk","Microsoft.VisualStudio.VsDevCmd.Core.WinSdk","Microsoft.VisualStudio.VsDevCmd.Core.DotNet","Microsoft.VisualStudio.VC.DevCmd","Microsoft.VisualStudio.VC.DevCmd.Resources","Microsoft.VisualStudio.VirtualTree","Microsoft.VisualStudio.PackageGroup.Progression","Microsoft.VisualStudio.PerformanceProvider","Microsoft.VisualStudio.GraphModel","Microsoft.VisualStudio.GraphProvider","Microsoft.DiaSymReader","Microsoft.Build.Dependencies","Microsoft.Build.FileTracker.Msi","Microsoft.Build","Microsoft.VisualStudio.TextMateGrammars","Microsoft.VisualStudio.PackageGroup.TeamExplorer","Microsoft.TeamFoundation.OfficeIntegration","Microsoft.TeamFoundation.OfficeIntegration.Resources","Microsoft.VisualStudio.TeamExplorer","Microsoft.ServiceHub","Microsoft.VisualStudio.ProjectServices","Microsoft.VisualStudio.SLNX.VSIX","Microsoft.VisualStudio.FileHandler.Msi","Microsoft.VisualStudio.FileHandler.Msi","Microsoft.VisualStudio.PackageGroup.MinShell","Microsoft.VisualStudio.MinShell.Interop","Microsoft.VisualStudio.Log","Microsoft.VisualStudio.Log.Targeted","Microsoft.VisualStudio.Log.Resources","Microsoft.VisualStudio.Finalizer","Microsoft.VisualStudio.WDExpress","Microsoft.VisualStudio.WDExpress.Resources","Microsoft.VisualStudio.CoreEditor","Microsoft.VisualStudio.Connected","Microsoft.VisualStudio.Connected.Resources","Microsoft.VisualStudio.MinShell","Microsoft.VisualStudio.Setup.Configuration","Microsoft.VisualStudio.MinShell.Platform","Microsoft.VisualStudio.MinShell.Platform.Resources","Microsoft.VisualStudio.MefHosting","Microsoft.VisualStudio.MefHosting.Resources","Microsoft.VisualStudio.Initializer","Microsoft.VisualStudio.ExtensionManager","Microsoft.VisualStudio.MinShell.x86","Microsoft.VisualStudio.NativeImageSupport","Microsoft.VisualStudio.MinShell.Msi","Microsoft.VisualStudio.MinShell.Msi.Resources","Microsoft.VisualStudio.LanguageServer","Microsoft.VisualStudio.MinShell.Resources","Microsoft.Net.PackageGroup.4.6.Redist","Microsoft.VisualStudio.Branding.WDExpress"]}] diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js index b00adf0227..1327cf8841 100644 --- a/test/test-find-visualstudio.js +++ b/test/test-find-visualstudio.js @@ -296,6 +296,34 @@ test('VS2017 Community with C++ workload', function (t) { finder.findVisualStudio() }) +test('VS2017 Express', function (t) { + t.plan(2) + + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info, { + msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\' + + 'WDExpress\\MSBuild\\15.0\\Bin\\MSBuild.exe', + path: + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\WDExpress', + sdk: '10.0.17763.0', + toolset: 'v141', + version: '15.9.28307.858', + versionMajor: 15, + versionMinor: 9, + versionYear: 2017 + }) + }) + + poison(finder, 'regSearchKeys') + finder.findVisualStudio2017OrNewer = (cb) => { + const file = path.join(__dirname, 'fixtures', 'VS_2017_Express.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '', cb) + } + finder.findVisualStudio() +}) + test('VS2019 Preview with C++ workload', function (t) { t.plan(2) @@ -392,13 +420,15 @@ function allVsVersions (t, finder) { const data2 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', 'VS_2017_Community_workload.txt'))) const data3 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', - 'VS_2019_Preview.txt'))) + 'VS_2017_Express.txt'))) const data4 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', - 'VS_2019_BuildTools_minimal.txt'))) + 'VS_2019_Preview.txt'))) const data5 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', + 'VS_2019_BuildTools_minimal.txt'))) + const data6 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', 'VS_2019_Community_workload.txt'))) const data = JSON.stringify(data0.concat(data1, data2, data3, data4, - data5)) + data5, data6)) finder.parseData(null, data, '', cb) } finder.regSearchKeys = (keys, value, addOpts, cb) => { From f60ed47d1449c32fa855c1a5ba4c75c0589d67d0 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Wed, 2 Oct 2019 17:55:08 +0000 Subject: [PATCH 095/551] travis: add Python 3.5 and 3.6 tests on Linux MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Support the same Python versions as the Python Core Team: https://devguide.python.org/#branchstatus PR-URL: https://github.com/nodejs/node-gyp/pull/1903 Reviewed-By: Rod Vagg Reviewed-By: Sam Roberts Reviewed-By: João Reis Reviewed-By: Richard Lau --- .travis.yml | 10 ++++++++++ README.md | 8 ++++---- 2 files changed, 14 insertions(+), 4 deletions(-) diff --git a/.travis.yml b/.travis.yml index 2d4b010df8..c390c45648 100644 --- a/.travis.yml +++ b/.travis.yml @@ -41,10 +41,20 @@ matrix: python: 3.7 env: NODE_GYP_FORCE_PYTHON=python3 before_install: nvm install 10 + + - name: "Node.js 12 & Python 3.5 on Linux" + python: 3.5 + env: NODE_GYP_FORCE_PYTHON=python3 + before_install: nvm install 12 + - name: "Node.js 12 & Python 3.6 on Linux" + python: 3.6 + env: NODE_GYP_FORCE_PYTHON=python3 + before_install: nvm install 12 - name: "Node.js 12 & Python 3.7 on Linux" python: 3.7 env: NODE_GYP_FORCE_PYTHON=python3 before_install: nvm install 12 + - name: "Python 3.7 on macOS" os: osx #osx_image: xcode11 diff --git a/README.md b/README.md index d54098773b..118e720964 100644 --- a/README.md +++ b/README.md @@ -25,23 +25,23 @@ $ npm install -g node-gyp You will also need to install: -NOTE: node-gyp is compatible with Python 2.7 and 3.7 but node itself is not yet compatible with Python 3. +NOTE: node-gyp is compatible with Python v2.7, v3.5, v3.6, or v3.7 but node itself is not yet compatible with Python 3. ### On Unix - * `Python v2.7 or v3.7` (v3.5 and 3.6 may work but are not currently tested) + * `Python v2.7, v3.5, v3.6, or v3.7` * `make` * A proper C/C++ compiler toolchain, like [GCC](https://gcc.gnu.org) ### On macOS - * `Python v2.7 or v3.7` (v3.5 and 3.6 may work but are not currently tested) + * `Python v2.7, v3.5, v3.6, or v3.7` * [Xcode](https://developer.apple.com/xcode/download/) * You also need to install the `XCode Command Line Tools` by running `xcode-select --install`. Alternatively, if you already have the full Xcode installed, you can find them under the menu `Xcode -> Open Developer Tool -> More Developer Tools...`. This step will install `clang`, `clang++`, and `make`. ### On Windows -Install the current version of Python from the [Microsoft Store package](https://docs.python.org/3.7/using/windows.html#the-microsoft-store-package). +Install the current version of Python from the [Microsoft Store package](https://docs.python.org/3/using/windows.html#the-microsoft-store-package). #### Option 1 From f36bd228a40d1d8b6fedab8ed650a4fb196ba228 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Wed, 2 Oct 2019 20:20:56 +0100 Subject: [PATCH 096/551] gyp: add __lt__ to MSVSSolutionEntry PR-URL: https://github.com/nodejs/node-gyp/pull/1904 Reviewed-By: Richard Lau Reviewed-By: Christian Clauss --- gyp/pylib/gyp/MSVSNew.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/gyp/pylib/gyp/MSVSNew.py b/gyp/pylib/gyp/MSVSNew.py index 9b64e2c1c8..76c4b95c0c 100644 --- a/gyp/pylib/gyp/MSVSNew.py +++ b/gyp/pylib/gyp/MSVSNew.py @@ -59,6 +59,9 @@ def __cmp__(self, other): # Sort by name then guid (so things are in order on vs2008). return cmp((self.name, self.get_guid()), (other.name, other.get_guid())) + def __lt__(self, other): + return self.__cmp__(other) < 0 + class MSVSFolder(MSVSSolutionEntry): """Folder in a Visual Studio project or solution.""" From dd0e97ef0bd168fcc05c379b819666074466d330 Mon Sep 17 00:00:00 2001 From: Sam Roberts Date: Thu, 3 Oct 2019 08:18:36 -0700 Subject: [PATCH 097/551] lib: try to find `python` after `python3` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Unadorned `python` can be either Python 2 or Python 3, but it is likely to be Python 2 for quite a while. To find Python3, it is recommended to use the explicit name `python3`. See: - https://www.python.org/dev/peps/pep-0394/#for-python-runtime-distributors - https://github.com/nodejs/node-gyp/pull/1892#issuecomment-537637433 PR-URL: https://github.com/nodejs/node-gyp/pull/1907 Reviewed-By: Christian Clauss Reviewed-By: Rod Vagg Reviewed-By: João Reis --- lib/find-python.js | 26 +++++++++++++------------- test/test-find-python.js | 15 +++++++-------- 2 files changed, 20 insertions(+), 21 deletions(-) diff --git a/lib/find-python.js b/lib/find-python.js index c12ccc3bb2..9d918a881b 100644 --- a/lib/find-python.js +++ b/lib/find-python.js @@ -26,8 +26,8 @@ PythonFinder.prototype = { win: win, pyLauncher: 'py.exe', winDefaultLocations: [ - path.join(process.env.SystemDrive || 'C:', 'Python27', 'python.exe'), - path.join(process.env.SystemDrive || 'C:', 'Python37', 'python.exe') + path.join(process.env.SystemDrive || 'C:', 'Python37', 'python.exe'), + path.join(process.env.SystemDrive || 'C:', 'Python27', 'python.exe') ], // Logs a message at verbose level, but also saves it to be displayed later @@ -88,14 +88,14 @@ PythonFinder.prototype = { arg: this.env.PYTHON }, { - before: () => { this.addLog('checking if "python" can be used') }, + before: () => { this.addLog('checking if "python3" can be used') }, check: this.checkCommand, - arg: 'python' + arg: 'python3' }, { - before: () => { this.addLog('checking if "python3" can be used') }, + before: () => { this.addLog('checking if "python" can be used') }, check: this.checkCommand, - arg: 'python3' + arg: 'python' }, { before: () => { this.addLog('checking if "python2" can be used') }, @@ -105,13 +105,6 @@ PythonFinder.prototype = { ] if (this.win) { - checks.push({ - before: () => { - this.addLog( - 'checking if the py launcher can be used to find Python 2') - }, - check: this.checkPyLauncher - }) for (var i = 0; i < this.winDefaultLocations.length; ++i) { const location = this.winDefaultLocations[i] checks.push({ @@ -123,6 +116,13 @@ PythonFinder.prototype = { arg: location }) } + checks.push({ + before: () => { + this.addLog( + 'checking if the py launcher can be used to find Python 2') + }, + check: this.checkPyLauncher + }) } return checks diff --git a/test/test-find-python.js b/test/test-find-python.js index 1c86f45b73..6ca522a04c 100644 --- a/test/test-find-python.js +++ b/test/test-find-python.js @@ -159,6 +159,8 @@ test('find python - no python, use python launcher', function (t) { } if (/sys\.executable/.test(args[args.length - 1])) { cb(new Error('not found')) + } else if (f.winDefaultLocations.includes(program)) { + cb(new Error('not found')) } else if (/sys\.version_info/.test(args[args.length - 1])) { if (program === 'Z:\\snake.exe') { cb(null, '2.7.14') @@ -178,24 +180,21 @@ test('find python - no python, use python launcher', function (t) { }) test('find python - no python, no python launcher, good guess', function (t) { - t.plan(4) + t.plan(2) - var re = /C:[\\/]Python27[\\/]python[.]exe/ + var re = /C:[\\/]Python37[\\/]python[.]exe/ var f = new TestPythonFinder(null, done) f.win = true f.execFile = function (program, args, opts, cb) { if (program === 'py.exe') { - f.execFile = function (program, args, opts, cb) { - poison(f, 'execFile') - t.ok(re.test(program)) - t.ok(/sys\.version_info/.test(args[args.length - 1])) - cb(null, '2.7.14') - } return cb(new Error('not found')) } if (/sys\.executable/.test(args[args.length - 1])) { cb(new Error('not found')) + } else if (re.test(program) && + /sys\.version_info/.test(args[args.length - 1])) { + cb(null, '3.7.3') } else { t.fail() } From b1bf32ed1b633d31bad078b2ab2fd5daaa9c4a2c Mon Sep 17 00:00:00 2001 From: Sam Roberts Date: Thu, 3 Oct 2019 08:42:36 -0700 Subject: [PATCH 098/551] doc: clarify Python configuration, etc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Clarify that: - node-gyp is not used to build Node.js - PYTHON can be used to add a Python search path - NODE_GYP_FORCE_PYTHON can be used to override all Python search paths - That a compatible Python is searched for PR-URL: https://github.com/nodejs/node-gyp/pull/1908 Reviewed-By: Christian Clauss Reviewed-By: Rod Vagg Reviewed-By: João Reis Reviewed-By: Richard Lau --- README.md | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 118e720964..7597f03c85 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,12 @@ # `node-gyp` - Node.js native addon build tool -`node-gyp` is a cross-platform command-line tool written in Node.js for compiling -native addon modules for Node.js. It bundles the [gyp](https://gyp.gsrc.io) -project used by the Chromium team and takes away the pain of dealing with the -various differences in build platforms. +`node-gyp` is a cross-platform command-line tool written in Node.js for +compiling native addon modules for Node.js. It contains a fork of the +[gyp](https://gyp.gsrc.io) project that was previously used by the Chromium +team and takes away the pain of dealing with the various differences in build +platforms. + +Note that `node-gyp` is _not_ used to build Node.js itself. Multiple target versions of Node.js are supported (i.e. `0.8`, ..., `4`, `5`, `6`, etc.), regardless of what version of Node.js is actually installed on your system @@ -25,7 +28,9 @@ $ npm install -g node-gyp You will also need to install: -NOTE: node-gyp is compatible with Python v2.7, v3.5, v3.6, or v3.7 but node itself is not yet compatible with Python 3. +NOTE: node-gyp is compatible with Python v2.7, v3.5, v3.6, or v3.7. If the +Python to use is not explicitly configured (see "Configuring Python Dependency" +below) it will attempt to find a compatible Python executable. ### On Unix @@ -76,6 +81,14 @@ value: $ npm config set python /path/to/executable/python ``` +If the `PYTHON` environment variable is set to the path of a Python executable, +it will be used if it is a compatible Python. + +If the `NODE_GYP_FORCE_PYTHON` environment variable is set to the path of a +Python executable, it will be used instead of any of the other configured or +builtin Python search paths. If its not a compatible Python, no further +searching will be done. + ## How to Use To compile your native addon, first go to its root directory: From 77803c314b5360e17de76f22058676625d2a9652 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Thu, 3 Oct 2019 11:29:43 +1000 Subject: [PATCH 099/551] v5.0.5: bump version and update changelog PR-URL: https://github.com/nodejs/node-gyp/pull/1905 --- CHANGELOG.md | 13 +++++++++++++ package.json | 2 +- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3dbc5d9244..ecc51b360d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,16 @@ +v5.0.5 2019-10-04 +================= + +* [[`3891391746`](https://github.com/nodejs/node-gyp/commit/3891391746)] - **doc**: reconcile README with Python 3 compat changes (Rod Vagg) [#1911](https://github.com/nodejs/node-gyp/pull/1911) +* [[`07f81f1920`](https://github.com/nodejs/node-gyp/commit/07f81f1920)] - **lib**: accept Python 3 after Python 2 (Sam Roberts) [#1910](https://github.com/nodejs/node-gyp/pull/1910) +* [[`04ce59f4a2`](https://github.com/nodejs/node-gyp/commit/04ce59f4a2)] - **doc**: clarify Python configuration, etc (Sam Roberts) [#1908](https://github.com/nodejs/node-gyp/pull/1908) +* [[`01c46ee3df`](https://github.com/nodejs/node-gyp/commit/01c46ee3df)] - **gyp**: add \_\_lt\_\_ to MSVSSolutionEntry (João Reis) [#1904](https://github.com/nodejs/node-gyp/pull/1904) +* [[`735d961b99`](https://github.com/nodejs/node-gyp/commit/735d961b99)] - **win**: support VS 2017 Desktop Express (João Reis) [#1902](https://github.com/nodejs/node-gyp/pull/1902) +* [[`3834156a92`](https://github.com/nodejs/node-gyp/commit/3834156a92)] - **test**: add Python 3.5 and 3.6 tests on Linux (cclauss) [#1909](https://github.com/nodejs/node-gyp/pull/1909) +* [[`1196e990d8`](https://github.com/nodejs/node-gyp/commit/1196e990d8)] - **src**: update to standard@14 (Rod Vagg) [#1899](https://github.com/nodejs/node-gyp/pull/1899) +* [[`53ee7dfe89`](https://github.com/nodejs/node-gyp/commit/53ee7dfe89)] - **gyp**: fix undefined name: cflags --\> ldflags (Christian Clauss) [#1901](https://github.com/nodejs/node-gyp/pull/1901) +* [[`5871dcf6c9`](https://github.com/nodejs/node-gyp/commit/5871dcf6c9)] - **src,win**: add support for fetching arm64 node.lib (Richard Townsend) [#1875](https://github.com/nodejs/node-gyp/pull/1875) + v5.0.4 2019-09-27 ================= diff --git a/package.json b/package.json index 634febf186..781cea9d7d 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "5.0.4", + "version": "5.0.5", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From 1a4ff636d598ebdfcea5cd468608e9acf1bd176c Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Thu, 26 Sep 2019 12:39:57 +1000 Subject: [PATCH 100/551] v6.0.0: bump version and update changelog PR-URL: https://github.com/nodejs/node-gyp/pull/1892 --- CHANGELOG.md | 9 +++++++++ package.json | 2 +- 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ecc51b360d..6f7809c625 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,12 @@ +v6.0.0 2019-10-04 +================= + +* [[`dd0e97ef0b`](https://github.com/nodejs/node-gyp/commit/dd0e97ef0b)] - **(SEMVER-MAJOR)** **lib**: try to find `python` after `python3` (Sam Roberts) [#1907](https://github.com/nodejs/node-gyp/pull/1907) +* [[`f60ed47d14`](https://github.com/nodejs/node-gyp/commit/f60ed47d14)] - **travis**: add Python 3.5 and 3.6 tests on Linux (Christian Clauss) [#1903](https://github.com/nodejs/node-gyp/pull/1903) +* [[`c763ca1838`](https://github.com/nodejs/node-gyp/commit/c763ca1838)] - **(SEMVER-MAJOR)** **doc**: Declare that node-gyp is Python 3 compatible (cclauss) [#1811](https://github.com/nodejs/node-gyp/pull/1811) +* [[`3d1c60ab81`](https://github.com/nodejs/node-gyp/commit/3d1c60ab81)] - **(SEMVER-MAJOR)** **lib**: accept Python 3 by default (João Reis) [#1844](https://github.com/nodejs/node-gyp/pull/1844) +* [[`c6e3b65a23`](https://github.com/nodejs/node-gyp/commit/c6e3b65a23)] - **(SEMVER-MAJOR)** **lib**: raise the minimum Python version from 2.6 to 2.7 (cclauss) [#1818](https://github.com/nodejs/node-gyp/pull/1818) + v5.0.5 2019-10-04 ================= diff --git a/package.json b/package.json index 781cea9d7d..7014dadf56 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "5.0.5", + "version": "6.0.0", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From 5a83630c337eebff52211e19f233bf294128b416 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Tue, 15 Oct 2019 19:38:02 +1100 Subject: [PATCH 101/551] travis: add Windows + Python 3.8 to the mix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR-URL: https://github.com/nodejs/node-gyp/pull/1921 Reviewed-By: Christian Clauss Reviewed-By: João Reis --- .travis.yml | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.travis.yml b/.travis.yml index c390c45648..87cbbdcc46 100644 --- a/.travis.yml +++ b/.travis.yml @@ -68,6 +68,14 @@ matrix: env: >- PATH=/c/Python37:/c/Python37/Scripts:$PATH NODE_GYP_FORCE_PYTHON=/c/Python37/python.exe + before_install: choco install python --version=3.7.4 + - name: "Node.js 12 & Python 3.8 on Windows" + os: windows + language: node_js + node_js: 12 # node + env: >- + PATH=/c/Python38:/c/Python38/Scripts:$PATH + NODE_GYP_FORCE_PYTHON=/c/Python38/python.exe before_install: choco install python install: From 032db2a2d09d42c3a259638b287da2f17449f03d Mon Sep 17 00:00:00 2001 From: Sam Hughes Date: Wed, 16 Oct 2019 16:24:37 +0100 Subject: [PATCH 102/551] lib,install: always download SHA sums on Windows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR-URL: https://github.com/nodejs/node-gyp/pull/1926 Reviewed-By: João Reis Reviewed-By: Richard Lau --- lib/install.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/install.js b/lib/install.js index b870532880..06e82871f7 100644 --- a/lib/install.js +++ b/lib/install.js @@ -224,8 +224,8 @@ function install (fs, gyp, argv, callback) { var installVersionPath = path.resolve(devDir, 'installVersion') fs.writeFile(installVersionPath, gyp.package.installVersion + '\n', deref) - // Only download SHASUMS.txt if not using tarPath override - if (!tarPath) { + // Only download SHASUMS.txt if we downloaded something in need of SHA verification + if (!tarPath || win) { // download SHASUMS.txt async++ downloadShasums(deref) From 60e4488f08e44e99a8a8a343bbf384997d02f4dd Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Mon, 21 Oct 2019 13:06:08 +0200 Subject: [PATCH 103/551] build: avoid bare exceptions in xcode_emulation.py https://realpython.com/the-most-diabolical-python-antipattern/ PR-URL: https://github.com/nodejs/node-gyp/pull/1932 Reviewed-By: Rod Vagg --- gyp/pylib/gyp/xcode_emulation.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py index 74735c033a..13964beec6 100644 --- a/gyp/pylib/gyp/xcode_emulation.py +++ b/gyp/pylib/gyp/xcode_emulation.py @@ -437,7 +437,7 @@ def _GetSdkVersionInfoItem(self, sdk, infoitem): # most sensible route and should still do the right thing. try: return GetStdoutQuiet(['xcodebuild', '-version', '-sdk', sdk, infoitem]) - except: + except GypError: pass def _SdkRoot(self, configname): @@ -1135,7 +1135,7 @@ def _DefaultSdkRoot(self): return default_sdk_root try: all_sdks = GetStdout(['xcodebuild', '-showsdks']) - except: + except GypError: # If xcodebuild fails, there will be no valid SDKs return '' for line in all_sdks.splitlines(): @@ -1276,7 +1276,7 @@ def XcodeVersion(): # checking that version. if len(version_list) < 2: raise GypError("xcodebuild returned unexpected results") - except: + except GypError: version = CLTVersion() if version: version = ".".join(version.split(".")[:3]) @@ -1314,7 +1314,7 @@ def CLTVersion(): try: output = GetStdout(['/usr/sbin/pkgutil', '--pkg-info', key]) return re.search(regex, output).groupdict()['version'] - except: + except GypError: continue From 4fff8458c07698783959182bf82ebb4ae36d53b0 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Tue, 22 Oct 2019 12:25:18 +0200 Subject: [PATCH 104/551] travis: ignore failed `brew upgrade npm`, update xcode PR-URL: https://github.com/nodejs/node-gyp/pull/1932 Reviewed-By: Rod Vagg --- .travis.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.travis.yml b/.travis.yml index 87cbbdcc46..518543a3c4 100644 --- a/.travis.yml +++ b/.travis.yml @@ -8,7 +8,7 @@ matrix: python: 2.7 - name: "Python 2.7 on macOS" os: osx - osx_image: xcode11 + osx_image: xcode11.2 language: shell # 'language: python' is not yet supported on macOS env: NODE_GYP_FORCE_PYTHON=python2 before_install: HOMEBREW_NO_AUTO_UPDATE=1 brew install npm @@ -57,10 +57,10 @@ matrix: - name: "Python 3.7 on macOS" os: osx - #osx_image: xcode11 + osx_image: xcode11.2 language: shell # 'language: python' is not yet supported on macOS env: NODE_GYP_FORCE_PYTHON=python3 - before_install: HOMEBREW_NO_AUTO_UPDATE=1 brew install npm + before_install: HOMEBREW_NO_AUTO_UPDATE=1 brew upgrade npm || true - name: "Node.js 12 & Python 3.7 on Windows" os: windows language: node_js From 3538a317b6f2bd863668fb2640f625a4fd4fb3b3 Mon Sep 17 00:00:00 2001 From: Dan Pike Date: Mon, 14 Oct 2019 18:12:43 +0100 Subject: [PATCH 105/551] doc: adjustments to the README.md for new users PR-URL: https://github.com/nodejs/node-gyp/pull/1919 Reviewed-By: Rod Vagg --- README.md | 38 ++++++++++++++++++-------------------- 1 file changed, 18 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index 7597f03c85..5d4beaae3a 100644 --- a/README.md +++ b/README.md @@ -3,8 +3,7 @@ `node-gyp` is a cross-platform command-line tool written in Node.js for compiling native addon modules for Node.js. It contains a fork of the [gyp](https://gyp.gsrc.io) project that was previously used by the Chromium -team and takes away the pain of dealing with the various differences in build -platforms. +team, extended to support the development of Node.js native addons. Note that `node-gyp` is _not_ used to build Node.js itself. @@ -14,23 +13,18 @@ etc.), regardless of what version of Node.js is actually installed on your syste ## Features - * Easy to use, consistent interface - * Same commands to build your module on every platform - * Supports multiple target versions of Node.js + * The same build commands work on any of the supported platforms + * Supports the targetting of different versions of Node.js ## Installation -You can install with `npm`: +You can install `node-gyp` using `npm`: ``` bash $ npm install -g node-gyp ``` -You will also need to install: - -NOTE: node-gyp is compatible with Python v2.7, v3.5, v3.6, or v3.7. If the -Python to use is not explicitly configured (see "Configuring Python Dependency" -below) it will attempt to find a compatible Python executable. +Depending on your operating system, you will need to install: ### On Unix @@ -66,14 +60,17 @@ Install tools and configuration manually: ### Configuring Python Dependency -If you have multiple Python versions installed, you can identify which Python -version `node-gyp` uses by setting the `--python` variable: +`node-gyp` requires that you have installed a compatible version of Python, one of: v2.7, v3.5, v3.6, +or v3.7. If you have multiple Python versions installed, you can identify which Python +version `node-gyp` should use in one of the following ways: + +1. by setting the `--python` command-line option, e.g.: ``` bash $ node-gyp --python /path/to/executable/python ``` -If `node-gyp` is called by way of `npm`, *and* you have multiple versions of +2. If `node-gyp` is called by way of `npm`, *and* you have multiple versions of Python installed, then you can set `npm`'s 'python' config key to the appropriate value: @@ -81,12 +78,12 @@ value: $ npm config set python /path/to/executable/python ``` -If the `PYTHON` environment variable is set to the path of a Python executable, -it will be used if it is a compatible Python. +3. If the `PYTHON` environment variable is set to the path of a Python executable, +then that version will be used, if it is a compatible version. -If the `NODE_GYP_FORCE_PYTHON` environment variable is set to the path of a +4. If the `NODE_GYP_FORCE_PYTHON` environment variable is set to the path of a Python executable, it will be used instead of any of the other configured or -builtin Python search paths. If its not a compatible Python, no further +builtin Python search paths. If it's not a compatible version, no further searching will be done. ## How to Use @@ -146,7 +143,9 @@ A barebones `gyp` file appropriate for building a Node.js addon could look like: } ``` -Some additional resources for addons and writing `gyp` files: +## Further reading + +Some additional resources for Node.js native addons and writing `gyp` configuration files: * ["Going Native" a nodeschool.io tutorial](http://nodeschool.io/#goingnative) * ["Hello World" node addon example](https://github.com/nodejs/node/tree/master/test/addons/hello-world) @@ -154,7 +153,6 @@ Some additional resources for addons and writing `gyp` files: * [gyp input format reference](https://gyp.gsrc.io/docs/InputFormatReference.md) * [*"binding.gyp" files out in the wild* wiki page](https://github.com/nodejs/node-gyp/wiki/%22binding.gyp%22-files-out-in-the-wild) - ## Commands `node-gyp` responds to the following commands: From b91718eefc5ef589a58a384ee1454dcbe6637462 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Wed, 16 Oct 2019 00:19:08 +0500 Subject: [PATCH 106/551] test: upgrade Linux Travis CI to Python 3.8 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ensure that we remain compatible with the newest version of Python. PR-URL: https://github.com/nodejs/node-gyp/pull/1923 Reviewed-By: Rod Vagg Reviewed-By: Sam Roberts Reviewed-By: João Reis Reviewed-By: Richard Lau --- .travis.yml | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/.travis.yml b/.travis.yml index 518543a3c4..172744f4cf 100644 --- a/.travis.yml +++ b/.travis.yml @@ -29,16 +29,16 @@ matrix: NODE_GYP_FORCE_PYTHON=/c/Python27/python.exe before_install: choco install python2 - - name: "Node.js 6 & Python 3.7 on Linux" - python: 3.7 + - name: "Node.js 6 & Python 3.8 on Linux" + python: 3.8 env: NODE_GYP_FORCE_PYTHON=python3 before_install: nvm install 6 - - name: "Node.js 8 & Python 3.7 on Linux" - python: 3.7 + - name: "Node.js 8 & Python 3.8 on Linux" + python: 3.8 env: NODE_GYP_FORCE_PYTHON=python3 before_install: nvm install 8 - - name: "Node.js 10 & Python 3.7 on Linux" - python: 3.7 + - name: "Node.js 10 & Python 3.8 on Linux" + python: 3.8 env: NODE_GYP_FORCE_PYTHON=python3 before_install: nvm install 10 @@ -54,6 +54,10 @@ matrix: python: 3.7 env: NODE_GYP_FORCE_PYTHON=python3 before_install: nvm install 12 + - name: "Node.js 12 & Python 3.8 on Linux" + python: 3.8 + env: NODE_GYP_FORCE_PYTHON=python3 + before_install: nvm install 12 - name: "Python 3.7 on macOS" os: osx From c60c22de58ffee1db0fbe68baba34f3719f3741d Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Tue, 15 Oct 2019 17:11:09 +1100 Subject: [PATCH 107/551] deps: update deps to roughly match current npm@6 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR-URL: https://github.com/nodejs/node-gyp/pull/1920 Reviewed-By: João Reis Reviewed-By: Richard Lau --- package.json | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/package.json b/package.json index 7014dadf56..92b46000ae 100644 --- a/package.json +++ b/package.json @@ -22,26 +22,26 @@ "bin": "./bin/node-gyp.js", "main": "./lib/node-gyp.js", "dependencies": { - "env-paths": "^1.0.0", - "glob": "^7.0.3", - "graceful-fs": "^4.1.2", - "mkdirp": "^0.5.0", - "nopt": "2 || 3", - "npmlog": "0 || 1 || 2 || 3 || 4", - "request": "^2.87.0", - "rimraf": "2", - "semver": "~5.3.0", + "env-paths": "^2.2.0", + "glob": "^7.1.4", + "graceful-fs": "^4.2.2", + "mkdirp": "^0.5.1", + "nopt": "^4.0.1", + "npmlog": "^4.1.2", + "request": "^2.88.0", + "rimraf": "^2.6.3", + "semver": "^5.7.1", "tar": "^4.4.12", - "which": "1" + "which": "^1.3.1" }, "engines": { "node": ">= 6.0.0" }, "devDependencies": { - "bindings": "~1.2.1", - "nan": "^2.0.0", - "require-inject": "~1.3.0", - "standard": "~14.3.1", + "bindings": "^1.5.0", + "nan": "^2.14.0", + "require-inject": "^1.4.4", + "standard": "^14.3.1", "tap": "~12.7.0" }, "scripts": { From f0693413d953aee9830751d0b3a63787fc6f78d8 Mon Sep 17 00:00:00 2001 From: Richard Lau Date: Wed, 23 Oct 2019 18:33:49 -0400 Subject: [PATCH 108/551] src,win: allow 403 errors for arm64 node.lib MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The servers hosting the header packages for Electron return 403 instead of 404 for the constructed URL for arm64 node.lib for older releases that do not support arm64. PR-URL: https://github.com/nodejs/node-gyp/pull/1934 Reviewed-By: Rod Vagg Reviewed-By: João Reis --- lib/install.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/install.js b/lib/install.js index 06e82871f7..f68cd7fd6d 100644 --- a/lib/install.js +++ b/lib/install.js @@ -323,7 +323,7 @@ function install (fs, gyp, argv, callback) { req.on('error', done) req.on('response', function (res) { - if (res.statusCode === 404) { + if (res.statusCode === 403 || res.statusCode === 404) { if (arch === 'arm64') { // Arm64 is a newer platform on Windows and not all node distributions provide it. log.verbose(`${name} was not found in ${libUrl}`) From bb2eb72a3f3bef0d8f8ed5fce402ccd2daaba5a3 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Sun, 27 Oct 2019 10:32:20 +0100 Subject: [PATCH 109/551] gyp: finish decode stdout on Python 3 PR-URL: https://github.com/nodejs/node-gyp/pull/1937 Reviewed-By: Jiawen Geng Reviewed-By: Richard Lau Reviewed-By: Sam Roberts --- gyp/pylib/gyp/MSVSVersion.py | 6 ++++++ gyp/pylib/gyp/generator/eclipse.py | 6 ++++++ gyp/pylib/gyp/generator/msvs.py | 4 ++++ gyp/pylib/gyp/input.py | 4 ++++ gyp/pylib/gyp/mac_tool.py | 4 ++++ gyp/pylib/gyp/msvs_emulation.py | 10 +++++++++- gyp/pylib/gyp/win_tool.py | 11 +++++++++++ 7 files changed, 44 insertions(+), 1 deletion(-) diff --git a/gyp/pylib/gyp/MSVSVersion.py b/gyp/pylib/gyp/MSVSVersion.py index 13d9777f0e..c7cf68d3a1 100644 --- a/gyp/pylib/gyp/MSVSVersion.py +++ b/gyp/pylib/gyp/MSVSVersion.py @@ -12,6 +12,8 @@ import gyp import glob +PY3 = bytes != str + class VisualStudioVersion(object): """Information regarding a version of Visual Studio.""" @@ -132,6 +134,8 @@ def _RegistryQueryBase(sysdir, key, value): # Obtain the stdout from reg.exe, reading to the end so p.returncode is valid # Note that the error text may be in [1] in some cases text = p.communicate()[0] + if PY3: + text = text.decode('utf-8') # Check return code from reg.exe; officially 0==success and 1==error if p.returncode: return None @@ -334,6 +338,8 @@ def _ConvertToCygpath(path): if sys.platform == 'cygwin': p = subprocess.Popen(['cygpath', path], stdout=subprocess.PIPE) path = p.communicate()[0].strip() + if PY3: + path = path.decode('utf-8') return path diff --git a/gyp/pylib/gyp/generator/eclipse.py b/gyp/pylib/gyp/generator/eclipse.py index 6b49ad6760..91f187d685 100644 --- a/gyp/pylib/gyp/generator/eclipse.py +++ b/gyp/pylib/gyp/generator/eclipse.py @@ -26,6 +26,8 @@ import shlex import xml.etree.cElementTree as ET +PY3 = bytes != str + generator_wants_static_library_dependencies_adjusted = False generator_default_variables = { @@ -97,6 +99,8 @@ def GetAllIncludeDirectories(target_list, target_dicts, proc = subprocess.Popen(args=command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) output = proc.communicate()[1] + if PY3: + output = output.decode('utf-8') # Extract the list of include dirs from the output, which has this format: # ... # #include "..." search starts here: @@ -234,6 +238,8 @@ def GetAllDefines(target_list, target_dicts, data, config_name, params, cpp_proc = subprocess.Popen(args=command, cwd='.', stdin=subprocess.PIPE, stdout=subprocess.PIPE) cpp_output = cpp_proc.communicate()[0] + if PY3: + cpp_output = cpp_output.decode('utf-8') cpp_lines = cpp_output.split('\n') for cpp_line in cpp_lines: if not cpp_line.strip(): diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index b3b4bc1053..8f3e487c48 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -25,6 +25,8 @@ from gyp.common import GypError from gyp.common import OrderedSet +PY3 = bytes != str + # TODO: Remove once bots are on 2.7, http://crbug.com/241769 def _import_OrderedDict(): import collections @@ -124,6 +126,8 @@ def _GetDomainAndUserName(): call = subprocess.Popen(['net', 'config', 'Workstation'], stdout=subprocess.PIPE) config = call.communicate()[0] + if PY3: + config = config.decode('utf-8') username_re = re.compile(r'^User name\s+(\S+)', re.MULTILINE) username_match = username_re.search(config) if username_match: diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index aba81cf654..6a0a4db964 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -22,6 +22,7 @@ from gyp.common import GypError from gyp.common import OrderedSet +PY3 = bytes != str # A list of types that are treated as linkable. linkable_types = [ @@ -909,6 +910,9 @@ def ExpandVariables(input, phase, variables, build_file): (e, contents, build_file)) p_stdout, p_stderr = p.communicate('') + if PY3: + p_stdout = p_stdout.decode('utf-8') + p_stderr = p_stderr.decode('utf-8') if p.wait() != 0 or p_stderr: sys.stderr.write(p_stderr) diff --git a/gyp/pylib/gyp/mac_tool.py b/gyp/pylib/gyp/mac_tool.py index b8b7344eff..222befb982 100755 --- a/gyp/pylib/gyp/mac_tool.py +++ b/gyp/pylib/gyp/mac_tool.py @@ -23,6 +23,8 @@ import sys import tempfile +PY3 = bytes != str + def main(args): executor = MacTool() @@ -243,6 +245,8 @@ def ExecFilterLibtool(self, *cmd_list): env['ZERO_AR_DATE'] = '1' libtoolout = subprocess.Popen(cmd_list, stderr=subprocess.PIPE, env=env) _, err = libtoolout.communicate() + if PY3: + err = err.decode('utf-8') for line in err.splitlines(): if not libtool_re.match(line) and not libtool_re5.match(line): print(line, file=sys.stderr) diff --git a/gyp/pylib/gyp/msvs_emulation.py b/gyp/pylib/gyp/msvs_emulation.py index 9d50bad1f5..9f1f547b90 100644 --- a/gyp/pylib/gyp/msvs_emulation.py +++ b/gyp/pylib/gyp/msvs_emulation.py @@ -16,6 +16,7 @@ import gyp.MSVSUtil import gyp.MSVSVersion +PY3 = bytes != str windows_quoter_regex = re.compile(r'(\\*)"') @@ -126,7 +127,10 @@ def _FindDirectXInstallation(): # Setup params to pass to and attempt to launch reg.exe. cmd = ['reg.exe', 'query', r'HKLM\Software\Microsoft\DirectX', '/s'] p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) - for line in p.communicate()[0].splitlines(): + stdout = p.communicate()[0] + if PY3: + stdout = stdout.decode('utf-8') + for line in stdout.splitlines(): if 'InstallPath' in line: dxsdk_dir = line.split(' ')[3] + "\\" @@ -1038,6 +1042,8 @@ def GenerateEnvironmentFiles(toplevel_build_dir, generator_flags, popen = subprocess.Popen( args, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) variables, _ = popen.communicate() + if PY3: + variables = variables.decode('utf-8') env = _ExtractImportantEnvironment(variables) # Inject system includes from gyp files into INCLUDE. @@ -1057,6 +1063,8 @@ def GenerateEnvironmentFiles(toplevel_build_dir, generator_flags, 'for', '%i', 'in', '(cl.exe)', 'do', '@echo', 'LOC:%~$PATH:i')) popen = subprocess.Popen(args, shell=True, stdout=subprocess.PIPE) output, _ = popen.communicate() + if PY3: + output = output.decode('utf-8') cl_paths[arch] = _ExtractCLPath(output) return cl_paths diff --git a/gyp/pylib/gyp/win_tool.py b/gyp/pylib/gyp/win_tool.py index bca0b9e346..0a6daf2039 100755 --- a/gyp/pylib/gyp/win_tool.py +++ b/gyp/pylib/gyp/win_tool.py @@ -20,6 +20,7 @@ import sys BASE_DIR = os.path.dirname(os.path.abspath(__file__)) +PY3 = bytes != str # A regex matching an argument corresponding to the output filename passed to # link.exe. @@ -124,6 +125,8 @@ def ExecLinkWrapper(self, arch, use_separate_mspdbsrv, *args): stdout=subprocess.PIPE, stderr=subprocess.STDOUT) out, _ = link.communicate() + if PY3: + out = out.decode('utf-8') for line in out.splitlines(): if (not line.startswith(' Creating library ') and not line.startswith('Generating code') and @@ -215,6 +218,8 @@ def ExecManifestWrapper(self, arch, *args): popen = subprocess.Popen(args, shell=True, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) out, _ = popen.communicate() + if PY3: + out = out.decode('utf-8') for line in out.splitlines(): if line and 'manifest authoring warning 81010002' not in line: print(line) @@ -247,6 +252,8 @@ def ExecMidlWrapper(self, arch, outdir, tlb, h, dlldata, iid, proxy, idl, popen = subprocess.Popen(args, shell=True, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) out, _ = popen.communicate() + if PY3: + out = out.decode('utf-8') # Filter junk out of stdout, and write filtered versions. Output we want # to filter is pairs of lines that look like this: # Processing C:\Program Files (x86)\Microsoft SDKs\...\include\objidl.idl @@ -266,6 +273,8 @@ def ExecAsmWrapper(self, arch, *args): popen = subprocess.Popen(args, shell=True, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) out, _ = popen.communicate() + if PY3: + out = out.decode('utf-8') for line in out.splitlines(): if (not line.startswith('Copyright (C) Microsoft Corporation') and not line.startswith('Microsoft (R) Macro Assembler') and @@ -281,6 +290,8 @@ def ExecRcWrapper(self, arch, *args): popen = subprocess.Popen(args, shell=True, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) out, _ = popen.communicate() + if PY3: + out = out.decode('utf-8') for line in out.splitlines(): if (not line.startswith('Microsoft (R) Windows (R) Resource Compiler') and not line.startswith('Copyright (C) Microsoft Corporation') and From 9c0f3404f03aa1c3413ce896f6365f8889fb74a2 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Sun, 27 Oct 2019 10:36:53 +0100 Subject: [PATCH 110/551] gyp: fix TypeError in XcodeVersion() PR-URL: https://github.com/nodejs/node-gyp/pull/1939 Reviewed-By: Jiawen Geng Reviewed-By: Richard Lau Reviewed-By: Sam Roberts --- gyp/pylib/gyp/xcode_emulation.py | 41 ++++++++++++++++---------------- 1 file changed, 20 insertions(+), 21 deletions(-) diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py index 13964beec6..6017164990 100644 --- a/gyp/pylib/gyp/xcode_emulation.py +++ b/gyp/pylib/gyp/xcode_emulation.py @@ -855,7 +855,8 @@ def GetLdflags(self, configname, product_dir, gyp_to_build_path, arch=None): # These flags reflect the compilation options used by xcode to compile # extensions. ldflags.append('-lpkstart') - if XcodeVersion() < '0900': + xcode_version, _ = XcodeVersion() + if xcode_version < '0900': ldflags.append(sdk_root + '/System/Library/PrivateFrameworks/PlugInKit.framework/PlugInKit') ldflags.append('-fapplication-extension') @@ -1088,15 +1089,15 @@ def GetExtraPlistItems(self, configname=None): cache = {} cache['BuildMachineOSBuild'] = self._BuildMachineOSBuild() - xcode, xcode_build = XcodeVersion() - cache['DTXcode'] = xcode + xcode_version, xcode_build = XcodeVersion() + cache['DTXcode'] = xcode_version cache['DTXcodeBuild'] = xcode_build sdk_root = self._SdkRoot(configname) if not sdk_root: sdk_root = self._DefaultSdkRoot() cache['DTSDKName'] = sdk_root - if xcode >= '0430': + if xcode_version >= '0430': cache['DTSDKBuild'] = self._GetSdkVersionInfoItem( sdk_root, 'ProductBuildVersion') else: @@ -1126,7 +1127,7 @@ def _DefaultSdkRoot(self): project, then the environment variable was empty. Starting with this version, Xcode uses the name of the newest SDK installed. """ - xcode_version, xcode_build = XcodeVersion() + xcode_version, _ = XcodeVersion() if xcode_version < '0500': return '' default_sdk_path = self._XcodeSdkPath('') @@ -1263,10 +1264,12 @@ def XcodeVersion(): # Xcode 3.2.6 # Component versions: DevToolsCore-1809.0; DevToolsSupport-1806.0 # BuildVersion: 10M2518 - # Convert that to '0463', '4H1503'. + # Convert that to ('0463', '4H1503') or ('0326', '10M2518'). global XCODE_VERSION_CACHE if XCODE_VERSION_CACHE: return XCODE_VERSION_CACHE + version = "" + build = "" try: version_list = GetStdoutQuiet(['xcodebuild', '-version']).splitlines() # In some circumstances xcodebuild exits 0 but doesn't return @@ -1276,21 +1279,16 @@ def XcodeVersion(): # checking that version. if len(version_list) < 2: raise GypError("xcodebuild returned unexpected results") - except GypError: - version = CLTVersion() - if version: - version = ".".join(version.split(".")[:3]) - else: + version = version_list[0].split()[-1] # Last word on first line + build = version_list[-1].split()[-1] # Last word on last line + except GypError: # Xcode not installed so look for XCode Command Line Tools + version = CLTVersion() # macOS Catalina returns 11.0.0.0.1.1567737322 + if not version: raise GypError("No Xcode or CLT version detected!") - # The CLT has no build information, so we return an empty string. - version_list = [version, ''] - version = version_list[0] - build = version_list[-1] - # Be careful to convert "4.2" to "0420" and "10.0" to "1000": - version = format(''.join((version.split()[-1].split('.') + ['0', '0'])[:3]), - '>04s') - if build: - build = build.split()[-1] + # Be careful to convert "4.2.3" to "0423" and "11.0.0" to "1100": + version = version.split(".")[:3] # Just major, minor, micro + version[0] = version[0].zfill(2) # Add a leading zero if major is one digit + version = ("".join(version) + "00")[:4] # Limit to exactly four characters XCODE_VERSION_CACHE = (version, build) return XCODE_VERSION_CACHE @@ -1524,7 +1522,8 @@ def _GetXcodeEnv(xcode_settings, built_products_dir, srcroot, configuration, install_name_base = xcode_settings.GetInstallNameBase() if install_name_base: env['DYLIB_INSTALL_NAME_BASE'] = install_name_base - if XcodeVersion() >= '0500' and not env.get('SDKROOT'): + xcode_version, _ = XcodeVersion() + if xcode_version >= '0500' and not env.get('SDKROOT'): sdk_root = xcode_settings._SdkRoot(configuration) if not sdk_root: sdk_root = xcode_settings._XcodeSdkPath('') From d8e09a1b6aef83431b2b0b6cb93b548b75978d06 Mon Sep 17 00:00:00 2001 From: gengjiawen Date: Mon, 28 Oct 2019 20:39:01 +0800 Subject: [PATCH 111/551] gyp: make cmake python3 compatible PR-URL: https://github.com/nodejs/node-gyp/pull/1944 Reviewed-By: Christian Clauss Reviewed-By: Richard Lau --- gyp/pylib/gyp/generator/cmake.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/gyp/pylib/gyp/generator/cmake.py b/gyp/pylib/gyp/generator/cmake.py index 996b6f25fd..5601a6657e 100644 --- a/gyp/pylib/gyp/generator/cmake.py +++ b/gyp/pylib/gyp/generator/cmake.py @@ -239,7 +239,10 @@ def StringToCMakeTargetName(a): Invalid for make: ':' Invalid for unknown reasons but cause failures: '.' """ - return a.translate(string.maketrans(' /():."', '_______')) + try: + return a.translate(str.maketrans(' /():."', '_______')) + except AttributeError: + return a.translate(string.maketrans(' /():."', '_______')) def WriteActions(target_name, actions, extra_sources, extra_deps, From c0282daa48cf1803c4283998304c563390866c8d Mon Sep 17 00:00:00 2001 From: cclauss Date: Sun, 27 Oct 2019 17:42:31 +0100 Subject: [PATCH 112/551] gyp: iteritems() -> items() in compile_commands_json.py PR-URL: https://github.com/nodejs/node-gyp/pull/1947 Reviewed-By: Jiawen Geng Reviewed-By: Richard Lau --- gyp/pylib/gyp/generator/compile_commands_json.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/gyp/pylib/gyp/generator/compile_commands_json.py b/gyp/pylib/gyp/generator/compile_commands_json.py index 575db63c4e..1b8490451f 100644 --- a/gyp/pylib/gyp/generator/compile_commands_json.py +++ b/gyp/pylib/gyp/generator/compile_commands_json.py @@ -43,7 +43,7 @@ def CalculateVariables(default_variables, params): def AddCommandsForTarget(cwd, target, params, per_config_commands): output_dir = params['generator_flags']['output_dir'] - for configuration_name, configuration in target['configurations'].iteritems(): + for configuration_name, configuration in target['configurations'].items(): builddir_name = os.path.join(output_dir, configuration_name) if IsMac(params): @@ -92,7 +92,7 @@ def resolve(filename): def GenerateOutput(target_list, target_dicts, data, params): per_config_commands = {} - for qualified_target, target in target_dicts.iteritems(): + for qualified_target, target in target_dicts.items(): build_file, target_name, toolset = ( gyp.common.ParseQualifiedTarget(qualified_target)) if IsMac(params): @@ -102,7 +102,7 @@ def GenerateOutput(target_list, target_dicts, data, params): AddCommandsForTarget(cwd, target, params, per_config_commands) output_dir = params['generator_flags']['output_dir'] - for configuration_name, commands in per_config_commands.iteritems(): + for configuration_name, commands in per_config_commands.items(): filename = os.path.join(output_dir, configuration_name, 'compile_commands.json') From 1b11be63cc85edf779b4dc12612f868752996329 Mon Sep 17 00:00:00 2001 From: Wilfried Goesgens Date: Wed, 16 Oct 2019 16:29:38 +0200 Subject: [PATCH 113/551] gyp: python3 fixes: utf8 decode, use of 'None' in eval PR-URL: https://github.com/nodejs/node-gyp/pull/1925 Reviewed-By: Christian Clauss Reviewed-By: Richard Lau --- gyp/pylib/gyp/generator/make.py | 2 +- gyp/pylib/gyp/input.py | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index bdf7134537..1960536794 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -1776,7 +1776,7 @@ def WriteMakeRule(self, outputs, inputs, actions=None, comment=None, # - The multi-output rule will have an do-nothing recipe. # Hash the target name to avoid generating overlong filenames. - cmddigest = hashlib.sha1(command if command else self.target).hexdigest() + cmddigest = hashlib.sha1((command or self.target).encode('utf-8')).hexdigest() intermediate = "%s.intermediate" % cmddigest self.WriteLn('%s: %s' % (' '.join(outputs), intermediate)) self.WriteLn('\t%s' % '@:') diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index 6a0a4db964..2973c078fc 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -240,7 +240,7 @@ def LoadOneBuildFile(build_file_path, data, aux_data, includes, if check: build_file_data = CheckedEval(build_file_contents) else: - build_file_data = eval(build_file_contents, {'__builtins__': None}, + build_file_data = eval(build_file_contents, {'__builtins__': {}}, None) except SyntaxError as e: e.filename = build_file_path @@ -1094,7 +1094,7 @@ def EvalSingleCondition( else: ast_code = compile(cond_expr_expanded, '', 'eval') cached_conditions_asts[cond_expr_expanded] = ast_code - if eval(ast_code, {'__builtins__': None}, variables): + if eval(ast_code, {'__builtins__': {}}, variables): return true_dict return false_dict except SyntaxError as e: From 8ec2e681d536db65e394e4d3cad0738fe4003ffb Mon Sep 17 00:00:00 2001 From: cclauss Date: Fri, 25 Oct 2019 12:50:43 +0200 Subject: [PATCH 114/551] doc: add macOS_Catalina.md document PR-URL: https://github.com/nodejs/node-gyp/pull/1940 Reviewed-By: Rod Vagg Reviewed-By: Jiawen Geng --- README.md | 1 + macOS_Catalina.md | 69 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+) create mode 100644 macOS_Catalina.md diff --git a/README.md b/README.md index 5d4beaae3a..9b1c171305 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,7 @@ Depending on your operating system, you will need to install: * `Python v2.7, v3.5, v3.6, or v3.7` * [Xcode](https://developer.apple.com/xcode/download/) * You also need to install the `XCode Command Line Tools` by running `xcode-select --install`. Alternatively, if you already have the full Xcode installed, you can find them under the menu `Xcode -> Open Developer Tool -> More Developer Tools...`. This step will install `clang`, `clang++`, and `make`. + * If your Mac has been _upgraded_ to macOS Catalina (10.15), please read [macOS_Catalina.md](macOS_Catalina.md). ### On Windows diff --git a/macOS_Catalina.md b/macOS_Catalina.md new file mode 100644 index 0000000000..848c2c5c84 --- /dev/null +++ b/macOS_Catalina.md @@ -0,0 +1,69 @@ +# Installation notes for macOS Catalina (v10.15) + +_This document specifically refers to upgrades from previous versions of macOS to Catalina (10.15). It should be removed from the source repository when Catalina ceases to be the latest macOS version or updated to deal with challenges involved in upgrades to the next version of macOS._ + +Lessons learned from: +* https://github.com/nodejs/node-gyp/issues/1779 +* https://github.com/nodejs/node-gyp/issues/1861 +* https://github.com/nodejs/node-gyp/issues/1927 and elsewhere + +Installing `node-gyp` on macOS can be found at https://github.com/nodejs/node-gyp#on-macos + +However, upgrading to macOS Catalina changes some settings that may cause normal `node-gyp` installations to fail. + +### Is my Mac running macOS Catalina? +Let's make first make sure that your Mac is currently running Catalina: +% `sw_vers` + ProductName: Mac OS X + ProductVersion: 10.15 + BuildVersion: 19A602 + +If `ProductVersion` is less then `10.15` then this document is not really for you. + +### The acid test +Next, lets see if `Xcode Command Line Tools` are installed: +1. `/usr/sbin/pkgutil --packages | grep CL` + * If nothing is listed, then [skip to the next section](#Two-roads). + * If `com.apple.pkg.CLTools_Executables` is listed then try: +2. `/usr/sbin/pkgutil --pkg-info com.apple.pkg.CLTools_Executables` + * If `version: 11.0.0` or later is listed then _you are done_! Your Mac should be ready to install `node-gyp`. Doing `clang -v` should show `Apple clang version 11.0.0` or later. + +As you go through the remainder of this document, at anytime you can try these `acid test` commands. If they pass then your Mac should be ready to install `node-gyp`. + +### Two roads +There are two main ways to install `node-gyp` on macOS: +1. With the full Xcode (~7.6 GB download) from the `App Store` app. +2. With the _much_ smaller Xcode Command Line Tools via `xcode-select --install` + +### Installing `node-gyp` using the full Xcode +1. `xcodebuild -version` should show `Xcode 11.1` or later. + * If not, then install/upgrade Xcode from the App Store app. +2. Open the Xcode app and allow it to do an essential install of the most recent compiler tools. +3. Once all installations are _complete_, quit out of Xcode. +4. `sudo xcodebuild -license accept` # If you agree with the licensing terms. +5. `softwareupdate -l` # No listing is a good sign. + * If Xcode or Tools upgrades are listed, use "Software Upgrade" to install them. +6. `xcode-select -version` # Should return `xcode-select version 2370` or later. +7. `xcode-select -print-path` # Should return `/Applications/Xcode.app/Contents/Developer` +8. Try the [_acid test_ steps above](#The-acid-test) to see if your Mac is ready. +9. If the _acid test_ does _not_ pass then... +10. `sudo xcode-select --reset` # Enter root password. No output is normal. +11. Repeat step 7 above. Is the path different this time? Repeat the _acid test_. + +### Installing `node-gyp` using the Xcode Command Line Tools +1. If the _acid test_ has not succeeded, then try `xcode-select --install` +2. Wait until the install process is _complete_. +3. `softwareupdate -l` # No listing is a good sign. + * If Xcode or Tools upgrades are listed, use "Software Update" to install them. +4. `xcode-select -version` # Should return `xcode-select version 2370` or later. +5. `xcode-select -print-path` # Should return `/Library/Developer/CommandLineTools` +6. Try the [_acid test_ steps above](#The-acid-test) to see if your Mac is ready. +7. If the _acid test_ does _not_ pass then... +8. `sudo xcode-select --reset` # Enter root password. No output is normal. +9. Repeat step 5 above. Is the path different this time? Repeat the _acid test_. + +### I did all that and the acid test still does not pass :-( +1. `sudo rm -rf $(xcode-select -print-path)` # Enter root password. No output is normal. +2. `xcode-select --install` +3. If the [_acid test_](#The-acid-test) still does _not_ pass then... +4. Add a comment to https://github.com/nodejs/node-gyp/issues/1927 so we can improve. From 68319a2c344c822d48bd6d5dd32f82dd41384e19 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Thu, 24 Oct 2019 14:59:11 +1100 Subject: [PATCH 115/551] v6.0.1: bump version and update changelog MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR-URL: https://github.com/nodejs/node-gyp/pull/1935 Reviewed-By: João Reis Reviewed-By: Richard Lau Reviewed-By: Jiawen Geng Reviewed-By: Christian Clauss --- CHANGELOG.md | 18 ++++++++++++++++++ package.json | 2 +- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6f7809c625..d173ab7ec6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,21 @@ +v6.0.1 2019-11-01 +================= + +* [[`8ec2e681d5`](https://github.com/nodejs/node-gyp/commit/8ec2e681d5)] - **doc**: add macOS\_Catalina.md document (cclauss) [#1940](https://github.com/nodejs/node-gyp/pull/1940) +* [[`1b11be63cc`](https://github.com/nodejs/node-gyp/commit/1b11be63cc)] - **gyp**: python3 fixes: utf8 decode, use of 'None' in eval (Wilfried Goesgens) [#1925](https://github.com/nodejs/node-gyp/pull/1925) +* [[`c0282daa48`](https://github.com/nodejs/node-gyp/commit/c0282daa48)] - **gyp**: iteritems() -\> items() in compile\_commands\_json.py (cclauss) [#1947](https://github.com/nodejs/node-gyp/pull/1947) +* [[`d8e09a1b6a`](https://github.com/nodejs/node-gyp/commit/d8e09a1b6a)] - **gyp**: make cmake python3 compatible (gengjiawen) [#1944](https://github.com/nodejs/node-gyp/pull/1944) +* [[`9c0f3404f0`](https://github.com/nodejs/node-gyp/commit/9c0f3404f0)] - **gyp**: fix TypeError in XcodeVersion() (Christian Clauss) [#1939](https://github.com/nodejs/node-gyp/pull/1939) +* [[`bb2eb72a3f`](https://github.com/nodejs/node-gyp/commit/bb2eb72a3f)] - **gyp**: finish decode stdout on Python 3 (Christian Clauss) [#1937](https://github.com/nodejs/node-gyp/pull/1937) +* [[`f0693413d9`](https://github.com/nodejs/node-gyp/commit/f0693413d9)] - **src,win**: allow 403 errors for arm64 node.lib (Richard Lau) [#1934](https://github.com/nodejs/node-gyp/pull/1934) +* [[`c60c22de58`](https://github.com/nodejs/node-gyp/commit/c60c22de58)] - **deps**: update deps to roughly match current npm@6 (Rod Vagg) [#1920](https://github.com/nodejs/node-gyp/pull/1920) +* [[`b91718eefc`](https://github.com/nodejs/node-gyp/commit/b91718eefc)] - **test**: upgrade Linux Travis CI to Python 3.8 (Christian Clauss) [#1923](https://github.com/nodejs/node-gyp/pull/1923) +* [[`3538a317b6`](https://github.com/nodejs/node-gyp/commit/3538a317b6)] - **doc**: adjustments to the README.md for new users (Dan Pike) [#1919](https://github.com/nodejs/node-gyp/pull/1919) +* [[`4fff8458c0`](https://github.com/nodejs/node-gyp/commit/4fff8458c0)] - **travis**: ignore failed `brew upgrade npm`, update xcode (Christian Clauss) [#1932](https://github.com/nodejs/node-gyp/pull/1932) +* [[`60e4488f08`](https://github.com/nodejs/node-gyp/commit/60e4488f08)] - **build**: avoid bare exceptions in xcode\_emulation.py (Christian Clauss) [#1932](https://github.com/nodejs/node-gyp/pull/1932) +* [[`032db2a2d0`](https://github.com/nodejs/node-gyp/commit/032db2a2d0)] - **lib,install**: always download SHA sums on Windows (Sam Hughes) [#1926](https://github.com/nodejs/node-gyp/pull/1926) +* [[`5a83630c33`](https://github.com/nodejs/node-gyp/commit/5a83630c33)] - **travis**: add Windows + Python 3.8 to the mix (Rod Vagg) [#1921](https://github.com/nodejs/node-gyp/pull/1921) + v6.0.0 2019-10-04 ================= diff --git a/package.json b/package.json index 92b46000ae..243738506d 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "6.0.0", + "version": "6.0.1", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From c506a6a1504bc7c2da8a4e84bf785c9302aa61f2 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Sun, 23 Jun 2019 15:31:52 +1000 Subject: [PATCH 116/551] test: configure proper devDir for invoking configure() test/test-configure-python.js downloads a fresh set of headers to the package directory each time. By setting to the default global cache dir we get to re-use cached headers and skip the download step. PR-URL: https://github.com/nodejs/node-gyp/pull/1796 Reviewed-By: Richard Lau --- test/common.js | 3 +++ test/test-configure-python.js | 3 +++ 2 files changed, 6 insertions(+) create mode 100644 test/common.js diff --git a/test/common.js b/test/common.js new file mode 100644 index 0000000000..b714ee2902 --- /dev/null +++ b/test/common.js @@ -0,0 +1,3 @@ +const envPaths = require('env-paths') + +module.exports.devDir = () => envPaths('node-gyp', { suffix: '' }).cache diff --git a/test/test-configure-python.js b/test/test-configure-python.js index 518bf036d0..f28234134f 100644 --- a/test/test-configure-python.js +++ b/test/test-configure-python.js @@ -30,6 +30,7 @@ test('configure PYTHONPATH with no existing env', function (t) { t.equal(process.env.PYTHONPATH, EXPECTED_PYPATH) return SPAWN_RESULT } + prog.devDir = devDir configure(prog, [], t.fail) }) @@ -49,6 +50,7 @@ test('configure PYTHONPATH with existing env of one dir', function (t) { return SPAWN_RESULT } + prog.devDir = devDir configure(prog, [], t.fail) }) @@ -70,5 +72,6 @@ test('configure PYTHONPATH with existing env of multiple dirs', function (t) { return SPAWN_RESULT } + prog.devDir = devDir configure(prog, [], t.fail) }) From 0670e5189d00b5db3c3058220367a41d407b218a Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Wed, 30 Oct 2019 22:38:49 +1100 Subject: [PATCH 117/551] test: add header download test PR-URL: https://github.com/nodejs/node-gyp/pull/1796 Reviewed-By: Richard Lau --- test/test-configure-python.js | 1 + test/test-download.js | 70 ++++++++++++++++++++++++++++++++++- 2 files changed, 70 insertions(+), 1 deletion(-) diff --git a/test/test-configure-python.js b/test/test-configure-python.js index f28234134f..d08f9e5ed3 100644 --- a/test/test-configure-python.js +++ b/test/test-configure-python.js @@ -2,6 +2,7 @@ const test = require('tap').test const path = require('path') +const devDir = require('./common').devDir() const gyp = require('../lib/node-gyp') const requireInject = require('require-inject') const configure = requireInject('../lib/configure', { diff --git a/test/test-download.js b/test/test-download.js index dbffb20246..738a43f276 100644 --- a/test/test-download.js +++ b/test/test-download.js @@ -6,8 +6,13 @@ const path = require('path') const http = require('http') const https = require('https') const install = require('../lib/install') +const semver = require('semver') +const devDir = require('./common').devDir() +const rimraf = require('rimraf') +const gyp = require('../lib/node-gyp') +const log = require('npmlog') -require('npmlog').level = 'warn' +log.level = 'warn' test('download over http', function (t) { t.plan(2) @@ -103,3 +108,66 @@ test('check certificate splitting', function (t) { t.strictEqual(cas.length, 2) t.notStrictEqual(cas[0], cas[1]) }) + +// only run this test if we are running a version of Node with predictable version path behavior + +test('download headers (actual)', function (t) { + if (process.env.FAST_TEST || + process.release.name !== 'node' || + semver.prerelease(process.version) !== null || + semver.satisfies(process.version, '<10')) { + return t.skip('Skipping acutal download of headers due to test environment configuration') + } + + t.plan(17) + + const expectedDir = path.join(devDir, process.version.replace(/^v/, '')) + rimraf(expectedDir, (err) => { + t.ifError(err) + + const prog = gyp() + prog.parseArgv([]) + prog.devDir = devDir + log.level = 'warn' + install(prog, [], (err) => { + t.ifError(err) + + fs.readFile(path.join(expectedDir, 'installVersion'), 'utf8', (err, data) => { + t.ifError(err) + t.strictEqual(data, '9\n', 'correct installVersion') + }) + + fs.readdir(path.join(expectedDir, 'include/node'), (err, list) => { + t.ifError(err) + + t.ok(list.includes('common.gypi')) + t.ok(list.includes('config.gypi')) + t.ok(list.includes('node.h')) + t.ok(list.includes('node_version.h')) + t.ok(list.includes('openssl')) + t.ok(list.includes('uv')) + t.ok(list.includes('uv.h')) + t.ok(list.includes('v8-platform.h')) + t.ok(list.includes('v8.h')) + t.ok(list.includes('zlib.h')) + }) + + fs.readFile(path.join(expectedDir, 'include/node/node_version.h'), 'utf8', (err, contents) => { + t.ifError(err) + + const lines = contents.split('\n') + + // extract the 3 version parts from the defines to build a valid version string and + // and check them against our current env version + const version = ['major', 'minor', 'patch'].reduce((version, type) => { + const re = new RegExp(`^#define\\sNODE_${type.toUpperCase()}_VERSION`) + const line = lines.find((l) => re.test(l)) + const i = line ? parseInt(line.replace(/^[^0-9]+([0-9]+).*$/, '$1'), 10) : 'ERROR' + return `${version}${type !== 'major' ? '.' : 'v'}${i}` + }, '') + + t.strictEqual(version, process.version) + }) + }) + }) +}) From 20aa0b44f7114814747edb9c2b9690ca19681be0 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Sun, 10 Nov 2019 11:18:54 +0100 Subject: [PATCH 118/551] doc: macOS Catalina add two commands These steps have proven effective for some users. PR-URL: https://github.com/nodejs/node-gyp/pull/1962 Reviewed-By: Rod Vagg --- macOS_Catalina.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/macOS_Catalina.md b/macOS_Catalina.md index 848c2c5c84..5f02face57 100644 --- a/macOS_Catalina.md +++ b/macOS_Catalina.md @@ -38,7 +38,9 @@ There are two main ways to install `node-gyp` on macOS: ### Installing `node-gyp` using the full Xcode 1. `xcodebuild -version` should show `Xcode 11.1` or later. * If not, then install/upgrade Xcode from the App Store app. -2. Open the Xcode app and allow it to do an essential install of the most recent compiler tools. +2. Open the Xcode app and... + * Under __Preferences > Locations__ select the tools if their location is empty. + * Allow Xcode app to do an essential install of the most recent compiler tools. 3. Once all installations are _complete_, quit out of Xcode. 4. `sudo xcodebuild -license accept` # If you agree with the licensing terms. 5. `softwareupdate -l` # No listing is a good sign. @@ -66,4 +68,7 @@ There are two main ways to install `node-gyp` on macOS: 1. `sudo rm -rf $(xcode-select -print-path)` # Enter root password. No output is normal. 2. `xcode-select --install` 3. If the [_acid test_](#The-acid-test) still does _not_ pass then... -4. Add a comment to https://github.com/nodejs/node-gyp/issues/1927 so we can improve. +4. `npm explore npm -g -- npm install node-gyp@latest` +5. `npm explore npm -g -- npm explore npm-lifecycle -- npm install node-gyp@latest` +6. If the _acid test_ still does _not_ pass then... +7. Add a comment to https://github.com/nodejs/node-gyp/issues/1927 so we can improve. From 04da736d38f94246adce4c4e4298a070637d2597 Mon Sep 17 00:00:00 2001 From: cclauss Date: Fri, 8 Nov 2019 22:56:52 +0100 Subject: [PATCH 119/551] test: fix Python unittests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR-URL: https://github.com/nodejs/node-gyp/pull/1961 Reviewed-By: Ben Noordhuis Reviewed-By: João Reis --- .travis.yml | 4 +- gyp/gyp_main.py | 2 +- gyp/pylib/gyp/MSVSSettings_test.py | 2 + gyp/pylib/gyp/common.py | 8 +- gyp/pylib/gyp/generator/msvs.py | 14 +- gyp/pylib/gyp/generator/ninja_test.py | 1 - gyp/pylib/gyp/ordered_dict.py | 289 -------------------------- 7 files changed, 14 insertions(+), 306 deletions(-) delete mode 100644 gyp/pylib/gyp/ordered_dict.py diff --git a/.travis.yml b/.travis.yml index 172744f4cf..4c67788aaa 100644 --- a/.travis.yml +++ b/.travis.yml @@ -84,7 +84,7 @@ matrix: install: #- pip install -r requirements.txt - - pip install flake8 # pytest # add another testing frameworks later + - pip install --upgrade flake8 pytest==4.6.6 # pytest 5 no longer supports legacy Python before_script: - flake8 --version # stop the build if there are Python syntax errors or undefined names @@ -96,7 +96,7 @@ before_script: script: - node -e 'require("npmlog").level="verbose"; require("./lib/find-python")(null,()=>{})' - npm test - #- pytest --capture=sys # add other tests here + - GYP_MSVS_VERSION=2015 GYP_MSVS_OVERRIDE_PATH="C:\\Dummy" pytest notifications: on_success: change on_failure: change # `always` will be the setting once code changes slow down diff --git a/gyp/gyp_main.py b/gyp/gyp_main.py index aece53c8eb..f738e8009f 100755 --- a/gyp/gyp_main.py +++ b/gyp/gyp_main.py @@ -31,7 +31,7 @@ def UnixifyPath(path): out = subprocess.Popen(["cygpath", "-u", path], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) - stdout, stderr = out.communicate() + stdout, _ = out.communicate() if PY3: stdout = stdout.decode("utf-8") return str(stdout) diff --git a/gyp/pylib/gyp/MSVSSettings_test.py b/gyp/pylib/gyp/MSVSSettings_test.py index c082bbea9b..ce71c38a9b 100755 --- a/gyp/pylib/gyp/MSVSSettings_test.py +++ b/gyp/pylib/gyp/MSVSSettings_test.py @@ -1085,6 +1085,7 @@ def testConvertToMSBuildSettings_full_synthetic(self): 'GenerateManifest': 'true', 'IgnoreImportLibrary': 'true', 'LinkIncremental': 'false'}} + self.maxDiff = 9999 # on failure display a long diff actual_msbuild_settings = MSVSSettings.ConvertToMSBuildSettings( msvs_settings, self.stderr) @@ -1476,6 +1477,7 @@ def testConvertToMSBuildSettings_actual(self): 'ResourceOutputFileName': '$(IntDir)$(TargetFileName).embed.manifest.resfdsf'} } + self.maxDiff = 9999 # on failure display a long diff actual_msbuild_settings = MSVSSettings.ConvertToMSBuildSettings( msvs_settings, self.stderr) diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py index 071ad7e242..d866e81d40 100644 --- a/gyp/pylib/gyp/common.py +++ b/gyp/pylib/gyp/common.py @@ -2,7 +2,6 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. -import collections import errno import filecmp import os.path @@ -11,6 +10,11 @@ import sys import subprocess +try: + from collections.abc import MutableSet +except ImportError: + from collections import MutableSet + PY3 = bytes != str @@ -496,7 +500,7 @@ def uniquer(seq, idfun=None): # Based on http://code.activestate.com/recipes/576694/. -class OrderedSet(collections.MutableSet): +class OrderedSet(MutableSet): def __init__(self, iterable=None): self.end = end = [] end += [None, end, end] # sentinel node for doubly linked list diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index 8f3e487c48..8dbe0dc05b 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -12,6 +12,8 @@ import subprocess import sys +from collections import OrderedDict + import gyp.common import gyp.easy_xml as easy_xml import gyp.generator.ninja as ninja_generator @@ -27,16 +29,6 @@ PY3 = bytes != str -# TODO: Remove once bots are on 2.7, http://crbug.com/241769 -def _import_OrderedDict(): - import collections - try: - return collections.OrderedDict - except AttributeError: - import gyp.ordered_dict - return gyp.ordered_dict.OrderedDict -OrderedDict = _import_OrderedDict() - # Regular expression for validating Visual Studio GUIDs. If the GUID # contains lowercase hex letters, MSVS will be fine. However, @@ -179,7 +171,7 @@ def _FixPath(path): def _IsWindowsAbsPath(path): - """ + r""" On Cygwin systems Python needs a little help determining if a path is an absolute Windows path or not, so that it does not treat those as relative, which results in bad paths like: diff --git a/gyp/pylib/gyp/generator/ninja_test.py b/gyp/pylib/gyp/generator/ninja_test.py index 1ad68e4fc9..5ecfbdf004 100644 --- a/gyp/pylib/gyp/generator/ninja_test.py +++ b/gyp/pylib/gyp/generator/ninja_test.py @@ -9,7 +9,6 @@ import gyp.generator.ninja as ninja import unittest import sys -import TestCommon class TestPrefixesAndSuffixes(unittest.TestCase): diff --git a/gyp/pylib/gyp/ordered_dict.py b/gyp/pylib/gyp/ordered_dict.py deleted file mode 100644 index 6fe9c1f6c7..0000000000 --- a/gyp/pylib/gyp/ordered_dict.py +++ /dev/null @@ -1,289 +0,0 @@ -# Unmodified from http://code.activestate.com/recipes/576693/ -# other than to add MIT license header (as specified on page, but not in code). -# Linked from Python documentation here: -# http://docs.python.org/2/library/collections.html#collections.OrderedDict -# -# This should be deleted once Py2.7 is available on all bots, see -# http://crbug.com/241769. -# -# Copyright (c) 2009 Raymond Hettinger. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN -# THE SOFTWARE. - -# Backport of OrderedDict() class that runs on Python 2.4, 2.5, 2.6, 2.7 and pypy. -# Passes Python2.7's test suite and incorporates all the latest updates. - -try: - from thread import get_ident as _get_ident -except ImportError: - from dummy_thread import get_ident as _get_ident - -try: - from _abcoll import KeysView, ValuesView, ItemsView -except ImportError: - pass - - -class OrderedDict(dict): - 'Dictionary that remembers insertion order' - # An inherited dict maps keys to values. - # The inherited dict provides __getitem__, __len__, __contains__, and get. - # The remaining methods are order-aware. - # Big-O running times for all methods are the same as for regular dictionaries. - - # The internal self.__map dictionary maps keys to links in a doubly linked list. - # The circular doubly linked list starts and ends with a sentinel element. - # The sentinel element never gets deleted (this simplifies the algorithm). - # Each link is stored as a list of length three: [PREV, NEXT, KEY]. - - def __init__(self, *args, **kwds): - '''Initialize an ordered dictionary. Signature is the same as for - regular dictionaries, but keyword arguments are not recommended - because their insertion order is arbitrary. - - ''' - if len(args) > 1: - raise TypeError('expected at most 1 arguments, got %d' % len(args)) - try: - self.__root - except AttributeError: - self.__root = root = [] # sentinel node - root[:] = [root, root, None] - self.__map = {} - self.__update(*args, **kwds) - - def __setitem__(self, key, value, dict_setitem=dict.__setitem__): - 'od.__setitem__(i, y) <==> od[i]=y' - # Setting a new item creates a new link which goes at the end of the linked - # list, and the inherited dictionary is updated with the new key/value pair. - if key not in self: - root = self.__root - last = root[0] - last[1] = root[0] = self.__map[key] = [last, root, key] - dict_setitem(self, key, value) - - def __delitem__(self, key, dict_delitem=dict.__delitem__): - 'od.__delitem__(y) <==> del od[y]' - # Deleting an existing item uses self.__map to find the link which is - # then removed by updating the links in the predecessor and successor nodes. - dict_delitem(self, key) - link_prev, link_next, key = self.__map.pop(key) - link_prev[1] = link_next - link_next[0] = link_prev - - def __iter__(self): - 'od.__iter__() <==> iter(od)' - root = self.__root - curr = root[1] - while curr is not root: - yield curr[2] - curr = curr[1] - - def __reversed__(self): - 'od.__reversed__() <==> reversed(od)' - root = self.__root - curr = root[0] - while curr is not root: - yield curr[2] - curr = curr[0] - - def clear(self): - 'od.clear() -> None. Remove all items from od.' - try: - for node in self.__map.itervalues(): - del node[:] - root = self.__root - root[:] = [root, root, None] - self.__map.clear() - except AttributeError: - pass - dict.clear(self) - - def popitem(self, last=True): - '''od.popitem() -> (k, v), return and remove a (key, value) pair. - Pairs are returned in LIFO order if last is true or FIFO order if false. - - ''' - if not self: - raise KeyError('dictionary is empty') - root = self.__root - if last: - link = root[0] - link_prev = link[0] - link_prev[1] = root - root[0] = link_prev - else: - link = root[1] - link_next = link[1] - root[1] = link_next - link_next[0] = root - key = link[2] - del self.__map[key] - value = dict.pop(self, key) - return key, value - - # -- the following methods do not depend on the internal structure -- - - def keys(self): - 'od.keys() -> list of keys in od' - return list(self) - - def values(self): - 'od.values() -> list of values in od' - return [self[key] for key in self] - - def items(self): - 'od.items() -> list of (key, value) pairs in od' - return [(key, self[key]) for key in self] - - def iterkeys(self): - 'od.iterkeys() -> an iterator over the keys in od' - return iter(self) - - def itervalues(self): - 'od.itervalues -> an iterator over the values in od' - for k in self: - yield self[k] - - def items(self): - 'od.items -> an iterator over the (key, value) items in od' - for k in self: - yield (k, self[k]) - - # Suppress 'OrderedDict.update: Method has no argument': - # pylint: disable=E0211 - def update(*args, **kwds): - '''od.update(E, **F) -> None. Update od from dict/iterable E and F. - - If E is a dict instance, does: for k in E: od[k] = E[k] - If E has a .keys() method, does: for k in E.keys(): od[k] = E[k] - Or if E is an iterable of items, does: for k, v in E: od[k] = v - In either case, this is followed by: for k, v in F.items(): od[k] = v - - ''' - if len(args) > 2: - raise TypeError('update() takes at most 2 positional ' - 'arguments (%d given)' % (len(args),)) - elif not args: - raise TypeError('update() takes at least 1 argument (0 given)') - self = args[0] - # Make progressively weaker assumptions about "other" - other = () - if len(args) == 2: - other = args[1] - if isinstance(other, dict): - for key in other: - self[key] = other[key] - elif hasattr(other, 'keys'): - for key in other.keys(): - self[key] = other[key] - else: - for key, value in other: - self[key] = value - for key, value in kwds.items(): - self[key] = value - - __update = update # let subclasses override update without breaking __init__ - - __marker = object() - - def pop(self, key, default=__marker): - '''od.pop(k[,d]) -> v, remove specified key and return the corresponding value. - If key is not found, d is returned if given, otherwise KeyError is raised. - - ''' - if key in self: - result = self[key] - del self[key] - return result - if default is self.__marker: - raise KeyError(key) - return default - - def setdefault(self, key, default=None): - 'od.setdefault(k[,d]) -> od.get(k,d), also set od[k]=d if k not in od' - if key in self: - return self[key] - self[key] = default - return default - - def __repr__(self, _repr_running={}): - 'od.__repr__() <==> repr(od)' - call_key = id(self), _get_ident() - if call_key in _repr_running: - return '...' - _repr_running[call_key] = 1 - try: - if not self: - return '%s()' % (self.__class__.__name__,) - return '%s(%r)' % (self.__class__.__name__, self.items()) - finally: - del _repr_running[call_key] - - def __reduce__(self): - 'Return state information for pickling' - items = [[k, self[k]] for k in self] - inst_dict = vars(self).copy() - for k in vars(OrderedDict()): - inst_dict.pop(k, None) - if inst_dict: - return (self.__class__, (items,), inst_dict) - return self.__class__, (items,) - - def copy(self): - 'od.copy() -> a shallow copy of od' - return self.__class__(self) - - @classmethod - def fromkeys(cls, iterable, value=None): - '''OD.fromkeys(S[, v]) -> New ordered dictionary with keys from S - and values equal to v (which defaults to None). - - ''' - d = cls() - for key in iterable: - d[key] = value - return d - - def __eq__(self, other): - '''od.__eq__(y) <==> od==y. Comparison to another OD is order-sensitive - while comparison to a regular mapping is order-insensitive. - - ''' - if isinstance(other, OrderedDict): - return len(self)==len(other) and self.items() == other.items() - return dict.__eq__(self, other) - - def __ne__(self, other): - return not self == other - - # -- the following methods are only used in Python 2.7 -- - - def viewkeys(self): - "od.viewkeys() -> a set-like object providing a view on od's keys" - return KeysView(self) - - def viewvalues(self): - "od.viewvalues() -> an object providing a view on od's values" - return ValuesView(self) - - def viewitems(self): - "od.viewitems() -> a set-like object providing a view on od's items" - return ItemsView(self) - From 6b8f2652dde9f687f4d98fb26cf923825c2776c1 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Mon, 18 Nov 2019 13:37:59 +1100 Subject: [PATCH 120/551] doc: add travis badge PR-URL: https://github.com/nodejs/node-gyp/pull/1971 Reviewed-By: Richard Lau Reviewed-By: Jiawen Geng Reviewed-By: Jiawen Geng --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 9b1c171305..f6dd0e9561 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ # `node-gyp` - Node.js native addon build tool +[![Build Status](https://travis-ci.com/nodejs/node-gyp.svg?branch=master)](https://travis-ci.com/nodejs/node-gyp) + `node-gyp` is a cross-platform command-line tool written in Node.js for compiling native addon modules for Node.js. It contains a fork of the [gyp](https://gyp.gsrc.io) project that was previously used by the Chromium From f7b6b6b77b973e16f5c6e54faebfaca3fb857e19 Mon Sep 17 00:00:00 2001 From: Suraneti Rodsuwan Date: Fri, 13 Dec 2019 13:19:46 +0700 Subject: [PATCH 121/551] doc: fix typo in README.md (#1985) PR-URL: https://github.com/nodejs/node-gyp/pull/1985 Reviewed-By: Christian Clauss Reviewed-By: Jiawen Geng --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f6dd0e9561..a28324fe27 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ etc.), regardless of what version of Node.js is actually installed on your syste ## Features * The same build commands work on any of the supported platforms - * Supports the targetting of different versions of Node.js + * Supports the targeting of different versions of Node.js ## Installation From 5a64e9bd327cc48adaeb2259d6b8b6715179319b Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Mon, 2 Dec 2019 17:55:45 +0100 Subject: [PATCH 122/551] test: initial Github Actions with Ubuntu & macOS Running Python standalone tests on multiple OSes would free up Travis CI for tests of various combinations of Node.js and Python as well as tests on other [CPU architectures](https://docs.travis-ci.com/user/multi-cpu-architectures). __arch: amd64, arm64, ppc64le, s390x__ PR-URL: https://github.com/nodejs/node-gyp/pull/1985 Reviewed-By: Rod Vagg Reviewed-By: Jiawen Geng --- .github/workflows/Python_tests.yml | 36 ++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 .github/workflows/Python_tests.yml diff --git a/.github/workflows/Python_tests.yml b/.github/workflows/Python_tests.yml new file mode 100644 index 0000000000..235127f21a --- /dev/null +++ b/.github/workflows/Python_tests.yml @@ -0,0 +1,36 @@ +# TODO: Line 14, enable os: windows-latest +# TODO: Line 15, enable python-version: 3.5 +# TODO: Line 36, enable pytest --doctest-modules + +name: Python_tests +on: [push, pull_request] +jobs: + Python_tests: + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + max-parallel: 15 + matrix: + os: [macos-latest, ubuntu-latest] # , windows-latest] + python-version: [2.7, 3.6, 3.7, 3.8] # 3.5, + steps: + - uses: actions/checkout@v1 + - name: Set up Python ${{ matrix.python-version }} + uses: actions/setup-python@v1 + with: + python-version: ${{ matrix.python-version }} + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install flake8 pytest # -r requirements.txt + - name: Lint with flake8 + if: matrix.os == 'ubuntu-latest' + run: | + # stop the build if there are Python syntax errors or undefined names + flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics + # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide + flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics + - name: Test with pytest + run: pytest + # - name: Run doctests with pytest + # run: pytest --doctest-modules From d6a7e0e1fb9ba4f324fa100cce562e6a896e4429 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Thu, 28 Nov 2019 18:49:45 +0100 Subject: [PATCH 123/551] test: fix macOS Travis on Python 2.7 & 3.7 Uses `pyenv` to manage MacOS python versions since its not included in the environment. rvagg: landing this from #1979 even though it wasn't from the original author. Treating approval there as approval of this commit too. PR-URL: https://github.com/nodejs/node-gyp/pull/1979 Reviewed-By: Rod Vagg --- .travis.yml | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/.travis.yml b/.travis.yml index 4c67788aaa..e89fca16f2 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,7 +1,13 @@ dist: xenial language: python cache: pip -matrix: +addons: + homebrew: + update: true + packages: + - npm + - pyenv +jobs: include: - name: "Python 2.7 on Linux" env: NODE_GYP_FORCE_PYTHON=python2 @@ -11,7 +17,9 @@ matrix: osx_image: xcode11.2 language: shell # 'language: python' is not yet supported on macOS env: NODE_GYP_FORCE_PYTHON=python2 - before_install: HOMEBREW_NO_AUTO_UPDATE=1 brew install npm + before_install: + - pyenv install 2.7 + - pyenv global 2.7 - name: "Node.js 6 & Python 2.7 on Windows" os: windows language: node_js @@ -64,7 +72,6 @@ matrix: osx_image: xcode11.2 language: shell # 'language: python' is not yet supported on macOS env: NODE_GYP_FORCE_PYTHON=python3 - before_install: HOMEBREW_NO_AUTO_UPDATE=1 brew upgrade npm || true - name: "Node.js 12 & Python 3.7 on Windows" os: windows language: node_js From 345c70e56d2b10082d3ec84571f30c802941a607 Mon Sep 17 00:00:00 2001 From: Matias Lopez Date: Thu, 28 Nov 2019 13:10:55 -0500 Subject: [PATCH 124/551] test: direct python invocation & simpler pyenv Reorder Travis builds by OS. Replace `pyenv global` calls with properly set `PATH` and `PYENV_VERSION` env vars. Does not assume python modules are in the `PATH` so all python modules are prefixed with `python -m`. PR-URL: https://github.com/nodejs/node-gyp/pull/1979 Reviewed-By: Christian Clauss Reviewed-By: Rod Vagg --- .travis.yml | 64 ++++++++++++++++++++++++++--------------------------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/.travis.yml b/.travis.yml index e89fca16f2..c281373ecc 100644 --- a/.travis.yml +++ b/.travis.yml @@ -12,30 +12,6 @@ jobs: - name: "Python 2.7 on Linux" env: NODE_GYP_FORCE_PYTHON=python2 python: 2.7 - - name: "Python 2.7 on macOS" - os: osx - osx_image: xcode11.2 - language: shell # 'language: python' is not yet supported on macOS - env: NODE_GYP_FORCE_PYTHON=python2 - before_install: - - pyenv install 2.7 - - pyenv global 2.7 - - name: "Node.js 6 & Python 2.7 on Windows" - os: windows - language: node_js - node_js: 6 # node - env: >- - PATH=/c/Python27:/c/Python27/Scripts:$PATH - NODE_GYP_FORCE_PYTHON=/c/Python27/python.exe - before_install: choco install python2 - - name: "Node.js 12 & Python 2.7 on Windows" - os: windows - language: node_js - node_js: 12 # node - env: >- - PATH=/c/Python27:/c/Python27/Scripts:$PATH - NODE_GYP_FORCE_PYTHON=/c/Python27/python.exe - before_install: choco install python2 - name: "Node.js 6 & Python 3.8 on Linux" python: 3.8 @@ -67,11 +43,36 @@ jobs: env: NODE_GYP_FORCE_PYTHON=python3 before_install: nvm install 12 - - name: "Python 3.7 on macOS" + - name: "Python 2.7 on macOS" os: osx osx_image: xcode11.2 language: shell # 'language: python' is not yet supported on macOS - env: NODE_GYP_FORCE_PYTHON=python3 + env: NODE_GYP_FORCE_PYTHON=python2 PATH=$HOME/.pyenv/shims:$PATH PYENV_VERSION=2.7.17 + before_install: pyenv install $PYENV_VERSION + - name: "Python 3.8 on macOS" + os: osx + osx_image: xcode11.2 + language: shell # 'language: python' is not yet supported on macOS + env: NODE_GYP_FORCE_PYTHON=python3 PATH=$HOME/.pyenv/shims:$PATH PYENV_VERSION=3.8.0 + before_install: pyenv install $PYENV_VERSION + + - name: "Node.js 6 & Python 2.7 on Windows" + os: windows + language: node_js + node_js: 6 # node + env: >- + PATH=/c/Python27:/c/Python27/Scripts:$PATH + NODE_GYP_FORCE_PYTHON=/c/Python27/python.exe + before_install: choco install python2 + - name: "Node.js 12 & Python 2.7 on Windows" + os: windows + language: node_js + node_js: 12 # node + env: >- + PATH=/c/Python27:/c/Python27/Scripts:$PATH + NODE_GYP_FORCE_PYTHON=/c/Python27/python.exe + before_install: choco install python2 + - name: "Node.js 12 & Python 3.7 on Windows" os: windows language: node_js @@ -90,20 +91,19 @@ jobs: before_install: choco install python install: - #- pip install -r requirements.txt - - pip install --upgrade flake8 pytest==4.6.6 # pytest 5 no longer supports legacy Python + - python -m pip install --upgrade flake8 pytest==4.6.6 # pytest 5 no longer supports legacy Python before_script: - - flake8 --version + - python -m flake8 --version # stop the build if there are Python syntax errors or undefined names - - flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics + - python -m flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics # exit-zero treats all errors as warnings. Two space indentation is OK. The GitHub editor is 127 chars wide - - flake8 . --count --exit-zero --ignore=E111,E114,W503 --max-complexity=10 --max-line-length=127 --statistics + - python -m flake8 . --count --exit-zero --ignore=E111,E114,W503 --max-complexity=10 --max-line-length=127 --statistics - npm install - npm list script: - node -e 'require("npmlog").level="verbose"; require("./lib/find-python")(null,()=>{})' - npm test - - GYP_MSVS_VERSION=2015 GYP_MSVS_OVERRIDE_PATH="C:\\Dummy" pytest + - GYP_MSVS_VERSION=2015 GYP_MSVS_OVERRIDE_PATH="C:\\Dummy" python -m pytest notifications: on_success: change on_failure: change # `always` will be the setting once code changes slow down From 3bcba2a01ab4a3577fa1a56e543bba138d64d9f1 Mon Sep 17 00:00:00 2001 From: Matias Lopez Date: Wed, 27 Nov 2019 19:40:00 -0500 Subject: [PATCH 125/551] lib: noproxy support, match proxy detection to `request` PR-URL: https://github.com/nodejs/node-gyp/pull/1978 Reviewed-By: Rod Vagg --- README.md | 3 +- lib/install.js | 6 +-- lib/node-gyp.js | 1 + lib/proxy.js | 92 +++++++++++++++++++++++++++++++++++++++++ test/test-download.js | 95 +++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 192 insertions(+), 5 deletions(-) create mode 100644 lib/proxy.js diff --git a/README.md b/README.md index a28324fe27..d5f263d292 100644 --- a/README.md +++ b/README.md @@ -193,7 +193,8 @@ Some additional resources for Node.js native addons and writing `gyp` configurat | `--devdir=$path` | SDK download directory (default is OS cache directory) | `--ensure` | Don't reinstall headers if already present | `--dist-url=$url` | Download header tarball from custom URL -| `--proxy=$url` | Set HTTP proxy for downloading header tarball +| `--proxy=$url` | Set HTTP(S) proxy for downloading header tarball +| `--noproxy=$urls` | Set urls to ignore proxies when downloading header tarball | `--cafile=$cafile` | Override default CA chain (to download tarball) | `--nodedir=$path` | Set the path to the node source code | `--python=$path` | Set path to the Python binary diff --git a/lib/install.js b/lib/install.js index f68cd7fd6d..c919c10588 100644 --- a/lib/install.js +++ b/lib/install.js @@ -11,6 +11,7 @@ const request = require('request') const mkdir = require('mkdirp') const processRelease = require('./process-release') const win = process.platform === 'win32' +const getProxyFromURI = require('./proxy') function install (fs, gyp, argv, callback) { var release = processRelease(argv, gyp, process.version, process.release) @@ -410,10 +411,7 @@ function download (gyp, env, url) { } // basic support for a proxy server - var proxyUrl = gyp.opts.proxy || - env.http_proxy || - env.HTTP_PROXY || - env.npm_config_proxy + var proxyUrl = getProxyFromURI(gyp, env, url) if (proxyUrl) { if (/^https?:\/\//i.test(proxyUrl)) { log.verbose('download', 'using proxy url: "%s"', proxyUrl) diff --git a/lib/node-gyp.js b/lib/node-gyp.js index 9d24103900..81fc590919 100644 --- a/lib/node-gyp.js +++ b/lib/node-gyp.js @@ -67,6 +67,7 @@ proto.configDefs = { ensure: Boolean, // 'install' solution: String, // 'build' (windows only) proxy: String, // 'install' + noproxy: String, // 'install' devdir: String, // everywhere nodedir: String, // 'configure' loglevel: String, // everywhere diff --git a/lib/proxy.js b/lib/proxy.js new file mode 100644 index 0000000000..92d9ed2f7f --- /dev/null +++ b/lib/proxy.js @@ -0,0 +1,92 @@ +'use strict' +// Taken from https://github.com/request/request/blob/212570b/lib/getProxyFromURI.js + +const url = require('url') + +function formatHostname (hostname) { + // canonicalize the hostname, so that 'oogle.com' won't match 'google.com' + return hostname.replace(/^\.*/, '.').toLowerCase() +} + +function parseNoProxyZone (zone) { + zone = zone.trim().toLowerCase() + + var zoneParts = zone.split(':', 2) + var zoneHost = formatHostname(zoneParts[0]) + var zonePort = zoneParts[1] + var hasPort = zone.indexOf(':') > -1 + + return { hostname: zoneHost, port: zonePort, hasPort: hasPort } +} + +function uriInNoProxy (uri, noProxy) { + var port = uri.port || (uri.protocol === 'https:' ? '443' : '80') + var hostname = formatHostname(uri.hostname) + var noProxyList = noProxy.split(',') + + // iterate through the noProxyList until it finds a match. + return noProxyList.map(parseNoProxyZone).some(function (noProxyZone) { + var isMatchedAt = hostname.indexOf(noProxyZone.hostname) + var hostnameMatched = ( + isMatchedAt > -1 && + (isMatchedAt === hostname.length - noProxyZone.hostname.length) + ) + + if (noProxyZone.hasPort) { + return (port === noProxyZone.port) && hostnameMatched + } + + return hostnameMatched + }) +} + +function getProxyFromURI (gyp, env, uri) { + // If a string URI/URL was given, parse it into a URL object + if (typeof uri === 'string') { + // eslint-disable-next-line + uri = url.parse(uri) + } + + // Decide the proper request proxy to use based on the request URI object and the + // environmental variables (NO_PROXY, HTTP_PROXY, etc.) + // respect NO_PROXY environment variables (see: https://lynx.invisible-island.net/lynx2.8.7/breakout/lynx_help/keystrokes/environments.html) + + var noProxy = gyp.opts.noproxy || env.NO_PROXY || env.no_proxy || env.npm_config_noproxy || '' + + // if the noProxy is a wildcard then return null + + if (noProxy === '*') { + return null + } + + // if the noProxy is not empty and the uri is found return null + + if (noProxy !== '' && uriInNoProxy(uri, noProxy)) { + return null + } + + // Check for HTTP or HTTPS Proxy in environment Else default to null + + if (uri.protocol === 'http:') { + return gyp.opts.proxy || + env.HTTP_PROXY || + env.http_proxy || + env.npm_config_proxy || null + } + + if (uri.protocol === 'https:') { + return gyp.opts.proxy || + env.HTTPS_PROXY || + env.https_proxy || + env.HTTP_PROXY || + env.http_proxy || + env.npm_config_proxy || null + } + + // if none of that works, return null + // (What uri protocol are you using then?) + + return null +} + +module.exports = getProxyFromURI diff --git a/test/test-download.js b/test/test-download.js index 738a43f276..c88f0c612c 100644 --- a/test/test-download.js +++ b/test/test-download.js @@ -90,6 +90,101 @@ test('download over https with custom ca', function (t) { }) }) +test('download over http with proxy', function (t) { + t.plan(2) + + var server = http.createServer(function (req, res) { + t.strictEqual(req.headers['user-agent'], + 'node-gyp v42 (node ' + process.version + ')') + res.end('ok') + pserver.close(function () { + server.close() + }) + }) + + var pserver = http.createServer(function (req, res) { + t.strictEqual(req.headers['user-agent'], + 'node-gyp v42 (node ' + process.version + ')') + res.end('proxy ok') + server.close(function () { + pserver.close() + }) + }) + + var host = 'localhost' + server.listen(0, host, function () { + var port = this.address().port + pserver.listen(port + 1, host, function () { + var gyp = { + opts: { + proxy: 'http://' + host + ':' + (port + 1) + }, + version: '42' + } + var url = 'http://' + host + ':' + port + var req = install.test.download(gyp, {}, url) + req.on('response', function (res) { + var body = '' + res.setEncoding('utf8') + res.on('data', function (data) { + body += data + }) + res.on('end', function () { + t.strictEqual(body, 'proxy ok') + }) + }) + }) + }) +}) + +test('download over http with noproxy', function (t) { + t.plan(2) + + var server = http.createServer(function (req, res) { + t.strictEqual(req.headers['user-agent'], + 'node-gyp v42 (node ' + process.version + ')') + res.end('ok') + pserver.close(function () { + server.close() + }) + }) + + var pserver = http.createServer(function (req, res) { + t.strictEqual(req.headers['user-agent'], + 'node-gyp v42 (node ' + process.version + ')') + res.end('proxy ok') + server.close(function () { + pserver.close() + }) + }) + + var host = 'localhost' + server.listen(0, host, function () { + var port = this.address().port + pserver.listen(port + 1, host, function () { + var gyp = { + opts: { + proxy: 'http://' + host + ':' + (port + 1), + noproxy: 'localhost' + }, + version: '42' + } + var url = 'http://' + host + ':' + port + var req = install.test.download(gyp, {}, url) + req.on('response', function (res) { + var body = '' + res.setEncoding('utf8') + res.on('data', function (data) { + body += data + }) + res.on('end', function () { + t.strictEqual(body, 'ok') + }) + }) + }) + }) +}) + test('download with missing cafile', function (t) { t.plan(1) var gyp = { From 038b4a37444aa3711327863ff6cc431242769c82 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Mon, 18 Nov 2019 13:45:53 +1100 Subject: [PATCH 126/551] v5.0.6: bump version and update changelog PR-URL: https://github.com/nodejs/node-gyp/pull/1972 --- CHANGELOG.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d173ab7ec6..c0401ded2d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,28 @@ v6.0.0 2019-10-04 * [[`3d1c60ab81`](https://github.com/nodejs/node-gyp/commit/3d1c60ab81)] - **(SEMVER-MAJOR)** **lib**: accept Python 3 by default (João Reis) [#1844](https://github.com/nodejs/node-gyp/pull/1844) * [[`c6e3b65a23`](https://github.com/nodejs/node-gyp/commit/c6e3b65a23)] - **(SEMVER-MAJOR)** **lib**: raise the minimum Python version from 2.6 to 2.7 (cclauss) [#1818](https://github.com/nodejs/node-gyp/pull/1818) +v5.0.6 2019-12-16 +================= + +* [[`cdec00286f`](https://github.com/nodejs/node-gyp/commit/cdec00286f)] - **doc**: adjustments to the README.md for new users (Dan Pike) [#1919](https://github.com/nodejs/node-gyp/pull/1919) +* [[`b7c8233ef2`](https://github.com/nodejs/node-gyp/commit/b7c8233ef2)] - **test**: fix Python unittests (cclauss) [#1961](https://github.com/nodejs/node-gyp/pull/1961) +* [[`e12b00ab0a`](https://github.com/nodejs/node-gyp/commit/e12b00ab0a)] - **doc**: macOS Catalina add two commands (Christian Clauss) [#1962](https://github.com/nodejs/node-gyp/pull/1962) +* [[`70b9890c0d`](https://github.com/nodejs/node-gyp/commit/70b9890c0d)] - **test**: add header download test (Rod Vagg) [#1796](https://github.com/nodejs/node-gyp/pull/1796) +* [[`4029fa8629`](https://github.com/nodejs/node-gyp/commit/4029fa8629)] - **test**: configure proper devDir for invoking configure() (Rod Vagg) [#1796](https://github.com/nodejs/node-gyp/pull/1796) +* [[`fe8b02cc8b`](https://github.com/nodejs/node-gyp/commit/fe8b02cc8b)] - **doc**: add macOS\_Catalina.md document (cclauss) [#1940](https://github.com/nodejs/node-gyp/pull/1940) +* [[`8ea47ce365`](https://github.com/nodejs/node-gyp/commit/8ea47ce365)] - **gyp**: python3 fixes: utf8 decode, use of 'None' in eval (Wilfried Goesgens) [#1925](https://github.com/nodejs/node-gyp/pull/1925) +* [[`c7229716ba`](https://github.com/nodejs/node-gyp/commit/c7229716ba)] - **gyp**: iteritems() -\> items() in compile\_commands\_json.py (cclauss) [#1947](https://github.com/nodejs/node-gyp/pull/1947) +* [[`2a18b2a0f8`](https://github.com/nodejs/node-gyp/commit/2a18b2a0f8)] - **gyp**: make cmake python3 compatible (gengjiawen) [#1944](https://github.com/nodejs/node-gyp/pull/1944) +* [[`70f391e844`](https://github.com/nodejs/node-gyp/commit/70f391e844)] - **gyp**: fix TypeError in XcodeVersion() (Christian Clauss) [#1939](https://github.com/nodejs/node-gyp/pull/1939) +* [[`9f4f0fa34e`](https://github.com/nodejs/node-gyp/commit/9f4f0fa34e)] - **gyp**: finish decode stdout on Python 3 (Christian Clauss) [#1937](https://github.com/nodejs/node-gyp/pull/1937) +* [[`7cf507906d`](https://github.com/nodejs/node-gyp/commit/7cf507906d)] - **src,win**: allow 403 errors for arm64 node.lib (Richard Lau) [#1934](https://github.com/nodejs/node-gyp/pull/1934) +* [[`ad0d182c01`](https://github.com/nodejs/node-gyp/commit/ad0d182c01)] - **deps**: update deps to roughly match current npm@6 (Rod Vagg) [#1920](https://github.com/nodejs/node-gyp/pull/1920) +* [[`1553081ed6`](https://github.com/nodejs/node-gyp/commit/1553081ed6)] - **test**: upgrade Linux Travis CI to Python 3.8 (Christian Clauss) [#1923](https://github.com/nodejs/node-gyp/pull/1923) +* [[`0705cae9aa`](https://github.com/nodejs/node-gyp/commit/0705cae9aa)] - **travis**: ignore failed `brew upgrade npm`, update xcode (Christian Clauss) [#1932](https://github.com/nodejs/node-gyp/pull/1932) +* [[`7bfdb6f5bf`](https://github.com/nodejs/node-gyp/commit/7bfdb6f5bf)] - **build**: avoid bare exceptions in xcode\_emulation.py (Christian Clauss) [#1932](https://github.com/nodejs/node-gyp/pull/1932) +* [[`7edf7658fa`](https://github.com/nodejs/node-gyp/commit/7edf7658fa)] - **lib,install**: always download SHA sums on Windows (Sam Hughes) [#1926](https://github.com/nodejs/node-gyp/pull/1926) +* [[`69056d04fe`](https://github.com/nodejs/node-gyp/commit/69056d04fe)] - **travis**: add Windows + Python 3.8 to the mix (Rod Vagg) [#1921](https://github.com/nodejs/node-gyp/pull/1921) + v5.0.5 2019-10-04 ================= From 5a729e86eef11c018d40faa21be1120debe59950 Mon Sep 17 00:00:00 2001 From: Richard Lau Date: Mon, 23 Dec 2019 02:58:15 +0000 Subject: [PATCH 127/551] test: fix typo in header download test (#2001) PR-URL: https://github.com/nodejs/node-gyp/pull/2001 Reviewed-By: Christian Clauss Reviewed-By: Rod Vagg --- test/test-download.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/test-download.js b/test/test-download.js index c88f0c612c..fe373e3280 100644 --- a/test/test-download.js +++ b/test/test-download.js @@ -211,7 +211,7 @@ test('download headers (actual)', function (t) { process.release.name !== 'node' || semver.prerelease(process.version) !== null || semver.satisfies(process.version, '<10')) { - return t.skip('Skipping acutal download of headers due to test environment configuration') + return t.skip('Skipping actual download of headers due to test environment configuration') } t.plan(17) From 31ecc8421d0276ae5ef6d7d926cf85a428ea017f Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Sun, 29 Dec 2019 08:43:40 +0100 Subject: [PATCH 128/551] test: add Windows to GitHub Actions testing (#1996) PR-URL: https://github.com/nodejs/node-gyp/pull/1996 Reviewed-By: Jiawen Geng --- .github/workflows/Python_tests.yml | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/.github/workflows/Python_tests.yml b/.github/workflows/Python_tests.yml index 235127f21a..dba9d10933 100644 --- a/.github/workflows/Python_tests.yml +++ b/.github/workflows/Python_tests.yml @@ -1,4 +1,3 @@ -# TODO: Line 14, enable os: windows-latest # TODO: Line 15, enable python-version: 3.5 # TODO: Line 36, enable pytest --doctest-modules @@ -11,7 +10,7 @@ jobs: fail-fast: false max-parallel: 15 matrix: - os: [macos-latest, ubuntu-latest] # , windows-latest] + os: [macos-latest, ubuntu-latest, windows-latest] python-version: [2.7, 3.6, 3.7, 3.8] # 3.5, steps: - uses: actions/checkout@v1 @@ -30,7 +29,12 @@ jobs: flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics - - name: Test with pytest + - name: Test with pytest (Linux and macOS) + if: matrix.os != 'windows-latest' run: pytest + - name: Test with pytest (Windows) + if: matrix.os == 'windows-latest' + shell: bash + run: GYP_MSVS_VERSION=2015 GYP_MSVS_OVERRIDE_PATH="C:\\Dummy" pytest # - name: Run doctests with pytest # run: pytest --doctest-modules From 312c12ef4f8f11b90133590716d42adf32cdebf4 Mon Sep 17 00:00:00 2001 From: James Home Date: Sun, 29 Dec 2019 01:47:45 -0600 Subject: [PATCH 129/551] doc: update macOS_Catalina.md (#1992) PR-URL: https://github.com/nodejs/node-gyp/pull/1992 Reviewed-By: Christian Clauss Reviewed-By: Jiawen Geng --- macOS_Catalina.md | 49 ++++++++++++++++++++++++++++------------------- 1 file changed, 29 insertions(+), 20 deletions(-) diff --git a/macOS_Catalina.md b/macOS_Catalina.md index 5f02face57..530cbf25c3 100644 --- a/macOS_Catalina.md +++ b/macOS_Catalina.md @@ -1,39 +1,37 @@ # Installation notes for macOS Catalina (v10.15) -_This document specifically refers to upgrades from previous versions of macOS to Catalina (10.15). It should be removed from the source repository when Catalina ceases to be the latest macOS version or updated to deal with challenges involved in upgrades to the next version of macOS._ +_This document specifically refers to upgrades from previous versions of macOS to Catalina (10.15). It should be removed from the source repository when Catalina ceases to be the latest macOS version or when future Catalina versions no longer raise these issues._ -Lessons learned from: -* https://github.com/nodejs/node-gyp/issues/1779 -* https://github.com/nodejs/node-gyp/issues/1861 -* https://github.com/nodejs/node-gyp/issues/1927 and elsewhere - -Installing `node-gyp` on macOS can be found at https://github.com/nodejs/node-gyp#on-macos - -However, upgrading to macOS Catalina changes some settings that may cause normal `node-gyp` installations to fail. +**Upgrading to macOS Catalina may cause normal `node-gyp` installations to fail.** ### Is my Mac running macOS Catalina? -Let's make first make sure that your Mac is currently running Catalina: -% `sw_vers` +Let's first make sure that your Mac is running Catalina: +``` +% sw_vers ProductName: Mac OS X ProductVersion: 10.15 BuildVersion: 19A602 +``` +If `ProductVersion` is less then `10.15` then this document is not for you. Normal install docs for `node-gyp` on macOS can be found at https://github.com/nodejs/node-gyp#on-macos -If `ProductVersion` is less then `10.15` then this document is not really for you. ### The acid test -Next, lets see if `Xcode Command Line Tools` are installed: +To see if `Xcode Command Line Tools` is installed in a way that will work with `node-gyp`, run: 1. `/usr/sbin/pkgutil --packages | grep CL` - * If nothing is listed, then [skip to the next section](#Two-roads). - * If `com.apple.pkg.CLTools_Executables` is listed then try: + * `com.apple.pkg.CLTools_Executables` should be listed. If it isn't, this test failed. 2. `/usr/sbin/pkgutil --pkg-info com.apple.pkg.CLTools_Executables` - * If `version: 11.0.0` or later is listed then _you are done_! Your Mac should be ready to install `node-gyp`. Doing `clang -v` should show `Apple clang version 11.0.0` or later. + * `version: 11.0.0` (or later) should be listed. If it isn't, this test failed. + +If both tests succeeded, _you are done_! You should be ready to install `node-gyp`. + +If either test failed, there is a problem with your Xcode Command Line Tools installation. [Continue to Solutions](#Solutions). -As you go through the remainder of this document, at anytime you can try these `acid test` commands. If they pass then your Mac should be ready to install `node-gyp`. +### Solutions +There are three ways to install the Xcode libraries `node-gyp` needs on macOS. People running Catalina have had success with some but not others in a way that has been unpredictable. -### Two roads -There are two main ways to install `node-gyp` on macOS: 1. With the full Xcode (~7.6 GB download) from the `App Store` app. 2. With the _much_ smaller Xcode Command Line Tools via `xcode-select --install` +3. With the _much_ smaller Xcode Command Line Tools via manual download. **For people running the latest version of Catalina (10.15.2 at the time of this writing), this has worked when the other two solutions haven't.** ### Installing `node-gyp` using the full Xcode 1. `xcodebuild -version` should show `Xcode 11.1` or later. @@ -52,7 +50,7 @@ There are two main ways to install `node-gyp` on macOS: 10. `sudo xcode-select --reset` # Enter root password. No output is normal. 11. Repeat step 7 above. Is the path different this time? Repeat the _acid test_. -### Installing `node-gyp` using the Xcode Command Line Tools +### Installing `node-gyp` using the Xcode Command Line Tools via `xcode-select --install` 1. If the _acid test_ has not succeeded, then try `xcode-select --install` 2. Wait until the install process is _complete_. 3. `softwareupdate -l` # No listing is a good sign. @@ -64,6 +62,11 @@ There are two main ways to install `node-gyp` on macOS: 8. `sudo xcode-select --reset` # Enter root password. No output is normal. 9. Repeat step 5 above. Is the path different this time? Repeat the _acid test_. +### Installing `node-gyp` using the Xcode Command Line Tools via manual download +1. Download the appropriate version of the "Command Line Tools for Xcode" for your version of Catalina from developer.apple.com/download. As of MacOS 10.15.2, that's Command_Line_Tools_for_Xcode_11.3.dmg +2. Install the package. +3. Run the _acid test_. + ### I did all that and the acid test still does not pass :-( 1. `sudo rm -rf $(xcode-select -print-path)` # Enter root password. No output is normal. 2. `xcode-select --install` @@ -72,3 +75,9 @@ There are two main ways to install `node-gyp` on macOS: 5. `npm explore npm -g -- npm explore npm-lifecycle -- npm install node-gyp@latest` 6. If the _acid test_ still does _not_ pass then... 7. Add a comment to https://github.com/nodejs/node-gyp/issues/1927 so we can improve. + +Lessons learned from: +* https://github.com/nodejs/node-gyp/issues/1779 +* https://github.com/nodejs/node-gyp/issues/1861 +* https://github.com/nodejs/node-gyp/issues/1927 and elsewhere +* Thanks to @rrrix for discovering Solution 3 From 470cc2178e95ee30ee2e6a3b0985ad8000266ccf Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Sun, 29 Dec 2019 18:49:13 +1100 Subject: [PATCH 130/551] test: remove old docker test harness (#1993) PR-URL: https://github.com/nodejs/node-gyp/pull/1993 Reviewed-By: Richard Lau Reviewed-By: Jiawen Geng --- test/docker.sh | 112 ------------------------------------------------- 1 file changed, 112 deletions(-) delete mode 100755 test/docker.sh diff --git a/test/docker.sh b/test/docker.sh deleted file mode 100755 index 67014209d0..0000000000 --- a/test/docker.sh +++ /dev/null @@ -1,112 +0,0 @@ -#!/bin/bash - -#set -e - -test_node_versions="6.17.0 8.15.1 10.15.3 11.12.0" - -myuid=$(id -u) -mygid=$(id -g) -__dirname="$(CDPATH= cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -dot_node_gyp=${__dirname}/.node-gyp/ - -# borrows from https://github.com/rvagg/dnt/ - -# Simple setup function for a container: -# setup_container(image id, base image, commands to run to set up) -setup_container() { - local container_id="$1" - local base_container="$2" - local run_cmd="$3" - - # Does this image exist? If yes, ignore - docker inspect "$container_id" &> /dev/null - if [[ $? -eq 0 ]]; then - echo "Found existing container [$container_id]" - else - # No such image, so make it - echo "Did not find container [$container_id], creating..." - docker run -i $base_container /bin/bash -c "$run_cmd" - sleep 2 - docker commit $(docker ps -l -q) $container_id - fi -} - -# Run tests inside each of the versioned containers, copy cwd into npm's copy of node-gyp -# so it'll be invoked by npm when a compile is needed -# run_tests(version, test-commands) -run_tests() { - local version="$1" - local run_cmd="$2" - - run_cmd="rsync -aAXx --delete --exclude .git --exclude build /node-gyp-src/ /usr/lib/node_modules/npm/node_modules/node-gyp/; - /bin/su -s /bin/bash node-gyp -c 'cd && ${run_cmd}'" - - rm -rf $dot_node_gyp - mkdir $dot_node_gyp - - docker run \ - --rm -i \ - -v ~/.npm/:/node-gyp/.npm/ \ - -v ${dot_node_gyp}:/node-gyp/.node-gyp/ \ - -v $(pwd):/node-gyp-src/:ro \ - node-gyp-test/${version} /bin/bash -c "${run_cmd}" -} - -# A base image with build tools and a user account -setup_container "node-gyp-test/base" "ubuntu:14.04" " - adduser --gecos node-gyp --home /node-gyp/ --disabled-login node-gyp --uid $myuid && - echo "node-gyp:node-gyp" | chpasswd && - apt-get update && - apt-get install -y build-essential python git rsync curl -" - -# An image on top of the base containing clones of repos we want to use for testing -setup_container "node-gyp-test/clones" "node-gyp-test/base" " - cd /node-gyp/ && git clone https://github.com/justmoon/node-bignum.git && - cd /node-gyp/ && git clone https://github.com/bnoordhuis/node-buffertools.git && - chown -R node-gyp.node-gyp /node-gyp/ -" - -# An image for each of the node versions we want to test with that version installed and the latest npm -for v in $test_node_versions; do - setup_container "node-gyp-test/${v}" "node-gyp-test/clones" " - curl -sL https://nodejs.org/dist/v${v}/node-v${v}-linux-x64.tar.gz | tar -zxv --strip-components=1 -C /usr/ && - npm install npm@latest -g && - node -v && npm -v - " -done - -# Test use of --target=x.y.z to compile against alternate versions -test_download_node_version() { - local run_with_ver="$1" - local expected_dir="$2" - local expected_ver="$3" - run_tests $run_with_ver "cd node-buffertools && npm install --loglevel=info --target=${expected_ver}" - local node_ver=$(cat "${dot_node_gyp}${expected_dir}/node_version.h" | grep '#define NODE_\w*_VERSION [0-9]*$') - node_ver=$(echo $node_ver | sed 's/#define NODE_[A-Z]*_VERSION //g' | sed 's/ /./g') - if [ "X$(echo $node_ver)" != "X${expected_ver}" ]; then - echo "Did not download v${expected_ver} using --target, instead got: $(echo $node_ver)" - exit 1 - fi - echo "Verified correct download of [v${node_ver}]" -} - -test_download_node_version "0.12.7" "0.10.30/src" "0.10.30" -# should download the headers file -test_download_node_version "4.3.0" "4.3.0/include/node" "4.3.0" -test_download_node_version "5.6.0" "5.6.0/include/node" "5.6.0" - -# TODO: test --dist-url by starting up a localhost server and serving up tarballs - -# testing --dist-url, using simple-proxy.js to make localhost work as a distribution -# point for tarballs -# we can test whether it uses the proxy because after 2 connections the proxy will -# die and therefore should not be running at the end of the test, `nc` can tell us this -run_tests "0.12.7" " - (node /node-gyp-src/test/simple-proxy.js 8080 /boombar/ https://nodejs.org/dist/ &) && - cd node-buffertools && - NODEJS_ORG_MIRROR=http://localhost:8080/boombar/ /node-gyp-src/bin/node-gyp.js --loglevel=info rebuild && - nc -z localhost 8080 && echo -e \"\\n\\n\\033[31mFAILED TO USE LOCAL PROXY\\033[39m\\n\\n\" -" - -rm -rf $dot_node_gyp From 26cd6eaea614b4e5e48142eb352038ebb55f0e90 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Sun, 29 Dec 2019 18:50:15 +1100 Subject: [PATCH 131/551] doc: add GitHub Actions badge (#1994) PR-URL: https://github.com/nodejs/node-gyp/pull/1994 Reviewed-By: Richard Lau Reviewed-By: Christian Clauss Reviewed-By: Jiawen Geng --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index d5f263d292..0c15d33b48 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,7 @@ # `node-gyp` - Node.js native addon build tool -[![Build Status](https://travis-ci.com/nodejs/node-gyp.svg?branch=master)](https://travis-ci.com/nodejs/node-gyp) +[![Travis CI](https://travis-ci.com/nodejs/node-gyp.svg?branch=master)](https://travis-ci.com/nodejs/node-gyp) +[![Build Status](https://github.com/nodejs/node-gyp/workflows/Python_tests/badge.svg)](https://github.com/nodejs/node-gyp/actions?workflow=Python_tests) `node-gyp` is a cross-platform command-line tool written in Node.js for compiling native addon modules for Node.js. It contains a fork of the From 14f2a07a390328d8cdd4da043a2246d3e1ec01dd Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Tue, 31 Dec 2019 21:39:18 +0100 Subject: [PATCH 132/551] gyp: list(dict) so we can del dict(key) while iterating Fixes #1998 Reviewed-By: Rod Vagg PR-URL: https://github.com/nodejs/node-gyp/pull/2009 --- gyp/pylib/gyp/input.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index 2973c078fc..d1742800ac 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -2286,7 +2286,7 @@ def SetUpConfigurations(target, target_dict): merged_configurations[configuration]) # Now drop all the abstract ones. - for configuration in target_dict['configurations'].keys(): + for configuration in list(target_dict['configurations']): old_configuration_dict = target_dict['configurations'][configuration] if old_configuration_dict.get('abstract'): del target_dict['configurations'][configuration] From 9a7dd16b76b44cdb83b1c4f097fb1100cb31dab1 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Fri, 3 Jan 2020 11:35:37 +1100 Subject: [PATCH 133/551] doc: remove backticks from Python version list Reviewed-By: Richard Lau Reviewed-By: Christian Clauss PR-URL: https://github.com/nodejs/node-gyp/pull/2011 --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 0c15d33b48..79abf4ba71 100644 --- a/README.md +++ b/README.md @@ -31,13 +31,13 @@ Depending on your operating system, you will need to install: ### On Unix - * `Python v2.7, v3.5, v3.6, or v3.7` + * Python v2.7, v3.5, v3.6, or v3.7 * `make` * A proper C/C++ compiler toolchain, like [GCC](https://gcc.gnu.org) ### On macOS - * `Python v2.7, v3.5, v3.6, or v3.7` + * Python v2.7, v3.5, v3.6, or v3.7 * [Xcode](https://developer.apple.com/xcode/download/) * You also need to install the `XCode Command Line Tools` by running `xcode-select --install`. Alternatively, if you already have the full Xcode installed, you can find them under the menu `Xcode -> Open Developer Tool -> More Developer Tools...`. This step will install `clang`, `clang++`, and `make`. * If your Mac has been _upgraded_ to macOS Catalina (10.15), please read [macOS_Catalina.md](macOS_Catalina.md). From f242ce4d2c93d38fd91af77880e971e7bbfe8eaf Mon Sep 17 00:00:00 2001 From: Xavier Guimard Date: Sun, 29 Dec 2019 09:15:08 +0100 Subject: [PATCH 134/551] =?UTF-8?q?lib:=20compatibility=20with=20semver=20?= =?UTF-8?q?=E2=89=A5=207=20(`new`=20for=20semver.Range)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes: #2005 Reviewed-By: Rod Vagg PR-URL: https://github.com/nodejs/node-gyp/pull/2006 --- lib/find-python.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/find-python.js b/lib/find-python.js index 9d918a881b..43bf85985a 100644 --- a/lib/find-python.js +++ b/lib/find-python.js @@ -226,7 +226,7 @@ PythonFinder.prototype = { } this.addLog(`- version is "${version}"`) - const range = semver.Range(this.semverRange) + const range = new semver.Range(this.semverRange) var valid = false try { valid = range.test(version) From a79d866ac3bf6224b3a69268ebc86a0758e8c7c5 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Fri, 3 Jan 2020 11:43:33 +1100 Subject: [PATCH 135/551] v6.1.0: bump version and update changelog PR-URL: https://github.com/nodejs/node-gyp/pull/2013 --- CHANGELOG.md | 22 ++++++++++++++++++++++ package.json | 2 +- 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c0401ded2d..80da1b1e9e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,25 @@ +v6.1.0 2019-01-08 +================= + +* [[`9a7dd16b76`](https://github.com/nodejs/node-gyp/commit/9a7dd16b76)] - **doc**: remove backticks from Python version list (Rod Vagg) [#2011](https://github.com/nodejs/node-gyp/pull/2011) +* [[`26cd6eaea6`](https://github.com/nodejs/node-gyp/commit/26cd6eaea6)] - **doc**: add GitHub Actions badge (#1994) (Rod Vagg) [#1994](https://github.com/nodejs/node-gyp/pull/1994) +* [[`312c12ef4f`](https://github.com/nodejs/node-gyp/commit/312c12ef4f)] - **doc**: update macOS\_Catalina.md (#1992) (James Home) [#1992](https://github.com/nodejs/node-gyp/pull/1992) +* [[`f7b6b6b77b`](https://github.com/nodejs/node-gyp/commit/f7b6b6b77b)] - **doc**: fix typo in README.md (#1985) (Suraneti Rodsuwan) [#1985](https://github.com/nodejs/node-gyp/pull/1985) +* [[`6b8f2652dd`](https://github.com/nodejs/node-gyp/commit/6b8f2652dd)] - **doc**: add travis badge (Rod Vagg) [#1971](https://github.com/nodejs/node-gyp/pull/1971) +* [[`20aa0b44f7`](https://github.com/nodejs/node-gyp/commit/20aa0b44f7)] - **doc**: macOS Catalina add two commands (Christian Clauss) [#1962](https://github.com/nodejs/node-gyp/pull/1962) +* [[`14f2a07a39`](https://github.com/nodejs/node-gyp/commit/14f2a07a39)] - **gyp**: list(dict) so we can del dict(key) while iterating (Christian Clauss) [#2009](https://github.com/nodejs/node-gyp/pull/2009) +* [[`f242ce4d2c`](https://github.com/nodejs/node-gyp/commit/f242ce4d2c)] - **lib**: compatibility with semver ≥ 7 (`new` for semver.Range) (Xavier Guimard) [#2006](https://github.com/nodejs/node-gyp/pull/2006) +* [[`3bcba2a01a`](https://github.com/nodejs/node-gyp/commit/3bcba2a01a)] - **(SEMVER-MINOR)** **lib**: noproxy support, match proxy detection to `request` (Matias Lopez) [#1978](https://github.com/nodejs/node-gyp/pull/1978) +* [[`470cc2178e`](https://github.com/nodejs/node-gyp/commit/470cc2178e)] - **test**: remove old docker test harness (#1993) (Rod Vagg) [#1993](https://github.com/nodejs/node-gyp/pull/1993) +* [[`31ecc8421d`](https://github.com/nodejs/node-gyp/commit/31ecc8421d)] - **test**: add Windows to GitHub Actions testing (#1996) (Christian Clauss) [#1996](https://github.com/nodejs/node-gyp/pull/1996) +* [[`5a729e86ee`](https://github.com/nodejs/node-gyp/commit/5a729e86ee)] - **test**: fix typo in header download test (#2001) (Richard Lau) [#2001](https://github.com/nodejs/node-gyp/pull/2001) +* [[`345c70e56d`](https://github.com/nodejs/node-gyp/commit/345c70e56d)] - **test**: direct python invocation & simpler pyenv (Matias Lopez) [#1979](https://github.com/nodejs/node-gyp/pull/1979) +* [[`d6a7e0e1fb`](https://github.com/nodejs/node-gyp/commit/d6a7e0e1fb)] - **test**: fix macOS Travis on Python 2.7 & 3.7 (Christian Clauss) [#1979](https://github.com/nodejs/node-gyp/pull/1979) +* [[`5a64e9bd32`](https://github.com/nodejs/node-gyp/commit/5a64e9bd32)] - **test**: initial Github Actions with Ubuntu & macOS (Christian Clauss) [#1985](https://github.com/nodejs/node-gyp/pull/1985) +* [[`04da736d38`](https://github.com/nodejs/node-gyp/commit/04da736d38)] - **test**: fix Python unittests (cclauss) [#1961](https://github.com/nodejs/node-gyp/pull/1961) +* [[`0670e5189d`](https://github.com/nodejs/node-gyp/commit/0670e5189d)] - **test**: add header download test (Rod Vagg) [#1796](https://github.com/nodejs/node-gyp/pull/1796) +* [[`c506a6a150`](https://github.com/nodejs/node-gyp/commit/c506a6a150)] - **test**: configure proper devDir for invoking configure() (Rod Vagg) [#1796](https://github.com/nodejs/node-gyp/pull/1796) + v6.0.1 2019-11-01 ================= diff --git a/package.json b/package.json index 243738506d..478f43cb81 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "6.0.1", + "version": "6.1.0", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From 1f7e1e93b5116e727d0719dcca87ee38c18042ae Mon Sep 17 00:00:00 2001 From: Andrew Casey Date: Wed, 8 Jan 2020 13:49:39 -0800 Subject: [PATCH 136/551] lib: ignore VS instances that cause COMExceptions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I have quite a few instances of VS installed and it looks like Find-VisualStudio.cs enumerates all of them, even when find-visualstudio.js already knows which one it wants (from the environment variable in the developer command prompt). One of them (from 15.7.2, if that's interesting) causes a COMException on the ISetupInstance2.GetPackages call. Ignoring such packages seems harmless and unblocks the rest of the run. PR-URL: https://github.com/nodejs/node-gyp/pull/2018 Reviewed-By: João Reis --- lib/Find-VisualStudio.cs | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/lib/Find-VisualStudio.cs b/lib/Find-VisualStudio.cs index 0a533f42d6..d2e45a7627 100644 --- a/lib/Find-VisualStudio.cs +++ b/lib/Find-VisualStudio.cs @@ -205,7 +205,14 @@ public static void PrintJson() return; } - instances.Add(InstanceJson(rgelt[0])); + try + { + instances.Add(InstanceJson(rgelt[0])); + } + catch (COMException) + { + // Ignore instances that can't be queried. + } } } From d1dea13fe44bc65ee3ea8441c08c9a3d03d6d63d Mon Sep 17 00:00:00 2001 From: Quentin Vernot <6038707+quentinvernot@users.noreply.github.com> Date: Tue, 14 Jan 2020 18:21:42 +0100 Subject: [PATCH 137/551] doc: fix changelog 6.1.0 release year to be 2020 PR-URL: https://github.com/nodejs/node-gyp/pull/2021 Reviewed-By: Richard Lau Reviewed-By: Christian Clauss Reviewed-By: Rod Vagg --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 80da1b1e9e..5314f7106e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,4 @@ -v6.1.0 2019-01-08 +v6.1.0 2020-01-08 ================= * [[`9a7dd16b76`](https://github.com/nodejs/node-gyp/commit/9a7dd16b76)] - **doc**: remove backticks from Python version list (Rod Vagg) [#2011](https://github.com/nodejs/node-gyp/pull/2011) From e505aa8642c18a0b14297be2327253377a9ca77a Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Mon, 16 Dec 2019 11:59:40 +1100 Subject: [PATCH 138/551] v5.0.7: bump version and update changelog Republish of v5.0.6 but with node-gyp-v5.0.6.tar.gz removed from pack file PR-URL: https://github.com/nodejs/node-gyp/pull/1972 --- CHANGELOG.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5314f7106e..878dabcfa4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,6 +47,11 @@ v6.0.0 2019-10-04 * [[`3d1c60ab81`](https://github.com/nodejs/node-gyp/commit/3d1c60ab81)] - **(SEMVER-MAJOR)** **lib**: accept Python 3 by default (João Reis) [#1844](https://github.com/nodejs/node-gyp/pull/1844) * [[`c6e3b65a23`](https://github.com/nodejs/node-gyp/commit/c6e3b65a23)] - **(SEMVER-MAJOR)** **lib**: raise the minimum Python version from 2.6 to 2.7 (cclauss) [#1818](https://github.com/nodejs/node-gyp/pull/1818) +v5.0.7 2019-12-16 +================= + +Republish of v5.0.6 with unnecessary tarball removed from pack file. + v5.0.6 2019-12-16 ================= From ae5b150051ab6249e57a3153b288e66e74d7bc7b Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Wed, 15 Jan 2020 10:36:31 +0100 Subject: [PATCH 139/551] doc: Catalina suggestion: remove /Library/Developer/CommandLineTools PR-URL: https://github.com/nodejs/node-gyp/pull/2022 Reviewed-By: Rod Vagg --- macOS_Catalina.md | 1 + 1 file changed, 1 insertion(+) diff --git a/macOS_Catalina.md b/macOS_Catalina.md index 530cbf25c3..9e83e2eabe 100644 --- a/macOS_Catalina.md +++ b/macOS_Catalina.md @@ -69,6 +69,7 @@ There are three ways to install the Xcode libraries `node-gyp` needs on macOS. P ### I did all that and the acid test still does not pass :-( 1. `sudo rm -rf $(xcode-select -print-path)` # Enter root password. No output is normal. +2. `sudo rm -rf /Library/Developer/CommandLineTools` # Enter root password. 2. `xcode-select --install` 3. If the [_acid test_](#The-acid-test) still does _not_ pass then... 4. `npm explore npm -g -- npm install node-gyp@latest` From 48642191f5b3d7026fd96b3505b3b4e82ce25d65 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Przemys=C5=82aw=20Bitkowski?= Date: Sat, 18 Jan 2020 17:45:00 +0100 Subject: [PATCH 140/551] doc: add download link for Command Line Tools for Xcode PR-URL: https://github.com/nodejs/node-gyp/pull/2029 Reviewed-By: Rod Vagg --- macOS_Catalina.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/macOS_Catalina.md b/macOS_Catalina.md index 9e83e2eabe..4fdbb4a893 100644 --- a/macOS_Catalina.md +++ b/macOS_Catalina.md @@ -63,7 +63,7 @@ There are three ways to install the Xcode libraries `node-gyp` needs on macOS. P 9. Repeat step 5 above. Is the path different this time? Repeat the _acid test_. ### Installing `node-gyp` using the Xcode Command Line Tools via manual download -1. Download the appropriate version of the "Command Line Tools for Xcode" for your version of Catalina from developer.apple.com/download. As of MacOS 10.15.2, that's Command_Line_Tools_for_Xcode_11.3.dmg +1. Download the appropriate version of the "Command Line Tools for Xcode" for your version of Catalina from developer.apple.com/download. As of MacOS 10.15.2, that's [Command_Line_Tools_for_Xcode_11.3.dmg](https://download.developer.apple.com/Developer_Tools/Command_Line_Tools_for_Xcode_11.3/Command_Line_Tools_for_Xcode_11.3.dmg) 2. Install the package. 3. Run the _acid test_. From 35de45984fa00b4b5bdcbf9b3d92e333a8039fe0 Mon Sep 17 00:00:00 2001 From: Jonathan Hult Date: Wed, 22 Jan 2020 23:57:54 -0500 Subject: [PATCH 141/551] doc: update catalina xcode cli tools download link; formatting PR-URL: https://github.com/nodejs/node-gyp/pull/2034 Reviewed-By: Rod Vagg Reviewed-By: Christian Clauss --- macOS_Catalina.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/macOS_Catalina.md b/macOS_Catalina.md index 4fdbb4a893..8c2ce9ad59 100644 --- a/macOS_Catalina.md +++ b/macOS_Catalina.md @@ -63,15 +63,15 @@ There are three ways to install the Xcode libraries `node-gyp` needs on macOS. P 9. Repeat step 5 above. Is the path different this time? Repeat the _acid test_. ### Installing `node-gyp` using the Xcode Command Line Tools via manual download -1. Download the appropriate version of the "Command Line Tools for Xcode" for your version of Catalina from developer.apple.com/download. As of MacOS 10.15.2, that's [Command_Line_Tools_for_Xcode_11.3.dmg](https://download.developer.apple.com/Developer_Tools/Command_Line_Tools_for_Xcode_11.3/Command_Line_Tools_for_Xcode_11.3.dmg) +1. Download the appropriate version of the "Command Line Tools for Xcode" for your version of Catalina from . As of MacOS 10.15.2, that's [Command_Line_Tools_for_Xcode_11.3.dmg](https://download.developer.apple.com/Developer_Tools/Command_Line_Tools_for_Xcode_11.3/Command_Line_Tools_for_Xcode_11.3.dmg) 2. Install the package. -3. Run the _acid test_. +3. Run the [_acid test_ steps above](#The-acid-test). ### I did all that and the acid test still does not pass :-( 1. `sudo rm -rf $(xcode-select -print-path)` # Enter root password. No output is normal. 2. `sudo rm -rf /Library/Developer/CommandLineTools` # Enter root password. 2. `xcode-select --install` -3. If the [_acid test_](#The-acid-test) still does _not_ pass then... +3. If the [_acid test_ steps above](#The-acid-test) still does _not_ pass then... 4. `npm explore npm -g -- npm install node-gyp@latest` 5. `npm explore npm -g -- npm explore npm-lifecycle -- npm install node-gyp@latest` 6. If the _acid test_ still does _not_ pass then... From dab030536b6a70ecae37debc74c581db9e5280fd Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Mon, 3 Feb 2020 14:48:20 +1100 Subject: [PATCH 142/551] v5.1.0: bump version and update changelog PR-URL: https://github.com/nodejs/node-gyp/pull/2012 --- CHANGELOG.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 878dabcfa4..d27e181b42 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,6 +47,24 @@ v6.0.0 2019-10-04 * [[`3d1c60ab81`](https://github.com/nodejs/node-gyp/commit/3d1c60ab81)] - **(SEMVER-MAJOR)** **lib**: accept Python 3 by default (João Reis) [#1844](https://github.com/nodejs/node-gyp/pull/1844) * [[`c6e3b65a23`](https://github.com/nodejs/node-gyp/commit/c6e3b65a23)] - **(SEMVER-MAJOR)** **lib**: raise the minimum Python version from 2.6 to 2.7 (cclauss) [#1818](https://github.com/nodejs/node-gyp/pull/1818) +v5.1.0 2020-02-05 +================= + +* [[`f37a8b40d0`](https://github.com/nodejs/node-gyp/commit/f37a8b40d0)] - **doc**: add GitHub Actions badge (#1994) (Rod Vagg) [#1994](https://github.com/nodejs/node-gyp/pull/1994) +* [[`cb3f6aae5e`](https://github.com/nodejs/node-gyp/commit/cb3f6aae5e)] - **doc**: update macOS\_Catalina.md (#1992) (James Home) [#1992](https://github.com/nodejs/node-gyp/pull/1992) +* [[`0607596a4c`](https://github.com/nodejs/node-gyp/commit/0607596a4c)] - **doc**: fix typo in README.md (#1985) (Suraneti Rodsuwan) [#1985](https://github.com/nodejs/node-gyp/pull/1985) +* [[`0d5a415a14`](https://github.com/nodejs/node-gyp/commit/0d5a415a14)] - **doc**: add travis badge (Rod Vagg) [#1971](https://github.com/nodejs/node-gyp/pull/1971) +* [[`103740cd95`](https://github.com/nodejs/node-gyp/commit/103740cd95)] - **gyp**: list(dict) so we can del dict(key) while iterating (Christian Clauss) [#2009](https://github.com/nodejs/node-gyp/pull/2009) +* [[`278dcddbdd`](https://github.com/nodejs/node-gyp/commit/278dcddbdd)] - **lib**: ignore VS instances that cause COMExceptions (Andrew Casey) [#2018](https://github.com/nodejs/node-gyp/pull/2018) +* [[`1694907bbf`](https://github.com/nodejs/node-gyp/commit/1694907bbf)] - **lib**: compatibility with semver ≥ 7 (`new` for semver.Range) (Xavier Guimard) [#2006](https://github.com/nodejs/node-gyp/pull/2006) +* [[`a3f1143514`](https://github.com/nodejs/node-gyp/commit/a3f1143514)] - **(SEMVER-MINOR)** **lib**: noproxy support, match proxy detection to `request` (Matias Lopez) [#1978](https://github.com/nodejs/node-gyp/pull/1978) +* [[`52365819c7`](https://github.com/nodejs/node-gyp/commit/52365819c7)] - **test**: remove old docker test harness (#1993) (Rod Vagg) [#1993](https://github.com/nodejs/node-gyp/pull/1993) +* [[`bc509c511d`](https://github.com/nodejs/node-gyp/commit/bc509c511d)] - **test**: add Windows to GitHub Actions testing (#1996) (Christian Clauss) [#1996](https://github.com/nodejs/node-gyp/pull/1996) +* [[`91ee26dd48`](https://github.com/nodejs/node-gyp/commit/91ee26dd48)] - **test**: fix typo in header download test (#2001) (Richard Lau) [#2001](https://github.com/nodejs/node-gyp/pull/2001) +* [[`0923f344c9`](https://github.com/nodejs/node-gyp/commit/0923f344c9)] - **test**: direct python invocation & simpler pyenv (Matias Lopez) [#1979](https://github.com/nodejs/node-gyp/pull/1979) +* [[`32c8744b34`](https://github.com/nodejs/node-gyp/commit/32c8744b34)] - **test**: fix macOS Travis on Python 2.7 & 3.7 (Christian Clauss) [#1979](https://github.com/nodejs/node-gyp/pull/1979) +* [[`fd4b1351e4`](https://github.com/nodejs/node-gyp/commit/fd4b1351e4)] - **test**: initial Github Actions with Ubuntu & macOS (Christian Clauss) [#1985](https://github.com/nodejs/node-gyp/pull/1985) + v5.0.7 2019-12-16 ================= From 972780bde72b9a61153f3ea7ea9700cec001ffd7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C3=ABl=20Zasso?= Date: Wed, 19 Feb 2020 04:42:32 +0100 Subject: [PATCH 143/551] gyp: sync code base with nodejs repo (#1975) PR-URL: https://github.com/nodejs/node-gyp/pull/1975 Reviewed-By: Ujjwal Sharma Reviewed-By: Christian Clauss Reviewed-By: Rod Vagg --- gyp/AUTHORS | 2 + gyp/DEPS | 24 --- gyp/README.md | 4 + gyp/codereview.settings | 10 -- gyp/pylib/gyp/MSVSNew.py | 8 +- gyp/pylib/gyp/MSVSSettings.py | 5 +- gyp/pylib/gyp/MSVSSettings_test.py | 10 +- gyp/pylib/gyp/MSVSUtil.py | 3 +- gyp/pylib/gyp/MSVSVersion.py | 147 +++++++++++----- gyp/pylib/gyp/common.py | 18 +- gyp/pylib/gyp/easy_xml.py | 2 +- gyp/pylib/gyp/easy_xml_test.py | 5 +- gyp/pylib/gyp/generator/cmake.py | 53 ++++-- gyp/pylib/gyp/generator/eclipse.py | 4 +- gyp/pylib/gyp/generator/make.py | 30 ++-- gyp/pylib/gyp/generator/msvs.py | 149 +++++++++++----- gyp/pylib/gyp/generator/msvs_test.py | 5 +- gyp/pylib/gyp/generator/ninja.py | 139 +++++++++++---- gyp/pylib/gyp/generator/ninja_test.py | 5 +- gyp/pylib/gyp/generator/xcode.py | 16 +- gyp/pylib/gyp/input.py | 43 +++-- gyp/pylib/gyp/mac_tool.py | 242 ++++++++++++++++++-------- gyp/pylib/gyp/msvs_emulation.py | 34 +++- gyp/pylib/gyp/win_tool.py | 18 +- gyp/pylib/gyp/xcode_emulation.py | 241 ++++++++++++++++++++----- gyp/pylib/gyp/xcode_ninja.py | 25 ++- gyp/pylib/gyp/xcodeproj_file.py | 115 +++++++++--- gyp/tools/pretty_vcproj.py | 4 +- 28 files changed, 971 insertions(+), 390 deletions(-) delete mode 100644 gyp/DEPS create mode 100644 gyp/README.md delete mode 100644 gyp/codereview.settings diff --git a/gyp/AUTHORS b/gyp/AUTHORS index d76d8cd768..130c816058 100644 --- a/gyp/AUTHORS +++ b/gyp/AUTHORS @@ -11,3 +11,5 @@ Ryan Norton David J. Sankel Eric N. Vander Weele Tom Freudenberg +Julien Brianceau +Refael Ackermann diff --git a/gyp/DEPS b/gyp/DEPS deleted file mode 100644 index 2e1120f274..0000000000 --- a/gyp/DEPS +++ /dev/null @@ -1,24 +0,0 @@ -# DEPS file for gclient use in buildbot execution of gyp tests. -# -# (You don't need to use gclient for normal GYP development work.) - -vars = { - "chrome_trunk": "http://src.chromium.org/svn/trunk", - "googlecode_url": "http://%s.googlecode.com/svn", -} - -deps = { -} - -deps_os = { - "win": { - "third_party/cygwin": - Var("chrome_trunk") + "/deps/third_party/cygwin@66844", - - "third_party/python_26": - Var("chrome_trunk") + "/tools/third_party/python_26@89111", - - "src/third_party/pefile": - (Var("googlecode_url") % "pefile") + "/trunk@63", - }, -} diff --git a/gyp/README.md b/gyp/README.md new file mode 100644 index 0000000000..c0d73ac958 --- /dev/null +++ b/gyp/README.md @@ -0,0 +1,4 @@ +GYP can Generate Your Projects. +=================================== + +Documents are available at [gyp.gsrc.io](https://gyp.gsrc.io), or you can check out ```md-pages``` branch to read those documents offline. diff --git a/gyp/codereview.settings b/gyp/codereview.settings deleted file mode 100644 index faf37f1145..0000000000 --- a/gyp/codereview.settings +++ /dev/null @@ -1,10 +0,0 @@ -# This file is used by gcl to get repository specific information. -CODE_REVIEW_SERVER: codereview.chromium.org -CC_LIST: gyp-developer@googlegroups.com -VIEW_VC: https://chromium.googlesource.com/external/gyp/+/ -TRY_ON_UPLOAD: False -TRYSERVER_PROJECT: gyp -TRYSERVER_PATCHLEVEL: 1 -TRYSERVER_ROOT: gyp -TRYSERVER_SVN_URL: svn://svn.chromium.org/chrome-try/try-nacl -PROJECT: gyp diff --git a/gyp/pylib/gyp/MSVSNew.py b/gyp/pylib/gyp/MSVSNew.py index 76c4b95c0c..740ef2c73f 100644 --- a/gyp/pylib/gyp/MSVSNew.py +++ b/gyp/pylib/gyp/MSVSNew.py @@ -7,6 +7,7 @@ import hashlib import os import random +from operator import attrgetter import gyp.common @@ -59,9 +60,6 @@ def __cmp__(self, other): # Sort by name then guid (so things are in order on vs2008). return cmp((self.name, self.get_guid()), (other.name, other.get_guid())) - def __lt__(self, other): - return self.__cmp__(other) < 0 - class MSVSFolder(MSVSSolutionEntry): """Folder in a Visual Studio project or solution.""" @@ -89,7 +87,7 @@ def __init__(self, path, name = None, entries = None, self.guid = guid # Copy passed lists (or set to empty lists) - self.entries = sorted(list(entries or [])) + self.entries = sorted(entries or [], key=attrgetter('path')) self.items = list(items or []) self.entry_type_guid = ENTRY_TYPE_GUIDS['folder'] @@ -233,7 +231,7 @@ def Write(self, writer=gyp.common.WriteOnDiff): if isinstance(e, MSVSFolder): entries_to_check += e.entries - all_entries = sorted(all_entries) + all_entries = sorted(all_entries, key=attrgetter('path')) # Open file and print header f = writer(self.path) diff --git a/gyp/pylib/gyp/MSVSSettings.py b/gyp/pylib/gyp/MSVSSettings.py index 065a339a80..5dd8f8c1e6 100644 --- a/gyp/pylib/gyp/MSVSSettings.py +++ b/gyp/pylib/gyp/MSVSSettings.py @@ -467,7 +467,7 @@ def ConvertToMSBuildSettings(msvs_settings, stderr=sys.stderr): msvs_tool[msvs_setting](msvs_value, msbuild_settings) except ValueError as e: print('Warning: while converting %s/%s to MSBuild, ' - '%s' % (msvs_tool_name, msvs_setting, e), file=stderr) + '%s' % (msvs_tool_name, msvs_setting, e), file=stderr) else: _ValidateExclusionSetting(msvs_setting, msvs_tool, @@ -477,7 +477,7 @@ def ConvertToMSBuildSettings(msvs_settings, stderr=sys.stderr): stderr) else: print('Warning: unrecognized tool %s while converting to ' - 'MSBuild.' % msvs_tool_name, file=stderr) + 'MSBuild.' % msvs_tool_name, file=stderr) return msbuild_settings @@ -598,6 +598,7 @@ def _ValidateSettings(validators, settings, stderr): _Same(_compile, 'UseFullPaths', _boolean) # /FC _Same(_compile, 'WholeProgramOptimization', _boolean) # /GL _Same(_compile, 'XMLDocumentationFileName', _file_name) +_Same(_compile, 'CompileAsWinRT', _boolean) # /ZW _Same(_compile, 'AssemblerOutput', _Enumeration(['NoListing', diff --git a/gyp/pylib/gyp/MSVSSettings_test.py b/gyp/pylib/gyp/MSVSSettings_test.py index ce71c38a9b..77b79e650d 100755 --- a/gyp/pylib/gyp/MSVSSettings_test.py +++ b/gyp/pylib/gyp/MSVSSettings_test.py @@ -6,14 +6,14 @@ """Unit tests for the MSVSSettings.py file.""" -try: - from cStringIO import StringIO -except ImportError: - from io import StringIO - import unittest import gyp.MSVSSettings as MSVSSettings +try: + from StringIO import StringIO # Python 2 +except ImportError: + from io import StringIO # Python 3 + class TestSequenceFunctions(unittest.TestCase): diff --git a/gyp/pylib/gyp/MSVSUtil.py b/gyp/pylib/gyp/MSVSUtil.py index c8187eb331..f24530b275 100644 --- a/gyp/pylib/gyp/MSVSUtil.py +++ b/gyp/pylib/gyp/MSVSUtil.py @@ -14,6 +14,7 @@ 'loadable_module': 'dll', 'shared_library': 'dll', 'static_library': 'lib', + 'windows_driver': 'sys', } @@ -110,7 +111,7 @@ def ShardTargets(target_list, target_dicts): else: new_target_dicts[t] = target_dicts[t] # Shard dependencies. - for t in new_target_dicts: + for t in sorted(new_target_dicts): for deptype in ('dependencies', 'dependencies_original'): dependencies = copy.copy(new_target_dicts[t].get(deptype, [])) new_dependencies = [] diff --git a/gyp/pylib/gyp/MSVSVersion.py b/gyp/pylib/gyp/MSVSVersion.py index c7cf68d3a1..ce9b349834 100644 --- a/gyp/pylib/gyp/MSVSVersion.py +++ b/gyp/pylib/gyp/MSVSVersion.py @@ -15,12 +15,16 @@ PY3 = bytes != str +def JoinPath(*args): + return os.path.normpath(os.path.join(*args)) + + class VisualStudioVersion(object): """Information regarding a version of Visual Studio.""" def __init__(self, short_name, description, solution_version, project_version, flat_sln, uses_vcxproj, - path, sdk_based, default_toolset=None): + path, sdk_based, default_toolset=None, compatible_sdks=None): self.short_name = short_name self.description = description self.solution_version = solution_version @@ -30,6 +34,9 @@ def __init__(self, short_name, description, self.path = path self.sdk_based = sdk_based self.default_toolset = default_toolset + compatible_sdks = compatible_sdks or [] + compatible_sdks.sort(key=lambda v: float(v.replace('v', '')), reverse=True) + self.compatible_sdks = compatible_sdks def ShortName(self): return self.short_name @@ -70,43 +77,67 @@ def DefaultToolset(self): of a user override.""" return self.default_toolset - def SetupScript(self, target_arch): + + def _SetupScriptInternal(self, target_arch): """Returns a command (with arguments) to be used to set up the environment.""" - # Check if we are running in the SDK command line environment and use - # the setup script from the SDK if so. |target_arch| should be either - # 'x86' or 'x64'. - assert target_arch in ('x86', 'x64') - sdk_dir = os.environ.get('WindowsSDKDir') - if self.sdk_based and sdk_dir: - return [os.path.normpath(os.path.join(sdk_dir, 'Bin/SetEnv.Cmd')), - '/' + target_arch] - else: - # We don't use VC/vcvarsall.bat for x86 because vcvarsall calls - # vcvars32, which it can only find if VS??COMNTOOLS is set, which it - # isn't always. - if target_arch == 'x86': - if self.short_name >= '2013' and self.short_name[-1] != 'e' and ( - os.environ.get('PROCESSOR_ARCHITECTURE') == 'AMD64' or - os.environ.get('PROCESSOR_ARCHITEW6432') == 'AMD64'): - # VS2013 and later, non-Express have a x64-x86 cross that we want - # to prefer. - return [os.path.normpath( - os.path.join(self.path, 'VC/vcvarsall.bat')), 'amd64_x86'] - # Otherwise, the standard x86 compiler. - return [os.path.normpath( - os.path.join(self.path, 'Common7/Tools/vsvars32.bat'))] + assert target_arch in ('x86', 'x64'), "target_arch not supported" + # If WindowsSDKDir is set and SetEnv.Cmd exists then we are using the + # depot_tools build tools and should run SetEnv.Cmd to set up the + # environment. The check for WindowsSDKDir alone is not sufficient because + # this is set by running vcvarsall.bat. + sdk_dir = os.environ.get('WindowsSDKDir', '') + setup_path = JoinPath(sdk_dir, 'Bin', 'SetEnv.Cmd') + if self.sdk_based and sdk_dir and os.path.exists(setup_path): + return [setup_path, '/' + target_arch] + + is_host_arch_x64 = ( + os.environ.get('PROCESSOR_ARCHITECTURE') == 'AMD64' or + os.environ.get('PROCESSOR_ARCHITEW6432') == 'AMD64' + ) + + # For VS2017 (and newer) it's fairly easy + if self.short_name >= '2017': + script_path = JoinPath(self.path, + 'VC', 'Auxiliary', 'Build', 'vcvarsall.bat') + + # Always use a native executable, cross-compiling if necessary. + host_arch = 'amd64' if is_host_arch_x64 else 'x86' + msvc_target_arch = 'amd64' if target_arch == 'x64' else 'x86' + arg = host_arch + if host_arch != msvc_target_arch: + arg += '_' + msvc_target_arch + + return [script_path, arg] + + # We try to find the best version of the env setup batch. + vcvarsall = JoinPath(self.path, 'VC', 'vcvarsall.bat') + if target_arch == 'x86': + if self.short_name >= '2013' and self.short_name[-1] != 'e' and \ + is_host_arch_x64: + # VS2013 and later, non-Express have a x64-x86 cross that we want + # to prefer. + return [vcvarsall, 'amd64_x86'] else: - assert target_arch == 'x64' - arg = 'x86_amd64' - # Use the 64-on-64 compiler if we're not using an express - # edition and we're running on a 64bit OS. - if self.short_name[-1] != 'e' and ( - os.environ.get('PROCESSOR_ARCHITECTURE') == 'AMD64' or - os.environ.get('PROCESSOR_ARCHITEW6432') == 'AMD64'): - arg = 'amd64' - return [os.path.normpath( - os.path.join(self.path, 'VC/vcvarsall.bat')), arg] + # Otherwise, the standard x86 compiler. We don't use VC/vcvarsall.bat + # for x86 because vcvarsall calls vcvars32, which it can only find if + # VS??COMNTOOLS is set, which isn't guaranteed. + return [JoinPath(self.path, 'Common7', 'Tools', 'vsvars32.bat')] + elif target_arch == 'x64': + arg = 'x86_amd64' + # Use the 64-on-64 compiler if we're not using an express edition and + # we're running on a 64bit OS. + if self.short_name[-1] != 'e' and is_host_arch_x64: + arg = 'amd64' + return [vcvarsall, arg] + + def SetupScript(self, target_arch): + script_data = self._SetupScriptInternal(target_arch) + script_path = script_data[0] + if not os.path.exists(script_path): + raise Exception('%s is missing - make sure VC++ tools are installed.' % + script_path) + return script_data def _RegistryQueryBase(sysdir, key, value): @@ -181,11 +212,11 @@ def _RegistryGetValueUsingWinReg(key, value): ImportError if _winreg is unavailable. """ try: - # Python 2 - from _winreg import HKEY_LOCAL_MACHINE, OpenKey, QueryValueEx + # Python 2 + from _winreg import HKEY_LOCAL_MACHINE, OpenKey, QueryValueEx except ImportError: - # Python 3 - from winreg import HKEY_LOCAL_MACHINE, OpenKey, QueryValueEx + # Python 3 + from winreg import HKEY_LOCAL_MACHINE, OpenKey, QueryValueEx try: root, subkey = key.split('\\', 1) @@ -236,6 +267,26 @@ def _CreateVersion(name, path, sdk_based=False): if path: path = os.path.normpath(path) versions = { + '2019': VisualStudioVersion('2019', + 'Visual Studio 2019', + solution_version='12.00', + project_version='16.0', + flat_sln=False, + uses_vcxproj=True, + path=path, + sdk_based=sdk_based, + default_toolset='v142', + compatible_sdks=['v8.1', 'v10.0']), + '2017': VisualStudioVersion('2017', + 'Visual Studio 2017', + solution_version='12.00', + project_version='15.0', + flat_sln=False, + uses_vcxproj=True, + path=path, + sdk_based=sdk_based, + default_toolset='v141', + compatible_sdks=['v8.1', 'v10.0']), '2015': VisualStudioVersion('2015', 'Visual Studio 2015', solution_version='12.00', @@ -350,7 +401,6 @@ def _DetectVisualStudioVersions(versions_to_check, force_express): A list of visual studio versions installed in descending order of usage preference. Base this on the registry and a quick check if devenv.exe exists. - Only versions 8-10 are considered. Possibilities are: 2005(e) - Visual Studio 2005 (8) 2008(e) - Visual Studio 2008 (9) @@ -358,6 +408,8 @@ def _DetectVisualStudioVersions(versions_to_check, force_express): 2012(e) - Visual Studio 2012 (11) 2013(e) - Visual Studio 2013 (12) 2015 - Visual Studio 2015 (14) + 2017 - Visual Studio 2017 (15) + 2019 - Visual Studio 2019 (16) Where (e) is e for express editions of MSVS and blank otherwise. """ version_to_year = { @@ -367,6 +419,8 @@ def _DetectVisualStudioVersions(versions_to_check, force_express): '11.0': '2012', '12.0': '2013', '14.0': '2015', + '15.0': '2017', + '16.0': '2019', } versions = [] for version in versions_to_check: @@ -397,13 +451,18 @@ def _DetectVisualStudioVersions(versions_to_check, force_express): # The old method above does not work when only SDK is installed. keys = [r'HKLM\Software\Microsoft\VisualStudio\SxS\VC7', - r'HKLM\Software\Wow6432Node\Microsoft\VisualStudio\SxS\VC7'] + r'HKLM\Software\Wow6432Node\Microsoft\VisualStudio\SxS\VC7', + r'HKLM\Software\Microsoft\VisualStudio\SxS\VS7', + r'HKLM\Software\Wow6432Node\Microsoft\VisualStudio\SxS\VS7'] for index in range(len(keys)): path = _RegistryGetValue(keys[index], version) if not path: continue path = _ConvertToCygpath(path) - if version != '14.0': # There is no Express edition for 2015. + if version == '15.0': + if os.path.exists(path): + versions.append(_CreateVersion('2017', path)) + elif version != '14.0': # There is no Express edition for 2015. versions.append(_CreateVersion(version_to_year[version] + 'e', os.path.join(path, '..'), sdk_based=True)) @@ -422,7 +481,7 @@ def SelectVisualStudioVersion(version='auto', allow_fallback=True): if version == 'auto': version = os.environ.get('GYP_MSVS_VERSION', 'auto') version_map = { - 'auto': ('14.0', '12.0', '10.0', '9.0', '8.0', '11.0'), + 'auto': ('16.0', '15.0', '14.0', '12.0', '10.0', '9.0', '8.0', '11.0'), '2005': ('8.0',), '2005e': ('8.0',), '2008': ('9.0',), @@ -434,6 +493,8 @@ def SelectVisualStudioVersion(version='auto', allow_fallback=True): '2013': ('12.0',), '2013e': ('12.0',), '2015': ('14.0',), + '2017': ('15.0',), + '2019': ('16.0',), } override_path = os.environ.get('GYP_MSVS_OVERRIDE_PATH') if override_path: diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py index d866e81d40..aa410e1dfd 100644 --- a/gyp/pylib/gyp/common.py +++ b/gyp/pylib/gyp/common.py @@ -434,7 +434,7 @@ def GetFlavor(params): return flavors[sys.platform] if sys.platform.startswith('sunos'): return 'solaris' - if sys.platform.startswith('freebsd'): + if sys.platform.startswith(('dragonfly', 'freebsd')): return 'freebsd' if sys.platform.startswith('openbsd'): return 'openbsd' @@ -442,15 +442,13 @@ def GetFlavor(params): return 'netbsd' if sys.platform.startswith('aix'): return 'aix' - if sys.platform.startswith('zos'): - return 'zos' - if sys.platform.startswith('os390'): + if sys.platform.startswith(('os390', 'zos')): return 'zos' return 'linux' -def CopyTool(flavor, out_path): +def CopyTool(flavor, out_path, generator_flags={}): """Finds (flock|mac|win)_tool.gyp in the gyp directory and copies it to |out_path|.""" # aix and solaris just need flock emulation. mac and win use more complicated @@ -470,11 +468,18 @@ def CopyTool(flavor, out_path): with open(source_path) as source_file: source = source_file.readlines() + # Set custom header flags. + header = '# Generated by gyp. Do not edit.\n' + mac_toolchain_dir = generator_flags.get('mac_toolchain_dir', None) + if flavor == 'mac' and mac_toolchain_dir: + header += "import os;\nos.environ['DEVELOPER_DIR']='%s'\n" \ + % mac_toolchain_dir + # Add header and write it out. tool_path = os.path.join(out_path, 'gyp-%s-tool' % prefix) with open(tool_path, 'w') as tool_file: tool_file.write( - ''.join([source[0], '# Generated by gyp. Do not edit.\n'] + source[1:])) + ''.join([source[0], header] + source[1:])) # Make file executable. os.chmod(tool_path, 0o755) @@ -635,4 +640,3 @@ def IsCygwin(): return "CYGWIN" in str(stdout) except Exception: return False - diff --git a/gyp/pylib/gyp/easy_xml.py b/gyp/pylib/gyp/easy_xml.py index 1ddd909175..86d0ba6c0c 100644 --- a/gyp/pylib/gyp/easy_xml.py +++ b/gyp/pylib/gyp/easy_xml.py @@ -119,7 +119,7 @@ def WriteXmlIfChanged(content, path, encoding='utf-8', pretty=False, xml_string = xml_string.replace('\n', '\r\n') default_encoding = locale.getdefaultlocale()[1] - if default_encoding.upper() != encoding.upper(): + if default_encoding and default_encoding.upper() != encoding.upper(): xml_string = xml_string.encode(encoding) # Get the old content diff --git a/gyp/pylib/gyp/easy_xml_test.py b/gyp/pylib/gyp/easy_xml_test.py index 2a80b8a456..664b538a58 100755 --- a/gyp/pylib/gyp/easy_xml_test.py +++ b/gyp/pylib/gyp/easy_xml_test.py @@ -8,10 +8,11 @@ import gyp.easy_xml as easy_xml import unittest + try: - from cStringIO import StringIO + from StringIO import StringIO # Python 2 except ImportError: - from io import StringIO + from io import StringIO # Python 3 class TestSequenceFunctions(unittest.TestCase): diff --git a/gyp/pylib/gyp/generator/cmake.py b/gyp/pylib/gyp/generator/cmake.py index 5601a6657e..e966a8f23e 100644 --- a/gyp/pylib/gyp/generator/cmake.py +++ b/gyp/pylib/gyp/generator/cmake.py @@ -36,6 +36,7 @@ import string import subprocess import gyp.common +import gyp.xcode_emulation generator_default_variables = { 'EXECUTABLE_PREFIX': '', @@ -613,8 +614,8 @@ def CreateCMakeTargetName(self, qualified_target): def WriteTarget(namer, qualified_target, target_dicts, build_dir, config_to_use, - options, generator_flags, all_qualified_targets, output): - + options, generator_flags, all_qualified_targets, flavor, + output): # The make generator does this always. # TODO: It would be nice to be able to tell CMake all dependencies. circular_libs = generator_flags.get('circular', True) @@ -638,6 +639,10 @@ def WriteTarget(namer, qualified_target, target_dicts, build_dir, config_to_use, spec = target_dicts.get(qualified_target, {}) config = spec.get('configurations', {}).get(config_to_use, {}) + xcode_settings = None + if flavor == 'mac': + xcode_settings = gyp.xcode_emulation.XcodeSettings(spec) + target_name = spec.get('target_name', '') target_type = spec.get('type', '') target_toolset = spec.get('toolset') @@ -869,7 +874,7 @@ def WriteTarget(namer, qualified_target, target_dicts, build_dir, config_to_use, elif target_type != 'executable': print('ERROR: What output file should be generated?', - 'type', target_type, 'target', target_name) + 'type', target_type, 'target', target_name) product_prefix = spec.get('product_prefix', default_product_prefix) product_name = spec.get('product_name', default_product_name) @@ -909,10 +914,10 @@ def WriteTarget(namer, qualified_target, target_dicts, build_dir, config_to_use, defines = config.get('defines') if defines is not None: SetTargetProperty(output, - cmake_target_name, - 'COMPILE_DEFINITIONS', - defines, - ';') + cmake_target_name, + 'COMPILE_DEFINITIONS', + defines, + ';') # Compile Flags - http://www.cmake.org/Bug/view.php?id=6493 # CMake currently does not have target C and CXX flags. @@ -932,6 +937,13 @@ def WriteTarget(namer, qualified_target, target_dicts, build_dir, config_to_use, cflags = config.get('cflags', []) cflags_c = config.get('cflags_c', []) cflags_cxx = config.get('cflags_cc', []) + if xcode_settings: + cflags = xcode_settings.GetCflags(config_to_use) + cflags_c = xcode_settings.GetCflagsC(config_to_use) + cflags_cxx = xcode_settings.GetCflagsCC(config_to_use) + #cflags_objc = xcode_settings.GetCflagsObjC(config_to_use) + #cflags_objcc = xcode_settings.GetCflagsObjCC(config_to_use) + if (not cflags_c or not c_sources) and (not cflags_cxx or not cxx_sources): SetTargetProperty(output, cmake_target_name, 'COMPILE_FLAGS', cflags, ' ') @@ -970,6 +982,13 @@ def WriteTarget(namer, qualified_target, target_dicts, build_dir, config_to_use, if ldflags is not None: SetTargetProperty(output, cmake_target_name, 'LINK_FLAGS', ldflags, ' ') + # XCode settings + xcode_settings = config.get('xcode_settings', {}) + for xcode_setting, xcode_value in xcode_settings.viewitems(): + SetTargetProperty(output, cmake_target_name, + "XCODE_ATTRIBUTE_%s" % xcode_setting, xcode_value, + '' if isinstance(xcode_value, str) else ' ') + # Note on Dependencies and Libraries: # CMake wants to handle link order, resolving the link line up front. # Gyp does not retain or enforce specifying enough information to do so. @@ -1034,7 +1053,7 @@ def WriteTarget(namer, qualified_target, target_dicts, build_dir, config_to_use, output.write(cmake_target_name) output.write('\n') if static_deps: - write_group = circular_libs and len(static_deps) > 1 + write_group = circular_libs and len(static_deps) > 1 and flavor != 'mac' if write_group: output.write('-Wl,--start-group\n') for dep in gyp.common.uniquer(static_deps): @@ -1050,9 +1069,9 @@ def WriteTarget(namer, qualified_target, target_dicts, build_dir, config_to_use, output.write('\n') if external_libs: for lib in gyp.common.uniquer(external_libs): - output.write(' ') - output.write(lib) - output.write('\n') + output.write(' "') + output.write(RemovePrefix(lib, "$(SDKROOT)")) + output.write('"\n') output.write(')\n') @@ -1064,6 +1083,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_to_use): options = params['options'] generator_flags = params['generator_flags'] + flavor = gyp.common.GetFlavor(params) # generator_dir: relative path from pwd to where make puts build files. # Makes migrating from make to cmake easier, cmake doesn't put anything here. @@ -1146,7 +1166,9 @@ def GenerateOutputForConfig(target_list, target_dicts, data, # Force ninja to use rsp files. Otherwise link and ar lines can get too long, # resulting in 'Argument list too long' errors. - output.write('set(CMAKE_NINJA_FORCE_RESPONSE_FILE 1)\n') + # However, rsp files don't work correctly on Mac. + if flavor != 'mac': + output.write('set(CMAKE_NINJA_FORCE_RESPONSE_FILE 1)\n') output.write('\n') namer = CMakeNamer(target_list) @@ -1161,8 +1183,13 @@ def GenerateOutputForConfig(target_list, target_dicts, data, all_qualified_targets.add(qualified_target) for qualified_target in target_list: + if flavor == 'mac': + gyp_file, _, _ = gyp.common.ParseQualifiedTarget(qualified_target) + spec = target_dicts[qualified_target] + gyp.xcode_emulation.MergeGlobalXcodeSettingsToSpec(data[gyp_file], spec) + WriteTarget(namer, qualified_target, target_dicts, build_dir, config_to_use, - options, generator_flags, all_qualified_targets, output) + options, generator_flags, all_qualified_targets, flavor, output) output.close() diff --git a/gyp/pylib/gyp/generator/eclipse.py b/gyp/pylib/gyp/generator/eclipse.py index 91f187d685..80e5fb6302 100644 --- a/gyp/pylib/gyp/generator/eclipse.py +++ b/gyp/pylib/gyp/generator/eclipse.py @@ -199,8 +199,8 @@ def GetAllDefines(target_list, target_dicts, data, config_name, params, """Calculate the defines for a project. Returns: - A dict that includes explicit defines declared in gyp files along with all of - the default defines that the compiler uses. + A dict that includes explicit defines declared in gyp files along with all + of the default defines that the compiler uses. """ # Get defines declared in the gyp files. diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index 1960536794..26cf88cccf 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -149,7 +149,7 @@ def CalculateGeneratorInputInfo(params): # special "figure out circular dependencies" flags around the entire # input list during linking. quiet_cmd_link = LINK($(TOOLSET)) $@ -cmd_link = $(LINK.$(TOOLSET)) $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ -Wl,--start-group $(LD_INPUTS) $(LIBS) -Wl,--end-group +cmd_link = $(LINK.$(TOOLSET)) -o $@ $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -Wl,--start-group $(LD_INPUTS) $(LIBS) -Wl,--end-group # We support two kinds of shared objects (.so): # 1) shared_library, which is just bundling together many dependent libraries @@ -168,10 +168,10 @@ def CalculateGeneratorInputInfo(params): # - Set SONAME to the library filename so our binaries don't reference # the local, absolute paths used on the link command-line. quiet_cmd_solink = SOLINK($(TOOLSET)) $@ -cmd_solink = $(LINK.$(TOOLSET)) -shared $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -Wl,-soname=$(@F) -o $@ -Wl,--whole-archive $(LD_INPUTS) -Wl,--no-whole-archive $(LIBS) +cmd_solink = $(LINK.$(TOOLSET)) -o $@ -shared $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -Wl,-soname=$(@F) -Wl,--whole-archive $(LD_INPUTS) -Wl,--no-whole-archive $(LIBS) quiet_cmd_solink_module = SOLINK_MODULE($(TOOLSET)) $@ -cmd_solink_module = $(LINK.$(TOOLSET)) -shared $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -Wl,-soname=$(@F) -o $@ -Wl,--start-group $(filter-out FORCE_DO_CMD, $^) -Wl,--end-group $(LIBS) +cmd_solink_module = $(LINK.$(TOOLSET)) -o $@ -shared $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -Wl,-soname=$(@F) -Wl,--start-group $(filter-out FORCE_DO_CMD, $^) -Wl,--end-group $(LIBS) """ LINK_COMMANDS_MAC = """\ @@ -381,17 +381,17 @@ def CalculateGeneratorInputInfo(params): # - quiet_cmd_foo is the brief-output summary of the command. quiet_cmd_cc = CC($(TOOLSET)) $@ -cmd_cc = $(CC.$(TOOLSET)) $(GYP_CFLAGS) $(DEPFLAGS) $(CFLAGS.$(TOOLSET)) -c -o $@ $< +cmd_cc = $(CC.$(TOOLSET)) -o $@ $< $(GYP_CFLAGS) $(DEPFLAGS) $(CFLAGS.$(TOOLSET)) -c quiet_cmd_cxx = CXX($(TOOLSET)) $@ -cmd_cxx = $(CXX.$(TOOLSET)) $(GYP_CXXFLAGS) $(DEPFLAGS) $(CXXFLAGS.$(TOOLSET)) -c -o $@ $< +cmd_cxx = $(CXX.$(TOOLSET)) -o $@ $< $(GYP_CXXFLAGS) $(DEPFLAGS) $(CXXFLAGS.$(TOOLSET)) -c %(extra_commands)s quiet_cmd_touch = TOUCH $@ cmd_touch = touch $@ quiet_cmd_copy = COPY $@ # send stderr to /dev/null to ignore messages when linking directories. -cmd_copy = rm -rf "$@" && cp %(copy_archive_args)s "$<" "$@" +cmd_copy = ln -f "$<" "$@" 2>/dev/null || (rm -rf "$@" && cp %(copy_archive_args)s "$<" "$@") %(link_commands)s """ @@ -950,7 +950,7 @@ def WriteActions(self, actions, extra_sources, extra_outputs, '%s%s' % (name, cd_action, command)) self.WriteLn() - outputs = [self.Absolutify(output) for output in outputs] + outputs = [self.Absolutify(o) for o in outputs] # The makefile rules are all relative to the top dir, but the gyp actions # are defined relative to their containing dir. This replaces the obj # variable for the action rule with an absolute version so that the output @@ -1040,7 +1040,7 @@ def WriteRules(self, rules, extra_sources, extra_outputs, outputs = [gyp.xcode_emulation.ExpandEnvVars(o, env) for o in outputs] inputs = [gyp.xcode_emulation.ExpandEnvVars(i, env) for i in inputs] - outputs = [self.Absolutify(output) for output in outputs] + outputs = [self.Absolutify(o) for o in outputs] all_outputs += outputs # Only write the 'obj' and 'builddir' rules for the "primary" output # (:1); it's superfluous for the "extra outputs", and this avoids @@ -1383,7 +1383,7 @@ def ComputeOutputBasename(self, spec): target = '%s.stamp' % target elif self.type != 'executable': print("ERROR: What output file should be generated?", - "type", self.type, "target", target) + "type", self.type, "target", target) target_prefix = spec.get('product_prefix', target_prefix) target = spec.get('product_name', target) @@ -1951,13 +1951,11 @@ def _InstallableTargetInstallPath(self): """Returns the location of the final output for an installable target.""" # Xcode puts shared_library results into PRODUCT_DIR, and some gyp files # rely on this. Emulate this behavior for mac. - - # XXX(TooTallNate): disabling this code since we don't want this behavior... - #if (self.type == 'shared_library' and - # (self.flavor != 'mac' or self.toolset != 'target')): - # # Install all shared libs into a common directory (per toolset) for - # # convenient access with LD_LIBRARY_PATH. - # return '$(builddir)/lib.%s/%s' % (self.toolset, self.alias) + if (self.type == 'shared_library' and + (self.flavor != 'mac' or self.toolset != 'target')): + # Install all shared libs into a common directory (per toolset) for + # convenient access with LD_LIBRARY_PATH. + return '$(builddir)/lib.%s/%s' % (self.toolset, self.alias) return '$(builddir)/' + self.alias diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index 8dbe0dc05b..933042c711 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -42,6 +42,8 @@ generator_default_variables = { + 'DRIVER_PREFIX': '', + 'DRIVER_SUFFIX': '.sys', 'EXECUTABLE_PREFIX': '', 'EXECUTABLE_SUFFIX': '.exe', 'STATIC_LIB_PREFIX': '', @@ -88,6 +90,7 @@ 'msvs_target_platform_minversion', ] +generator_filelist_paths = None # List of precompiled header related keys. precomp_keys = [ @@ -171,10 +174,9 @@ def _FixPath(path): def _IsWindowsAbsPath(path): - r""" + """ On Cygwin systems Python needs a little help determining if a path is an absolute Windows path or not, so that it does not treat those as relative, which results in bad paths like: - '..\C:\\some_source_code_file.cc' """ return path.startswith('c:') or path.startswith('C:') @@ -265,6 +267,8 @@ def _ToolSetOrAppend(tools, tool_name, setting, value, only_if_unset=False): if not tools.get(tool_name): tools[tool_name] = dict() tool = tools[tool_name] + if 'CompileAsWinRT' == setting: + return if tool.get(setting): if only_if_unset: return if type(tool[setting]) == list and type(value) == list: @@ -278,6 +282,10 @@ def _ToolSetOrAppend(tools, tool_name, setting, value, only_if_unset=False): tool[setting] = value +def _ConfigTargetVersion(config_data): + return config_data.get('msvs_target_version', 'Windows7') + + def _ConfigPlatform(config_data): return config_data.get('msvs_configuration_platform', 'Win32') @@ -294,20 +302,32 @@ def _ConfigFullName(config_name, config_data): return '%s|%s' % (_ConfigBaseName(config_name, platform_name), platform_name) -def _ConfigWindowsTargetPlatformVersion(config_data): - ver = config_data.get('msvs_windows_target_platform_version') - if not ver or re.match(r'^\d+', ver): - return ver - for key in [r'HKLM\Software\Microsoft\Microsoft SDKs\Windows\%s', - r'HKLM\Software\Wow6432Node\Microsoft\Microsoft SDKs\Windows\%s']: - sdkdir = MSVSVersion._RegistryGetValue(key % ver, 'InstallationFolder') - if not sdkdir: - continue - version = MSVSVersion._RegistryGetValue(key % ver, 'ProductVersion') or '' - # find a matching entry in sdkdir\include - names = sorted([x for x in os.listdir(r'%s\include' % sdkdir) \ - if x.startswith(version)], reverse = True) - return names[0] +def _ConfigWindowsTargetPlatformVersion(config_data, version): + target_ver = config_data.get('msvs_windows_target_platform_version') + if target_ver and re.match(r'^\d+', target_ver): + return target_ver + config_ver = config_data.get('msvs_windows_sdk_version') + vers = [config_ver] if config_ver else version.compatible_sdks + for ver in vers: + for key in [ + r'HKLM\Software\Microsoft\Microsoft SDKs\Windows\%s', + r'HKLM\Software\Wow6432Node\Microsoft\Microsoft SDKs\Windows\%s']: + sdk_dir = MSVSVersion._RegistryGetValue(key % ver, 'InstallationFolder') + if not sdk_dir: + continue + version = MSVSVersion._RegistryGetValue(key % ver, 'ProductVersion') or '' + # Find a matching entry in sdk_dir\include. + expected_sdk_dir=r'%s\include' % sdk_dir + names = sorted([x for x in (os.listdir(expected_sdk_dir) + if os.path.isdir(expected_sdk_dir) + else [] + ) + if x.startswith(version)], reverse=True) + if names: + return names[0] + else: + print('Warning: No include files found for detected ' + 'Windows SDK version %s' % (version), file=sys.stdout) def _BuildCommandLineForRuleRaw(spec, cmd, cygwin_shell, has_input_path, @@ -926,6 +946,8 @@ def _GetMsbuildToolsetOfProject(proj_path, spec, version): toolset = default_config.get('msbuild_toolset') if not toolset and version.DefaultToolset(): toolset = version.DefaultToolset() + if spec['type'] == 'windows_driver': + toolset = 'WindowsKernelModeDriver10.0' return toolset @@ -1109,6 +1131,7 @@ def _GetMSVSConfigurationType(spec, build_file): 'shared_library': '2', # .dll 'loadable_module': '2', # .dll 'static_library': '4', # .lib + 'windows_driver': '5', # .sys 'none': '10', # Utility type }[spec['type']] except KeyError: @@ -1293,6 +1316,7 @@ def _GetOutputFilePathAndTool(spec, msbuild): 'executable': ('VCLinkerTool', 'Link', '$(OutDir)', '.exe'), 'shared_library': ('VCLinkerTool', 'Link', '$(OutDir)', '.dll'), 'loadable_module': ('VCLinkerTool', 'Link', '$(OutDir)', '.dll'), + 'windows_driver': ('VCLinkerTool', 'Link', '$(OutDir)', '.sys'), 'static_library': ('VCLibrarianTool', 'Lib', '$(OutDir)lib\\', '.lib'), } output_file_props = output_file_map.get(spec['type']) @@ -1355,7 +1379,8 @@ def _GetDisabledWarnings(config): def _GetModuleDefinition(spec): def_file = '' - if spec['type'] in ['shared_library', 'loadable_module', 'executable']: + if spec['type'] in ['shared_library', 'loadable_module', 'executable', + 'windows_driver']: def_files = [s for s in spec.get('sources', []) if s.endswith('.def')] if len(def_files) == 1: def_file = _FixPath(def_files[0]) @@ -1711,14 +1736,17 @@ def _GetCopies(spec): src_bare = src[:-1] base_dir = posixpath.split(src_bare)[0] outer_dir = posixpath.split(src_bare)[1] - cmd = 'cd "%s" && xcopy /e /f /y "%s" "%s\\%s\\"' % ( - _FixPath(base_dir), outer_dir, _FixPath(dst), outer_dir) + fixed_dst = _FixPath(dst) + full_dst = '"%s\\%s\\"' % (fixed_dst, outer_dir) + cmd = 'mkdir %s 2>nul & cd "%s" && xcopy /e /f /y "%s" %s' % ( + full_dst, _FixPath(base_dir), outer_dir, full_dst) copies.append(([src], ['dummy_copies', dst], cmd, - 'Copying %s to %s' % (src, dst))) + 'Copying %s to %s' % (src, fixed_dst))) else: + fix_dst = _FixPath(cpy['destination']) cmd = 'mkdir "%s" 2>nul & set ERRORLEVEL=0 & copy /Y "%s" "%s"' % ( - _FixPath(cpy['destination']), _FixPath(src), _FixPath(dst)) - copies.append(([src], [dst], cmd, 'Copying %s to %s' % (src, dst))) + fix_dst, _FixPath(src), _FixPath(dst)) + copies.append(([src], [dst], cmd, 'Copying %s to %s' % (src, fix_dst))) return copies @@ -1964,6 +1992,19 @@ def PerformBuild(data, configurations, params): rtn = subprocess.check_call(arguments) +def CalculateGeneratorInputInfo(params): + if params.get('flavor') == 'ninja': + toplevel = params['options'].toplevel_dir + qualified_out_dir = os.path.normpath(os.path.join( + toplevel, ninja_generator.ComputeOutputDir(params), + 'gypfiles-msvs-ninja')) + + global generator_filelist_paths + generator_filelist_paths = { + 'toplevel': toplevel, + 'qualified_out_dir': qualified_out_dir, + } + def GenerateOutput(target_list, target_dicts, data, params): """Generate .sln and .vcproj files. @@ -2129,6 +2170,7 @@ def _MapFileToMsBuildSourceType(source, rule_dependencies, A pair of (group this file should be part of, the label of element) """ _, ext = os.path.splitext(source) + ext = ext.lower() if ext in extension_to_rule_name: group = 'rule' element = extension_to_rule_name[ext] @@ -2141,12 +2183,12 @@ def _MapFileToMsBuildSourceType(source, rule_dependencies, elif ext == '.rc': group = 'resource' element = 'ResourceCompile' - elif ext == '.asm': + elif ext in ['.s', '.asm']: group = 'masm' element = 'MASM' for platform in platforms: if platform.lower() in ['arm', 'arm64']: - element = 'MARMASM' + element = 'MARMASM' elif ext == '.idl': group = 'midl' element = 'Midl' @@ -2655,7 +2697,7 @@ def _GetMSBuildProjectConfigurations(configurations): return [group] -def _GetMSBuildGlobalProperties(spec, guid, gyp_file_name): +def _GetMSBuildGlobalProperties(spec, version, guid, gyp_file_name): namespace = os.path.splitext(gyp_file_name)[0] properties = [ ['PropertyGroup', {'Label': 'Globals'}, @@ -2670,6 +2712,18 @@ def _GetMSBuildGlobalProperties(spec, guid, gyp_file_name): os.environ.get('PROCESSOR_ARCHITEW6432') == 'AMD64': properties[0].append(['PreferredToolArchitecture', 'x64']) + if spec.get('msvs_target_platform_version'): + target_platform_version = spec.get('msvs_target_platform_version') + properties[0].append(['WindowsTargetPlatformVersion', + target_platform_version]) + if spec.get('msvs_target_platform_minversion'): + target_platform_minversion = spec.get('msvs_target_platform_minversion') + properties[0].append(['WindowsTargetPlatformMinVersion', + target_platform_minversion]) + else: + properties[0].append(['WindowsTargetPlatformMinVersion', + target_platform_version]) + if spec.get('msvs_enable_winrt'): properties[0].append(['DefaultLanguage', 'en-US']) properties[0].append(['AppContainerApplication', 'true']) @@ -2678,49 +2732,45 @@ def _GetMSBuildGlobalProperties(spec, guid, gyp_file_name): properties[0].append(['ApplicationTypeRevision', app_type_revision]) else: properties[0].append(['ApplicationTypeRevision', '8.1']) - - if spec.get('msvs_target_platform_version'): - target_platform_version = spec.get('msvs_target_platform_version') - properties[0].append(['WindowsTargetPlatformVersion', - target_platform_version]) - if spec.get('msvs_target_platform_minversion'): - target_platform_minversion = spec.get('msvs_target_platform_minversion') - properties[0].append(['WindowsTargetPlatformMinVersion', - target_platform_minversion]) - else: - properties[0].append(['WindowsTargetPlatformMinVersion', - target_platform_version]) if spec.get('msvs_enable_winphone'): properties[0].append(['ApplicationType', 'Windows Phone']) else: properties[0].append(['ApplicationType', 'Windows Store']) platform_name = None - msvs_windows_target_platform_version = None + msvs_windows_sdk_version = None for configuration in spec['configurations'].values(): platform_name = platform_name or _ConfigPlatform(configuration) - msvs_windows_target_platform_version = \ - msvs_windows_target_platform_version or \ - _ConfigWindowsTargetPlatformVersion(configuration) - if platform_name and msvs_windows_target_platform_version: + msvs_windows_sdk_version = (msvs_windows_sdk_version or + _ConfigWindowsTargetPlatformVersion(configuration, version)) + if platform_name and msvs_windows_sdk_version: break + if msvs_windows_sdk_version: + properties[0].append(['WindowsTargetPlatformVersion', + str(msvs_windows_sdk_version)]) + elif version.compatible_sdks: + raise GypError('%s requires any SDK of %s version, but none were found' % + (version.description, version.compatible_sdks)) if platform_name == 'ARM': properties[0].append(['WindowsSDKDesktopARMSupport', 'true']) - if msvs_windows_target_platform_version: - properties[0].append(['WindowsTargetPlatformVersion', \ - str(msvs_windows_target_platform_version)]) return properties + def _GetMSBuildConfigurationDetails(spec, build_file): properties = {} for name, settings in spec['configurations'].items(): msbuild_attributes = _GetMSBuildAttributes(spec, settings, build_file) condition = _GetConfigurationCondition(name, settings) character_set = msbuild_attributes.get('CharacterSet') + config_type = msbuild_attributes.get('ConfigurationType') _AddConditionalProperty(properties, condition, 'ConfigurationType', - msbuild_attributes['ConfigurationType']) + config_type) + if config_type == 'Driver': + _AddConditionalProperty(properties, condition, 'DriverType', 'WDM') + _AddConditionalProperty(properties, condition, 'TargetVersion', + _ConfigTargetVersion(settings)) if character_set: if 'msvs_enable_winrt' not in spec : _AddConditionalProperty(properties, condition, 'CharacterSet', @@ -2819,6 +2869,7 @@ def _ConvertMSVSConfigurationType(config_type): '1': 'Application', '2': 'DynamicLibrary', '4': 'StaticLibrary', + '5': 'Driver', '10': 'Utility' }[config_type] return config_type @@ -2861,6 +2912,7 @@ def _GetMSBuildAttributes(spec, config, build_file): 'executable': 'Link', 'shared_library': 'Link', 'loadable_module': 'Link', + 'windows_driver': 'Link', 'static_library': 'Lib', } msbuild_tool = msbuild_tool_map.get(spec['type']) @@ -3045,7 +3097,7 @@ def _FinalizeMSBuildSettings(spec, configuration): value = configuration.get(ignored_setting) if value: print('Warning: The automatic conversion to MSBuild does not handle ' - '%s. Ignoring setting of %s' % (ignored_setting, str(value))) + '%s. Ignoring setting of %s' % (ignored_setting, str(value))) defines = [_EscapeCppDefineForMSBuild(d) for d in defines] disabled_warnings = _GetDisabledWarnings(configuration) @@ -3360,7 +3412,8 @@ def _GenerateMSBuildProject(project, options, version, generator_flags): }] content += _GetMSBuildProjectConfigurations(configurations) - content += _GetMSBuildGlobalProperties(spec, project.guid, project_file_name) + content += _GetMSBuildGlobalProperties(spec, version, project.guid, + project_file_name) content += import_default_section content += _GetMSBuildConfigurationDetails(spec, project.build_file) if spec.get('msvs_enable_winphone'): diff --git a/gyp/pylib/gyp/generator/msvs_test.py b/gyp/pylib/gyp/generator/msvs_test.py index daf4f411bc..1b0cdd1720 100755 --- a/gyp/pylib/gyp/generator/msvs_test.py +++ b/gyp/pylib/gyp/generator/msvs_test.py @@ -7,10 +7,11 @@ import gyp.generator.msvs as msvs import unittest + try: - from cStringIO import StringIO + from StringIO import StringIO # Python 2 except ImportError: - from io import StringIO + from io import StringIO # Python 3 class TestSequenceFunctions(unittest.TestCase): diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py index 33cc253aba..d5006bf84a 100644 --- a/gyp/pylib/gyp/generator/ninja.py +++ b/gyp/pylib/gyp/generator/ninja.py @@ -1,8 +1,9 @@ -from __future__ import print_function # Copyright (c) 2013 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. +from __future__ import print_function + import collections import copy import hashlib @@ -152,6 +153,9 @@ def __init__(self, type): # because dependents only link against the lib (not both the lib and the # dll) we keep track of the import library here. self.import_lib = None + # Track if this target contains any C++ files, to decide if gcc or g++ + # should be used for linking. + self.uses_cpp = False def Linkable(self): """Return true if this is a target that can be linked against.""" @@ -351,7 +355,7 @@ def WriteCollapsedDependencies(self, name, targets, order_only=None): Uses a stamp file if necessary.""" - assert targets == filter(None, targets), targets + assert targets == [item for item in targets if item], targets if len(targets) == 0: assert not order_only return None @@ -379,14 +383,17 @@ def WriteSpec(self, spec, config_name, generator_flags): self.target = Target(spec['type']) self.is_standalone_static_library = bool( spec.get('standalone_static_library', 0)) - # Track if this target contains any C++ files, to decide if gcc or g++ - # should be used for linking. - self.uses_cpp = False + + self.target_rpath = generator_flags.get('target_rpath', r'\$$ORIGIN/lib/') self.is_mac_bundle = gyp.xcode_emulation.IsMacBundle(self.flavor, spec) self.xcode_settings = self.msvs_settings = None if self.flavor == 'mac': self.xcode_settings = gyp.xcode_emulation.XcodeSettings(spec) + mac_toolchain_dir = generator_flags.get('mac_toolchain_dir', None) + if mac_toolchain_dir: + self.xcode_settings.mac_toolchain_dir = mac_toolchain_dir + if self.flavor == 'win': self.msvs_settings = gyp.msvs_emulation.MsvsSettings(spec, generator_flags) @@ -423,8 +430,10 @@ def WriteSpec(self, spec, config_name, generator_flags): target = self.target_outputs[dep] actions_depends.append(target.PreActionInput(self.flavor)) compile_depends.append(target.PreCompileInput()) - actions_depends = filter(None, actions_depends) - compile_depends = filter(None, compile_depends) + if target.uses_cpp: + self.target.uses_cpp = True + actions_depends = [item for item in actions_depends if item] + compile_depends = [item for item in compile_depends if item] actions_depends = self.WriteCollapsedDependencies('actions_depends', actions_depends) compile_depends = self.WriteCollapsedDependencies('compile_depends', @@ -448,7 +457,12 @@ def WriteSpec(self, spec, config_name, generator_flags): # Write out the compilation steps, if any. link_deps = [] - sources = extra_sources + spec.get('sources', []) + try: + sources = extra_sources + spec.get('sources', []) + except TypeError: + print('extra_sources: ', str(extra_sources)) + print('spec.get("sources"): ', str(spec.get('sources'))) + raise if sources: if self.flavor == 'mac' and len(self.archs) > 1: # Write subninja file containing compile and link commands scoped to @@ -476,7 +490,7 @@ def WriteSpec(self, spec, config_name, generator_flags): if self.flavor != 'mac' or len(self.archs) == 1: link_deps += [self.GypPathToNinja(o) for o in obj_outputs] else: - print("Warning: Actions/rules writing object files don't work with " \ + print("Warning: Actions/rules writing object files don't work with " "multiarch targets, dropping. (target %s)" % spec['target_name']) elif self.flavor == 'mac' and len(self.archs) > 1: link_deps = collections.defaultdict(list) @@ -563,6 +577,9 @@ def WriteActionsRulesCopies(self, spec, extra_sources, prebuild, if 'sources' in spec and self.flavor == 'win': outputs += self.WriteWinIdlFiles(spec, prebuild) + if self.xcode_settings and self.xcode_settings.IsIosFramework(): + self.WriteiOSFrameworkHeaders(spec, outputs, prebuild) + stamp = self.WriteCollapsedDependencies('actions_rules_copies', outputs) if self.is_mac_bundle: @@ -660,6 +677,7 @@ def WriteRules(self, rules, extra_sources, prebuild, for var in special_locals: if '${%s}' % var in argument: needed_variables.add(var) + needed_variables = sorted(needed_variables) def cygwin_munge(path): # pylint: disable=cell-var-from-loop @@ -733,6 +751,7 @@ def cygwin_munge(path): # WriteNewNinjaRule uses unique_name for creating an rsp file on win. extra_bindings.append(('unique_name', hashlib.md5(outputs[0]).hexdigest())) + self.ninja.build(outputs, rule_name, self.GypPathToNinja(source), implicit=inputs, order_only=prebuild, @@ -744,7 +763,11 @@ def cygwin_munge(path): def WriteCopies(self, copies, prebuild, mac_bundle_depends): outputs = [] - env = self.GetToolchainEnv() + if self.xcode_settings: + extra_env = self.xcode_settings.GetPerTargetSettings() + env = self.GetToolchainEnv(additional_settings=extra_env) + else: + env = self.GetToolchainEnv() for copy in copies: for path in copy['files']: # Normalize the path so trailing slashes don't confuse us. @@ -766,18 +789,38 @@ def WriteCopies(self, copies, prebuild, mac_bundle_depends): return outputs + def WriteiOSFrameworkHeaders(self, spec, outputs, prebuild): + """Prebuild steps to generate hmap files and copy headers to destination.""" + framework = self.ComputeMacBundleOutput() + all_sources = spec['sources'] + copy_headers = spec['mac_framework_headers'] + output = self.GypPathToUniqueOutput('headers.hmap') + self.xcode_settings.header_map_path = output + all_headers = map(self.GypPathToNinja, + filter(lambda x:x.endswith(('.h')), all_sources)) + variables = [('framework', framework), + ('copy_headers', map(self.GypPathToNinja, copy_headers))] + outputs.extend(self.ninja.build( + output, 'compile_ios_framework_headers', all_headers, + variables=variables, order_only=prebuild)) + def WriteMacBundleResources(self, resources, bundle_depends): """Writes ninja edges for 'mac_bundle_resources'.""" xcassets = [] + + extra_env = self.xcode_settings.GetPerTargetSettings() + env = self.GetSortedXcodeEnv(additional_settings=extra_env) + env = self.ComputeExportEnvString(env) + isBinary = self.xcode_settings.IsBinaryOutputFormat(self.config_name) + for output, res in gyp.xcode_emulation.GetMacBundleResources( generator_default_variables['PRODUCT_DIR'], self.xcode_settings, map(self.GypPathToNinja, resources)): output = self.ExpandSpecial(output) if os.path.splitext(output)[-1] != '.xcassets': - isBinary = self.xcode_settings.IsBinaryOutputFormat(self.config_name) self.ninja.build(output, 'mac_tool', res, variables=[('mactool_cmd', 'copy-bundle-resource'), \ - ('binary', isBinary)]) + ('env', env), ('binary', isBinary)]) bundle_depends.append(output) else: xcassets.append(res) @@ -996,7 +1039,7 @@ def WriteSourcesForArch(self, ninja_file, config_name, config, sources, obj_ext = self.obj_ext if ext in ('cc', 'cpp', 'cxx'): command = 'cxx' - self.uses_cpp = True + self.target.uses_cpp = True elif ext == 'c' or (ext == 'S' and self.flavor != 'win'): command = 'cc' elif ext == 's' and self.flavor != 'win': # Doesn't generate .o.d files. @@ -1011,7 +1054,7 @@ def WriteSourcesForArch(self, ninja_file, config_name, config, sources, command = 'objc' elif self.flavor == 'mac' and ext == 'mm': command = 'objcxx' - self.uses_cpp = True + self.target.uses_cpp = True elif self.flavor == 'win' and ext == 'rc': command = 'rc' obj_ext = '.res' @@ -1062,16 +1105,16 @@ def WritePchTargets(self, ninja_file, pch_commands): cmd = map.get(lang) ninja_file.build(gch, cmd, input, variables=[(var_name, lang_flag)]) - def WriteLink(self, spec, config_name, config, link_deps): + def WriteLink(self, spec, config_name, config, link_deps, compile_deps): """Write out a link step. Fills out target.binary. """ if self.flavor != 'mac' or len(self.archs) == 1: return self.WriteLinkForArch( - self.ninja, spec, config_name, config, link_deps) + self.ninja, spec, config_name, config, link_deps, compile_deps) else: output = self.ComputeOutput(spec) inputs = [self.WriteLinkForArch(self.arch_subninjas[arch], spec, config_name, config, link_deps[arch], - arch=arch) + compile_deps, arch=arch) for arch in self.archs] extra_bindings = [] build_output = output @@ -1090,7 +1133,7 @@ def WriteLink(self, spec, config_name, config, link_deps): return output def WriteLinkForArch(self, ninja_file, spec, config_name, config, - link_deps, arch=None): + link_deps, compile_deps, arch=None): """Write out a link step. Fills out target.binary. """ command = { 'executable': 'link', @@ -1103,6 +1146,14 @@ def WriteLinkForArch(self, ninja_file, spec, config_name, config, solibs = set() order_deps = set() + if compile_deps: + # Normally, the compiles of the target already depend on compile_deps, + # but a shared_library target might have no sources and only link together + # a few static_library deps, so the link step also needs to depend + # on compile_deps to make sure actions in the shared_library target + # get run before the link. + order_deps.add(compile_deps) + if 'dependencies' in spec: # Two kinds of dependencies: # - Linkable dependencies (like a .a or a .so): add them to the link line. @@ -1139,7 +1190,7 @@ def WriteLinkForArch(self, ninja_file, spec, config_name, config, implicit_deps.add(final_output) extra_bindings = [] - if self.uses_cpp and self.flavor != 'win': + if self.target.uses_cpp and self.flavor != 'win': extra_bindings.append(('ld', '$ldxx')) output = self.ComputeOutput(spec, arch) @@ -1182,7 +1233,9 @@ def WriteLinkForArch(self, ninja_file, spec, config_name, config, rpath = 'lib/' if self.toolset != 'target': rpath += self.toolset - ldflags.append(r'-Wl,-rpath=\$$ORIGIN/%s' % rpath) + ldflags.append(r'-Wl,-rpath=\$$ORIGIN/%s' % rpath) + else: + ldflags.append('-Wl,-rpath=%s' % self.target_rpath) ldflags.append('-Wl,-rpath-link=%s' % rpath) self.WriteVariableList(ninja_file, 'ldflags', map(self.ExpandSpecial, ldflags)) @@ -1256,10 +1309,11 @@ def WriteLinkForArch(self, ninja_file, spec, config_name, config, if len(solibs): - extra_bindings.append(('solibs', gyp.common.EncodePOSIXShellList(solibs))) + extra_bindings.append(('solibs', + gyp.common.EncodePOSIXShellList(sorted(solibs)))) ninja_file.build(output, command + command_suffix, link_deps, - implicit=list(implicit_deps), + implicit=sorted(implicit_deps), order_only=list(order_deps), variables=extra_bindings) return linked_binary @@ -1312,7 +1366,8 @@ def WriteTarget(self, spec, config_name, config, link_deps, compile_deps): # needed. variables=variables) else: - self.target.binary = self.WriteLink(spec, config_name, config, link_deps) + self.target.binary = self.WriteLink(spec, config_name, config, link_deps, + compile_deps) return self.target.binary def WriteMacBundle(self, spec, mac_bundle_depends, is_empty): @@ -1325,9 +1380,13 @@ def WriteMacBundle(self, spec, mac_bundle_depends, is_empty): self.AppendPostbuildVariable(variables, spec, output, self.target.binary, is_command_start=not package_framework) if package_framework and not is_empty: - variables.append(('version', self.xcode_settings.GetFrameworkVersion())) - self.ninja.build(output, 'package_framework', mac_bundle_depends, - variables=variables) + if spec['type'] == 'shared_library' and self.xcode_settings.isIOS: + self.ninja.build(output, 'package_ios_framework', mac_bundle_depends, + variables=variables) + else: + variables.append(('version', self.xcode_settings.GetFrameworkVersion())) + self.ninja.build(output, 'package_framework', mac_bundle_depends, + variables=variables) else: self.ninja.build(output, 'stamp', mac_bundle_depends, variables=variables) @@ -1814,7 +1873,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, master_ninja = ninja_syntax.Writer(master_ninja_file, width=120) # Put build-time support tools in out/{config_name}. - gyp.common.CopyTool(flavor, toplevel_build) + gyp.common.CopyTool(flavor, toplevel_build, generator_flags) # Grab make settings for CC/CXX. # The rules are @@ -1840,7 +1899,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, ld_host = '$cc_host' ldxx_host = '$cxx_host' - ar_host = 'ar' + ar_host = ar cc_host = None cxx_host = None cc_host_global_setting = None @@ -1899,6 +1958,10 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, key_prefix = re.sub(r'\.HOST$', '.host', key_prefix) wrappers[key_prefix] = os.path.join(build_to_root, value) + mac_toolchain_dir = generator_flags.get('mac_toolchain_dir', None) + if mac_toolchain_dir: + wrappers['LINK'] = "export DEVELOPER_DIR='%s' &&" % mac_toolchain_dir + if flavor == 'win': configs = [target_dicts[qualified_target]['configurations'][config_name] for qualified_target in target_list] @@ -1909,7 +1972,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, configs, generator_flags) cl_paths = gyp.msvs_emulation.GenerateEnvironmentFiles( toplevel_build, generator_flags, shared_system_includes, OpenOutput) - for arch, path in cl_paths.items(): + for arch, path in sorted(cl_paths.items()): if clang_cl: # If we have selected clang-cl, use that instead. path = clang_cl @@ -2233,6 +2296,12 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, 'compile_xcassets', description='COMPILE XCASSETS $in', command='$env ./gyp-mac-tool compile-xcassets $keys $in') + master_ninja.rule( + 'compile_ios_framework_headers', + description='COMPILE HEADER MAPS AND COPY FRAMEWORK HEADERS $in', + command='$env ./gyp-mac-tool compile-ios-framework-header-map $out ' + '$framework $in && $env ./gyp-mac-tool ' + 'copy-ios-framework-headers $framework $copy_headers') master_ninja.rule( 'mac_tool', description='MACTOOL $mactool_cmd $in', @@ -2242,6 +2311,11 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, description='PACKAGE FRAMEWORK $out, POSTBUILDS', command='./gyp-mac-tool package-framework $out $version$postbuilds ' '&& touch $out') + master_ninja.rule( + 'package_ios_framework', + description='PACKAGE IOS FRAMEWORK $out, POSTBUILDS', + command='./gyp-mac-tool package-ios-framework $out $postbuilds ' + '&& touch $out') if flavor == 'win': master_ninja.rule( 'stamp', @@ -2266,7 +2340,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, master_ninja.rule( 'copy', description='COPY $in $out', - command='rm -rf $out && cp -af $in $out') + command='ln -f $in $out 2>/dev/null || (rm -rf $out && cp -af $in $out)') master_ninja.newline() all_targets = set() @@ -2314,6 +2388,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, qualified_target_for_hash = gyp.common.QualifiedTarget(build_file, name, toolset) + qualified_target_for_hash = qualified_target_for_hash.encode('utf-8') hash_for_rules = hashlib.md5(qualified_target_for_hash).hexdigest() base_path = os.path.dirname(build_file) @@ -2353,7 +2428,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, # able to run actions and build libraries by their short name. master_ninja.newline() master_ninja.comment('Short names for targets.') - for short_name in target_short_names: + for short_name in sorted(target_short_names): master_ninja.build(short_name, 'phony', [x.FinalOutput() for x in target_short_names[short_name]]) @@ -2369,7 +2444,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, if all_outputs: master_ninja.newline() - master_ninja.build('all', 'phony', list(all_outputs)) + master_ninja.build('all', 'phony', sorted(all_outputs)) master_ninja.default(generator_flags.get('default_target', 'all')) master_ninja_file.close() diff --git a/gyp/pylib/gyp/generator/ninja_test.py b/gyp/pylib/gyp/generator/ninja_test.py index 5ecfbdf004..c8adc251c9 100644 --- a/gyp/pylib/gyp/generator/ninja_test.py +++ b/gyp/pylib/gyp/generator/ninja_test.py @@ -6,9 +6,10 @@ """ Unit tests for the ninja.py file. """ -import gyp.generator.ninja as ninja -import unittest import sys +import unittest + +import gyp.generator.ninja as ninja class TestPrefixesAndSuffixes(unittest.TestCase): diff --git a/gyp/pylib/gyp/generator/xcode.py b/gyp/pylib/gyp/generator/xcode.py index 6317d04c70..4917ba77b9 100644 --- a/gyp/pylib/gyp/generator/xcode.py +++ b/gyp/pylib/gyp/generator/xcode.py @@ -1,8 +1,9 @@ -from __future__ import print_function # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. +from __future__ import print_function + import filecmp import gyp.common import gyp.xcodeproj_file @@ -78,6 +79,7 @@ 'mac_framework_headers', 'mac_framework_private_headers', 'mac_xctest_bundle', + 'mac_xcuitest_bundle', 'xcode_create_dependents_test_runner', ] @@ -692,6 +694,7 @@ def GenerateOutput(target_list, target_dicts, data, params): 'executable+bundle': 'com.apple.product-type.application', 'loadable_module+bundle': 'com.apple.product-type.bundle', 'loadable_module+xctest': 'com.apple.product-type.bundle.unit-test', + 'loadable_module+xcuitest': 'com.apple.product-type.bundle.ui-testing', 'shared_library+bundle': 'com.apple.product-type.framework', 'executable+extension+bundle': 'com.apple.product-type.app-extension', 'executable+watch+extension+bundle': @@ -708,13 +711,19 @@ def GenerateOutput(target_list, target_dicts, data, params): type = spec['type'] is_xctest = int(spec.get('mac_xctest_bundle', 0)) + is_xcuitest = int(spec.get('mac_xcuitest_bundle', 0)) is_bundle = int(spec.get('mac_bundle', 0)) or is_xctest is_app_extension = int(spec.get('ios_app_extension', 0)) is_watchkit_extension = int(spec.get('ios_watchkit_extension', 0)) is_watch_app = int(spec.get('ios_watch_app', 0)) if type != 'none': type_bundle_key = type - if is_xctest: + if is_xcuitest: + type_bundle_key += '+xcuitest' + assert type == 'loadable_module', ( + 'mac_xcuitest_bundle targets must have type loadable_module ' + '(target %s)' % target_name) + elif is_xctest: type_bundle_key += '+xctest' assert type == 'loadable_module', ( 'mac_xctest_bundle targets must have type loadable_module ' @@ -746,6 +755,9 @@ def GenerateOutput(target_list, target_dicts, data, params): assert not is_bundle, ( 'mac_bundle targets cannot have type none (target "%s")' % target_name) + assert not is_xcuitest, ( + 'mac_xcuitest_bundle targets cannot have type none (target "%s")' % + target_name) assert not is_xctest, ( 'mac_xctest_bundle targets cannot have type none (target "%s")' % target_name) diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index d1742800ac..1f40abb069 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -19,6 +19,7 @@ import threading import time import traceback +from distutils.version import StrictVersion from gyp.common import GypError from gyp.common import OrderedSet @@ -30,6 +31,7 @@ 'shared_library', 'loadable_module', 'mac_kernel_extension', + 'windows_driver', ] # A list of sections that contain links to other targets. @@ -172,10 +174,8 @@ def GetIncludedBuildFiles(build_file_path, aux_data, included=None): def CheckedEval(file_contents): """Return the eval of a gyp file. - The gyp file is restricted to dictionaries and lists only, and repeated keys are not allowed. - Note that this is slower than eval() is. """ @@ -214,7 +214,7 @@ def CheckNode(node, keypath): return node.s else: raise TypeError("Unknown AST node at key path '" + '.'.join(keypath) + - "': " + repr(node)) + "': " + repr(node)) def LoadOneBuildFile(build_file_path, data, aux_data, includes, @@ -881,6 +881,7 @@ def ExpandVariables(input, phase, variables, build_file): oldwd = os.getcwd() # Python doesn't like os.open('.'): no fchdir. if build_file_dir: # build_file_dir may be None (see above). os.chdir(build_file_dir) + sys.path.append(os.getcwd()) try: parsed_contents = shlex.split(contents) @@ -891,6 +892,7 @@ def ExpandVariables(input, phase, variables, build_file): "module (%s): %s" % (parsed_contents[0], e)) replacement = str(py_module.DoMain(parsed_contents[1:])).rstrip() finally: + sys.path.pop() os.chdir(oldwd) assert replacement != None elif command_string: @@ -949,7 +951,7 @@ def ExpandVariables(input, phase, variables, build_file): replacement = variables[contents] if isinstance(replacement, bytes) and not isinstance(replacement, str): - replacement = replacement.decode("utf-8") # done on Python 3 only + replacement = replacement.decode("utf-8") # done on Python 3 only if type(replacement) is list: for item in replacement: if isinstance(item, bytes) and not isinstance(item, str): @@ -1094,7 +1096,8 @@ def EvalSingleCondition( else: ast_code = compile(cond_expr_expanded, '', 'eval') cached_conditions_asts[cond_expr_expanded] = ast_code - if eval(ast_code, {'__builtins__': {}}, variables): + env = {'__builtins__': {}, 'v': StrictVersion} + if eval(ast_code, env, variables): return true_dict return false_dict except SyntaxError as e: @@ -1547,11 +1550,15 @@ def FlattenToList(self): # dependents. flat_list = OrderedSet() + def ExtractNodeRef(node): + """Extracts the object that the node represents from the given node.""" + return node.ref + # in_degree_zeros is the list of DependencyGraphNodes that have no # dependencies not in flat_list. Initially, it is a copy of the children # of this node, because when the graph was built, nodes with no # dependencies were made implicit dependents of the root node. - in_degree_zeros = set(self.dependents[:]) + in_degree_zeros = sorted(self.dependents[:], key=ExtractNodeRef) while in_degree_zeros: # Nodes in in_degree_zeros have no dependencies not in flat_list, so they @@ -1563,12 +1570,13 @@ def FlattenToList(self): # Look at dependents of the node just added to flat_list. Some of them # may now belong in in_degree_zeros. - for node_dependent in node.dependents: + for node_dependent in sorted(node.dependents, key=ExtractNodeRef): is_in_degree_zero = True # TODO: We want to check through the # node_dependent.dependencies list but if it's long and we # always start at the beginning, then we get O(n^2) behaviour. - for node_dependent_dependency in node_dependent.dependencies: + for node_dependent_dependency in (sorted(node_dependent.dependencies, + key=ExtractNodeRef)): if not node_dependent_dependency.ref in flat_list: # The dependent one or more dependencies not in flat_list. There # will be more chances to add it to flat_list when examining @@ -1581,7 +1589,7 @@ def FlattenToList(self): # All of the dependent's dependencies are already in flat_list. Add # it to in_degree_zeros where it will be processed in a future # iteration of the outer loop. - in_degree_zeros.add(node_dependent) + in_degree_zeros += [node_dependent] return list(flat_list) @@ -1737,12 +1745,13 @@ def _LinkDependenciesInternal(self, targets, include_shared_libraries, dependencies.add(self.ref) return dependencies - # Executables, mac kernel extensions and loadable modules are already fully - # and finally linked. Nothing else can be a link dependency of them, there - # can only be dependencies in the sense that a dependent target might run - # an executable or load the loadable_module. + # Executables, mac kernel extensions, windows drivers and loadable modules + # are already fully and finally linked. Nothing else can be a link + # dependency of them, there can only be dependencies in the sense that a + # dependent target might run an executable or load the loadable_module. if not initial and target_type in ('executable', 'loadable_module', - 'mac_kernel_extension'): + 'mac_kernel_extension', + 'windows_driver'): return dependencies # Shared libraries are already fully linked. They should only be included @@ -2038,7 +2047,7 @@ def MakePathRelative(to_file, fro_file, item): gyp.common.RelativePath(os.path.dirname(fro_file), os.path.dirname(to_file)), item)).replace('\\', '/') - if item[-1:] == '/': + if item.endswith('/'): ret += '/' return ret @@ -2493,7 +2502,7 @@ def ValidateTargetType(target, target_dict): """ VALID_TARGET_TYPES = ('executable', 'loadable_module', 'static_library', 'shared_library', - 'mac_kernel_extension', 'none') + 'mac_kernel_extension', 'none', 'windows_driver') target_type = target_dict.get('type', None) if target_type not in VALID_TARGET_TYPES: raise GypError("Target %s has an invalid target type '%s'. " @@ -2644,7 +2653,7 @@ def ValidateActionsInTarget(target, target_dict, build_file): def TurnIntIntoStrInDict(the_dict): """Given dict the_dict, recursively converts all integers into strings. """ - # Use items instead of items because there's no need to try to look at + # Use items instead of iteritems because there's no need to try to look at # reinserted keys and their associated values. for k, v in the_dict.items(): if type(v) is int: diff --git a/gyp/pylib/gyp/mac_tool.py b/gyp/pylib/gyp/mac_tool.py index 222befb982..781a8633bc 100755 --- a/gyp/pylib/gyp/mac_tool.py +++ b/gyp/pylib/gyp/mac_tool.py @@ -19,6 +19,7 @@ import re import shutil import string +import struct import subprocess import sys import tempfile @@ -52,6 +53,7 @@ def _CommandifyName(self, name_string): def ExecCopyBundleResource(self, source, dest, convert_to_binary): """Copies a resource file to the bundle/Resources directory, performing any necessary compilation on each resource.""" + convert_to_binary = convert_to_binary == 'True' extension = os.path.splitext(source)[1].lower() if os.path.isdir(source): # Copy tree. @@ -65,11 +67,16 @@ def ExecCopyBundleResource(self, source, dest, convert_to_binary): return self._CopyXIBFile(source, dest) elif extension == '.storyboard': return self._CopyXIBFile(source, dest) - elif extension == '.strings': - self._CopyStringsFile(source, dest, convert_to_binary) + elif extension == '.strings' and not convert_to_binary: + self._CopyStringsFile(source, dest) else: + if os.path.exists(dest): + os.unlink(dest) shutil.copy(source, dest) + if convert_to_binary and extension in ('.plist', '.strings'): + self._ConvertToBinary(dest) + def _CopyXIBFile(self, source, dest): """Compiles a XIB file with ibtool into a binary plist in the bundle.""" @@ -80,27 +87,49 @@ def _CopyXIBFile(self, source, dest): if os.path.relpath(dest): dest = os.path.join(base, dest) - args = ['xcrun', 'ibtool', '--errors', '--warnings', '--notices', - '--output-format', 'human-readable-text', '--compile', dest, source] + args = ['xcrun', 'ibtool', '--errors', '--warnings', '--notices'] + + if os.environ['XCODE_VERSION_ACTUAL'] > '0700': + args.extend(['--auto-activate-custom-fonts']) + if 'IPHONEOS_DEPLOYMENT_TARGET' in os.environ: + args.extend([ + '--target-device', 'iphone', '--target-device', 'ipad', + '--minimum-deployment-target', + os.environ['IPHONEOS_DEPLOYMENT_TARGET'], + ]) + else: + args.extend([ + '--target-device', 'mac', + '--minimum-deployment-target', + os.environ['MACOSX_DEPLOYMENT_TARGET'], + ]) + + args.extend(['--output-format', 'human-readable-text', '--compile', dest, + source]) + ibtool_section_re = re.compile(r'/\*.*\*/') ibtool_re = re.compile(r'.*note:.*is clipping its content') - ibtoolout = subprocess.Popen(args, stdout=subprocess.PIPE) + try: + stdout = subprocess.check_output(args) + except subprocess.CalledProcessError as e: + print(e.output) + raise current_section_header = None - for line in ibtoolout.stdout: + for line in stdout.splitlines(): if ibtool_section_re.match(line): current_section_header = line elif not ibtool_re.match(line): if current_section_header: - sys.stdout.write(current_section_header) + print(current_section_header) current_section_header = None - sys.stdout.write(line) - return ibtoolout.returncode + print(line) + return 0 def _ConvertToBinary(self, dest): subprocess.check_call([ 'xcrun', 'plutil', '-convert', 'binary1', '-o', dest, dest]) - def _CopyStringsFile(self, source, dest, convert_to_binary): + def _CopyStringsFile(self, source, dest): """Copies a .strings file using iconv to reconvert the input into UTF-16.""" input_code = self._DetectInputEncoding(source) or "UTF-8" @@ -110,32 +139,25 @@ def _CopyStringsFile(self, source, dest, convert_to_binary): # semicolon in dictionary. # on invalid files. Do the same kind of validation. import CoreFoundation - s = open(source, 'rb').read() + with open(source, 'rb') as in_file: + s = in_file.read() d = CoreFoundation.CFDataCreate(None, s, len(s)) _, error = CoreFoundation.CFPropertyListCreateFromXMLData(None, d, 0, None) if error: return - fp = open(dest, 'wb') - fp.write(s.decode(input_code).encode('UTF-16')) - fp.close() - - if convert_to_binary == 'True': - self._ConvertToBinary(dest) + with open(dest, 'wb') as fp: + fp.write(s.decode(input_code).encode('UTF-16')) def _DetectInputEncoding(self, file_name): """Reads the first few bytes from file_name and tries to guess the text encoding. Returns None as a guess if it can't detect it.""" - fp = open(file_name, 'rb') - try: - header = fp.read(3) - except Exception: - fp.close() - return None - fp.close() - if header.startswith("\xFE\xFF"): - return "UTF-16" - elif header.startswith("\xFF\xFE"): + with open(file_name, 'rb') as fp: + try: + header = fp.read(3) + except Exception: + return None + if header.startswith(("\xFE\xFF", "\xFF\xFE")): return "UTF-16" elif header.startswith("\xEF\xBB\xBF"): return "UTF-8" @@ -145,9 +167,8 @@ def _DetectInputEncoding(self, file_name): def ExecCopyInfoPlist(self, source, dest, convert_to_binary, *keys): """Copies the |source| Info.plist to the destination directory |dest|.""" # Read the source Info.plist into memory. - fd = open(source, 'r') - lines = fd.read() - fd.close() + with open(source, 'r') as fd: + lines = fd.read() # Insert synthesized key/value pairs (e.g. BuildMachineOSBuild). plist = plistlib.readPlistFromString(lines) @@ -157,7 +178,7 @@ def ExecCopyInfoPlist(self, source, dest, convert_to_binary, *keys): # Go through all the environment variables and replace them as variables in # the file. - IDENT_RE = re.compile(r'[/\s]') + IDENT_RE = re.compile(r'[_/\s]') for key in os.environ: if key.startswith('_'): continue @@ -180,17 +201,16 @@ def ExecCopyInfoPlist(self, source, dest, convert_to_binary, *keys): lines = string.replace(lines, evar, evalue) # Remove any keys with values that haven't been replaced. - lines = lines.split('\n') + lines = lines.splitlines() for i in range(len(lines)): if lines[i].strip().startswith("${"): lines[i] = None lines[i - 1] = None - lines = '\n'.join(filter(lambda x: x is not None, lines)) + lines = '\n'.join(line for line in lines if line is not None) # Write out the file with variables replaced. - fd = open(dest, 'w') - fd.write(lines) - fd.close() + with open(dest, 'w') as fd: + fd.write(lines) # Now write out PkgInfo file now that the Info.plist file has been # "compiled". @@ -218,9 +238,8 @@ def _WritePkgInfo(self, info_plist): signature_code = '?' * 4 dest = os.path.join(os.path.dirname(info_plist), 'PkgInfo') - fp = open(dest, 'w') - fp.write('%s%s' % (package_type, signature_code)) - fp.close() + with open(dest, 'w') as fp: + fp.write('%s%s' % (package_type, signature_code)) def ExecFlock(self, lockfile, *cmd_list): """Emulates the most basic behavior of Linux's flock(1).""" @@ -232,7 +251,8 @@ def ExecFlock(self, lockfile, *cmd_list): def ExecFilterLibtool(self, *cmd_list): """Calls libtool and filters out '/path/to/libtool: file: foo.o has no symbols'.""" - libtool_re = re.compile(r'^.*libtool: file: .* has no symbols$') + libtool_re = re.compile(r'^.*libtool: (?:for architecture: \S* )?' + r'file: .* has no symbols$') libtool_re5 = re.compile( r'^.*libtool: warning for library: ' + r'.* the table of contents is empty ' + @@ -259,6 +279,22 @@ def ExecFilterLibtool(self, *cmd_list): break return libtoolout.returncode + def ExecPackageIosFramework(self, framework): + # Find the name of the binary based on the part before the ".framework". + binary = os.path.basename(framework).split('.')[0] + module_path = os.path.join(framework, 'Modules') + if not os.path.exists(module_path): + os.mkdir(module_path) + module_template = 'framework module %s {\n' \ + ' umbrella header "%s.h"\n' \ + '\n' \ + ' export *\n' \ + ' module * { export * }\n' \ + '}\n' % (binary, binary) + + with open(os.path.join(module_path, 'module.modulemap'), "w") as module_file: + module_file.write(module_template) + def ExecPackageFramework(self, framework, version): """Takes a path to Something.framework and the Current version of that and sets up all the symlinks.""" @@ -295,6 +331,23 @@ def _Relink(self, dest, link): os.remove(link) os.symlink(dest, link) + def ExecCompileIosFrameworkHeaderMap(self, out, framework, *all_headers): + framework_name = os.path.basename(framework).split('.')[0] + all_headers = [os.path.abspath(header) for header in all_headers] + filelist = {} + for header in all_headers: + filename = os.path.basename(header) + filelist[filename] = header + filelist[os.path.join(framework_name, filename)] = header + WriteHmap(out, filelist) + + def ExecCopyIosFrameworkHeaders(self, framework, *copy_headers): + header_path = os.path.join(framework, 'Headers') + if not os.path.exists(header_path): + os.makedirs(header_path) + for header in copy_headers: + shutil.copy(header, os.path.join(header_path, os.path.basename(header))) + def ExecCompileXcassets(self, keys, *inputs): """Compiles multiple .xcassets files into a single .car file. @@ -355,49 +408,28 @@ def ExecMergeInfoPlist(self, output, *inputs): self._MergePlist(merged_plist, plist) plistlib.writePlist(merged_plist, output) - def ExecCodeSignBundle(self, key, resource_rules, entitlements, provisioning): + def ExecCodeSignBundle(self, key, entitlements, provisioning, path, preserve): """Code sign a bundle. This function tries to code sign an iOS bundle, following the same algorithm as Xcode: - 1. copy ResourceRules.plist from the user or the SDK into the bundle, - 2. pick the provisioning profile that best match the bundle identifier, + 1. pick the provisioning profile that best match the bundle identifier, and copy it into the bundle as embedded.mobileprovision, - 3. copy Entitlements.plist from user or SDK next to the bundle, - 4. code sign the bundle. + 2. copy Entitlements.plist from user or SDK next to the bundle, + 3. code sign the bundle. """ - resource_rules_path = self._InstallResourceRules(resource_rules) substitutions, overrides = self._InstallProvisioningProfile( provisioning, self._GetCFBundleIdentifier()) entitlements_path = self._InstallEntitlements( entitlements, substitutions, overrides) - subprocess.check_call([ - 'codesign', '--force', '--sign', key, '--resource-rules', - resource_rules_path, '--entitlements', entitlements_path, - os.path.join( - os.environ['TARGET_BUILD_DIR'], - os.environ['FULL_PRODUCT_NAME'])]) - - def _InstallResourceRules(self, resource_rules): - """Installs ResourceRules.plist from user or SDK into the bundle. - - Args: - resource_rules: string, optional, path to the ResourceRules.plist file - to use, default to "${SDKROOT}/ResourceRules.plist" - Returns: - Path to the copy of ResourceRules.plist into the bundle. - """ - source_path = resource_rules - target_path = os.path.join( - os.environ['BUILT_PRODUCTS_DIR'], - os.environ['CONTENTS_FOLDER_PATH'], - 'ResourceRules.plist') - if not source_path: - source_path = os.path.join( - os.environ['SDKROOT'], 'ResourceRules.plist') - shutil.copy2(source_path, target_path) - return target_path + args = ['codesign', '--force', '--sign', key] + if preserve == 'True': + args.extend(['--deep', '--preserve-metadata=identifier,entitlements']) + else: + args.extend(['--entitlements', entitlements_path]) + args.extend(['--timestamp=none', path]) + subprocess.check_call(args) def _InstallProvisioningProfile(self, profile, bundle_identifier): """Installs embedded.mobileprovision into the bundle. @@ -610,5 +642,71 @@ def _ExpandVariables(self, data, substitutions): return {k: self._ExpandVariables(data[k], substitutions) for k in data} return data +def NextGreaterPowerOf2(x): + return 2**(x).bit_length() + +def WriteHmap(output_name, filelist): + """Generates a header map based on |filelist|. + + Per Mark Mentovai: + A header map is structured essentially as a hash table, keyed by names used + in #includes, and providing pathnames to the actual files. + + The implementation below and the comment above comes from inspecting: + http://www.opensource.apple.com/source/distcc/distcc-2503/distcc_dist/include_server/headermap.py?txt + while also looking at the implementation in clang in: + https://llvm.org/svn/llvm-project/cfe/trunk/lib/Lex/HeaderMap.cpp + """ + magic = 1751998832 + version = 1 + _reserved = 0 + count = len(filelist) + capacity = NextGreaterPowerOf2(count) + strings_offset = 24 + (12 * capacity) + max_value_length = max(len(value) for value in filelist.values()) + + out = open(output_name, "wb") + out.write(struct.pack(' 0 or arg.count('/') > 1: + arg = os.path.normpath(arg) + # For a literal quote, CommandLineToArgvW requires 2n+1 backslashes # preceding it, and results in n backslashes + the quote. So we substitute # in 2* what we match, +1 more, plus the quote. @@ -277,7 +281,8 @@ def ConvertVSMacros(self, s, base_to_build=None, config=None): def AdjustLibraries(self, libraries): """Strip -l from library if it's specified with that.""" libs = [lib[2:] if lib.startswith('-l') else lib for lib in libraries] - return [lib + '.lib' if not lib.endswith('.lib') else lib for lib in libs] + return [lib + '.lib' if not lib.lower().endswith('.lib') \ + and not lib.lower().endswith('.obj') else lib for lib in libs] def _GetAndMunge(self, field, path, default, prefix, append, map): """Retrieve a value from |field| at |path| or return |default|. If @@ -314,7 +319,10 @@ def _TargetConfig(self, config): # There's two levels of architecture/platform specification in VS. The # first level is globally for the configuration (this is what we consider # "the" config at the gyp level, which will be something like 'Debug' or - # 'Release_x64'), and a second target-specific configuration, which is an + # 'Release'), VS2015 and later only use this level + if self.vs_version.short_name >= 2015: + return config + # and a second target-specific configuration, which is an # override for the global one. |config| is remapped here to take into # account the local target-specific overrides to the global configuration. arch = self.GetArch(config) @@ -476,8 +484,10 @@ def GetCflags(self, config): prefix='/arch:') cflags.extend(['/FI' + f for f in self._Setting( ('VCCLCompilerTool', 'ForcedIncludeFiles'), config, default=[])]) - if self.vs_version.short_name in ('2013', '2013e', '2015'): - # New flag required in 2013 to maintain previous PDB behavior. + if self.vs_version.project_version >= 12.0: + # New flag introduced in VS2013 (project version 12.0) Forces writes to + # the program database (PDB) to be serialized through MSPDBSRV.EXE. + # https://msdn.microsoft.com/en-us/library/dn502518.aspx cflags.append('/FS') # ninja handles parallelism by itself, don't have the compiler do it too. cflags = filter(lambda x: not x.startswith('/MP'), cflags) @@ -493,8 +503,9 @@ def _GetPchFlags(self, config, extension): if self.msvs_precompiled_header[config]: source_ext = os.path.splitext(self.msvs_precompiled_source[config])[1] if _LanguageMatchesForPch(source_ext, extension): - pch = os.path.split(self.msvs_precompiled_header[config])[1] - return ['/Yu' + pch, '/FI' + pch, '/Fp${pchprefix}.' + pch + '.pch'] + pch = self.msvs_precompiled_header[config] + pchbase = os.path.split(pch)[1] + return ['/Yu' + pch, '/FI' + pch, '/Fp${pchprefix}.' + pchbase + '.pch'] return [] def GetCflagsC(self, config): @@ -536,7 +547,8 @@ def GetDefFile(self, gyp_to_build_path): """Returns the .def file from sources, if any. Otherwise returns None.""" spec = self.spec if spec['type'] in ('shared_library', 'loadable_module', 'executable'): - def_files = [s for s in spec.get('sources', []) if s.endswith('.def')] + def_files = [s for s in spec.get('sources', []) + if s.lower().endswith('.def')] if len(def_files) == 1: return gyp_to_build_path(def_files[0]) elif len(def_files) > 1: @@ -901,7 +913,7 @@ def __init__( def _PchHeader(self): """Get the header that will appear in an #include line for all source files.""" - return os.path.split(self.settings.msvs_precompiled_header[self.config])[1] + return self.settings.msvs_precompiled_header[self.config] def GetObjDependencies(self, sources, objs, arch): """Given a list of sources files and the corresponding object files, @@ -974,6 +986,10 @@ def _ExtractImportantEnvironment(output_of_set): 'tmp', ) env = {} + # This occasionally happens and leads to misleading SYSTEMROOT error messages + # if not caught here. + if output_of_set.count('=') == 0: + raise Exception('Invalid output_of_set. Value is:\n%s' % output_of_set) for line in output_of_set.splitlines(): for envvar in envvars_to_save: if re.match(envvar + '=', line.lower()): @@ -1044,6 +1060,8 @@ def GenerateEnvironmentFiles(toplevel_build_dir, generator_flags, variables, _ = popen.communicate() if PY3: variables = variables.decode('utf-8') + if popen.returncode != 0: + raise Exception('"%s" failed with error %d' % (args, popen.returncode)) env = _ExtractImportantEnvironment(variables) # Inject system includes from gyp files into INCLUDE. diff --git a/gyp/pylib/gyp/win_tool.py b/gyp/pylib/gyp/win_tool.py index 0a6daf2039..cfdacb0d7c 100755 --- a/gyp/pylib/gyp/win_tool.py +++ b/gyp/pylib/gyp/win_tool.py @@ -119,11 +119,19 @@ def ExecLinkWrapper(self, arch, use_separate_mspdbsrv, *args): env = self._GetEnv(arch) if use_separate_mspdbsrv == 'True': self._UseSeparateMspdbsrv(env, args) - link = subprocess.Popen([args[0].replace('/', '\\')] + list(args[1:]), - shell=True, - env=env, - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + if sys.platform == 'win32': + args = list(args) # *args is a tuple by default, which is read-only. + args[0] = args[0].replace('/', '\\') + # https://docs.python.org/2/library/subprocess.html: + # "On Unix with shell=True [...] if args is a sequence, the first item + # specifies the command string, and any additional items will be treated as + # additional arguments to the shell itself. That is to say, Popen does the + # equivalent of: + # Popen(['/bin/sh', '-c', args[0], args[1], ...])" + # For that reason, since going through the shell doesn't seem necessary on + # non-Windows don't do that there. + link = subprocess.Popen(args, shell=sys.platform == 'win32', env=env, + stdout=subprocess.PIPE, stderr=subprocess.STDOUT) out, _ = link.communicate() if PY3: out = out.decode('utf-8') diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py index 6017164990..c3daba5fb8 100644 --- a/gyp/pylib/gyp/xcode_emulation.py +++ b/gyp/pylib/gyp/xcode_emulation.py @@ -151,6 +151,7 @@ class XcodeSettings(object): # Populated lazily by _SdkPath(). Shared by all XcodeSettings, so cached # at class-level for efficiency. _sdk_path_cache = {} + _platform_path_cache = {} _sdk_root_cache = {} # Populated lazily by GetExtraPlistItems(). Shared by all XcodeSettings, so @@ -165,6 +166,8 @@ def __init__(self, spec): self.spec = spec self.isIOS = False + self.mac_toolchain_dir = None + self.header_map_path = None # Per-target 'xcode_settings' are pushed down into configs earlier by gyp. # This means self.xcode_settings[config] always contains all settings @@ -198,7 +201,7 @@ def _ConvertConditionalKeys(self, configname): new_key = key.split("[")[0] settings[new_key] = settings[key] else: - print('Warning: Conditional keys not implemented, ignoring:', \ + print('Warning: Conditional keys not implemented, ignoring:', ' '.join(conditional_keys)) del settings[key] @@ -225,8 +228,19 @@ def IsBinaryOutputFormat(self, configname): default) return format == "binary" + def IsIosFramework(self): + return self.spec['type'] == 'shared_library' and self._IsBundle() and \ + self.isIOS + def _IsBundle(self): - return int(self.spec.get('mac_bundle', 0)) != 0 + return int(self.spec.get('mac_bundle', 0)) != 0 or self._IsXCTest() or \ + self._IsXCUiTest() + + def _IsXCTest(self): + return int(self.spec.get('mac_xctest_bundle', 0)) != 0 + + def _IsXCUiTest(self): + return int(self.spec.get('mac_xcuitest_bundle', 0)) != 0 def _IsIosAppExtension(self): return int(self.spec.get('ios_app_extension', 0)) != 0 @@ -237,9 +251,6 @@ def _IsIosWatchKitExtension(self): def _IsIosWatchApp(self): return int(self.spec.get('ios_watch_app', 0)) != 0 - def _IsXCTest(self): - return int(self.spec.get('mac_xctest_bundle', 0)) != 0 - def GetFrameworkVersion(self): """Returns the framework version of the current target. Only valid for bundles.""" @@ -305,11 +316,62 @@ def GetBundleResourceFolder(self): return self.GetBundleContentsFolderPath() return os.path.join(self.GetBundleContentsFolderPath(), 'Resources') + def GetBundleExecutableFolderPath(self): + """Returns the qualified path to the bundle's executables folder. E.g. + Chromium.app/Contents/MacOS. Only valid for bundles.""" + assert self._IsBundle() + if self.spec['type'] in ('shared_library') or self.isIOS: + return self.GetBundleContentsFolderPath() + elif self.spec['type'] in ('executable', 'loadable_module'): + return os.path.join(self.GetBundleContentsFolderPath(), 'MacOS') + + def GetBundleJavaFolderPath(self): + """Returns the qualified path to the bundle's Java resource folder. + E.g. Chromium.app/Contents/Resources/Java. Only valid for bundles.""" + assert self._IsBundle() + return os.path.join(self.GetBundleResourceFolder(), 'Java') + + def GetBundleFrameworksFolderPath(self): + """Returns the qualified path to the bundle's frameworks folder. E.g, + Chromium.app/Contents/Frameworks. Only valid for bundles.""" + assert self._IsBundle() + return os.path.join(self.GetBundleContentsFolderPath(), 'Frameworks') + + def GetBundleSharedFrameworksFolderPath(self): + """Returns the qualified path to the bundle's frameworks folder. E.g, + Chromium.app/Contents/SharedFrameworks. Only valid for bundles.""" + assert self._IsBundle() + return os.path.join(self.GetBundleContentsFolderPath(), + 'SharedFrameworks') + + def GetBundleSharedSupportFolderPath(self): + """Returns the qualified path to the bundle's shared support folder. E.g, + Chromium.app/Contents/SharedSupport. Only valid for bundles.""" + assert self._IsBundle() + if self.spec['type'] == 'shared_library': + return self.GetBundleResourceFolder() + else: + return os.path.join(self.GetBundleContentsFolderPath(), + 'SharedSupport') + + def GetBundlePlugInsFolderPath(self): + """Returns the qualified path to the bundle's plugins folder. E.g, + Chromium.app/Contents/PlugIns. Only valid for bundles.""" + assert self._IsBundle() + return os.path.join(self.GetBundleContentsFolderPath(), 'PlugIns') + + def GetBundleXPCServicesFolderPath(self): + """Returns the qualified path to the bundle's XPC services folder. E.g, + Chromium.app/Contents/XPCServices. Only valid for bundles.""" + assert self._IsBundle() + return os.path.join(self.GetBundleContentsFolderPath(), 'XPCServices') + def GetBundlePlistPath(self): """Returns the qualified path to the bundle's plist file. E.g. Chromium.app/Contents/Info.plist. Only valid for bundles.""" assert self._IsBundle() - if self.spec['type'] in ('executable', 'loadable_module'): + if self.spec['type'] in ('executable', 'loadable_module') or \ + self.IsIosFramework(): return os.path.join(self.GetBundleContentsFolderPath(), 'Info.plist') else: return os.path.join(self.GetBundleContentsFolderPath(), @@ -329,6 +391,10 @@ def GetProductType(self): assert self._IsBundle(), ('ios_watch_app flag requires mac_bundle ' '(target %s)' % self.spec['target_name']) return 'com.apple.product-type.application.watchapp' + if self._IsXCUiTest(): + assert self._IsBundle(), ('mac_xcuitest_bundle flag requires mac_bundle ' + '(target %s)' % self.spec['target_name']) + return 'com.apple.product-type.bundle.ui-testing' if self._IsBundle(): return { 'executable': 'com.apple.product-type.application', @@ -359,11 +425,8 @@ def _GetBundleBinaryPath(self): """Returns the name of the bundle binary of by this target. E.g. Chromium.app/Contents/MacOS/Chromium. Only valid for bundles.""" assert self._IsBundle() - if self.spec['type'] in ('shared_library') or self.isIOS: - path = self.GetBundleContentsFolderPath() - elif self.spec['type'] in ('executable', 'loadable_module'): - path = os.path.join(self.GetBundleContentsFolderPath(), 'MacOS') - return os.path.join(path, self.GetExecutableName()) + return os.path.join(self.GetBundleExecutableFolderPath(), \ + self.GetExecutableName()) def _GetStandaloneExecutableSuffix(self): if 'product_extension' in self.spec: @@ -414,8 +477,8 @@ def GetExecutableName(self): return self._GetStandaloneBinaryPath() def GetExecutablePath(self): - """Returns the directory name of the bundle represented by this target. E.g. - Chromium.app/Contents/MacOS/Chromium.""" + """Returns the qualified path to the primary executable of the bundle + represented by this target. E.g. Chromium.app/Contents/MacOS/Chromium.""" if self._IsBundle(): return self._GetBundleBinaryPath() else: @@ -436,7 +499,7 @@ def _GetSdkVersionInfoItem(self, sdk, infoitem): # Since the CLT has no SDK paths anyway, returning None is the # most sensible route and should still do the right thing. try: - return GetStdoutQuiet(['xcodebuild', '-version', '-sdk', sdk, infoitem]) + return GetStdoutQuiet(['xcrun', '--sdk', sdk, infoitem]) except GypError: pass @@ -445,6 +508,14 @@ def _SdkRoot(self, configname): configname = self.configname return self.GetPerConfigSetting('SDKROOT', configname, default='') + def _XcodePlatformPath(self, configname=None): + sdk_root = self._SdkRoot(configname) + if sdk_root not in XcodeSettings._platform_path_cache: + platform_path = self._GetSdkVersionInfoItem(sdk_root, + '--show-sdk-platform-path') + XcodeSettings._platform_path_cache[sdk_root] = platform_path + return XcodeSettings._platform_path_cache[sdk_root] + def _SdkPath(self, configname=None): sdk_root = self._SdkRoot(configname) if sdk_root.startswith('/'): @@ -453,7 +524,7 @@ def _SdkPath(self, configname=None): def _XcodeSdkPath(self, sdk_root): if sdk_root not in XcodeSettings._sdk_path_cache: - sdk_path = self._GetSdkVersionInfoItem(sdk_root, 'Path') + sdk_path = self._GetSdkVersionInfoItem(sdk_root, '--show-sdk-path') XcodeSettings._sdk_path_cache[sdk_root] = sdk_path if sdk_root: XcodeSettings._sdk_root_cache[sdk_path] = sdk_root @@ -484,6 +555,9 @@ def GetCflags(self, configname, arch=None): if 'SDKROOT' in self._Settings() and sdk_root: cflags.append('-isysroot %s' % sdk_root) + if self.header_map_path: + cflags.append('-I%s' % self.header_map_path) + if self._Test('CLANG_WARN_CONSTANT_CONVERSION', 'YES', default='NO'): cflags.append('-Wconstant-conversion') @@ -826,7 +900,8 @@ def GetLdflags(self, configname, product_dir, gyp_to_build_path, arch=None): ldflags.append('-arch ' + archs[0]) # Xcode adds the product directory by default. - ldflags.append('-L' + product_dir) + # Rewrite -L. to -L./ to work around http://www.openradar.me/25313838 + ldflags.append('-L' + (product_dir if product_dir != '.' else './')) install_name = self.GetInstallName() if install_name and self.spec['type'] != 'loadable_module': @@ -845,8 +920,9 @@ def GetLdflags(self, configname, product_dir, gyp_to_build_path, arch=None): if self._IsXCTest(): platform_root = self._XcodePlatformPath(configname) - if platform_root: + if sdk_root and platform_root: ldflags.append('-F' + platform_root + '/Developer/Library/Frameworks/') + ldflags.append('-framework XCTest') is_extension = self._IsIosAppExtension() or self._IsIosWatchKitExtension() if sdk_root and is_extension: @@ -854,14 +930,14 @@ def GetLdflags(self, configname, product_dir, gyp_to_build_path, arch=None): # extensions and provide loader and main function. # These flags reflect the compilation options used by xcode to compile # extensions. - ldflags.append('-lpkstart') xcode_version, _ = XcodeVersion() if xcode_version < '0900': + ldflags.append('-lpkstart') ldflags.append(sdk_root + '/System/Library/PrivateFrameworks/PlugInKit.framework/PlugInKit') + else: + ldflags.append('-e _NSExtensionMain') ldflags.append('-fapplication-extension') - ldflags.append('-Xlinker -rpath ' - '-Xlinker @executable_path/../../Frameworks') self._Appendf(ldflags, 'CLANG_CXX_LIBRARY', '-stdlib=%s') @@ -935,7 +1011,8 @@ def _GetStripPostbuilds(self, configname, output_binary, quiet): self._Test('STRIP_INSTALLED_PRODUCT', 'YES', default='NO')): default_strip_style = 'debugging' - if self.spec['type'] == 'loadable_module' and self._IsBundle(): + if ((self.spec['type'] == 'loadable_module' or self._IsIosAppExtension()) + and self._IsBundle()): default_strip_style = 'non-global' elif self.spec['type'] == 'executable': default_strip_style = 'all' @@ -990,27 +1067,68 @@ def _GetIOSPostbuilds(self, configname, output_binary): """Return a shell command to codesign the iOS output binary so it can be deployed to a device. This should be run as the very last step of the build.""" - if not (self.isIOS and self.spec['type'] == 'executable'): + if not (self.isIOS and + (self.spec['type'] == 'executable' or self._IsXCTest()) or + self.IsIosFramework()): return [] + postbuilds = [] + product_name = self.GetFullProductName() settings = self.xcode_settings[configname] + + # Xcode expects XCTests to be copied into the TEST_HOST dir. + if self._IsXCTest(): + source = os.path.join("${BUILT_PRODUCTS_DIR}", product_name) + test_host = os.path.dirname(settings.get('TEST_HOST')) + xctest_destination = os.path.join(test_host, 'PlugIns', product_name) + postbuilds.extend(['ditto %s %s' % (source, xctest_destination)]) + key = self._GetIOSCodeSignIdentityKey(settings) if not key: - return [] + return postbuilds # Warn for any unimplemented signing xcode keys. unimpl = ['OTHER_CODE_SIGN_FLAGS'] unimpl = set(unimpl) & set(self.xcode_settings[configname].keys()) if unimpl: - print('Warning: Some codesign keys not implemented, ignoring: %s' % ( - ', '.join(sorted(unimpl)))) + print('Warning: Some codesign keys not implemented, ignoring: %s' % + ', '.join(sorted(unimpl))) - return ['%s code-sign-bundle "%s" "%s" "%s" "%s"' % ( + if self._IsXCTest(): + # For device xctests, Xcode copies two extra frameworks into $TEST_HOST. + test_host = os.path.dirname(settings.get('TEST_HOST')) + frameworks_dir = os.path.join(test_host, 'Frameworks') + platform_root = self._XcodePlatformPath(configname) + frameworks = \ + ['Developer/Library/PrivateFrameworks/IDEBundleInjection.framework', + 'Developer/Library/Frameworks/XCTest.framework'] + for framework in frameworks: + source = os.path.join(platform_root, framework) + destination = os.path.join(frameworks_dir, os.path.basename(framework)) + postbuilds.extend(['ditto %s %s' % (source, destination)]) + + # Then re-sign everything with 'preserve=True' + postbuilds.extend(['%s code-sign-bundle "%s" "%s" "%s" "%s" %s' % ( + os.path.join('${TARGET_BUILD_DIR}', 'gyp-mac-tool'), key, + settings.get('CODE_SIGN_ENTITLEMENTS', ''), + settings.get('PROVISIONING_PROFILE', ''), destination, True) + ]) + plugin_dir = os.path.join(test_host, 'PlugIns') + targets = [os.path.join(plugin_dir, product_name), test_host] + for target in targets: + postbuilds.extend(['%s code-sign-bundle "%s" "%s" "%s" "%s" %s' % ( + os.path.join('${TARGET_BUILD_DIR}', 'gyp-mac-tool'), key, + settings.get('CODE_SIGN_ENTITLEMENTS', ''), + settings.get('PROVISIONING_PROFILE', ''), target, True) + ]) + + postbuilds.extend(['%s code-sign-bundle "%s" "%s" "%s" "%s" %s' % ( os.path.join('${TARGET_BUILD_DIR}', 'gyp-mac-tool'), key, - settings.get('CODE_SIGN_RESOURCE_RULES_PATH', ''), settings.get('CODE_SIGN_ENTITLEMENTS', ''), - settings.get('PROVISIONING_PROFILE', '')) - ] + settings.get('PROVISIONING_PROFILE', ''), + os.path.join("${BUILT_PRODUCTS_DIR}", product_name), False) + ]) + return postbuilds def _GetIOSCodeSignIdentityKey(self, settings): identity = settings.get('CODE_SIGN_IDENTITY') @@ -1092,25 +1210,37 @@ def GetExtraPlistItems(self, configname=None): xcode_version, xcode_build = XcodeVersion() cache['DTXcode'] = xcode_version cache['DTXcodeBuild'] = xcode_build + compiler = self.xcode_settings[configname].get('GCC_VERSION') + if compiler is not None: + cache['DTCompiler'] = compiler sdk_root = self._SdkRoot(configname) if not sdk_root: sdk_root = self._DefaultSdkRoot() - cache['DTSDKName'] = sdk_root - if xcode_version >= '0430': + sdk_version = self._GetSdkVersionInfoItem(sdk_root, '--show-sdk-version') + cache['DTSDKName'] = sdk_root + (sdk_version or '') + if xcode_version >= '0720': cache['DTSDKBuild'] = self._GetSdkVersionInfoItem( - sdk_root, 'ProductBuildVersion') + sdk_root, '--show-sdk-build-version') + elif xcode_version >= '0430': + cache['DTSDKBuild'] = sdk_version else: cache['DTSDKBuild'] = cache['BuildMachineOSBuild'] if self.isIOS: - cache['DTPlatformName'] = cache['DTSDKName'] + cache['MinimumOSVersion'] = self.xcode_settings[configname].get( + 'IPHONEOS_DEPLOYMENT_TARGET') + cache['DTPlatformName'] = sdk_root + cache['DTPlatformVersion'] = sdk_version + if configname.endswith("iphoneos"): - cache['DTPlatformVersion'] = self._GetSdkVersionInfoItem( - sdk_root, 'ProductVersion') cache['CFBundleSupportedPlatforms'] = ['iPhoneOS'] + cache['DTPlatformBuild'] = cache['DTSDKBuild'] else: cache['CFBundleSupportedPlatforms'] = ['iPhoneSimulator'] + # This is weird, but Xcode sets DTPlatformBuild to an empty field + # for simulator builds. + cache['DTPlatformBuild'] = "" XcodeSettings._plist_cache[configname] = cache # Include extra plist items that are per-target, not per global @@ -1320,7 +1450,8 @@ def GetStdoutQuiet(cmdlist): """Returns the content of standard output returned by invoking |cmdlist|. Ignores the stderr. Raises |GypError| if the command return with a non-zero return code.""" - job = subprocess.Popen(cmdlist, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + job = subprocess.Popen(cmdlist, stdout=subprocess.PIPE, + stderr=subprocess.PIPE) out = job.communicate()[0] if PY3: out = out.decode("utf-8") @@ -1364,7 +1495,10 @@ def IsMacBundle(flavor, spec): Bundles are directories with a certain subdirectory structure, instead of just a single file. Bundle rules do not produce a binary but also package resources into that directory.""" - is_mac_bundle = (int(spec.get('mac_bundle', 0)) != 0 and flavor == 'mac') + is_mac_bundle = int(spec.get('mac_xctest_bundle', 0)) != 0 or \ + int(spec.get('mac_xcuitest_bundle', 0)) != 0 or \ + (int(spec.get('mac_bundle', 0)) != 0 and flavor == 'mac') + if is_mac_bundle: assert spec['type'] != 'none', ( 'mac_bundle targets cannot have type none (target "%s")' % @@ -1474,13 +1608,14 @@ def _GetXcodeEnv(xcode_settings, built_products_dir, srcroot, configuration, additional_settings: An optional dict with more values to add to the result. """ + if not xcode_settings: return {} # This function is considered a friend of XcodeSettings, so let it reach into # its implementation details. spec = xcode_settings.spec - # These are filled in on a as-needed basis. + # These are filled in on an as-needed basis. env = { 'BUILT_FRAMEWORKS_DIR' : built_products_dir, 'BUILT_PRODUCTS_DIR' : built_products_dir, @@ -1493,12 +1628,16 @@ def _GetXcodeEnv(xcode_settings, built_products_dir, srcroot, configuration, # written for bundles: 'TARGET_BUILD_DIR' : built_products_dir, 'TEMP_DIR' : '${TMPDIR}', + 'XCODE_VERSION_ACTUAL' : XcodeVersion()[0], } if xcode_settings.GetPerConfigSetting('SDKROOT', configuration): env['SDKROOT'] = xcode_settings._SdkPath(configuration) else: env['SDKROOT'] = '' + if xcode_settings.mac_toolchain_dir: + env['DEVELOPER_DIR'] = xcode_settings.mac_toolchain_dir + if spec['type'] in ( 'executable', 'static_library', 'shared_library', 'loadable_module'): env['EXECUTABLE_NAME'] = xcode_settings.GetExecutableName() @@ -1509,10 +1648,27 @@ def _GetXcodeEnv(xcode_settings, built_products_dir, srcroot, configuration, env['MACH_O_TYPE'] = mach_o_type env['PRODUCT_TYPE'] = xcode_settings.GetProductType() if xcode_settings._IsBundle(): + # xcodeproj_file.py sets the same Xcode subfolder value for this as for + # FRAMEWORKS_FOLDER_PATH so Xcode builds will actually use FFP's value. + env['BUILT_FRAMEWORKS_DIR'] = \ + os.path.join(built_products_dir + os.sep \ + + xcode_settings.GetBundleFrameworksFolderPath()) env['CONTENTS_FOLDER_PATH'] = \ - xcode_settings.GetBundleContentsFolderPath() + xcode_settings.GetBundleContentsFolderPath() + env['EXECUTABLE_FOLDER_PATH'] = \ + xcode_settings.GetBundleExecutableFolderPath() env['UNLOCALIZED_RESOURCES_FOLDER_PATH'] = \ xcode_settings.GetBundleResourceFolder() + env['JAVA_FOLDER_PATH'] = xcode_settings.GetBundleJavaFolderPath() + env['FRAMEWORKS_FOLDER_PATH'] = \ + xcode_settings.GetBundleFrameworksFolderPath() + env['SHARED_FRAMEWORKS_FOLDER_PATH'] = \ + xcode_settings.GetBundleSharedFrameworksFolderPath() + env['SHARED_SUPPORT_FOLDER_PATH'] = \ + xcode_settings.GetBundleSharedSupportFolderPath() + env['PLUGINS_FOLDER_PATH'] = xcode_settings.GetBundlePlugInsFolderPath() + env['XPCSERVICES_FOLDER_PATH'] = \ + xcode_settings.GetBundleXPCServicesFolderPath() env['INFOPLIST_PATH'] = xcode_settings.GetBundlePlistPath() env['WRAPPER_NAME'] = xcode_settings.GetWrapperName() @@ -1644,11 +1800,12 @@ def _AddIOSDeviceConfigurations(targets): for target_dict in targets.values(): toolset = target_dict['toolset'] configs = target_dict['configurations'] - for config_name, config_dict in dict(configs).items(): - iphoneos_config_dict = copy.deepcopy(config_dict) + for config_name, simulator_config_dict in dict(configs).items(): + iphoneos_config_dict = copy.deepcopy(simulator_config_dict) configs[config_name + '-iphoneos'] = iphoneos_config_dict - configs[config_name + '-iphonesimulator'] = config_dict + configs[config_name + '-iphonesimulator'] = simulator_config_dict if toolset == 'target': + simulator_config_dict['xcode_settings']['SDKROOT'] = 'iphonesimulator' iphoneos_config_dict['xcode_settings']['SDKROOT'] = 'iphoneos' return targets diff --git a/gyp/pylib/gyp/xcode_ninja.py b/gyp/pylib/gyp/xcode_ninja.py index d70eddc90a..2bc2143340 100644 --- a/gyp/pylib/gyp/xcode_ninja.py +++ b/gyp/pylib/gyp/xcode_ninja.py @@ -92,11 +92,16 @@ def _TargetFromSpec(old_spec, params): new_xcode_settings['CODE_SIGNING_REQUIRED'] = "NO" new_xcode_settings['IPHONEOS_DEPLOYMENT_TARGET'] = \ old_xcode_settings['IPHONEOS_DEPLOYMENT_TARGET'] + for key in ['BUNDLE_LOADER', 'TEST_HOST']: + if key in old_xcode_settings: + new_xcode_settings[key] = old_xcode_settings[key] + ninja_target['configurations'][config] = {} ninja_target['configurations'][config]['xcode_settings'] = \ new_xcode_settings ninja_target['mac_bundle'] = old_spec.get('mac_bundle', 0) + ninja_target['mac_xctest_bundle'] = old_spec.get('mac_xctest_bundle', 0) ninja_target['ios_app_extension'] = old_spec.get('ios_app_extension', 0) ninja_target['ios_watchkit_extension'] = \ old_spec.get('ios_watchkit_extension', 0) @@ -138,9 +143,10 @@ def IsValidTargetForWrapper(target_extras, executable_target_pattern, spec): if target_extras is not None and re.search(target_extras, target_name): return True - # Otherwise just show executable targets. - if spec.get('type', '') == 'executable' and \ - spec.get('product_extension', '') != 'bundle': + # Otherwise just show executable targets and xc_tests. + if (int(spec.get('mac_xctest_bundle', 0)) != 0 or + (spec.get('type', '') == 'executable' and + spec.get('product_extension', '') != 'bundle')): # If there is a filter and the target does not match, exclude the target. if executable_target_pattern is not None: @@ -227,13 +233,26 @@ def CreateWrapper(target_list, target_dicts, data, params): # Tell Xcode to look everywhere for headers. sources_target['configurations'] = {'Default': { 'include_dirs': [ depth ] } } + # Put excluded files into the sources target so they can be opened in Xcode. + skip_excluded_files = \ + not generator_flags.get('xcode_ninja_list_excluded_files', True) + sources = [] for target, target_dict in target_dicts.items(): base = os.path.dirname(target) files = target_dict.get('sources', []) + \ target_dict.get('mac_bundle_resources', []) + + if not skip_excluded_files: + files.extend(target_dict.get('sources_excluded', []) + + target_dict.get('mac_bundle_resources_excluded', [])) + for action in target_dict.get('actions', []): files.extend(action.get('inputs', [])) + + if not skip_excluded_files: + files.extend(action.get('inputs_excluded', [])) + # Remove files starting with $. These are mostly intermediate files for the # build system. files = [ file for file in files if not file.startswith('$')] diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py index 93ffca7c90..1e950dce8f 100644 --- a/gyp/pylib/gyp/xcodeproj_file.py +++ b/gyp/pylib/gyp/xcodeproj_file.py @@ -1940,24 +1940,40 @@ class PBXCopyFilesBuildPhase(XCBuildPhase): 'name': [0, str, 0, 0], }) - # path_tree_re matches "$(DIR)/path" or just "$(DIR)". Match group 1 is - # "DIR", match group 3 is "path" or None. - path_tree_re = re.compile('^\\$\\((.*)\\)(/(.*)|)$') - - # path_tree_to_subfolder maps names of Xcode variables to the associated - # dstSubfolderSpec property value used in a PBXCopyFilesBuildPhase object. - path_tree_to_subfolder = { - 'BUILT_FRAMEWORKS_DIR': 10, # Frameworks Directory - 'BUILT_PRODUCTS_DIR': 16, # Products Directory - # Other types that can be chosen via the Xcode UI. - # TODO(mark): Map Xcode variable names to these. - # : 1, # Wrapper - # : 6, # Executables: 6 - # : 7, # Resources - # : 15, # Java Resources - # : 11, # Shared Frameworks - # : 12, # Shared Support - # : 13, # PlugIns + # path_tree_re matches "$(DIR)/path", "$(DIR)/$(DIR2)/path" or just "$(DIR)". + # Match group 1 is "DIR", group 3 is "path" or "$(DIR2") or "$(DIR2)/path" + # or None. If group 3 is "path", group 4 will be None otherwise group 4 is + # "DIR2" and group 6 is "path". + path_tree_re = re.compile(r'^\$\((.*?)\)(/(\$\((.*?)\)(/(.*)|)|(.*)|)|)$') + + # path_tree_{first,second}_to_subfolder map names of Xcode variables to the + # associated dstSubfolderSpec property value used in a PBXCopyFilesBuildPhase + # object. + path_tree_first_to_subfolder = { + # Types that can be chosen via the Xcode UI. + 'BUILT_PRODUCTS_DIR': 16, # Products Directory + 'BUILT_FRAMEWORKS_DIR': 10, # Not an official Xcode macro. + # Existed before support for the + # names below was added. Maps to + # "Frameworks". + } + + path_tree_second_to_subfolder = { + 'WRAPPER_NAME': 1, # Wrapper + # Although Xcode's friendly name is "Executables", the destination + # is demonstrably the value of the build setting + # EXECUTABLE_FOLDER_PATH not EXECUTABLES_FOLDER_PATH. + 'EXECUTABLE_FOLDER_PATH': 6, # Executables. + 'UNLOCALIZED_RESOURCES_FOLDER_PATH': 7, # Resources + 'JAVA_FOLDER_PATH': 15, # Java Resources + 'FRAMEWORKS_FOLDER_PATH': 10, # Frameworks + 'SHARED_FRAMEWORKS_FOLDER_PATH': 11, # Shared Frameworks + 'SHARED_SUPPORT_FOLDER_PATH': 12, # Shared Support + 'PLUGINS_FOLDER_PATH': 13, # PlugIns + # For XPC Services, Xcode sets both dstPath and dstSubfolderSpec. + # Note that it re-uses the BUILT_PRODUCTS_DIR value for + # dstSubfolderSpec. dstPath is set below. + 'XPCSERVICES_FOLDER_PATH': 16, # XPC Services. } def Name(self): @@ -1978,14 +1994,61 @@ def SetDestination(self, path): path_tree_match = self.path_tree_re.search(path) if path_tree_match: - # Everything else needs to be relative to an Xcode variable. path_tree = path_tree_match.group(1) - relative_path = path_tree_match.group(3) - - if path_tree in self.path_tree_to_subfolder: - subfolder = self.path_tree_to_subfolder[path_tree] + if path_tree in self.path_tree_first_to_subfolder: + subfolder = self.path_tree_first_to_subfolder[path_tree] + relative_path = path_tree_match.group(3) if relative_path is None: relative_path = '' + + if subfolder == 16 and path_tree_match.group(4) is not None: + # BUILT_PRODUCTS_DIR (16) is the first element in a path whose + # second element is possibly one of the variable names in + # path_tree_second_to_subfolder. Xcode sets the values of all these + # variables to relative paths so .gyp files must prefix them with + # BUILT_PRODUCTS_DIR, e.g. + # $(BUILT_PRODUCTS_DIR)/$(PLUGINS_FOLDER_PATH). Then + # xcode_emulation.py can export these variables with the same values + # as Xcode yet make & ninja files can determine the absolute path + # to the target. Xcode uses the dstSubfolderSpec value set here + # to determine the full path. + # + # An alternative of xcode_emulation.py setting the values to absolute + # paths when exporting these variables has been ruled out because + # then the values would be different depending on the build tool. + # + # Another alternative is to invent new names for the variables used + # to match to the subfolder indices in the second table. .gyp files + # then will not need to prepend $(BUILT_PRODUCTS_DIR) because + # xcode_emulation.py can set the values of those variables to + # the absolute paths when exporting. This is possibly the thinking + # behind BUILT_FRAMEWORKS_DIR which is used in exactly this manner. + # + # Requiring prepending BUILT_PRODUCTS_DIR has been chosen because + # this same way could be used to specify destinations in .gyp files + # that pre-date this addition to GYP. However they would only work + # with the Xcode generator. The previous version of xcode_emulation.py + # does not export these variables. Such files will get the benefit + # of the Xcode UI showing the proper destination name simply by + # regenerating the projects with this version of GYP. + path_tree = path_tree_match.group(4) + relative_path = path_tree_match.group(6) + separator = '/' + + if path_tree in self.path_tree_second_to_subfolder: + subfolder = self.path_tree_second_to_subfolder[path_tree] + if relative_path is None: + relative_path = '' + separator = '' + if path_tree == 'XPCSERVICES_FOLDER_PATH': + relative_path = '$(CONTENTS_FOLDER_PATH)/XPCServices' \ + + separator + relative_path + else: + # subfolder = 16 from above + # The second element of the path is an unrecognized variable. + # Include it and any remaining elements in relative_path. + relative_path = path_tree_match.group(3) + else: # The path starts with an unrecognized Xcode variable # name like $(SRCROOT). Xcode will still handle this @@ -2256,6 +2319,8 @@ class PBXNativeTarget(XCTarget): '', ''], 'com.apple.product-type.bundle.unit-test': ['wrapper.cfbundle', '', '.xctest'], + 'com.apple.product-type.bundle.ui-testing': ['wrapper.cfbundle', + '', '.xctest'], 'com.googlecode.gyp.xcode.bundle': ['compiled.mach-o.dylib', '', '.so'], 'com.apple.product-type.kernel-extension': ['wrapper.kext', @@ -2312,7 +2377,9 @@ def __init__(self, properties=None, id=None, parent=None, force_extension = suffix[1:] if self._properties['productType'] == \ - 'com.apple.product-type-bundle.unit.test': + 'com.apple.product-type-bundle.unit.test' or \ + self._properties['productType'] == \ + 'com.apple.product-type-bundle.ui-testing': if force_extension is None: force_extension = suffix[1:] diff --git a/gyp/tools/pretty_vcproj.py b/gyp/tools/pretty_vcproj.py index 24e99282da..3212626004 100755 --- a/gyp/tools/pretty_vcproj.py +++ b/gyp/tools/pretty_vcproj.py @@ -291,8 +291,8 @@ def main(argv): # check if we have exactly 1 parameter. if len(argv) < 2: - print(('Usage: %s "c:\\path\\to\\vcproj.vcproj" [key1=value1] ' - '[key2=value2]' % argv[0])) + print('Usage: %s "c:\\path\\to\\vcproj.vcproj" [key1=value1] ' + '[key2=value2]' % argv[0]) return 1 # Parse the keys From e7402b4a7c440af179c8bb999f82f6b89c829bd9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dario=20Vladovi=C4=87?= Date: Wed, 19 Feb 2020 04:52:26 +0100 Subject: [PATCH 144/551] doc: update catalina xcode cli tools download link (#2044) PR-URL: https://github.com/nodejs/node-gyp/pull/2044 Reviewed-By: Rod Vagg Reviewed-By: Jiawen Geng --- macOS_Catalina.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/macOS_Catalina.md b/macOS_Catalina.md index 8c2ce9ad59..640366aeb4 100644 --- a/macOS_Catalina.md +++ b/macOS_Catalina.md @@ -63,7 +63,7 @@ There are three ways to install the Xcode libraries `node-gyp` needs on macOS. P 9. Repeat step 5 above. Is the path different this time? Repeat the _acid test_. ### Installing `node-gyp` using the Xcode Command Line Tools via manual download -1. Download the appropriate version of the "Command Line Tools for Xcode" for your version of Catalina from . As of MacOS 10.15.2, that's [Command_Line_Tools_for_Xcode_11.3.dmg](https://download.developer.apple.com/Developer_Tools/Command_Line_Tools_for_Xcode_11.3/Command_Line_Tools_for_Xcode_11.3.dmg) +1. Download the appropriate version of the "Command Line Tools for Xcode" for your version of Catalina from . As of MacOS 10.15.3, that's [Command_Line_Tools_for_Xcode_11.3.1.dmg](https://download.developer.apple.com/Developer_Tools/Command_Line_Tools_for_Xcode_11.3.1/Command_Line_Tools_for_Xcode_11.3.1.dmg) 2. Install the package. 3. Run the [_acid test_ steps above](#The-acid-test). From ca86ef253971f29d1b3145dfa4a535845ae70e82 Mon Sep 17 00:00:00 2001 From: BSKY Date: Mon, 9 Mar 2020 09:39:33 +0900 Subject: [PATCH 145/551] test: bump actions/checkout from v1 to v2 PR-URL: https://github.com/nodejs/node-gyp/pull/2063 Reviewed-By: Richard Lau Reviewed-By: Christian Clauss --- .github/workflows/Python_tests.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/Python_tests.yml b/.github/workflows/Python_tests.yml index dba9d10933..067294515d 100644 --- a/.github/workflows/Python_tests.yml +++ b/.github/workflows/Python_tests.yml @@ -13,7 +13,7 @@ jobs: os: [macos-latest, ubuntu-latest, windows-latest] python-version: [2.7, 3.6, 3.7, 3.8] # 3.5, steps: - - uses: actions/checkout@v1 + - uses: actions/checkout@v2 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v1 with: From e18a61afc1669d4897e6c5c8a6694f4995a0f4d6 Mon Sep 17 00:00:00 2001 From: Shelley Vohr Date: Wed, 4 Mar 2020 07:50:45 -0800 Subject: [PATCH 146/551] build: shrink bloated addon binaries on windows PR-URL: https://github.com/nodejs/node-gyp/pull/2060 Reviewed-By: Bartosz Sosnowski --- addon.gypi | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/addon.gypi b/addon.gypi index 6462f539ff..9327b0d722 100644 --- a/addon.gypi +++ b/addon.gypi @@ -1,7 +1,8 @@ { 'variables' : { 'node_engine_include_dir%': 'deps/v8/include', - 'node_host_binary%': 'node' + 'node_host_binary%': 'node', + 'node_with_ltcg%': 'true', }, 'target_defaults': { 'type': 'loadable_module', @@ -126,6 +127,26 @@ 'library_dirs': [ '<(node_root_dir)/$(ConfigurationName)' ], 'libraries': [ '<@(node_engine_libs)' ], }], + ['node_with_ltcg=="true"', { + 'msvs_settings': { + 'VCCLCompilerTool': { + 'WholeProgramOptimization': 'true' # /GL, whole program optimization, needed for LTCG + }, + 'VCLibrarianTool': { + 'AdditionalOptions': [ + '/LTCG:INCREMENTAL', # incremental link-time code generation + ] + }, + 'VCLinkerTool': { + 'OptimizeReferences': 2, # /OPT:REF + 'EnableCOMDATFolding': 2, # /OPT:ICF + 'LinkIncremental': 1, # disable incremental linking + 'AdditionalOptions': [ + '/LTCG:INCREMENTAL', # incremental link-time code generation + ] + } + } + }] ], 'libraries': [ '-lkernel32.lib', From 473cfa283f0615e2b0db12be76ec1cd4792756c4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C3=ABl=20Zasso?= Date: Mon, 16 Mar 2020 10:45:32 +0100 Subject: [PATCH 147/551] doc: note in README that Python 3.8 is supported (#2072) PR-URL: https://github.com/nodejs/node-gyp/pull/2072 Refs: https://github.com/nodejs/node-gyp/issues/2071 Reviewed-By: Christian Clauss Reviewed-By: Jiawen Geng --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 79abf4ba71..6f44711871 100644 --- a/README.md +++ b/README.md @@ -31,13 +31,13 @@ Depending on your operating system, you will need to install: ### On Unix - * Python v2.7, v3.5, v3.6, or v3.7 + * Python v2.7, v3.5, v3.6, v3.7, or v3.8 * `make` * A proper C/C++ compiler toolchain, like [GCC](https://gcc.gnu.org) ### On macOS - * Python v2.7, v3.5, v3.6, or v3.7 + * Python v2.7, v3.5, v3.6, v3.7, or v3.8 * [Xcode](https://developer.apple.com/xcode/download/) * You also need to install the `XCode Command Line Tools` by running `xcode-select --install`. Alternatively, if you already have the full Xcode installed, you can find them under the menu `Xcode -> Open Developer Tool -> More Developer Tools...`. This step will install `clang`, `clang++`, and `make`. * If your Mac has been _upgraded_ to macOS Catalina (10.15), please read [macOS_Catalina.md](macOS_Catalina.md). @@ -65,7 +65,7 @@ Install tools and configuration manually: ### Configuring Python Dependency `node-gyp` requires that you have installed a compatible version of Python, one of: v2.7, v3.5, v3.6, -or v3.7. If you have multiple Python versions installed, you can identify which Python +v3.7, or v3.8. If you have multiple Python versions installed, you can identify which Python version `node-gyp` should use in one of the following ways: 1. by setting the `--python` command-line option, e.g.: From 4f23c7bee2851a3f821e003e9c1d4db0baa54dc7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C3=ABl=20Zasso?= Date: Mon, 16 Mar 2020 10:46:43 +0100 Subject: [PATCH 148/551] doc: update link to the code of conduct (#2073) PR-URL: https://github.com/nodejs/node-gyp/pull/2073 Reviewed-By: Christian Clauss Reviewed-By: Jiawen Geng --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f48786bd84..c1c50eab4e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -3,7 +3,7 @@ ## Code of Conduct Please read the -[Code of Conduct](https://github.com/nodejs/TSC/blob/master/CODE_OF_CONDUCT.md) +[Code of Conduct](https://github.com/nodejs/admin/blob/master/CODE_OF_CONDUCT.md) which explains the minimum behavior expectations for node-gyp contributors. From 7b75af349bb5354943cdfaa099454243d5a74d65 Mon Sep 17 00:00:00 2001 From: Karl Horky Date: Fri, 27 Mar 2020 11:03:50 +0100 Subject: [PATCH 149/551] doc: add macOS Catalina software update info PR-URL: https://github.com/nodejs/node-gyp/pull/2078 Reviewed-By: Christian Clauss Reviewed-By: Rod Vagg --- macOS_Catalina.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/macOS_Catalina.md b/macOS_Catalina.md index 640366aeb4..238ecff50d 100644 --- a/macOS_Catalina.md +++ b/macOS_Catalina.md @@ -2,7 +2,7 @@ _This document specifically refers to upgrades from previous versions of macOS to Catalina (10.15). It should be removed from the source repository when Catalina ceases to be the latest macOS version or when future Catalina versions no longer raise these issues._ -**Upgrading to macOS Catalina may cause normal `node-gyp` installations to fail.** +**Both upgrading to macOS Catalina and running a Software Update in Catalina may cause normal `node-gyp` installations to fail.** ### Is my Mac running macOS Catalina? Let's first make sure that your Mac is running Catalina: @@ -52,15 +52,16 @@ There are three ways to install the Xcode libraries `node-gyp` needs on macOS. P ### Installing `node-gyp` using the Xcode Command Line Tools via `xcode-select --install` 1. If the _acid test_ has not succeeded, then try `xcode-select --install` -2. Wait until the install process is _complete_. -3. `softwareupdate -l` # No listing is a good sign. +2. If the installation command returns `xcode-select: error: command line tools are already installed, use "Software Update" to install updates`, continue to [remove and reinstall](#i-did-all-that-and-the-acid-test-still-does-not-pass--) +3. Wait until the install process is _complete_. +4. `softwareupdate -l` # No listing is a good sign. * If Xcode or Tools upgrades are listed, use "Software Update" to install them. -4. `xcode-select -version` # Should return `xcode-select version 2370` or later. -5. `xcode-select -print-path` # Should return `/Library/Developer/CommandLineTools` -6. Try the [_acid test_ steps above](#The-acid-test) to see if your Mac is ready. -7. If the _acid test_ does _not_ pass then... -8. `sudo xcode-select --reset` # Enter root password. No output is normal. -9. Repeat step 5 above. Is the path different this time? Repeat the _acid test_. +5. `xcode-select -version` # Should return `xcode-select version 2370` or later. +6. `xcode-select -print-path` # Should return `/Library/Developer/CommandLineTools` +7. Try the [_acid test_ steps above](#The-acid-test) to see if your Mac is ready. +8. If the _acid test_ does _not_ pass then... +9. `sudo xcode-select --reset` # Enter root password. No output is normal. +10. Repeat step 5 above. Is the path different this time? Repeat the _acid test_. ### Installing `node-gyp` using the Xcode Command Line Tools via manual download 1. Download the appropriate version of the "Command Line Tools for Xcode" for your version of Catalina from . As of MacOS 10.15.3, that's [Command_Line_Tools_for_Xcode_11.3.1.dmg](https://download.developer.apple.com/Developer_Tools/Command_Line_Tools_for_Xcode_11.3.1/Command_Line_Tools_for_Xcode_11.3.1.dmg) From 6356117b08b3c8070c355862933b45533cc9461d Mon Sep 17 00:00:00 2001 From: Bartosz Sosnowski Date: Tue, 7 Apr 2020 15:08:43 +0200 Subject: [PATCH 150/551] doc, bin: stop suggesting opening node-gyp issues A lot of new issues in node-gyp are related to outdated node-gyp or broken modules. This removes the suggestion to open a new issue in the node-gyp, instead suggesting the user should open the issue in the module issue tracker. It also makes the issue template more explicit about providing the logs. PR-URL: https://github.com/nodejs/node-gyp/pull/2096 Reviewed-By: Christian Clauss Reviewed-By: Rod Vagg --- .github/ISSUE_TEMPLATE.md | 32 +++++++++++++++++++++++++++----- bin/node-gyp.js | 5 ++--- 2 files changed, 29 insertions(+), 8 deletions(-) diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md index b5bed7fdd1..485e26ecae 100644 --- a/.github/ISSUE_TEMPLATE.md +++ b/.github/ISSUE_TEMPLATE.md @@ -1,10 +1,16 @@ * **Node Version**: @@ -19,6 +25,22 @@ Paste your log here, between the backticks. It can be: - npm --verbose output, - or contents of npm-debug.log, - or output of node-gyp rebuild --verbose. +Include the command you were trying to run. + +This should look like this: + +>npm --verbose +npm info it worked if it ends with ok +npm verb cli [ +npm verb cli 'C:\\...\\node\\13.9.0\\x64\\node.exe', +npm verb cli 'C:\\...\\node\\13.9.0\\x64\\node_modules\\npm\\bin\\npm-cli.js', +npm verb cli '--verbose' +npm verb cli ] +npm info using npm@6.13.7 +npm info using node@v13.9.0 + +Usage: npm +(...) ``` diff --git a/bin/node-gyp.js b/bin/node-gyp.js index 49b5721d02..8652ea21ec 100755 --- a/bin/node-gyp.js +++ b/bin/node-gyp.js @@ -131,9 +131,8 @@ function errorMessage () { function issueMessage () { errorMessage() - log.error('', ['This is a bug in `node-gyp`.', - 'Try to update node-gyp and file an Issue if it does not help:', - ' ' + log.error('', ['Node-gyp failed to build your package.', + 'Try to update npm and/or node-gyp and if it does not help file an issue with the package author.' ].join('\n')) } From 741ab096d554f2d3e6b6a7232b06872a0c1375f8 Mon Sep 17 00:00:00 2001 From: Shelley Vohr Date: Mon, 11 May 2020 12:00:35 -0700 Subject: [PATCH 151/551] test: remove support for EOL versions of Node.js Reviewed-By: Ben Noordhuis Reviewed-By: Jiawen Geng Reviewed-By: Christian Clauss Reviewed-By: Rod Vagg --- .travis.yml | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/.travis.yml b/.travis.yml index c281373ecc..ae691bed48 100644 --- a/.travis.yml +++ b/.travis.yml @@ -13,14 +13,6 @@ jobs: env: NODE_GYP_FORCE_PYTHON=python2 python: 2.7 - - name: "Node.js 6 & Python 3.8 on Linux" - python: 3.8 - env: NODE_GYP_FORCE_PYTHON=python3 - before_install: nvm install 6 - - name: "Node.js 8 & Python 3.8 on Linux" - python: 3.8 - env: NODE_GYP_FORCE_PYTHON=python3 - before_install: nvm install 8 - name: "Node.js 10 & Python 3.8 on Linux" python: 3.8 env: NODE_GYP_FORCE_PYTHON=python3 @@ -56,14 +48,6 @@ jobs: env: NODE_GYP_FORCE_PYTHON=python3 PATH=$HOME/.pyenv/shims:$PATH PYENV_VERSION=3.8.0 before_install: pyenv install $PYENV_VERSION - - name: "Node.js 6 & Python 2.7 on Windows" - os: windows - language: node_js - node_js: 6 # node - env: >- - PATH=/c/Python27:/c/Python27/Scripts:$PATH - NODE_GYP_FORCE_PYTHON=/c/Python27/python.exe - before_install: choco install python2 - name: "Node.js 12 & Python 2.7 on Windows" os: windows language: node_js From c255ffbf6adc6b60cb39fa1aa40bfc8eb45f80ac Mon Sep 17 00:00:00 2001 From: DeeDeeG Date: Fri, 15 May 2020 01:38:58 -0400 Subject: [PATCH 152/551] lib: drop "-2" flag for "py.exe" launcher Now that node-gyp supports both Python 2 and Python 3, We don't need to explicitly try to find only Python 2. Fixes: https://github.com/nodejs/node-gyp/issues/2130 Refs: https://docs.python.org/3/using/windows.html#launcher PR-URL: https://github.com/nodejs/node-gyp/pull/2131 Reviewed-By: Christian Clauss Reviewed-By: Rod Vagg --- lib/find-python.js | 11 +++-------- test/test-find-python.js | 3 +-- 2 files changed, 4 insertions(+), 10 deletions(-) diff --git a/lib/find-python.js b/lib/find-python.js index 43bf85985a..af269de2fc 100644 --- a/lib/find-python.js +++ b/lib/find-python.js @@ -119,7 +119,7 @@ PythonFinder.prototype = { checks.push({ before: () => { this.addLog( - 'checking if the py launcher can be used to find Python 2') + 'checking if the py launcher can be used to find Python') }, check: this.checkPyLauncher }) @@ -188,15 +188,10 @@ PythonFinder.prototype = { // Distributions of Python on Windows by default install with the "py.exe" // Python launcher which is more likely to exist than the Python executable // being in the $PATH. - // Because the Python launcher supports all versions of Python, we have to - // explicitly request a Python 2 version. This is done by supplying "-2" as - // the first command line argument. Since "py.exe -2" would be an invalid - // executable for "execFile", we have to use the launcher to figure out - // where the actual "python.exe" executable is located. checkPyLauncher: function checkPyLauncher (errorCallback) { this.log.verbose( - `- executing "${this.pyLauncher}" to get Python 2 executable path`) - this.run(this.pyLauncher, ['-2', ...this.argsExecutable], false, + `- executing "${this.pyLauncher}" to get Python executable path`) + this.run(this.pyLauncher, this.argsExecutable, false, function (err, execPath) { // Possible outcomes: same as checkCommand if (err) { diff --git a/test/test-find-python.js b/test/test-find-python.js index 6ca522a04c..6be887f7eb 100644 --- a/test/test-find-python.js +++ b/test/test-find-python.js @@ -146,14 +146,13 @@ test('find python - no python2, no python, unix', function (t) { }) test('find python - no python, use python launcher', function (t) { - t.plan(4) + t.plan(3) var f = new TestPythonFinder(null, done) f.win = true f.execFile = function (program, args, opts, cb) { if (program === 'py.exe') { - t.notEqual(args.indexOf('-2'), -1) t.notEqual(args.indexOf('-c'), -1) return cb(null, 'Z:\\snake.exe') } From 5f47b7a18397dd8530176f380d4bc1c99e0839ac Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Wed, 13 May 2020 12:28:22 +1000 Subject: [PATCH 153/551] v5.1.1: bump version and update changelog --- CHANGELOG.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d27e181b42..eac853c3e3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,6 +47,20 @@ v6.0.0 2019-10-04 * [[`3d1c60ab81`](https://github.com/nodejs/node-gyp/commit/3d1c60ab81)] - **(SEMVER-MAJOR)** **lib**: accept Python 3 by default (João Reis) [#1844](https://github.com/nodejs/node-gyp/pull/1844) * [[`c6e3b65a23`](https://github.com/nodejs/node-gyp/commit/c6e3b65a23)] - **(SEMVER-MAJOR)** **lib**: raise the minimum Python version from 2.6 to 2.7 (cclauss) [#1818](https://github.com/nodejs/node-gyp/pull/1818) +v5.1.1 2020-05-25 +================= + +* [[`bdd3a79abe`](https://github.com/nodejs/node-gyp/commit/bdd3a79abe)] - **build**: shrink bloated addon binaries on windows (Shelley Vohr) [#2060](https://github.com/nodejs/node-gyp/pull/2060) +* [[`1f2ba75bc0`](https://github.com/nodejs/node-gyp/commit/1f2ba75bc0)] - **doc**: add macOS Catalina software update info (Karl Horky) [#2078](https://github.com/nodejs/node-gyp/pull/2078) +* [[`c106d915f5`](https://github.com/nodejs/node-gyp/commit/c106d915f5)] - **doc**: update catalina xcode cli tools download link (#2044) (Dario Vladović) [#2044](https://github.com/nodejs/node-gyp/pull/2044) +* [[`9a6fea92e2`](https://github.com/nodejs/node-gyp/commit/9a6fea92e2)] - **doc**: update catalina xcode cli tools download link; formatting (Jonathan Hult) [#2034](https://github.com/nodejs/node-gyp/pull/2034) +* [[`59b0b1add8`](https://github.com/nodejs/node-gyp/commit/59b0b1add8)] - **doc**: add download link for Command Line Tools for Xcode (Przemysław Bitkowski) [#2029](https://github.com/nodejs/node-gyp/pull/2029) +* [[`bb8d0e7b10`](https://github.com/nodejs/node-gyp/commit/bb8d0e7b10)] - **doc**: Catalina suggestion: remove /Library/Developer/CommandLineTools (Christian Clauss) [#2022](https://github.com/nodejs/node-gyp/pull/2022) +* [[`fb2e80d4e3`](https://github.com/nodejs/node-gyp/commit/fb2e80d4e3)] - **doc**: update link to the code of conduct (#2073) (Michaël Zasso) [#2073](https://github.com/nodejs/node-gyp/pull/2073) +* [[`251d9c885c`](https://github.com/nodejs/node-gyp/commit/251d9c885c)] - **doc**: note in README that Python 3.8 is supported (#2072) (Michaël Zasso) [#2072](https://github.com/nodejs/node-gyp/pull/2072) +* [[`2b6fc3c8d6`](https://github.com/nodejs/node-gyp/commit/2b6fc3c8d6)] - **doc, bin**: stop suggesting opening node-gyp issues (Bartosz Sosnowski) [#2096](https://github.com/nodejs/node-gyp/pull/2096) +* [[`a876ae58ad`](https://github.com/nodejs/node-gyp/commit/a876ae58ad)] - **test**: bump actions/checkout from v1 to v2 (BSKY) [#2063](https://github.com/nodejs/node-gyp/pull/2063) + v5.1.0 2020-02-05 ================= From d45438a047a30b6c6993553459e4544e810bb3f5 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Wed, 13 May 2020 12:43:07 +1000 Subject: [PATCH 154/551] deps: update deps, match to npm@7 PR-URL: https://github.com/nodejs/node-gyp/pull/2126 Reviewed-By: Richard Lau Reviewed-By: Christian Clauss --- package.json | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/package.json b/package.json index 478f43cb81..707524e846 100644 --- a/package.json +++ b/package.json @@ -24,25 +24,25 @@ "dependencies": { "env-paths": "^2.2.0", "glob": "^7.1.4", - "graceful-fs": "^4.2.2", + "graceful-fs": "^4.2.3", "mkdirp": "^0.5.1", - "nopt": "^4.0.1", + "nopt": "^4.0.3", "npmlog": "^4.1.2", - "request": "^2.88.0", + "request": "^2.88.2", "rimraf": "^2.6.3", - "semver": "^5.7.1", - "tar": "^4.4.12", - "which": "^1.3.1" + "semver": "^7.3.2", + "tar": "^6.0.1", + "which": "^2.0.2" }, "engines": { "node": ">= 6.0.0" }, "devDependencies": { "bindings": "^1.5.0", - "nan": "^2.14.0", + "nan": "^2.14.1", "require-inject": "^1.4.4", - "standard": "^14.3.1", - "tap": "~12.7.0" + "standard": "^14.3.4", + "tap": "^12.7.0" }, "scripts": { "lint": "standard */*.js test/**/*.js", From 963f2a7b481ac4b8dee7f8c1c582f0d78e207f03 Mon Sep 17 00:00:00 2001 From: Matheus Marchini Date: Mon, 18 May 2020 17:12:35 -0700 Subject: [PATCH 155/551] doc: improve cataline discoverability for search engines By showing the error in the guide, search engines should be able to find this document when users search for specific lines from the error. Co-authored-by: Christian Clauss PR-URL: https://github.com/nodejs/node-gyp/pull/2135 Reviewed-By: Rod Vagg Reviewed-By: Christian Clauss --- macOS_Catalina.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/macOS_Catalina.md b/macOS_Catalina.md index 238ecff50d..dbc8da4e7d 100644 --- a/macOS_Catalina.md +++ b/macOS_Catalina.md @@ -2,7 +2,11 @@ _This document specifically refers to upgrades from previous versions of macOS to Catalina (10.15). It should be removed from the source repository when Catalina ceases to be the latest macOS version or when future Catalina versions no longer raise these issues._ -**Both upgrading to macOS Catalina and running a Software Update in Catalina may cause normal `node-gyp` installations to fail.** +**Both upgrading to macOS Catalina and running a Software Update in Catalina may cause normal `node-gyp` installations to fail. This might manifest as the following error during `npm install`:** + +```console +gyp: No Xcode or CLT version detected! +``` ### Is my Mac running macOS Catalina? Let's first make sure that your Mac is running Catalina: From 9aed6286a3d6debbcbb6306cf6ef317fc50f4375 Mon Sep 17 00:00:00 2001 From: Matheus Marchini Date: Mon, 18 May 2020 16:48:00 -0700 Subject: [PATCH 156/551] doc: give more attention to Catalina issues doc It's easy to miss the Catalina issues doc when reading the readme. Since it can be a common issue among developers, it makes sense to give more attention to it in the readme. PR-URL: https://github.com/nodejs/node-gyp/pull/2134 Reviewed-By: Rod Vagg Reviewed-By: Christian Clauss --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 6f44711871..e3fab86ca6 100644 --- a/README.md +++ b/README.md @@ -37,10 +37,11 @@ Depending on your operating system, you will need to install: ### On macOS +**ATTENTION**: If your Mac has been _upgraded_ to macOS Catalina (10.15), please read [macOS_Catalina.md](macOS_Catalina.md). + * Python v2.7, v3.5, v3.6, v3.7, or v3.8 * [Xcode](https://developer.apple.com/xcode/download/) * You also need to install the `XCode Command Line Tools` by running `xcode-select --install`. Alternatively, if you already have the full Xcode installed, you can find them under the menu `Xcode -> Open Developer Tool -> More Developer Tools...`. This step will install `clang`, `clang++`, and `make`. - * If your Mac has been _upgraded_ to macOS Catalina (10.15), please read [macOS_Catalina.md](macOS_Catalina.md). ### On Windows From ebc34ec823d20b593ee9713bb0887daaa363cbe2 Mon Sep 17 00:00:00 2001 From: Ujjwal Sharma Date: Tue, 7 Apr 2020 07:13:04 +0530 Subject: [PATCH 157/551] gyp: update gyp to 0.2.0 PR-URL: https://github.com/nodejs/node-gyp/pull/2092 Reviewed-By: Rod Vagg --- gyp/.flake8 | 4 + gyp/.github/workflows/Python_tests.yml | 31 + gyp/.gitignore | 144 +- gyp/AUTHORS | 1 + gyp/CODE_OF_CONDUCT.md | 4 + gyp/CONTRIBUTING.md | 32 + gyp/LICENSE | 1 + gyp/OWNERS | 1 - gyp/PRESUBMIT.py | 138 - gyp/gyp.bat | 0 gyp/gyp_main.py | 55 +- gyp/pylib/gyp/MSVSNew.py | 510 +- gyp/pylib/gyp/MSVSProject.py | 226 +- gyp/pylib/gyp/MSVSSettings.py | 1605 +++-- gyp/pylib/gyp/MSVSSettings_test.py | 2882 ++++---- gyp/pylib/gyp/MSVSToolFile.py | 65 +- gyp/pylib/gyp/MSVSUserFile.py | 212 +- gyp/pylib/gyp/MSVSUtil.py | 350 +- gyp/pylib/gyp/MSVSVersion.py | 853 +-- gyp/pylib/gyp/__init__.py | 1054 +-- gyp/pylib/gyp/common.py | 837 +-- gyp/pylib/gyp/common_test.py | 98 +- gyp/pylib/gyp/easy_xml.py | 174 +- gyp/pylib/gyp/easy_xml_test.py | 181 +- gyp/pylib/gyp/flock_tool.py | 71 +- gyp/pylib/gyp/generator/analyzer.py | 1112 ++-- gyp/pylib/gyp/generator/android.py | 1847 +++--- gyp/pylib/gyp/generator/cmake.py | 2154 +++--- .../gyp/generator/compile_commands_json.py | 167 +- .../gyp/generator/dump_dependency_json.py | 156 +- gyp/pylib/gyp/generator/eclipse.py | 729 ++- gyp/pylib/gyp/generator/gypd.py | 73 +- gyp/pylib/gyp/generator/gypsh.py | 54 +- gyp/pylib/gyp/generator/make.py | 3489 +++++----- gyp/pylib/gyp/generator/msvs.py | 5799 +++++++++-------- gyp/pylib/gyp/generator/msvs_test.py | 58 +- gyp/pylib/gyp/generator/ninja.py | 4997 +++++++------- gyp/pylib/gyp/generator/ninja_test.py | 71 +- gyp/pylib/gyp/generator/xcode.py | 2477 +++---- gyp/pylib/gyp/generator/xcode_test.py | 16 +- gyp/pylib/gyp/input.py | 5206 ++++++++------- gyp/pylib/gyp/input_test.py | 162 +- gyp/pylib/gyp/mac_tool.py | 1154 ++-- gyp/pylib/gyp/msvs_emulation.py | 2017 +++--- gyp/pylib/gyp/ninja_syntax.py | 122 +- gyp/pylib/gyp/simple_copy.py | 46 +- gyp/pylib/gyp/win_tool.py | 626 +- gyp/pylib/gyp/xcode_emulation.py | 3114 ++++----- gyp/pylib/gyp/xcode_ninja.py | 501 +- gyp/pylib/gyp/xcodeproj_file.py | 4749 +++++++------- gyp/pylib/gyp/xml_fix.py | 80 +- gyp/requirements_dev.txt | 2 + gyp/samples/samples | 81 - gyp/samples/samples.bat | 5 - gyp/setup.py | 43 +- gyp/test_gyp.py | 269 + gyp/tools/graphviz.py | 131 +- gyp/tools/pretty_gyp.py | 182 +- gyp/tools/pretty_sln.py | 267 +- gyp/tools/pretty_vcproj.py | 528 +- 60 files changed, 27578 insertions(+), 24435 deletions(-) create mode 100644 gyp/.flake8 create mode 100644 gyp/.github/workflows/Python_tests.yml create mode 100644 gyp/CODE_OF_CONDUCT.md create mode 100644 gyp/CONTRIBUTING.md delete mode 100644 gyp/OWNERS delete mode 100644 gyp/PRESUBMIT.py mode change 100644 => 100755 gyp/gyp.bat create mode 100644 gyp/requirements_dev.txt delete mode 100755 gyp/samples/samples delete mode 100644 gyp/samples/samples.bat create mode 100755 gyp/test_gyp.py diff --git a/gyp/.flake8 b/gyp/.flake8 new file mode 100644 index 0000000000..139e952e7d --- /dev/null +++ b/gyp/.flake8 @@ -0,0 +1,4 @@ +[flake8] +max-complexity = 10 +max-line-length = 88 +extend-ignore = E203,C901,E501 diff --git a/gyp/.github/workflows/Python_tests.yml b/gyp/.github/workflows/Python_tests.yml new file mode 100644 index 0000000000..47c40343ad --- /dev/null +++ b/gyp/.github/workflows/Python_tests.yml @@ -0,0 +1,31 @@ +# TODO: Enable os: windows-latest +# TODO: Enable python-version: 3.5 +# TODO: Enable pytest --doctest-modules + +name: Python_tests +on: [push, pull_request] +jobs: + Python_tests: + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + max-parallel: 15 + matrix: + os: [macos-latest, ubuntu-latest] # , windows-latest] + python-version: [2.7, 3.6, 3.7, 3.8] # 3.5, + steps: + - uses: actions/checkout@v1 + - name: Set up Python ${{ matrix.python-version }} + uses: actions/setup-python@v1 + with: + python-version: ${{ matrix.python-version }} + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install -r requirements_dev.txt + - name: Lint with flake8 + run: flake8 . --count --show-source --statistics + - name: Test with pytest + run: pytest + # - name: Run doctests with pytest + # run: pytest --doctest-modules diff --git a/gyp/.gitignore b/gyp/.gitignore index 0d20b6487c..d82fa7a96c 100644 --- a/gyp/.gitignore +++ b/gyp/.gitignore @@ -1 +1,143 @@ -*.pyc +# Byte-compiled / optimized / DLL files +__pycache__/ +*.py[cod] +*$py.class + +# C extensions +*.so + +# Distribution / packaging +.Python +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +wheels/ +pip-wheel-metadata/ +share/python-wheels/ +*.egg-info/ +.installed.cfg +*.egg +MANIFEST + +# PyInstaller +# Usually these files are written by a python script from a template +# before PyInstaller builds the exe, so as to inject date/other infos into it. +*.manifest +*.spec + +# Installer logs +pip-log.txt +pip-delete-this-directory.txt + +# Unit test / coverage reports +htmlcov/ +.tox/ +.nox/ +.coverage +.coverage.* +.cache +nosetests.xml +coverage.xml +*.cover +*.py,cover +.hypothesis/ +.pytest_cache/ +cover/ + +# Translations +*.mo +*.pot + +# Django stuff: +*.log +local_settings.py +db.sqlite3 +db.sqlite3-journal + +# Flask stuff: +instance/ +.webassets-cache + +# Scrapy stuff: +.scrapy + +# Sphinx documentation +docs/_build/ + +# PyBuilder +.pybuilder/ +target/ + +# Jupyter Notebook +.ipynb_checkpoints + +# IPython +profile_default/ +ipython_config.py + +# pyenv +# For a library or package, you might want to ignore these files since the code is +# intended to run in multiple environments; otherwise, check them in: +# .python-version + +# pipenv +# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. +# However, in case of collaboration, if having platform-specific dependencies or dependencies +# having no cross-platform support, pipenv may install dependencies that don't work, or not +# install all needed dependencies. +#Pipfile.lock + +# PEP 582; used by e.g. github.com/David-OConnor/pyflow +__pypackages__/ + +# Celery stuff +celerybeat-schedule +celerybeat.pid + +# SageMath parsed files +*.sage.py + +# Environments +.env +.venv +env/ +venv/ +ENV/ +env.bak/ +venv.bak/ + +# Spyder project settings +.spyderproject +.spyproject + +# Rope project settings +.ropeproject + +# mkdocs documentation +/site + +# mypy +.mypy_cache/ +.dmypy.json +dmypy.json + +# Pyre type checker +.pyre/ + +# pytype static type analyzer +.pytype/ + +# Cython debug symbols +cython_debug/ + +# static files generated from Django application using `collectstatic` +media +static diff --git a/gyp/AUTHORS b/gyp/AUTHORS index 130c816058..f49a357b9e 100644 --- a/gyp/AUTHORS +++ b/gyp/AUTHORS @@ -13,3 +13,4 @@ Eric N. Vander Weele Tom Freudenberg Julien Brianceau Refael Ackermann +Ujjwal Sharma diff --git a/gyp/CODE_OF_CONDUCT.md b/gyp/CODE_OF_CONDUCT.md new file mode 100644 index 0000000000..4c21140559 --- /dev/null +++ b/gyp/CODE_OF_CONDUCT.md @@ -0,0 +1,4 @@ +# Code of Conduct + +* [Node.js Code of Conduct](https://github.com/nodejs/admin/blob/master/CODE_OF_CONDUCT.md) +* [Node.js Moderation Policy](https://github.com/nodejs/admin/blob/master/Moderation-Policy.md) diff --git a/gyp/CONTRIBUTING.md b/gyp/CONTRIBUTING.md new file mode 100644 index 0000000000..f9dd574a47 --- /dev/null +++ b/gyp/CONTRIBUTING.md @@ -0,0 +1,32 @@ +# Contributing to gyp-next + +## Code of Conduct + +This project is bound to the [Node.js Code of Conduct](https://github.com/nodejs/admin/blob/master/CODE_OF_CONDUCT.md). + + +## Developer's Certificate of Origin 1.1 + +By making a contribution to this project, I certify that: + +* (a) The contribution was created in whole or in part by me and I + have the right to submit it under the open source license + indicated in the file; or + +* (b) The contribution is based upon previous work that, to the best + of my knowledge, is covered under an appropriate open source + license and I have the right under that license to submit that + work with modifications, whether created in whole or in part + by me, under the same open source license (unless I am + permitted to submit under a different license), as indicated + in the file; or + +* (c) The contribution was provided directly to me by some other + person who certified (a), (b) or (c) and I have not modified + it. + +* (d) I understand and agree that this project and the contribution + are public and that a record of the contribution (including all + personal information I submit with it, including my sign-off) is + maintained indefinitely and may be redistributed consistent with + this project or the open source license(s) involved. diff --git a/gyp/LICENSE b/gyp/LICENSE index ab6b011a10..372b8a9bc0 100644 --- a/gyp/LICENSE +++ b/gyp/LICENSE @@ -1,3 +1,4 @@ +Copyright (c) 2019 Ujjwal Sharma. All rights reserved. Copyright (c) 2009 Google Inc. All rights reserved. Redistribution and use in source and binary forms, with or without diff --git a/gyp/OWNERS b/gyp/OWNERS deleted file mode 100644 index 72e8ffc0db..0000000000 --- a/gyp/OWNERS +++ /dev/null @@ -1 +0,0 @@ -* diff --git a/gyp/PRESUBMIT.py b/gyp/PRESUBMIT.py deleted file mode 100644 index e52f9d2d22..0000000000 --- a/gyp/PRESUBMIT.py +++ /dev/null @@ -1,138 +0,0 @@ -# Copyright (c) 2012 Google Inc. All rights reserved. -# Use of this source code is governed by a BSD-style license that can be -# found in the LICENSE file. - - -"""Top-level presubmit script for GYP. - -See http://dev.chromium.org/developers/how-tos/depottools/presubmit-scripts -for more details about the presubmit API built into gcl. -""" - - -PYLINT_BLACKLIST = [ - # TODO: fix me. - # From SCons, not done in google style. - 'test/lib/TestCmd.py', - 'test/lib/TestCommon.py', - 'test/lib/TestGyp.py', -] - - -PYLINT_DISABLED_WARNINGS = [ - # TODO: fix me. - # Many tests include modules they don't use. - 'W0611', - # Possible unbalanced tuple unpacking with sequence. - 'W0632', - # Attempting to unpack a non-sequence. - 'W0633', - # Include order doesn't properly include local files? - 'F0401', - # Some use of built-in names. - 'W0622', - # Some unused variables. - 'W0612', - # Operator not preceded/followed by space. - 'C0323', - 'C0322', - # Unnecessary semicolon. - 'W0301', - # Unused argument. - 'W0613', - # String has no effect (docstring in wrong place). - 'W0105', - # map/filter on lambda could be replaced by comprehension. - 'W0110', - # Use of eval. - 'W0123', - # Comma not followed by space. - 'C0324', - # Access to a protected member. - 'W0212', - # Bad indent. - 'W0311', - # Line too long. - 'C0301', - # Undefined variable. - 'E0602', - # Not exception type specified. - 'W0702', - # No member of that name. - 'E1101', - # Dangerous default {}. - 'W0102', - # Cyclic import. - 'R0401', - # Others, too many to sort. - 'W0201', 'W0232', 'E1103', 'W0621', 'W0108', 'W0223', 'W0231', - 'R0201', 'E0101', 'C0321', - # ************* Module copy - # W0104:427,12:_test.odict.__setitem__: Statement seems to have no effect - 'W0104', -] - - -def _LicenseHeader(input_api): - # Accept any year number from 2009 to the current year. - current_year = int(input_api.time.strftime('%Y')) - allowed_years = (str(s) for s in reversed(range(2009, current_year + 1))) - years_re = '(' + '|'.join(allowed_years) + ')' - - # The (c) is deprecated, but tolerate it until it's removed from all files. - return ( - r'.*? Copyright (\(c\) )?%(year)s Google Inc\. All rights reserved\.\n' - r'.*? Use of this source code is governed by a BSD-style license that ' - r'can be\n' - r'.*? found in the LICENSE file\.\n' - ) % { - 'year': years_re, - } - -def CheckChangeOnUpload(input_api, output_api): - report = [] - report.extend(input_api.canned_checks.PanProjectChecks( - input_api, output_api, license_header=_LicenseHeader(input_api))) - return report - - -def CheckChangeOnCommit(input_api, output_api): - report = [] - - report.extend(input_api.canned_checks.PanProjectChecks( - input_api, output_api, license_header=_LicenseHeader(input_api))) - report.extend(input_api.canned_checks.CheckTreeIsOpen( - input_api, output_api, - 'http://gyp-status.appspot.com/status', - 'http://gyp-status.appspot.com/current')) - - import os - import sys - old_sys_path = sys.path - try: - sys.path = ['pylib', 'test/lib'] + sys.path - blacklist = PYLINT_BLACKLIST - if sys.platform == 'win32': - blacklist = [os.path.normpath(x).replace('\\', '\\\\') - for x in PYLINT_BLACKLIST] - report.extend(input_api.canned_checks.RunPylint( - input_api, - output_api, - black_list=blacklist, - disabled_warnings=PYLINT_DISABLED_WARNINGS)) - finally: - sys.path = old_sys_path - return report - - -TRYBOTS = [ - 'linux_try', - 'mac_try', - 'win_try', -] - - -def GetPreferredTryMasters(_, change): - return { - 'client.gyp': { t: set(['defaulttests']) for t in TRYBOTS }, - } diff --git a/gyp/gyp.bat b/gyp/gyp.bat old mode 100644 new mode 100755 diff --git a/gyp/gyp_main.py b/gyp/gyp_main.py index f738e8009f..da696cfc4b 100755 --- a/gyp/gyp_main.py +++ b/gyp/gyp_main.py @@ -10,41 +10,42 @@ PY3 = bytes != str -# Below IsCygwin() function copied from pylib/gyp/common.py + def IsCygwin(): - try: - out = subprocess.Popen("uname", - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) - stdout, stderr = out.communicate() - if PY3: - stdout = stdout.decode("utf-8") - return "CYGWIN" in str(stdout) - except Exception: - return False + # Function copied from pylib/gyp/common.py + try: + out = subprocess.Popen( + "uname", stdout=subprocess.PIPE, stderr=subprocess.STDOUT + ) + stdout, stderr = out.communicate() + if PY3: + stdout = stdout.decode("utf-8") + return "CYGWIN" in str(stdout) + except Exception: + return False def UnixifyPath(path): - try: - if not IsCygwin(): - return path - out = subprocess.Popen(["cygpath", "-u", path], - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) - stdout, _ = out.communicate() - if PY3: - stdout = stdout.decode("utf-8") - return str(stdout) - except Exception: - return path + try: + if not IsCygwin(): + return path + out = subprocess.Popen( + ["cygpath", "-u", path], stdout=subprocess.PIPE, stderr=subprocess.STDOUT + ) + stdout, _ = out.communicate() + if PY3: + stdout = stdout.decode("utf-8") + return str(stdout) + except Exception: + return path # Make sure we're using the version of pylib in this repo, not one installed # elsewhere on the system. Also convert to Unix style path on Cygwin systems, # else the 'gyp' library will not be found path = UnixifyPath(sys.argv[0]) -sys.path.insert(0, os.path.join(os.path.dirname(path), 'pylib')) -import gyp +sys.path.insert(0, os.path.join(os.path.dirname(path), "pylib")) +import gyp # noqa: E402 -if __name__ == '__main__': - sys.exit(gyp.script_main()) +if __name__ == "__main__": + sys.exit(gyp.script_main()) diff --git a/gyp/pylib/gyp/MSVSNew.py b/gyp/pylib/gyp/MSVSNew.py index 740ef2c73f..04bbb3df71 100644 --- a/gyp/pylib/gyp/MSVSNew.py +++ b/gyp/pylib/gyp/MSVSNew.py @@ -12,26 +12,28 @@ import gyp.common try: - cmp + cmp except NameError: - def cmp(x, y): - return (x > y) - (x < y) + + def cmp(x, y): + return (x > y) - (x < y) + # Initialize random number generator random.seed() # GUIDs for project types ENTRY_TYPE_GUIDS = { - 'project': '{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}', - 'folder': '{2150E333-8FDC-42A3-9474-1A3956D46DE8}', + "project": "{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}", + "folder": "{2150E333-8FDC-42A3-9474-1A3956D46DE8}", } -#------------------------------------------------------------------------------ +# ------------------------------------------------------------------------------ # Helper functions -def MakeGuid(name, seed='msvs_new'): - """Returns a GUID for the specified target name. +def MakeGuid(name, seed="msvs_new"): + """Returns a GUID for the specified target name. Args: name: Target name. @@ -45,28 +47,39 @@ def MakeGuid(name, seed='msvs_new'): determine the GUID to refer to explicitly. It also means that the GUID will not change when the project for a target is rebuilt. """ - # Calculate a MD5 signature for the seed and name. - d = hashlib.md5((str(seed) + str(name)).encode('utf-8')).hexdigest().upper() - # Convert most of the signature to GUID form (discard the rest) - guid = ('{' + d[:8] + '-' + d[8:12] + '-' + d[12:16] + '-' + d[16:20] - + '-' + d[20:32] + '}') - return guid - -#------------------------------------------------------------------------------ + # Calculate a MD5 signature for the seed and name. + d = hashlib.md5((str(seed) + str(name)).encode("utf-8")).hexdigest().upper() + # Convert most of the signature to GUID form (discard the rest) + guid = ( + "{" + + d[:8] + + "-" + + d[8:12] + + "-" + + d[12:16] + + "-" + + d[16:20] + + "-" + + d[20:32] + + "}" + ) + return guid + + +# ------------------------------------------------------------------------------ class MSVSSolutionEntry(object): - def __cmp__(self, other): - # Sort by name then guid (so things are in order on vs2008). - return cmp((self.name, self.get_guid()), (other.name, other.get_guid())) + def __cmp__(self, other): + # Sort by name then guid (so things are in order on vs2008). + return cmp((self.name, self.get_guid()), (other.name, other.get_guid())) class MSVSFolder(MSVSSolutionEntry): - """Folder in a Visual Studio project or solution.""" + """Folder in a Visual Studio project or solution.""" - def __init__(self, path, name = None, entries = None, - guid = None, items = None): - """Initializes the folder. + def __init__(self, path, name=None, entries=None, guid=None, items=None): + """Initializes the folder. Args: path: Full path to the folder. @@ -77,38 +90,46 @@ def __init__(self, path, name = None, entries = None, items: List of solution items to include in the folder project. May be None, if the folder does not directly contain items. """ - if name: - self.name = name - else: - # Use last layer. - self.name = os.path.basename(path) + if name: + self.name = name + else: + # Use last layer. + self.name = os.path.basename(path) - self.path = path - self.guid = guid + self.path = path + self.guid = guid - # Copy passed lists (or set to empty lists) - self.entries = sorted(entries or [], key=attrgetter('path')) - self.items = list(items or []) + # Copy passed lists (or set to empty lists) + self.entries = sorted(entries or [], key=attrgetter("path")) + self.items = list(items or []) - self.entry_type_guid = ENTRY_TYPE_GUIDS['folder'] + self.entry_type_guid = ENTRY_TYPE_GUIDS["folder"] - def get_guid(self): - if self.guid is None: - # Use consistent guids for folders (so things don't regenerate). - self.guid = MakeGuid(self.path, seed='msvs_folder') - return self.guid + def get_guid(self): + if self.guid is None: + # Use consistent guids for folders (so things don't regenerate). + self.guid = MakeGuid(self.path, seed="msvs_folder") + return self.guid -#------------------------------------------------------------------------------ +# ------------------------------------------------------------------------------ class MSVSProject(MSVSSolutionEntry): - """Visual Studio project.""" - - def __init__(self, path, name = None, dependencies = None, guid = None, - spec = None, build_file = None, config_platform_overrides = None, - fixpath_prefix = None): - """Initializes the project. + """Visual Studio project.""" + + def __init__( + self, + path, + name=None, + dependencies=None, + guid=None, + spec=None, + build_file=None, + config_platform_overrides=None, + fixpath_prefix=None, + ): + """Initializes the project. Args: path: Absolute path to the project file. @@ -123,57 +144,59 @@ def __init__(self, path, name = None, dependencies = None, guid = None, used in place of the default for this target. fixpath_prefix: the path used to adjust the behavior of _fixpath """ - self.path = path - self.guid = guid - self.spec = spec - self.build_file = build_file - # Use project filename if name not specified - self.name = name or os.path.splitext(os.path.basename(path))[0] - - # Copy passed lists (or set to empty lists) - self.dependencies = list(dependencies or []) - - self.entry_type_guid = ENTRY_TYPE_GUIDS['project'] - - if config_platform_overrides: - self.config_platform_overrides = config_platform_overrides - else: - self.config_platform_overrides = {} - self.fixpath_prefix = fixpath_prefix - self.msbuild_toolset = None - - def set_dependencies(self, dependencies): - self.dependencies = list(dependencies or []) - - def get_guid(self): - if self.guid is None: - # Set GUID from path - # TODO(rspangler): This is fragile. - # 1. We can't just use the project filename sans path, since there could - # be multiple projects with the same base name (for example, - # foo/unittest.vcproj and bar/unittest.vcproj). - # 2. The path needs to be relative to $SOURCE_ROOT, so that the project - # GUID is the same whether it's included from base/base.sln or - # foo/bar/baz/baz.sln. - # 3. The GUID needs to be the same each time this builder is invoked, so - # that we don't need to rebuild the solution when the project changes. - # 4. We should be able to handle pre-built project files by reading the - # GUID from the files. - self.guid = MakeGuid(self.name) - return self.guid - - def set_msbuild_toolset(self, msbuild_toolset): - self.msbuild_toolset = msbuild_toolset - -#------------------------------------------------------------------------------ + self.path = path + self.guid = guid + self.spec = spec + self.build_file = build_file + # Use project filename if name not specified + self.name = name or os.path.splitext(os.path.basename(path))[0] + + # Copy passed lists (or set to empty lists) + self.dependencies = list(dependencies or []) + + self.entry_type_guid = ENTRY_TYPE_GUIDS["project"] + + if config_platform_overrides: + self.config_platform_overrides = config_platform_overrides + else: + self.config_platform_overrides = {} + self.fixpath_prefix = fixpath_prefix + self.msbuild_toolset = None + + def set_dependencies(self, dependencies): + self.dependencies = list(dependencies or []) + + def get_guid(self): + if self.guid is None: + # Set GUID from path + # TODO(rspangler): This is fragile. + # 1. We can't just use the project filename sans path, since there could + # be multiple projects with the same base name (for example, + # foo/unittest.vcproj and bar/unittest.vcproj). + # 2. The path needs to be relative to $SOURCE_ROOT, so that the project + # GUID is the same whether it's included from base/base.sln or + # foo/bar/baz/baz.sln. + # 3. The GUID needs to be the same each time this builder is invoked, so + # that we don't need to rebuild the solution when the project changes. + # 4. We should be able to handle pre-built project files by reading the + # GUID from the files. + self.guid = MakeGuid(self.name) + return self.guid + + def set_msbuild_toolset(self, msbuild_toolset): + self.msbuild_toolset = msbuild_toolset + + +# ------------------------------------------------------------------------------ class MSVSSolution(object): - """Visual Studio solution.""" + """Visual Studio solution.""" - def __init__(self, path, version, entries=None, variants=None, - websiteProperties=True): - """Initializes the solution. + def __init__( + self, path, version, entries=None, variants=None, websiteProperties=True + ): + """Initializes the solution. Args: path: Path to solution file. @@ -185,152 +208,163 @@ def __init__(self, path, version, entries=None, variants=None, websiteProperties: Flag to decide if the website properties section is generated. """ - self.path = path - self.websiteProperties = websiteProperties - self.version = version - - # Copy passed lists (or set to empty lists) - self.entries = list(entries or []) - - if variants: - # Copy passed list - self.variants = variants[:] - else: - # Use default - self.variants = ['Debug|Win32', 'Release|Win32'] - # TODO(rspangler): Need to be able to handle a mapping of solution config - # to project config. Should we be able to handle variants being a dict, - # or add a separate variant_map variable? If it's a dict, we can't - # guarantee the order of variants since dict keys aren't ordered. - - - # TODO(rspangler): Automatically write to disk for now; should delay until - # node-evaluation time. - self.Write() - - - def Write(self, writer=gyp.common.WriteOnDiff): - """Writes the solution file to disk. + self.path = path + self.websiteProperties = websiteProperties + self.version = version + + # Copy passed lists (or set to empty lists) + self.entries = list(entries or []) + + if variants: + # Copy passed list + self.variants = variants[:] + else: + # Use default + self.variants = ["Debug|Win32", "Release|Win32"] + # TODO(rspangler): Need to be able to handle a mapping of solution config + # to project config. Should we be able to handle variants being a dict, + # or add a separate variant_map variable? If it's a dict, we can't + # guarantee the order of variants since dict keys aren't ordered. + + # TODO(rspangler): Automatically write to disk for now; should delay until + # node-evaluation time. + self.Write() + + def Write(self, writer=gyp.common.WriteOnDiff): + """Writes the solution file to disk. Raises: IndexError: An entry appears multiple times. """ - # Walk the entry tree and collect all the folders and projects. - all_entries = set() - entries_to_check = self.entries[:] - while entries_to_check: - e = entries_to_check.pop(0) - - # If this entry has been visited, nothing to do. - if e in all_entries: - continue - - all_entries.add(e) - - # If this is a folder, check its entries too. - if isinstance(e, MSVSFolder): - entries_to_check += e.entries - - all_entries = sorted(all_entries, key=attrgetter('path')) - - # Open file and print header - f = writer(self.path) - f.write('Microsoft Visual Studio Solution File, ' - 'Format Version %s\r\n' % self.version.SolutionVersion()) - f.write('# %s\r\n' % self.version.Description()) - - # Project entries - sln_root = os.path.split(self.path)[0] - for e in all_entries: - relative_path = gyp.common.RelativePath(e.path, sln_root) - # msbuild does not accept an empty folder_name. - # use '.' in case relative_path is empty. - folder_name = relative_path.replace('/', '\\') or '.' - f.write('Project("%s") = "%s", "%s", "%s"\r\n' % ( - e.entry_type_guid, # Entry type GUID - e.name, # Folder name - folder_name, # Folder name (again) - e.get_guid(), # Entry GUID - )) - - # TODO(rspangler): Need a way to configure this stuff - if self.websiteProperties: - f.write('\tProjectSection(WebsiteProperties) = preProject\r\n' - '\t\tDebug.AspNetCompiler.Debug = "True"\r\n' - '\t\tRelease.AspNetCompiler.Debug = "False"\r\n' - '\tEndProjectSection\r\n') - - if isinstance(e, MSVSFolder): - if e.items: - f.write('\tProjectSection(SolutionItems) = preProject\r\n') - for i in e.items: - f.write('\t\t%s = %s\r\n' % (i, i)) - f.write('\tEndProjectSection\r\n') - - if isinstance(e, MSVSProject): - if e.dependencies: - f.write('\tProjectSection(ProjectDependencies) = postProject\r\n') - for d in e.dependencies: - f.write('\t\t%s = %s\r\n' % (d.get_guid(), d.get_guid())) - f.write('\tEndProjectSection\r\n') - - f.write('EndProject\r\n') - - # Global section - f.write('Global\r\n') - - # Configurations (variants) - f.write('\tGlobalSection(SolutionConfigurationPlatforms) = preSolution\r\n') - for v in self.variants: - f.write('\t\t%s = %s\r\n' % (v, v)) - f.write('\tEndGlobalSection\r\n') - - # Sort config guids for easier diffing of solution changes. - config_guids = [] - config_guids_overrides = {} - for e in all_entries: - if isinstance(e, MSVSProject): - config_guids.append(e.get_guid()) - config_guids_overrides[e.get_guid()] = e.config_platform_overrides - config_guids.sort() - - f.write('\tGlobalSection(ProjectConfigurationPlatforms) = postSolution\r\n') - for g in config_guids: - for v in self.variants: - nv = config_guids_overrides[g].get(v, v) - # Pick which project configuration to build for this solution - # configuration. - f.write('\t\t%s.%s.ActiveCfg = %s\r\n' % ( - g, # Project GUID - v, # Solution build configuration - nv, # Project build config for that solution config - )) - - # Enable project in this solution configuration. - f.write('\t\t%s.%s.Build.0 = %s\r\n' % ( - g, # Project GUID - v, # Solution build configuration - nv, # Project build config for that solution config - )) - f.write('\tEndGlobalSection\r\n') - - # TODO(rspangler): Should be able to configure this stuff too (though I've - # never seen this be any different) - f.write('\tGlobalSection(SolutionProperties) = preSolution\r\n') - f.write('\t\tHideSolutionNode = FALSE\r\n') - f.write('\tEndGlobalSection\r\n') - - # Folder mappings - # Omit this section if there are no folders - if any([e.entries for e in all_entries if isinstance(e, MSVSFolder)]): - f.write('\tGlobalSection(NestedProjects) = preSolution\r\n') - for e in all_entries: - if not isinstance(e, MSVSFolder): - continue # Does not apply to projects, only folders - for subentry in e.entries: - f.write('\t\t%s = %s\r\n' % (subentry.get_guid(), e.get_guid())) - f.write('\tEndGlobalSection\r\n') - - f.write('EndGlobal\r\n') - - f.close() + # Walk the entry tree and collect all the folders and projects. + all_entries = set() + entries_to_check = self.entries[:] + while entries_to_check: + e = entries_to_check.pop(0) + + # If this entry has been visited, nothing to do. + if e in all_entries: + continue + + all_entries.add(e) + + # If this is a folder, check its entries too. + if isinstance(e, MSVSFolder): + entries_to_check += e.entries + + all_entries = sorted(all_entries, key=attrgetter("path")) + + # Open file and print header + f = writer(self.path) + f.write( + "Microsoft Visual Studio Solution File, " + "Format Version %s\r\n" % self.version.SolutionVersion() + ) + f.write("# %s\r\n" % self.version.Description()) + + # Project entries + sln_root = os.path.split(self.path)[0] + for e in all_entries: + relative_path = gyp.common.RelativePath(e.path, sln_root) + # msbuild does not accept an empty folder_name. + # use '.' in case relative_path is empty. + folder_name = relative_path.replace("/", "\\") or "." + f.write( + 'Project("%s") = "%s", "%s", "%s"\r\n' + % ( + e.entry_type_guid, # Entry type GUID + e.name, # Folder name + folder_name, # Folder name (again) + e.get_guid(), # Entry GUID + ) + ) + + # TODO(rspangler): Need a way to configure this stuff + if self.websiteProperties: + f.write( + "\tProjectSection(WebsiteProperties) = preProject\r\n" + '\t\tDebug.AspNetCompiler.Debug = "True"\r\n' + '\t\tRelease.AspNetCompiler.Debug = "False"\r\n' + "\tEndProjectSection\r\n" + ) + + if isinstance(e, MSVSFolder): + if e.items: + f.write("\tProjectSection(SolutionItems) = preProject\r\n") + for i in e.items: + f.write("\t\t%s = %s\r\n" % (i, i)) + f.write("\tEndProjectSection\r\n") + + if isinstance(e, MSVSProject): + if e.dependencies: + f.write("\tProjectSection(ProjectDependencies) = postProject\r\n") + for d in e.dependencies: + f.write("\t\t%s = %s\r\n" % (d.get_guid(), d.get_guid())) + f.write("\tEndProjectSection\r\n") + + f.write("EndProject\r\n") + + # Global section + f.write("Global\r\n") + + # Configurations (variants) + f.write("\tGlobalSection(SolutionConfigurationPlatforms) = preSolution\r\n") + for v in self.variants: + f.write("\t\t%s = %s\r\n" % (v, v)) + f.write("\tEndGlobalSection\r\n") + + # Sort config guids for easier diffing of solution changes. + config_guids = [] + config_guids_overrides = {} + for e in all_entries: + if isinstance(e, MSVSProject): + config_guids.append(e.get_guid()) + config_guids_overrides[e.get_guid()] = e.config_platform_overrides + config_guids.sort() + + f.write("\tGlobalSection(ProjectConfigurationPlatforms) = postSolution\r\n") + for g in config_guids: + for v in self.variants: + nv = config_guids_overrides[g].get(v, v) + # Pick which project configuration to build for this solution + # configuration. + f.write( + "\t\t%s.%s.ActiveCfg = %s\r\n" + % ( + g, # Project GUID + v, # Solution build configuration + nv, # Project build config for that solution config + ) + ) + + # Enable project in this solution configuration. + f.write( + "\t\t%s.%s.Build.0 = %s\r\n" + % ( + g, # Project GUID + v, # Solution build configuration + nv, # Project build config for that solution config + ) + ) + f.write("\tEndGlobalSection\r\n") + + # TODO(rspangler): Should be able to configure this stuff too (though I've + # never seen this be any different) + f.write("\tGlobalSection(SolutionProperties) = preSolution\r\n") + f.write("\t\tHideSolutionNode = FALSE\r\n") + f.write("\tEndGlobalSection\r\n") + + # Folder mappings + # Omit this section if there are no folders + if any([e.entries for e in all_entries if isinstance(e, MSVSFolder)]): + f.write("\tGlobalSection(NestedProjects) = preSolution\r\n") + for e in all_entries: + if not isinstance(e, MSVSFolder): + continue # Does not apply to projects, only folders + for subentry in e.entries: + f.write("\t\t%s = %s\r\n" % (subentry.get_guid(), e.get_guid())) + f.write("\tEndGlobalSection\r\n") + + f.write("EndGlobal\r\n") + + f.close() diff --git a/gyp/pylib/gyp/MSVSProject.py b/gyp/pylib/gyp/MSVSProject.py index db1ceede34..f953d52cd0 100644 --- a/gyp/pylib/gyp/MSVSProject.py +++ b/gyp/pylib/gyp/MSVSProject.py @@ -4,55 +4,55 @@ """Visual Studio project reader/writer.""" -import gyp.common import gyp.easy_xml as easy_xml -#------------------------------------------------------------------------------ +# ------------------------------------------------------------------------------ class Tool(object): - """Visual Studio tool.""" + """Visual Studio tool.""" - def __init__(self, name, attrs=None): - """Initializes the tool. + def __init__(self, name, attrs=None): + """Initializes the tool. Args: name: Tool name. attrs: Dict of tool attributes; may be None. """ - self._attrs = attrs or {} - self._attrs['Name'] = name + self._attrs = attrs or {} + self._attrs["Name"] = name - def _GetSpecification(self): - """Creates an element for the tool. + def _GetSpecification(self): + """Creates an element for the tool. Returns: A new xml.dom.Element for the tool. """ - return ['Tool', self._attrs] + return ["Tool", self._attrs] + class Filter(object): - """Visual Studio filter - that is, a virtual folder.""" + """Visual Studio filter - that is, a virtual folder.""" - def __init__(self, name, contents=None): - """Initializes the folder. + def __init__(self, name, contents=None): + """Initializes the folder. Args: name: Filter (folder) name. contents: List of filenames and/or Filter objects contained. """ - self.name = name - self.contents = list(contents or []) + self.name = name + self.contents = list(contents or []) -#------------------------------------------------------------------------------ +# ------------------------------------------------------------------------------ class Writer(object): - """Visual Studio XML project writer.""" + """Visual Studio XML project writer.""" - def __init__(self, project_path, version, name, guid=None, platforms=None): - """Initializes the project. + def __init__(self, project_path, version, name, guid=None, platforms=None): + """Initializes the project. Args: project_path: Path to the project file. @@ -61,36 +61,36 @@ def __init__(self, project_path, version, name, guid=None, platforms=None): guid: GUID to use for project, if not None. platforms: Array of string, the supported platforms. If null, ['Win32'] """ - self.project_path = project_path - self.version = version - self.name = name - self.guid = guid + self.project_path = project_path + self.version = version + self.name = name + self.guid = guid - # Default to Win32 for platforms. - if not platforms: - platforms = ['Win32'] + # Default to Win32 for platforms. + if not platforms: + platforms = ["Win32"] - # Initialize the specifications of the various sections. - self.platform_section = ['Platforms'] - for platform in platforms: - self.platform_section.append(['Platform', {'Name': platform}]) - self.tool_files_section = ['ToolFiles'] - self.configurations_section = ['Configurations'] - self.files_section = ['Files'] + # Initialize the specifications of the various sections. + self.platform_section = ["Platforms"] + for platform in platforms: + self.platform_section.append(["Platform", {"Name": platform}]) + self.tool_files_section = ["ToolFiles"] + self.configurations_section = ["Configurations"] + self.files_section = ["Files"] - # Keep a dict keyed on filename to speed up access. - self.files_dict = dict() + # Keep a dict keyed on filename to speed up access. + self.files_dict = dict() - def AddToolFile(self, path): - """Adds a tool file to the project. + def AddToolFile(self, path): + """Adds a tool file to the project. Args: path: Relative path from project to tool file. """ - self.tool_files_section.append(['ToolFile', {'RelativePath': path}]) + self.tool_files_section.append(["ToolFile", {"RelativePath": path}]) - def _GetSpecForConfiguration(self, config_type, config_name, attrs, tools): - """Returns the specification for a configuration. + def _GetSpecForConfiguration(self, config_type, config_name, attrs, tools): + """Returns the specification for a configuration. Args: config_type: Type of configuration node. @@ -99,40 +99,39 @@ def _GetSpecForConfiguration(self, config_type, config_name, attrs, tools): tools: List of tools (strings or Tool objects); may be None. Returns: """ - # Handle defaults - if not attrs: - attrs = {} - if not tools: - tools = [] - - # Add configuration node and its attributes - node_attrs = attrs.copy() - node_attrs['Name'] = config_name - specification = [config_type, node_attrs] - - # Add tool nodes and their attributes - if tools: - for t in tools: - if isinstance(t, Tool): - specification.append(t._GetSpecification()) - else: - specification.append(Tool(t)._GetSpecification()) - return specification - - - def AddConfig(self, name, attrs=None, tools=None): - """Adds a configuration to the project. + # Handle defaults + if not attrs: + attrs = {} + if not tools: + tools = [] + + # Add configuration node and its attributes + node_attrs = attrs.copy() + node_attrs["Name"] = config_name + specification = [config_type, node_attrs] + + # Add tool nodes and their attributes + if tools: + for t in tools: + if isinstance(t, Tool): + specification.append(t._GetSpecification()) + else: + specification.append(Tool(t)._GetSpecification()) + return specification + + def AddConfig(self, name, attrs=None, tools=None): + """Adds a configuration to the project. Args: name: Configuration name. attrs: Dict of configuration attributes; may be None. tools: List of tools (strings or Tool objects); may be None. """ - spec = self._GetSpecForConfiguration('Configuration', name, attrs, tools) - self.configurations_section.append(spec) + spec = self._GetSpecForConfiguration("Configuration", name, attrs, tools) + self.configurations_section.append(spec) - def _AddFilesToNode(self, parent, files): - """Adds files and/or filters to the parent node. + def _AddFilesToNode(self, parent, files): + """Adds files and/or filters to the parent node. Args: parent: Destination node @@ -140,17 +139,17 @@ def _AddFilesToNode(self, parent, files): Will call itself recursively, if the files list contains Filter objects. """ - for f in files: - if isinstance(f, Filter): - node = ['Filter', {'Name': f.name}] - self._AddFilesToNode(node, f.contents) - else: - node = ['File', {'RelativePath': f}] - self.files_dict[f] = node - parent.append(node) - - def AddFiles(self, files): - """Adds files to the project. + for f in files: + if isinstance(f, Filter): + node = ["Filter", {"Name": f.name}] + self._AddFilesToNode(node, f.contents) + else: + node = ["File", {"RelativePath": f}] + self.files_dict[f] = node + parent.append(node) + + def AddFiles(self, files): + """Adds files to the project. Args: files: A list of Filter objects and/or relative paths to files. @@ -159,12 +158,12 @@ def AddFiles(self, files): later add files to a Filter object which was passed into a previous call to AddFiles(), it will not be reflected in this project. """ - self._AddFilesToNode(self.files_section, files) - # TODO(rspangler) This also doesn't handle adding files to an existing - # filter. That is, it doesn't merge the trees. + self._AddFilesToNode(self.files_section, files) + # TODO(rspangler) This also doesn't handle adding files to an existing + # filter. That is, it doesn't merge the trees. - def AddFileConfig(self, path, config, attrs=None, tools=None): - """Adds a configuration to a file. + def AddFileConfig(self, path, config, attrs=None, tools=None): + """Adds a configuration to a file. Args: path: Relative path to the file. @@ -175,34 +174,33 @@ def AddFileConfig(self, path, config, attrs=None, tools=None): Raises: ValueError: Relative path does not match any file added via AddFiles(). """ - # Find the file node with the right relative path - parent = self.files_dict.get(path) - if not parent: - raise ValueError('AddFileConfig: file "%s" not in project.' % path) - - # Add the config to the file node - spec = self._GetSpecForConfiguration('FileConfiguration', config, attrs, - tools) - parent.append(spec) - - def WriteIfChanged(self): - """Writes the project file.""" - # First create XML content definition - content = [ - 'VisualStudioProject', - {'ProjectType': 'Visual C++', - 'Version': self.version.ProjectVersion(), - 'Name': self.name, - 'ProjectGUID': self.guid, - 'RootNamespace': self.name, - 'Keyword': 'Win32Proj' - }, - self.platform_section, - self.tool_files_section, - self.configurations_section, - ['References'], # empty section - self.files_section, - ['Globals'] # empty section - ] - easy_xml.WriteXmlIfChanged(content, self.project_path, - encoding="Windows-1252") + # Find the file node with the right relative path + parent = self.files_dict.get(path) + if not parent: + raise ValueError('AddFileConfig: file "%s" not in project.' % path) + + # Add the config to the file node + spec = self._GetSpecForConfiguration("FileConfiguration", config, attrs, tools) + parent.append(spec) + + def WriteIfChanged(self): + """Writes the project file.""" + # First create XML content definition + content = [ + "VisualStudioProject", + { + "ProjectType": "Visual C++", + "Version": self.version.ProjectVersion(), + "Name": self.name, + "ProjectGUID": self.guid, + "RootNamespace": self.name, + "Keyword": "Win32Proj", + }, + self.platform_section, + self.tool_files_section, + self.configurations_section, + ["References"], # empty section + self.files_section, + ["Globals"], # empty section + ] + easy_xml.WriteXmlIfChanged(content, self.project_path, encoding="Windows-1252") diff --git a/gyp/pylib/gyp/MSVSSettings.py b/gyp/pylib/gyp/MSVSSettings.py index 5dd8f8c1e6..6ef16f2a0b 100644 --- a/gyp/pylib/gyp/MSVSSettings.py +++ b/gyp/pylib/gyp/MSVSSettings.py @@ -37,42 +37,42 @@ class _Tool(object): - """Represents a tool used by MSVS or MSBuild. + """Represents a tool used by MSVS or MSBuild. Attributes: msvs_name: The name of the tool in MSVS. msbuild_name: The name of the tool in MSBuild. """ - def __init__(self, msvs_name, msbuild_name): - self.msvs_name = msvs_name - self.msbuild_name = msbuild_name + def __init__(self, msvs_name, msbuild_name): + self.msvs_name = msvs_name + self.msbuild_name = msbuild_name def _AddTool(tool): - """Adds a tool to the four dictionaries used to process settings. + """Adds a tool to the four dictionaries used to process settings. This only defines the tool. Each setting also needs to be added. Args: tool: The _Tool object to be added. """ - _msvs_validators[tool.msvs_name] = {} - _msbuild_validators[tool.msbuild_name] = {} - _msvs_to_msbuild_converters[tool.msvs_name] = {} - _msbuild_name_of_tool[tool.msvs_name] = tool.msbuild_name + _msvs_validators[tool.msvs_name] = {} + _msbuild_validators[tool.msbuild_name] = {} + _msvs_to_msbuild_converters[tool.msvs_name] = {} + _msbuild_name_of_tool[tool.msvs_name] = tool.msbuild_name def _GetMSBuildToolSettings(msbuild_settings, tool): - """Returns an MSBuild tool dictionary. Creates it if needed.""" - return msbuild_settings.setdefault(tool.msbuild_name, {}) + """Returns an MSBuild tool dictionary. Creates it if needed.""" + return msbuild_settings.setdefault(tool.msbuild_name, {}) class _Type(object): - """Type of settings (Base class).""" + """Type of settings (Base class).""" - def ValidateMSVS(self, value): - """Verifies that the value is legal for MSVS. + def ValidateMSVS(self, value): + """Verifies that the value is legal for MSVS. Args: value: the value to check for this type. @@ -81,8 +81,8 @@ def ValidateMSVS(self, value): ValueError if value is not valid for MSVS. """ - def ValidateMSBuild(self, value): - """Verifies that the value is legal for MSBuild. + def ValidateMSBuild(self, value): + """Verifies that the value is legal for MSBuild. Args: value: the value to check for this type. @@ -91,8 +91,8 @@ def ValidateMSBuild(self, value): ValueError if value is not valid for MSBuild. """ - def ConvertToMSBuild(self, value): - """Returns the MSBuild equivalent of the MSVS value given. + def ConvertToMSBuild(self, value): + """Returns the MSBuild equivalent of the MSVS value given. Args: value: the MSVS value to convert. @@ -103,84 +103,84 @@ def ConvertToMSBuild(self, value): Raises: ValueError if value is not valid. """ - return value + return value class _String(_Type): - """A setting that's just a string.""" + """A setting that's just a string.""" - def ValidateMSVS(self, value): - if not isinstance(value, string_types): - raise ValueError('expected string; got %r' % value) + def ValidateMSVS(self, value): + if not isinstance(value, string_types): + raise ValueError("expected string; got %r" % value) - def ValidateMSBuild(self, value): - if not isinstance(value, string_types): - raise ValueError('expected string; got %r' % value) + def ValidateMSBuild(self, value): + if not isinstance(value, string_types): + raise ValueError("expected string; got %r" % value) - def ConvertToMSBuild(self, value): - # Convert the macros - return ConvertVCMacrosToMSBuild(value) + def ConvertToMSBuild(self, value): + # Convert the macros + return ConvertVCMacrosToMSBuild(value) class _StringList(_Type): - """A settings that's a list of strings.""" + """A settings that's a list of strings.""" - def ValidateMSVS(self, value): - if not isinstance(value, string_types) and not isinstance(value, list): - raise ValueError('expected string list; got %r' % value) + def ValidateMSVS(self, value): + if not isinstance(value, string_types) and not isinstance(value, list): + raise ValueError("expected string list; got %r" % value) - def ValidateMSBuild(self, value): - if not isinstance(value, string_types) and not isinstance(value, list): - raise ValueError('expected string list; got %r' % value) + def ValidateMSBuild(self, value): + if not isinstance(value, string_types) and not isinstance(value, list): + raise ValueError("expected string list; got %r" % value) - def ConvertToMSBuild(self, value): - # Convert the macros - if isinstance(value, list): - return [ConvertVCMacrosToMSBuild(i) for i in value] - else: - return ConvertVCMacrosToMSBuild(value) + def ConvertToMSBuild(self, value): + # Convert the macros + if isinstance(value, list): + return [ConvertVCMacrosToMSBuild(i) for i in value] + else: + return ConvertVCMacrosToMSBuild(value) class _Boolean(_Type): - """Boolean settings, can have the values 'false' or 'true'.""" + """Boolean settings, can have the values 'false' or 'true'.""" - def _Validate(self, value): - if value != 'true' and value != 'false': - raise ValueError('expected bool; got %r' % value) + def _Validate(self, value): + if value != "true" and value != "false": + raise ValueError("expected bool; got %r" % value) - def ValidateMSVS(self, value): - self._Validate(value) + def ValidateMSVS(self, value): + self._Validate(value) - def ValidateMSBuild(self, value): - self._Validate(value) + def ValidateMSBuild(self, value): + self._Validate(value) - def ConvertToMSBuild(self, value): - self._Validate(value) - return value + def ConvertToMSBuild(self, value): + self._Validate(value) + return value class _Integer(_Type): - """Integer settings.""" + """Integer settings.""" - def __init__(self, msbuild_base=10): - _Type.__init__(self) - self._msbuild_base = msbuild_base + def __init__(self, msbuild_base=10): + _Type.__init__(self) + self._msbuild_base = msbuild_base - def ValidateMSVS(self, value): - # Try to convert, this will raise ValueError if invalid. - self.ConvertToMSBuild(value) + def ValidateMSVS(self, value): + # Try to convert, this will raise ValueError if invalid. + self.ConvertToMSBuild(value) - def ValidateMSBuild(self, value): - # Try to convert, this will raise ValueError if invalid. - int(value, self._msbuild_base) + def ValidateMSBuild(self, value): + # Try to convert, this will raise ValueError if invalid. + int(value, self._msbuild_base) - def ConvertToMSBuild(self, value): - msbuild_format = (self._msbuild_base == 10) and '%d' or '0x%04x' - return msbuild_format % int(value) + def ConvertToMSBuild(self, value): + msbuild_format = (self._msbuild_base == 10) and "%d" or "0x%04x" + return msbuild_format % int(value) class _Enumeration(_Type): - """Type of settings that is an enumeration. + """Type of settings that is an enumeration. In MSVS, the values are indexes like '0', '1', and '2'. MSBuild uses text labels that are more representative, like 'Win32'. @@ -192,31 +192,32 @@ class _Enumeration(_Type): new: an array of labels that are new to MSBuild. """ - def __init__(self, label_list, new=None): - _Type.__init__(self) - self._label_list = label_list - self._msbuild_values = set(value for value in label_list - if value is not None) - if new is not None: - self._msbuild_values.update(new) - - def ValidateMSVS(self, value): - # Try to convert. It will raise an exception if not valid. - self.ConvertToMSBuild(value) - - def ValidateMSBuild(self, value): - if value not in self._msbuild_values: - raise ValueError('unrecognized enumerated value %s' % value) - - def ConvertToMSBuild(self, value): - index = int(value) - if index < 0 or index >= len(self._label_list): - raise ValueError('index value (%d) not in expected range [0, %d)' % - (index, len(self._label_list))) - label = self._label_list[index] - if label is None: - raise ValueError('converted value for %s not specified.' % value) - return label + def __init__(self, label_list, new=None): + _Type.__init__(self) + self._label_list = label_list + self._msbuild_values = set(value for value in label_list if value is not None) + if new is not None: + self._msbuild_values.update(new) + + def ValidateMSVS(self, value): + # Try to convert. It will raise an exception if not valid. + self.ConvertToMSBuild(value) + + def ValidateMSBuild(self, value): + if value not in self._msbuild_values: + raise ValueError("unrecognized enumerated value %s" % value) + + def ConvertToMSBuild(self, value): + index = int(value) + if index < 0 or index >= len(self._label_list): + raise ValueError( + "index value (%d) not in expected range [0, %d)" + % (index, len(self._label_list)) + ) + label = self._label_list[index] + if label is None: + raise ValueError("converted value for %s not specified." % value) + return label # Instantiate the various generic types. @@ -231,22 +232,22 @@ def ConvertToMSBuild(self, value): _string_list = _StringList() # Some boolean settings went from numerical values to boolean. The # mapping is 0: default, 1: false, 2: true. -_newly_boolean = _Enumeration(['', 'false', 'true']) +_newly_boolean = _Enumeration(["", "false", "true"]) def _Same(tool, name, setting_type): - """Defines a setting that has the same name in MSVS and MSBuild. + """Defines a setting that has the same name in MSVS and MSBuild. Args: tool: a dictionary that gives the names of the tool for MSVS and MSBuild. name: the name of the setting. setting_type: the type of this setting. """ - _Renamed(tool, name, name, setting_type) + _Renamed(tool, name, name, setting_type) def _Renamed(tool, msvs_name, msbuild_name, setting_type): - """Defines a setting for which the name has changed. + """Defines a setting for which the name has changed. Args: tool: a dictionary that gives the names of the tool for MSVS and MSBuild. @@ -255,24 +256,25 @@ def _Renamed(tool, msvs_name, msbuild_name, setting_type): setting_type: the type of this setting. """ - def _Translate(value, msbuild_settings): - msbuild_tool_settings = _GetMSBuildToolSettings(msbuild_settings, tool) - msbuild_tool_settings[msbuild_name] = setting_type.ConvertToMSBuild(value) + def _Translate(value, msbuild_settings): + msbuild_tool_settings = _GetMSBuildToolSettings(msbuild_settings, tool) + msbuild_tool_settings[msbuild_name] = setting_type.ConvertToMSBuild(value) - _msvs_validators[tool.msvs_name][msvs_name] = setting_type.ValidateMSVS - _msbuild_validators[tool.msbuild_name][msbuild_name] = ( - setting_type.ValidateMSBuild) - _msvs_to_msbuild_converters[tool.msvs_name][msvs_name] = _Translate + _msvs_validators[tool.msvs_name][msvs_name] = setting_type.ValidateMSVS + _msbuild_validators[tool.msbuild_name][msbuild_name] = setting_type.ValidateMSBuild + _msvs_to_msbuild_converters[tool.msvs_name][msvs_name] = _Translate def _Moved(tool, settings_name, msbuild_tool_name, setting_type): - _MovedAndRenamed(tool, settings_name, msbuild_tool_name, settings_name, - setting_type) + _MovedAndRenamed( + tool, settings_name, msbuild_tool_name, settings_name, setting_type + ) -def _MovedAndRenamed(tool, msvs_settings_name, msbuild_tool_name, - msbuild_settings_name, setting_type): - """Defines a setting that may have moved to a new section. +def _MovedAndRenamed( + tool, msvs_settings_name, msbuild_tool_name, msbuild_settings_name, setting_type +): + """Defines a setting that may have moved to a new section. Args: tool: a dictionary that gives the names of the tool for MSVS and MSBuild. @@ -282,19 +284,18 @@ def _MovedAndRenamed(tool, msvs_settings_name, msbuild_tool_name, setting_type: the type of this setting. """ - def _Translate(value, msbuild_settings): - tool_settings = msbuild_settings.setdefault(msbuild_tool_name, {}) - tool_settings[msbuild_settings_name] = setting_type.ConvertToMSBuild(value) + def _Translate(value, msbuild_settings): + tool_settings = msbuild_settings.setdefault(msbuild_tool_name, {}) + tool_settings[msbuild_settings_name] = setting_type.ConvertToMSBuild(value) - _msvs_validators[tool.msvs_name][msvs_settings_name] = ( - setting_type.ValidateMSVS) - validator = setting_type.ValidateMSBuild - _msbuild_validators[msbuild_tool_name][msbuild_settings_name] = validator - _msvs_to_msbuild_converters[tool.msvs_name][msvs_settings_name] = _Translate + _msvs_validators[tool.msvs_name][msvs_settings_name] = setting_type.ValidateMSVS + validator = setting_type.ValidateMSBuild + _msbuild_validators[msbuild_tool_name][msbuild_settings_name] = validator + _msvs_to_msbuild_converters[tool.msvs_name][msvs_settings_name] = _Translate def _MSVSOnly(tool, name, setting_type): - """Defines a setting that is only found in MSVS. + """Defines a setting that is only found in MSVS. Args: tool: a dictionary that gives the names of the tool for MSVS and MSBuild. @@ -302,16 +303,16 @@ def _MSVSOnly(tool, name, setting_type): setting_type: the type of this setting. """ - def _Translate(unused_value, unused_msbuild_settings): - # Since this is for MSVS only settings, no translation will happen. - pass + def _Translate(unused_value, unused_msbuild_settings): + # Since this is for MSVS only settings, no translation will happen. + pass - _msvs_validators[tool.msvs_name][name] = setting_type.ValidateMSVS - _msvs_to_msbuild_converters[tool.msvs_name][name] = _Translate + _msvs_validators[tool.msvs_name][name] = setting_type.ValidateMSVS + _msvs_to_msbuild_converters[tool.msvs_name][name] = _Translate def _MSBuildOnly(tool, name, setting_type): - """Defines a setting that is only found in MSBuild. + """Defines a setting that is only found in MSBuild. Args: tool: a dictionary that gives the names of the tool for MSVS and MSBuild. @@ -319,17 +320,17 @@ def _MSBuildOnly(tool, name, setting_type): setting_type: the type of this setting. """ - def _Translate(value, msbuild_settings): - # Let msbuild-only properties get translated as-is from msvs_settings. - tool_settings = msbuild_settings.setdefault(tool.msbuild_name, {}) - tool_settings[name] = value + def _Translate(value, msbuild_settings): + # Let msbuild-only properties get translated as-is from msvs_settings. + tool_settings = msbuild_settings.setdefault(tool.msbuild_name, {}) + tool_settings[name] = value - _msbuild_validators[tool.msbuild_name][name] = setting_type.ValidateMSBuild - _msvs_to_msbuild_converters[tool.msvs_name][name] = _Translate + _msbuild_validators[tool.msbuild_name][name] = setting_type.ValidateMSBuild + _msvs_to_msbuild_converters[tool.msvs_name][name] = _Translate def _ConvertedToAdditionalOption(tool, msvs_name, flag): - """Defines a setting that's handled via a command line option in MSBuild. + """Defines a setting that's handled via a command line option in MSBuild. Args: tool: a dictionary that gives the names of the tool for MSVS and MSBuild. @@ -337,53 +338,55 @@ def _ConvertedToAdditionalOption(tool, msvs_name, flag): flag: the flag to insert at the end of the AdditionalOptions """ - def _Translate(value, msbuild_settings): - if value == 'true': - tool_settings = _GetMSBuildToolSettings(msbuild_settings, tool) - if 'AdditionalOptions' in tool_settings: - new_flags = '%s %s' % (tool_settings['AdditionalOptions'], flag) - else: - new_flags = flag - tool_settings['AdditionalOptions'] = new_flags - _msvs_validators[tool.msvs_name][msvs_name] = _boolean.ValidateMSVS - _msvs_to_msbuild_converters[tool.msvs_name][msvs_name] = _Translate + def _Translate(value, msbuild_settings): + if value == "true": + tool_settings = _GetMSBuildToolSettings(msbuild_settings, tool) + if "AdditionalOptions" in tool_settings: + new_flags = "%s %s" % (tool_settings["AdditionalOptions"], flag) + else: + new_flags = flag + tool_settings["AdditionalOptions"] = new_flags + + _msvs_validators[tool.msvs_name][msvs_name] = _boolean.ValidateMSVS + _msvs_to_msbuild_converters[tool.msvs_name][msvs_name] = _Translate def _CustomGeneratePreprocessedFile(tool, msvs_name): - def _Translate(value, msbuild_settings): - tool_settings = _GetMSBuildToolSettings(msbuild_settings, tool) - if value == '0': - tool_settings['PreprocessToFile'] = 'false' - tool_settings['PreprocessSuppressLineNumbers'] = 'false' - elif value == '1': # /P - tool_settings['PreprocessToFile'] = 'true' - tool_settings['PreprocessSuppressLineNumbers'] = 'false' - elif value == '2': # /EP /P - tool_settings['PreprocessToFile'] = 'true' - tool_settings['PreprocessSuppressLineNumbers'] = 'true' - else: - raise ValueError('value must be one of [0, 1, 2]; got %s' % value) - # Create a bogus validator that looks for '0', '1', or '2' - msvs_validator = _Enumeration(['a', 'b', 'c']).ValidateMSVS - _msvs_validators[tool.msvs_name][msvs_name] = msvs_validator - msbuild_validator = _boolean.ValidateMSBuild - msbuild_tool_validators = _msbuild_validators[tool.msbuild_name] - msbuild_tool_validators['PreprocessToFile'] = msbuild_validator - msbuild_tool_validators['PreprocessSuppressLineNumbers'] = msbuild_validator - _msvs_to_msbuild_converters[tool.msvs_name][msvs_name] = _Translate - - -fix_vc_macro_slashes_regex_list = ('IntDir', 'OutDir') + def _Translate(value, msbuild_settings): + tool_settings = _GetMSBuildToolSettings(msbuild_settings, tool) + if value == "0": + tool_settings["PreprocessToFile"] = "false" + tool_settings["PreprocessSuppressLineNumbers"] = "false" + elif value == "1": # /P + tool_settings["PreprocessToFile"] = "true" + tool_settings["PreprocessSuppressLineNumbers"] = "false" + elif value == "2": # /EP /P + tool_settings["PreprocessToFile"] = "true" + tool_settings["PreprocessSuppressLineNumbers"] = "true" + else: + raise ValueError("value must be one of [0, 1, 2]; got %s" % value) + + # Create a bogus validator that looks for '0', '1', or '2' + msvs_validator = _Enumeration(["a", "b", "c"]).ValidateMSVS + _msvs_validators[tool.msvs_name][msvs_name] = msvs_validator + msbuild_validator = _boolean.ValidateMSBuild + msbuild_tool_validators = _msbuild_validators[tool.msbuild_name] + msbuild_tool_validators["PreprocessToFile"] = msbuild_validator + msbuild_tool_validators["PreprocessSuppressLineNumbers"] = msbuild_validator + _msvs_to_msbuild_converters[tool.msvs_name][msvs_name] = _Translate + + +fix_vc_macro_slashes_regex_list = ("IntDir", "OutDir") fix_vc_macro_slashes_regex = re.compile( - r'(\$\((?:%s)\))(?:[\\/]+)' % "|".join(fix_vc_macro_slashes_regex_list) + r"(\$\((?:%s)\))(?:[\\/]+)" % "|".join(fix_vc_macro_slashes_regex_list) ) # Regular expression to detect keys that were generated by exclusion lists -_EXCLUDED_SUFFIX_RE = re.compile('^(.*)_excluded$') +_EXCLUDED_SUFFIX_RE = re.compile("^(.*)_excluded$") def _ValidateExclusionSetting(setting, settings, error_msg, stderr=sys.stderr): - """Verify that 'setting' is valid if it is generated from an exclusion list. + """Verify that 'setting' is valid if it is generated from an exclusion list. If the setting appears to be generated from an exclusion list, the root name is checked. @@ -394,57 +397,57 @@ def _ValidateExclusionSetting(setting, settings, error_msg, stderr=sys.stderr): error_msg: The message to emit in the event of error stderr: The stream receiving the error messages. """ - # This may be unrecognized because it's an exclusion list. If the - # setting name has the _excluded suffix, then check the root name. - unrecognized = True - m = re.match(_EXCLUDED_SUFFIX_RE, setting) - if m: - root_setting = m.group(1) - unrecognized = root_setting not in settings + # This may be unrecognized because it's an exclusion list. If the + # setting name has the _excluded suffix, then check the root name. + unrecognized = True + m = re.match(_EXCLUDED_SUFFIX_RE, setting) + if m: + root_setting = m.group(1) + unrecognized = root_setting not in settings - if unrecognized: - # We don't know this setting. Give a warning. - print(error_msg, file=stderr) + if unrecognized: + # We don't know this setting. Give a warning. + print(error_msg, file=stderr) def FixVCMacroSlashes(s): - """Replace macros which have excessive following slashes. + """Replace macros which have excessive following slashes. These macros are known to have a built-in trailing slash. Furthermore, many scripts hiccup on processing paths with extra slashes in the middle. This list is probably not exhaustive. Add as needed. """ - if '$' in s: - s = fix_vc_macro_slashes_regex.sub(r'\1', s) - return s + if "$" in s: + s = fix_vc_macro_slashes_regex.sub(r"\1", s) + return s def ConvertVCMacrosToMSBuild(s): - """Convert the MSVS macros found in the string to the MSBuild equivalent. + """Convert the MSVS macros found in the string to the MSBuild equivalent. This list is probably not exhaustive. Add as needed. """ - if '$' in s: - replace_map = { - '$(ConfigurationName)': '$(Configuration)', - '$(InputDir)': '%(RelativeDir)', - '$(InputExt)': '%(Extension)', - '$(InputFileName)': '%(Filename)%(Extension)', - '$(InputName)': '%(Filename)', - '$(InputPath)': '%(Identity)', - '$(ParentName)': '$(ProjectFileName)', - '$(PlatformName)': '$(Platform)', - '$(SafeInputName)': '%(Filename)', - } - for old, new in replace_map.items(): - s = s.replace(old, new) - s = FixVCMacroSlashes(s) - return s + if "$" in s: + replace_map = { + "$(ConfigurationName)": "$(Configuration)", + "$(InputDir)": "%(RelativeDir)", + "$(InputExt)": "%(Extension)", + "$(InputFileName)": "%(Filename)%(Extension)", + "$(InputName)": "%(Filename)", + "$(InputPath)": "%(Identity)", + "$(ParentName)": "$(ProjectFileName)", + "$(PlatformName)": "$(Platform)", + "$(SafeInputName)": "%(Filename)", + } + for old, new in replace_map.items(): + s = s.replace(old, new) + s = FixVCMacroSlashes(s) + return s def ConvertToMSBuildSettings(msvs_settings, stderr=sys.stderr): - """Converts MSVS settings (VS2008 and earlier) to MSBuild settings (VS2010+). + """Converts MSVS settings (VS2008 and earlier) to MSBuild settings (VS2010+). Args: msvs_settings: A dictionary. The key is the tool name. The values are @@ -456,55 +459,65 @@ def ConvertToMSBuildSettings(msvs_settings, stderr=sys.stderr): or the empty string (for the global settings). The values are themselves dictionaries of settings and their values. """ - msbuild_settings = {} - for msvs_tool_name, msvs_tool_settings in msvs_settings.items(): - if msvs_tool_name in _msvs_to_msbuild_converters: - msvs_tool = _msvs_to_msbuild_converters[msvs_tool_name] - for msvs_setting, msvs_value in msvs_tool_settings.items(): - if msvs_setting in msvs_tool: - # Invoke the translation function. - try: - msvs_tool[msvs_setting](msvs_value, msbuild_settings) - except ValueError as e: - print('Warning: while converting %s/%s to MSBuild, ' - '%s' % (msvs_tool_name, msvs_setting, e), file=stderr) + msbuild_settings = {} + for msvs_tool_name, msvs_tool_settings in msvs_settings.items(): + if msvs_tool_name in _msvs_to_msbuild_converters: + msvs_tool = _msvs_to_msbuild_converters[msvs_tool_name] + for msvs_setting, msvs_value in msvs_tool_settings.items(): + if msvs_setting in msvs_tool: + # Invoke the translation function. + try: + msvs_tool[msvs_setting](msvs_value, msbuild_settings) + except ValueError as e: + print( + "Warning: while converting %s/%s to MSBuild, " + "%s" % (msvs_tool_name, msvs_setting, e), + file=stderr, + ) + else: + _ValidateExclusionSetting( + msvs_setting, + msvs_tool, + ( + "Warning: unrecognized setting %s/%s " + "while converting to MSBuild." + % (msvs_tool_name, msvs_setting) + ), + stderr, + ) else: - _ValidateExclusionSetting(msvs_setting, - msvs_tool, - ('Warning: unrecognized setting %s/%s ' - 'while converting to MSBuild.' % - (msvs_tool_name, msvs_setting)), - stderr) - else: - print('Warning: unrecognized tool %s while converting to ' - 'MSBuild.' % msvs_tool_name, file=stderr) - return msbuild_settings + print( + "Warning: unrecognized tool %s while converting to " + "MSBuild." % msvs_tool_name, + file=stderr, + ) + return msbuild_settings def ValidateMSVSSettings(settings, stderr=sys.stderr): - """Validates that the names of the settings are valid for MSVS. + """Validates that the names of the settings are valid for MSVS. Args: settings: A dictionary. The key is the tool name. The values are themselves dictionaries of settings and their values. stderr: The stream receiving the error messages. """ - _ValidateSettings(_msvs_validators, settings, stderr) + _ValidateSettings(_msvs_validators, settings, stderr) def ValidateMSBuildSettings(settings, stderr=sys.stderr): - """Validates that the names of the settings are valid for MSBuild. + """Validates that the names of the settings are valid for MSBuild. Args: settings: A dictionary. The key is the tool name. The values are themselves dictionaries of settings and their values. stderr: The stream receiving the error messages. """ - _ValidateSettings(_msbuild_validators, settings, stderr) + _ValidateSettings(_msbuild_validators, settings, stderr) def _ValidateSettings(validators, settings, stderr): - """Validates that the settings are valid for MSBuild or MSVS. + """Validates that the settings are valid for MSBuild or MSVS. We currently only validate the names of the settings, not their values. @@ -514,36 +527,39 @@ def _ValidateSettings(validators, settings, stderr): themselves dictionaries of settings and their values. stderr: The stream receiving the error messages. """ - for tool_name in settings: - if tool_name in validators: - tool_validators = validators[tool_name] - for setting, value in settings[tool_name].items(): - if setting in tool_validators: - try: - tool_validators[setting](value) - except ValueError as e: - print('Warning: for %s/%s, %s' % - (tool_name, setting, e), file=stderr) - else: - _ValidateExclusionSetting(setting, - tool_validators, - ('Warning: unrecognized setting %s/%s' % - (tool_name, setting)), - stderr) + for tool_name in settings: + if tool_name in validators: + tool_validators = validators[tool_name] + for setting, value in settings[tool_name].items(): + if setting in tool_validators: + try: + tool_validators[setting](value) + except ValueError as e: + print( + "Warning: for %s/%s, %s" % (tool_name, setting, e), + file=stderr, + ) + else: + _ValidateExclusionSetting( + setting, + tool_validators, + ("Warning: unrecognized setting %s/%s" % (tool_name, setting)), + stderr, + ) - else: - print('Warning: unrecognized tool %s' % (tool_name), file=stderr) + else: + print("Warning: unrecognized tool %s" % (tool_name), file=stderr) # MSVS and MBuild names of the tools. -_compile = _Tool('VCCLCompilerTool', 'ClCompile') -_link = _Tool('VCLinkerTool', 'Link') -_midl = _Tool('VCMIDLTool', 'Midl') -_rc = _Tool('VCResourceCompilerTool', 'ResourceCompile') -_lib = _Tool('VCLibrarianTool', 'Lib') -_manifest = _Tool('VCManifestTool', 'Manifest') -_masm = _Tool('MASM', 'MASM') -_armasm = _Tool('ARMASM', 'ARMASM') +_compile = _Tool("VCCLCompilerTool", "ClCompile") +_link = _Tool("VCLinkerTool", "Link") +_midl = _Tool("VCMIDLTool", "Midl") +_rc = _Tool("VCResourceCompilerTool", "ResourceCompile") +_lib = _Tool("VCLibrarianTool", "Lib") +_manifest = _Tool("VCManifestTool", "Manifest") +_masm = _Tool("MASM", "MASM") +_armasm = _Tool("ARMASM", "ARMASM") _AddTool(_compile) @@ -555,9 +571,9 @@ def _ValidateSettings(validators, settings, stderr): _AddTool(_masm) _AddTool(_armasm) # Add sections only found in the MSBuild settings. -_msbuild_validators[''] = {} -_msbuild_validators['ProjectReference'] = {} -_msbuild_validators['ManifestResourceCompile'] = {} +_msbuild_validators[""] = {} +_msbuild_validators["ProjectReference"] = {} +_msbuild_validators["ManifestResourceCompile"] = {} # Descriptions of the compiler options, i.e. VCCLCompilerTool in MSVS and # ClCompile in MSBuild. @@ -565,167 +581,231 @@ def _ValidateSettings(validators, settings, stderr): # the schema of the MSBuild ClCompile settings. # Options that have the same name in MSVS and MSBuild -_Same(_compile, 'AdditionalIncludeDirectories', _folder_list) # /I -_Same(_compile, 'AdditionalOptions', _string_list) -_Same(_compile, 'AdditionalUsingDirectories', _folder_list) # /AI -_Same(_compile, 'AssemblerListingLocation', _file_name) # /Fa -_Same(_compile, 'BrowseInformationFile', _file_name) -_Same(_compile, 'BufferSecurityCheck', _boolean) # /GS -_Same(_compile, 'DisableLanguageExtensions', _boolean) # /Za -_Same(_compile, 'DisableSpecificWarnings', _string_list) # /wd -_Same(_compile, 'EnableFiberSafeOptimizations', _boolean) # /GT -_Same(_compile, 'EnablePREfast', _boolean) # /analyze Visible='false' -_Same(_compile, 'ExpandAttributedSource', _boolean) # /Fx -_Same(_compile, 'FloatingPointExceptions', _boolean) # /fp:except -_Same(_compile, 'ForceConformanceInForLoopScope', _boolean) # /Zc:forScope -_Same(_compile, 'ForcedIncludeFiles', _file_list) # /FI -_Same(_compile, 'ForcedUsingFiles', _file_list) # /FU -_Same(_compile, 'GenerateXMLDocumentationFiles', _boolean) # /doc -_Same(_compile, 'IgnoreStandardIncludePath', _boolean) # /X -_Same(_compile, 'MinimalRebuild', _boolean) # /Gm -_Same(_compile, 'OmitDefaultLibName', _boolean) # /Zl -_Same(_compile, 'OmitFramePointers', _boolean) # /Oy -_Same(_compile, 'PreprocessorDefinitions', _string_list) # /D -_Same(_compile, 'ProgramDataBaseFileName', _file_name) # /Fd -_Same(_compile, 'RuntimeTypeInfo', _boolean) # /GR -_Same(_compile, 'ShowIncludes', _boolean) # /showIncludes -_Same(_compile, 'SmallerTypeCheck', _boolean) # /RTCc -_Same(_compile, 'StringPooling', _boolean) # /GF -_Same(_compile, 'SuppressStartupBanner', _boolean) # /nologo -_Same(_compile, 'TreatWChar_tAsBuiltInType', _boolean) # /Zc:wchar_t -_Same(_compile, 'UndefineAllPreprocessorDefinitions', _boolean) # /u -_Same(_compile, 'UndefinePreprocessorDefinitions', _string_list) # /U -_Same(_compile, 'UseFullPaths', _boolean) # /FC -_Same(_compile, 'WholeProgramOptimization', _boolean) # /GL -_Same(_compile, 'XMLDocumentationFileName', _file_name) -_Same(_compile, 'CompileAsWinRT', _boolean) # /ZW - -_Same(_compile, 'AssemblerOutput', - _Enumeration(['NoListing', - 'AssemblyCode', # /FA - 'All', # /FAcs - 'AssemblyAndMachineCode', # /FAc - 'AssemblyAndSourceCode'])) # /FAs -_Same(_compile, 'BasicRuntimeChecks', - _Enumeration(['Default', - 'StackFrameRuntimeCheck', # /RTCs - 'UninitializedLocalUsageCheck', # /RTCu - 'EnableFastChecks'])) # /RTC1 -_Same(_compile, 'BrowseInformation', - _Enumeration(['false', - 'true', # /FR - 'true'])) # /Fr -_Same(_compile, 'CallingConvention', - _Enumeration(['Cdecl', # /Gd - 'FastCall', # /Gr - 'StdCall', # /Gz - 'VectorCall'])) # /Gv -_Same(_compile, 'CompileAs', - _Enumeration(['Default', - 'CompileAsC', # /TC - 'CompileAsCpp'])) # /TP -_Same(_compile, 'DebugInformationFormat', - _Enumeration(['', # Disabled - 'OldStyle', # /Z7 - None, - 'ProgramDatabase', # /Zi - 'EditAndContinue'])) # /ZI -_Same(_compile, 'EnableEnhancedInstructionSet', - _Enumeration(['NotSet', - 'StreamingSIMDExtensions', # /arch:SSE - 'StreamingSIMDExtensions2', # /arch:SSE2 - 'AdvancedVectorExtensions', # /arch:AVX (vs2012+) - 'NoExtensions', # /arch:IA32 (vs2012+) - # This one only exists in the new msbuild format. - 'AdvancedVectorExtensions2', # /arch:AVX2 (vs2013r2+) - ])) -_Same(_compile, 'ErrorReporting', - _Enumeration(['None', # /errorReport:none - 'Prompt', # /errorReport:prompt - 'Queue'], # /errorReport:queue - new=['Send'])) # /errorReport:send" -_Same(_compile, 'ExceptionHandling', - _Enumeration(['false', - 'Sync', # /EHsc - 'Async'], # /EHa - new=['SyncCThrow'])) # /EHs -_Same(_compile, 'FavorSizeOrSpeed', - _Enumeration(['Neither', - 'Speed', # /Ot - 'Size'])) # /Os -_Same(_compile, 'FloatingPointModel', - _Enumeration(['Precise', # /fp:precise - 'Strict', # /fp:strict - 'Fast'])) # /fp:fast -_Same(_compile, 'InlineFunctionExpansion', - _Enumeration(['Default', - 'OnlyExplicitInline', # /Ob1 - 'AnySuitable'], # /Ob2 - new=['Disabled'])) # /Ob0 -_Same(_compile, 'Optimization', - _Enumeration(['Disabled', # /Od - 'MinSpace', # /O1 - 'MaxSpeed', # /O2 - 'Full'])) # /Ox -_Same(_compile, 'RuntimeLibrary', - _Enumeration(['MultiThreaded', # /MT - 'MultiThreadedDebug', # /MTd - 'MultiThreadedDLL', # /MD - 'MultiThreadedDebugDLL'])) # /MDd -_Same(_compile, 'StructMemberAlignment', - _Enumeration(['Default', - '1Byte', # /Zp1 - '2Bytes', # /Zp2 - '4Bytes', # /Zp4 - '8Bytes', # /Zp8 - '16Bytes'])) # /Zp16 -_Same(_compile, 'WarningLevel', - _Enumeration(['TurnOffAllWarnings', # /W0 - 'Level1', # /W1 - 'Level2', # /W2 - 'Level3', # /W3 - 'Level4'], # /W4 - new=['EnableAllWarnings'])) # /Wall +_Same(_compile, "AdditionalIncludeDirectories", _folder_list) # /I +_Same(_compile, "AdditionalOptions", _string_list) +_Same(_compile, "AdditionalUsingDirectories", _folder_list) # /AI +_Same(_compile, "AssemblerListingLocation", _file_name) # /Fa +_Same(_compile, "BrowseInformationFile", _file_name) +_Same(_compile, "BufferSecurityCheck", _boolean) # /GS +_Same(_compile, "DisableLanguageExtensions", _boolean) # /Za +_Same(_compile, "DisableSpecificWarnings", _string_list) # /wd +_Same(_compile, "EnableFiberSafeOptimizations", _boolean) # /GT +_Same(_compile, "EnablePREfast", _boolean) # /analyze Visible='false' +_Same(_compile, "ExpandAttributedSource", _boolean) # /Fx +_Same(_compile, "FloatingPointExceptions", _boolean) # /fp:except +_Same(_compile, "ForceConformanceInForLoopScope", _boolean) # /Zc:forScope +_Same(_compile, "ForcedIncludeFiles", _file_list) # /FI +_Same(_compile, "ForcedUsingFiles", _file_list) # /FU +_Same(_compile, "GenerateXMLDocumentationFiles", _boolean) # /doc +_Same(_compile, "IgnoreStandardIncludePath", _boolean) # /X +_Same(_compile, "MinimalRebuild", _boolean) # /Gm +_Same(_compile, "OmitDefaultLibName", _boolean) # /Zl +_Same(_compile, "OmitFramePointers", _boolean) # /Oy +_Same(_compile, "PreprocessorDefinitions", _string_list) # /D +_Same(_compile, "ProgramDataBaseFileName", _file_name) # /Fd +_Same(_compile, "RuntimeTypeInfo", _boolean) # /GR +_Same(_compile, "ShowIncludes", _boolean) # /showIncludes +_Same(_compile, "SmallerTypeCheck", _boolean) # /RTCc +_Same(_compile, "StringPooling", _boolean) # /GF +_Same(_compile, "SuppressStartupBanner", _boolean) # /nologo +_Same(_compile, "TreatWChar_tAsBuiltInType", _boolean) # /Zc:wchar_t +_Same(_compile, "UndefineAllPreprocessorDefinitions", _boolean) # /u +_Same(_compile, "UndefinePreprocessorDefinitions", _string_list) # /U +_Same(_compile, "UseFullPaths", _boolean) # /FC +_Same(_compile, "WholeProgramOptimization", _boolean) # /GL +_Same(_compile, "XMLDocumentationFileName", _file_name) +_Same(_compile, "CompileAsWinRT", _boolean) # /ZW + +_Same( + _compile, + "AssemblerOutput", + _Enumeration( + [ + "NoListing", + "AssemblyCode", # /FA + "All", # /FAcs + "AssemblyAndMachineCode", # /FAc + "AssemblyAndSourceCode", + ] + ), +) # /FAs +_Same( + _compile, + "BasicRuntimeChecks", + _Enumeration( + [ + "Default", + "StackFrameRuntimeCheck", # /RTCs + "UninitializedLocalUsageCheck", # /RTCu + "EnableFastChecks", + ] + ), +) # /RTC1 +_Same( + _compile, "BrowseInformation", _Enumeration(["false", "true", "true"]) # /FR +) # /Fr +_Same( + _compile, + "CallingConvention", + _Enumeration(["Cdecl", "FastCall", "StdCall", "VectorCall"]), # /Gd # /Gr # /Gz +) # /Gv +_Same( + _compile, + "CompileAs", + _Enumeration(["Default", "CompileAsC", "CompileAsCpp"]), # /TC +) # /TP +_Same( + _compile, + "DebugInformationFormat", + _Enumeration( + [ + "", # Disabled + "OldStyle", # /Z7 + None, + "ProgramDatabase", # /Zi + "EditAndContinue", + ] + ), +) # /ZI +_Same( + _compile, + "EnableEnhancedInstructionSet", + _Enumeration( + [ + "NotSet", + "StreamingSIMDExtensions", # /arch:SSE + "StreamingSIMDExtensions2", # /arch:SSE2 + "AdvancedVectorExtensions", # /arch:AVX (vs2012+) + "NoExtensions", # /arch:IA32 (vs2012+) + # This one only exists in the new msbuild format. + "AdvancedVectorExtensions2", # /arch:AVX2 (vs2013r2+) + ] + ), +) +_Same( + _compile, + "ErrorReporting", + _Enumeration( + [ + "None", # /errorReport:none + "Prompt", # /errorReport:prompt + "Queue", + ], # /errorReport:queue + new=["Send"], + ), +) # /errorReport:send" +_Same( + _compile, + "ExceptionHandling", + _Enumeration(["false", "Sync", "Async"], new=["SyncCThrow"]), # /EHsc # /EHa +) # /EHs +_Same( + _compile, "FavorSizeOrSpeed", _Enumeration(["Neither", "Speed", "Size"]) # /Ot +) # /Os +_Same( + _compile, + "FloatingPointModel", + _Enumeration(["Precise", "Strict", "Fast"]), # /fp:precise # /fp:strict +) # /fp:fast +_Same( + _compile, + "InlineFunctionExpansion", + _Enumeration( + ["Default", "OnlyExplicitInline", "AnySuitable"], # /Ob1 # /Ob2 + new=["Disabled"], + ), +) # /Ob0 +_Same( + _compile, + "Optimization", + _Enumeration(["Disabled", "MinSpace", "MaxSpeed", "Full"]), # /Od # /O1 # /O2 +) # /Ox +_Same( + _compile, + "RuntimeLibrary", + _Enumeration( + [ + "MultiThreaded", # /MT + "MultiThreadedDebug", # /MTd + "MultiThreadedDLL", # /MD + "MultiThreadedDebugDLL", + ] + ), +) # /MDd +_Same( + _compile, + "StructMemberAlignment", + _Enumeration( + [ + "Default", + "1Byte", # /Zp1 + "2Bytes", # /Zp2 + "4Bytes", # /Zp4 + "8Bytes", # /Zp8 + "16Bytes", + ] + ), +) # /Zp16 +_Same( + _compile, + "WarningLevel", + _Enumeration( + [ + "TurnOffAllWarnings", # /W0 + "Level1", # /W1 + "Level2", # /W2 + "Level3", # /W3 + "Level4", + ], # /W4 + new=["EnableAllWarnings"], + ), +) # /Wall # Options found in MSVS that have been renamed in MSBuild. -_Renamed(_compile, 'EnableFunctionLevelLinking', 'FunctionLevelLinking', - _boolean) # /Gy -_Renamed(_compile, 'EnableIntrinsicFunctions', 'IntrinsicFunctions', - _boolean) # /Oi -_Renamed(_compile, 'KeepComments', 'PreprocessKeepComments', _boolean) # /C -_Renamed(_compile, 'ObjectFile', 'ObjectFileName', _file_name) # /Fo -_Renamed(_compile, 'OpenMP', 'OpenMPSupport', _boolean) # /openmp -_Renamed(_compile, 'PrecompiledHeaderThrough', 'PrecompiledHeaderFile', - _file_name) # Used with /Yc and /Yu -_Renamed(_compile, 'PrecompiledHeaderFile', 'PrecompiledHeaderOutputFile', - _file_name) # /Fp -_Renamed(_compile, 'UsePrecompiledHeader', 'PrecompiledHeader', - _Enumeration(['NotUsing', # VS recognized '' for this value too. - 'Create', # /Yc - 'Use'])) # /Yu -_Renamed(_compile, 'WarnAsError', 'TreatWarningAsError', _boolean) # /WX - -_ConvertedToAdditionalOption(_compile, 'DefaultCharIsUnsigned', '/J') +_Renamed( + _compile, "EnableFunctionLevelLinking", "FunctionLevelLinking", _boolean +) # /Gy +_Renamed(_compile, "EnableIntrinsicFunctions", "IntrinsicFunctions", _boolean) # /Oi +_Renamed(_compile, "KeepComments", "PreprocessKeepComments", _boolean) # /C +_Renamed(_compile, "ObjectFile", "ObjectFileName", _file_name) # /Fo +_Renamed(_compile, "OpenMP", "OpenMPSupport", _boolean) # /openmp +_Renamed( + _compile, "PrecompiledHeaderThrough", "PrecompiledHeaderFile", _file_name +) # Used with /Yc and /Yu +_Renamed( + _compile, "PrecompiledHeaderFile", "PrecompiledHeaderOutputFile", _file_name +) # /Fp +_Renamed( + _compile, + "UsePrecompiledHeader", + "PrecompiledHeader", + _Enumeration( + ["NotUsing", "Create", "Use"] # VS recognized '' for this value too. # /Yc + ), +) # /Yu +_Renamed(_compile, "WarnAsError", "TreatWarningAsError", _boolean) # /WX + +_ConvertedToAdditionalOption(_compile, "DefaultCharIsUnsigned", "/J") # MSVS options not found in MSBuild. -_MSVSOnly(_compile, 'Detect64BitPortabilityProblems', _boolean) -_MSVSOnly(_compile, 'UseUnicodeResponseFiles', _boolean) +_MSVSOnly(_compile, "Detect64BitPortabilityProblems", _boolean) +_MSVSOnly(_compile, "UseUnicodeResponseFiles", _boolean) # MSBuild options not found in MSVS. -_MSBuildOnly(_compile, 'BuildingInIDE', _boolean) -_MSBuildOnly(_compile, 'CompileAsManaged', - _Enumeration([], new=['false', - 'true'])) # /clr -_MSBuildOnly(_compile, 'CreateHotpatchableImage', _boolean) # /hotpatch -_MSBuildOnly(_compile, 'MultiProcessorCompilation', _boolean) # /MP -_MSBuildOnly(_compile, 'PreprocessOutputPath', _string) # /Fi -_MSBuildOnly(_compile, 'ProcessorNumber', _integer) # the number of processors -_MSBuildOnly(_compile, 'TrackerLogDirectory', _folder_name) -_MSBuildOnly(_compile, 'TreatSpecificWarningsAsErrors', _string_list) # /we -_MSBuildOnly(_compile, 'UseUnicodeForAssemblerListing', _boolean) # /FAu +_MSBuildOnly(_compile, "BuildingInIDE", _boolean) +_MSBuildOnly( + _compile, "CompileAsManaged", _Enumeration([], new=["false", "true"]) +) # /clr +_MSBuildOnly(_compile, "CreateHotpatchableImage", _boolean) # /hotpatch +_MSBuildOnly(_compile, "MultiProcessorCompilation", _boolean) # /MP +_MSBuildOnly(_compile, "PreprocessOutputPath", _string) # /Fi +_MSBuildOnly(_compile, "ProcessorNumber", _integer) # the number of processors +_MSBuildOnly(_compile, "TrackerLogDirectory", _folder_name) +_MSBuildOnly(_compile, "TreatSpecificWarningsAsErrors", _string_list) # /we +_MSBuildOnly(_compile, "UseUnicodeForAssemblerListing", _boolean) # /FAu # Defines a setting that needs very customized processing -_CustomGeneratePreprocessedFile(_compile, 'GeneratePreprocessedFile') +_CustomGeneratePreprocessedFile(_compile, "GeneratePreprocessedFile") # Directives for converting MSVS VCLinkerTool to MSBuild Link. @@ -733,326 +813,411 @@ def _ValidateSettings(validators, settings, stderr): # the schema of the MSBuild Link settings. # Options that have the same name in MSVS and MSBuild -_Same(_link, 'AdditionalDependencies', _file_list) -_Same(_link, 'AdditionalLibraryDirectories', _folder_list) # /LIBPATH +_Same(_link, "AdditionalDependencies", _file_list) +_Same(_link, "AdditionalLibraryDirectories", _folder_list) # /LIBPATH # /MANIFESTDEPENDENCY: -_Same(_link, 'AdditionalManifestDependencies', _file_list) -_Same(_link, 'AdditionalOptions', _string_list) -_Same(_link, 'AddModuleNamesToAssembly', _file_list) # /ASSEMBLYMODULE -_Same(_link, 'AllowIsolation', _boolean) # /ALLOWISOLATION -_Same(_link, 'AssemblyLinkResource', _file_list) # /ASSEMBLYLINKRESOURCE -_Same(_link, 'BaseAddress', _string) # /BASE -_Same(_link, 'CLRUnmanagedCodeCheck', _boolean) # /CLRUNMANAGEDCODECHECK -_Same(_link, 'DelayLoadDLLs', _file_list) # /DELAYLOAD -_Same(_link, 'DelaySign', _boolean) # /DELAYSIGN -_Same(_link, 'EmbedManagedResourceFile', _file_list) # /ASSEMBLYRESOURCE -_Same(_link, 'EnableUAC', _boolean) # /MANIFESTUAC -_Same(_link, 'EntryPointSymbol', _string) # /ENTRY -_Same(_link, 'ForceSymbolReferences', _file_list) # /INCLUDE -_Same(_link, 'FunctionOrder', _file_name) # /ORDER -_Same(_link, 'GenerateDebugInformation', _boolean) # /DEBUG -_Same(_link, 'GenerateMapFile', _boolean) # /MAP -_Same(_link, 'HeapCommitSize', _string) -_Same(_link, 'HeapReserveSize', _string) # /HEAP -_Same(_link, 'IgnoreAllDefaultLibraries', _boolean) # /NODEFAULTLIB -_Same(_link, 'IgnoreEmbeddedIDL', _boolean) # /IGNOREIDL -_Same(_link, 'ImportLibrary', _file_name) # /IMPLIB -_Same(_link, 'KeyContainer', _file_name) # /KEYCONTAINER -_Same(_link, 'KeyFile', _file_name) # /KEYFILE -_Same(_link, 'ManifestFile', _file_name) # /ManifestFile -_Same(_link, 'MapExports', _boolean) # /MAPINFO:EXPORTS -_Same(_link, 'MapFileName', _file_name) -_Same(_link, 'MergedIDLBaseFileName', _file_name) # /IDLOUT -_Same(_link, 'MergeSections', _string) # /MERGE -_Same(_link, 'MidlCommandFile', _file_name) # /MIDL -_Same(_link, 'ModuleDefinitionFile', _file_name) # /DEF -_Same(_link, 'OutputFile', _file_name) # /OUT -_Same(_link, 'PerUserRedirection', _boolean) -_Same(_link, 'Profile', _boolean) # /PROFILE -_Same(_link, 'ProfileGuidedDatabase', _file_name) # /PGD -_Same(_link, 'ProgramDatabaseFile', _file_name) # /PDB -_Same(_link, 'RegisterOutput', _boolean) -_Same(_link, 'SetChecksum', _boolean) # /RELEASE -_Same(_link, 'StackCommitSize', _string) -_Same(_link, 'StackReserveSize', _string) # /STACK -_Same(_link, 'StripPrivateSymbols', _file_name) # /PDBSTRIPPED -_Same(_link, 'SupportUnloadOfDelayLoadedDLL', _boolean) # /DELAY:UNLOAD -_Same(_link, 'SuppressStartupBanner', _boolean) # /NOLOGO -_Same(_link, 'SwapRunFromCD', _boolean) # /SWAPRUN:CD -_Same(_link, 'TurnOffAssemblyGeneration', _boolean) # /NOASSEMBLY -_Same(_link, 'TypeLibraryFile', _file_name) # /TLBOUT -_Same(_link, 'TypeLibraryResourceID', _integer) # /TLBID -_Same(_link, 'UACUIAccess', _boolean) # /uiAccess='true' -_Same(_link, 'Version', _string) # /VERSION - -_Same(_link, 'EnableCOMDATFolding', _newly_boolean) # /OPT:ICF -_Same(_link, 'FixedBaseAddress', _newly_boolean) # /FIXED -_Same(_link, 'LargeAddressAware', _newly_boolean) # /LARGEADDRESSAWARE -_Same(_link, 'OptimizeReferences', _newly_boolean) # /OPT:REF -_Same(_link, 'RandomizedBaseAddress', _newly_boolean) # /DYNAMICBASE -_Same(_link, 'TerminalServerAware', _newly_boolean) # /TSAWARE +_Same(_link, "AdditionalManifestDependencies", _file_list) +_Same(_link, "AdditionalOptions", _string_list) +_Same(_link, "AddModuleNamesToAssembly", _file_list) # /ASSEMBLYMODULE +_Same(_link, "AllowIsolation", _boolean) # /ALLOWISOLATION +_Same(_link, "AssemblyLinkResource", _file_list) # /ASSEMBLYLINKRESOURCE +_Same(_link, "BaseAddress", _string) # /BASE +_Same(_link, "CLRUnmanagedCodeCheck", _boolean) # /CLRUNMANAGEDCODECHECK +_Same(_link, "DelayLoadDLLs", _file_list) # /DELAYLOAD +_Same(_link, "DelaySign", _boolean) # /DELAYSIGN +_Same(_link, "EmbedManagedResourceFile", _file_list) # /ASSEMBLYRESOURCE +_Same(_link, "EnableUAC", _boolean) # /MANIFESTUAC +_Same(_link, "EntryPointSymbol", _string) # /ENTRY +_Same(_link, "ForceSymbolReferences", _file_list) # /INCLUDE +_Same(_link, "FunctionOrder", _file_name) # /ORDER +_Same(_link, "GenerateDebugInformation", _boolean) # /DEBUG +_Same(_link, "GenerateMapFile", _boolean) # /MAP +_Same(_link, "HeapCommitSize", _string) +_Same(_link, "HeapReserveSize", _string) # /HEAP +_Same(_link, "IgnoreAllDefaultLibraries", _boolean) # /NODEFAULTLIB +_Same(_link, "IgnoreEmbeddedIDL", _boolean) # /IGNOREIDL +_Same(_link, "ImportLibrary", _file_name) # /IMPLIB +_Same(_link, "KeyContainer", _file_name) # /KEYCONTAINER +_Same(_link, "KeyFile", _file_name) # /KEYFILE +_Same(_link, "ManifestFile", _file_name) # /ManifestFile +_Same(_link, "MapExports", _boolean) # /MAPINFO:EXPORTS +_Same(_link, "MapFileName", _file_name) +_Same(_link, "MergedIDLBaseFileName", _file_name) # /IDLOUT +_Same(_link, "MergeSections", _string) # /MERGE +_Same(_link, "MidlCommandFile", _file_name) # /MIDL +_Same(_link, "ModuleDefinitionFile", _file_name) # /DEF +_Same(_link, "OutputFile", _file_name) # /OUT +_Same(_link, "PerUserRedirection", _boolean) +_Same(_link, "Profile", _boolean) # /PROFILE +_Same(_link, "ProfileGuidedDatabase", _file_name) # /PGD +_Same(_link, "ProgramDatabaseFile", _file_name) # /PDB +_Same(_link, "RegisterOutput", _boolean) +_Same(_link, "SetChecksum", _boolean) # /RELEASE +_Same(_link, "StackCommitSize", _string) +_Same(_link, "StackReserveSize", _string) # /STACK +_Same(_link, "StripPrivateSymbols", _file_name) # /PDBSTRIPPED +_Same(_link, "SupportUnloadOfDelayLoadedDLL", _boolean) # /DELAY:UNLOAD +_Same(_link, "SuppressStartupBanner", _boolean) # /NOLOGO +_Same(_link, "SwapRunFromCD", _boolean) # /SWAPRUN:CD +_Same(_link, "TurnOffAssemblyGeneration", _boolean) # /NOASSEMBLY +_Same(_link, "TypeLibraryFile", _file_name) # /TLBOUT +_Same(_link, "TypeLibraryResourceID", _integer) # /TLBID +_Same(_link, "UACUIAccess", _boolean) # /uiAccess='true' +_Same(_link, "Version", _string) # /VERSION + +_Same(_link, "EnableCOMDATFolding", _newly_boolean) # /OPT:ICF +_Same(_link, "FixedBaseAddress", _newly_boolean) # /FIXED +_Same(_link, "LargeAddressAware", _newly_boolean) # /LARGEADDRESSAWARE +_Same(_link, "OptimizeReferences", _newly_boolean) # /OPT:REF +_Same(_link, "RandomizedBaseAddress", _newly_boolean) # /DYNAMICBASE +_Same(_link, "TerminalServerAware", _newly_boolean) # /TSAWARE _subsystem_enumeration = _Enumeration( - ['NotSet', - 'Console', # /SUBSYSTEM:CONSOLE - 'Windows', # /SUBSYSTEM:WINDOWS - 'Native', # /SUBSYSTEM:NATIVE - 'EFI Application', # /SUBSYSTEM:EFI_APPLICATION - 'EFI Boot Service Driver', # /SUBSYSTEM:EFI_BOOT_SERVICE_DRIVER - 'EFI ROM', # /SUBSYSTEM:EFI_ROM - 'EFI Runtime', # /SUBSYSTEM:EFI_RUNTIME_DRIVER - 'WindowsCE'], # /SUBSYSTEM:WINDOWSCE - new=['POSIX']) # /SUBSYSTEM:POSIX + [ + "NotSet", + "Console", # /SUBSYSTEM:CONSOLE + "Windows", # /SUBSYSTEM:WINDOWS + "Native", # /SUBSYSTEM:NATIVE + "EFI Application", # /SUBSYSTEM:EFI_APPLICATION + "EFI Boot Service Driver", # /SUBSYSTEM:EFI_BOOT_SERVICE_DRIVER + "EFI ROM", # /SUBSYSTEM:EFI_ROM + "EFI Runtime", # /SUBSYSTEM:EFI_RUNTIME_DRIVER + "WindowsCE", + ], # /SUBSYSTEM:WINDOWSCE + new=["POSIX"], +) # /SUBSYSTEM:POSIX _target_machine_enumeration = _Enumeration( - ['NotSet', - 'MachineX86', # /MACHINE:X86 - None, - 'MachineARM', # /MACHINE:ARM - 'MachineEBC', # /MACHINE:EBC - 'MachineIA64', # /MACHINE:IA64 - None, - 'MachineMIPS', # /MACHINE:MIPS - 'MachineMIPS16', # /MACHINE:MIPS16 - 'MachineMIPSFPU', # /MACHINE:MIPSFPU - 'MachineMIPSFPU16', # /MACHINE:MIPSFPU16 - None, - None, - None, - 'MachineSH4', # /MACHINE:SH4 - None, - 'MachineTHUMB', # /MACHINE:THUMB - 'MachineX64']) # /MACHINE:X64 - -_Same(_link, 'AssemblyDebug', - _Enumeration(['', - 'true', # /ASSEMBLYDEBUG - 'false'])) # /ASSEMBLYDEBUG:DISABLE -_Same(_link, 'CLRImageType', - _Enumeration(['Default', - 'ForceIJWImage', # /CLRIMAGETYPE:IJW - 'ForcePureILImage', # /Switch="CLRIMAGETYPE:PURE - 'ForceSafeILImage'])) # /Switch="CLRIMAGETYPE:SAFE -_Same(_link, 'CLRThreadAttribute', - _Enumeration(['DefaultThreadingAttribute', # /CLRTHREADATTRIBUTE:NONE - 'MTAThreadingAttribute', # /CLRTHREADATTRIBUTE:MTA - 'STAThreadingAttribute'])) # /CLRTHREADATTRIBUTE:STA -_Same(_link, 'DataExecutionPrevention', - _Enumeration(['', - 'false', # /NXCOMPAT:NO - 'true'])) # /NXCOMPAT -_Same(_link, 'Driver', - _Enumeration(['NotSet', - 'Driver', # /Driver - 'UpOnly', # /DRIVER:UPONLY - 'WDM'])) # /DRIVER:WDM -_Same(_link, 'LinkTimeCodeGeneration', - _Enumeration(['Default', - 'UseLinkTimeCodeGeneration', # /LTCG - 'PGInstrument', # /LTCG:PGInstrument - 'PGOptimization', # /LTCG:PGOptimize - 'PGUpdate'])) # /LTCG:PGUpdate -_Same(_link, 'ShowProgress', - _Enumeration(['NotSet', - 'LinkVerbose', # /VERBOSE - 'LinkVerboseLib'], # /VERBOSE:Lib - new=['LinkVerboseICF', # /VERBOSE:ICF - 'LinkVerboseREF', # /VERBOSE:REF - 'LinkVerboseSAFESEH', # /VERBOSE:SAFESEH - 'LinkVerboseCLR'])) # /VERBOSE:CLR -_Same(_link, 'SubSystem', _subsystem_enumeration) -_Same(_link, 'TargetMachine', _target_machine_enumeration) -_Same(_link, 'UACExecutionLevel', - _Enumeration(['AsInvoker', # /level='asInvoker' - 'HighestAvailable', # /level='highestAvailable' - 'RequireAdministrator'])) # /level='requireAdministrator' -_Same(_link, 'MinimumRequiredVersion', _string) -_Same(_link, 'TreatLinkerWarningAsErrors', _boolean) # /WX + [ + "NotSet", + "MachineX86", # /MACHINE:X86 + None, + "MachineARM", # /MACHINE:ARM + "MachineEBC", # /MACHINE:EBC + "MachineIA64", # /MACHINE:IA64 + None, + "MachineMIPS", # /MACHINE:MIPS + "MachineMIPS16", # /MACHINE:MIPS16 + "MachineMIPSFPU", # /MACHINE:MIPSFPU + "MachineMIPSFPU16", # /MACHINE:MIPSFPU16 + None, + None, + None, + "MachineSH4", # /MACHINE:SH4 + None, + "MachineTHUMB", # /MACHINE:THUMB + "MachineX64", + ] +) # /MACHINE:X64 + +_Same( + _link, "AssemblyDebug", _Enumeration(["", "true", "false"]) # /ASSEMBLYDEBUG +) # /ASSEMBLYDEBUG:DISABLE +_Same( + _link, + "CLRImageType", + _Enumeration( + [ + "Default", + "ForceIJWImage", # /CLRIMAGETYPE:IJW + "ForcePureILImage", # /Switch="CLRIMAGETYPE:PURE + "ForceSafeILImage", + ] + ), +) # /Switch="CLRIMAGETYPE:SAFE +_Same( + _link, + "CLRThreadAttribute", + _Enumeration( + [ + "DefaultThreadingAttribute", # /CLRTHREADATTRIBUTE:NONE + "MTAThreadingAttribute", # /CLRTHREADATTRIBUTE:MTA + "STAThreadingAttribute", + ] + ), +) # /CLRTHREADATTRIBUTE:STA +_Same( + _link, + "DataExecutionPrevention", + _Enumeration(["", "false", "true"]), # /NXCOMPAT:NO +) # /NXCOMPAT +_Same( + _link, + "Driver", + _Enumeration(["NotSet", "Driver", "UpOnly", "WDM"]), # /Driver # /DRIVER:UPONLY +) # /DRIVER:WDM +_Same( + _link, + "LinkTimeCodeGeneration", + _Enumeration( + [ + "Default", + "UseLinkTimeCodeGeneration", # /LTCG + "PGInstrument", # /LTCG:PGInstrument + "PGOptimization", # /LTCG:PGOptimize + "PGUpdate", + ] + ), +) # /LTCG:PGUpdate +_Same( + _link, + "ShowProgress", + _Enumeration( + ["NotSet", "LinkVerbose", "LinkVerboseLib"], # /VERBOSE # /VERBOSE:Lib + new=[ + "LinkVerboseICF", # /VERBOSE:ICF + "LinkVerboseREF", # /VERBOSE:REF + "LinkVerboseSAFESEH", # /VERBOSE:SAFESEH + "LinkVerboseCLR", + ], + ), +) # /VERBOSE:CLR +_Same(_link, "SubSystem", _subsystem_enumeration) +_Same(_link, "TargetMachine", _target_machine_enumeration) +_Same( + _link, + "UACExecutionLevel", + _Enumeration( + [ + "AsInvoker", # /level='asInvoker' + "HighestAvailable", # /level='highestAvailable' + "RequireAdministrator", + ] + ), +) # /level='requireAdministrator' +_Same(_link, "MinimumRequiredVersion", _string) +_Same(_link, "TreatLinkerWarningAsErrors", _boolean) # /WX # Options found in MSVS that have been renamed in MSBuild. -_Renamed(_link, 'ErrorReporting', 'LinkErrorReporting', - _Enumeration(['NoErrorReport', # /ERRORREPORT:NONE - 'PromptImmediately', # /ERRORREPORT:PROMPT - 'QueueForNextLogin'], # /ERRORREPORT:QUEUE - new=['SendErrorReport'])) # /ERRORREPORT:SEND -_Renamed(_link, 'IgnoreDefaultLibraryNames', 'IgnoreSpecificDefaultLibraries', - _file_list) # /NODEFAULTLIB -_Renamed(_link, 'ResourceOnlyDLL', 'NoEntryPoint', _boolean) # /NOENTRY -_Renamed(_link, 'SwapRunFromNet', 'SwapRunFromNET', _boolean) # /SWAPRUN:NET - -_Moved(_link, 'GenerateManifest', '', _boolean) -_Moved(_link, 'IgnoreImportLibrary', '', _boolean) -_Moved(_link, 'LinkIncremental', '', _newly_boolean) -_Moved(_link, 'LinkLibraryDependencies', 'ProjectReference', _boolean) -_Moved(_link, 'UseLibraryDependencyInputs', 'ProjectReference', _boolean) +_Renamed( + _link, + "ErrorReporting", + "LinkErrorReporting", + _Enumeration( + [ + "NoErrorReport", # /ERRORREPORT:NONE + "PromptImmediately", # /ERRORREPORT:PROMPT + "QueueForNextLogin", + ], # /ERRORREPORT:QUEUE + new=["SendErrorReport"], + ), +) # /ERRORREPORT:SEND +_Renamed( + _link, "IgnoreDefaultLibraryNames", "IgnoreSpecificDefaultLibraries", _file_list +) # /NODEFAULTLIB +_Renamed(_link, "ResourceOnlyDLL", "NoEntryPoint", _boolean) # /NOENTRY +_Renamed(_link, "SwapRunFromNet", "SwapRunFromNET", _boolean) # /SWAPRUN:NET + +_Moved(_link, "GenerateManifest", "", _boolean) +_Moved(_link, "IgnoreImportLibrary", "", _boolean) +_Moved(_link, "LinkIncremental", "", _newly_boolean) +_Moved(_link, "LinkLibraryDependencies", "ProjectReference", _boolean) +_Moved(_link, "UseLibraryDependencyInputs", "ProjectReference", _boolean) # MSVS options not found in MSBuild. -_MSVSOnly(_link, 'OptimizeForWindows98', _newly_boolean) -_MSVSOnly(_link, 'UseUnicodeResponseFiles', _boolean) +_MSVSOnly(_link, "OptimizeForWindows98", _newly_boolean) +_MSVSOnly(_link, "UseUnicodeResponseFiles", _boolean) # MSBuild options not found in MSVS. -_MSBuildOnly(_link, 'BuildingInIDE', _boolean) -_MSBuildOnly(_link, 'ImageHasSafeExceptionHandlers', _boolean) # /SAFESEH -_MSBuildOnly(_link, 'LinkDLL', _boolean) # /DLL Visible='false' -_MSBuildOnly(_link, 'LinkStatus', _boolean) # /LTCG:STATUS -_MSBuildOnly(_link, 'PreventDllBinding', _boolean) # /ALLOWBIND -_MSBuildOnly(_link, 'SupportNobindOfDelayLoadedDLL', _boolean) # /DELAY:NOBIND -_MSBuildOnly(_link, 'TrackerLogDirectory', _folder_name) -_MSBuildOnly(_link, 'MSDOSStubFileName', _file_name) # /STUB Visible='false' -_MSBuildOnly(_link, 'SectionAlignment', _integer) # /ALIGN -_MSBuildOnly(_link, 'SpecifySectionAttributes', _string) # /SECTION -_MSBuildOnly(_link, 'ForceFileOutput', - _Enumeration([], new=['Enabled', # /FORCE - # /FORCE:MULTIPLE - 'MultiplyDefinedSymbolOnly', - 'UndefinedSymbolOnly'])) # /FORCE:UNRESOLVED -_MSBuildOnly(_link, 'CreateHotPatchableImage', - _Enumeration([], new=['Enabled', # /FUNCTIONPADMIN - 'X86Image', # /FUNCTIONPADMIN:5 - 'X64Image', # /FUNCTIONPADMIN:6 - 'ItaniumImage'])) # /FUNCTIONPADMIN:16 -_MSBuildOnly(_link, 'CLRSupportLastError', - _Enumeration([], new=['Enabled', # /CLRSupportLastError - 'Disabled', # /CLRSupportLastError:NO - # /CLRSupportLastError:SYSTEMDLL - 'SystemDlls'])) +_MSBuildOnly(_link, "BuildingInIDE", _boolean) +_MSBuildOnly(_link, "ImageHasSafeExceptionHandlers", _boolean) # /SAFESEH +_MSBuildOnly(_link, "LinkDLL", _boolean) # /DLL Visible='false' +_MSBuildOnly(_link, "LinkStatus", _boolean) # /LTCG:STATUS +_MSBuildOnly(_link, "PreventDllBinding", _boolean) # /ALLOWBIND +_MSBuildOnly(_link, "SupportNobindOfDelayLoadedDLL", _boolean) # /DELAY:NOBIND +_MSBuildOnly(_link, "TrackerLogDirectory", _folder_name) +_MSBuildOnly(_link, "MSDOSStubFileName", _file_name) # /STUB Visible='false' +_MSBuildOnly(_link, "SectionAlignment", _integer) # /ALIGN +_MSBuildOnly(_link, "SpecifySectionAttributes", _string) # /SECTION +_MSBuildOnly( + _link, + "ForceFileOutput", + _Enumeration( + [], + new=[ + "Enabled", # /FORCE + # /FORCE:MULTIPLE + "MultiplyDefinedSymbolOnly", + "UndefinedSymbolOnly", + ], + ), +) # /FORCE:UNRESOLVED +_MSBuildOnly( + _link, + "CreateHotPatchableImage", + _Enumeration( + [], + new=[ + "Enabled", # /FUNCTIONPADMIN + "X86Image", # /FUNCTIONPADMIN:5 + "X64Image", # /FUNCTIONPADMIN:6 + "ItaniumImage", + ], + ), +) # /FUNCTIONPADMIN:16 +_MSBuildOnly( + _link, + "CLRSupportLastError", + _Enumeration( + [], + new=[ + "Enabled", # /CLRSupportLastError + "Disabled", # /CLRSupportLastError:NO + # /CLRSupportLastError:SYSTEMDLL + "SystemDlls", + ], + ), +) # Directives for converting VCResourceCompilerTool to ResourceCompile. # See "c:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\1033\rc.xml" for # the schema of the MSBuild ResourceCompile settings. -_Same(_rc, 'AdditionalOptions', _string_list) -_Same(_rc, 'AdditionalIncludeDirectories', _folder_list) # /I -_Same(_rc, 'Culture', _Integer(msbuild_base=16)) -_Same(_rc, 'IgnoreStandardIncludePath', _boolean) # /X -_Same(_rc, 'PreprocessorDefinitions', _string_list) # /D -_Same(_rc, 'ResourceOutputFileName', _string) # /fo -_Same(_rc, 'ShowProgress', _boolean) # /v +_Same(_rc, "AdditionalOptions", _string_list) +_Same(_rc, "AdditionalIncludeDirectories", _folder_list) # /I +_Same(_rc, "Culture", _Integer(msbuild_base=16)) +_Same(_rc, "IgnoreStandardIncludePath", _boolean) # /X +_Same(_rc, "PreprocessorDefinitions", _string_list) # /D +_Same(_rc, "ResourceOutputFileName", _string) # /fo +_Same(_rc, "ShowProgress", _boolean) # /v # There is no UI in VisualStudio 2008 to set the following properties. # However they are found in CL and other tools. Include them here for # completeness, as they are very likely to have the same usage pattern. -_Same(_rc, 'SuppressStartupBanner', _boolean) # /nologo -_Same(_rc, 'UndefinePreprocessorDefinitions', _string_list) # /u +_Same(_rc, "SuppressStartupBanner", _boolean) # /nologo +_Same(_rc, "UndefinePreprocessorDefinitions", _string_list) # /u # MSBuild options not found in MSVS. -_MSBuildOnly(_rc, 'NullTerminateStrings', _boolean) # /n -_MSBuildOnly(_rc, 'TrackerLogDirectory', _folder_name) +_MSBuildOnly(_rc, "NullTerminateStrings", _boolean) # /n +_MSBuildOnly(_rc, "TrackerLogDirectory", _folder_name) # Directives for converting VCMIDLTool to Midl. # See "c:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\1033\midl.xml" for # the schema of the MSBuild Midl settings. -_Same(_midl, 'AdditionalIncludeDirectories', _folder_list) # /I -_Same(_midl, 'AdditionalOptions', _string_list) -_Same(_midl, 'CPreprocessOptions', _string) # /cpp_opt -_Same(_midl, 'ErrorCheckAllocations', _boolean) # /error allocation -_Same(_midl, 'ErrorCheckBounds', _boolean) # /error bounds_check -_Same(_midl, 'ErrorCheckEnumRange', _boolean) # /error enum -_Same(_midl, 'ErrorCheckRefPointers', _boolean) # /error ref -_Same(_midl, 'ErrorCheckStubData', _boolean) # /error stub_data -_Same(_midl, 'GenerateStublessProxies', _boolean) # /Oicf -_Same(_midl, 'GenerateTypeLibrary', _boolean) -_Same(_midl, 'HeaderFileName', _file_name) # /h -_Same(_midl, 'IgnoreStandardIncludePath', _boolean) # /no_def_idir -_Same(_midl, 'InterfaceIdentifierFileName', _file_name) # /iid -_Same(_midl, 'MkTypLibCompatible', _boolean) # /mktyplib203 -_Same(_midl, 'OutputDirectory', _string) # /out -_Same(_midl, 'PreprocessorDefinitions', _string_list) # /D -_Same(_midl, 'ProxyFileName', _file_name) # /proxy -_Same(_midl, 'RedirectOutputAndErrors', _file_name) # /o -_Same(_midl, 'SuppressStartupBanner', _boolean) # /nologo -_Same(_midl, 'TypeLibraryName', _file_name) # /tlb -_Same(_midl, 'UndefinePreprocessorDefinitions', _string_list) # /U -_Same(_midl, 'WarnAsError', _boolean) # /WX - -_Same(_midl, 'DefaultCharType', - _Enumeration(['Unsigned', # /char unsigned - 'Signed', # /char signed - 'Ascii'])) # /char ascii7 -_Same(_midl, 'TargetEnvironment', - _Enumeration(['NotSet', - 'Win32', # /env win32 - 'Itanium', # /env ia64 - 'X64', # /env x64 - 'ARM64', # /env arm64 - ])) -_Same(_midl, 'EnableErrorChecks', - _Enumeration(['EnableCustom', - 'None', # /error none - 'All'])) # /error all -_Same(_midl, 'StructMemberAlignment', - _Enumeration(['NotSet', - '1', # Zp1 - '2', # Zp2 - '4', # Zp4 - '8'])) # Zp8 -_Same(_midl, 'WarningLevel', - _Enumeration(['0', # /W0 - '1', # /W1 - '2', # /W2 - '3', # /W3 - '4'])) # /W4 - -_Renamed(_midl, 'DLLDataFileName', 'DllDataFileName', _file_name) # /dlldata -_Renamed(_midl, 'ValidateParameters', 'ValidateAllParameters', - _boolean) # /robust +_Same(_midl, "AdditionalIncludeDirectories", _folder_list) # /I +_Same(_midl, "AdditionalOptions", _string_list) +_Same(_midl, "CPreprocessOptions", _string) # /cpp_opt +_Same(_midl, "ErrorCheckAllocations", _boolean) # /error allocation +_Same(_midl, "ErrorCheckBounds", _boolean) # /error bounds_check +_Same(_midl, "ErrorCheckEnumRange", _boolean) # /error enum +_Same(_midl, "ErrorCheckRefPointers", _boolean) # /error ref +_Same(_midl, "ErrorCheckStubData", _boolean) # /error stub_data +_Same(_midl, "GenerateStublessProxies", _boolean) # /Oicf +_Same(_midl, "GenerateTypeLibrary", _boolean) +_Same(_midl, "HeaderFileName", _file_name) # /h +_Same(_midl, "IgnoreStandardIncludePath", _boolean) # /no_def_idir +_Same(_midl, "InterfaceIdentifierFileName", _file_name) # /iid +_Same(_midl, "MkTypLibCompatible", _boolean) # /mktyplib203 +_Same(_midl, "OutputDirectory", _string) # /out +_Same(_midl, "PreprocessorDefinitions", _string_list) # /D +_Same(_midl, "ProxyFileName", _file_name) # /proxy +_Same(_midl, "RedirectOutputAndErrors", _file_name) # /o +_Same(_midl, "SuppressStartupBanner", _boolean) # /nologo +_Same(_midl, "TypeLibraryName", _file_name) # /tlb +_Same(_midl, "UndefinePreprocessorDefinitions", _string_list) # /U +_Same(_midl, "WarnAsError", _boolean) # /WX + +_Same( + _midl, + "DefaultCharType", + _Enumeration(["Unsigned", "Signed", "Ascii"]), # /char unsigned # /char signed +) # /char ascii7 +_Same( + _midl, + "TargetEnvironment", + _Enumeration( + [ + "NotSet", + "Win32", # /env win32 + "Itanium", # /env ia64 + "X64", # /env x64 + "ARM64", # /env arm64 + ] + ), +) +_Same( + _midl, + "EnableErrorChecks", + _Enumeration(["EnableCustom", "None", "All"]), # /error none +) # /error all +_Same( + _midl, + "StructMemberAlignment", + _Enumeration(["NotSet", "1", "2", "4", "8"]), # Zp1 # Zp2 # Zp4 +) # Zp8 +_Same( + _midl, + "WarningLevel", + _Enumeration(["0", "1", "2", "3", "4"]), # /W0 # /W1 # /W2 # /W3 +) # /W4 + +_Renamed(_midl, "DLLDataFileName", "DllDataFileName", _file_name) # /dlldata +_Renamed(_midl, "ValidateParameters", "ValidateAllParameters", _boolean) # /robust # MSBuild options not found in MSVS. -_MSBuildOnly(_midl, 'ApplicationConfigurationMode', _boolean) # /app_config -_MSBuildOnly(_midl, 'ClientStubFile', _file_name) # /cstub -_MSBuildOnly(_midl, 'GenerateClientFiles', - _Enumeration([], new=['Stub', # /client stub - 'None'])) # /client none -_MSBuildOnly(_midl, 'GenerateServerFiles', - _Enumeration([], new=['Stub', # /client stub - 'None'])) # /client none -_MSBuildOnly(_midl, 'LocaleID', _integer) # /lcid DECIMAL -_MSBuildOnly(_midl, 'ServerStubFile', _file_name) # /sstub -_MSBuildOnly(_midl, 'SuppressCompilerWarnings', _boolean) # /no_warn -_MSBuildOnly(_midl, 'TrackerLogDirectory', _folder_name) -_MSBuildOnly(_midl, 'TypeLibFormat', - _Enumeration([], new=['NewFormat', # /newtlb - 'OldFormat'])) # /oldtlb +_MSBuildOnly(_midl, "ApplicationConfigurationMode", _boolean) # /app_config +_MSBuildOnly(_midl, "ClientStubFile", _file_name) # /cstub +_MSBuildOnly( + _midl, "GenerateClientFiles", _Enumeration([], new=["Stub", "None"]) # /client stub +) # /client none +_MSBuildOnly( + _midl, "GenerateServerFiles", _Enumeration([], new=["Stub", "None"]) # /client stub +) # /client none +_MSBuildOnly(_midl, "LocaleID", _integer) # /lcid DECIMAL +_MSBuildOnly(_midl, "ServerStubFile", _file_name) # /sstub +_MSBuildOnly(_midl, "SuppressCompilerWarnings", _boolean) # /no_warn +_MSBuildOnly(_midl, "TrackerLogDirectory", _folder_name) +_MSBuildOnly( + _midl, "TypeLibFormat", _Enumeration([], new=["NewFormat", "OldFormat"]) # /newtlb +) # /oldtlb # Directives for converting VCLibrarianTool to Lib. # See "c:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\1033\lib.xml" for # the schema of the MSBuild Lib settings. -_Same(_lib, 'AdditionalDependencies', _file_list) -_Same(_lib, 'AdditionalLibraryDirectories', _folder_list) # /LIBPATH -_Same(_lib, 'AdditionalOptions', _string_list) -_Same(_lib, 'ExportNamedFunctions', _string_list) # /EXPORT -_Same(_lib, 'ForceSymbolReferences', _string) # /INCLUDE -_Same(_lib, 'IgnoreAllDefaultLibraries', _boolean) # /NODEFAULTLIB -_Same(_lib, 'IgnoreSpecificDefaultLibraries', _file_list) # /NODEFAULTLIB -_Same(_lib, 'ModuleDefinitionFile', _file_name) # /DEF -_Same(_lib, 'OutputFile', _file_name) # /OUT -_Same(_lib, 'SuppressStartupBanner', _boolean) # /NOLOGO -_Same(_lib, 'UseUnicodeResponseFiles', _boolean) -_Same(_lib, 'LinkTimeCodeGeneration', _boolean) # /LTCG -_Same(_lib, 'TargetMachine', _target_machine_enumeration) +_Same(_lib, "AdditionalDependencies", _file_list) +_Same(_lib, "AdditionalLibraryDirectories", _folder_list) # /LIBPATH +_Same(_lib, "AdditionalOptions", _string_list) +_Same(_lib, "ExportNamedFunctions", _string_list) # /EXPORT +_Same(_lib, "ForceSymbolReferences", _string) # /INCLUDE +_Same(_lib, "IgnoreAllDefaultLibraries", _boolean) # /NODEFAULTLIB +_Same(_lib, "IgnoreSpecificDefaultLibraries", _file_list) # /NODEFAULTLIB +_Same(_lib, "ModuleDefinitionFile", _file_name) # /DEF +_Same(_lib, "OutputFile", _file_name) # /OUT +_Same(_lib, "SuppressStartupBanner", _boolean) # /NOLOGO +_Same(_lib, "UseUnicodeResponseFiles", _boolean) +_Same(_lib, "LinkTimeCodeGeneration", _boolean) # /LTCG +_Same(_lib, "TargetMachine", _target_machine_enumeration) # TODO(jeanluc) _link defines the same value that gets moved to # ProjectReference. We may want to validate that they are consistent. -_Moved(_lib, 'LinkLibraryDependencies', 'ProjectReference', _boolean) - -_MSBuildOnly(_lib, 'DisplayLibrary', _string) # /LIST Visible='false' -_MSBuildOnly(_lib, 'ErrorReporting', - _Enumeration([], new=['PromptImmediately', # /ERRORREPORT:PROMPT - 'QueueForNextLogin', # /ERRORREPORT:QUEUE - 'SendErrorReport', # /ERRORREPORT:SEND - 'NoErrorReport'])) # /ERRORREPORT:NONE -_MSBuildOnly(_lib, 'MinimumRequiredVersion', _string) -_MSBuildOnly(_lib, 'Name', _file_name) # /NAME -_MSBuildOnly(_lib, 'RemoveObjects', _file_list) # /REMOVE -_MSBuildOnly(_lib, 'SubSystem', _subsystem_enumeration) -_MSBuildOnly(_lib, 'TrackerLogDirectory', _folder_name) -_MSBuildOnly(_lib, 'TreatLibWarningAsErrors', _boolean) # /WX -_MSBuildOnly(_lib, 'Verbose', _boolean) +_Moved(_lib, "LinkLibraryDependencies", "ProjectReference", _boolean) + +_MSBuildOnly(_lib, "DisplayLibrary", _string) # /LIST Visible='false' +_MSBuildOnly( + _lib, + "ErrorReporting", + _Enumeration( + [], + new=[ + "PromptImmediately", # /ERRORREPORT:PROMPT + "QueueForNextLogin", # /ERRORREPORT:QUEUE + "SendErrorReport", # /ERRORREPORT:SEND + "NoErrorReport", + ], + ), +) # /ERRORREPORT:NONE +_MSBuildOnly(_lib, "MinimumRequiredVersion", _string) +_MSBuildOnly(_lib, "Name", _file_name) # /NAME +_MSBuildOnly(_lib, "RemoveObjects", _file_list) # /REMOVE +_MSBuildOnly(_lib, "SubSystem", _subsystem_enumeration) +_MSBuildOnly(_lib, "TrackerLogDirectory", _folder_name) +_MSBuildOnly(_lib, "TreatLibWarningAsErrors", _boolean) # /WX +_MSBuildOnly(_lib, "Verbose", _boolean) # Directives for converting VCManifestTool to Mt. @@ -1060,41 +1225,45 @@ def _ValidateSettings(validators, settings, stderr): # the schema of the MSBuild Lib settings. # Options that have the same name in MSVS and MSBuild -_Same(_manifest, 'AdditionalManifestFiles', _file_list) # /manifest -_Same(_manifest, 'AdditionalOptions', _string_list) -_Same(_manifest, 'AssemblyIdentity', _string) # /identity: -_Same(_manifest, 'ComponentFileName', _file_name) # /dll -_Same(_manifest, 'GenerateCatalogFiles', _boolean) # /makecdfs -_Same(_manifest, 'InputResourceManifests', _string) # /inputresource -_Same(_manifest, 'OutputManifestFile', _file_name) # /out -_Same(_manifest, 'RegistrarScriptFile', _file_name) # /rgs -_Same(_manifest, 'ReplacementsFile', _file_name) # /replacements -_Same(_manifest, 'SuppressStartupBanner', _boolean) # /nologo -_Same(_manifest, 'TypeLibraryFile', _file_name) # /tlb: -_Same(_manifest, 'UpdateFileHashes', _boolean) # /hashupdate -_Same(_manifest, 'UpdateFileHashesSearchPath', _file_name) -_Same(_manifest, 'VerboseOutput', _boolean) # /verbose +_Same(_manifest, "AdditionalManifestFiles", _file_list) # /manifest +_Same(_manifest, "AdditionalOptions", _string_list) +_Same(_manifest, "AssemblyIdentity", _string) # /identity: +_Same(_manifest, "ComponentFileName", _file_name) # /dll +_Same(_manifest, "GenerateCatalogFiles", _boolean) # /makecdfs +_Same(_manifest, "InputResourceManifests", _string) # /inputresource +_Same(_manifest, "OutputManifestFile", _file_name) # /out +_Same(_manifest, "RegistrarScriptFile", _file_name) # /rgs +_Same(_manifest, "ReplacementsFile", _file_name) # /replacements +_Same(_manifest, "SuppressStartupBanner", _boolean) # /nologo +_Same(_manifest, "TypeLibraryFile", _file_name) # /tlb: +_Same(_manifest, "UpdateFileHashes", _boolean) # /hashupdate +_Same(_manifest, "UpdateFileHashesSearchPath", _file_name) +_Same(_manifest, "VerboseOutput", _boolean) # /verbose # Options that have moved location. -_MovedAndRenamed(_manifest, 'ManifestResourceFile', - 'ManifestResourceCompile', - 'ResourceOutputFileName', - _file_name) -_Moved(_manifest, 'EmbedManifest', '', _boolean) +_MovedAndRenamed( + _manifest, + "ManifestResourceFile", + "ManifestResourceCompile", + "ResourceOutputFileName", + _file_name, +) +_Moved(_manifest, "EmbedManifest", "", _boolean) # MSVS options not found in MSBuild. -_MSVSOnly(_manifest, 'DependencyInformationFile', _file_name) -_MSVSOnly(_manifest, 'UseFAT32Workaround', _boolean) -_MSVSOnly(_manifest, 'UseUnicodeResponseFiles', _boolean) +_MSVSOnly(_manifest, "DependencyInformationFile", _file_name) +_MSVSOnly(_manifest, "UseFAT32Workaround", _boolean) +_MSVSOnly(_manifest, "UseUnicodeResponseFiles", _boolean) # MSBuild options not found in MSVS. -_MSBuildOnly(_manifest, 'EnableDPIAwareness', _boolean) -_MSBuildOnly(_manifest, 'GenerateCategoryTags', _boolean) # /category -_MSBuildOnly(_manifest, 'ManifestFromManagedAssembly', - _file_name) # /managedassemblyname -_MSBuildOnly(_manifest, 'OutputResourceManifests', _string) # /outputresource -_MSBuildOnly(_manifest, 'SuppressDependencyElement', _boolean) # /nodependency -_MSBuildOnly(_manifest, 'TrackerLogDirectory', _folder_name) +_MSBuildOnly(_manifest, "EnableDPIAwareness", _boolean) +_MSBuildOnly(_manifest, "GenerateCategoryTags", _boolean) # /category +_MSBuildOnly( + _manifest, "ManifestFromManagedAssembly", _file_name +) # /managedassemblyname +_MSBuildOnly(_manifest, "OutputResourceManifests", _string) # /outputresource +_MSBuildOnly(_manifest, "SuppressDependencyElement", _boolean) # /nodependency +_MSBuildOnly(_manifest, "TrackerLogDirectory", _folder_name) # Directives for MASM. @@ -1102,4 +1271,4 @@ def _ValidateSettings(validators, settings, stderr): # MSBuild MASM settings. # Options that have the same name in MSVS and MSBuild. -_Same(_masm, 'UseSafeExceptionHandlers', _boolean) # /safeseh +_Same(_masm, "UseSafeExceptionHandlers", _boolean) # /safeseh diff --git a/gyp/pylib/gyp/MSVSSettings_test.py b/gyp/pylib/gyp/MSVSSettings_test.py index 77b79e650d..c53c88e824 100755 --- a/gyp/pylib/gyp/MSVSSettings_test.py +++ b/gyp/pylib/gyp/MSVSSettings_test.py @@ -10,1090 +10,1140 @@ import gyp.MSVSSettings as MSVSSettings try: - from StringIO import StringIO # Python 2 + from StringIO import StringIO # Python 2 except ImportError: - from io import StringIO # Python 3 + from io import StringIO # Python 3 class TestSequenceFunctions(unittest.TestCase): + def setUp(self): + self.stderr = StringIO() - def setUp(self): - self.stderr = StringIO() + def _ExpectedWarnings(self, expected): + """Compares recorded lines to expected warnings.""" + self.stderr.seek(0) + actual = self.stderr.read().split("\n") + actual = [line for line in actual if line] + self.assertEqual(sorted(expected), sorted(actual)) - def _ExpectedWarnings(self, expected): - """Compares recorded lines to expected warnings.""" - self.stderr.seek(0) - actual = self.stderr.read().split('\n') - actual = [line for line in actual if line] - self.assertEqual(sorted(expected), sorted(actual)) - - def testValidateMSVSSettings_tool_names(self): - """Tests that only MSVS tool names are allowed.""" - MSVSSettings.ValidateMSVSSettings( - {'VCCLCompilerTool': {}, - 'VCLinkerTool': {}, - 'VCMIDLTool': {}, - 'foo': {}, - 'VCResourceCompilerTool': {}, - 'VCLibrarianTool': {}, - 'VCManifestTool': {}, - 'ClCompile': {}}, - self.stderr) - self._ExpectedWarnings([ - 'Warning: unrecognized tool foo', - 'Warning: unrecognized tool ClCompile']) + def testValidateMSVSSettings_tool_names(self): + """Tests that only MSVS tool names are allowed.""" + MSVSSettings.ValidateMSVSSettings( + { + "VCCLCompilerTool": {}, + "VCLinkerTool": {}, + "VCMIDLTool": {}, + "foo": {}, + "VCResourceCompilerTool": {}, + "VCLibrarianTool": {}, + "VCManifestTool": {}, + "ClCompile": {}, + }, + self.stderr, + ) + self._ExpectedWarnings( + ["Warning: unrecognized tool foo", "Warning: unrecognized tool ClCompile"] + ) - def testValidateMSVSSettings_settings(self): - """Tests that for invalid MSVS settings.""" - MSVSSettings.ValidateMSVSSettings( - {'VCCLCompilerTool': { - 'AdditionalIncludeDirectories': 'folder1;folder2', - 'AdditionalOptions': ['string1', 'string2'], - 'AdditionalUsingDirectories': 'folder1;folder2', - 'AssemblerListingLocation': 'a_file_name', - 'AssemblerOutput': '0', - 'BasicRuntimeChecks': '5', - 'BrowseInformation': 'fdkslj', - 'BrowseInformationFile': 'a_file_name', - 'BufferSecurityCheck': 'true', - 'CallingConvention': '-1', - 'CompileAs': '1', - 'DebugInformationFormat': '2', - 'DefaultCharIsUnsigned': 'true', - 'Detect64BitPortabilityProblems': 'true', - 'DisableLanguageExtensions': 'true', - 'DisableSpecificWarnings': 'string1;string2', - 'EnableEnhancedInstructionSet': '1', - 'EnableFiberSafeOptimizations': 'true', - 'EnableFunctionLevelLinking': 'true', - 'EnableIntrinsicFunctions': 'true', - 'EnablePREfast': 'true', - 'Enableprefast': 'bogus', - 'ErrorReporting': '1', - 'ExceptionHandling': '1', - 'ExpandAttributedSource': 'true', - 'FavorSizeOrSpeed': '1', - 'FloatingPointExceptions': 'true', - 'FloatingPointModel': '1', - 'ForceConformanceInForLoopScope': 'true', - 'ForcedIncludeFiles': 'file1;file2', - 'ForcedUsingFiles': 'file1;file2', - 'GeneratePreprocessedFile': '1', - 'GenerateXMLDocumentationFiles': 'true', - 'IgnoreStandardIncludePath': 'true', - 'InlineFunctionExpansion': '1', - 'KeepComments': 'true', - 'MinimalRebuild': 'true', - 'ObjectFile': 'a_file_name', - 'OmitDefaultLibName': 'true', - 'OmitFramePointers': 'true', - 'OpenMP': 'true', - 'Optimization': '1', - 'PrecompiledHeaderFile': 'a_file_name', - 'PrecompiledHeaderThrough': 'a_file_name', - 'PreprocessorDefinitions': 'string1;string2', - 'ProgramDataBaseFileName': 'a_file_name', - 'RuntimeLibrary': '1', - 'RuntimeTypeInfo': 'true', - 'ShowIncludes': 'true', - 'SmallerTypeCheck': 'true', - 'StringPooling': 'true', - 'StructMemberAlignment': '1', - 'SuppressStartupBanner': 'true', - 'TreatWChar_tAsBuiltInType': 'true', - 'UndefineAllPreprocessorDefinitions': 'true', - 'UndefinePreprocessorDefinitions': 'string1;string2', - 'UseFullPaths': 'true', - 'UsePrecompiledHeader': '1', - 'UseUnicodeResponseFiles': 'true', - 'WarnAsError': 'true', - 'WarningLevel': '1', - 'WholeProgramOptimization': 'true', - 'XMLDocumentationFileName': 'a_file_name', - 'ZZXYZ': 'bogus'}, - 'VCLinkerTool': { - 'AdditionalDependencies': 'file1;file2', - 'AdditionalDependencies_excluded': 'file3', - 'AdditionalLibraryDirectories': 'folder1;folder2', - 'AdditionalManifestDependencies': 'file1;file2', - 'AdditionalOptions': 'a string1', - 'AddModuleNamesToAssembly': 'file1;file2', - 'AllowIsolation': 'true', - 'AssemblyDebug': '2', - 'AssemblyLinkResource': 'file1;file2', - 'BaseAddress': 'a string1', - 'CLRImageType': '2', - 'CLRThreadAttribute': '2', - 'CLRUnmanagedCodeCheck': 'true', - 'DataExecutionPrevention': '2', - 'DelayLoadDLLs': 'file1;file2', - 'DelaySign': 'true', - 'Driver': '2', - 'EmbedManagedResourceFile': 'file1;file2', - 'EnableCOMDATFolding': '2', - 'EnableUAC': 'true', - 'EntryPointSymbol': 'a string1', - 'ErrorReporting': '2', - 'FixedBaseAddress': '2', - 'ForceSymbolReferences': 'file1;file2', - 'FunctionOrder': 'a_file_name', - 'GenerateDebugInformation': 'true', - 'GenerateManifest': 'true', - 'GenerateMapFile': 'true', - 'HeapCommitSize': 'a string1', - 'HeapReserveSize': 'a string1', - 'IgnoreAllDefaultLibraries': 'true', - 'IgnoreDefaultLibraryNames': 'file1;file2', - 'IgnoreEmbeddedIDL': 'true', - 'IgnoreImportLibrary': 'true', - 'ImportLibrary': 'a_file_name', - 'KeyContainer': 'a_file_name', - 'KeyFile': 'a_file_name', - 'LargeAddressAware': '2', - 'LinkIncremental': '2', - 'LinkLibraryDependencies': 'true', - 'LinkTimeCodeGeneration': '2', - 'ManifestFile': 'a_file_name', - 'MapExports': 'true', - 'MapFileName': 'a_file_name', - 'MergedIDLBaseFileName': 'a_file_name', - 'MergeSections': 'a string1', - 'MidlCommandFile': 'a_file_name', - 'ModuleDefinitionFile': 'a_file_name', - 'OptimizeForWindows98': '1', - 'OptimizeReferences': '2', - 'OutputFile': 'a_file_name', - 'PerUserRedirection': 'true', - 'Profile': 'true', - 'ProfileGuidedDatabase': 'a_file_name', - 'ProgramDatabaseFile': 'a_file_name', - 'RandomizedBaseAddress': '2', - 'RegisterOutput': 'true', - 'ResourceOnlyDLL': 'true', - 'SetChecksum': 'true', - 'ShowProgress': '2', - 'StackCommitSize': 'a string1', - 'StackReserveSize': 'a string1', - 'StripPrivateSymbols': 'a_file_name', - 'SubSystem': '2', - 'SupportUnloadOfDelayLoadedDLL': 'true', - 'SuppressStartupBanner': 'true', - 'SwapRunFromCD': 'true', - 'SwapRunFromNet': 'true', - 'TargetMachine': '2', - 'TerminalServerAware': '2', - 'TurnOffAssemblyGeneration': 'true', - 'TypeLibraryFile': 'a_file_name', - 'TypeLibraryResourceID': '33', - 'UACExecutionLevel': '2', - 'UACUIAccess': 'true', - 'UseLibraryDependencyInputs': 'true', - 'UseUnicodeResponseFiles': 'true', - 'Version': 'a string1'}, - 'VCMIDLTool': { - 'AdditionalIncludeDirectories': 'folder1;folder2', - 'AdditionalOptions': 'a string1', - 'CPreprocessOptions': 'a string1', - 'DefaultCharType': '1', - 'DLLDataFileName': 'a_file_name', - 'EnableErrorChecks': '1', - 'ErrorCheckAllocations': 'true', - 'ErrorCheckBounds': 'true', - 'ErrorCheckEnumRange': 'true', - 'ErrorCheckRefPointers': 'true', - 'ErrorCheckStubData': 'true', - 'GenerateStublessProxies': 'true', - 'GenerateTypeLibrary': 'true', - 'HeaderFileName': 'a_file_name', - 'IgnoreStandardIncludePath': 'true', - 'InterfaceIdentifierFileName': 'a_file_name', - 'MkTypLibCompatible': 'true', - 'notgood': 'bogus', - 'OutputDirectory': 'a string1', - 'PreprocessorDefinitions': 'string1;string2', - 'ProxyFileName': 'a_file_name', - 'RedirectOutputAndErrors': 'a_file_name', - 'StructMemberAlignment': '1', - 'SuppressStartupBanner': 'true', - 'TargetEnvironment': '1', - 'TypeLibraryName': 'a_file_name', - 'UndefinePreprocessorDefinitions': 'string1;string2', - 'ValidateParameters': 'true', - 'WarnAsError': 'true', - 'WarningLevel': '1'}, - 'VCResourceCompilerTool': { - 'AdditionalOptions': 'a string1', - 'AdditionalIncludeDirectories': 'folder1;folder2', - 'Culture': '1003', - 'IgnoreStandardIncludePath': 'true', - 'notgood2': 'bogus', - 'PreprocessorDefinitions': 'string1;string2', - 'ResourceOutputFileName': 'a string1', - 'ShowProgress': 'true', - 'SuppressStartupBanner': 'true', - 'UndefinePreprocessorDefinitions': 'string1;string2'}, - 'VCLibrarianTool': { - 'AdditionalDependencies': 'file1;file2', - 'AdditionalLibraryDirectories': 'folder1;folder2', - 'AdditionalOptions': 'a string1', - 'ExportNamedFunctions': 'string1;string2', - 'ForceSymbolReferences': 'a string1', - 'IgnoreAllDefaultLibraries': 'true', - 'IgnoreSpecificDefaultLibraries': 'file1;file2', - 'LinkLibraryDependencies': 'true', - 'ModuleDefinitionFile': 'a_file_name', - 'OutputFile': 'a_file_name', - 'SuppressStartupBanner': 'true', - 'UseUnicodeResponseFiles': 'true'}, - 'VCManifestTool': { - 'AdditionalManifestFiles': 'file1;file2', - 'AdditionalOptions': 'a string1', - 'AssemblyIdentity': 'a string1', - 'ComponentFileName': 'a_file_name', - 'DependencyInformationFile': 'a_file_name', - 'GenerateCatalogFiles': 'true', - 'InputResourceManifests': 'a string1', - 'ManifestResourceFile': 'a_file_name', - 'OutputManifestFile': 'a_file_name', - 'RegistrarScriptFile': 'a_file_name', - 'ReplacementsFile': 'a_file_name', - 'SuppressStartupBanner': 'true', - 'TypeLibraryFile': 'a_file_name', - 'UpdateFileHashes': 'truel', - 'UpdateFileHashesSearchPath': 'a_file_name', - 'UseFAT32Workaround': 'true', - 'UseUnicodeResponseFiles': 'true', - 'VerboseOutput': 'true'}}, - self.stderr) - self._ExpectedWarnings([ - 'Warning: for VCCLCompilerTool/BasicRuntimeChecks, ' - 'index value (5) not in expected range [0, 4)', - 'Warning: for VCCLCompilerTool/BrowseInformation, ' - "invalid literal for int() with base 10: 'fdkslj'", - 'Warning: for VCCLCompilerTool/CallingConvention, ' - 'index value (-1) not in expected range [0, 4)', - 'Warning: for VCCLCompilerTool/DebugInformationFormat, ' - 'converted value for 2 not specified.', - 'Warning: unrecognized setting VCCLCompilerTool/Enableprefast', - 'Warning: unrecognized setting VCCLCompilerTool/ZZXYZ', - 'Warning: for VCLinkerTool/TargetMachine, ' - 'converted value for 2 not specified.', - 'Warning: unrecognized setting VCMIDLTool/notgood', - 'Warning: unrecognized setting VCResourceCompilerTool/notgood2', - 'Warning: for VCManifestTool/UpdateFileHashes, ' - "expected bool; got 'truel'" - '']) + def testValidateMSVSSettings_settings(self): + """Tests that for invalid MSVS settings.""" + MSVSSettings.ValidateMSVSSettings( + { + "VCCLCompilerTool": { + "AdditionalIncludeDirectories": "folder1;folder2", + "AdditionalOptions": ["string1", "string2"], + "AdditionalUsingDirectories": "folder1;folder2", + "AssemblerListingLocation": "a_file_name", + "AssemblerOutput": "0", + "BasicRuntimeChecks": "5", + "BrowseInformation": "fdkslj", + "BrowseInformationFile": "a_file_name", + "BufferSecurityCheck": "true", + "CallingConvention": "-1", + "CompileAs": "1", + "DebugInformationFormat": "2", + "DefaultCharIsUnsigned": "true", + "Detect64BitPortabilityProblems": "true", + "DisableLanguageExtensions": "true", + "DisableSpecificWarnings": "string1;string2", + "EnableEnhancedInstructionSet": "1", + "EnableFiberSafeOptimizations": "true", + "EnableFunctionLevelLinking": "true", + "EnableIntrinsicFunctions": "true", + "EnablePREfast": "true", + "Enableprefast": "bogus", + "ErrorReporting": "1", + "ExceptionHandling": "1", + "ExpandAttributedSource": "true", + "FavorSizeOrSpeed": "1", + "FloatingPointExceptions": "true", + "FloatingPointModel": "1", + "ForceConformanceInForLoopScope": "true", + "ForcedIncludeFiles": "file1;file2", + "ForcedUsingFiles": "file1;file2", + "GeneratePreprocessedFile": "1", + "GenerateXMLDocumentationFiles": "true", + "IgnoreStandardIncludePath": "true", + "InlineFunctionExpansion": "1", + "KeepComments": "true", + "MinimalRebuild": "true", + "ObjectFile": "a_file_name", + "OmitDefaultLibName": "true", + "OmitFramePointers": "true", + "OpenMP": "true", + "Optimization": "1", + "PrecompiledHeaderFile": "a_file_name", + "PrecompiledHeaderThrough": "a_file_name", + "PreprocessorDefinitions": "string1;string2", + "ProgramDataBaseFileName": "a_file_name", + "RuntimeLibrary": "1", + "RuntimeTypeInfo": "true", + "ShowIncludes": "true", + "SmallerTypeCheck": "true", + "StringPooling": "true", + "StructMemberAlignment": "1", + "SuppressStartupBanner": "true", + "TreatWChar_tAsBuiltInType": "true", + "UndefineAllPreprocessorDefinitions": "true", + "UndefinePreprocessorDefinitions": "string1;string2", + "UseFullPaths": "true", + "UsePrecompiledHeader": "1", + "UseUnicodeResponseFiles": "true", + "WarnAsError": "true", + "WarningLevel": "1", + "WholeProgramOptimization": "true", + "XMLDocumentationFileName": "a_file_name", + "ZZXYZ": "bogus", + }, + "VCLinkerTool": { + "AdditionalDependencies": "file1;file2", + "AdditionalDependencies_excluded": "file3", + "AdditionalLibraryDirectories": "folder1;folder2", + "AdditionalManifestDependencies": "file1;file2", + "AdditionalOptions": "a string1", + "AddModuleNamesToAssembly": "file1;file2", + "AllowIsolation": "true", + "AssemblyDebug": "2", + "AssemblyLinkResource": "file1;file2", + "BaseAddress": "a string1", + "CLRImageType": "2", + "CLRThreadAttribute": "2", + "CLRUnmanagedCodeCheck": "true", + "DataExecutionPrevention": "2", + "DelayLoadDLLs": "file1;file2", + "DelaySign": "true", + "Driver": "2", + "EmbedManagedResourceFile": "file1;file2", + "EnableCOMDATFolding": "2", + "EnableUAC": "true", + "EntryPointSymbol": "a string1", + "ErrorReporting": "2", + "FixedBaseAddress": "2", + "ForceSymbolReferences": "file1;file2", + "FunctionOrder": "a_file_name", + "GenerateDebugInformation": "true", + "GenerateManifest": "true", + "GenerateMapFile": "true", + "HeapCommitSize": "a string1", + "HeapReserveSize": "a string1", + "IgnoreAllDefaultLibraries": "true", + "IgnoreDefaultLibraryNames": "file1;file2", + "IgnoreEmbeddedIDL": "true", + "IgnoreImportLibrary": "true", + "ImportLibrary": "a_file_name", + "KeyContainer": "a_file_name", + "KeyFile": "a_file_name", + "LargeAddressAware": "2", + "LinkIncremental": "2", + "LinkLibraryDependencies": "true", + "LinkTimeCodeGeneration": "2", + "ManifestFile": "a_file_name", + "MapExports": "true", + "MapFileName": "a_file_name", + "MergedIDLBaseFileName": "a_file_name", + "MergeSections": "a string1", + "MidlCommandFile": "a_file_name", + "ModuleDefinitionFile": "a_file_name", + "OptimizeForWindows98": "1", + "OptimizeReferences": "2", + "OutputFile": "a_file_name", + "PerUserRedirection": "true", + "Profile": "true", + "ProfileGuidedDatabase": "a_file_name", + "ProgramDatabaseFile": "a_file_name", + "RandomizedBaseAddress": "2", + "RegisterOutput": "true", + "ResourceOnlyDLL": "true", + "SetChecksum": "true", + "ShowProgress": "2", + "StackCommitSize": "a string1", + "StackReserveSize": "a string1", + "StripPrivateSymbols": "a_file_name", + "SubSystem": "2", + "SupportUnloadOfDelayLoadedDLL": "true", + "SuppressStartupBanner": "true", + "SwapRunFromCD": "true", + "SwapRunFromNet": "true", + "TargetMachine": "2", + "TerminalServerAware": "2", + "TurnOffAssemblyGeneration": "true", + "TypeLibraryFile": "a_file_name", + "TypeLibraryResourceID": "33", + "UACExecutionLevel": "2", + "UACUIAccess": "true", + "UseLibraryDependencyInputs": "true", + "UseUnicodeResponseFiles": "true", + "Version": "a string1", + }, + "VCMIDLTool": { + "AdditionalIncludeDirectories": "folder1;folder2", + "AdditionalOptions": "a string1", + "CPreprocessOptions": "a string1", + "DefaultCharType": "1", + "DLLDataFileName": "a_file_name", + "EnableErrorChecks": "1", + "ErrorCheckAllocations": "true", + "ErrorCheckBounds": "true", + "ErrorCheckEnumRange": "true", + "ErrorCheckRefPointers": "true", + "ErrorCheckStubData": "true", + "GenerateStublessProxies": "true", + "GenerateTypeLibrary": "true", + "HeaderFileName": "a_file_name", + "IgnoreStandardIncludePath": "true", + "InterfaceIdentifierFileName": "a_file_name", + "MkTypLibCompatible": "true", + "notgood": "bogus", + "OutputDirectory": "a string1", + "PreprocessorDefinitions": "string1;string2", + "ProxyFileName": "a_file_name", + "RedirectOutputAndErrors": "a_file_name", + "StructMemberAlignment": "1", + "SuppressStartupBanner": "true", + "TargetEnvironment": "1", + "TypeLibraryName": "a_file_name", + "UndefinePreprocessorDefinitions": "string1;string2", + "ValidateParameters": "true", + "WarnAsError": "true", + "WarningLevel": "1", + }, + "VCResourceCompilerTool": { + "AdditionalOptions": "a string1", + "AdditionalIncludeDirectories": "folder1;folder2", + "Culture": "1003", + "IgnoreStandardIncludePath": "true", + "notgood2": "bogus", + "PreprocessorDefinitions": "string1;string2", + "ResourceOutputFileName": "a string1", + "ShowProgress": "true", + "SuppressStartupBanner": "true", + "UndefinePreprocessorDefinitions": "string1;string2", + }, + "VCLibrarianTool": { + "AdditionalDependencies": "file1;file2", + "AdditionalLibraryDirectories": "folder1;folder2", + "AdditionalOptions": "a string1", + "ExportNamedFunctions": "string1;string2", + "ForceSymbolReferences": "a string1", + "IgnoreAllDefaultLibraries": "true", + "IgnoreSpecificDefaultLibraries": "file1;file2", + "LinkLibraryDependencies": "true", + "ModuleDefinitionFile": "a_file_name", + "OutputFile": "a_file_name", + "SuppressStartupBanner": "true", + "UseUnicodeResponseFiles": "true", + }, + "VCManifestTool": { + "AdditionalManifestFiles": "file1;file2", + "AdditionalOptions": "a string1", + "AssemblyIdentity": "a string1", + "ComponentFileName": "a_file_name", + "DependencyInformationFile": "a_file_name", + "GenerateCatalogFiles": "true", + "InputResourceManifests": "a string1", + "ManifestResourceFile": "a_file_name", + "OutputManifestFile": "a_file_name", + "RegistrarScriptFile": "a_file_name", + "ReplacementsFile": "a_file_name", + "SuppressStartupBanner": "true", + "TypeLibraryFile": "a_file_name", + "UpdateFileHashes": "truel", + "UpdateFileHashesSearchPath": "a_file_name", + "UseFAT32Workaround": "true", + "UseUnicodeResponseFiles": "true", + "VerboseOutput": "true", + }, + }, + self.stderr, + ) + self._ExpectedWarnings( + [ + "Warning: for VCCLCompilerTool/BasicRuntimeChecks, " + "index value (5) not in expected range [0, 4)", + "Warning: for VCCLCompilerTool/BrowseInformation, " + "invalid literal for int() with base 10: 'fdkslj'", + "Warning: for VCCLCompilerTool/CallingConvention, " + "index value (-1) not in expected range [0, 4)", + "Warning: for VCCLCompilerTool/DebugInformationFormat, " + "converted value for 2 not specified.", + "Warning: unrecognized setting VCCLCompilerTool/Enableprefast", + "Warning: unrecognized setting VCCLCompilerTool/ZZXYZ", + "Warning: for VCLinkerTool/TargetMachine, " + "converted value for 2 not specified.", + "Warning: unrecognized setting VCMIDLTool/notgood", + "Warning: unrecognized setting VCResourceCompilerTool/notgood2", + "Warning: for VCManifestTool/UpdateFileHashes, " + "expected bool; got 'truel'" + "", + ] + ) - def testValidateMSBuildSettings_settings(self): - """Tests that for invalid MSBuild settings.""" - MSVSSettings.ValidateMSBuildSettings( - {'ClCompile': { - 'AdditionalIncludeDirectories': 'folder1;folder2', - 'AdditionalOptions': ['string1', 'string2'], - 'AdditionalUsingDirectories': 'folder1;folder2', - 'AssemblerListingLocation': 'a_file_name', - 'AssemblerOutput': 'NoListing', - 'BasicRuntimeChecks': 'StackFrameRuntimeCheck', - 'BrowseInformation': 'false', - 'BrowseInformationFile': 'a_file_name', - 'BufferSecurityCheck': 'true', - 'BuildingInIDE': 'true', - 'CallingConvention': 'Cdecl', - 'CompileAs': 'CompileAsC', - 'CompileAsManaged': 'true', - 'CreateHotpatchableImage': 'true', - 'DebugInformationFormat': 'ProgramDatabase', - 'DisableLanguageExtensions': 'true', - 'DisableSpecificWarnings': 'string1;string2', - 'EnableEnhancedInstructionSet': 'StreamingSIMDExtensions', - 'EnableFiberSafeOptimizations': 'true', - 'EnablePREfast': 'true', - 'Enableprefast': 'bogus', - 'ErrorReporting': 'Prompt', - 'ExceptionHandling': 'SyncCThrow', - 'ExpandAttributedSource': 'true', - 'FavorSizeOrSpeed': 'Neither', - 'FloatingPointExceptions': 'true', - 'FloatingPointModel': 'Precise', - 'ForceConformanceInForLoopScope': 'true', - 'ForcedIncludeFiles': 'file1;file2', - 'ForcedUsingFiles': 'file1;file2', - 'FunctionLevelLinking': 'false', - 'GenerateXMLDocumentationFiles': 'true', - 'IgnoreStandardIncludePath': 'true', - 'InlineFunctionExpansion': 'OnlyExplicitInline', - 'IntrinsicFunctions': 'false', - 'MinimalRebuild': 'true', - 'MultiProcessorCompilation': 'true', - 'ObjectFileName': 'a_file_name', - 'OmitDefaultLibName': 'true', - 'OmitFramePointers': 'true', - 'OpenMPSupport': 'true', - 'Optimization': 'Disabled', - 'PrecompiledHeader': 'NotUsing', - 'PrecompiledHeaderFile': 'a_file_name', - 'PrecompiledHeaderOutputFile': 'a_file_name', - 'PreprocessKeepComments': 'true', - 'PreprocessorDefinitions': 'string1;string2', - 'PreprocessOutputPath': 'a string1', - 'PreprocessSuppressLineNumbers': 'false', - 'PreprocessToFile': 'false', - 'ProcessorNumber': '33', - 'ProgramDataBaseFileName': 'a_file_name', - 'RuntimeLibrary': 'MultiThreaded', - 'RuntimeTypeInfo': 'true', - 'ShowIncludes': 'true', - 'SmallerTypeCheck': 'true', - 'StringPooling': 'true', - 'StructMemberAlignment': '1Byte', - 'SuppressStartupBanner': 'true', - 'TrackerLogDirectory': 'a_folder', - 'TreatSpecificWarningsAsErrors': 'string1;string2', - 'TreatWarningAsError': 'true', - 'TreatWChar_tAsBuiltInType': 'true', - 'UndefineAllPreprocessorDefinitions': 'true', - 'UndefinePreprocessorDefinitions': 'string1;string2', - 'UseFullPaths': 'true', - 'UseUnicodeForAssemblerListing': 'true', - 'WarningLevel': 'TurnOffAllWarnings', - 'WholeProgramOptimization': 'true', - 'XMLDocumentationFileName': 'a_file_name', - 'ZZXYZ': 'bogus'}, - 'Link': { - 'AdditionalDependencies': 'file1;file2', - 'AdditionalLibraryDirectories': 'folder1;folder2', - 'AdditionalManifestDependencies': 'file1;file2', - 'AdditionalOptions': 'a string1', - 'AddModuleNamesToAssembly': 'file1;file2', - 'AllowIsolation': 'true', - 'AssemblyDebug': '', - 'AssemblyLinkResource': 'file1;file2', - 'BaseAddress': 'a string1', - 'BuildingInIDE': 'true', - 'CLRImageType': 'ForceIJWImage', - 'CLRSupportLastError': 'Enabled', - 'CLRThreadAttribute': 'MTAThreadingAttribute', - 'CLRUnmanagedCodeCheck': 'true', - 'CreateHotPatchableImage': 'X86Image', - 'DataExecutionPrevention': 'false', - 'DelayLoadDLLs': 'file1;file2', - 'DelaySign': 'true', - 'Driver': 'NotSet', - 'EmbedManagedResourceFile': 'file1;file2', - 'EnableCOMDATFolding': 'false', - 'EnableUAC': 'true', - 'EntryPointSymbol': 'a string1', - 'FixedBaseAddress': 'false', - 'ForceFileOutput': 'Enabled', - 'ForceSymbolReferences': 'file1;file2', - 'FunctionOrder': 'a_file_name', - 'GenerateDebugInformation': 'true', - 'GenerateMapFile': 'true', - 'HeapCommitSize': 'a string1', - 'HeapReserveSize': 'a string1', - 'IgnoreAllDefaultLibraries': 'true', - 'IgnoreEmbeddedIDL': 'true', - 'IgnoreSpecificDefaultLibraries': 'a_file_list', - 'ImageHasSafeExceptionHandlers': 'true', - 'ImportLibrary': 'a_file_name', - 'KeyContainer': 'a_file_name', - 'KeyFile': 'a_file_name', - 'LargeAddressAware': 'false', - 'LinkDLL': 'true', - 'LinkErrorReporting': 'SendErrorReport', - 'LinkStatus': 'true', - 'LinkTimeCodeGeneration': 'UseLinkTimeCodeGeneration', - 'ManifestFile': 'a_file_name', - 'MapExports': 'true', - 'MapFileName': 'a_file_name', - 'MergedIDLBaseFileName': 'a_file_name', - 'MergeSections': 'a string1', - 'MidlCommandFile': 'a_file_name', - 'MinimumRequiredVersion': 'a string1', - 'ModuleDefinitionFile': 'a_file_name', - 'MSDOSStubFileName': 'a_file_name', - 'NoEntryPoint': 'true', - 'OptimizeReferences': 'false', - 'OutputFile': 'a_file_name', - 'PerUserRedirection': 'true', - 'PreventDllBinding': 'true', - 'Profile': 'true', - 'ProfileGuidedDatabase': 'a_file_name', - 'ProgramDatabaseFile': 'a_file_name', - 'RandomizedBaseAddress': 'false', - 'RegisterOutput': 'true', - 'SectionAlignment': '33', - 'SetChecksum': 'true', - 'ShowProgress': 'LinkVerboseREF', - 'SpecifySectionAttributes': 'a string1', - 'StackCommitSize': 'a string1', - 'StackReserveSize': 'a string1', - 'StripPrivateSymbols': 'a_file_name', - 'SubSystem': 'Console', - 'SupportNobindOfDelayLoadedDLL': 'true', - 'SupportUnloadOfDelayLoadedDLL': 'true', - 'SuppressStartupBanner': 'true', - 'SwapRunFromCD': 'true', - 'SwapRunFromNET': 'true', - 'TargetMachine': 'MachineX86', - 'TerminalServerAware': 'false', - 'TrackerLogDirectory': 'a_folder', - 'TreatLinkerWarningAsErrors': 'true', - 'TurnOffAssemblyGeneration': 'true', - 'TypeLibraryFile': 'a_file_name', - 'TypeLibraryResourceID': '33', - 'UACExecutionLevel': 'AsInvoker', - 'UACUIAccess': 'true', - 'Version': 'a string1'}, - 'ResourceCompile': { - 'AdditionalIncludeDirectories': 'folder1;folder2', - 'AdditionalOptions': 'a string1', - 'Culture': '0x236', - 'IgnoreStandardIncludePath': 'true', - 'NullTerminateStrings': 'true', - 'PreprocessorDefinitions': 'string1;string2', - 'ResourceOutputFileName': 'a string1', - 'ShowProgress': 'true', - 'SuppressStartupBanner': 'true', - 'TrackerLogDirectory': 'a_folder', - 'UndefinePreprocessorDefinitions': 'string1;string2'}, - 'Midl': { - 'AdditionalIncludeDirectories': 'folder1;folder2', - 'AdditionalOptions': 'a string1', - 'ApplicationConfigurationMode': 'true', - 'ClientStubFile': 'a_file_name', - 'CPreprocessOptions': 'a string1', - 'DefaultCharType': 'Signed', - 'DllDataFileName': 'a_file_name', - 'EnableErrorChecks': 'EnableCustom', - 'ErrorCheckAllocations': 'true', - 'ErrorCheckBounds': 'true', - 'ErrorCheckEnumRange': 'true', - 'ErrorCheckRefPointers': 'true', - 'ErrorCheckStubData': 'true', - 'GenerateClientFiles': 'Stub', - 'GenerateServerFiles': 'None', - 'GenerateStublessProxies': 'true', - 'GenerateTypeLibrary': 'true', - 'HeaderFileName': 'a_file_name', - 'IgnoreStandardIncludePath': 'true', - 'InterfaceIdentifierFileName': 'a_file_name', - 'LocaleID': '33', - 'MkTypLibCompatible': 'true', - 'OutputDirectory': 'a string1', - 'PreprocessorDefinitions': 'string1;string2', - 'ProxyFileName': 'a_file_name', - 'RedirectOutputAndErrors': 'a_file_name', - 'ServerStubFile': 'a_file_name', - 'StructMemberAlignment': 'NotSet', - 'SuppressCompilerWarnings': 'true', - 'SuppressStartupBanner': 'true', - 'TargetEnvironment': 'Itanium', - 'TrackerLogDirectory': 'a_folder', - 'TypeLibFormat': 'NewFormat', - 'TypeLibraryName': 'a_file_name', - 'UndefinePreprocessorDefinitions': 'string1;string2', - 'ValidateAllParameters': 'true', - 'WarnAsError': 'true', - 'WarningLevel': '1'}, - 'Lib': { - 'AdditionalDependencies': 'file1;file2', - 'AdditionalLibraryDirectories': 'folder1;folder2', - 'AdditionalOptions': 'a string1', - 'DisplayLibrary': 'a string1', - 'ErrorReporting': 'PromptImmediately', - 'ExportNamedFunctions': 'string1;string2', - 'ForceSymbolReferences': 'a string1', - 'IgnoreAllDefaultLibraries': 'true', - 'IgnoreSpecificDefaultLibraries': 'file1;file2', - 'LinkTimeCodeGeneration': 'true', - 'MinimumRequiredVersion': 'a string1', - 'ModuleDefinitionFile': 'a_file_name', - 'Name': 'a_file_name', - 'OutputFile': 'a_file_name', - 'RemoveObjects': 'file1;file2', - 'SubSystem': 'Console', - 'SuppressStartupBanner': 'true', - 'TargetMachine': 'MachineX86i', - 'TrackerLogDirectory': 'a_folder', - 'TreatLibWarningAsErrors': 'true', - 'UseUnicodeResponseFiles': 'true', - 'Verbose': 'true'}, - 'Manifest': { - 'AdditionalManifestFiles': 'file1;file2', - 'AdditionalOptions': 'a string1', - 'AssemblyIdentity': 'a string1', - 'ComponentFileName': 'a_file_name', - 'EnableDPIAwareness': 'fal', - 'GenerateCatalogFiles': 'truel', - 'GenerateCategoryTags': 'true', - 'InputResourceManifests': 'a string1', - 'ManifestFromManagedAssembly': 'a_file_name', - 'notgood3': 'bogus', - 'OutputManifestFile': 'a_file_name', - 'OutputResourceManifests': 'a string1', - 'RegistrarScriptFile': 'a_file_name', - 'ReplacementsFile': 'a_file_name', - 'SuppressDependencyElement': 'true', - 'SuppressStartupBanner': 'true', - 'TrackerLogDirectory': 'a_folder', - 'TypeLibraryFile': 'a_file_name', - 'UpdateFileHashes': 'true', - 'UpdateFileHashesSearchPath': 'a_file_name', - 'VerboseOutput': 'true'}, - 'ProjectReference': { - 'LinkLibraryDependencies': 'true', - 'UseLibraryDependencyInputs': 'true'}, - 'ManifestResourceCompile': { - 'ResourceOutputFileName': 'a_file_name'}, - '': { - 'EmbedManifest': 'true', - 'GenerateManifest': 'true', - 'IgnoreImportLibrary': 'true', - 'LinkIncremental': 'false'}}, - self.stderr) - self._ExpectedWarnings([ - 'Warning: unrecognized setting ClCompile/Enableprefast', - 'Warning: unrecognized setting ClCompile/ZZXYZ', - 'Warning: unrecognized setting Manifest/notgood3', - 'Warning: for Manifest/GenerateCatalogFiles, ' - "expected bool; got 'truel'", - 'Warning: for Lib/TargetMachine, unrecognized enumerated value ' - 'MachineX86i', - "Warning: for Manifest/EnableDPIAwareness, expected bool; got 'fal'"]) + def testValidateMSBuildSettings_settings(self): + """Tests that for invalid MSBuild settings.""" + MSVSSettings.ValidateMSBuildSettings( + { + "ClCompile": { + "AdditionalIncludeDirectories": "folder1;folder2", + "AdditionalOptions": ["string1", "string2"], + "AdditionalUsingDirectories": "folder1;folder2", + "AssemblerListingLocation": "a_file_name", + "AssemblerOutput": "NoListing", + "BasicRuntimeChecks": "StackFrameRuntimeCheck", + "BrowseInformation": "false", + "BrowseInformationFile": "a_file_name", + "BufferSecurityCheck": "true", + "BuildingInIDE": "true", + "CallingConvention": "Cdecl", + "CompileAs": "CompileAsC", + "CompileAsManaged": "true", + "CreateHotpatchableImage": "true", + "DebugInformationFormat": "ProgramDatabase", + "DisableLanguageExtensions": "true", + "DisableSpecificWarnings": "string1;string2", + "EnableEnhancedInstructionSet": "StreamingSIMDExtensions", + "EnableFiberSafeOptimizations": "true", + "EnablePREfast": "true", + "Enableprefast": "bogus", + "ErrorReporting": "Prompt", + "ExceptionHandling": "SyncCThrow", + "ExpandAttributedSource": "true", + "FavorSizeOrSpeed": "Neither", + "FloatingPointExceptions": "true", + "FloatingPointModel": "Precise", + "ForceConformanceInForLoopScope": "true", + "ForcedIncludeFiles": "file1;file2", + "ForcedUsingFiles": "file1;file2", + "FunctionLevelLinking": "false", + "GenerateXMLDocumentationFiles": "true", + "IgnoreStandardIncludePath": "true", + "InlineFunctionExpansion": "OnlyExplicitInline", + "IntrinsicFunctions": "false", + "MinimalRebuild": "true", + "MultiProcessorCompilation": "true", + "ObjectFileName": "a_file_name", + "OmitDefaultLibName": "true", + "OmitFramePointers": "true", + "OpenMPSupport": "true", + "Optimization": "Disabled", + "PrecompiledHeader": "NotUsing", + "PrecompiledHeaderFile": "a_file_name", + "PrecompiledHeaderOutputFile": "a_file_name", + "PreprocessKeepComments": "true", + "PreprocessorDefinitions": "string1;string2", + "PreprocessOutputPath": "a string1", + "PreprocessSuppressLineNumbers": "false", + "PreprocessToFile": "false", + "ProcessorNumber": "33", + "ProgramDataBaseFileName": "a_file_name", + "RuntimeLibrary": "MultiThreaded", + "RuntimeTypeInfo": "true", + "ShowIncludes": "true", + "SmallerTypeCheck": "true", + "StringPooling": "true", + "StructMemberAlignment": "1Byte", + "SuppressStartupBanner": "true", + "TrackerLogDirectory": "a_folder", + "TreatSpecificWarningsAsErrors": "string1;string2", + "TreatWarningAsError": "true", + "TreatWChar_tAsBuiltInType": "true", + "UndefineAllPreprocessorDefinitions": "true", + "UndefinePreprocessorDefinitions": "string1;string2", + "UseFullPaths": "true", + "UseUnicodeForAssemblerListing": "true", + "WarningLevel": "TurnOffAllWarnings", + "WholeProgramOptimization": "true", + "XMLDocumentationFileName": "a_file_name", + "ZZXYZ": "bogus", + }, + "Link": { + "AdditionalDependencies": "file1;file2", + "AdditionalLibraryDirectories": "folder1;folder2", + "AdditionalManifestDependencies": "file1;file2", + "AdditionalOptions": "a string1", + "AddModuleNamesToAssembly": "file1;file2", + "AllowIsolation": "true", + "AssemblyDebug": "", + "AssemblyLinkResource": "file1;file2", + "BaseAddress": "a string1", + "BuildingInIDE": "true", + "CLRImageType": "ForceIJWImage", + "CLRSupportLastError": "Enabled", + "CLRThreadAttribute": "MTAThreadingAttribute", + "CLRUnmanagedCodeCheck": "true", + "CreateHotPatchableImage": "X86Image", + "DataExecutionPrevention": "false", + "DelayLoadDLLs": "file1;file2", + "DelaySign": "true", + "Driver": "NotSet", + "EmbedManagedResourceFile": "file1;file2", + "EnableCOMDATFolding": "false", + "EnableUAC": "true", + "EntryPointSymbol": "a string1", + "FixedBaseAddress": "false", + "ForceFileOutput": "Enabled", + "ForceSymbolReferences": "file1;file2", + "FunctionOrder": "a_file_name", + "GenerateDebugInformation": "true", + "GenerateMapFile": "true", + "HeapCommitSize": "a string1", + "HeapReserveSize": "a string1", + "IgnoreAllDefaultLibraries": "true", + "IgnoreEmbeddedIDL": "true", + "IgnoreSpecificDefaultLibraries": "a_file_list", + "ImageHasSafeExceptionHandlers": "true", + "ImportLibrary": "a_file_name", + "KeyContainer": "a_file_name", + "KeyFile": "a_file_name", + "LargeAddressAware": "false", + "LinkDLL": "true", + "LinkErrorReporting": "SendErrorReport", + "LinkStatus": "true", + "LinkTimeCodeGeneration": "UseLinkTimeCodeGeneration", + "ManifestFile": "a_file_name", + "MapExports": "true", + "MapFileName": "a_file_name", + "MergedIDLBaseFileName": "a_file_name", + "MergeSections": "a string1", + "MidlCommandFile": "a_file_name", + "MinimumRequiredVersion": "a string1", + "ModuleDefinitionFile": "a_file_name", + "MSDOSStubFileName": "a_file_name", + "NoEntryPoint": "true", + "OptimizeReferences": "false", + "OutputFile": "a_file_name", + "PerUserRedirection": "true", + "PreventDllBinding": "true", + "Profile": "true", + "ProfileGuidedDatabase": "a_file_name", + "ProgramDatabaseFile": "a_file_name", + "RandomizedBaseAddress": "false", + "RegisterOutput": "true", + "SectionAlignment": "33", + "SetChecksum": "true", + "ShowProgress": "LinkVerboseREF", + "SpecifySectionAttributes": "a string1", + "StackCommitSize": "a string1", + "StackReserveSize": "a string1", + "StripPrivateSymbols": "a_file_name", + "SubSystem": "Console", + "SupportNobindOfDelayLoadedDLL": "true", + "SupportUnloadOfDelayLoadedDLL": "true", + "SuppressStartupBanner": "true", + "SwapRunFromCD": "true", + "SwapRunFromNET": "true", + "TargetMachine": "MachineX86", + "TerminalServerAware": "false", + "TrackerLogDirectory": "a_folder", + "TreatLinkerWarningAsErrors": "true", + "TurnOffAssemblyGeneration": "true", + "TypeLibraryFile": "a_file_name", + "TypeLibraryResourceID": "33", + "UACExecutionLevel": "AsInvoker", + "UACUIAccess": "true", + "Version": "a string1", + }, + "ResourceCompile": { + "AdditionalIncludeDirectories": "folder1;folder2", + "AdditionalOptions": "a string1", + "Culture": "0x236", + "IgnoreStandardIncludePath": "true", + "NullTerminateStrings": "true", + "PreprocessorDefinitions": "string1;string2", + "ResourceOutputFileName": "a string1", + "ShowProgress": "true", + "SuppressStartupBanner": "true", + "TrackerLogDirectory": "a_folder", + "UndefinePreprocessorDefinitions": "string1;string2", + }, + "Midl": { + "AdditionalIncludeDirectories": "folder1;folder2", + "AdditionalOptions": "a string1", + "ApplicationConfigurationMode": "true", + "ClientStubFile": "a_file_name", + "CPreprocessOptions": "a string1", + "DefaultCharType": "Signed", + "DllDataFileName": "a_file_name", + "EnableErrorChecks": "EnableCustom", + "ErrorCheckAllocations": "true", + "ErrorCheckBounds": "true", + "ErrorCheckEnumRange": "true", + "ErrorCheckRefPointers": "true", + "ErrorCheckStubData": "true", + "GenerateClientFiles": "Stub", + "GenerateServerFiles": "None", + "GenerateStublessProxies": "true", + "GenerateTypeLibrary": "true", + "HeaderFileName": "a_file_name", + "IgnoreStandardIncludePath": "true", + "InterfaceIdentifierFileName": "a_file_name", + "LocaleID": "33", + "MkTypLibCompatible": "true", + "OutputDirectory": "a string1", + "PreprocessorDefinitions": "string1;string2", + "ProxyFileName": "a_file_name", + "RedirectOutputAndErrors": "a_file_name", + "ServerStubFile": "a_file_name", + "StructMemberAlignment": "NotSet", + "SuppressCompilerWarnings": "true", + "SuppressStartupBanner": "true", + "TargetEnvironment": "Itanium", + "TrackerLogDirectory": "a_folder", + "TypeLibFormat": "NewFormat", + "TypeLibraryName": "a_file_name", + "UndefinePreprocessorDefinitions": "string1;string2", + "ValidateAllParameters": "true", + "WarnAsError": "true", + "WarningLevel": "1", + }, + "Lib": { + "AdditionalDependencies": "file1;file2", + "AdditionalLibraryDirectories": "folder1;folder2", + "AdditionalOptions": "a string1", + "DisplayLibrary": "a string1", + "ErrorReporting": "PromptImmediately", + "ExportNamedFunctions": "string1;string2", + "ForceSymbolReferences": "a string1", + "IgnoreAllDefaultLibraries": "true", + "IgnoreSpecificDefaultLibraries": "file1;file2", + "LinkTimeCodeGeneration": "true", + "MinimumRequiredVersion": "a string1", + "ModuleDefinitionFile": "a_file_name", + "Name": "a_file_name", + "OutputFile": "a_file_name", + "RemoveObjects": "file1;file2", + "SubSystem": "Console", + "SuppressStartupBanner": "true", + "TargetMachine": "MachineX86i", + "TrackerLogDirectory": "a_folder", + "TreatLibWarningAsErrors": "true", + "UseUnicodeResponseFiles": "true", + "Verbose": "true", + }, + "Manifest": { + "AdditionalManifestFiles": "file1;file2", + "AdditionalOptions": "a string1", + "AssemblyIdentity": "a string1", + "ComponentFileName": "a_file_name", + "EnableDPIAwareness": "fal", + "GenerateCatalogFiles": "truel", + "GenerateCategoryTags": "true", + "InputResourceManifests": "a string1", + "ManifestFromManagedAssembly": "a_file_name", + "notgood3": "bogus", + "OutputManifestFile": "a_file_name", + "OutputResourceManifests": "a string1", + "RegistrarScriptFile": "a_file_name", + "ReplacementsFile": "a_file_name", + "SuppressDependencyElement": "true", + "SuppressStartupBanner": "true", + "TrackerLogDirectory": "a_folder", + "TypeLibraryFile": "a_file_name", + "UpdateFileHashes": "true", + "UpdateFileHashesSearchPath": "a_file_name", + "VerboseOutput": "true", + }, + "ProjectReference": { + "LinkLibraryDependencies": "true", + "UseLibraryDependencyInputs": "true", + }, + "ManifestResourceCompile": {"ResourceOutputFileName": "a_file_name"}, + "": { + "EmbedManifest": "true", + "GenerateManifest": "true", + "IgnoreImportLibrary": "true", + "LinkIncremental": "false", + }, + }, + self.stderr, + ) + self._ExpectedWarnings( + [ + "Warning: unrecognized setting ClCompile/Enableprefast", + "Warning: unrecognized setting ClCompile/ZZXYZ", + "Warning: unrecognized setting Manifest/notgood3", + "Warning: for Manifest/GenerateCatalogFiles, " + "expected bool; got 'truel'", + "Warning: for Lib/TargetMachine, unrecognized enumerated value " + "MachineX86i", + "Warning: for Manifest/EnableDPIAwareness, expected bool; got 'fal'", + ] + ) - def testConvertToMSBuildSettings_empty(self): - """Tests an empty conversion.""" - msvs_settings = {} - expected_msbuild_settings = {} - actual_msbuild_settings = MSVSSettings.ConvertToMSBuildSettings( - msvs_settings, - self.stderr) - self.assertEqual(expected_msbuild_settings, actual_msbuild_settings) - self._ExpectedWarnings([]) + def testConvertToMSBuildSettings_empty(self): + """Tests an empty conversion.""" + msvs_settings = {} + expected_msbuild_settings = {} + actual_msbuild_settings = MSVSSettings.ConvertToMSBuildSettings( + msvs_settings, self.stderr + ) + self.assertEqual(expected_msbuild_settings, actual_msbuild_settings) + self._ExpectedWarnings([]) - def testConvertToMSBuildSettings_minimal(self): - """Tests a minimal conversion.""" - msvs_settings = { - 'VCCLCompilerTool': { - 'AdditionalIncludeDirectories': 'dir1', - 'AdditionalOptions': '/foo', - 'BasicRuntimeChecks': '0', + def testConvertToMSBuildSettings_minimal(self): + """Tests a minimal conversion.""" + msvs_settings = { + "VCCLCompilerTool": { + "AdditionalIncludeDirectories": "dir1", + "AdditionalOptions": "/foo", + "BasicRuntimeChecks": "0", }, - 'VCLinkerTool': { - 'LinkTimeCodeGeneration': '1', - 'ErrorReporting': '1', - 'DataExecutionPrevention': '2', + "VCLinkerTool": { + "LinkTimeCodeGeneration": "1", + "ErrorReporting": "1", + "DataExecutionPrevention": "2", }, } - expected_msbuild_settings = { - 'ClCompile': { - 'AdditionalIncludeDirectories': 'dir1', - 'AdditionalOptions': '/foo', - 'BasicRuntimeChecks': 'Default', + expected_msbuild_settings = { + "ClCompile": { + "AdditionalIncludeDirectories": "dir1", + "AdditionalOptions": "/foo", + "BasicRuntimeChecks": "Default", }, - 'Link': { - 'LinkTimeCodeGeneration': 'UseLinkTimeCodeGeneration', - 'LinkErrorReporting': 'PromptImmediately', - 'DataExecutionPrevention': 'true', + "Link": { + "LinkTimeCodeGeneration": "UseLinkTimeCodeGeneration", + "LinkErrorReporting": "PromptImmediately", + "DataExecutionPrevention": "true", }, } - actual_msbuild_settings = MSVSSettings.ConvertToMSBuildSettings( - msvs_settings, - self.stderr) - self.assertEqual(expected_msbuild_settings, actual_msbuild_settings) - self._ExpectedWarnings([]) - - def testConvertToMSBuildSettings_warnings(self): - """Tests conversion that generates warnings.""" - msvs_settings = { - 'VCCLCompilerTool': { - 'AdditionalIncludeDirectories': '1', - 'AdditionalOptions': '2', - # These are incorrect values: - 'BasicRuntimeChecks': '12', - 'BrowseInformation': '21', - 'UsePrecompiledHeader': '13', - 'GeneratePreprocessedFile': '14'}, - 'VCLinkerTool': { - # These are incorrect values: - 'Driver': '10', - 'LinkTimeCodeGeneration': '31', - 'ErrorReporting': '21', - 'FixedBaseAddress': '6'}, - 'VCResourceCompilerTool': { - # Custom - 'Culture': '1003'}} - expected_msbuild_settings = { - 'ClCompile': { - 'AdditionalIncludeDirectories': '1', - 'AdditionalOptions': '2'}, - 'Link': {}, - 'ResourceCompile': { - # Custom - 'Culture': '0x03eb'}} - actual_msbuild_settings = MSVSSettings.ConvertToMSBuildSettings( - msvs_settings, - self.stderr) - self.assertEqual(expected_msbuild_settings, actual_msbuild_settings) - self._ExpectedWarnings([ - 'Warning: while converting VCCLCompilerTool/BasicRuntimeChecks to ' - 'MSBuild, index value (12) not in expected range [0, 4)', - 'Warning: while converting VCCLCompilerTool/BrowseInformation to ' - 'MSBuild, index value (21) not in expected range [0, 3)', - 'Warning: while converting VCCLCompilerTool/UsePrecompiledHeader to ' - 'MSBuild, index value (13) not in expected range [0, 3)', - 'Warning: while converting VCCLCompilerTool/GeneratePreprocessedFile to ' - 'MSBuild, value must be one of [0, 1, 2]; got 14', + actual_msbuild_settings = MSVSSettings.ConvertToMSBuildSettings( + msvs_settings, self.stderr + ) + self.assertEqual(expected_msbuild_settings, actual_msbuild_settings) + self._ExpectedWarnings([]) - 'Warning: while converting VCLinkerTool/Driver to ' - 'MSBuild, index value (10) not in expected range [0, 4)', - 'Warning: while converting VCLinkerTool/LinkTimeCodeGeneration to ' - 'MSBuild, index value (31) not in expected range [0, 5)', - 'Warning: while converting VCLinkerTool/ErrorReporting to ' - 'MSBuild, index value (21) not in expected range [0, 3)', - 'Warning: while converting VCLinkerTool/FixedBaseAddress to ' - 'MSBuild, index value (6) not in expected range [0, 3)', - ]) + def testConvertToMSBuildSettings_warnings(self): + """Tests conversion that generates warnings.""" + msvs_settings = { + "VCCLCompilerTool": { + "AdditionalIncludeDirectories": "1", + "AdditionalOptions": "2", + # These are incorrect values: + "BasicRuntimeChecks": "12", + "BrowseInformation": "21", + "UsePrecompiledHeader": "13", + "GeneratePreprocessedFile": "14", + }, + "VCLinkerTool": { + # These are incorrect values: + "Driver": "10", + "LinkTimeCodeGeneration": "31", + "ErrorReporting": "21", + "FixedBaseAddress": "6", + }, + "VCResourceCompilerTool": { + # Custom + "Culture": "1003" + }, + } + expected_msbuild_settings = { + "ClCompile": { + "AdditionalIncludeDirectories": "1", + "AdditionalOptions": "2", + }, + "Link": {}, + "ResourceCompile": { + # Custom + "Culture": "0x03eb" + }, + } + actual_msbuild_settings = MSVSSettings.ConvertToMSBuildSettings( + msvs_settings, self.stderr + ) + self.assertEqual(expected_msbuild_settings, actual_msbuild_settings) + self._ExpectedWarnings( + [ + "Warning: while converting VCCLCompilerTool/BasicRuntimeChecks to " + "MSBuild, index value (12) not in expected range [0, 4)", + "Warning: while converting VCCLCompilerTool/BrowseInformation to " + "MSBuild, index value (21) not in expected range [0, 3)", + "Warning: while converting VCCLCompilerTool/UsePrecompiledHeader to " + "MSBuild, index value (13) not in expected range [0, 3)", + "Warning: while converting VCCLCompilerTool/GeneratePreprocessedFile to " + "MSBuild, value must be one of [0, 1, 2]; got 14", + "Warning: while converting VCLinkerTool/Driver to " + "MSBuild, index value (10) not in expected range [0, 4)", + "Warning: while converting VCLinkerTool/LinkTimeCodeGeneration to " + "MSBuild, index value (31) not in expected range [0, 5)", + "Warning: while converting VCLinkerTool/ErrorReporting to " + "MSBuild, index value (21) not in expected range [0, 3)", + "Warning: while converting VCLinkerTool/FixedBaseAddress to " + "MSBuild, index value (6) not in expected range [0, 3)", + ] + ) - def testConvertToMSBuildSettings_full_synthetic(self): - """Tests conversion of all the MSBuild settings.""" - msvs_settings = { - 'VCCLCompilerTool': { - 'AdditionalIncludeDirectories': 'folder1;folder2;folder3', - 'AdditionalOptions': 'a_string', - 'AdditionalUsingDirectories': 'folder1;folder2;folder3', - 'AssemblerListingLocation': 'a_file_name', - 'AssemblerOutput': '0', - 'BasicRuntimeChecks': '1', - 'BrowseInformation': '2', - 'BrowseInformationFile': 'a_file_name', - 'BufferSecurityCheck': 'true', - 'CallingConvention': '0', - 'CompileAs': '1', - 'DebugInformationFormat': '4', - 'DefaultCharIsUnsigned': 'true', - 'Detect64BitPortabilityProblems': 'true', - 'DisableLanguageExtensions': 'true', - 'DisableSpecificWarnings': 'd1;d2;d3', - 'EnableEnhancedInstructionSet': '0', - 'EnableFiberSafeOptimizations': 'true', - 'EnableFunctionLevelLinking': 'true', - 'EnableIntrinsicFunctions': 'true', - 'EnablePREfast': 'true', - 'ErrorReporting': '1', - 'ExceptionHandling': '2', - 'ExpandAttributedSource': 'true', - 'FavorSizeOrSpeed': '0', - 'FloatingPointExceptions': 'true', - 'FloatingPointModel': '1', - 'ForceConformanceInForLoopScope': 'true', - 'ForcedIncludeFiles': 'file1;file2;file3', - 'ForcedUsingFiles': 'file1;file2;file3', - 'GeneratePreprocessedFile': '1', - 'GenerateXMLDocumentationFiles': 'true', - 'IgnoreStandardIncludePath': 'true', - 'InlineFunctionExpansion': '2', - 'KeepComments': 'true', - 'MinimalRebuild': 'true', - 'ObjectFile': 'a_file_name', - 'OmitDefaultLibName': 'true', - 'OmitFramePointers': 'true', - 'OpenMP': 'true', - 'Optimization': '3', - 'PrecompiledHeaderFile': 'a_file_name', - 'PrecompiledHeaderThrough': 'a_file_name', - 'PreprocessorDefinitions': 'd1;d2;d3', - 'ProgramDataBaseFileName': 'a_file_name', - 'RuntimeLibrary': '0', - 'RuntimeTypeInfo': 'true', - 'ShowIncludes': 'true', - 'SmallerTypeCheck': 'true', - 'StringPooling': 'true', - 'StructMemberAlignment': '1', - 'SuppressStartupBanner': 'true', - 'TreatWChar_tAsBuiltInType': 'true', - 'UndefineAllPreprocessorDefinitions': 'true', - 'UndefinePreprocessorDefinitions': 'd1;d2;d3', - 'UseFullPaths': 'true', - 'UsePrecompiledHeader': '1', - 'UseUnicodeResponseFiles': 'true', - 'WarnAsError': 'true', - 'WarningLevel': '2', - 'WholeProgramOptimization': 'true', - 'XMLDocumentationFileName': 'a_file_name'}, - 'VCLinkerTool': { - 'AdditionalDependencies': 'file1;file2;file3', - 'AdditionalLibraryDirectories': 'folder1;folder2;folder3', - 'AdditionalLibraryDirectories_excluded': 'folder1;folder2;folder3', - 'AdditionalManifestDependencies': 'file1;file2;file3', - 'AdditionalOptions': 'a_string', - 'AddModuleNamesToAssembly': 'file1;file2;file3', - 'AllowIsolation': 'true', - 'AssemblyDebug': '0', - 'AssemblyLinkResource': 'file1;file2;file3', - 'BaseAddress': 'a_string', - 'CLRImageType': '1', - 'CLRThreadAttribute': '2', - 'CLRUnmanagedCodeCheck': 'true', - 'DataExecutionPrevention': '0', - 'DelayLoadDLLs': 'file1;file2;file3', - 'DelaySign': 'true', - 'Driver': '1', - 'EmbedManagedResourceFile': 'file1;file2;file3', - 'EnableCOMDATFolding': '0', - 'EnableUAC': 'true', - 'EntryPointSymbol': 'a_string', - 'ErrorReporting': '0', - 'FixedBaseAddress': '1', - 'ForceSymbolReferences': 'file1;file2;file3', - 'FunctionOrder': 'a_file_name', - 'GenerateDebugInformation': 'true', - 'GenerateManifest': 'true', - 'GenerateMapFile': 'true', - 'HeapCommitSize': 'a_string', - 'HeapReserveSize': 'a_string', - 'IgnoreAllDefaultLibraries': 'true', - 'IgnoreDefaultLibraryNames': 'file1;file2;file3', - 'IgnoreEmbeddedIDL': 'true', - 'IgnoreImportLibrary': 'true', - 'ImportLibrary': 'a_file_name', - 'KeyContainer': 'a_file_name', - 'KeyFile': 'a_file_name', - 'LargeAddressAware': '2', - 'LinkIncremental': '1', - 'LinkLibraryDependencies': 'true', - 'LinkTimeCodeGeneration': '2', - 'ManifestFile': 'a_file_name', - 'MapExports': 'true', - 'MapFileName': 'a_file_name', - 'MergedIDLBaseFileName': 'a_file_name', - 'MergeSections': 'a_string', - 'MidlCommandFile': 'a_file_name', - 'ModuleDefinitionFile': 'a_file_name', - 'OptimizeForWindows98': '1', - 'OptimizeReferences': '0', - 'OutputFile': 'a_file_name', - 'PerUserRedirection': 'true', - 'Profile': 'true', - 'ProfileGuidedDatabase': 'a_file_name', - 'ProgramDatabaseFile': 'a_file_name', - 'RandomizedBaseAddress': '1', - 'RegisterOutput': 'true', - 'ResourceOnlyDLL': 'true', - 'SetChecksum': 'true', - 'ShowProgress': '0', - 'StackCommitSize': 'a_string', - 'StackReserveSize': 'a_string', - 'StripPrivateSymbols': 'a_file_name', - 'SubSystem': '2', - 'SupportUnloadOfDelayLoadedDLL': 'true', - 'SuppressStartupBanner': 'true', - 'SwapRunFromCD': 'true', - 'SwapRunFromNet': 'true', - 'TargetMachine': '3', - 'TerminalServerAware': '2', - 'TurnOffAssemblyGeneration': 'true', - 'TypeLibraryFile': 'a_file_name', - 'TypeLibraryResourceID': '33', - 'UACExecutionLevel': '1', - 'UACUIAccess': 'true', - 'UseLibraryDependencyInputs': 'false', - 'UseUnicodeResponseFiles': 'true', - 'Version': 'a_string'}, - 'VCResourceCompilerTool': { - 'AdditionalIncludeDirectories': 'folder1;folder2;folder3', - 'AdditionalOptions': 'a_string', - 'Culture': '1003', - 'IgnoreStandardIncludePath': 'true', - 'PreprocessorDefinitions': 'd1;d2;d3', - 'ResourceOutputFileName': 'a_string', - 'ShowProgress': 'true', - 'SuppressStartupBanner': 'true', - 'UndefinePreprocessorDefinitions': 'd1;d2;d3'}, - 'VCMIDLTool': { - 'AdditionalIncludeDirectories': 'folder1;folder2;folder3', - 'AdditionalOptions': 'a_string', - 'CPreprocessOptions': 'a_string', - 'DefaultCharType': '0', - 'DLLDataFileName': 'a_file_name', - 'EnableErrorChecks': '2', - 'ErrorCheckAllocations': 'true', - 'ErrorCheckBounds': 'true', - 'ErrorCheckEnumRange': 'true', - 'ErrorCheckRefPointers': 'true', - 'ErrorCheckStubData': 'true', - 'GenerateStublessProxies': 'true', - 'GenerateTypeLibrary': 'true', - 'HeaderFileName': 'a_file_name', - 'IgnoreStandardIncludePath': 'true', - 'InterfaceIdentifierFileName': 'a_file_name', - 'MkTypLibCompatible': 'true', - 'OutputDirectory': 'a_string', - 'PreprocessorDefinitions': 'd1;d2;d3', - 'ProxyFileName': 'a_file_name', - 'RedirectOutputAndErrors': 'a_file_name', - 'StructMemberAlignment': '3', - 'SuppressStartupBanner': 'true', - 'TargetEnvironment': '1', - 'TypeLibraryName': 'a_file_name', - 'UndefinePreprocessorDefinitions': 'd1;d2;d3', - 'ValidateParameters': 'true', - 'WarnAsError': 'true', - 'WarningLevel': '4'}, - 'VCLibrarianTool': { - 'AdditionalDependencies': 'file1;file2;file3', - 'AdditionalLibraryDirectories': 'folder1;folder2;folder3', - 'AdditionalLibraryDirectories_excluded': 'folder1;folder2;folder3', - 'AdditionalOptions': 'a_string', - 'ExportNamedFunctions': 'd1;d2;d3', - 'ForceSymbolReferences': 'a_string', - 'IgnoreAllDefaultLibraries': 'true', - 'IgnoreSpecificDefaultLibraries': 'file1;file2;file3', - 'LinkLibraryDependencies': 'true', - 'ModuleDefinitionFile': 'a_file_name', - 'OutputFile': 'a_file_name', - 'SuppressStartupBanner': 'true', - 'UseUnicodeResponseFiles': 'true'}, - 'VCManifestTool': { - 'AdditionalManifestFiles': 'file1;file2;file3', - 'AdditionalOptions': 'a_string', - 'AssemblyIdentity': 'a_string', - 'ComponentFileName': 'a_file_name', - 'DependencyInformationFile': 'a_file_name', - 'EmbedManifest': 'true', - 'GenerateCatalogFiles': 'true', - 'InputResourceManifests': 'a_string', - 'ManifestResourceFile': 'my_name', - 'OutputManifestFile': 'a_file_name', - 'RegistrarScriptFile': 'a_file_name', - 'ReplacementsFile': 'a_file_name', - 'SuppressStartupBanner': 'true', - 'TypeLibraryFile': 'a_file_name', - 'UpdateFileHashes': 'true', - 'UpdateFileHashesSearchPath': 'a_file_name', - 'UseFAT32Workaround': 'true', - 'UseUnicodeResponseFiles': 'true', - 'VerboseOutput': 'true'}} - expected_msbuild_settings = { - 'ClCompile': { - 'AdditionalIncludeDirectories': 'folder1;folder2;folder3', - 'AdditionalOptions': 'a_string /J', - 'AdditionalUsingDirectories': 'folder1;folder2;folder3', - 'AssemblerListingLocation': 'a_file_name', - 'AssemblerOutput': 'NoListing', - 'BasicRuntimeChecks': 'StackFrameRuntimeCheck', - 'BrowseInformation': 'true', - 'BrowseInformationFile': 'a_file_name', - 'BufferSecurityCheck': 'true', - 'CallingConvention': 'Cdecl', - 'CompileAs': 'CompileAsC', - 'DebugInformationFormat': 'EditAndContinue', - 'DisableLanguageExtensions': 'true', - 'DisableSpecificWarnings': 'd1;d2;d3', - 'EnableEnhancedInstructionSet': 'NotSet', - 'EnableFiberSafeOptimizations': 'true', - 'EnablePREfast': 'true', - 'ErrorReporting': 'Prompt', - 'ExceptionHandling': 'Async', - 'ExpandAttributedSource': 'true', - 'FavorSizeOrSpeed': 'Neither', - 'FloatingPointExceptions': 'true', - 'FloatingPointModel': 'Strict', - 'ForceConformanceInForLoopScope': 'true', - 'ForcedIncludeFiles': 'file1;file2;file3', - 'ForcedUsingFiles': 'file1;file2;file3', - 'FunctionLevelLinking': 'true', - 'GenerateXMLDocumentationFiles': 'true', - 'IgnoreStandardIncludePath': 'true', - 'InlineFunctionExpansion': 'AnySuitable', - 'IntrinsicFunctions': 'true', - 'MinimalRebuild': 'true', - 'ObjectFileName': 'a_file_name', - 'OmitDefaultLibName': 'true', - 'OmitFramePointers': 'true', - 'OpenMPSupport': 'true', - 'Optimization': 'Full', - 'PrecompiledHeader': 'Create', - 'PrecompiledHeaderFile': 'a_file_name', - 'PrecompiledHeaderOutputFile': 'a_file_name', - 'PreprocessKeepComments': 'true', - 'PreprocessorDefinitions': 'd1;d2;d3', - 'PreprocessSuppressLineNumbers': 'false', - 'PreprocessToFile': 'true', - 'ProgramDataBaseFileName': 'a_file_name', - 'RuntimeLibrary': 'MultiThreaded', - 'RuntimeTypeInfo': 'true', - 'ShowIncludes': 'true', - 'SmallerTypeCheck': 'true', - 'StringPooling': 'true', - 'StructMemberAlignment': '1Byte', - 'SuppressStartupBanner': 'true', - 'TreatWarningAsError': 'true', - 'TreatWChar_tAsBuiltInType': 'true', - 'UndefineAllPreprocessorDefinitions': 'true', - 'UndefinePreprocessorDefinitions': 'd1;d2;d3', - 'UseFullPaths': 'true', - 'WarningLevel': 'Level2', - 'WholeProgramOptimization': 'true', - 'XMLDocumentationFileName': 'a_file_name'}, - 'Link': { - 'AdditionalDependencies': 'file1;file2;file3', - 'AdditionalLibraryDirectories': 'folder1;folder2;folder3', - 'AdditionalManifestDependencies': 'file1;file2;file3', - 'AdditionalOptions': 'a_string', - 'AddModuleNamesToAssembly': 'file1;file2;file3', - 'AllowIsolation': 'true', - 'AssemblyDebug': '', - 'AssemblyLinkResource': 'file1;file2;file3', - 'BaseAddress': 'a_string', - 'CLRImageType': 'ForceIJWImage', - 'CLRThreadAttribute': 'STAThreadingAttribute', - 'CLRUnmanagedCodeCheck': 'true', - 'DataExecutionPrevention': '', - 'DelayLoadDLLs': 'file1;file2;file3', - 'DelaySign': 'true', - 'Driver': 'Driver', - 'EmbedManagedResourceFile': 'file1;file2;file3', - 'EnableCOMDATFolding': '', - 'EnableUAC': 'true', - 'EntryPointSymbol': 'a_string', - 'FixedBaseAddress': 'false', - 'ForceSymbolReferences': 'file1;file2;file3', - 'FunctionOrder': 'a_file_name', - 'GenerateDebugInformation': 'true', - 'GenerateMapFile': 'true', - 'HeapCommitSize': 'a_string', - 'HeapReserveSize': 'a_string', - 'IgnoreAllDefaultLibraries': 'true', - 'IgnoreEmbeddedIDL': 'true', - 'IgnoreSpecificDefaultLibraries': 'file1;file2;file3', - 'ImportLibrary': 'a_file_name', - 'KeyContainer': 'a_file_name', - 'KeyFile': 'a_file_name', - 'LargeAddressAware': 'true', - 'LinkErrorReporting': 'NoErrorReport', - 'LinkTimeCodeGeneration': 'PGInstrument', - 'ManifestFile': 'a_file_name', - 'MapExports': 'true', - 'MapFileName': 'a_file_name', - 'MergedIDLBaseFileName': 'a_file_name', - 'MergeSections': 'a_string', - 'MidlCommandFile': 'a_file_name', - 'ModuleDefinitionFile': 'a_file_name', - 'NoEntryPoint': 'true', - 'OptimizeReferences': '', - 'OutputFile': 'a_file_name', - 'PerUserRedirection': 'true', - 'Profile': 'true', - 'ProfileGuidedDatabase': 'a_file_name', - 'ProgramDatabaseFile': 'a_file_name', - 'RandomizedBaseAddress': 'false', - 'RegisterOutput': 'true', - 'SetChecksum': 'true', - 'ShowProgress': 'NotSet', - 'StackCommitSize': 'a_string', - 'StackReserveSize': 'a_string', - 'StripPrivateSymbols': 'a_file_name', - 'SubSystem': 'Windows', - 'SupportUnloadOfDelayLoadedDLL': 'true', - 'SuppressStartupBanner': 'true', - 'SwapRunFromCD': 'true', - 'SwapRunFromNET': 'true', - 'TargetMachine': 'MachineARM', - 'TerminalServerAware': 'true', - 'TurnOffAssemblyGeneration': 'true', - 'TypeLibraryFile': 'a_file_name', - 'TypeLibraryResourceID': '33', - 'UACExecutionLevel': 'HighestAvailable', - 'UACUIAccess': 'true', - 'Version': 'a_string'}, - 'ResourceCompile': { - 'AdditionalIncludeDirectories': 'folder1;folder2;folder3', - 'AdditionalOptions': 'a_string', - 'Culture': '0x03eb', - 'IgnoreStandardIncludePath': 'true', - 'PreprocessorDefinitions': 'd1;d2;d3', - 'ResourceOutputFileName': 'a_string', - 'ShowProgress': 'true', - 'SuppressStartupBanner': 'true', - 'UndefinePreprocessorDefinitions': 'd1;d2;d3'}, - 'Midl': { - 'AdditionalIncludeDirectories': 'folder1;folder2;folder3', - 'AdditionalOptions': 'a_string', - 'CPreprocessOptions': 'a_string', - 'DefaultCharType': 'Unsigned', - 'DllDataFileName': 'a_file_name', - 'EnableErrorChecks': 'All', - 'ErrorCheckAllocations': 'true', - 'ErrorCheckBounds': 'true', - 'ErrorCheckEnumRange': 'true', - 'ErrorCheckRefPointers': 'true', - 'ErrorCheckStubData': 'true', - 'GenerateStublessProxies': 'true', - 'GenerateTypeLibrary': 'true', - 'HeaderFileName': 'a_file_name', - 'IgnoreStandardIncludePath': 'true', - 'InterfaceIdentifierFileName': 'a_file_name', - 'MkTypLibCompatible': 'true', - 'OutputDirectory': 'a_string', - 'PreprocessorDefinitions': 'd1;d2;d3', - 'ProxyFileName': 'a_file_name', - 'RedirectOutputAndErrors': 'a_file_name', - 'StructMemberAlignment': '4', - 'SuppressStartupBanner': 'true', - 'TargetEnvironment': 'Win32', - 'TypeLibraryName': 'a_file_name', - 'UndefinePreprocessorDefinitions': 'd1;d2;d3', - 'ValidateAllParameters': 'true', - 'WarnAsError': 'true', - 'WarningLevel': '4'}, - 'Lib': { - 'AdditionalDependencies': 'file1;file2;file3', - 'AdditionalLibraryDirectories': 'folder1;folder2;folder3', - 'AdditionalOptions': 'a_string', - 'ExportNamedFunctions': 'd1;d2;d3', - 'ForceSymbolReferences': 'a_string', - 'IgnoreAllDefaultLibraries': 'true', - 'IgnoreSpecificDefaultLibraries': 'file1;file2;file3', - 'ModuleDefinitionFile': 'a_file_name', - 'OutputFile': 'a_file_name', - 'SuppressStartupBanner': 'true', - 'UseUnicodeResponseFiles': 'true'}, - 'Manifest': { - 'AdditionalManifestFiles': 'file1;file2;file3', - 'AdditionalOptions': 'a_string', - 'AssemblyIdentity': 'a_string', - 'ComponentFileName': 'a_file_name', - 'GenerateCatalogFiles': 'true', - 'InputResourceManifests': 'a_string', - 'OutputManifestFile': 'a_file_name', - 'RegistrarScriptFile': 'a_file_name', - 'ReplacementsFile': 'a_file_name', - 'SuppressStartupBanner': 'true', - 'TypeLibraryFile': 'a_file_name', - 'UpdateFileHashes': 'true', - 'UpdateFileHashesSearchPath': 'a_file_name', - 'VerboseOutput': 'true'}, - 'ManifestResourceCompile': { - 'ResourceOutputFileName': 'my_name'}, - 'ProjectReference': { - 'LinkLibraryDependencies': 'true', - 'UseLibraryDependencyInputs': 'false'}, - '': { - 'EmbedManifest': 'true', - 'GenerateManifest': 'true', - 'IgnoreImportLibrary': 'true', - 'LinkIncremental': 'false'}} - self.maxDiff = 9999 # on failure display a long diff - actual_msbuild_settings = MSVSSettings.ConvertToMSBuildSettings( - msvs_settings, - self.stderr) - self.assertEqual(expected_msbuild_settings, actual_msbuild_settings) - self._ExpectedWarnings([]) + def testConvertToMSBuildSettings_full_synthetic(self): + """Tests conversion of all the MSBuild settings.""" + msvs_settings = { + "VCCLCompilerTool": { + "AdditionalIncludeDirectories": "folder1;folder2;folder3", + "AdditionalOptions": "a_string", + "AdditionalUsingDirectories": "folder1;folder2;folder3", + "AssemblerListingLocation": "a_file_name", + "AssemblerOutput": "0", + "BasicRuntimeChecks": "1", + "BrowseInformation": "2", + "BrowseInformationFile": "a_file_name", + "BufferSecurityCheck": "true", + "CallingConvention": "0", + "CompileAs": "1", + "DebugInformationFormat": "4", + "DefaultCharIsUnsigned": "true", + "Detect64BitPortabilityProblems": "true", + "DisableLanguageExtensions": "true", + "DisableSpecificWarnings": "d1;d2;d3", + "EnableEnhancedInstructionSet": "0", + "EnableFiberSafeOptimizations": "true", + "EnableFunctionLevelLinking": "true", + "EnableIntrinsicFunctions": "true", + "EnablePREfast": "true", + "ErrorReporting": "1", + "ExceptionHandling": "2", + "ExpandAttributedSource": "true", + "FavorSizeOrSpeed": "0", + "FloatingPointExceptions": "true", + "FloatingPointModel": "1", + "ForceConformanceInForLoopScope": "true", + "ForcedIncludeFiles": "file1;file2;file3", + "ForcedUsingFiles": "file1;file2;file3", + "GeneratePreprocessedFile": "1", + "GenerateXMLDocumentationFiles": "true", + "IgnoreStandardIncludePath": "true", + "InlineFunctionExpansion": "2", + "KeepComments": "true", + "MinimalRebuild": "true", + "ObjectFile": "a_file_name", + "OmitDefaultLibName": "true", + "OmitFramePointers": "true", + "OpenMP": "true", + "Optimization": "3", + "PrecompiledHeaderFile": "a_file_name", + "PrecompiledHeaderThrough": "a_file_name", + "PreprocessorDefinitions": "d1;d2;d3", + "ProgramDataBaseFileName": "a_file_name", + "RuntimeLibrary": "0", + "RuntimeTypeInfo": "true", + "ShowIncludes": "true", + "SmallerTypeCheck": "true", + "StringPooling": "true", + "StructMemberAlignment": "1", + "SuppressStartupBanner": "true", + "TreatWChar_tAsBuiltInType": "true", + "UndefineAllPreprocessorDefinitions": "true", + "UndefinePreprocessorDefinitions": "d1;d2;d3", + "UseFullPaths": "true", + "UsePrecompiledHeader": "1", + "UseUnicodeResponseFiles": "true", + "WarnAsError": "true", + "WarningLevel": "2", + "WholeProgramOptimization": "true", + "XMLDocumentationFileName": "a_file_name", + }, + "VCLinkerTool": { + "AdditionalDependencies": "file1;file2;file3", + "AdditionalLibraryDirectories": "folder1;folder2;folder3", + "AdditionalLibraryDirectories_excluded": "folder1;folder2;folder3", + "AdditionalManifestDependencies": "file1;file2;file3", + "AdditionalOptions": "a_string", + "AddModuleNamesToAssembly": "file1;file2;file3", + "AllowIsolation": "true", + "AssemblyDebug": "0", + "AssemblyLinkResource": "file1;file2;file3", + "BaseAddress": "a_string", + "CLRImageType": "1", + "CLRThreadAttribute": "2", + "CLRUnmanagedCodeCheck": "true", + "DataExecutionPrevention": "0", + "DelayLoadDLLs": "file1;file2;file3", + "DelaySign": "true", + "Driver": "1", + "EmbedManagedResourceFile": "file1;file2;file3", + "EnableCOMDATFolding": "0", + "EnableUAC": "true", + "EntryPointSymbol": "a_string", + "ErrorReporting": "0", + "FixedBaseAddress": "1", + "ForceSymbolReferences": "file1;file2;file3", + "FunctionOrder": "a_file_name", + "GenerateDebugInformation": "true", + "GenerateManifest": "true", + "GenerateMapFile": "true", + "HeapCommitSize": "a_string", + "HeapReserveSize": "a_string", + "IgnoreAllDefaultLibraries": "true", + "IgnoreDefaultLibraryNames": "file1;file2;file3", + "IgnoreEmbeddedIDL": "true", + "IgnoreImportLibrary": "true", + "ImportLibrary": "a_file_name", + "KeyContainer": "a_file_name", + "KeyFile": "a_file_name", + "LargeAddressAware": "2", + "LinkIncremental": "1", + "LinkLibraryDependencies": "true", + "LinkTimeCodeGeneration": "2", + "ManifestFile": "a_file_name", + "MapExports": "true", + "MapFileName": "a_file_name", + "MergedIDLBaseFileName": "a_file_name", + "MergeSections": "a_string", + "MidlCommandFile": "a_file_name", + "ModuleDefinitionFile": "a_file_name", + "OptimizeForWindows98": "1", + "OptimizeReferences": "0", + "OutputFile": "a_file_name", + "PerUserRedirection": "true", + "Profile": "true", + "ProfileGuidedDatabase": "a_file_name", + "ProgramDatabaseFile": "a_file_name", + "RandomizedBaseAddress": "1", + "RegisterOutput": "true", + "ResourceOnlyDLL": "true", + "SetChecksum": "true", + "ShowProgress": "0", + "StackCommitSize": "a_string", + "StackReserveSize": "a_string", + "StripPrivateSymbols": "a_file_name", + "SubSystem": "2", + "SupportUnloadOfDelayLoadedDLL": "true", + "SuppressStartupBanner": "true", + "SwapRunFromCD": "true", + "SwapRunFromNet": "true", + "TargetMachine": "3", + "TerminalServerAware": "2", + "TurnOffAssemblyGeneration": "true", + "TypeLibraryFile": "a_file_name", + "TypeLibraryResourceID": "33", + "UACExecutionLevel": "1", + "UACUIAccess": "true", + "UseLibraryDependencyInputs": "false", + "UseUnicodeResponseFiles": "true", + "Version": "a_string", + }, + "VCResourceCompilerTool": { + "AdditionalIncludeDirectories": "folder1;folder2;folder3", + "AdditionalOptions": "a_string", + "Culture": "1003", + "IgnoreStandardIncludePath": "true", + "PreprocessorDefinitions": "d1;d2;d3", + "ResourceOutputFileName": "a_string", + "ShowProgress": "true", + "SuppressStartupBanner": "true", + "UndefinePreprocessorDefinitions": "d1;d2;d3", + }, + "VCMIDLTool": { + "AdditionalIncludeDirectories": "folder1;folder2;folder3", + "AdditionalOptions": "a_string", + "CPreprocessOptions": "a_string", + "DefaultCharType": "0", + "DLLDataFileName": "a_file_name", + "EnableErrorChecks": "2", + "ErrorCheckAllocations": "true", + "ErrorCheckBounds": "true", + "ErrorCheckEnumRange": "true", + "ErrorCheckRefPointers": "true", + "ErrorCheckStubData": "true", + "GenerateStublessProxies": "true", + "GenerateTypeLibrary": "true", + "HeaderFileName": "a_file_name", + "IgnoreStandardIncludePath": "true", + "InterfaceIdentifierFileName": "a_file_name", + "MkTypLibCompatible": "true", + "OutputDirectory": "a_string", + "PreprocessorDefinitions": "d1;d2;d3", + "ProxyFileName": "a_file_name", + "RedirectOutputAndErrors": "a_file_name", + "StructMemberAlignment": "3", + "SuppressStartupBanner": "true", + "TargetEnvironment": "1", + "TypeLibraryName": "a_file_name", + "UndefinePreprocessorDefinitions": "d1;d2;d3", + "ValidateParameters": "true", + "WarnAsError": "true", + "WarningLevel": "4", + }, + "VCLibrarianTool": { + "AdditionalDependencies": "file1;file2;file3", + "AdditionalLibraryDirectories": "folder1;folder2;folder3", + "AdditionalLibraryDirectories_excluded": "folder1;folder2;folder3", + "AdditionalOptions": "a_string", + "ExportNamedFunctions": "d1;d2;d3", + "ForceSymbolReferences": "a_string", + "IgnoreAllDefaultLibraries": "true", + "IgnoreSpecificDefaultLibraries": "file1;file2;file3", + "LinkLibraryDependencies": "true", + "ModuleDefinitionFile": "a_file_name", + "OutputFile": "a_file_name", + "SuppressStartupBanner": "true", + "UseUnicodeResponseFiles": "true", + }, + "VCManifestTool": { + "AdditionalManifestFiles": "file1;file2;file3", + "AdditionalOptions": "a_string", + "AssemblyIdentity": "a_string", + "ComponentFileName": "a_file_name", + "DependencyInformationFile": "a_file_name", + "EmbedManifest": "true", + "GenerateCatalogFiles": "true", + "InputResourceManifests": "a_string", + "ManifestResourceFile": "my_name", + "OutputManifestFile": "a_file_name", + "RegistrarScriptFile": "a_file_name", + "ReplacementsFile": "a_file_name", + "SuppressStartupBanner": "true", + "TypeLibraryFile": "a_file_name", + "UpdateFileHashes": "true", + "UpdateFileHashesSearchPath": "a_file_name", + "UseFAT32Workaround": "true", + "UseUnicodeResponseFiles": "true", + "VerboseOutput": "true", + }, + } + expected_msbuild_settings = { + "ClCompile": { + "AdditionalIncludeDirectories": "folder1;folder2;folder3", + "AdditionalOptions": "a_string /J", + "AdditionalUsingDirectories": "folder1;folder2;folder3", + "AssemblerListingLocation": "a_file_name", + "AssemblerOutput": "NoListing", + "BasicRuntimeChecks": "StackFrameRuntimeCheck", + "BrowseInformation": "true", + "BrowseInformationFile": "a_file_name", + "BufferSecurityCheck": "true", + "CallingConvention": "Cdecl", + "CompileAs": "CompileAsC", + "DebugInformationFormat": "EditAndContinue", + "DisableLanguageExtensions": "true", + "DisableSpecificWarnings": "d1;d2;d3", + "EnableEnhancedInstructionSet": "NotSet", + "EnableFiberSafeOptimizations": "true", + "EnablePREfast": "true", + "ErrorReporting": "Prompt", + "ExceptionHandling": "Async", + "ExpandAttributedSource": "true", + "FavorSizeOrSpeed": "Neither", + "FloatingPointExceptions": "true", + "FloatingPointModel": "Strict", + "ForceConformanceInForLoopScope": "true", + "ForcedIncludeFiles": "file1;file2;file3", + "ForcedUsingFiles": "file1;file2;file3", + "FunctionLevelLinking": "true", + "GenerateXMLDocumentationFiles": "true", + "IgnoreStandardIncludePath": "true", + "InlineFunctionExpansion": "AnySuitable", + "IntrinsicFunctions": "true", + "MinimalRebuild": "true", + "ObjectFileName": "a_file_name", + "OmitDefaultLibName": "true", + "OmitFramePointers": "true", + "OpenMPSupport": "true", + "Optimization": "Full", + "PrecompiledHeader": "Create", + "PrecompiledHeaderFile": "a_file_name", + "PrecompiledHeaderOutputFile": "a_file_name", + "PreprocessKeepComments": "true", + "PreprocessorDefinitions": "d1;d2;d3", + "PreprocessSuppressLineNumbers": "false", + "PreprocessToFile": "true", + "ProgramDataBaseFileName": "a_file_name", + "RuntimeLibrary": "MultiThreaded", + "RuntimeTypeInfo": "true", + "ShowIncludes": "true", + "SmallerTypeCheck": "true", + "StringPooling": "true", + "StructMemberAlignment": "1Byte", + "SuppressStartupBanner": "true", + "TreatWarningAsError": "true", + "TreatWChar_tAsBuiltInType": "true", + "UndefineAllPreprocessorDefinitions": "true", + "UndefinePreprocessorDefinitions": "d1;d2;d3", + "UseFullPaths": "true", + "WarningLevel": "Level2", + "WholeProgramOptimization": "true", + "XMLDocumentationFileName": "a_file_name", + }, + "Link": { + "AdditionalDependencies": "file1;file2;file3", + "AdditionalLibraryDirectories": "folder1;folder2;folder3", + "AdditionalManifestDependencies": "file1;file2;file3", + "AdditionalOptions": "a_string", + "AddModuleNamesToAssembly": "file1;file2;file3", + "AllowIsolation": "true", + "AssemblyDebug": "", + "AssemblyLinkResource": "file1;file2;file3", + "BaseAddress": "a_string", + "CLRImageType": "ForceIJWImage", + "CLRThreadAttribute": "STAThreadingAttribute", + "CLRUnmanagedCodeCheck": "true", + "DataExecutionPrevention": "", + "DelayLoadDLLs": "file1;file2;file3", + "DelaySign": "true", + "Driver": "Driver", + "EmbedManagedResourceFile": "file1;file2;file3", + "EnableCOMDATFolding": "", + "EnableUAC": "true", + "EntryPointSymbol": "a_string", + "FixedBaseAddress": "false", + "ForceSymbolReferences": "file1;file2;file3", + "FunctionOrder": "a_file_name", + "GenerateDebugInformation": "true", + "GenerateMapFile": "true", + "HeapCommitSize": "a_string", + "HeapReserveSize": "a_string", + "IgnoreAllDefaultLibraries": "true", + "IgnoreEmbeddedIDL": "true", + "IgnoreSpecificDefaultLibraries": "file1;file2;file3", + "ImportLibrary": "a_file_name", + "KeyContainer": "a_file_name", + "KeyFile": "a_file_name", + "LargeAddressAware": "true", + "LinkErrorReporting": "NoErrorReport", + "LinkTimeCodeGeneration": "PGInstrument", + "ManifestFile": "a_file_name", + "MapExports": "true", + "MapFileName": "a_file_name", + "MergedIDLBaseFileName": "a_file_name", + "MergeSections": "a_string", + "MidlCommandFile": "a_file_name", + "ModuleDefinitionFile": "a_file_name", + "NoEntryPoint": "true", + "OptimizeReferences": "", + "OutputFile": "a_file_name", + "PerUserRedirection": "true", + "Profile": "true", + "ProfileGuidedDatabase": "a_file_name", + "ProgramDatabaseFile": "a_file_name", + "RandomizedBaseAddress": "false", + "RegisterOutput": "true", + "SetChecksum": "true", + "ShowProgress": "NotSet", + "StackCommitSize": "a_string", + "StackReserveSize": "a_string", + "StripPrivateSymbols": "a_file_name", + "SubSystem": "Windows", + "SupportUnloadOfDelayLoadedDLL": "true", + "SuppressStartupBanner": "true", + "SwapRunFromCD": "true", + "SwapRunFromNET": "true", + "TargetMachine": "MachineARM", + "TerminalServerAware": "true", + "TurnOffAssemblyGeneration": "true", + "TypeLibraryFile": "a_file_name", + "TypeLibraryResourceID": "33", + "UACExecutionLevel": "HighestAvailable", + "UACUIAccess": "true", + "Version": "a_string", + }, + "ResourceCompile": { + "AdditionalIncludeDirectories": "folder1;folder2;folder3", + "AdditionalOptions": "a_string", + "Culture": "0x03eb", + "IgnoreStandardIncludePath": "true", + "PreprocessorDefinitions": "d1;d2;d3", + "ResourceOutputFileName": "a_string", + "ShowProgress": "true", + "SuppressStartupBanner": "true", + "UndefinePreprocessorDefinitions": "d1;d2;d3", + }, + "Midl": { + "AdditionalIncludeDirectories": "folder1;folder2;folder3", + "AdditionalOptions": "a_string", + "CPreprocessOptions": "a_string", + "DefaultCharType": "Unsigned", + "DllDataFileName": "a_file_name", + "EnableErrorChecks": "All", + "ErrorCheckAllocations": "true", + "ErrorCheckBounds": "true", + "ErrorCheckEnumRange": "true", + "ErrorCheckRefPointers": "true", + "ErrorCheckStubData": "true", + "GenerateStublessProxies": "true", + "GenerateTypeLibrary": "true", + "HeaderFileName": "a_file_name", + "IgnoreStandardIncludePath": "true", + "InterfaceIdentifierFileName": "a_file_name", + "MkTypLibCompatible": "true", + "OutputDirectory": "a_string", + "PreprocessorDefinitions": "d1;d2;d3", + "ProxyFileName": "a_file_name", + "RedirectOutputAndErrors": "a_file_name", + "StructMemberAlignment": "4", + "SuppressStartupBanner": "true", + "TargetEnvironment": "Win32", + "TypeLibraryName": "a_file_name", + "UndefinePreprocessorDefinitions": "d1;d2;d3", + "ValidateAllParameters": "true", + "WarnAsError": "true", + "WarningLevel": "4", + }, + "Lib": { + "AdditionalDependencies": "file1;file2;file3", + "AdditionalLibraryDirectories": "folder1;folder2;folder3", + "AdditionalOptions": "a_string", + "ExportNamedFunctions": "d1;d2;d3", + "ForceSymbolReferences": "a_string", + "IgnoreAllDefaultLibraries": "true", + "IgnoreSpecificDefaultLibraries": "file1;file2;file3", + "ModuleDefinitionFile": "a_file_name", + "OutputFile": "a_file_name", + "SuppressStartupBanner": "true", + "UseUnicodeResponseFiles": "true", + }, + "Manifest": { + "AdditionalManifestFiles": "file1;file2;file3", + "AdditionalOptions": "a_string", + "AssemblyIdentity": "a_string", + "ComponentFileName": "a_file_name", + "GenerateCatalogFiles": "true", + "InputResourceManifests": "a_string", + "OutputManifestFile": "a_file_name", + "RegistrarScriptFile": "a_file_name", + "ReplacementsFile": "a_file_name", + "SuppressStartupBanner": "true", + "TypeLibraryFile": "a_file_name", + "UpdateFileHashes": "true", + "UpdateFileHashesSearchPath": "a_file_name", + "VerboseOutput": "true", + }, + "ManifestResourceCompile": {"ResourceOutputFileName": "my_name"}, + "ProjectReference": { + "LinkLibraryDependencies": "true", + "UseLibraryDependencyInputs": "false", + }, + "": { + "EmbedManifest": "true", + "GenerateManifest": "true", + "IgnoreImportLibrary": "true", + "LinkIncremental": "false", + }, + } + self.maxDiff = 9999 # on failure display a long diff + actual_msbuild_settings = MSVSSettings.ConvertToMSBuildSettings( + msvs_settings, self.stderr + ) + self.assertEqual(expected_msbuild_settings, actual_msbuild_settings) + self._ExpectedWarnings([]) - def testConvertToMSBuildSettings_actual(self): - """Tests the conversion of an actual project. + def testConvertToMSBuildSettings_actual(self): + """Tests the conversion of an actual project. A VS2008 project with most of the options defined was created through the VS2008 IDE. It was then converted to VS2010. The tool settings found in @@ -1136,354 +1186,362 @@ def testConvertToMSBuildSettings_actual(self): AdditionalOptions: ' %(AdditionalOptions)', InputResourceManifests: ';%(InputResourceManifests)', """ - msvs_settings = { - 'VCCLCompilerTool': { - 'AdditionalIncludeDirectories': 'dir1', - 'AdditionalOptions': '/more', - 'AdditionalUsingDirectories': 'test', - 'AssemblerListingLocation': '$(IntDir)\\a', - 'AssemblerOutput': '1', - 'BasicRuntimeChecks': '3', - 'BrowseInformation': '1', - 'BrowseInformationFile': '$(IntDir)\\e', - 'BufferSecurityCheck': 'false', - 'CallingConvention': '1', - 'CompileAs': '1', - 'DebugInformationFormat': '4', - 'DefaultCharIsUnsigned': 'true', - 'Detect64BitPortabilityProblems': 'true', - 'DisableLanguageExtensions': 'true', - 'DisableSpecificWarnings': 'abc', - 'EnableEnhancedInstructionSet': '1', - 'EnableFiberSafeOptimizations': 'true', - 'EnableFunctionLevelLinking': 'true', - 'EnableIntrinsicFunctions': 'true', - 'EnablePREfast': 'true', - 'ErrorReporting': '2', - 'ExceptionHandling': '2', - 'ExpandAttributedSource': 'true', - 'FavorSizeOrSpeed': '2', - 'FloatingPointExceptions': 'true', - 'FloatingPointModel': '1', - 'ForceConformanceInForLoopScope': 'false', - 'ForcedIncludeFiles': 'def', - 'ForcedUsingFiles': 'ge', - 'GeneratePreprocessedFile': '2', - 'GenerateXMLDocumentationFiles': 'true', - 'IgnoreStandardIncludePath': 'true', - 'InlineFunctionExpansion': '1', - 'KeepComments': 'true', - 'MinimalRebuild': 'true', - 'ObjectFile': '$(IntDir)\\b', - 'OmitDefaultLibName': 'true', - 'OmitFramePointers': 'true', - 'OpenMP': 'true', - 'Optimization': '3', - 'PrecompiledHeaderFile': '$(IntDir)\\$(TargetName).pche', - 'PrecompiledHeaderThrough': 'StdAfx.hd', - 'PreprocessorDefinitions': 'WIN32;_DEBUG;_CONSOLE', - 'ProgramDataBaseFileName': '$(IntDir)\\vc90b.pdb', - 'RuntimeLibrary': '3', - 'RuntimeTypeInfo': 'false', - 'ShowIncludes': 'true', - 'SmallerTypeCheck': 'true', - 'StringPooling': 'true', - 'StructMemberAlignment': '3', - 'SuppressStartupBanner': 'false', - 'TreatWChar_tAsBuiltInType': 'false', - 'UndefineAllPreprocessorDefinitions': 'true', - 'UndefinePreprocessorDefinitions': 'wer', - 'UseFullPaths': 'true', - 'UsePrecompiledHeader': '0', - 'UseUnicodeResponseFiles': 'false', - 'WarnAsError': 'true', - 'WarningLevel': '3', - 'WholeProgramOptimization': 'true', - 'XMLDocumentationFileName': '$(IntDir)\\c'}, - 'VCLinkerTool': { - 'AdditionalDependencies': 'zx', - 'AdditionalLibraryDirectories': 'asd', - 'AdditionalManifestDependencies': 's2', - 'AdditionalOptions': '/mor2', - 'AddModuleNamesToAssembly': 'd1', - 'AllowIsolation': 'false', - 'AssemblyDebug': '1', - 'AssemblyLinkResource': 'd5', - 'BaseAddress': '23423', - 'CLRImageType': '3', - 'CLRThreadAttribute': '1', - 'CLRUnmanagedCodeCheck': 'true', - 'DataExecutionPrevention': '0', - 'DelayLoadDLLs': 'd4', - 'DelaySign': 'true', - 'Driver': '2', - 'EmbedManagedResourceFile': 'd2', - 'EnableCOMDATFolding': '1', - 'EnableUAC': 'false', - 'EntryPointSymbol': 'f5', - 'ErrorReporting': '2', - 'FixedBaseAddress': '1', - 'ForceSymbolReferences': 'd3', - 'FunctionOrder': 'fssdfsd', - 'GenerateDebugInformation': 'true', - 'GenerateManifest': 'false', - 'GenerateMapFile': 'true', - 'HeapCommitSize': '13', - 'HeapReserveSize': '12', - 'IgnoreAllDefaultLibraries': 'true', - 'IgnoreDefaultLibraryNames': 'flob;flok', - 'IgnoreEmbeddedIDL': 'true', - 'IgnoreImportLibrary': 'true', - 'ImportLibrary': 'f4', - 'KeyContainer': 'f7', - 'KeyFile': 'f6', - 'LargeAddressAware': '2', - 'LinkIncremental': '0', - 'LinkLibraryDependencies': 'false', - 'LinkTimeCodeGeneration': '1', - 'ManifestFile': - '$(IntDir)\\$(TargetFileName).2intermediate.manifest', - 'MapExports': 'true', - 'MapFileName': 'd5', - 'MergedIDLBaseFileName': 'f2', - 'MergeSections': 'f5', - 'MidlCommandFile': 'f1', - 'ModuleDefinitionFile': 'sdsd', - 'OptimizeForWindows98': '2', - 'OptimizeReferences': '2', - 'OutputFile': '$(OutDir)\\$(ProjectName)2.exe', - 'PerUserRedirection': 'true', - 'Profile': 'true', - 'ProfileGuidedDatabase': '$(TargetDir)$(TargetName).pgdd', - 'ProgramDatabaseFile': 'Flob.pdb', - 'RandomizedBaseAddress': '1', - 'RegisterOutput': 'true', - 'ResourceOnlyDLL': 'true', - 'SetChecksum': 'false', - 'ShowProgress': '1', - 'StackCommitSize': '15', - 'StackReserveSize': '14', - 'StripPrivateSymbols': 'd3', - 'SubSystem': '1', - 'SupportUnloadOfDelayLoadedDLL': 'true', - 'SuppressStartupBanner': 'false', - 'SwapRunFromCD': 'true', - 'SwapRunFromNet': 'true', - 'TargetMachine': '1', - 'TerminalServerAware': '1', - 'TurnOffAssemblyGeneration': 'true', - 'TypeLibraryFile': 'f3', - 'TypeLibraryResourceID': '12', - 'UACExecutionLevel': '2', - 'UACUIAccess': 'true', - 'UseLibraryDependencyInputs': 'true', - 'UseUnicodeResponseFiles': 'false', - 'Version': '333'}, - 'VCResourceCompilerTool': { - 'AdditionalIncludeDirectories': 'f3', - 'AdditionalOptions': '/more3', - 'Culture': '3084', - 'IgnoreStandardIncludePath': 'true', - 'PreprocessorDefinitions': '_UNICODE;UNICODE2', - 'ResourceOutputFileName': '$(IntDir)/$(InputName)3.res', - 'ShowProgress': 'true'}, - 'VCManifestTool': { - 'AdditionalManifestFiles': 'sfsdfsd', - 'AdditionalOptions': 'afdsdafsd', - 'AssemblyIdentity': 'sddfdsadfsa', - 'ComponentFileName': 'fsdfds', - 'DependencyInformationFile': '$(IntDir)\\mt.depdfd', - 'EmbedManifest': 'false', - 'GenerateCatalogFiles': 'true', - 'InputResourceManifests': 'asfsfdafs', - 'ManifestResourceFile': - '$(IntDir)\\$(TargetFileName).embed.manifest.resfdsf', - 'OutputManifestFile': '$(TargetPath).manifestdfs', - 'RegistrarScriptFile': 'sdfsfd', - 'ReplacementsFile': 'sdffsd', - 'SuppressStartupBanner': 'false', - 'TypeLibraryFile': 'sfsd', - 'UpdateFileHashes': 'true', - 'UpdateFileHashesSearchPath': 'sfsd', - 'UseFAT32Workaround': 'true', - 'UseUnicodeResponseFiles': 'false', - 'VerboseOutput': 'true'}} - expected_msbuild_settings = { - 'ClCompile': { - 'AdditionalIncludeDirectories': 'dir1', - 'AdditionalOptions': '/more /J', - 'AdditionalUsingDirectories': 'test', - 'AssemblerListingLocation': '$(IntDir)a', - 'AssemblerOutput': 'AssemblyCode', - 'BasicRuntimeChecks': 'EnableFastChecks', - 'BrowseInformation': 'true', - 'BrowseInformationFile': '$(IntDir)e', - 'BufferSecurityCheck': 'false', - 'CallingConvention': 'FastCall', - 'CompileAs': 'CompileAsC', - 'DebugInformationFormat': 'EditAndContinue', - 'DisableLanguageExtensions': 'true', - 'DisableSpecificWarnings': 'abc', - 'EnableEnhancedInstructionSet': 'StreamingSIMDExtensions', - 'EnableFiberSafeOptimizations': 'true', - 'EnablePREfast': 'true', - 'ErrorReporting': 'Queue', - 'ExceptionHandling': 'Async', - 'ExpandAttributedSource': 'true', - 'FavorSizeOrSpeed': 'Size', - 'FloatingPointExceptions': 'true', - 'FloatingPointModel': 'Strict', - 'ForceConformanceInForLoopScope': 'false', - 'ForcedIncludeFiles': 'def', - 'ForcedUsingFiles': 'ge', - 'FunctionLevelLinking': 'true', - 'GenerateXMLDocumentationFiles': 'true', - 'IgnoreStandardIncludePath': 'true', - 'InlineFunctionExpansion': 'OnlyExplicitInline', - 'IntrinsicFunctions': 'true', - 'MinimalRebuild': 'true', - 'ObjectFileName': '$(IntDir)b', - 'OmitDefaultLibName': 'true', - 'OmitFramePointers': 'true', - 'OpenMPSupport': 'true', - 'Optimization': 'Full', - 'PrecompiledHeader': 'NotUsing', # Actual conversion gives '' - 'PrecompiledHeaderFile': 'StdAfx.hd', - 'PrecompiledHeaderOutputFile': '$(IntDir)$(TargetName).pche', - 'PreprocessKeepComments': 'true', - 'PreprocessorDefinitions': 'WIN32;_DEBUG;_CONSOLE', - 'PreprocessSuppressLineNumbers': 'true', - 'PreprocessToFile': 'true', - 'ProgramDataBaseFileName': '$(IntDir)vc90b.pdb', - 'RuntimeLibrary': 'MultiThreadedDebugDLL', - 'RuntimeTypeInfo': 'false', - 'ShowIncludes': 'true', - 'SmallerTypeCheck': 'true', - 'StringPooling': 'true', - 'StructMemberAlignment': '4Bytes', - 'SuppressStartupBanner': 'false', - 'TreatWarningAsError': 'true', - 'TreatWChar_tAsBuiltInType': 'false', - 'UndefineAllPreprocessorDefinitions': 'true', - 'UndefinePreprocessorDefinitions': 'wer', - 'UseFullPaths': 'true', - 'WarningLevel': 'Level3', - 'WholeProgramOptimization': 'true', - 'XMLDocumentationFileName': '$(IntDir)c'}, - 'Link': { - 'AdditionalDependencies': 'zx', - 'AdditionalLibraryDirectories': 'asd', - 'AdditionalManifestDependencies': 's2', - 'AdditionalOptions': '/mor2', - 'AddModuleNamesToAssembly': 'd1', - 'AllowIsolation': 'false', - 'AssemblyDebug': 'true', - 'AssemblyLinkResource': 'd5', - 'BaseAddress': '23423', - 'CLRImageType': 'ForceSafeILImage', - 'CLRThreadAttribute': 'MTAThreadingAttribute', - 'CLRUnmanagedCodeCheck': 'true', - 'DataExecutionPrevention': '', - 'DelayLoadDLLs': 'd4', - 'DelaySign': 'true', - 'Driver': 'UpOnly', - 'EmbedManagedResourceFile': 'd2', - 'EnableCOMDATFolding': 'false', - 'EnableUAC': 'false', - 'EntryPointSymbol': 'f5', - 'FixedBaseAddress': 'false', - 'ForceSymbolReferences': 'd3', - 'FunctionOrder': 'fssdfsd', - 'GenerateDebugInformation': 'true', - 'GenerateMapFile': 'true', - 'HeapCommitSize': '13', - 'HeapReserveSize': '12', - 'IgnoreAllDefaultLibraries': 'true', - 'IgnoreEmbeddedIDL': 'true', - 'IgnoreSpecificDefaultLibraries': 'flob;flok', - 'ImportLibrary': 'f4', - 'KeyContainer': 'f7', - 'KeyFile': 'f6', - 'LargeAddressAware': 'true', - 'LinkErrorReporting': 'QueueForNextLogin', - 'LinkTimeCodeGeneration': 'UseLinkTimeCodeGeneration', - 'ManifestFile': '$(IntDir)$(TargetFileName).2intermediate.manifest', - 'MapExports': 'true', - 'MapFileName': 'd5', - 'MergedIDLBaseFileName': 'f2', - 'MergeSections': 'f5', - 'MidlCommandFile': 'f1', - 'ModuleDefinitionFile': 'sdsd', - 'NoEntryPoint': 'true', - 'OptimizeReferences': 'true', - 'OutputFile': '$(OutDir)$(ProjectName)2.exe', - 'PerUserRedirection': 'true', - 'Profile': 'true', - 'ProfileGuidedDatabase': '$(TargetDir)$(TargetName).pgdd', - 'ProgramDatabaseFile': 'Flob.pdb', - 'RandomizedBaseAddress': 'false', - 'RegisterOutput': 'true', - 'SetChecksum': 'false', - 'ShowProgress': 'LinkVerbose', - 'StackCommitSize': '15', - 'StackReserveSize': '14', - 'StripPrivateSymbols': 'd3', - 'SubSystem': 'Console', - 'SupportUnloadOfDelayLoadedDLL': 'true', - 'SuppressStartupBanner': 'false', - 'SwapRunFromCD': 'true', - 'SwapRunFromNET': 'true', - 'TargetMachine': 'MachineX86', - 'TerminalServerAware': 'false', - 'TurnOffAssemblyGeneration': 'true', - 'TypeLibraryFile': 'f3', - 'TypeLibraryResourceID': '12', - 'UACExecutionLevel': 'RequireAdministrator', - 'UACUIAccess': 'true', - 'Version': '333'}, - 'ResourceCompile': { - 'AdditionalIncludeDirectories': 'f3', - 'AdditionalOptions': '/more3', - 'Culture': '0x0c0c', - 'IgnoreStandardIncludePath': 'true', - 'PreprocessorDefinitions': '_UNICODE;UNICODE2', - 'ResourceOutputFileName': '$(IntDir)%(Filename)3.res', - 'ShowProgress': 'true'}, - 'Manifest': { - 'AdditionalManifestFiles': 'sfsdfsd', - 'AdditionalOptions': 'afdsdafsd', - 'AssemblyIdentity': 'sddfdsadfsa', - 'ComponentFileName': 'fsdfds', - 'GenerateCatalogFiles': 'true', - 'InputResourceManifests': 'asfsfdafs', - 'OutputManifestFile': '$(TargetPath).manifestdfs', - 'RegistrarScriptFile': 'sdfsfd', - 'ReplacementsFile': 'sdffsd', - 'SuppressStartupBanner': 'false', - 'TypeLibraryFile': 'sfsd', - 'UpdateFileHashes': 'true', - 'UpdateFileHashesSearchPath': 'sfsd', - 'VerboseOutput': 'true'}, - 'ProjectReference': { - 'LinkLibraryDependencies': 'false', - 'UseLibraryDependencyInputs': 'true'}, - '': { - 'EmbedManifest': 'false', - 'GenerateManifest': 'false', - 'IgnoreImportLibrary': 'true', - 'LinkIncremental': '' + msvs_settings = { + "VCCLCompilerTool": { + "AdditionalIncludeDirectories": "dir1", + "AdditionalOptions": "/more", + "AdditionalUsingDirectories": "test", + "AssemblerListingLocation": "$(IntDir)\\a", + "AssemblerOutput": "1", + "BasicRuntimeChecks": "3", + "BrowseInformation": "1", + "BrowseInformationFile": "$(IntDir)\\e", + "BufferSecurityCheck": "false", + "CallingConvention": "1", + "CompileAs": "1", + "DebugInformationFormat": "4", + "DefaultCharIsUnsigned": "true", + "Detect64BitPortabilityProblems": "true", + "DisableLanguageExtensions": "true", + "DisableSpecificWarnings": "abc", + "EnableEnhancedInstructionSet": "1", + "EnableFiberSafeOptimizations": "true", + "EnableFunctionLevelLinking": "true", + "EnableIntrinsicFunctions": "true", + "EnablePREfast": "true", + "ErrorReporting": "2", + "ExceptionHandling": "2", + "ExpandAttributedSource": "true", + "FavorSizeOrSpeed": "2", + "FloatingPointExceptions": "true", + "FloatingPointModel": "1", + "ForceConformanceInForLoopScope": "false", + "ForcedIncludeFiles": "def", + "ForcedUsingFiles": "ge", + "GeneratePreprocessedFile": "2", + "GenerateXMLDocumentationFiles": "true", + "IgnoreStandardIncludePath": "true", + "InlineFunctionExpansion": "1", + "KeepComments": "true", + "MinimalRebuild": "true", + "ObjectFile": "$(IntDir)\\b", + "OmitDefaultLibName": "true", + "OmitFramePointers": "true", + "OpenMP": "true", + "Optimization": "3", + "PrecompiledHeaderFile": "$(IntDir)\\$(TargetName).pche", + "PrecompiledHeaderThrough": "StdAfx.hd", + "PreprocessorDefinitions": "WIN32;_DEBUG;_CONSOLE", + "ProgramDataBaseFileName": "$(IntDir)\\vc90b.pdb", + "RuntimeLibrary": "3", + "RuntimeTypeInfo": "false", + "ShowIncludes": "true", + "SmallerTypeCheck": "true", + "StringPooling": "true", + "StructMemberAlignment": "3", + "SuppressStartupBanner": "false", + "TreatWChar_tAsBuiltInType": "false", + "UndefineAllPreprocessorDefinitions": "true", + "UndefinePreprocessorDefinitions": "wer", + "UseFullPaths": "true", + "UsePrecompiledHeader": "0", + "UseUnicodeResponseFiles": "false", + "WarnAsError": "true", + "WarningLevel": "3", + "WholeProgramOptimization": "true", + "XMLDocumentationFileName": "$(IntDir)\\c", + }, + "VCLinkerTool": { + "AdditionalDependencies": "zx", + "AdditionalLibraryDirectories": "asd", + "AdditionalManifestDependencies": "s2", + "AdditionalOptions": "/mor2", + "AddModuleNamesToAssembly": "d1", + "AllowIsolation": "false", + "AssemblyDebug": "1", + "AssemblyLinkResource": "d5", + "BaseAddress": "23423", + "CLRImageType": "3", + "CLRThreadAttribute": "1", + "CLRUnmanagedCodeCheck": "true", + "DataExecutionPrevention": "0", + "DelayLoadDLLs": "d4", + "DelaySign": "true", + "Driver": "2", + "EmbedManagedResourceFile": "d2", + "EnableCOMDATFolding": "1", + "EnableUAC": "false", + "EntryPointSymbol": "f5", + "ErrorReporting": "2", + "FixedBaseAddress": "1", + "ForceSymbolReferences": "d3", + "FunctionOrder": "fssdfsd", + "GenerateDebugInformation": "true", + "GenerateManifest": "false", + "GenerateMapFile": "true", + "HeapCommitSize": "13", + "HeapReserveSize": "12", + "IgnoreAllDefaultLibraries": "true", + "IgnoreDefaultLibraryNames": "flob;flok", + "IgnoreEmbeddedIDL": "true", + "IgnoreImportLibrary": "true", + "ImportLibrary": "f4", + "KeyContainer": "f7", + "KeyFile": "f6", + "LargeAddressAware": "2", + "LinkIncremental": "0", + "LinkLibraryDependencies": "false", + "LinkTimeCodeGeneration": "1", + "ManifestFile": "$(IntDir)\\$(TargetFileName).2intermediate.manifest", + "MapExports": "true", + "MapFileName": "d5", + "MergedIDLBaseFileName": "f2", + "MergeSections": "f5", + "MidlCommandFile": "f1", + "ModuleDefinitionFile": "sdsd", + "OptimizeForWindows98": "2", + "OptimizeReferences": "2", + "OutputFile": "$(OutDir)\\$(ProjectName)2.exe", + "PerUserRedirection": "true", + "Profile": "true", + "ProfileGuidedDatabase": "$(TargetDir)$(TargetName).pgdd", + "ProgramDatabaseFile": "Flob.pdb", + "RandomizedBaseAddress": "1", + "RegisterOutput": "true", + "ResourceOnlyDLL": "true", + "SetChecksum": "false", + "ShowProgress": "1", + "StackCommitSize": "15", + "StackReserveSize": "14", + "StripPrivateSymbols": "d3", + "SubSystem": "1", + "SupportUnloadOfDelayLoadedDLL": "true", + "SuppressStartupBanner": "false", + "SwapRunFromCD": "true", + "SwapRunFromNet": "true", + "TargetMachine": "1", + "TerminalServerAware": "1", + "TurnOffAssemblyGeneration": "true", + "TypeLibraryFile": "f3", + "TypeLibraryResourceID": "12", + "UACExecutionLevel": "2", + "UACUIAccess": "true", + "UseLibraryDependencyInputs": "true", + "UseUnicodeResponseFiles": "false", + "Version": "333", + }, + "VCResourceCompilerTool": { + "AdditionalIncludeDirectories": "f3", + "AdditionalOptions": "/more3", + "Culture": "3084", + "IgnoreStandardIncludePath": "true", + "PreprocessorDefinitions": "_UNICODE;UNICODE2", + "ResourceOutputFileName": "$(IntDir)/$(InputName)3.res", + "ShowProgress": "true", + }, + "VCManifestTool": { + "AdditionalManifestFiles": "sfsdfsd", + "AdditionalOptions": "afdsdafsd", + "AssemblyIdentity": "sddfdsadfsa", + "ComponentFileName": "fsdfds", + "DependencyInformationFile": "$(IntDir)\\mt.depdfd", + "EmbedManifest": "false", + "GenerateCatalogFiles": "true", + "InputResourceManifests": "asfsfdafs", + "ManifestResourceFile": "$(IntDir)\\$(TargetFileName).embed.manifest.resfdsf", + "OutputManifestFile": "$(TargetPath).manifestdfs", + "RegistrarScriptFile": "sdfsfd", + "ReplacementsFile": "sdffsd", + "SuppressStartupBanner": "false", + "TypeLibraryFile": "sfsd", + "UpdateFileHashes": "true", + "UpdateFileHashesSearchPath": "sfsd", + "UseFAT32Workaround": "true", + "UseUnicodeResponseFiles": "false", + "VerboseOutput": "true", + }, + } + expected_msbuild_settings = { + "ClCompile": { + "AdditionalIncludeDirectories": "dir1", + "AdditionalOptions": "/more /J", + "AdditionalUsingDirectories": "test", + "AssemblerListingLocation": "$(IntDir)a", + "AssemblerOutput": "AssemblyCode", + "BasicRuntimeChecks": "EnableFastChecks", + "BrowseInformation": "true", + "BrowseInformationFile": "$(IntDir)e", + "BufferSecurityCheck": "false", + "CallingConvention": "FastCall", + "CompileAs": "CompileAsC", + "DebugInformationFormat": "EditAndContinue", + "DisableLanguageExtensions": "true", + "DisableSpecificWarnings": "abc", + "EnableEnhancedInstructionSet": "StreamingSIMDExtensions", + "EnableFiberSafeOptimizations": "true", + "EnablePREfast": "true", + "ErrorReporting": "Queue", + "ExceptionHandling": "Async", + "ExpandAttributedSource": "true", + "FavorSizeOrSpeed": "Size", + "FloatingPointExceptions": "true", + "FloatingPointModel": "Strict", + "ForceConformanceInForLoopScope": "false", + "ForcedIncludeFiles": "def", + "ForcedUsingFiles": "ge", + "FunctionLevelLinking": "true", + "GenerateXMLDocumentationFiles": "true", + "IgnoreStandardIncludePath": "true", + "InlineFunctionExpansion": "OnlyExplicitInline", + "IntrinsicFunctions": "true", + "MinimalRebuild": "true", + "ObjectFileName": "$(IntDir)b", + "OmitDefaultLibName": "true", + "OmitFramePointers": "true", + "OpenMPSupport": "true", + "Optimization": "Full", + "PrecompiledHeader": "NotUsing", # Actual conversion gives '' + "PrecompiledHeaderFile": "StdAfx.hd", + "PrecompiledHeaderOutputFile": "$(IntDir)$(TargetName).pche", + "PreprocessKeepComments": "true", + "PreprocessorDefinitions": "WIN32;_DEBUG;_CONSOLE", + "PreprocessSuppressLineNumbers": "true", + "PreprocessToFile": "true", + "ProgramDataBaseFileName": "$(IntDir)vc90b.pdb", + "RuntimeLibrary": "MultiThreadedDebugDLL", + "RuntimeTypeInfo": "false", + "ShowIncludes": "true", + "SmallerTypeCheck": "true", + "StringPooling": "true", + "StructMemberAlignment": "4Bytes", + "SuppressStartupBanner": "false", + "TreatWarningAsError": "true", + "TreatWChar_tAsBuiltInType": "false", + "UndefineAllPreprocessorDefinitions": "true", + "UndefinePreprocessorDefinitions": "wer", + "UseFullPaths": "true", + "WarningLevel": "Level3", + "WholeProgramOptimization": "true", + "XMLDocumentationFileName": "$(IntDir)c", + }, + "Link": { + "AdditionalDependencies": "zx", + "AdditionalLibraryDirectories": "asd", + "AdditionalManifestDependencies": "s2", + "AdditionalOptions": "/mor2", + "AddModuleNamesToAssembly": "d1", + "AllowIsolation": "false", + "AssemblyDebug": "true", + "AssemblyLinkResource": "d5", + "BaseAddress": "23423", + "CLRImageType": "ForceSafeILImage", + "CLRThreadAttribute": "MTAThreadingAttribute", + "CLRUnmanagedCodeCheck": "true", + "DataExecutionPrevention": "", + "DelayLoadDLLs": "d4", + "DelaySign": "true", + "Driver": "UpOnly", + "EmbedManagedResourceFile": "d2", + "EnableCOMDATFolding": "false", + "EnableUAC": "false", + "EntryPointSymbol": "f5", + "FixedBaseAddress": "false", + "ForceSymbolReferences": "d3", + "FunctionOrder": "fssdfsd", + "GenerateDebugInformation": "true", + "GenerateMapFile": "true", + "HeapCommitSize": "13", + "HeapReserveSize": "12", + "IgnoreAllDefaultLibraries": "true", + "IgnoreEmbeddedIDL": "true", + "IgnoreSpecificDefaultLibraries": "flob;flok", + "ImportLibrary": "f4", + "KeyContainer": "f7", + "KeyFile": "f6", + "LargeAddressAware": "true", + "LinkErrorReporting": "QueueForNextLogin", + "LinkTimeCodeGeneration": "UseLinkTimeCodeGeneration", + "ManifestFile": "$(IntDir)$(TargetFileName).2intermediate.manifest", + "MapExports": "true", + "MapFileName": "d5", + "MergedIDLBaseFileName": "f2", + "MergeSections": "f5", + "MidlCommandFile": "f1", + "ModuleDefinitionFile": "sdsd", + "NoEntryPoint": "true", + "OptimizeReferences": "true", + "OutputFile": "$(OutDir)$(ProjectName)2.exe", + "PerUserRedirection": "true", + "Profile": "true", + "ProfileGuidedDatabase": "$(TargetDir)$(TargetName).pgdd", + "ProgramDatabaseFile": "Flob.pdb", + "RandomizedBaseAddress": "false", + "RegisterOutput": "true", + "SetChecksum": "false", + "ShowProgress": "LinkVerbose", + "StackCommitSize": "15", + "StackReserveSize": "14", + "StripPrivateSymbols": "d3", + "SubSystem": "Console", + "SupportUnloadOfDelayLoadedDLL": "true", + "SuppressStartupBanner": "false", + "SwapRunFromCD": "true", + "SwapRunFromNET": "true", + "TargetMachine": "MachineX86", + "TerminalServerAware": "false", + "TurnOffAssemblyGeneration": "true", + "TypeLibraryFile": "f3", + "TypeLibraryResourceID": "12", + "UACExecutionLevel": "RequireAdministrator", + "UACUIAccess": "true", + "Version": "333", + }, + "ResourceCompile": { + "AdditionalIncludeDirectories": "f3", + "AdditionalOptions": "/more3", + "Culture": "0x0c0c", + "IgnoreStandardIncludePath": "true", + "PreprocessorDefinitions": "_UNICODE;UNICODE2", + "ResourceOutputFileName": "$(IntDir)%(Filename)3.res", + "ShowProgress": "true", + }, + "Manifest": { + "AdditionalManifestFiles": "sfsdfsd", + "AdditionalOptions": "afdsdafsd", + "AssemblyIdentity": "sddfdsadfsa", + "ComponentFileName": "fsdfds", + "GenerateCatalogFiles": "true", + "InputResourceManifests": "asfsfdafs", + "OutputManifestFile": "$(TargetPath).manifestdfs", + "RegistrarScriptFile": "sdfsfd", + "ReplacementsFile": "sdffsd", + "SuppressStartupBanner": "false", + "TypeLibraryFile": "sfsd", + "UpdateFileHashes": "true", + "UpdateFileHashesSearchPath": "sfsd", + "VerboseOutput": "true", + }, + "ProjectReference": { + "LinkLibraryDependencies": "false", + "UseLibraryDependencyInputs": "true", + }, + "": { + "EmbedManifest": "false", + "GenerateManifest": "false", + "IgnoreImportLibrary": "true", + "LinkIncremental": "", + }, + "ManifestResourceCompile": { + "ResourceOutputFileName": "$(IntDir)$(TargetFileName).embed.manifest.resfdsf" }, - 'ManifestResourceCompile': { - 'ResourceOutputFileName': - '$(IntDir)$(TargetFileName).embed.manifest.resfdsf'} } - self.maxDiff = 9999 # on failure display a long diff - actual_msbuild_settings = MSVSSettings.ConvertToMSBuildSettings( - msvs_settings, - self.stderr) - self.assertEqual(expected_msbuild_settings, actual_msbuild_settings) - self._ExpectedWarnings([]) + self.maxDiff = 9999 # on failure display a long diff + actual_msbuild_settings = MSVSSettings.ConvertToMSBuildSettings( + msvs_settings, self.stderr + ) + self.assertEqual(expected_msbuild_settings, actual_msbuild_settings) + self._ExpectedWarnings([]) -if __name__ == '__main__': - unittest.main() +if __name__ == "__main__": + unittest.main() diff --git a/gyp/pylib/gyp/MSVSToolFile.py b/gyp/pylib/gyp/MSVSToolFile.py index 74e529a17f..2c08589e06 100644 --- a/gyp/pylib/gyp/MSVSToolFile.py +++ b/gyp/pylib/gyp/MSVSToolFile.py @@ -4,28 +4,27 @@ """Visual Studio project reader/writer.""" -import gyp.common import gyp.easy_xml as easy_xml class Writer(object): - """Visual Studio XML tool file writer.""" + """Visual Studio XML tool file writer.""" - def __init__(self, tool_file_path, name): - """Initializes the tool file. + def __init__(self, tool_file_path, name): + """Initializes the tool file. Args: tool_file_path: Path to the tool file. name: Name of the tool file. """ - self.tool_file_path = tool_file_path - self.name = name - self.rules_section = ['Rules'] + self.tool_file_path = tool_file_path + self.name = name + self.rules_section = ["Rules"] - def AddCustomBuildRule(self, name, cmd, description, - additional_dependencies, - outputs, extensions): - """Adds a rule to the tool file. + def AddCustomBuildRule( + self, name, cmd, description, additional_dependencies, outputs, extensions + ): + """Adds a rule to the tool file. Args: name: Name of the rule. @@ -35,24 +34,26 @@ def AddCustomBuildRule(self, name, cmd, description, outputs: outputs of the rule. extensions: extensions handled by the rule. """ - rule = ['CustomBuildRule', - {'Name': name, - 'ExecutionDescription': description, - 'CommandLine': cmd, - 'Outputs': ';'.join(outputs), - 'FileExtensions': ';'.join(extensions), - 'AdditionalDependencies': - ';'.join(additional_dependencies) - }] - self.rules_section.append(rule) - - def WriteIfChanged(self): - """Writes the tool file.""" - content = ['VisualStudioToolFile', - {'Version': '8.00', - 'Name': self.name - }, - self.rules_section - ] - easy_xml.WriteXmlIfChanged(content, self.tool_file_path, - encoding="Windows-1252") + rule = [ + "CustomBuildRule", + { + "Name": name, + "ExecutionDescription": description, + "CommandLine": cmd, + "Outputs": ";".join(outputs), + "FileExtensions": ";".join(extensions), + "AdditionalDependencies": ";".join(additional_dependencies), + }, + ] + self.rules_section.append(rule) + + def WriteIfChanged(self): + """Writes the tool file.""" + content = [ + "VisualStudioToolFile", + {"Version": "8.00", "Name": self.name}, + self.rules_section, + ] + easy_xml.WriteXmlIfChanged( + content, self.tool_file_path, encoding="Windows-1252" + ) diff --git a/gyp/pylib/gyp/MSVSUserFile.py b/gyp/pylib/gyp/MSVSUserFile.py index 2264d64015..de0896e693 100644 --- a/gyp/pylib/gyp/MSVSUserFile.py +++ b/gyp/pylib/gyp/MSVSUserFile.py @@ -6,78 +6,81 @@ import os import re -import socket # for gethostname +import socket # for gethostname -import gyp.common import gyp.easy_xml as easy_xml -#------------------------------------------------------------------------------ +# ------------------------------------------------------------------------------ + def _FindCommandInPath(command): - """If there are no slashes in the command given, this function + """If there are no slashes in the command given, this function searches the PATH env to find the given command, and converts it to an absolute path. We have to do this because MSVS is looking for an actual file to launch a debugger on, not just a command line. Note that this happens at GYP time, so anything needing to be built needs to have a full path.""" - if '/' in command or '\\' in command: - # If the command already has path elements (either relative or - # absolute), then assume it is constructed properly. + if "/" in command or "\\" in command: + # If the command already has path elements (either relative or + # absolute), then assume it is constructed properly. + return command + else: + # Search through the path list and find an existing file that + # we can access. + paths = os.environ.get("PATH", "").split(os.pathsep) + for path in paths: + item = os.path.join(path, command) + if os.path.isfile(item) and os.access(item, os.X_OK): + return item return command - else: - # Search through the path list and find an existing file that - # we can access. - paths = os.environ.get('PATH','').split(os.pathsep) - for path in paths: - item = os.path.join(path, command) - if os.path.isfile(item) and os.access(item, os.X_OK): - return item - return command + def _QuoteWin32CommandLineArgs(args): - new_args = [] - for arg in args: - # Replace all double-quotes with double-double-quotes to escape - # them for cmd shell, and then quote the whole thing if there - # are any. - if arg.find('"') != -1: - arg = '""'.join(arg.split('"')) - arg = '"%s"' % arg - - # Otherwise, if there are any spaces, quote the whole arg. - elif re.search(r'[ \t\n]', arg): - arg = '"%s"' % arg - new_args.append(arg) - return new_args + new_args = [] + for arg in args: + # Replace all double-quotes with double-double-quotes to escape + # them for cmd shell, and then quote the whole thing if there + # are any. + if arg.find('"') != -1: + arg = '""'.join(arg.split('"')) + arg = '"%s"' % arg + + # Otherwise, if there are any spaces, quote the whole arg. + elif re.search(r"[ \t\n]", arg): + arg = '"%s"' % arg + new_args.append(arg) + return new_args + class Writer(object): - """Visual Studio XML user user file writer.""" + """Visual Studio XML user user file writer.""" - def __init__(self, user_file_path, version, name): - """Initializes the user file. + def __init__(self, user_file_path, version, name): + """Initializes the user file. Args: user_file_path: Path to the user file. version: Version info. name: Name of the user file. """ - self.user_file_path = user_file_path - self.version = version - self.name = name - self.configurations = {} + self.user_file_path = user_file_path + self.version = version + self.name = name + self.configurations = {} - def AddConfig(self, name): - """Adds a configuration to the project. + def AddConfig(self, name): + """Adds a configuration to the project. Args: name: Configuration name. """ - self.configurations[name] = ['Configuration', {'Name': name}] + self.configurations[name] = ["Configuration", {"Name": name}] - def AddDebugSettings(self, config_name, command, environment = {}, - working_directory=""): - """Adds a DebugSettings node to the user file for a particular config. + def AddDebugSettings( + self, config_name, command, environment={}, working_directory="" + ): + """Adds a DebugSettings node to the user file for a particular config. Args: command: command line to run. First element in the list is the @@ -85,63 +88,66 @@ def AddDebugSettings(self, config_name, command, environment = {}, necessary. working_directory: other files which may trigger the rule. (optional) """ - command = _QuoteWin32CommandLineArgs(command) - - abs_command = _FindCommandInPath(command[0]) - - if environment and isinstance(environment, dict): - env_list = ['%s="%s"' % (key, val) - for (key,val) in environment.items()] - environment = ' '.join(env_list) - else: - environment = '' - - n_cmd = ['DebugSettings', - {'Command': abs_command, - 'WorkingDirectory': working_directory, - 'CommandArguments': " ".join(command[1:]), - 'RemoteMachine': socket.gethostname(), - 'Environment': environment, - 'EnvironmentMerge': 'true', - # Currently these are all "dummy" values that we're just setting - # in the default manner that MSVS does it. We could use some of - # these to add additional capabilities, I suppose, but they might - # not have parity with other platforms then. - 'Attach': 'false', - 'DebuggerType': '3', # 'auto' debugger - 'Remote': '1', - 'RemoteCommand': '', - 'HttpUrl': '', - 'PDBPath': '', - 'SQLDebugging': '', - 'DebuggerFlavor': '0', - 'MPIRunCommand': '', - 'MPIRunArguments': '', - 'MPIRunWorkingDirectory': '', - 'ApplicationCommand': '', - 'ApplicationArguments': '', - 'ShimCommand': '', - 'MPIAcceptMode': '', - 'MPIAcceptFilter': '' - }] - - # Find the config, and add it if it doesn't exist. - if config_name not in self.configurations: - self.AddConfig(config_name) - - # Add the DebugSettings onto the appropriate config. - self.configurations[config_name].append(n_cmd) - - def WriteIfChanged(self): - """Writes the user file.""" - configs = ['Configurations'] - for config, spec in sorted(self.configurations.items()): - configs.append(spec) - - content = ['VisualStudioUserFile', - {'Version': self.version.ProjectVersion(), - 'Name': self.name - }, - configs] - easy_xml.WriteXmlIfChanged(content, self.user_file_path, - encoding="Windows-1252") + command = _QuoteWin32CommandLineArgs(command) + + abs_command = _FindCommandInPath(command[0]) + + if environment and isinstance(environment, dict): + env_list = ['%s="%s"' % (key, val) for (key, val) in environment.items()] + environment = " ".join(env_list) + else: + environment = "" + + n_cmd = [ + "DebugSettings", + { + "Command": abs_command, + "WorkingDirectory": working_directory, + "CommandArguments": " ".join(command[1:]), + "RemoteMachine": socket.gethostname(), + "Environment": environment, + "EnvironmentMerge": "true", + # Currently these are all "dummy" values that we're just setting + # in the default manner that MSVS does it. We could use some of + # these to add additional capabilities, I suppose, but they might + # not have parity with other platforms then. + "Attach": "false", + "DebuggerType": "3", # 'auto' debugger + "Remote": "1", + "RemoteCommand": "", + "HttpUrl": "", + "PDBPath": "", + "SQLDebugging": "", + "DebuggerFlavor": "0", + "MPIRunCommand": "", + "MPIRunArguments": "", + "MPIRunWorkingDirectory": "", + "ApplicationCommand": "", + "ApplicationArguments": "", + "ShimCommand": "", + "MPIAcceptMode": "", + "MPIAcceptFilter": "", + }, + ] + + # Find the config, and add it if it doesn't exist. + if config_name not in self.configurations: + self.AddConfig(config_name) + + # Add the DebugSettings onto the appropriate config. + self.configurations[config_name].append(n_cmd) + + def WriteIfChanged(self): + """Writes the user file.""" + configs = ["Configurations"] + for config, spec in sorted(self.configurations.items()): + configs.append(spec) + + content = [ + "VisualStudioUserFile", + {"Version": self.version.ProjectVersion(), "Name": self.name}, + configs, + ] + easy_xml.WriteXmlIfChanged( + content, self.user_file_path, encoding="Windows-1252" + ) diff --git a/gyp/pylib/gyp/MSVSUtil.py b/gyp/pylib/gyp/MSVSUtil.py index f24530b275..83a9c297ed 100644 --- a/gyp/pylib/gyp/MSVSUtil.py +++ b/gyp/pylib/gyp/MSVSUtil.py @@ -10,25 +10,25 @@ # A dictionary mapping supported target types to extensions. TARGET_TYPE_EXT = { - 'executable': 'exe', - 'loadable_module': 'dll', - 'shared_library': 'dll', - 'static_library': 'lib', - 'windows_driver': 'sys', + "executable": "exe", + "loadable_module": "dll", + "shared_library": "dll", + "static_library": "lib", + "windows_driver": "sys", } def _GetLargePdbShimCcPath(): - """Returns the path of the large_pdb_shim.cc file.""" - this_dir = os.path.abspath(os.path.dirname(__file__)) - src_dir = os.path.abspath(os.path.join(this_dir, '..', '..')) - win_data_dir = os.path.join(src_dir, 'data', 'win') - large_pdb_shim_cc = os.path.join(win_data_dir, 'large-pdb-shim.cc') - return large_pdb_shim_cc + """Returns the path of the large_pdb_shim.cc file.""" + this_dir = os.path.abspath(os.path.dirname(__file__)) + src_dir = os.path.abspath(os.path.join(this_dir, "..", "..")) + win_data_dir = os.path.join(src_dir, "data", "win") + large_pdb_shim_cc = os.path.join(win_data_dir, "large-pdb-shim.cc") + return large_pdb_shim_cc def _DeepCopySomeKeys(in_dict, keys): - """Performs a partial deep-copy on |in_dict|, only copying the keys in |keys|. + """Performs a partial deep-copy on |in_dict|, only copying the keys in |keys|. Arguments: in_dict: The dictionary to copy. @@ -37,16 +37,16 @@ def _DeepCopySomeKeys(in_dict, keys): Returns: The partially deep-copied dictionary. """ - d = {} - for key in keys: - if key not in in_dict: - continue - d[key] = copy.deepcopy(in_dict[key]) - return d + d = {} + for key in keys: + if key not in in_dict: + continue + d[key] = copy.deepcopy(in_dict[key]) + return d def _SuffixName(name, suffix): - """Add a suffix to the end of a target. + """Add a suffix to the end of a target. Arguments: name: name of the target (foo#target) @@ -54,13 +54,13 @@ def _SuffixName(name, suffix): Returns: Target name with suffix added (foo_suffix#target) """ - parts = name.rsplit('#', 1) - parts[0] = '%s_%s' % (parts[0], suffix) - return '#'.join(parts) + parts = name.rsplit("#", 1) + parts[0] = "%s_%s" % (parts[0], suffix) + return "#".join(parts) def _ShardName(name, number): - """Add a shard number to the end of a target. + """Add a shard number to the end of a target. Arguments: name: name of the target (foo#target) @@ -68,11 +68,11 @@ def _ShardName(name, number): Returns: Target name with shard added (foo_1#target) """ - return _SuffixName(name, str(number)) + return _SuffixName(name, str(number)) def ShardTargets(target_list, target_dicts): - """Shard some targets apart to work around the linkers limits. + """Shard some targets apart to work around the linkers limits. Arguments: target_list: List of target pairs: 'base/base.gyp:base'. @@ -80,54 +80,55 @@ def ShardTargets(target_list, target_dicts): Returns: Tuple of the new sharded versions of the inputs. """ - # Gather the targets to shard, and how many pieces. - targets_to_shard = {} - for t in target_dicts: - shards = int(target_dicts[t].get('msvs_shard', 0)) - if shards: - targets_to_shard[t] = shards - # Shard target_list. - new_target_list = [] - for t in target_list: - if t in targets_to_shard: - for i in range(targets_to_shard[t]): - new_target_list.append(_ShardName(t, i)) - else: - new_target_list.append(t) - # Shard target_dict. - new_target_dicts = {} - for t in target_dicts: - if t in targets_to_shard: - for i in range(targets_to_shard[t]): - name = _ShardName(t, i) - new_target_dicts[name] = copy.copy(target_dicts[t]) - new_target_dicts[name]['target_name'] = _ShardName( - new_target_dicts[name]['target_name'], i) - sources = new_target_dicts[name].get('sources', []) - new_sources = [] - for pos in range(i, len(sources), targets_to_shard[t]): - new_sources.append(sources[pos]) - new_target_dicts[name]['sources'] = new_sources - else: - new_target_dicts[t] = target_dicts[t] - # Shard dependencies. - for t in sorted(new_target_dicts): - for deptype in ('dependencies', 'dependencies_original'): - dependencies = copy.copy(new_target_dicts[t].get(deptype, [])) - new_dependencies = [] - for d in dependencies: - if d in targets_to_shard: - for i in range(targets_to_shard[d]): - new_dependencies.append(_ShardName(d, i)) + # Gather the targets to shard, and how many pieces. + targets_to_shard = {} + for t in target_dicts: + shards = int(target_dicts[t].get("msvs_shard", 0)) + if shards: + targets_to_shard[t] = shards + # Shard target_list. + new_target_list = [] + for t in target_list: + if t in targets_to_shard: + for i in range(targets_to_shard[t]): + new_target_list.append(_ShardName(t, i)) else: - new_dependencies.append(d) - new_target_dicts[t][deptype] = new_dependencies - - return (new_target_list, new_target_dicts) + new_target_list.append(t) + # Shard target_dict. + new_target_dicts = {} + for t in target_dicts: + if t in targets_to_shard: + for i in range(targets_to_shard[t]): + name = _ShardName(t, i) + new_target_dicts[name] = copy.copy(target_dicts[t]) + new_target_dicts[name]["target_name"] = _ShardName( + new_target_dicts[name]["target_name"], i + ) + sources = new_target_dicts[name].get("sources", []) + new_sources = [] + for pos in range(i, len(sources), targets_to_shard[t]): + new_sources.append(sources[pos]) + new_target_dicts[name]["sources"] = new_sources + else: + new_target_dicts[t] = target_dicts[t] + # Shard dependencies. + for t in sorted(new_target_dicts): + for deptype in ("dependencies", "dependencies_original"): + dependencies = copy.copy(new_target_dicts[t].get(deptype, [])) + new_dependencies = [] + for d in dependencies: + if d in targets_to_shard: + for i in range(targets_to_shard[d]): + new_dependencies.append(_ShardName(d, i)) + else: + new_dependencies.append(d) + new_target_dicts[t][deptype] = new_dependencies + + return (new_target_list, new_target_dicts) def _GetPdbPath(target_dict, config_name, vars): - """Returns the path to the PDB file that will be generated by a given + """Returns the path to the PDB file that will be generated by a given configuration. The lookup proceeds as follows: @@ -144,30 +145,29 @@ def _GetPdbPath(target_dict, config_name, vars): Returns: The path of the corresponding PDB file. """ - config = target_dict['configurations'][config_name] - msvs = config.setdefault('msvs_settings', {}) - - linker = msvs.get('VCLinkerTool', {}) + config = target_dict["configurations"][config_name] + msvs = config.setdefault("msvs_settings", {}) - pdb_path = linker.get('ProgramDatabaseFile') - if pdb_path: - return pdb_path + linker = msvs.get("VCLinkerTool", {}) - variables = target_dict.get('variables', {}) - pdb_path = variables.get('msvs_large_pdb_path', None) - if pdb_path: - return pdb_path + pdb_path = linker.get("ProgramDatabaseFile") + if pdb_path: + return pdb_path + variables = target_dict.get("variables", {}) + pdb_path = variables.get("msvs_large_pdb_path", None) + if pdb_path: + return pdb_path - pdb_base = target_dict.get('product_name', target_dict['target_name']) - pdb_base = '%s.%s.pdb' % (pdb_base, TARGET_TYPE_EXT[target_dict['type']]) - pdb_path = vars['PRODUCT_DIR'] + '/' + pdb_base + pdb_base = target_dict.get("product_name", target_dict["target_name"]) + pdb_base = "%s.%s.pdb" % (pdb_base, TARGET_TYPE_EXT[target_dict["type"]]) + pdb_path = vars["PRODUCT_DIR"] + "/" + pdb_base - return pdb_path + return pdb_path def InsertLargePdbShims(target_list, target_dicts, vars): - """Insert a shim target that forces the linker to use 4KB pagesize PDBs. + """Insert a shim target that forces the linker to use 4KB pagesize PDBs. This is a workaround for targets with PDBs greater than 1GB in size, the limit for the 1KB pagesize PDBs created by the linker by default. @@ -179,93 +179,93 @@ def InsertLargePdbShims(target_list, target_dicts, vars): Returns: Tuple of the shimmed version of the inputs. """ - # Determine which targets need shimming. - targets_to_shim = [] - for t in target_dicts: - target_dict = target_dicts[t] - - # We only want to shim targets that have msvs_large_pdb enabled. - if not int(target_dict.get('msvs_large_pdb', 0)): - continue - # This is intended for executable, shared_library and loadable_module - # targets where every configuration is set up to produce a PDB output. - # If any of these conditions is not true then the shim logic will fail - # below. - targets_to_shim.append(t) - - large_pdb_shim_cc = _GetLargePdbShimCcPath() - - for t in targets_to_shim: - target_dict = target_dicts[t] - target_name = target_dict.get('target_name') - - base_dict = _DeepCopySomeKeys(target_dict, - ['configurations', 'default_configuration', 'toolset']) - - # This is the dict for copying the source file (part of the GYP tree) - # to the intermediate directory of the project. This is necessary because - # we can't always build a relative path to the shim source file (on Windows - # GYP and the project may be on different drives), and Ninja hates absolute - # paths (it ends up generating the .obj and .obj.d alongside the source - # file, polluting GYPs tree). - copy_suffix = 'large_pdb_copy' - copy_target_name = target_name + '_' + copy_suffix - full_copy_target_name = _SuffixName(t, copy_suffix) - shim_cc_basename = os.path.basename(large_pdb_shim_cc) - shim_cc_dir = vars['SHARED_INTERMEDIATE_DIR'] + '/' + copy_target_name - shim_cc_path = shim_cc_dir + '/' + shim_cc_basename - copy_dict = copy.deepcopy(base_dict) - copy_dict['target_name'] = copy_target_name - copy_dict['type'] = 'none' - copy_dict['sources'] = [ large_pdb_shim_cc ] - copy_dict['copies'] = [{ - 'destination': shim_cc_dir, - 'files': [ large_pdb_shim_cc ] - }] - - # This is the dict for the PDB generating shim target. It depends on the - # copy target. - shim_suffix = 'large_pdb_shim' - shim_target_name = target_name + '_' + shim_suffix - full_shim_target_name = _SuffixName(t, shim_suffix) - shim_dict = copy.deepcopy(base_dict) - shim_dict['target_name'] = shim_target_name - shim_dict['type'] = 'static_library' - shim_dict['sources'] = [ shim_cc_path ] - shim_dict['dependencies'] = [ full_copy_target_name ] - - # Set up the shim to output its PDB to the same location as the final linker - # target. - for config_name, config in shim_dict.get('configurations').items(): - pdb_path = _GetPdbPath(target_dict, config_name, vars) - - # A few keys that we don't want to propagate. - for key in ['msvs_precompiled_header', 'msvs_precompiled_source', 'test']: - config.pop(key, None) - - msvs = config.setdefault('msvs_settings', {}) - - # Update the compiler directives in the shim target. - compiler = msvs.setdefault('VCCLCompilerTool', {}) - compiler['DebugInformationFormat'] = '3' - compiler['ProgramDataBaseFileName'] = pdb_path - - # Set the explicit PDB path in the appropriate configuration of the - # original target. - config = target_dict['configurations'][config_name] - msvs = config.setdefault('msvs_settings', {}) - linker = msvs.setdefault('VCLinkerTool', {}) - linker['GenerateDebugInformation'] = 'true' - linker['ProgramDatabaseFile'] = pdb_path - - # Add the new targets. They must go to the beginning of the list so that - # the dependency generation works as expected in ninja. - target_list.insert(0, full_copy_target_name) - target_list.insert(0, full_shim_target_name) - target_dicts[full_copy_target_name] = copy_dict - target_dicts[full_shim_target_name] = shim_dict - - # Update the original target to depend on the shim target. - target_dict.setdefault('dependencies', []).append(full_shim_target_name) - - return (target_list, target_dicts) + # Determine which targets need shimming. + targets_to_shim = [] + for t in target_dicts: + target_dict = target_dicts[t] + + # We only want to shim targets that have msvs_large_pdb enabled. + if not int(target_dict.get("msvs_large_pdb", 0)): + continue + # This is intended for executable, shared_library and loadable_module + # targets where every configuration is set up to produce a PDB output. + # If any of these conditions is not true then the shim logic will fail + # below. + targets_to_shim.append(t) + + large_pdb_shim_cc = _GetLargePdbShimCcPath() + + for t in targets_to_shim: + target_dict = target_dicts[t] + target_name = target_dict.get("target_name") + + base_dict = _DeepCopySomeKeys( + target_dict, ["configurations", "default_configuration", "toolset"] + ) + + # This is the dict for copying the source file (part of the GYP tree) + # to the intermediate directory of the project. This is necessary because + # we can't always build a relative path to the shim source file (on Windows + # GYP and the project may be on different drives), and Ninja hates absolute + # paths (it ends up generating the .obj and .obj.d alongside the source + # file, polluting GYPs tree). + copy_suffix = "large_pdb_copy" + copy_target_name = target_name + "_" + copy_suffix + full_copy_target_name = _SuffixName(t, copy_suffix) + shim_cc_basename = os.path.basename(large_pdb_shim_cc) + shim_cc_dir = vars["SHARED_INTERMEDIATE_DIR"] + "/" + copy_target_name + shim_cc_path = shim_cc_dir + "/" + shim_cc_basename + copy_dict = copy.deepcopy(base_dict) + copy_dict["target_name"] = copy_target_name + copy_dict["type"] = "none" + copy_dict["sources"] = [large_pdb_shim_cc] + copy_dict["copies"] = [ + {"destination": shim_cc_dir, "files": [large_pdb_shim_cc]} + ] + + # This is the dict for the PDB generating shim target. It depends on the + # copy target. + shim_suffix = "large_pdb_shim" + shim_target_name = target_name + "_" + shim_suffix + full_shim_target_name = _SuffixName(t, shim_suffix) + shim_dict = copy.deepcopy(base_dict) + shim_dict["target_name"] = shim_target_name + shim_dict["type"] = "static_library" + shim_dict["sources"] = [shim_cc_path] + shim_dict["dependencies"] = [full_copy_target_name] + + # Set up the shim to output its PDB to the same location as the final linker + # target. + for config_name, config in shim_dict.get("configurations").items(): + pdb_path = _GetPdbPath(target_dict, config_name, vars) + + # A few keys that we don't want to propagate. + for key in ["msvs_precompiled_header", "msvs_precompiled_source", "test"]: + config.pop(key, None) + + msvs = config.setdefault("msvs_settings", {}) + + # Update the compiler directives in the shim target. + compiler = msvs.setdefault("VCCLCompilerTool", {}) + compiler["DebugInformationFormat"] = "3" + compiler["ProgramDataBaseFileName"] = pdb_path + + # Set the explicit PDB path in the appropriate configuration of the + # original target. + config = target_dict["configurations"][config_name] + msvs = config.setdefault("msvs_settings", {}) + linker = msvs.setdefault("VCLinkerTool", {}) + linker["GenerateDebugInformation"] = "true" + linker["ProgramDatabaseFile"] = pdb_path + + # Add the new targets. They must go to the beginning of the list so that + # the dependency generation works as expected in ninja. + target_list.insert(0, full_copy_target_name) + target_list.insert(0, full_shim_target_name) + target_dicts[full_copy_target_name] = copy_dict + target_dicts[full_shim_target_name] = shim_dict + + # Update the original target to depend on the shim target. + target_dict.setdefault("dependencies", []).append(full_shim_target_name) + + return (target_list, target_dicts) diff --git a/gyp/pylib/gyp/MSVSVersion.py b/gyp/pylib/gyp/MSVSVersion.py index ce9b349834..36b006aaa9 100644 --- a/gyp/pylib/gyp/MSVSVersion.py +++ b/gyp/pylib/gyp/MSVSVersion.py @@ -9,139 +9,152 @@ import re import subprocess import sys -import gyp import glob PY3 = bytes != str def JoinPath(*args): - return os.path.normpath(os.path.join(*args)) + return os.path.normpath(os.path.join(*args)) class VisualStudioVersion(object): - """Information regarding a version of Visual Studio.""" - - def __init__(self, short_name, description, - solution_version, project_version, flat_sln, uses_vcxproj, - path, sdk_based, default_toolset=None, compatible_sdks=None): - self.short_name = short_name - self.description = description - self.solution_version = solution_version - self.project_version = project_version - self.flat_sln = flat_sln - self.uses_vcxproj = uses_vcxproj - self.path = path - self.sdk_based = sdk_based - self.default_toolset = default_toolset - compatible_sdks = compatible_sdks or [] - compatible_sdks.sort(key=lambda v: float(v.replace('v', '')), reverse=True) - self.compatible_sdks = compatible_sdks - - def ShortName(self): - return self.short_name - - def Description(self): - """Get the full description of the version.""" - return self.description - - def SolutionVersion(self): - """Get the version number of the sln files.""" - return self.solution_version - - def ProjectVersion(self): - """Get the version number of the vcproj or vcxproj files.""" - return self.project_version - - def FlatSolution(self): - return self.flat_sln - - def UsesVcxproj(self): - """Returns true if this version uses a vcxproj file.""" - return self.uses_vcxproj - - def ProjectExtension(self): - """Returns the file extension for the project.""" - return self.uses_vcxproj and '.vcxproj' or '.vcproj' - - def Path(self): - """Returns the path to Visual Studio installation.""" - return self.path - - def ToolPath(self, tool): - """Returns the path to a given compiler tool. """ - return os.path.normpath(os.path.join(self.path, "VC/bin", tool)) - - def DefaultToolset(self): - """Returns the msbuild toolset version that will be used in the absence + """Information regarding a version of Visual Studio.""" + + def __init__( + self, + short_name, + description, + solution_version, + project_version, + flat_sln, + uses_vcxproj, + path, + sdk_based, + default_toolset=None, + compatible_sdks=None, + ): + self.short_name = short_name + self.description = description + self.solution_version = solution_version + self.project_version = project_version + self.flat_sln = flat_sln + self.uses_vcxproj = uses_vcxproj + self.path = path + self.sdk_based = sdk_based + self.default_toolset = default_toolset + compatible_sdks = compatible_sdks or [] + compatible_sdks.sort(key=lambda v: float(v.replace("v", "")), reverse=True) + self.compatible_sdks = compatible_sdks + + def ShortName(self): + return self.short_name + + def Description(self): + """Get the full description of the version.""" + return self.description + + def SolutionVersion(self): + """Get the version number of the sln files.""" + return self.solution_version + + def ProjectVersion(self): + """Get the version number of the vcproj or vcxproj files.""" + return self.project_version + + def FlatSolution(self): + return self.flat_sln + + def UsesVcxproj(self): + """Returns true if this version uses a vcxproj file.""" + return self.uses_vcxproj + + def ProjectExtension(self): + """Returns the file extension for the project.""" + return self.uses_vcxproj and ".vcxproj" or ".vcproj" + + def Path(self): + """Returns the path to Visual Studio installation.""" + return self.path + + def ToolPath(self, tool): + """Returns the path to a given compiler tool. """ + return os.path.normpath(os.path.join(self.path, "VC/bin", tool)) + + def DefaultToolset(self): + """Returns the msbuild toolset version that will be used in the absence of a user override.""" - return self.default_toolset + return self.default_toolset - - def _SetupScriptInternal(self, target_arch): - """Returns a command (with arguments) to be used to set up the + def _SetupScriptInternal(self, target_arch): + """Returns a command (with arguments) to be used to set up the environment.""" - assert target_arch in ('x86', 'x64'), "target_arch not supported" - # If WindowsSDKDir is set and SetEnv.Cmd exists then we are using the - # depot_tools build tools and should run SetEnv.Cmd to set up the - # environment. The check for WindowsSDKDir alone is not sufficient because - # this is set by running vcvarsall.bat. - sdk_dir = os.environ.get('WindowsSDKDir', '') - setup_path = JoinPath(sdk_dir, 'Bin', 'SetEnv.Cmd') - if self.sdk_based and sdk_dir and os.path.exists(setup_path): - return [setup_path, '/' + target_arch] - - is_host_arch_x64 = ( - os.environ.get('PROCESSOR_ARCHITECTURE') == 'AMD64' or - os.environ.get('PROCESSOR_ARCHITEW6432') == 'AMD64' - ) - - # For VS2017 (and newer) it's fairly easy - if self.short_name >= '2017': - script_path = JoinPath(self.path, - 'VC', 'Auxiliary', 'Build', 'vcvarsall.bat') - - # Always use a native executable, cross-compiling if necessary. - host_arch = 'amd64' if is_host_arch_x64 else 'x86' - msvc_target_arch = 'amd64' if target_arch == 'x64' else 'x86' - arg = host_arch - if host_arch != msvc_target_arch: - arg += '_' + msvc_target_arch - - return [script_path, arg] - - # We try to find the best version of the env setup batch. - vcvarsall = JoinPath(self.path, 'VC', 'vcvarsall.bat') - if target_arch == 'x86': - if self.short_name >= '2013' and self.short_name[-1] != 'e' and \ - is_host_arch_x64: - # VS2013 and later, non-Express have a x64-x86 cross that we want - # to prefer. - return [vcvarsall, 'amd64_x86'] - else: - # Otherwise, the standard x86 compiler. We don't use VC/vcvarsall.bat - # for x86 because vcvarsall calls vcvars32, which it can only find if - # VS??COMNTOOLS is set, which isn't guaranteed. - return [JoinPath(self.path, 'Common7', 'Tools', 'vsvars32.bat')] - elif target_arch == 'x64': - arg = 'x86_amd64' - # Use the 64-on-64 compiler if we're not using an express edition and - # we're running on a 64bit OS. - if self.short_name[-1] != 'e' and is_host_arch_x64: - arg = 'amd64' - return [vcvarsall, arg] - - def SetupScript(self, target_arch): - script_data = self._SetupScriptInternal(target_arch) - script_path = script_data[0] - if not os.path.exists(script_path): - raise Exception('%s is missing - make sure VC++ tools are installed.' % - script_path) - return script_data + assert target_arch in ("x86", "x64"), "target_arch not supported" + # If WindowsSDKDir is set and SetEnv.Cmd exists then we are using the + # depot_tools build tools and should run SetEnv.Cmd to set up the + # environment. The check for WindowsSDKDir alone is not sufficient because + # this is set by running vcvarsall.bat. + sdk_dir = os.environ.get("WindowsSDKDir", "") + setup_path = JoinPath(sdk_dir, "Bin", "SetEnv.Cmd") + if self.sdk_based and sdk_dir and os.path.exists(setup_path): + return [setup_path, "/" + target_arch] + + is_host_arch_x64 = ( + os.environ.get("PROCESSOR_ARCHITECTURE") == "AMD64" + or os.environ.get("PROCESSOR_ARCHITEW6432") == "AMD64" + ) + + # For VS2017 (and newer) it's fairly easy + if self.short_name >= "2017": + script_path = JoinPath( + self.path, "VC", "Auxiliary", "Build", "vcvarsall.bat" + ) + + # Always use a native executable, cross-compiling if necessary. + host_arch = "amd64" if is_host_arch_x64 else "x86" + msvc_target_arch = "amd64" if target_arch == "x64" else "x86" + arg = host_arch + if host_arch != msvc_target_arch: + arg += "_" + msvc_target_arch + + return [script_path, arg] + + # We try to find the best version of the env setup batch. + vcvarsall = JoinPath(self.path, "VC", "vcvarsall.bat") + if target_arch == "x86": + if ( + self.short_name >= "2013" + and self.short_name[-1] != "e" + and is_host_arch_x64 + ): + # VS2013 and later, non-Express have a x64-x86 cross that we want + # to prefer. + return [vcvarsall, "amd64_x86"] + else: + # Otherwise, the standard x86 compiler. We don't use VC/vcvarsall.bat + # for x86 because vcvarsall calls vcvars32, which it can only find if + # VS??COMNTOOLS is set, which isn't guaranteed. + return [JoinPath(self.path, "Common7", "Tools", "vsvars32.bat")] + elif target_arch == "x64": + arg = "x86_amd64" + # Use the 64-on-64 compiler if we're not using an express edition and + # we're running on a 64bit OS. + if self.short_name[-1] != "e" and is_host_arch_x64: + arg = "amd64" + return [vcvarsall, arg] + + def SetupScript(self, target_arch): + script_data = self._SetupScriptInternal(target_arch) + script_path = script_data[0] + if not os.path.exists(script_path): + raise Exception( + "%s is missing - make sure VC++ tools are installed." % script_path + ) + return script_data def _RegistryQueryBase(sysdir, key, value): - """Use reg.exe to read a particular key. + """Use reg.exe to read a particular key. While ideally we might use the win32 module, we would like gyp to be python neutral, so for instance cygwin python lacks this module. @@ -153,28 +166,27 @@ def _RegistryQueryBase(sysdir, key, value): Return: stdout from reg.exe, or None for failure. """ - # Skip if not on Windows or Python Win32 setup issue - if sys.platform not in ('win32', 'cygwin'): - return None - # Setup params to pass to and attempt to launch reg.exe - cmd = [os.path.join(os.environ.get('WINDIR', ''), sysdir, 'reg.exe'), - 'query', key] - if value: - cmd.extend(['/v', value]) - p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) - # Obtain the stdout from reg.exe, reading to the end so p.returncode is valid - # Note that the error text may be in [1] in some cases - text = p.communicate()[0] - if PY3: - text = text.decode('utf-8') - # Check return code from reg.exe; officially 0==success and 1==error - if p.returncode: - return None - return text + # Skip if not on Windows or Python Win32 setup issue + if sys.platform not in ("win32", "cygwin"): + return None + # Setup params to pass to and attempt to launch reg.exe + cmd = [os.path.join(os.environ.get("WINDIR", ""), sysdir, "reg.exe"), "query", key] + if value: + cmd.extend(["/v", value]) + p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + # Obtain the stdout from reg.exe, reading to the end so p.returncode is valid + # Note that the error text may be in [1] in some cases + text = p.communicate()[0] + if PY3: + text = text.decode("utf-8") + # Check return code from reg.exe; officially 0==success and 1==error + if p.returncode: + return None + return text def _RegistryQuery(key, value=None): - r"""Use reg.exe to read a particular key through _RegistryQueryBase. + r"""Use reg.exe to read a particular key through _RegistryQueryBase. First tries to launch from %WinDir%\Sysnative to avoid WoW64 redirection. If that fails, it falls back to System32. Sysnative is available on Vista and @@ -190,19 +202,19 @@ def _RegistryQuery(key, value=None): Return: stdout from reg.exe, or None for failure. """ - text = None - try: - text = _RegistryQueryBase('Sysnative', key, value) - except OSError as e: - if e.errno == errno.ENOENT: - text = _RegistryQueryBase('System32', key, value) - else: - raise - return text + text = None + try: + text = _RegistryQueryBase("Sysnative", key, value) + except OSError as e: + if e.errno == errno.ENOENT: + text = _RegistryQueryBase("System32", key, value) + else: + raise + return text def _RegistryGetValueUsingWinReg(key, value): - """Use the _winreg module to obtain the value of a registry key. + """Use the _winreg module to obtain the value of a registry key. Args: key: The registry key. @@ -211,24 +223,24 @@ def _RegistryGetValueUsingWinReg(key, value): contents of the registry key's value, or None on failure. Throws ImportError if _winreg is unavailable. """ - try: - # Python 2 - from _winreg import HKEY_LOCAL_MACHINE, OpenKey, QueryValueEx - except ImportError: - # Python 3 - from winreg import HKEY_LOCAL_MACHINE, OpenKey, QueryValueEx - - try: - root, subkey = key.split('\\', 1) - assert root == 'HKLM' # Only need HKLM for now. - with OpenKey(HKEY_LOCAL_MACHINE, subkey) as hkey: - return QueryValueEx(hkey, value)[0] - except WindowsError: - return None + try: + # Python 2 + from _winreg import HKEY_LOCAL_MACHINE, OpenKey, QueryValueEx + except ImportError: + # Python 3 + from winreg import HKEY_LOCAL_MACHINE, OpenKey, QueryValueEx + + try: + root, subkey = key.split("\\", 1) + assert root == "HKLM" # Only need HKLM for now. + with OpenKey(HKEY_LOCAL_MACHINE, subkey) as hkey: + return QueryValueEx(hkey, value)[0] + except WindowsError: + return None def _RegistryGetValue(key, value): - """Use _winreg or reg.exe to obtain the value of a registry key. + """Use _winreg or reg.exe to obtain the value of a registry key. Using _winreg is preferable because it solves an issue on some corporate environments where access to reg.exe is locked down. However, we still need @@ -241,161 +253,187 @@ def _RegistryGetValue(key, value): Return: contents of the registry key's value, or None on failure. """ - try: - return _RegistryGetValueUsingWinReg(key, value) - except ImportError: - pass - - # Fallback to reg.exe if we fail to import _winreg. - text = _RegistryQuery(key, value) - if not text: - return None - # Extract value. - match = re.search(r'REG_\w+\s+([^\r]+)\r\n', text) - if not match: - return None - return match.group(1) + try: + return _RegistryGetValueUsingWinReg(key, value) + except ImportError: + pass + + # Fallback to reg.exe if we fail to import _winreg. + text = _RegistryQuery(key, value) + if not text: + return None + # Extract value. + match = re.search(r"REG_\w+\s+([^\r]+)\r\n", text) + if not match: + return None + return match.group(1) def _CreateVersion(name, path, sdk_based=False): - """Sets up MSVS project generation. + """Sets up MSVS project generation. Setup is based off the GYP_MSVS_VERSION environment variable or whatever is autodetected if GYP_MSVS_VERSION is not explicitly specified. If a version is passed in that doesn't match a value in versions python will throw a error. """ - if path: - path = os.path.normpath(path) - versions = { - '2019': VisualStudioVersion('2019', - 'Visual Studio 2019', - solution_version='12.00', - project_version='16.0', - flat_sln=False, - uses_vcxproj=True, - path=path, - sdk_based=sdk_based, - default_toolset='v142', - compatible_sdks=['v8.1', 'v10.0']), - '2017': VisualStudioVersion('2017', - 'Visual Studio 2017', - solution_version='12.00', - project_version='15.0', - flat_sln=False, - uses_vcxproj=True, - path=path, - sdk_based=sdk_based, - default_toolset='v141', - compatible_sdks=['v8.1', 'v10.0']), - '2015': VisualStudioVersion('2015', - 'Visual Studio 2015', - solution_version='12.00', - project_version='14.0', - flat_sln=False, - uses_vcxproj=True, - path=path, - sdk_based=sdk_based, - default_toolset='v140'), - '2013': VisualStudioVersion('2013', - 'Visual Studio 2013', - solution_version='13.00', - project_version='12.0', - flat_sln=False, - uses_vcxproj=True, - path=path, - sdk_based=sdk_based, - default_toolset='v120'), - '2013e': VisualStudioVersion('2013e', - 'Visual Studio 2013', - solution_version='13.00', - project_version='12.0', - flat_sln=True, - uses_vcxproj=True, - path=path, - sdk_based=sdk_based, - default_toolset='v120'), - '2012': VisualStudioVersion('2012', - 'Visual Studio 2012', - solution_version='12.00', - project_version='4.0', - flat_sln=False, - uses_vcxproj=True, - path=path, - sdk_based=sdk_based, - default_toolset='v110'), - '2012e': VisualStudioVersion('2012e', - 'Visual Studio 2012', - solution_version='12.00', - project_version='4.0', - flat_sln=True, - uses_vcxproj=True, - path=path, - sdk_based=sdk_based, - default_toolset='v110'), - '2010': VisualStudioVersion('2010', - 'Visual Studio 2010', - solution_version='11.00', - project_version='4.0', - flat_sln=False, - uses_vcxproj=True, - path=path, - sdk_based=sdk_based), - '2010e': VisualStudioVersion('2010e', - 'Visual C++ Express 2010', - solution_version='11.00', - project_version='4.0', - flat_sln=True, - uses_vcxproj=True, - path=path, - sdk_based=sdk_based), - '2008': VisualStudioVersion('2008', - 'Visual Studio 2008', - solution_version='10.00', - project_version='9.00', - flat_sln=False, - uses_vcxproj=False, - path=path, - sdk_based=sdk_based), - '2008e': VisualStudioVersion('2008e', - 'Visual Studio 2008', - solution_version='10.00', - project_version='9.00', - flat_sln=True, - uses_vcxproj=False, - path=path, - sdk_based=sdk_based), - '2005': VisualStudioVersion('2005', - 'Visual Studio 2005', - solution_version='9.00', - project_version='8.00', - flat_sln=False, - uses_vcxproj=False, - path=path, - sdk_based=sdk_based), - '2005e': VisualStudioVersion('2005e', - 'Visual Studio 2005', - solution_version='9.00', - project_version='8.00', - flat_sln=True, - uses_vcxproj=False, - path=path, - sdk_based=sdk_based), - } - return versions[str(name)] + if path: + path = os.path.normpath(path) + versions = { + "2019": VisualStudioVersion( + "2019", + "Visual Studio 2019", + solution_version="12.00", + project_version="16.0", + flat_sln=False, + uses_vcxproj=True, + path=path, + sdk_based=sdk_based, + default_toolset="v142", + compatible_sdks=["v8.1", "v10.0"], + ), + "2017": VisualStudioVersion( + "2017", + "Visual Studio 2017", + solution_version="12.00", + project_version="15.0", + flat_sln=False, + uses_vcxproj=True, + path=path, + sdk_based=sdk_based, + default_toolset="v141", + compatible_sdks=["v8.1", "v10.0"], + ), + "2015": VisualStudioVersion( + "2015", + "Visual Studio 2015", + solution_version="12.00", + project_version="14.0", + flat_sln=False, + uses_vcxproj=True, + path=path, + sdk_based=sdk_based, + default_toolset="v140", + ), + "2013": VisualStudioVersion( + "2013", + "Visual Studio 2013", + solution_version="13.00", + project_version="12.0", + flat_sln=False, + uses_vcxproj=True, + path=path, + sdk_based=sdk_based, + default_toolset="v120", + ), + "2013e": VisualStudioVersion( + "2013e", + "Visual Studio 2013", + solution_version="13.00", + project_version="12.0", + flat_sln=True, + uses_vcxproj=True, + path=path, + sdk_based=sdk_based, + default_toolset="v120", + ), + "2012": VisualStudioVersion( + "2012", + "Visual Studio 2012", + solution_version="12.00", + project_version="4.0", + flat_sln=False, + uses_vcxproj=True, + path=path, + sdk_based=sdk_based, + default_toolset="v110", + ), + "2012e": VisualStudioVersion( + "2012e", + "Visual Studio 2012", + solution_version="12.00", + project_version="4.0", + flat_sln=True, + uses_vcxproj=True, + path=path, + sdk_based=sdk_based, + default_toolset="v110", + ), + "2010": VisualStudioVersion( + "2010", + "Visual Studio 2010", + solution_version="11.00", + project_version="4.0", + flat_sln=False, + uses_vcxproj=True, + path=path, + sdk_based=sdk_based, + ), + "2010e": VisualStudioVersion( + "2010e", + "Visual C++ Express 2010", + solution_version="11.00", + project_version="4.0", + flat_sln=True, + uses_vcxproj=True, + path=path, + sdk_based=sdk_based, + ), + "2008": VisualStudioVersion( + "2008", + "Visual Studio 2008", + solution_version="10.00", + project_version="9.00", + flat_sln=False, + uses_vcxproj=False, + path=path, + sdk_based=sdk_based, + ), + "2008e": VisualStudioVersion( + "2008e", + "Visual Studio 2008", + solution_version="10.00", + project_version="9.00", + flat_sln=True, + uses_vcxproj=False, + path=path, + sdk_based=sdk_based, + ), + "2005": VisualStudioVersion( + "2005", + "Visual Studio 2005", + solution_version="9.00", + project_version="8.00", + flat_sln=False, + uses_vcxproj=False, + path=path, + sdk_based=sdk_based, + ), + "2005e": VisualStudioVersion( + "2005e", + "Visual Studio 2005", + solution_version="9.00", + project_version="8.00", + flat_sln=True, + uses_vcxproj=False, + path=path, + sdk_based=sdk_based, + ), + } + return versions[str(name)] def _ConvertToCygpath(path): - """Convert to cygwin path if we are using cygwin.""" - if sys.platform == 'cygwin': - p = subprocess.Popen(['cygpath', path], stdout=subprocess.PIPE) - path = p.communicate()[0].strip() - if PY3: - path = path.decode('utf-8') - return path + """Convert to cygwin path if we are using cygwin.""" + if sys.platform == "cygwin": + p = subprocess.Popen(["cygpath", path], stdout=subprocess.PIPE) + path = p.communicate()[0].strip() + if PY3: + path = path.decode("utf-8") + return path def _DetectVisualStudioVersions(versions_to_check, force_express): - """Collect the list of installed visual studio versions. + """Collect the list of installed visual studio versions. Returns: A list of visual studio versions installed in descending order of @@ -412,105 +450,122 @@ def _DetectVisualStudioVersions(versions_to_check, force_express): 2019 - Visual Studio 2019 (16) Where (e) is e for express editions of MSVS and blank otherwise. """ - version_to_year = { - '8.0': '2005', - '9.0': '2008', - '10.0': '2010', - '11.0': '2012', - '12.0': '2013', - '14.0': '2015', - '15.0': '2017', - '16.0': '2019', - } - versions = [] - for version in versions_to_check: - # Old method of searching for which VS version is installed - # We don't use the 2010-encouraged-way because we also want to get the - # path to the binaries, which it doesn't offer. - keys = [r'HKLM\Software\Microsoft\VisualStudio\%s' % version, - r'HKLM\Software\Wow6432Node\Microsoft\VisualStudio\%s' % version, - r'HKLM\Software\Microsoft\VCExpress\%s' % version, - r'HKLM\Software\Wow6432Node\Microsoft\VCExpress\%s' % version] - for index in range(len(keys)): - path = _RegistryGetValue(keys[index], 'InstallDir') - if not path: - continue - path = _ConvertToCygpath(path) - # Check for full. - full_path = os.path.join(path, 'devenv.exe') - express_path = os.path.join(path, '*express.exe') - if not force_express and os.path.exists(full_path): - # Add this one. - versions.append(_CreateVersion(version_to_year[version], - os.path.join(path, '..', '..'))) - # Check for express. - elif glob.glob(express_path): - # Add this one. - versions.append(_CreateVersion(version_to_year[version] + 'e', - os.path.join(path, '..', '..'))) - - # The old method above does not work when only SDK is installed. - keys = [r'HKLM\Software\Microsoft\VisualStudio\SxS\VC7', - r'HKLM\Software\Wow6432Node\Microsoft\VisualStudio\SxS\VC7', - r'HKLM\Software\Microsoft\VisualStudio\SxS\VS7', - r'HKLM\Software\Wow6432Node\Microsoft\VisualStudio\SxS\VS7'] - for index in range(len(keys)): - path = _RegistryGetValue(keys[index], version) - if not path: - continue - path = _ConvertToCygpath(path) - if version == '15.0': - if os.path.exists(path): - versions.append(_CreateVersion('2017', path)) - elif version != '14.0': # There is no Express edition for 2015. - versions.append(_CreateVersion(version_to_year[version] + 'e', - os.path.join(path, '..'), sdk_based=True)) - - return versions - - -def SelectVisualStudioVersion(version='auto', allow_fallback=True): - """Select which version of Visual Studio projects to generate. + version_to_year = { + "8.0": "2005", + "9.0": "2008", + "10.0": "2010", + "11.0": "2012", + "12.0": "2013", + "14.0": "2015", + "15.0": "2017", + "16.0": "2019", + } + versions = [] + for version in versions_to_check: + # Old method of searching for which VS version is installed + # We don't use the 2010-encouraged-way because we also want to get the + # path to the binaries, which it doesn't offer. + keys = [ + r"HKLM\Software\Microsoft\VisualStudio\%s" % version, + r"HKLM\Software\Wow6432Node\Microsoft\VisualStudio\%s" % version, + r"HKLM\Software\Microsoft\VCExpress\%s" % version, + r"HKLM\Software\Wow6432Node\Microsoft\VCExpress\%s" % version, + ] + for index in range(len(keys)): + path = _RegistryGetValue(keys[index], "InstallDir") + if not path: + continue + path = _ConvertToCygpath(path) + # Check for full. + full_path = os.path.join(path, "devenv.exe") + express_path = os.path.join(path, "*express.exe") + if not force_express and os.path.exists(full_path): + # Add this one. + versions.append( + _CreateVersion( + version_to_year[version], os.path.join(path, "..", "..") + ) + ) + # Check for express. + elif glob.glob(express_path): + # Add this one. + versions.append( + _CreateVersion( + version_to_year[version] + "e", os.path.join(path, "..", "..") + ) + ) + + # The old method above does not work when only SDK is installed. + keys = [ + r"HKLM\Software\Microsoft\VisualStudio\SxS\VC7", + r"HKLM\Software\Wow6432Node\Microsoft\VisualStudio\SxS\VC7", + r"HKLM\Software\Microsoft\VisualStudio\SxS\VS7", + r"HKLM\Software\Wow6432Node\Microsoft\VisualStudio\SxS\VS7", + ] + for index in range(len(keys)): + path = _RegistryGetValue(keys[index], version) + if not path: + continue + path = _ConvertToCygpath(path) + if version == "15.0": + if os.path.exists(path): + versions.append(_CreateVersion("2017", path)) + elif version != "14.0": # There is no Express edition for 2015. + versions.append( + _CreateVersion( + version_to_year[version] + "e", + os.path.join(path, ".."), + sdk_based=True, + ) + ) + + return versions + + +def SelectVisualStudioVersion(version="auto", allow_fallback=True): + """Select which version of Visual Studio projects to generate. Arguments: version: Hook to allow caller to force a particular version (vs auto). Returns: An object representing a visual studio project format version. """ - # In auto mode, check environment variable for override. - if version == 'auto': - version = os.environ.get('GYP_MSVS_VERSION', 'auto') - version_map = { - 'auto': ('16.0', '15.0', '14.0', '12.0', '10.0', '9.0', '8.0', '11.0'), - '2005': ('8.0',), - '2005e': ('8.0',), - '2008': ('9.0',), - '2008e': ('9.0',), - '2010': ('10.0',), - '2010e': ('10.0',), - '2012': ('11.0',), - '2012e': ('11.0',), - '2013': ('12.0',), - '2013e': ('12.0',), - '2015': ('14.0',), - '2017': ('15.0',), - '2019': ('16.0',), - } - override_path = os.environ.get('GYP_MSVS_OVERRIDE_PATH') - if override_path: - msvs_version = os.environ.get('GYP_MSVS_VERSION') - if not msvs_version: - raise ValueError('GYP_MSVS_OVERRIDE_PATH requires GYP_MSVS_VERSION to be ' - 'set to a particular version (e.g. 2010e).') - return _CreateVersion(msvs_version, override_path, sdk_based=True) - version = str(version) - versions = _DetectVisualStudioVersions(version_map[version], 'e' in version) - if not versions: - if not allow_fallback: - raise ValueError('Could not locate Visual Studio installation.') - if version == 'auto': - # Default to 2005 if we couldn't find anything - return _CreateVersion('2005', None) - else: - return _CreateVersion(version, None) - return versions[0] + # In auto mode, check environment variable for override. + if version == "auto": + version = os.environ.get("GYP_MSVS_VERSION", "auto") + version_map = { + "auto": ("16.0", "15.0", "14.0", "12.0", "10.0", "9.0", "8.0", "11.0"), + "2005": ("8.0",), + "2005e": ("8.0",), + "2008": ("9.0",), + "2008e": ("9.0",), + "2010": ("10.0",), + "2010e": ("10.0",), + "2012": ("11.0",), + "2012e": ("11.0",), + "2013": ("12.0",), + "2013e": ("12.0",), + "2015": ("14.0",), + "2017": ("15.0",), + "2019": ("16.0",), + } + override_path = os.environ.get("GYP_MSVS_OVERRIDE_PATH") + if override_path: + msvs_version = os.environ.get("GYP_MSVS_VERSION") + if not msvs_version: + raise ValueError( + "GYP_MSVS_OVERRIDE_PATH requires GYP_MSVS_VERSION to be " + "set to a particular version (e.g. 2010e)." + ) + return _CreateVersion(msvs_version, override_path, sdk_based=True) + version = str(version) + versions = _DetectVisualStudioVersions(version_map[version], "e" in version) + if not versions: + if not allow_fallback: + raise ValueError("Could not locate Visual Studio installation.") + if version == "auto": + # Default to 2005 if we couldn't find anything + return _CreateVersion("2005", None) + else: + return _CreateVersion(version, None) + return versions[0] diff --git a/gyp/pylib/gyp/__init__.py b/gyp/pylib/gyp/__init__.py index dee834013f..e249536429 100755 --- a/gyp/pylib/gyp/__init__.py +++ b/gyp/pylib/gyp/__init__.py @@ -27,153 +27,180 @@ debug = {} # List of "official" debug modes, but you can use anything you like. -DEBUG_GENERAL = 'general' -DEBUG_VARIABLES = 'variables' -DEBUG_INCLUDES = 'includes' +DEBUG_GENERAL = "general" +DEBUG_VARIABLES = "variables" +DEBUG_INCLUDES = "includes" def DebugOutput(mode, message, *args): - if 'all' in gyp.debug or mode in gyp.debug: - ctx = ('unknown', 0, 'unknown') - try: - f = traceback.extract_stack(limit=2) - if f: - ctx = f[0][:3] - except: - pass - if args: - message %= args - print('%s:%s:%d:%s %s' % (mode.upper(), os.path.basename(ctx[0]), - ctx[1], ctx[2], message)) + if "all" in gyp.debug or mode in gyp.debug: + ctx = ("unknown", 0, "unknown") + try: + f = traceback.extract_stack(limit=2) + if f: + ctx = f[0][:3] + except Exception: + pass + if args: + message %= args + print( + "%s:%s:%d:%s %s" + % (mode.upper(), os.path.basename(ctx[0]), ctx[1], ctx[2], message) + ) + def FindBuildFiles(): - extension = '.gyp' - files = os.listdir(os.getcwd()) - build_files = [] - for file in files: - if file.endswith(extension): - build_files.append(file) - return build_files - - -def Load(build_files, format, default_variables={}, - includes=[], depth='.', params=None, check=False, - circular_check=True, duplicate_basename_check=True): - """ + extension = ".gyp" + files = os.listdir(os.getcwd()) + build_files = [] + for file in files: + if file.endswith(extension): + build_files.append(file) + return build_files + + +def Load( + build_files, + format, + default_variables={}, + includes=[], + depth=".", + params=None, + check=False, + circular_check=True, + duplicate_basename_check=True, +): + """ Loads one or more specified build files. default_variables and includes will be copied before use. Returns the generator for the specified format and the data returned by loading the specified build files. """ - if params is None: - params = {} - - if '-' in format: - format, params['flavor'] = format.split('-', 1) - - default_variables = copy.copy(default_variables) - - # Default variables provided by this program and its modules should be - # named WITH_CAPITAL_LETTERS to provide a distinct "best practice" namespace, - # avoiding collisions with user and automatic variables. - default_variables['GENERATOR'] = format - default_variables['GENERATOR_FLAVOR'] = params.get('flavor', '') - - # Format can be a custom python file, or by default the name of a module - # within gyp.generator. - if format.endswith('.py'): - generator_name = os.path.splitext(format)[0] - path, generator_name = os.path.split(generator_name) - - # Make sure the path to the custom generator is in sys.path - # Don't worry about removing it once we are done. Keeping the path - # to each generator that is used in sys.path is likely harmless and - # arguably a good idea. - path = os.path.abspath(path) - if path not in sys.path: - sys.path.insert(0, path) - else: - generator_name = 'gyp.generator.' + format - - # These parameters are passed in order (as opposed to by key) - # because ActivePython cannot handle key parameters to __import__. - generator = __import__(generator_name, globals(), locals(), generator_name) - for (key, val) in generator.generator_default_variables.items(): - default_variables.setdefault(key, val) - - # Give the generator the opportunity to set additional variables based on - # the params it will receive in the output phase. - if getattr(generator, 'CalculateVariables', None): - generator.CalculateVariables(default_variables, params) - - # Give the generator the opportunity to set generator_input_info based on - # the params it will receive in the output phase. - if getattr(generator, 'CalculateGeneratorInputInfo', None): - generator.CalculateGeneratorInputInfo(params) - - # Fetch the generator specific info that gets fed to input, we use getattr - # so we can default things and the generators only have to provide what - # they need. - generator_input_info = { - 'non_configuration_keys': - getattr(generator, 'generator_additional_non_configuration_keys', []), - 'path_sections': - getattr(generator, 'generator_additional_path_sections', []), - 'extra_sources_for_rules': - getattr(generator, 'generator_extra_sources_for_rules', []), - 'generator_supports_multiple_toolsets': - getattr(generator, 'generator_supports_multiple_toolsets', False), - 'generator_wants_static_library_dependencies_adjusted': - getattr(generator, - 'generator_wants_static_library_dependencies_adjusted', True), - 'generator_wants_sorted_dependencies': - getattr(generator, 'generator_wants_sorted_dependencies', False), - 'generator_filelist_paths': - getattr(generator, 'generator_filelist_paths', None), - } - - # Process the input specific to this generator. - result = gyp.input.Load(build_files, default_variables, includes[:], - depth, generator_input_info, check, circular_check, - duplicate_basename_check, - params['parallel'], params['root_targets']) - return [generator] + result + if params is None: + params = {} + + if "-" in format: + format, params["flavor"] = format.split("-", 1) + + default_variables = copy.copy(default_variables) + + # Default variables provided by this program and its modules should be + # named WITH_CAPITAL_LETTERS to provide a distinct "best practice" namespace, + # avoiding collisions with user and automatic variables. + default_variables["GENERATOR"] = format + default_variables["GENERATOR_FLAVOR"] = params.get("flavor", "") + + # Format can be a custom python file, or by default the name of a module + # within gyp.generator. + if format.endswith(".py"): + generator_name = os.path.splitext(format)[0] + path, generator_name = os.path.split(generator_name) + + # Make sure the path to the custom generator is in sys.path + # Don't worry about removing it once we are done. Keeping the path + # to each generator that is used in sys.path is likely harmless and + # arguably a good idea. + path = os.path.abspath(path) + if path not in sys.path: + sys.path.insert(0, path) + else: + generator_name = "gyp.generator." + format + + # These parameters are passed in order (as opposed to by key) + # because ActivePython cannot handle key parameters to __import__. + generator = __import__(generator_name, globals(), locals(), generator_name) + for (key, val) in generator.generator_default_variables.items(): + default_variables.setdefault(key, val) + + # Give the generator the opportunity to set additional variables based on + # the params it will receive in the output phase. + if getattr(generator, "CalculateVariables", None): + generator.CalculateVariables(default_variables, params) + + # Give the generator the opportunity to set generator_input_info based on + # the params it will receive in the output phase. + if getattr(generator, "CalculateGeneratorInputInfo", None): + generator.CalculateGeneratorInputInfo(params) + + # Fetch the generator specific info that gets fed to input, we use getattr + # so we can default things and the generators only have to provide what + # they need. + generator_input_info = { + "non_configuration_keys": getattr( + generator, "generator_additional_non_configuration_keys", [] + ), + "path_sections": getattr(generator, "generator_additional_path_sections", []), + "extra_sources_for_rules": getattr( + generator, "generator_extra_sources_for_rules", [] + ), + "generator_supports_multiple_toolsets": getattr( + generator, "generator_supports_multiple_toolsets", False + ), + "generator_wants_static_library_dependencies_adjusted": getattr( + generator, "generator_wants_static_library_dependencies_adjusted", True + ), + "generator_wants_sorted_dependencies": getattr( + generator, "generator_wants_sorted_dependencies", False + ), + "generator_filelist_paths": getattr( + generator, "generator_filelist_paths", None + ), + } + + # Process the input specific to this generator. + result = gyp.input.Load( + build_files, + default_variables, + includes[:], + depth, + generator_input_info, + check, + circular_check, + duplicate_basename_check, + params["parallel"], + params["root_targets"], + ) + return [generator] + result + def NameValueListToDict(name_value_list): - """ + """ Takes an array of strings of the form 'NAME=VALUE' and creates a dictionary of the pairs. If a string is simply NAME, then the value in the dictionary is set to True. If VALUE can be converted to an integer, it is. """ - result = { } - for item in name_value_list: - tokens = item.split('=', 1) - if len(tokens) == 2: - # If we can make it an int, use that, otherwise, use the string. - try: - token_value = int(tokens[1]) - except ValueError: - token_value = tokens[1] - # Set the variable to the supplied value. - result[tokens[0]] = token_value - else: - # No value supplied, treat it as a boolean and set it. - result[tokens[0]] = True - return result + result = {} + for item in name_value_list: + tokens = item.split("=", 1) + if len(tokens) == 2: + # If we can make it an int, use that, otherwise, use the string. + try: + token_value = int(tokens[1]) + except ValueError: + token_value = tokens[1] + # Set the variable to the supplied value. + result[tokens[0]] = token_value + else: + # No value supplied, treat it as a boolean and set it. + result[tokens[0]] = True + return result + def ShlexEnv(env_name): - flags = os.environ.get(env_name, []) - if flags: - flags = shlex.split(flags) - return flags + flags = os.environ.get(env_name, []) + if flags: + flags = shlex.split(flags) + return flags + def FormatOpt(opt, value): - if opt.startswith('--'): - return '%s=%s' % (opt, value) - return opt + value + if opt.startswith("--"): + return "%s=%s" % (opt, value) + return opt + value + def RegenerateAppendFlag(flag, values, predicate, env_name, options): - """Regenerate a list of command line flags, for an option of action='append'. + """Regenerate a list of command line flags, for an option of action='append'. The |env_name|, if given, is checked in the environment and used to generate an initial list of options, then the options that were specified on the @@ -182,20 +209,21 @@ def RegenerateAppendFlag(flag, values, predicate, env_name, options): the environment, while not requiring the environment to be set when the flags are used again. """ - flags = [] - if options.use_environment and env_name: - for flag_value in ShlexEnv(env_name): - value = FormatOpt(flag, predicate(flag_value)) - if value in flags: - flags.remove(value) - flags.append(value) - if values: - for flag_value in values: - flags.append(FormatOpt(flag, predicate(flag_value))) - return flags + flags = [] + if options.use_environment and env_name: + for flag_value in ShlexEnv(env_name): + value = FormatOpt(flag, predicate(flag_value)) + if value in flags: + flags.remove(value) + flags.append(value) + if values: + for flag_value in values: + flags.append(FormatOpt(flag, predicate(flag_value))) + return flags + def RegenerateFlags(options): - """Given a parsed options object, and taking the environment variables into + """Given a parsed options object, and taking the environment variables into account, returns a list of flags that should regenerate an equivalent options object (even in the absence of the environment variables.) @@ -204,53 +232,62 @@ def RegenerateFlags(options): The format flag is not included, as it is assumed the calling generator will set that as appropriate. """ - def FixPath(path): - path = gyp.common.FixIfRelativePath(path, options.depth) - if not path: - return os.path.curdir - return path - - def Noop(value): - return value - - # We always want to ignore the environment when regenerating, to avoid - # duplicate or changed flags in the environment at the time of regeneration. - flags = ['--ignore-environment'] - for name, metadata in options._regeneration_metadata.items(): - opt = metadata['opt'] - value = getattr(options, name) - value_predicate = metadata['type'] == 'path' and FixPath or Noop - action = metadata['action'] - env_name = metadata['env_name'] - if action == 'append': - flags.extend(RegenerateAppendFlag(opt, value, value_predicate, - env_name, options)) - elif action in ('store', None): # None is a synonym for 'store'. - if value: - flags.append(FormatOpt(opt, value_predicate(value))) - elif options.use_environment and env_name and os.environ.get(env_name): - flags.append(FormatOpt(opt, value_predicate(os.environ.get(env_name)))) - elif action in ('store_true', 'store_false'): - if ((action == 'store_true' and value) or - (action == 'store_false' and not value)): - flags.append(opt) - elif options.use_environment and env_name: - print('Warning: environment regeneration unimplemented ' - 'for %s flag %r env_name %r' % (action, opt, - env_name), file=sys.stderr) - else: - print('Warning: regeneration unimplemented for action %r ' - 'flag %r' % (action, opt), file=sys.stderr) - return flags + def FixPath(path): + path = gyp.common.FixIfRelativePath(path, options.depth) + if not path: + return os.path.curdir + return path + + def Noop(value): + return value + + # We always want to ignore the environment when regenerating, to avoid + # duplicate or changed flags in the environment at the time of regeneration. + flags = ["--ignore-environment"] + for name, metadata in options._regeneration_metadata.items(): + opt = metadata["opt"] + value = getattr(options, name) + value_predicate = metadata["type"] == "path" and FixPath or Noop + action = metadata["action"] + env_name = metadata["env_name"] + if action == "append": + flags.extend( + RegenerateAppendFlag(opt, value, value_predicate, env_name, options) + ) + elif action in ("store", None): # None is a synonym for 'store'. + if value: + flags.append(FormatOpt(opt, value_predicate(value))) + elif options.use_environment and env_name and os.environ.get(env_name): + flags.append(FormatOpt(opt, value_predicate(os.environ.get(env_name)))) + elif action in ("store_true", "store_false"): + if (action == "store_true" and value) or ( + action == "store_false" and not value + ): + flags.append(opt) + elif options.use_environment and env_name: + print( + "Warning: environment regeneration unimplemented " + "for %s flag %r env_name %r" % (action, opt, env_name), + file=sys.stderr, + ) + else: + print( + "Warning: regeneration unimplemented for action %r " + "flag %r" % (action, opt), + file=sys.stderr, + ) + + return flags + class RegeneratableOptionParser(argparse.ArgumentParser): - def __init__(self, usage): - self.__regeneratable_options = {} - argparse.ArgumentParser.__init__(self, usage=usage) + def __init__(self, usage): + self.__regeneratable_options = {} + argparse.ArgumentParser.__init__(self, usage=usage) - def add_argument(self, *args, **kw): - """Add an option to the parser. + def add_argument(self, *args, **kw): + """Add an option to the parser. This accepts the same arguments as ArgumentParser.add_argument, plus the following: @@ -261,297 +298,394 @@ def add_argument(self, *args, **kw): type: adds type='path', to tell the regenerator that the values of this option need to be made relative to options.depth """ - env_name = kw.pop('env_name', None) - if 'dest' in kw and kw.pop('regenerate', True): - dest = kw['dest'] - - # The path type is needed for regenerating, for optparse we can just treat - # it as a string. - type = kw.get('type') - if type == 'path': - kw['type'] = str - - self.__regeneratable_options[dest] = { - 'action': kw.get('action'), - 'type': type, - 'env_name': env_name, - 'opt': args[0], - } + env_name = kw.pop("env_name", None) + if "dest" in kw and kw.pop("regenerate", True): + dest = kw["dest"] + + # The path type is needed for regenerating, for optparse we can just treat + # it as a string. + type = kw.get("type") + if type == "path": + kw["type"] = str + + self.__regeneratable_options[dest] = { + "action": kw.get("action"), + "type": type, + "env_name": env_name, + "opt": args[0], + } - argparse.ArgumentParser.add_argument(self, *args, **kw) + argparse.ArgumentParser.add_argument(self, *args, **kw) + + def parse_args(self, *args): + values, args = argparse.ArgumentParser.parse_known_args(self, *args) + values._regeneration_metadata = self.__regeneratable_options + return values, args - def parse_args(self, *args): - values, args = argparse.ArgumentParser.parse_known_args(self, *args) - values._regeneration_metadata = self.__regeneratable_options - return values, args def gyp_main(args): - my_name = os.path.basename(sys.argv[0]) - usage = 'usage: %(prog)s [options ...] [build_file ...]' - - - parser = RegeneratableOptionParser(usage=usage.replace('%s', '%(prog)s')) - parser.add_argument('--build', dest='configs', action='append', - help='configuration for build after project generation') - parser.add_argument('--check', dest='check', action='store_true', - help='check format of gyp files') - parser.add_argument('--config-dir', dest='config_dir', action='store', - env_name='GYP_CONFIG_DIR', default=None, - help='The location for configuration files like ' - 'include.gypi.') - parser.add_argument('-d', '--debug', dest='debug', metavar='DEBUGMODE', - action='append', default=[], help='turn on a debugging ' - 'mode for debugging GYP. Supported modes are "variables", ' - '"includes" and "general" or "all" for all of them.') - parser.add_argument('-D', dest='defines', action='append', metavar='VAR=VAL', - env_name='GYP_DEFINES', - help='sets variable VAR to value VAL') - parser.add_argument('--depth', dest='depth', metavar='PATH', type='path', - help='set DEPTH gyp variable to a relative path to PATH') - parser.add_argument('-f', '--format', dest='formats', action='append', - env_name='GYP_GENERATORS', regenerate=False, - help='output formats to generate') - parser.add_argument('-G', dest='generator_flags', action='append', default=[], - metavar='FLAG=VAL', env_name='GYP_GENERATOR_FLAGS', - help='sets generator flag FLAG to VAL') - parser.add_argument('--generator-output', dest='generator_output', - action='store', default=None, metavar='DIR', type='path', - env_name='GYP_GENERATOR_OUTPUT', - help='puts generated build files under DIR') - parser.add_argument('--ignore-environment', dest='use_environment', - action='store_false', default=True, regenerate=False, - help='do not read options from environment variables') - parser.add_argument('-I', '--include', dest='includes', action='append', - metavar='INCLUDE', type='path', - help='files to include in all loaded .gyp files') - # --no-circular-check disables the check for circular relationships between - # .gyp files. These relationships should not exist, but they've only been - # observed to be harmful with the Xcode generator. Chromium's .gyp files - # currently have some circular relationships on non-Mac platforms, so this - # option allows the strict behavior to be used on Macs and the lenient - # behavior to be used elsewhere. - # TODO(mark): Remove this option when http://crbug.com/35878 is fixed. - parser.add_argument('--no-circular-check', dest='circular_check', - action='store_false', default=True, regenerate=False, - help="don't check for circular relationships between files") - # --no-duplicate-basename-check disables the check for duplicate basenames - # in a static_library/shared_library project. Visual C++ 2008 generator - # doesn't support this configuration. Libtool on Mac also generates warnings - # when duplicate basenames are passed into Make generator on Mac. - # TODO(yukawa): Remove this option when these legacy generators are - # deprecated. - parser.add_argument('--no-duplicate-basename-check', - dest='duplicate_basename_check', action='store_false', - default=True, regenerate=False, - help="don't check for duplicate basenames") - parser.add_argument('--no-parallel', action='store_true', default=False, - help='Disable multiprocessing') - parser.add_argument('-S', '--suffix', dest='suffix', default='', - help='suffix to add to generated files') - parser.add_argument('--toplevel-dir', dest='toplevel_dir', action='store', - default=None, metavar='DIR', type='path', - help='directory to use as the root of the source tree') - parser.add_argument('-R', '--root-target', dest='root_targets', - action='append', metavar='TARGET', - help='include only TARGET and its deep dependencies') - - options, build_files_arg = parser.parse_args(args) - build_files = build_files_arg - - # Set up the configuration directory (defaults to ~/.gyp) - if not options.config_dir: - home = None - home_dot_gyp = None + my_name = os.path.basename(sys.argv[0]) + usage = "usage: %(prog)s [options ...] [build_file ...]" + + parser = RegeneratableOptionParser(usage=usage.replace("%s", "%(prog)s")) + parser.add_argument( + "--build", + dest="configs", + action="append", + help="configuration for build after project generation", + ) + parser.add_argument( + "--check", dest="check", action="store_true", help="check format of gyp files" + ) + parser.add_argument( + "--config-dir", + dest="config_dir", + action="store", + env_name="GYP_CONFIG_DIR", + default=None, + help="The location for configuration files like " "include.gypi.", + ) + parser.add_argument( + "-d", + "--debug", + dest="debug", + metavar="DEBUGMODE", + action="append", + default=[], + help="turn on a debugging " + 'mode for debugging GYP. Supported modes are "variables", ' + '"includes" and "general" or "all" for all of them.', + ) + parser.add_argument( + "-D", + dest="defines", + action="append", + metavar="VAR=VAL", + env_name="GYP_DEFINES", + help="sets variable VAR to value VAL", + ) + parser.add_argument( + "--depth", + dest="depth", + metavar="PATH", + type="path", + help="set DEPTH gyp variable to a relative path to PATH", + ) + parser.add_argument( + "-f", + "--format", + dest="formats", + action="append", + env_name="GYP_GENERATORS", + regenerate=False, + help="output formats to generate", + ) + parser.add_argument( + "-G", + dest="generator_flags", + action="append", + default=[], + metavar="FLAG=VAL", + env_name="GYP_GENERATOR_FLAGS", + help="sets generator flag FLAG to VAL", + ) + parser.add_argument( + "--generator-output", + dest="generator_output", + action="store", + default=None, + metavar="DIR", + type="path", + env_name="GYP_GENERATOR_OUTPUT", + help="puts generated build files under DIR", + ) + parser.add_argument( + "--ignore-environment", + dest="use_environment", + action="store_false", + default=True, + regenerate=False, + help="do not read options from environment variables", + ) + parser.add_argument( + "-I", + "--include", + dest="includes", + action="append", + metavar="INCLUDE", + type="path", + help="files to include in all loaded .gyp files", + ) + # --no-circular-check disables the check for circular relationships between + # .gyp files. These relationships should not exist, but they've only been + # observed to be harmful with the Xcode generator. Chromium's .gyp files + # currently have some circular relationships on non-Mac platforms, so this + # option allows the strict behavior to be used on Macs and the lenient + # behavior to be used elsewhere. + # TODO(mark): Remove this option when http://crbug.com/35878 is fixed. + parser.add_argument( + "--no-circular-check", + dest="circular_check", + action="store_false", + default=True, + regenerate=False, + help="don't check for circular relationships between files", + ) + # --no-duplicate-basename-check disables the check for duplicate basenames + # in a static_library/shared_library project. Visual C++ 2008 generator + # doesn't support this configuration. Libtool on Mac also generates warnings + # when duplicate basenames are passed into Make generator on Mac. + # TODO(yukawa): Remove this option when these legacy generators are + # deprecated. + parser.add_argument( + "--no-duplicate-basename-check", + dest="duplicate_basename_check", + action="store_false", + default=True, + regenerate=False, + help="don't check for duplicate basenames", + ) + parser.add_argument( + "--no-parallel", + action="store_true", + default=False, + help="Disable multiprocessing", + ) + parser.add_argument( + "-S", + "--suffix", + dest="suffix", + default="", + help="suffix to add to generated files", + ) + parser.add_argument( + "--toplevel-dir", + dest="toplevel_dir", + action="store", + default=None, + metavar="DIR", + type="path", + help="directory to use as the root of the source tree", + ) + parser.add_argument( + "-R", + "--root-target", + dest="root_targets", + action="append", + metavar="TARGET", + help="include only TARGET and its deep dependencies", + ) + + options, build_files_arg = parser.parse_args(args) + build_files = build_files_arg + + # Set up the configuration directory (defaults to ~/.gyp) + if not options.config_dir: + home = None + home_dot_gyp = None + if options.use_environment: + home_dot_gyp = os.environ.get("GYP_CONFIG_DIR", None) + if home_dot_gyp: + home_dot_gyp = os.path.expanduser(home_dot_gyp) + + if not home_dot_gyp: + home_vars = ["HOME"] + if sys.platform in ("cygwin", "win32"): + home_vars.append("USERPROFILE") + for home_var in home_vars: + home = os.getenv(home_var) + if home: + home_dot_gyp = os.path.join(home, ".gyp") + if not os.path.exists(home_dot_gyp): + home_dot_gyp = None + else: + break + else: + home_dot_gyp = os.path.expanduser(options.config_dir) + + if home_dot_gyp and not os.path.exists(home_dot_gyp): + home_dot_gyp = None + + if not options.formats: + # If no format was given on the command line, then check the env variable. + generate_formats = [] + if options.use_environment: + generate_formats = os.environ.get("GYP_GENERATORS", []) + if generate_formats: + generate_formats = re.split(r"[\s,]", generate_formats) + if generate_formats: + options.formats = generate_formats + else: + # Nothing in the variable, default based on platform. + if sys.platform == "darwin": + options.formats = ["xcode"] + elif sys.platform in ("win32", "cygwin"): + options.formats = ["msvs"] + else: + options.formats = ["make"] + + if not options.generator_output and options.use_environment: + g_o = os.environ.get("GYP_GENERATOR_OUTPUT") + if g_o: + options.generator_output = g_o + + options.parallel = not options.no_parallel + + for mode in options.debug: + gyp.debug[mode] = 1 + + # Do an extra check to avoid work when we're not debugging. + if DEBUG_GENERAL in gyp.debug: + DebugOutput(DEBUG_GENERAL, "running with these options:") + for option, value in sorted(options.__dict__.items()): + if option[0] == "_": + continue + if isinstance(value, string_types): + DebugOutput(DEBUG_GENERAL, " %s: '%s'", option, value) + else: + DebugOutput(DEBUG_GENERAL, " %s: %s", option, value) + + if not build_files: + build_files = FindBuildFiles() + if not build_files: + raise GypError((usage + "\n\n%s: error: no build_file") % (my_name, my_name)) + + # TODO(mark): Chromium-specific hack! + # For Chromium, the gyp "depth" variable should always be a relative path + # to Chromium's top-level "src" directory. If no depth variable was set + # on the command line, try to find a "src" directory by looking at the + # absolute path to each build file's directory. The first "src" component + # found will be treated as though it were the path used for --depth. + if not options.depth: + for build_file in build_files: + build_file_dir = os.path.abspath(os.path.dirname(build_file)) + build_file_dir_components = build_file_dir.split(os.path.sep) + components_len = len(build_file_dir_components) + for index in range(components_len - 1, -1, -1): + if build_file_dir_components[index] == "src": + options.depth = os.path.sep.join(build_file_dir_components) + break + del build_file_dir_components[index] + + # If the inner loop found something, break without advancing to another + # build file. + if options.depth: + break + + if not options.depth: + raise GypError( + "Could not automatically locate src directory. This is" + "a temporary Chromium feature that will be removed. Use" + "--depth as a workaround." + ) + + # If toplevel-dir is not set, we assume that depth is the root of our source + # tree. + if not options.toplevel_dir: + options.toplevel_dir = options.depth + + # -D on the command line sets variable defaults - D isn't just for define, + # it's for default. Perhaps there should be a way to force (-F?) a + # variable's value so that it can't be overridden by anything else. + cmdline_default_variables = {} + defines = [] if options.use_environment: - home_dot_gyp = os.environ.get('GYP_CONFIG_DIR', None) - if home_dot_gyp: - home_dot_gyp = os.path.expanduser(home_dot_gyp) - - if not home_dot_gyp: - home_vars = ['HOME'] - if sys.platform in ('cygwin', 'win32'): - home_vars.append('USERPROFILE') - for home_var in home_vars: - home = os.getenv(home_var) - if home != None: - home_dot_gyp = os.path.join(home, '.gyp') - if not os.path.exists(home_dot_gyp): - home_dot_gyp = None - else: - break - else: - home_dot_gyp = os.path.expanduser(options.config_dir) - - if home_dot_gyp and not os.path.exists(home_dot_gyp): - home_dot_gyp = None - - if not options.formats: - # If no format was given on the command line, then check the env variable. - generate_formats = [] + defines += ShlexEnv("GYP_DEFINES") + if options.defines: + defines += options.defines + cmdline_default_variables = NameValueListToDict(defines) + if DEBUG_GENERAL in gyp.debug: + DebugOutput( + DEBUG_GENERAL, "cmdline_default_variables: %s", cmdline_default_variables + ) + + # Set up includes. + includes = [] + + # If ~/.gyp/include.gypi exists, it'll be forcibly included into every + # .gyp file that's loaded, before anything else is included. + if home_dot_gyp: + default_include = os.path.join(home_dot_gyp, "include.gypi") + if os.path.exists(default_include): + print("Using overrides found in " + default_include) + includes.append(default_include) + + # Command-line --include files come after the default include. + if options.includes: + includes.extend(options.includes) + + # Generator flags should be prefixed with the target generator since they + # are global across all generator runs. + gen_flags = [] if options.use_environment: - generate_formats = os.environ.get('GYP_GENERATORS', []) - if generate_formats: - generate_formats = re.split(r'[\s,]', generate_formats) - if generate_formats: - options.formats = generate_formats - else: - # Nothing in the variable, default based on platform. - if sys.platform == 'darwin': - options.formats = ['xcode'] - elif sys.platform in ('win32', 'cygwin'): - options.formats = ['msvs'] - else: - options.formats = ['make'] - - if not options.generator_output and options.use_environment: - g_o = os.environ.get('GYP_GENERATOR_OUTPUT') - if g_o: - options.generator_output = g_o - - options.parallel = not options.no_parallel - - for mode in options.debug: - gyp.debug[mode] = 1 - - # Do an extra check to avoid work when we're not debugging. - if DEBUG_GENERAL in gyp.debug: - DebugOutput(DEBUG_GENERAL, 'running with these options:') - for option, value in sorted(options.__dict__.items()): - if option[0] == '_': - continue - if isinstance(value, string_types): - DebugOutput(DEBUG_GENERAL, " %s: '%s'", option, value) - else: - DebugOutput(DEBUG_GENERAL, " %s: %s", option, value) - - if not build_files: - build_files = FindBuildFiles() - if not build_files: - raise GypError((usage + '\n\n%s: error: no build_file') % - (my_name, my_name)) - - # TODO(mark): Chromium-specific hack! - # For Chromium, the gyp "depth" variable should always be a relative path - # to Chromium's top-level "src" directory. If no depth variable was set - # on the command line, try to find a "src" directory by looking at the - # absolute path to each build file's directory. The first "src" component - # found will be treated as though it were the path used for --depth. - if not options.depth: - for build_file in build_files: - build_file_dir = os.path.abspath(os.path.dirname(build_file)) - build_file_dir_components = build_file_dir.split(os.path.sep) - components_len = len(build_file_dir_components) - for index in range(components_len - 1, -1, -1): - if build_file_dir_components[index] == 'src': - options.depth = os.path.sep.join(build_file_dir_components) - break - del build_file_dir_components[index] - - # If the inner loop found something, break without advancing to another - # build file. - if options.depth: - break + gen_flags += ShlexEnv("GYP_GENERATOR_FLAGS") + if options.generator_flags: + gen_flags += options.generator_flags + generator_flags = NameValueListToDict(gen_flags) + if DEBUG_GENERAL in gyp.debug.keys(): + DebugOutput(DEBUG_GENERAL, "generator_flags: %s", generator_flags) + + # Generate all requested formats (use a set in case we got one format request + # twice) + for format in set(options.formats): + params = { + "options": options, + "build_files": build_files, + "generator_flags": generator_flags, + "cwd": os.getcwd(), + "build_files_arg": build_files_arg, + "gyp_binary": sys.argv[0], + "home_dot_gyp": home_dot_gyp, + "parallel": options.parallel, + "root_targets": options.root_targets, + "target_arch": cmdline_default_variables.get("target_arch", ""), + } - if not options.depth: - raise GypError('Could not automatically locate src directory. This is' - 'a temporary Chromium feature that will be removed. Use' - '--depth as a workaround.') - - # If toplevel-dir is not set, we assume that depth is the root of our source - # tree. - if not options.toplevel_dir: - options.toplevel_dir = options.depth - - # -D on the command line sets variable defaults - D isn't just for define, - # it's for default. Perhaps there should be a way to force (-F?) a - # variable's value so that it can't be overridden by anything else. - cmdline_default_variables = {} - defines = [] - if options.use_environment: - defines += ShlexEnv('GYP_DEFINES') - if options.defines: - defines += options.defines - cmdline_default_variables = NameValueListToDict(defines) - if DEBUG_GENERAL in gyp.debug: - DebugOutput(DEBUG_GENERAL, - "cmdline_default_variables: %s", cmdline_default_variables) - - # Set up includes. - includes = [] - - # If ~/.gyp/include.gypi exists, it'll be forcibly included into every - # .gyp file that's loaded, before anything else is included. - if home_dot_gyp != None: - default_include = os.path.join(home_dot_gyp, 'include.gypi') - if os.path.exists(default_include): - print('Using overrides found in ' + default_include) - includes.append(default_include) - - # Command-line --include files come after the default include. - if options.includes: - includes.extend(options.includes) - - # Generator flags should be prefixed with the target generator since they - # are global across all generator runs. - gen_flags = [] - if options.use_environment: - gen_flags += ShlexEnv('GYP_GENERATOR_FLAGS') - if options.generator_flags: - gen_flags += options.generator_flags - generator_flags = NameValueListToDict(gen_flags) - if DEBUG_GENERAL in gyp.debug.keys(): - DebugOutput(DEBUG_GENERAL, "generator_flags: %s", generator_flags) - - # Generate all requested formats (use a set in case we got one format request - # twice) - for format in set(options.formats): - params = {'options': options, - 'build_files': build_files, - 'generator_flags': generator_flags, - 'cwd': os.getcwd(), - 'build_files_arg': build_files_arg, - 'gyp_binary': sys.argv[0], - 'home_dot_gyp': home_dot_gyp, - 'parallel': options.parallel, - 'root_targets': options.root_targets, - 'target_arch': cmdline_default_variables.get('target_arch', '')} - - # Start with the default variables from the command line. - [generator, flat_list, targets, data] = Load( - build_files, format, cmdline_default_variables, includes, options.depth, - params, options.check, options.circular_check, - options.duplicate_basename_check) - - # TODO(mark): Pass |data| for now because the generator needs a list of - # build files that came in. In the future, maybe it should just accept - # a list, and not the whole data dict. - # NOTE: flat_list is the flattened dependency graph specifying the order - # that targets may be built. Build systems that operate serially or that - # need to have dependencies defined before dependents reference them should - # generate targets in the order specified in flat_list. - generator.GenerateOutput(flat_list, targets, data, params) - - if options.configs: - valid_configs = targets[flat_list[0]]['configurations'].keys() - for conf in options.configs: - if conf not in valid_configs: - raise GypError('Invalid config specified via --build: %s' % conf) - generator.PerformBuild(data, options.configs, params) - - # Done - return 0 + # Start with the default variables from the command line. + [generator, flat_list, targets, data] = Load( + build_files, + format, + cmdline_default_variables, + includes, + options.depth, + params, + options.check, + options.circular_check, + options.duplicate_basename_check, + ) + + # TODO(mark): Pass |data| for now because the generator needs a list of + # build files that came in. In the future, maybe it should just accept + # a list, and not the whole data dict. + # NOTE: flat_list is the flattened dependency graph specifying the order + # that targets may be built. Build systems that operate serially or that + # need to have dependencies defined before dependents reference them should + # generate targets in the order specified in flat_list. + generator.GenerateOutput(flat_list, targets, data, params) + + if options.configs: + valid_configs = targets[flat_list[0]]["configurations"] + for conf in options.configs: + if conf not in valid_configs: + raise GypError("Invalid config specified via --build: %s" % conf) + generator.PerformBuild(data, options.configs, params) + + # Done + return 0 def main(args): - try: - return gyp_main(args) - except GypError as e: - sys.stderr.write("gyp: %s\n" % e) - return 1 + try: + return gyp_main(args) + except GypError as e: + sys.stderr.write("gyp: %s\n" % e) + return 1 + # NOTE: setuptools generated console_scripts calls function with no arguments def script_main(): - return main(sys.argv[1:]) + return main(sys.argv[1:]) + -if __name__ == '__main__': - sys.exit(script_main()) +if __name__ == "__main__": + sys.exit(script_main()) diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py index aa410e1dfd..3f2329bda4 100644 --- a/gyp/pylib/gyp/common.py +++ b/gyp/pylib/gyp/common.py @@ -11,9 +11,9 @@ import subprocess try: - from collections.abc import MutableSet + from collections.abc import MutableSet except ImportError: - from collections import MutableSet + from collections import MutableSet PY3 = bytes != str @@ -21,191 +21,197 @@ # A minimal memoizing decorator. It'll blow up if the args aren't immutable, # among other "problems". class memoize(object): - def __init__(self, func): - self.func = func - self.cache = {} - def __call__(self, *args): - try: - return self.cache[args] - except KeyError: - result = self.func(*args) - self.cache[args] = result - return result + def __init__(self, func): + self.func = func + self.cache = {} + + def __call__(self, *args): + try: + return self.cache[args] + except KeyError: + result = self.func(*args) + self.cache[args] = result + return result class GypError(Exception): - """Error class representing an error, which is to be presented + """Error class representing an error, which is to be presented to the user. The main entry point will catch and display this. """ - pass + + pass def ExceptionAppend(e, msg): - """Append a message to the given exception's message.""" - if not e.args: - e.args = (msg,) - elif len(e.args) == 1: - e.args = (str(e.args[0]) + ' ' + msg,) - else: - e.args = (str(e.args[0]) + ' ' + msg,) + e.args[1:] + """Append a message to the given exception's message.""" + if not e.args: + e.args = (msg,) + elif len(e.args) == 1: + e.args = (str(e.args[0]) + " " + msg,) + else: + e.args = (str(e.args[0]) + " " + msg,) + e.args[1:] def FindQualifiedTargets(target, qualified_list): - """ + """ Given a list of qualified targets, return the qualified targets for the specified |target|. """ - return [t for t in qualified_list if ParseQualifiedTarget(t)[1] == target] + return [t for t in qualified_list if ParseQualifiedTarget(t)[1] == target] def ParseQualifiedTarget(target): - # Splits a qualified target into a build file, target name and toolset. + # Splits a qualified target into a build file, target name and toolset. - # NOTE: rsplit is used to disambiguate the Windows drive letter separator. - target_split = target.rsplit(':', 1) - if len(target_split) == 2: - [build_file, target] = target_split - else: - build_file = None + # NOTE: rsplit is used to disambiguate the Windows drive letter separator. + target_split = target.rsplit(":", 1) + if len(target_split) == 2: + [build_file, target] = target_split + else: + build_file = None - target_split = target.rsplit('#', 1) - if len(target_split) == 2: - [target, toolset] = target_split - else: - toolset = None + target_split = target.rsplit("#", 1) + if len(target_split) == 2: + [target, toolset] = target_split + else: + toolset = None - return [build_file, target, toolset] + return [build_file, target, toolset] def ResolveTarget(build_file, target, toolset): - # This function resolves a target into a canonical form: - # - a fully defined build file, either absolute or relative to the current - # directory - # - a target name - # - a toolset - # - # build_file is the file relative to which 'target' is defined. - # target is the qualified target. - # toolset is the default toolset for that target. - [parsed_build_file, target, parsed_toolset] = ParseQualifiedTarget(target) - - if parsed_build_file: - if build_file: - # If a relative path, parsed_build_file is relative to the directory - # containing build_file. If build_file is not in the current directory, - # parsed_build_file is not a usable path as-is. Resolve it by - # interpreting it as relative to build_file. If parsed_build_file is - # absolute, it is usable as a path regardless of the current directory, - # and os.path.join will return it as-is. - build_file = os.path.normpath(os.path.join(os.path.dirname(build_file), - parsed_build_file)) - # Further (to handle cases like ../cwd), make it relative to cwd) - if not os.path.isabs(build_file): - build_file = RelativePath(build_file, '.') - else: - build_file = parsed_build_file + # This function resolves a target into a canonical form: + # - a fully defined build file, either absolute or relative to the current + # directory + # - a target name + # - a toolset + # + # build_file is the file relative to which 'target' is defined. + # target is the qualified target. + # toolset is the default toolset for that target. + [parsed_build_file, target, parsed_toolset] = ParseQualifiedTarget(target) + + if parsed_build_file: + if build_file: + # If a relative path, parsed_build_file is relative to the directory + # containing build_file. If build_file is not in the current directory, + # parsed_build_file is not a usable path as-is. Resolve it by + # interpreting it as relative to build_file. If parsed_build_file is + # absolute, it is usable as a path regardless of the current directory, + # and os.path.join will return it as-is. + build_file = os.path.normpath( + os.path.join(os.path.dirname(build_file), parsed_build_file) + ) + # Further (to handle cases like ../cwd), make it relative to cwd) + if not os.path.isabs(build_file): + build_file = RelativePath(build_file, ".") + else: + build_file = parsed_build_file - if parsed_toolset: - toolset = parsed_toolset + if parsed_toolset: + toolset = parsed_toolset - return [build_file, target, toolset] + return [build_file, target, toolset] def BuildFile(fully_qualified_target): - # Extracts the build file from the fully qualified target. - return ParseQualifiedTarget(fully_qualified_target)[0] + # Extracts the build file from the fully qualified target. + return ParseQualifiedTarget(fully_qualified_target)[0] def GetEnvironFallback(var_list, default): - """Look up a key in the environment, with fallback to secondary keys + """Look up a key in the environment, with fallback to secondary keys and finally falling back to a default value.""" - for var in var_list: - if var in os.environ: - return os.environ[var] - return default + for var in var_list: + if var in os.environ: + return os.environ[var] + return default def QualifiedTarget(build_file, target, toolset): - # "Qualified" means the file that a target was defined in and the target - # name, separated by a colon, suffixed by a # and the toolset name: - # /path/to/file.gyp:target_name#toolset - fully_qualified = build_file + ':' + target - if toolset: - fully_qualified = fully_qualified + '#' + toolset - return fully_qualified + # "Qualified" means the file that a target was defined in and the target + # name, separated by a colon, suffixed by a # and the toolset name: + # /path/to/file.gyp:target_name#toolset + fully_qualified = build_file + ":" + target + if toolset: + fully_qualified = fully_qualified + "#" + toolset + return fully_qualified @memoize def RelativePath(path, relative_to, follow_path_symlink=True): - # Assuming both |path| and |relative_to| are relative to the current - # directory, returns a relative path that identifies path relative to - # relative_to. - # If |follow_symlink_path| is true (default) and |path| is a symlink, then - # this method returns a path to the real file represented by |path|. If it is - # false, this method returns a path to the symlink. If |path| is not a - # symlink, this option has no effect. - - # Convert to normalized (and therefore absolute paths). - if follow_path_symlink: - path = os.path.realpath(path) - else: - path = os.path.abspath(path) - relative_to = os.path.realpath(relative_to) - - # On Windows, we can't create a relative path to a different drive, so just - # use the absolute path. - if sys.platform == 'win32': - if (os.path.splitdrive(path)[0].lower() != - os.path.splitdrive(relative_to)[0].lower()): - return path - - # Split the paths into components. - path_split = path.split(os.path.sep) - relative_to_split = relative_to.split(os.path.sep) - - # Determine how much of the prefix the two paths share. - prefix_len = len(os.path.commonprefix([path_split, relative_to_split])) - - # Put enough ".." components to back up out of relative_to to the common - # prefix, and then append the part of path_split after the common prefix. - relative_split = [os.path.pardir] * (len(relative_to_split) - prefix_len) + \ - path_split[prefix_len:] - - if len(relative_split) == 0: - # The paths were the same. - return '' - - # Turn it back into a string and we're done. - return os.path.join(*relative_split) + # Assuming both |path| and |relative_to| are relative to the current + # directory, returns a relative path that identifies path relative to + # relative_to. + # If |follow_symlink_path| is true (default) and |path| is a symlink, then + # this method returns a path to the real file represented by |path|. If it is + # false, this method returns a path to the symlink. If |path| is not a + # symlink, this option has no effect. + + # Convert to normalized (and therefore absolute paths). + if follow_path_symlink: + path = os.path.realpath(path) + else: + path = os.path.abspath(path) + relative_to = os.path.realpath(relative_to) + + # On Windows, we can't create a relative path to a different drive, so just + # use the absolute path. + if sys.platform == "win32": + if ( + os.path.splitdrive(path)[0].lower() + != os.path.splitdrive(relative_to)[0].lower() + ): + return path + + # Split the paths into components. + path_split = path.split(os.path.sep) + relative_to_split = relative_to.split(os.path.sep) + + # Determine how much of the prefix the two paths share. + prefix_len = len(os.path.commonprefix([path_split, relative_to_split])) + + # Put enough ".." components to back up out of relative_to to the common + # prefix, and then append the part of path_split after the common prefix. + relative_split = [os.path.pardir] * ( + len(relative_to_split) - prefix_len + ) + path_split[prefix_len:] + + if len(relative_split) == 0: + # The paths were the same. + return "" + + # Turn it back into a string and we're done. + return os.path.join(*relative_split) @memoize def InvertRelativePath(path, toplevel_dir=None): - """Given a path like foo/bar that is relative to toplevel_dir, return + """Given a path like foo/bar that is relative to toplevel_dir, return the inverse relative path back to the toplevel_dir. E.g. os.path.normpath(os.path.join(path, InvertRelativePath(path))) should always produce the empty string, unless the path contains symlinks. """ - if not path: - return path - toplevel_dir = '.' if toplevel_dir is None else toplevel_dir - return RelativePath(toplevel_dir, os.path.join(toplevel_dir, path)) + if not path: + return path + toplevel_dir = "." if toplevel_dir is None else toplevel_dir + return RelativePath(toplevel_dir, os.path.join(toplevel_dir, path)) def FixIfRelativePath(path, relative_to): - # Like RelativePath but returns |path| unchanged if it is absolute. - if os.path.isabs(path): - return path - return RelativePath(path, relative_to) + # Like RelativePath but returns |path| unchanged if it is absolute. + if os.path.isabs(path): + return path + return RelativePath(path, relative_to) def UnrelativePath(path, relative_to): - # Assuming that |relative_to| is relative to the current directory, and |path| - # is a path relative to the dirname of |relative_to|, returns a path that - # identifies |path| relative to the current directory. - rel_dir = os.path.dirname(relative_to) - return os.path.normpath(os.path.join(rel_dir, path)) + # Assuming that |relative_to| is relative to the current directory, and |path| + # is a path relative to the dirname of |relative_to|, returns a path that + # identifies |path| relative to the current directory. + rel_dir = os.path.dirname(relative_to) + return os.path.normpath(os.path.join(rel_dir, path)) # re objects used by EncodePOSIXShellArgument. See IEEE 1003.1 XCU.2.2 at @@ -234,7 +240,7 @@ def UnrelativePath(path, relative_to): # This does not match the characters in _escape, because those need to be # backslash-escaped regardless of whether they appear in a double-quoted # string. -_quote = re.compile('[\t\n #$%&\'()*;<=>?[{|}~]|^$') +_quote = re.compile("[\t\n #$%&'()*;<=>?[{|}~]|^$") # _escape is a pattern that should match any character that needs to be # escaped with a backslash, whether or not the argument matched the _quote @@ -262,8 +268,9 @@ def UnrelativePath(path, relative_to): # shells, there is no room for error here by ignoring !. _escape = re.compile(r'(["\\`])') + def EncodePOSIXShellArgument(argument): - """Encodes |argument| suitably for consumption by POSIX shells. + """Encodes |argument| suitably for consumption by POSIX shells. argument may be quoted and escaped as necessary to ensure that POSIX shells treat the returned value as a literal representing the argument passed to @@ -272,67 +279,67 @@ def EncodePOSIXShellArgument(argument): references to variables to be expanded by the shell. """ - if not isinstance(argument, str): - argument = str(argument) + if not isinstance(argument, str): + argument = str(argument) - if _quote.search(argument): - quote = '"' - else: - quote = '' + if _quote.search(argument): + quote = '"' + else: + quote = "" - encoded = quote + re.sub(_escape, r'\\\1', argument) + quote + encoded = quote + re.sub(_escape, r"\\\1", argument) + quote - return encoded + return encoded def EncodePOSIXShellList(list): - """Encodes |list| suitably for consumption by POSIX shells. + """Encodes |list| suitably for consumption by POSIX shells. Returns EncodePOSIXShellArgument for each item in list, and joins them together using the space character as an argument separator. """ - encoded_arguments = [] - for argument in list: - encoded_arguments.append(EncodePOSIXShellArgument(argument)) - return ' '.join(encoded_arguments) + encoded_arguments = [] + for argument in list: + encoded_arguments.append(EncodePOSIXShellArgument(argument)) + return " ".join(encoded_arguments) def DeepDependencyTargets(target_dicts, roots): - """Returns the recursive list of target dependencies.""" - dependencies = set() - pending = set(roots) - while pending: - # Pluck out one. - r = pending.pop() - # Skip if visited already. - if r in dependencies: - continue - # Add it. - dependencies.add(r) - # Add its children. - spec = target_dicts[r] - pending.update(set(spec.get('dependencies', []))) - pending.update(set(spec.get('dependencies_original', []))) - return list(dependencies - set(roots)) + """Returns the recursive list of target dependencies.""" + dependencies = set() + pending = set(roots) + while pending: + # Pluck out one. + r = pending.pop() + # Skip if visited already. + if r in dependencies: + continue + # Add it. + dependencies.add(r) + # Add its children. + spec = target_dicts[r] + pending.update(set(spec.get("dependencies", []))) + pending.update(set(spec.get("dependencies_original", []))) + return list(dependencies - set(roots)) def BuildFileTargets(target_list, build_file): - """From a target_list, returns the subset from the specified build_file. + """From a target_list, returns the subset from the specified build_file. """ - return [p for p in target_list if BuildFile(p) == build_file] + return [p for p in target_list if BuildFile(p) == build_file] def AllTargets(target_list, target_dicts, build_file): - """Returns all targets (direct and dependencies) for the specified build_file. + """Returns all targets (direct and dependencies) for the specified build_file. """ - bftargets = BuildFileTargets(target_list, build_file) - deptargets = DeepDependencyTargets(target_dicts, bftargets) - return bftargets + deptargets + bftargets = BuildFileTargets(target_list, build_file) + deptargets = DeepDependencyTargets(target_dicts, bftargets) + return bftargets + deptargets def WriteOnDiff(filename): - """Write to a file only if the new contents differ. + """Write to a file only if the new contents differ. Arguments: filename: name of the file to potentially write to. @@ -341,148 +348,146 @@ def WriteOnDiff(filename): the target if it differs (on close). """ - class Writer(object): - """Wrapper around file which only covers the target if it differs.""" - def __init__(self): - # On Cygwin remove the "dir" argument because `C:` prefixed paths are treated as relative, - # consequently ending up with current dir "/cygdrive/c/..." being prefixed to those, which was - # obviously a non-existent path, for example: "/cygdrive/c//C:\". - # See https://docs.python.org/2/library/tempfile.html#tempfile.mkstemp for more details - base_temp_dir = "" if IsCygwin() else os.path.dirname(filename) - # Pick temporary file. - tmp_fd, self.tmp_path = tempfile.mkstemp( - suffix='.tmp', - prefix=os.path.split(filename)[1] + '.gyp.', - dir=base_temp_dir) - try: - self.tmp_file = os.fdopen(tmp_fd, 'wb') - except Exception: - # Don't leave turds behind. - os.unlink(self.tmp_path) - raise - - def __getattr__(self, attrname): - # Delegate everything else to self.tmp_file - return getattr(self.tmp_file, attrname) - - def close(self): - try: - # Close tmp file. - self.tmp_file.close() - # Determine if different. - same = False - try: - same = filecmp.cmp(self.tmp_path, filename, False) - except OSError as e: - if e.errno != errno.ENOENT: - raise - - if same: - # The new file is identical to the old one, just get rid of the new - # one. - os.unlink(self.tmp_path) - else: - # The new file is different from the old one, or there is no old one. - # Rename the new file to the permanent name. - # - # tempfile.mkstemp uses an overly restrictive mode, resulting in a - # file that can only be read by the owner, regardless of the umask. - # There's no reason to not respect the umask here, which means that - # an extra hoop is required to fetch it and reset the new file's mode. - # - # No way to get the umask without setting a new one? Set a safe one - # and then set it back to the old value. - umask = os.umask(0o77) - os.umask(umask) - os.chmod(self.tmp_path, 0o666 & ~umask) - if sys.platform == 'win32' and os.path.exists(filename): - # NOTE: on windows (but not cygwin) rename will not replace an - # existing file, so it must be preceded with a remove. Sadly there - # is no way to make the switch atomic. - os.remove(filename) - os.rename(self.tmp_path, filename) - except Exception: - # Don't leave turds behind. - os.unlink(self.tmp_path) - raise - - def write(self, s): - self.tmp_file.write(s.encode('utf-8')) - - return Writer() + class Writer(object): + """Wrapper around file which only covers the target if it differs.""" + + def __init__(self): + # On Cygwin remove the "dir" argument because `C:` prefixed paths are treated as relative, + # consequently ending up with current dir "/cygdrive/c/..." being prefixed to those, which was + # obviously a non-existent path, for example: "/cygdrive/c//C:\". + # See https://docs.python.org/2/library/tempfile.html#tempfile.mkstemp for more details + base_temp_dir = "" if IsCygwin() else os.path.dirname(filename) + # Pick temporary file. + tmp_fd, self.tmp_path = tempfile.mkstemp( + suffix=".tmp", + prefix=os.path.split(filename)[1] + ".gyp.", + dir=base_temp_dir, + ) + try: + self.tmp_file = os.fdopen(tmp_fd, "w") + except Exception: + # Don't leave turds behind. + os.unlink(self.tmp_path) + raise + + def __getattr__(self, attrname): + # Delegate everything else to self.tmp_file + return getattr(self.tmp_file, attrname) + + def close(self): + try: + # Close tmp file. + self.tmp_file.close() + # Determine if different. + same = False + try: + same = filecmp.cmp(self.tmp_path, filename, False) + except OSError as e: + if e.errno != errno.ENOENT: + raise + + if same: + # The new file is identical to the old one, just get rid of the new + # one. + os.unlink(self.tmp_path) + else: + # The new file is different from the old one, or there is no old one. + # Rename the new file to the permanent name. + # + # tempfile.mkstemp uses an overly restrictive mode, resulting in a + # file that can only be read by the owner, regardless of the umask. + # There's no reason to not respect the umask here, which means that + # an extra hoop is required to fetch it and reset the new file's mode. + # + # No way to get the umask without setting a new one? Set a safe one + # and then set it back to the old value. + umask = os.umask(0o77) + os.umask(umask) + os.chmod(self.tmp_path, 0o666 & ~umask) + if sys.platform == "win32" and os.path.exists(filename): + # NOTE: on windows (but not cygwin) rename will not replace an + # existing file, so it must be preceded with a remove. Sadly there + # is no way to make the switch atomic. + os.remove(filename) + os.rename(self.tmp_path, filename) + except Exception: + # Don't leave turds behind. + os.unlink(self.tmp_path) + raise + + def write(self, s): + self.tmp_file.write(s.encode("utf-8")) + + return Writer() def EnsureDirExists(path): - """Make sure the directory for |path| exists.""" - try: - os.makedirs(os.path.dirname(path)) - except OSError: - pass + """Make sure the directory for |path| exists.""" + try: + os.makedirs(os.path.dirname(path)) + except OSError: + pass def GetFlavor(params): - """Returns |params.flavor| if it's set, the system's default flavor else.""" - flavors = { - 'cygwin': 'win', - 'win32': 'win', - 'darwin': 'mac', - } - - if 'flavor' in params: - return params['flavor'] - if sys.platform in flavors: - return flavors[sys.platform] - if sys.platform.startswith('sunos'): - return 'solaris' - if sys.platform.startswith(('dragonfly', 'freebsd')): - return 'freebsd' - if sys.platform.startswith('openbsd'): - return 'openbsd' - if sys.platform.startswith('netbsd'): - return 'netbsd' - if sys.platform.startswith('aix'): - return 'aix' - if sys.platform.startswith(('os390', 'zos')): - return 'zos' - - return 'linux' + """Returns |params.flavor| if it's set, the system's default flavor else.""" + flavors = { + "cygwin": "win", + "win32": "win", + "darwin": "mac", + } + + if "flavor" in params: + return params["flavor"] + if sys.platform in flavors: + return flavors[sys.platform] + if sys.platform.startswith("sunos"): + return "solaris" + if sys.platform.startswith(("dragonfly", "freebsd")): + return "freebsd" + if sys.platform.startswith("openbsd"): + return "openbsd" + if sys.platform.startswith("netbsd"): + return "netbsd" + if sys.platform.startswith("aix"): + return "aix" + if sys.platform.startswith(("os390", "zos")): + return "zos" + + return "linux" def CopyTool(flavor, out_path, generator_flags={}): - """Finds (flock|mac|win)_tool.gyp in the gyp directory and copies it + """Finds (flock|mac|win)_tool.gyp in the gyp directory and copies it to |out_path|.""" - # aix and solaris just need flock emulation. mac and win use more complicated - # support scripts. - prefix = { - 'aix': 'flock', - 'solaris': 'flock', - 'mac': 'mac', - 'win': 'win' - }.get(flavor, None) - if not prefix: - return - - # Slurp input file. - source_path = os.path.join( - os.path.dirname(os.path.abspath(__file__)), '%s_tool.py' % prefix) - with open(source_path) as source_file: - source = source_file.readlines() - - # Set custom header flags. - header = '# Generated by gyp. Do not edit.\n' - mac_toolchain_dir = generator_flags.get('mac_toolchain_dir', None) - if flavor == 'mac' and mac_toolchain_dir: - header += "import os;\nos.environ['DEVELOPER_DIR']='%s'\n" \ - % mac_toolchain_dir - - # Add header and write it out. - tool_path = os.path.join(out_path, 'gyp-%s-tool' % prefix) - with open(tool_path, 'w') as tool_file: - tool_file.write( - ''.join([source[0], header] + source[1:])) - - # Make file executable. - os.chmod(tool_path, 0o755) + # aix and solaris just need flock emulation. mac and win use more complicated + # support scripts. + prefix = {"aix": "flock", "solaris": "flock", "mac": "mac", "win": "win"}.get( + flavor, None + ) + if not prefix: + return + + # Slurp input file. + source_path = os.path.join( + os.path.dirname(os.path.abspath(__file__)), "%s_tool.py" % prefix + ) + with open(source_path) as source_file: + source = source_file.readlines() + + # Set custom header flags. + header = "# Generated by gyp. Do not edit.\n" + mac_toolchain_dir = generator_flags.get("mac_toolchain_dir", None) + if flavor == "mac" and mac_toolchain_dir: + header += "import os;\nos.environ['DEVELOPER_DIR']='%s'\n" % mac_toolchain_dir + + # Add header and write it out. + tool_path = os.path.join(out_path, "gyp-%s-tool" % prefix) + with open(tool_path, "w") as tool_file: + tool_file.write("".join([source[0], header] + source[1:])) + + # Make file executable. + os.chmod(tool_path, 0o755) # From Alex Martelli, @@ -491,14 +496,14 @@ def CopyTool(flavor, out_path, generator_flags={}): # First comment, dated 2001/10/13. # (Also in the printed Python Cookbook.) -def uniquer(seq, idfun=None): - if idfun is None: - idfun = lambda x: x + +def uniquer(seq, idfun=lambda x: x): seen = {} result = [] for item in seq: marker = idfun(item) - if marker in seen: continue + if marker in seen: + continue seen[marker] = 1 result.append(item) return result @@ -506,80 +511,82 @@ def uniquer(seq, idfun=None): # Based on http://code.activestate.com/recipes/576694/. class OrderedSet(MutableSet): - def __init__(self, iterable=None): - self.end = end = [] - end += [None, end, end] # sentinel node for doubly linked list - self.map = {} # key --> [key, prev, next] - if iterable is not None: - self |= iterable - - def __len__(self): - return len(self.map) - - def __contains__(self, key): - return key in self.map - - def add(self, key): - if key not in self.map: - end = self.end - curr = end[1] - curr[2] = end[1] = self.map[key] = [key, curr, end] - - def discard(self, key): - if key in self.map: - key, prev_item, next_item = self.map.pop(key) - prev_item[2] = next_item - next_item[1] = prev_item - - def __iter__(self): - end = self.end - curr = end[2] - while curr is not end: - yield curr[0] - curr = curr[2] - - def __reversed__(self): - end = self.end - curr = end[1] - while curr is not end: - yield curr[0] - curr = curr[1] - - # The second argument is an addition that causes a pylint warning. - def pop(self, last=True): # pylint: disable=W0221 - if not self: - raise KeyError('set is empty') - key = self.end[1][0] if last else self.end[2][0] - self.discard(key) - return key - - def __repr__(self): - if not self: - return '%s()' % (self.__class__.__name__,) - return '%s(%r)' % (self.__class__.__name__, list(self)) - - def __eq__(self, other): - if isinstance(other, OrderedSet): - return len(self) == len(other) and list(self) == list(other) - return set(self) == set(other) - - # Extensions to the recipe. - def update(self, iterable): - for i in iterable: - if i not in self: - self.add(i) + def __init__(self, iterable=None): + self.end = end = [] + end += [None, end, end] # sentinel node for doubly linked list + self.map = {} # key --> [key, prev, next] + if iterable is not None: + self |= iterable + + def __len__(self): + return len(self.map) + + def __contains__(self, key): + return key in self.map + + def add(self, key): + if key not in self.map: + end = self.end + curr = end[1] + curr[2] = end[1] = self.map[key] = [key, curr, end] + + def discard(self, key): + if key in self.map: + key, prev_item, next_item = self.map.pop(key) + prev_item[2] = next_item + next_item[1] = prev_item + + def __iter__(self): + end = self.end + curr = end[2] + while curr is not end: + yield curr[0] + curr = curr[2] + + def __reversed__(self): + end = self.end + curr = end[1] + while curr is not end: + yield curr[0] + curr = curr[1] + + # The second argument is an addition that causes a pylint warning. + def pop(self, last=True): # pylint: disable=W0221 + if not self: + raise KeyError("set is empty") + key = self.end[1][0] if last else self.end[2][0] + self.discard(key) + return key + + def __repr__(self): + if not self: + return "%s()" % (self.__class__.__name__,) + return "%s(%r)" % (self.__class__.__name__, list(self)) + + def __eq__(self, other): + if isinstance(other, OrderedSet): + return len(self) == len(other) and list(self) == list(other) + return set(self) == set(other) + + # Extensions to the recipe. + def update(self, iterable): + for i in iterable: + if i not in self: + self.add(i) class CycleError(Exception): - """An exception raised when an unexpected cycle is detected.""" - def __init__(self, nodes): - self.nodes = nodes - def __str__(self): - return 'CycleError: cycle involving: ' + str(self.nodes) + """An exception raised when an unexpected cycle is detected.""" + + def __init__(self, nodes): + self.nodes = nodes + + def __str__(self): + return "CycleError: cycle involving: " + str(self.nodes) def TopologicallySorted(graph, get_edges): - r"""Topologically sort based on a user provided edge definition. + r"""Topologically sort based on a user provided edge definition. Args: graph: A list of node names. @@ -599,44 +606,50 @@ def GetEdges(node): ==> ['a', 'c', b'] """ - get_edges = memoize(get_edges) - visited = set() - visiting = set() - ordered_nodes = [] - def Visit(node): - if node in visiting: - raise CycleError(visiting) - if node in visited: - return - visited.add(node) - visiting.add(node) - for neighbor in get_edges(node): - Visit(neighbor) - visiting.remove(node) - ordered_nodes.insert(0, node) - for node in sorted(graph): - Visit(node) - return ordered_nodes + get_edges = memoize(get_edges) + visited = set() + visiting = set() + ordered_nodes = [] + + def Visit(node): + if node in visiting: + raise CycleError(visiting) + if node in visited: + return + visited.add(node) + visiting.add(node) + for neighbor in get_edges(node): + Visit(neighbor) + visiting.remove(node) + ordered_nodes.insert(0, node) + + for node in sorted(graph): + Visit(node) + return ordered_nodes + def CrossCompileRequested(): - # TODO: figure out how to not build extra host objects in the - # non-cross-compile case when this is enabled, and enable unconditionally. - return (os.environ.get('GYP_CROSSCOMPILE') or - os.environ.get('AR_host') or - os.environ.get('CC_host') or - os.environ.get('CXX_host') or - os.environ.get('AR_target') or - os.environ.get('CC_target') or - os.environ.get('CXX_target')) + # TODO: figure out how to not build extra host objects in the + # non-cross-compile case when this is enabled, and enable unconditionally. + return ( + os.environ.get("GYP_CROSSCOMPILE") + or os.environ.get("AR_host") + or os.environ.get("CC_host") + or os.environ.get("CXX_host") + or os.environ.get("AR_target") + or os.environ.get("CC_target") + or os.environ.get("CXX_target") + ) + def IsCygwin(): - try: - out = subprocess.Popen("uname", - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) - stdout, stderr = out.communicate() - if PY3: - stdout = stdout.decode("utf-8") - return "CYGWIN" in str(stdout) - except Exception: - return False + try: + out = subprocess.Popen( + "uname", stdout=subprocess.PIPE, stderr=subprocess.STDOUT + ) + stdout, stderr = out.communicate() + if PY3: + stdout = stdout.decode("utf-8") + return "CYGWIN" in str(stdout) + except Exception: + return False diff --git a/gyp/pylib/gyp/common_test.py b/gyp/pylib/gyp/common_test.py index b75bbb8412..0310fb266c 100755 --- a/gyp/pylib/gyp/common_test.py +++ b/gyp/pylib/gyp/common_test.py @@ -12,61 +12,67 @@ class TestTopologicallySorted(unittest.TestCase): - def test_Valid(self): - """Test that sorting works on a valid graph with one possible order.""" - graph = { - 'a': ['b', 'c'], - 'b': [], - 'c': ['d'], - 'd': ['b'], + def test_Valid(self): + """Test that sorting works on a valid graph with one possible order.""" + graph = { + "a": ["b", "c"], + "b": [], + "c": ["d"], + "d": ["b"], } - def GetEdge(node): - return tuple(graph[node]) - self.assertEqual( - gyp.common.TopologicallySorted(graph.keys(), GetEdge), - ['a', 'c', 'd', 'b']) - - def test_Cycle(self): - """Test that an exception is thrown on a cyclic graph.""" - graph = { - 'a': ['b'], - 'b': ['c'], - 'c': ['d'], - 'd': ['a'], + + def GetEdge(node): + return tuple(graph[node]) + + self.assertEqual( + gyp.common.TopologicallySorted(graph.keys(), GetEdge), ["a", "c", "d", "b"] + ) + + def test_Cycle(self): + """Test that an exception is thrown on a cyclic graph.""" + graph = { + "a": ["b"], + "b": ["c"], + "c": ["d"], + "d": ["a"], } - def GetEdge(node): - return tuple(graph[node]) - self.assertRaises( - gyp.common.CycleError, gyp.common.TopologicallySorted, - graph.keys(), GetEdge) + + def GetEdge(node): + return tuple(graph[node]) + + self.assertRaises( + gyp.common.CycleError, gyp.common.TopologicallySorted, graph.keys(), GetEdge + ) class TestGetFlavor(unittest.TestCase): - """Test that gyp.common.GetFlavor works as intended""" - original_platform = '' + """Test that gyp.common.GetFlavor works as intended""" + + original_platform = "" - def setUp(self): - self.original_platform = sys.platform + def setUp(self): + self.original_platform = sys.platform - def tearDown(self): - sys.platform = self.original_platform + def tearDown(self): + sys.platform = self.original_platform - def assertFlavor(self, expected, argument, param): - sys.platform = argument - self.assertEqual(expected, gyp.common.GetFlavor(param)) + def assertFlavor(self, expected, argument, param): + sys.platform = argument + self.assertEqual(expected, gyp.common.GetFlavor(param)) - def test_platform_default(self): - self.assertFlavor('freebsd', 'freebsd9' , {}) - self.assertFlavor('freebsd', 'freebsd10', {}) - self.assertFlavor('openbsd', 'openbsd5' , {}) - self.assertFlavor('solaris', 'sunos5' , {}) - self.assertFlavor('solaris', 'sunos' , {}) - self.assertFlavor('linux' , 'linux2' , {}) - self.assertFlavor('linux' , 'linux3' , {}) + def test_platform_default(self): + self.assertFlavor("freebsd", "freebsd9", {}) + self.assertFlavor("freebsd", "freebsd10", {}) + self.assertFlavor("openbsd", "openbsd5", {}) + self.assertFlavor("solaris", "sunos5", {}) + self.assertFlavor("solaris", "sunos", {}) + self.assertFlavor("linux", "linux2", {}) + self.assertFlavor("linux", "linux3", {}) + self.assertFlavor("linux", "linux", {}) - def test_param(self): - self.assertFlavor('foobar', 'linux2' , {'flavor': 'foobar'}) + def test_param(self): + self.assertFlavor("foobar", "linux2", {"flavor": "foobar"}) -if __name__ == '__main__': - unittest.main() +if __name__ == "__main__": + unittest.main() diff --git a/gyp/pylib/gyp/easy_xml.py b/gyp/pylib/gyp/easy_xml.py index 86d0ba6c0c..e0628ef4d8 100644 --- a/gyp/pylib/gyp/easy_xml.py +++ b/gyp/pylib/gyp/easy_xml.py @@ -8,8 +8,8 @@ from functools import reduce -def XmlToString(content, encoding='utf-8', pretty=False): - """ Writes the XML content to disk, touching the file only if it has changed. +def XmlToString(content, encoding="utf-8", pretty=False): + """ Writes the XML content to disk, touching the file only if it has changed. Visual Studio files have a lot of pre-defined structures. This function makes it easy to represent these structures as Python data structures, instead of @@ -46,18 +46,18 @@ def XmlToString(content, encoding='utf-8', pretty=False): Returns: The XML content as a string. """ - # We create a huge list of all the elements of the file. - xml_parts = ['' % encoding] - if pretty: - xml_parts.append('\n') - _ConstructContentList(xml_parts, content, pretty) + # We create a huge list of all the elements of the file. + xml_parts = ['' % encoding] + if pretty: + xml_parts.append("\n") + _ConstructContentList(xml_parts, content, pretty) - # Convert it to a string - return ''.join(xml_parts) + # Convert it to a string + return "".join(xml_parts) def _ConstructContentList(xml_parts, specification, pretty, level=0): - """ Appends the XML parts corresponding to the specification. + """ Appends the XML parts corresponding to the specification. Args: xml_parts: A list of XML parts to be appended to. @@ -65,48 +65,49 @@ def _ConstructContentList(xml_parts, specification, pretty, level=0): pretty: True if we want pretty printing with indents and new lines. level: Indentation level. """ - # The first item in a specification is the name of the element. - if pretty: - indentation = ' ' * level - new_line = '\n' - else: - indentation = '' - new_line = '' - name = specification[0] - if not isinstance(name, str): - raise Exception('The first item of an EasyXml specification should be ' - 'a string. Specification was ' + str(specification)) - xml_parts.append(indentation + '<' + name) - - # Optionally in second position is a dictionary of the attributes. - rest = specification[1:] - if rest and isinstance(rest[0], dict): - for at, val in sorted(rest[0].items()): - xml_parts.append(' %s="%s"' % (at, _XmlEscape(val, attr=True))) - rest = rest[1:] - if rest: - xml_parts.append('>') - all_strings = reduce(lambda x, y: x and isinstance(y, str), rest, True) - multi_line = not all_strings - if multi_line and new_line: - xml_parts.append(new_line) - for child_spec in rest: - # If it's a string, append a text node. - # Otherwise recurse over that child definition - if isinstance(child_spec, str): - xml_parts.append(_XmlEscape(child_spec)) - else: - _ConstructContentList(xml_parts, child_spec, pretty, level + 1) - if multi_line and indentation: - xml_parts.append(indentation) - xml_parts.append('%s' % (name, new_line)) - else: - xml_parts.append('/>%s' % new_line) - - -def WriteXmlIfChanged(content, path, encoding='utf-8', pretty=False, - win32=False): - """ Writes the XML content to disk, touching the file only if it has changed. + # The first item in a specification is the name of the element. + if pretty: + indentation = " " * level + new_line = "\n" + else: + indentation = "" + new_line = "" + name = specification[0] + if not isinstance(name, str): + raise Exception( + "The first item of an EasyXml specification should be " + "a string. Specification was " + str(specification) + ) + xml_parts.append(indentation + "<" + name) + + # Optionally in second position is a dictionary of the attributes. + rest = specification[1:] + if rest and isinstance(rest[0], dict): + for at, val in sorted(rest[0].items()): + xml_parts.append(' %s="%s"' % (at, _XmlEscape(val, attr=True))) + rest = rest[1:] + if rest: + xml_parts.append(">") + all_strings = reduce(lambda x, y: x and isinstance(y, str), rest, True) + multi_line = not all_strings + if multi_line and new_line: + xml_parts.append(new_line) + for child_spec in rest: + # If it's a string, append a text node. + # Otherwise recurse over that child definition + if isinstance(child_spec, str): + xml_parts.append(_XmlEscape(child_spec)) + else: + _ConstructContentList(xml_parts, child_spec, pretty, level + 1) + if multi_line and indentation: + xml_parts.append(indentation) + xml_parts.append("%s" % (name, new_line)) + else: + xml_parts.append("/>%s" % new_line) + + +def WriteXmlIfChanged(content, path, encoding="utf-8", pretty=False, win32=False): + """ Writes the XML content to disk, touching the file only if it has changed. Args: content: The structured content to be written. @@ -114,50 +115,49 @@ def WriteXmlIfChanged(content, path, encoding='utf-8', pretty=False, encoding: The encoding to report on the first line of the XML file. pretty: True if we want pretty printing with indents and new lines. """ - xml_string = XmlToString(content, encoding, pretty) - if win32 and os.linesep != '\r\n': - xml_string = xml_string.replace('\n', '\r\n') + xml_string = XmlToString(content, encoding, pretty) + if win32 and os.linesep != "\r\n": + xml_string = xml_string.replace("\n", "\r\n") - default_encoding = locale.getdefaultlocale()[1] - if default_encoding and default_encoding.upper() != encoding.upper(): - xml_string = xml_string.encode(encoding) + default_encoding = locale.getdefaultlocale()[1] + if default_encoding and default_encoding.upper() != encoding.upper(): + xml_string = xml_string.encode(encoding) - # Get the old content - try: - f = open(path, 'r') - existing = f.read() - f.close() - except: - existing = None + # Get the old content + try: + with open(path, "r") as file: + existing = file.read() + except IOError: + existing = None - # It has changed, write it - if existing != xml_string: - f = open(path, 'wb') - f.write(xml_string) - f.close() + # It has changed, write it + if existing != xml_string: + with open(path, "wb") as file: + file.write(xml_string) _xml_escape_map = { - '"': '"', - "'": ''', - '<': '<', - '>': '>', - '&': '&', - '\n': ' ', - '\r': ' ', + '"': """, + "'": "'", + "<": "<", + ">": ">", + "&": "&", + "\n": " ", + "\r": " ", } -_xml_escape_re = re.compile( - "(%s)" % "|".join(map(re.escape, _xml_escape_map.keys()))) +_xml_escape_re = re.compile("(%s)" % "|".join(map(re.escape, _xml_escape_map.keys()))) def _XmlEscape(value, attr=False): - """ Escape a string for inclusion in XML.""" - def replace(match): - m = match.string[match.start() : match.end()] - # don't replace single quotes in attrs - if attr and m == "'": - return m - return _xml_escape_map[m] - return _xml_escape_re.sub(replace, value) + """ Escape a string for inclusion in XML.""" + + def replace(match): + m = match.string[match.start() : match.end()] + # don't replace single quotes in attrs + if attr and m == "'": + return m + return _xml_escape_map[m] + + return _xml_escape_re.sub(replace, value) diff --git a/gyp/pylib/gyp/easy_xml_test.py b/gyp/pylib/gyp/easy_xml_test.py index 664b538a58..5bc795ddbe 100755 --- a/gyp/pylib/gyp/easy_xml_test.py +++ b/gyp/pylib/gyp/easy_xml_test.py @@ -10,98 +10,103 @@ import unittest try: - from StringIO import StringIO # Python 2 + from StringIO import StringIO # Python 2 except ImportError: - from io import StringIO # Python 3 + from io import StringIO # Python 3 class TestSequenceFunctions(unittest.TestCase): - - def setUp(self): - self.stderr = StringIO() - - def test_EasyXml_simple(self): - self.assertEqual( - easy_xml.XmlToString(['test']), - '') - - self.assertEqual( - easy_xml.XmlToString(['test'], encoding='Windows-1252'), - '') - - def test_EasyXml_simple_with_attributes(self): - self.assertEqual( - easy_xml.XmlToString(['test2', {'a': 'value1', 'b': 'value2'}]), - '') - - def test_EasyXml_escaping(self): - original = '\'"\r&\nfoo' - converted = '<test>\'" & foo' - converted_apos = converted.replace("'", ''') - self.assertEqual( - easy_xml.XmlToString(['test3', {'a': original}, original]), - '%s' % - (converted, converted_apos)) - - def test_EasyXml_pretty(self): - self.assertEqual( - easy_xml.XmlToString( - ['test3', - ['GrandParent', - ['Parent1', - ['Child'] - ], - ['Parent2'] + def setUp(self): + self.stderr = StringIO() + + def test_EasyXml_simple(self): + self.assertEqual( + easy_xml.XmlToString(["test"]), + '', + ) + + self.assertEqual( + easy_xml.XmlToString(["test"], encoding="Windows-1252"), + '', + ) + + def test_EasyXml_simple_with_attributes(self): + self.assertEqual( + easy_xml.XmlToString(["test2", {"a": "value1", "b": "value2"}]), + '', + ) + + def test_EasyXml_escaping(self): + original = "'\"\r&\nfoo" + converted = "<test>'" & foo" + converted_apos = converted.replace("'", "'") + self.assertEqual( + easy_xml.XmlToString(["test3", {"a": original}, original]), + '%s' + % (converted, converted_apos), + ) + + def test_EasyXml_pretty(self): + self.assertEqual( + easy_xml.XmlToString( + ["test3", ["GrandParent", ["Parent1", ["Child"]], ["Parent2"]]], + pretty=True, + ), + '\n' + "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + "\n", + ) + + def test_EasyXml_complex(self): + # We want to create: + target = ( + '' + "" + '' + "{D2250C20-3A94-4FB9-AF73-11BC5B73884B}" + "Win32Proj" + "automated_ui_tests" + "" + '' + "' + "Application" + "Unicode" + "" + "" + ) + + xml = easy_xml.XmlToString( + [ + "Project", + [ + "PropertyGroup", + {"Label": "Globals"}, + ["ProjectGuid", "{D2250C20-3A94-4FB9-AF73-11BC5B73884B}"], + ["Keyword", "Win32Proj"], + ["RootNamespace", "automated_ui_tests"], + ], + ["Import", {"Project": "$(VCTargetsPath)\\Microsoft.Cpp.props"}], + [ + "PropertyGroup", + { + "Condition": "'$(Configuration)|$(Platform)'=='Debug|Win32'", + "Label": "Configuration", + }, + ["ConfigurationType", "Application"], + ["CharacterSet", "Unicode"], + ], ] - ], - pretty=True), - '\n' - '\n' - ' \n' - ' \n' - ' \n' - ' \n' - ' \n' - ' \n' - '\n') - - - def test_EasyXml_complex(self): - # We want to create: - target = ( - '' - '' - '' - '{D2250C20-3A94-4FB9-AF73-11BC5B73884B}' - 'Win32Proj' - 'automated_ui_tests' - '' - '' - '' - 'Application' - 'Unicode' - '' - '') - - xml = easy_xml.XmlToString( - ['Project', - ['PropertyGroup', {'Label': 'Globals'}, - ['ProjectGuid', '{D2250C20-3A94-4FB9-AF73-11BC5B73884B}'], - ['Keyword', 'Win32Proj'], - ['RootNamespace', 'automated_ui_tests'] - ], - ['Import', {'Project': '$(VCTargetsPath)\\Microsoft.Cpp.props'}], - ['PropertyGroup', - {'Condition': "'$(Configuration)|$(Platform)'=='Debug|Win32'", - 'Label': 'Configuration'}, - ['ConfigurationType', 'Application'], - ['CharacterSet', 'Unicode'] - ] - ]) - self.assertEqual(xml, target) + ) + self.assertEqual(xml, target) -if __name__ == '__main__': - unittest.main() +if __name__ == "__main__": + unittest.main() diff --git a/gyp/pylib/gyp/flock_tool.py b/gyp/pylib/gyp/flock_tool.py index 81fb79d136..f9f89e520a 100755 --- a/gyp/pylib/gyp/flock_tool.py +++ b/gyp/pylib/gyp/flock_tool.py @@ -14,41 +14,42 @@ def main(args): - executor = FlockTool() - executor.Dispatch(args) + executor = FlockTool() + executor.Dispatch(args) class FlockTool(object): - """This class emulates the 'flock' command.""" - def Dispatch(self, args): - """Dispatches a string command to a method.""" - if len(args) < 1: - raise Exception("Not enough arguments") - - method = "Exec%s" % self._CommandifyName(args[0]) - getattr(self, method)(*args[1:]) - - def _CommandifyName(self, name_string): - """Transforms a tool name like copy-info-plist to CopyInfoPlist""" - return name_string.title().replace('-', '') - - def ExecFlock(self, lockfile, *cmd_list): - """Emulates the most basic behavior of Linux's flock(1).""" - # Rely on exception handling to report errors. - # Note that the stock python on SunOS has a bug - # where fcntl.flock(fd, LOCK_EX) always fails - # with EBADF, that's why we use this F_SETLK - # hack instead. - fd = os.open(lockfile, os.O_WRONLY|os.O_NOCTTY|os.O_CREAT, 0o666) - if sys.platform.startswith('aix'): - # Python on AIX is compiled with LARGEFILE support, which changes the - # struct size. - op = struct.pack('hhIllqq', fcntl.F_WRLCK, 0, 0, 0, 0, 0, 0) - else: - op = struct.pack('hhllhhl', fcntl.F_WRLCK, 0, 0, 0, 0, 0, 0) - fcntl.fcntl(fd, fcntl.F_SETLK, op) - return subprocess.call(cmd_list) - - -if __name__ == '__main__': - sys.exit(main(sys.argv[1:])) + """This class emulates the 'flock' command.""" + + def Dispatch(self, args): + """Dispatches a string command to a method.""" + if len(args) < 1: + raise Exception("Not enough arguments") + + method = "Exec%s" % self._CommandifyName(args[0]) + getattr(self, method)(*args[1:]) + + def _CommandifyName(self, name_string): + """Transforms a tool name like copy-info-plist to CopyInfoPlist""" + return name_string.title().replace("-", "") + + def ExecFlock(self, lockfile, *cmd_list): + """Emulates the most basic behavior of Linux's flock(1).""" + # Rely on exception handling to report errors. + # Note that the stock python on SunOS has a bug + # where fcntl.flock(fd, LOCK_EX) always fails + # with EBADF, that's why we use this F_SETLK + # hack instead. + fd = os.open(lockfile, os.O_WRONLY | os.O_NOCTTY | os.O_CREAT, 0o666) + if sys.platform.startswith("aix"): + # Python on AIX is compiled with LARGEFILE support, which changes the + # struct size. + op = struct.pack("hhIllqq", fcntl.F_WRLCK, 0, 0, 0, 0, 0, 0) + else: + op = struct.pack("hhllhhl", fcntl.F_WRLCK, 0, 0, 0, 0, 0, 0) + fcntl.fcntl(fd, fcntl.F_SETLK, op) + return subprocess.call(cmd_list) + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/gyp/pylib/gyp/generator/analyzer.py b/gyp/pylib/gyp/generator/analyzer.py index 59d73dbedb..7a393c1f93 100644 --- a/gyp/pylib/gyp/generator/analyzer.py +++ b/gyp/pylib/gyp/generator/analyzer.py @@ -65,18 +65,16 @@ from __future__ import print_function import gyp.common -import gyp.ninja_syntax as ninja_syntax import json import os import posixpath -import sys debug = False -found_dependency_string = 'Found dependency' -no_dependency_string = 'No dependencies' +found_dependency_string = "Found dependency" +no_dependency_string = "No dependencies" # Status when it should be assumed that everything has changed. -all_changed_string = 'Found dependency (all)' +all_changed_string = "Found dependency (all)" # MatchStatus is used indicate if and how a target depends upon the supplied # sources. @@ -96,118 +94,130 @@ generator_wants_static_library_dependencies_adjusted = False -generator_default_variables = { -} -for dirname in ['INTERMEDIATE_DIR', 'SHARED_INTERMEDIATE_DIR', 'PRODUCT_DIR', - 'LIB_DIR', 'SHARED_LIB_DIR']: - generator_default_variables[dirname] = '!!!' - -for unused in ['RULE_INPUT_PATH', 'RULE_INPUT_ROOT', 'RULE_INPUT_NAME', - 'RULE_INPUT_DIRNAME', 'RULE_INPUT_EXT', - 'EXECUTABLE_PREFIX', 'EXECUTABLE_SUFFIX', - 'STATIC_LIB_PREFIX', 'STATIC_LIB_SUFFIX', - 'SHARED_LIB_PREFIX', 'SHARED_LIB_SUFFIX', - 'CONFIGURATION_NAME']: - generator_default_variables[unused] = '' +generator_default_variables = {} +for dirname in [ + "INTERMEDIATE_DIR", + "SHARED_INTERMEDIATE_DIR", + "PRODUCT_DIR", + "LIB_DIR", + "SHARED_LIB_DIR", +]: + generator_default_variables[dirname] = "!!!" + +for unused in [ + "RULE_INPUT_PATH", + "RULE_INPUT_ROOT", + "RULE_INPUT_NAME", + "RULE_INPUT_DIRNAME", + "RULE_INPUT_EXT", + "EXECUTABLE_PREFIX", + "EXECUTABLE_SUFFIX", + "STATIC_LIB_PREFIX", + "STATIC_LIB_SUFFIX", + "SHARED_LIB_PREFIX", + "SHARED_LIB_SUFFIX", + "CONFIGURATION_NAME", +]: + generator_default_variables[unused] = "" def _ToGypPath(path): - """Converts a path to the format used by gyp.""" - if os.sep == '\\' and os.altsep == '/': - return path.replace('\\', '/') - return path + """Converts a path to the format used by gyp.""" + if os.sep == "\\" and os.altsep == "/": + return path.replace("\\", "/") + return path def _ResolveParent(path, base_path_components): - """Resolves |path|, which starts with at least one '../'. Returns an empty + """Resolves |path|, which starts with at least one '../'. Returns an empty string if the path shouldn't be considered. See _AddSources() for a description of |base_path_components|.""" - depth = 0 - while path.startswith('../'): - depth += 1 - path = path[3:] - # Relative includes may go outside the source tree. For example, an action may - # have inputs in /usr/include, which are not in the source tree. - if depth > len(base_path_components): - return '' - if depth == len(base_path_components): - return path - return '/'.join(base_path_components[0:len(base_path_components) - depth]) + \ - '/' + path + depth = 0 + while path.startswith("../"): + depth += 1 + path = path[3:] + # Relative includes may go outside the source tree. For example, an action may + # have inputs in /usr/include, which are not in the source tree. + if depth > len(base_path_components): + return "" + if depth == len(base_path_components): + return path + return ( + "/".join(base_path_components[0 : len(base_path_components) - depth]) + + "/" + + path + ) def _AddSources(sources, base_path, base_path_components, result): - """Extracts valid sources from |sources| and adds them to |result|. Each + """Extracts valid sources from |sources| and adds them to |result|. Each source file is relative to |base_path|, but may contain '..'. To make resolving '..' easier |base_path_components| contains each of the directories in |base_path|. Additionally each source may contain variables. Such sources are ignored as it is assumed dependencies on them are expressed and tracked in some other means.""" - # NOTE: gyp paths are always posix style. - for source in sources: - if not len(source) or source.startswith('!!!') or source.startswith('$'): - continue - # variable expansion may lead to //. - org_source = source - source = source[0] + source[1:].replace('//', '/') - if source.startswith('../'): - source = _ResolveParent(source, base_path_components) - if len(source): - result.append(source) - continue - result.append(base_path + source) - if debug: - print('AddSource', org_source, result[len(result) - 1]) - - -def _ExtractSourcesFromAction(action, base_path, base_path_components, - results): - if 'inputs' in action: - _AddSources(action['inputs'], base_path, base_path_components, results) + # NOTE: gyp paths are always posix style. + for source in sources: + if not len(source) or source.startswith("!!!") or source.startswith("$"): + continue + # variable expansion may lead to //. + org_source = source + source = source[0] + source[1:].replace("//", "/") + if source.startswith("../"): + source = _ResolveParent(source, base_path_components) + if len(source): + result.append(source) + continue + result.append(base_path + source) + if debug: + print("AddSource", org_source, result[len(result) - 1]) + + +def _ExtractSourcesFromAction(action, base_path, base_path_components, results): + if "inputs" in action: + _AddSources(action["inputs"], base_path, base_path_components, results) def _ToLocalPath(toplevel_dir, path): - """Converts |path| to a path relative to |toplevel_dir|.""" - if path == toplevel_dir: - return '' - if path.startswith(toplevel_dir + '/'): - return path[len(toplevel_dir) + len('/'):] - return path + """Converts |path| to a path relative to |toplevel_dir|.""" + if path == toplevel_dir: + return "" + if path.startswith(toplevel_dir + "/"): + return path[len(toplevel_dir) + len("/") :] + return path def _ExtractSources(target, target_dict, toplevel_dir): - # |target| is either absolute or relative and in the format of the OS. Gyp - # source paths are always posix. Convert |target| to a posix path relative to - # |toplevel_dir_|. This is done to make it easy to build source paths. - base_path = posixpath.dirname(_ToLocalPath(toplevel_dir, _ToGypPath(target))) - base_path_components = base_path.split('/') - - # Add a trailing '/' so that _AddSources() can easily build paths. - if len(base_path): - base_path += '/' - - if debug: - print('ExtractSources', target, base_path) - - results = [] - if 'sources' in target_dict: - _AddSources(target_dict['sources'], base_path, base_path_components, - results) - # Include the inputs from any actions. Any changes to these affect the - # resulting output. - if 'actions' in target_dict: - for action in target_dict['actions']: - _ExtractSourcesFromAction(action, base_path, base_path_components, - results) - if 'rules' in target_dict: - for rule in target_dict['rules']: - _ExtractSourcesFromAction(rule, base_path, base_path_components, results) - - return results + # |target| is either absolute or relative and in the format of the OS. Gyp + # source paths are always posix. Convert |target| to a posix path relative to + # |toplevel_dir_|. This is done to make it easy to build source paths. + base_path = posixpath.dirname(_ToLocalPath(toplevel_dir, _ToGypPath(target))) + base_path_components = base_path.split("/") + + # Add a trailing '/' so that _AddSources() can easily build paths. + if len(base_path): + base_path += "/" + + if debug: + print("ExtractSources", target, base_path) + + results = [] + if "sources" in target_dict: + _AddSources(target_dict["sources"], base_path, base_path_components, results) + # Include the inputs from any actions. Any changes to these affect the + # resulting output. + if "actions" in target_dict: + for action in target_dict["actions"]: + _ExtractSourcesFromAction(action, base_path, base_path_components, results) + if "rules" in target_dict: + for rule in target_dict["rules"]: + _ExtractSourcesFromAction(rule, base_path, base_path_components, results) + + return results class Target(object): - """Holds information about a particular target: + """Holds information about a particular target: deps: set of Targets this Target depends upon. This is not recursive, only the direct dependent Targets. match_status: one of the MatchStatus values. @@ -225,101 +235,111 @@ class Target(object): is_static_library: true if the type of target is static_library. is_or_has_linked_ancestor: true if the target does a link (eg executable), or if there is a target in back_deps that does a link.""" - def __init__(self, name): - self.deps = set() - self.match_status = MATCH_STATUS_TBD - self.back_deps = set() - self.name = name - # TODO(sky): I don't like hanging this off Target. This state is specific - # to certain functions and should be isolated there. - self.visited = False - self.requires_build = False - self.added_to_compile_targets = False - self.in_roots = False - self.is_executable = False - self.is_static_library = False - self.is_or_has_linked_ancestor = False + + def __init__(self, name): + self.deps = set() + self.match_status = MATCH_STATUS_TBD + self.back_deps = set() + self.name = name + # TODO(sky): I don't like hanging this off Target. This state is specific + # to certain functions and should be isolated there. + self.visited = False + self.requires_build = False + self.added_to_compile_targets = False + self.in_roots = False + self.is_executable = False + self.is_static_library = False + self.is_or_has_linked_ancestor = False class Config(object): - """Details what we're looking for + """Details what we're looking for files: set of files to search for targets: see file description for details.""" - def __init__(self): - self.files = [] - self.targets = set() - self.additional_compile_target_names = set() - self.test_target_names = set() - - def Init(self, params): - """Initializes Config. This is a separate method as it raises an exception + + def __init__(self): + self.files = [] + self.targets = set() + self.additional_compile_target_names = set() + self.test_target_names = set() + + def Init(self, params): + """Initializes Config. This is a separate method as it raises an exception if there is a parse error.""" - generator_flags = params.get('generator_flags', {}) - config_path = generator_flags.get('config_path', None) - if not config_path: - return - try: - f = open(config_path, 'r') - config = json.load(f) - f.close() - except IOError: - raise Exception('Unable to open file ' + config_path) - except ValueError as e: - raise Exception('Unable to parse config file ' + config_path + str(e)) - if not isinstance(config, dict): - raise Exception('config_path must be a JSON file containing a dictionary') - self.files = config.get('files', []) - self.additional_compile_target_names = set( - config.get('additional_compile_targets', [])) - self.test_target_names = set(config.get('test_targets', [])) + generator_flags = params.get("generator_flags", {}) + config_path = generator_flags.get("config_path", None) + if not config_path: + return + try: + f = open(config_path, "r") + config = json.load(f) + f.close() + except IOError: + raise Exception("Unable to open file " + config_path) + except ValueError as e: + raise Exception("Unable to parse config file " + config_path + str(e)) + if not isinstance(config, dict): + raise Exception("config_path must be a JSON file containing a dictionary") + self.files = config.get("files", []) + self.additional_compile_target_names = set( + config.get("additional_compile_targets", []) + ) + self.test_target_names = set(config.get("test_targets", [])) def _WasBuildFileModified(build_file, data, files, toplevel_dir): - """Returns true if the build file |build_file| is either in |files| or + """Returns true if the build file |build_file| is either in |files| or one of the files included by |build_file| is in |files|. |toplevel_dir| is the root of the source tree.""" - if _ToLocalPath(toplevel_dir, _ToGypPath(build_file)) in files: - if debug: - print('gyp file modified', build_file) - return True + if _ToLocalPath(toplevel_dir, _ToGypPath(build_file)) in files: + if debug: + print("gyp file modified", build_file) + return True - # First element of included_files is the file itself. - if len(data[build_file]['included_files']) <= 1: + # First element of included_files is the file itself. + if len(data[build_file]["included_files"]) <= 1: + return False + + for include_file in data[build_file]["included_files"][1:]: + # |included_files| are relative to the directory of the |build_file|. + rel_include_file = _ToGypPath( + gyp.common.UnrelativePath(include_file, build_file) + ) + if _ToLocalPath(toplevel_dir, rel_include_file) in files: + if debug: + print( + "included gyp file modified, gyp_file=", + build_file, + "included file=", + rel_include_file, + ) + return True return False - for include_file in data[build_file]['included_files'][1:]: - # |included_files| are relative to the directory of the |build_file|. - rel_include_file = \ - _ToGypPath(gyp.common.UnrelativePath(include_file, build_file)) - if _ToLocalPath(toplevel_dir, rel_include_file) in files: - if debug: - print('included gyp file modified, gyp_file=', build_file, - 'included file=', rel_include_file) - return True - return False - def _GetOrCreateTargetByName(targets, target_name): - """Creates or returns the Target at targets[target_name]. If there is no + """Creates or returns the Target at targets[target_name]. If there is no Target for |target_name| one is created. Returns a tuple of whether a new Target was created and the Target.""" - if target_name in targets: - return False, targets[target_name] - target = Target(target_name) - targets[target_name] = target - return True, target + if target_name in targets: + return False, targets[target_name] + target = Target(target_name) + targets[target_name] = target + return True, target def _DoesTargetTypeRequireBuild(target_dict): - """Returns true if the target type is such that it needs to be built.""" - # If a 'none' target has rules or actions we assume it requires a build. - return bool(target_dict['type'] != 'none' or - target_dict.get('actions') or target_dict.get('rules')) + """Returns true if the target type is such that it needs to be built.""" + # If a 'none' target has rules or actions we assume it requires a build. + return bool( + target_dict["type"] != "none" + or target_dict.get("actions") + or target_dict.get("rules") + ) -def _GenerateTargets(data, target_list, target_dicts, toplevel_dir, files, - build_files): - """Returns a tuple of the following: +def _GenerateTargets(data, target_list, target_dicts, toplevel_dir, files, build_files): + """Returns a tuple of the following: . A dictionary mapping from fully qualified name to Target. . A list of the targets that have a source file in |files|. . Targets that constitute the 'all' target. See description at top of file @@ -327,417 +347,463 @@ def _GenerateTargets(data, target_list, target_dicts, toplevel_dir, files, This sets the |match_status| of the targets that contain any of the source files in |files| to MATCH_STATUS_MATCHES. |toplevel_dir| is the root of the source tree.""" - # Maps from target name to Target. - name_to_target = {} - - # Targets that matched. - matching_targets = [] - - # Queue of targets to visit. - targets_to_visit = target_list[:] - - # Maps from build file to a boolean indicating whether the build file is in - # |files|. - build_file_in_files = {} - - # Root targets across all files. - roots = set() - - # Set of Targets in |build_files|. - build_file_targets = set() - - while len(targets_to_visit) > 0: - target_name = targets_to_visit.pop() - created_target, target = _GetOrCreateTargetByName(name_to_target, - target_name) - if created_target: - roots.add(target) - elif target.visited: - continue - - target.visited = True - target.requires_build = _DoesTargetTypeRequireBuild( - target_dicts[target_name]) - target_type = target_dicts[target_name]['type'] - target.is_executable = target_type == 'executable' - target.is_static_library = target_type == 'static_library' - target.is_or_has_linked_ancestor = (target_type == 'executable' or - target_type == 'shared_library') - - build_file = gyp.common.ParseQualifiedTarget(target_name)[0] - if not build_file in build_file_in_files: - build_file_in_files[build_file] = \ - _WasBuildFileModified(build_file, data, files, toplevel_dir) - - if build_file in build_files: - build_file_targets.add(target) - - # If a build file (or any of its included files) is modified we assume all - # targets in the file are modified. - if build_file_in_files[build_file]: - print('matching target from modified build file', target_name) - target.match_status = MATCH_STATUS_MATCHES - matching_targets.append(target) - else: - sources = _ExtractSources(target_name, target_dicts[target_name], - toplevel_dir) - for source in sources: - if _ToGypPath(os.path.normpath(source)) in files: - print('target', target_name, 'matches', source) - target.match_status = MATCH_STATUS_MATCHES - matching_targets.append(target) - break - - # Add dependencies to visit as well as updating back pointers for deps. - for dep in target_dicts[target_name].get('dependencies', []): - targets_to_visit.append(dep) - - created_dep_target, dep_target = _GetOrCreateTargetByName(name_to_target, - dep) - if not created_dep_target: - roots.discard(dep_target) - - target.deps.add(dep_target) - dep_target.back_deps.add(target) - - return name_to_target, matching_targets, roots & build_file_targets + # Maps from target name to Target. + name_to_target = {} + + # Targets that matched. + matching_targets = [] + + # Queue of targets to visit. + targets_to_visit = target_list[:] + + # Maps from build file to a boolean indicating whether the build file is in + # |files|. + build_file_in_files = {} + + # Root targets across all files. + roots = set() + + # Set of Targets in |build_files|. + build_file_targets = set() + + while len(targets_to_visit) > 0: + target_name = targets_to_visit.pop() + created_target, target = _GetOrCreateTargetByName(name_to_target, target_name) + if created_target: + roots.add(target) + elif target.visited: + continue + + target.visited = True + target.requires_build = _DoesTargetTypeRequireBuild(target_dicts[target_name]) + target_type = target_dicts[target_name]["type"] + target.is_executable = target_type == "executable" + target.is_static_library = target_type == "static_library" + target.is_or_has_linked_ancestor = ( + target_type == "executable" or target_type == "shared_library" + ) + + build_file = gyp.common.ParseQualifiedTarget(target_name)[0] + if build_file not in build_file_in_files: + build_file_in_files[build_file] = _WasBuildFileModified( + build_file, data, files, toplevel_dir + ) + + if build_file in build_files: + build_file_targets.add(target) + + # If a build file (or any of its included files) is modified we assume all + # targets in the file are modified. + if build_file_in_files[build_file]: + print("matching target from modified build file", target_name) + target.match_status = MATCH_STATUS_MATCHES + matching_targets.append(target) + else: + sources = _ExtractSources( + target_name, target_dicts[target_name], toplevel_dir + ) + for source in sources: + if _ToGypPath(os.path.normpath(source)) in files: + print("target", target_name, "matches", source) + target.match_status = MATCH_STATUS_MATCHES + matching_targets.append(target) + break + + # Add dependencies to visit as well as updating back pointers for deps. + for dep in target_dicts[target_name].get("dependencies", []): + targets_to_visit.append(dep) + + created_dep_target, dep_target = _GetOrCreateTargetByName( + name_to_target, dep + ) + if not created_dep_target: + roots.discard(dep_target) + + target.deps.add(dep_target) + dep_target.back_deps.add(target) + + return name_to_target, matching_targets, roots & build_file_targets def _GetUnqualifiedToTargetMapping(all_targets, to_find): - """Returns a tuple of the following: + """Returns a tuple of the following: . mapping (dictionary) from unqualified name to Target for all the Targets in |to_find|. . any target names not found. If this is empty all targets were found.""" - result = {} - if not to_find: - return {}, [] - to_find = set(to_find) - for target_name in all_targets.keys(): - extracted = gyp.common.ParseQualifiedTarget(target_name) - if len(extracted) > 1 and extracted[1] in to_find: - to_find.remove(extracted[1]) - result[extracted[1]] = all_targets[target_name] - if not to_find: - return result, [] - return result, [x for x in to_find] + result = {} + if not to_find: + return {}, [] + to_find = set(to_find) + for target_name in all_targets.keys(): + extracted = gyp.common.ParseQualifiedTarget(target_name) + if len(extracted) > 1 and extracted[1] in to_find: + to_find.remove(extracted[1]) + result[extracted[1]] = all_targets[target_name] + if not to_find: + return result, [] + return result, [x for x in to_find] def _DoesTargetDependOnMatchingTargets(target): - """Returns true if |target| or any of its dependencies is one of the + """Returns true if |target| or any of its dependencies is one of the targets containing the files supplied as input to analyzer. This updates |matches| of the Targets as it recurses. target: the Target to look for.""" - if target.match_status == MATCH_STATUS_DOESNT_MATCH: + if target.match_status == MATCH_STATUS_DOESNT_MATCH: + return False + if ( + target.match_status == MATCH_STATUS_MATCHES + or target.match_status == MATCH_STATUS_MATCHES_BY_DEPENDENCY + ): + return True + for dep in target.deps: + if _DoesTargetDependOnMatchingTargets(dep): + target.match_status = MATCH_STATUS_MATCHES_BY_DEPENDENCY + print("\t", target.name, "matches by dep", dep.name) + return True + target.match_status = MATCH_STATUS_DOESNT_MATCH return False - if target.match_status == MATCH_STATUS_MATCHES or \ - target.match_status == MATCH_STATUS_MATCHES_BY_DEPENDENCY: - return True - for dep in target.deps: - if _DoesTargetDependOnMatchingTargets(dep): - target.match_status = MATCH_STATUS_MATCHES_BY_DEPENDENCY - print('\t', target.name, 'matches by dep', dep.name) - return True - target.match_status = MATCH_STATUS_DOESNT_MATCH - return False def _GetTargetsDependingOnMatchingTargets(possible_targets): - """Returns the list of Targets in |possible_targets| that depend (either + """Returns the list of Targets in |possible_targets| that depend (either directly on indirectly) on at least one of the targets containing the files supplied as input to analyzer. possible_targets: targets to search from.""" - found = [] - print('Targets that matched by dependency:') - for target in possible_targets: - if _DoesTargetDependOnMatchingTargets(target): - found.append(target) - return found + found = [] + print("Targets that matched by dependency:") + for target in possible_targets: + if _DoesTargetDependOnMatchingTargets(target): + found.append(target) + return found def _AddCompileTargets(target, roots, add_if_no_ancestor, result): - """Recurses through all targets that depend on |target|, adding all targets + """Recurses through all targets that depend on |target|, adding all targets that need to be built (and are in |roots|) to |result|. roots: set of root targets. add_if_no_ancestor: If true and there are no ancestors of |target| then add |target| to |result|. |target| must still be in |roots|. result: targets that need to be built are added here.""" - if target.visited: - return - - target.visited = True - target.in_roots = target in roots - - for back_dep_target in target.back_deps: - _AddCompileTargets(back_dep_target, roots, False, result) - target.added_to_compile_targets |= back_dep_target.added_to_compile_targets - target.in_roots |= back_dep_target.in_roots - target.is_or_has_linked_ancestor |= ( - back_dep_target.is_or_has_linked_ancestor) - - # Always add 'executable' targets. Even though they may be built by other - # targets that depend upon them it makes detection of what is going to be - # built easier. - # And always add static_libraries that have no dependencies on them from - # linkables. This is necessary as the other dependencies on them may be - # static libraries themselves, which are not compile time dependencies. - if target.in_roots and \ - (target.is_executable or - (not target.added_to_compile_targets and - (add_if_no_ancestor or target.requires_build)) or - (target.is_static_library and add_if_no_ancestor and - not target.is_or_has_linked_ancestor)): - print('\t\tadding to compile targets', target.name, 'executable', - target.is_executable, 'added_to_compile_targets', - target.added_to_compile_targets, 'add_if_no_ancestor', - add_if_no_ancestor, 'requires_build', target.requires_build, - 'is_static_library', target.is_static_library, - 'is_or_has_linked_ancestor', target.is_or_has_linked_ancestor) - result.add(target) - target.added_to_compile_targets = True + if target.visited: + return + + target.visited = True + target.in_roots = target in roots + + for back_dep_target in target.back_deps: + _AddCompileTargets(back_dep_target, roots, False, result) + target.added_to_compile_targets |= back_dep_target.added_to_compile_targets + target.in_roots |= back_dep_target.in_roots + target.is_or_has_linked_ancestor |= back_dep_target.is_or_has_linked_ancestor + + # Always add 'executable' targets. Even though they may be built by other + # targets that depend upon them it makes detection of what is going to be + # built easier. + # And always add static_libraries that have no dependencies on them from + # linkables. This is necessary as the other dependencies on them may be + # static libraries themselves, which are not compile time dependencies. + if target.in_roots and ( + target.is_executable + or ( + not target.added_to_compile_targets + and (add_if_no_ancestor or target.requires_build) + ) + or ( + target.is_static_library + and add_if_no_ancestor + and not target.is_or_has_linked_ancestor + ) + ): + print( + "\t\tadding to compile targets", + target.name, + "executable", + target.is_executable, + "added_to_compile_targets", + target.added_to_compile_targets, + "add_if_no_ancestor", + add_if_no_ancestor, + "requires_build", + target.requires_build, + "is_static_library", + target.is_static_library, + "is_or_has_linked_ancestor", + target.is_or_has_linked_ancestor, + ) + result.add(target) + target.added_to_compile_targets = True def _GetCompileTargets(matching_targets, supplied_targets): - """Returns the set of Targets that require a build. + """Returns the set of Targets that require a build. matching_targets: targets that changed and need to be built. supplied_targets: set of targets supplied to analyzer to search from.""" - result = set() - for target in matching_targets: - print('finding compile targets for match', target.name) - _AddCompileTargets(target, supplied_targets, True, result) - return result + result = set() + for target in matching_targets: + print("finding compile targets for match", target.name) + _AddCompileTargets(target, supplied_targets, True, result) + return result def _WriteOutput(params, **values): - """Writes the output, either to stdout or a file is specified.""" - if 'error' in values: - print('Error:', values['error']) - if 'status' in values: - print(values['status']) - if 'targets' in values: - values['targets'].sort() - print('Supplied targets that depend on changed files:') - for target in values['targets']: - print('\t', target) - if 'invalid_targets' in values: - values['invalid_targets'].sort() - print('The following targets were not found:') - for target in values['invalid_targets']: - print('\t', target) - if 'build_targets' in values: - values['build_targets'].sort() - print('Targets that require a build:') - for target in values['build_targets']: - print('\t', target) - if 'compile_targets' in values: - values['compile_targets'].sort() - print('Targets that need to be built:') - for target in values['compile_targets']: - print('\t', target) - if 'test_targets' in values: - values['test_targets'].sort() - print('Test targets:') - for target in values['test_targets']: - print('\t', target) - - output_path = params.get('generator_flags', {}).get( - 'analyzer_output_path', None) - if not output_path: - print(json.dumps(values)) - return - try: - f = open(output_path, 'w') - f.write(json.dumps(values) + '\n') - f.close() - except IOError as e: - print('Error writing to output file', output_path, str(e)) + """Writes the output, either to stdout or a file is specified.""" + if "error" in values: + print("Error:", values["error"]) + if "status" in values: + print(values["status"]) + if "targets" in values: + values["targets"].sort() + print("Supplied targets that depend on changed files:") + for target in values["targets"]: + print("\t", target) + if "invalid_targets" in values: + values["invalid_targets"].sort() + print("The following targets were not found:") + for target in values["invalid_targets"]: + print("\t", target) + if "build_targets" in values: + values["build_targets"].sort() + print("Targets that require a build:") + for target in values["build_targets"]: + print("\t", target) + if "compile_targets" in values: + values["compile_targets"].sort() + print("Targets that need to be built:") + for target in values["compile_targets"]: + print("\t", target) + if "test_targets" in values: + values["test_targets"].sort() + print("Test targets:") + for target in values["test_targets"]: + print("\t", target) + + output_path = params.get("generator_flags", {}).get("analyzer_output_path", None) + if not output_path: + print(json.dumps(values)) + return + try: + f = open(output_path, "w") + f.write(json.dumps(values) + "\n") + f.close() + except IOError as e: + print("Error writing to output file", output_path, str(e)) def _WasGypIncludeFileModified(params, files): - """Returns true if one of the files in |files| is in the set of included + """Returns true if one of the files in |files| is in the set of included files.""" - if params['options'].includes: - for include in params['options'].includes: - if _ToGypPath(os.path.normpath(include)) in files: - print('Include file modified, assuming all changed', include) - return True - return False + if params["options"].includes: + for include in params["options"].includes: + if _ToGypPath(os.path.normpath(include)) in files: + print("Include file modified, assuming all changed", include) + return True + return False def _NamesNotIn(names, mapping): - """Returns a list of the values in |names| that are not in |mapping|.""" - return [name for name in names if name not in mapping] + """Returns a list of the values in |names| that are not in |mapping|.""" + return [name for name in names if name not in mapping] def _LookupTargets(names, mapping): - """Returns a list of the mapping[name] for each value in |names| that is in + """Returns a list of the mapping[name] for each value in |names| that is in |mapping|.""" - return [mapping[name] for name in names if name in mapping] + return [mapping[name] for name in names if name in mapping] def CalculateVariables(default_variables, params): - """Calculate additional variables for use in the build (called by gyp).""" - flavor = gyp.common.GetFlavor(params) - if flavor == 'mac': - default_variables.setdefault('OS', 'mac') - elif flavor == 'win': - default_variables.setdefault('OS', 'win') - # Copy additional generator configuration data from VS, which is shared - # by the Windows Ninja generator. - import gyp.generator.msvs as msvs_generator - generator_additional_non_configuration_keys = getattr(msvs_generator, - 'generator_additional_non_configuration_keys', []) - generator_additional_path_sections = getattr(msvs_generator, - 'generator_additional_path_sections', []) - - gyp.msvs_emulation.CalculateCommonVariables(default_variables, params) - else: - operating_system = flavor - if flavor == 'android': - operating_system = 'linux' # Keep this legacy behavior for now. - default_variables.setdefault('OS', operating_system) + """Calculate additional variables for use in the build (called by gyp).""" + flavor = gyp.common.GetFlavor(params) + if flavor == "mac": + default_variables.setdefault("OS", "mac") + elif flavor == "win": + default_variables.setdefault("OS", "win") + gyp.msvs_emulation.CalculateCommonVariables(default_variables, params) + else: + operating_system = flavor + if flavor == "android": + operating_system = "linux" # Keep this legacy behavior for now. + default_variables.setdefault("OS", operating_system) class TargetCalculator(object): - """Calculates the matching test_targets and matching compile_targets.""" - def __init__(self, files, additional_compile_target_names, test_target_names, - data, target_list, target_dicts, toplevel_dir, build_files): - self._additional_compile_target_names = set(additional_compile_target_names) - self._test_target_names = set(test_target_names) - self._name_to_target, self._changed_targets, self._root_targets = ( - _GenerateTargets(data, target_list, target_dicts, toplevel_dir, - frozenset(files), build_files)) - self._unqualified_mapping, self.invalid_targets = ( - _GetUnqualifiedToTargetMapping(self._name_to_target, - self._supplied_target_names_no_all())) - - def _supplied_target_names(self): - return self._additional_compile_target_names | self._test_target_names - - def _supplied_target_names_no_all(self): - """Returns the supplied test targets without 'all'.""" - result = self._supplied_target_names() - result.discard('all') - return result - - def is_build_impacted(self): - """Returns true if the supplied files impact the build at all.""" - return self._changed_targets - - def find_matching_test_target_names(self): - """Returns the set of output test targets.""" - assert self.is_build_impacted() - # Find the test targets first. 'all' is special cased to mean all the - # root targets. To deal with all the supplied |test_targets| are expanded - # to include the root targets during lookup. If any of the root targets - # match, we remove it and replace it with 'all'. - test_target_names_no_all = set(self._test_target_names) - test_target_names_no_all.discard('all') - test_targets_no_all = _LookupTargets(test_target_names_no_all, - self._unqualified_mapping) - test_target_names_contains_all = 'all' in self._test_target_names - if test_target_names_contains_all: - test_targets = [x for x in (set(test_targets_no_all) | - set(self._root_targets))] - else: - test_targets = [x for x in test_targets_no_all] - print('supplied test_targets') - for target_name in self._test_target_names: - print('\t', target_name) - print('found test_targets') - for target in test_targets: - print('\t', target.name) - print('searching for matching test targets') - matching_test_targets = _GetTargetsDependingOnMatchingTargets(test_targets) - matching_test_targets_contains_all = (test_target_names_contains_all and - set(matching_test_targets) & - set(self._root_targets)) - if matching_test_targets_contains_all: - # Remove any of the targets for all that were not explicitly supplied, - # 'all' is subsequentely added to the matching names below. - matching_test_targets = [x for x in (set(matching_test_targets) & - set(test_targets_no_all))] - print('matched test_targets') - for target in matching_test_targets: - print('\t', target.name) - matching_target_names = [gyp.common.ParseQualifiedTarget(target.name)[1] - for target in matching_test_targets] - if matching_test_targets_contains_all: - matching_target_names.append('all') - print('\tall') - return matching_target_names - - def find_matching_compile_target_names(self): - """Returns the set of output compile targets.""" - assert self.is_build_impacted() - # Compile targets are found by searching up from changed targets. - # Reset the visited status for _GetBuildTargets. - for target in self._name_to_target.values(): - target.visited = False - - supplied_targets = _LookupTargets(self._supplied_target_names_no_all(), - self._unqualified_mapping) - if 'all' in self._supplied_target_names(): - supplied_targets = [x for x in (set(supplied_targets) | - set(self._root_targets))] - print('Supplied test_targets & compile_targets') - for target in supplied_targets: - print('\t', target.name) - print('Finding compile targets') - compile_targets = _GetCompileTargets(self._changed_targets, - supplied_targets) - return [gyp.common.ParseQualifiedTarget(target.name)[1] - for target in compile_targets] + """Calculates the matching test_targets and matching compile_targets.""" + + def __init__( + self, + files, + additional_compile_target_names, + test_target_names, + data, + target_list, + target_dicts, + toplevel_dir, + build_files, + ): + self._additional_compile_target_names = set(additional_compile_target_names) + self._test_target_names = set(test_target_names) + ( + self._name_to_target, + self._changed_targets, + self._root_targets, + ) = _GenerateTargets( + data, target_list, target_dicts, toplevel_dir, frozenset(files), build_files + ) + ( + self._unqualified_mapping, + self.invalid_targets, + ) = _GetUnqualifiedToTargetMapping( + self._name_to_target, self._supplied_target_names_no_all() + ) + + def _supplied_target_names(self): + return self._additional_compile_target_names | self._test_target_names + + def _supplied_target_names_no_all(self): + """Returns the supplied test targets without 'all'.""" + result = self._supplied_target_names() + result.discard("all") + return result + + def is_build_impacted(self): + """Returns true if the supplied files impact the build at all.""" + return self._changed_targets + + def find_matching_test_target_names(self): + """Returns the set of output test targets.""" + assert self.is_build_impacted() + # Find the test targets first. 'all' is special cased to mean all the + # root targets. To deal with all the supplied |test_targets| are expanded + # to include the root targets during lookup. If any of the root targets + # match, we remove it and replace it with 'all'. + test_target_names_no_all = set(self._test_target_names) + test_target_names_no_all.discard("all") + test_targets_no_all = _LookupTargets( + test_target_names_no_all, self._unqualified_mapping + ) + test_target_names_contains_all = "all" in self._test_target_names + if test_target_names_contains_all: + test_targets = [ + x for x in (set(test_targets_no_all) | set(self._root_targets)) + ] + else: + test_targets = [x for x in test_targets_no_all] + print("supplied test_targets") + for target_name in self._test_target_names: + print("\t", target_name) + print("found test_targets") + for target in test_targets: + print("\t", target.name) + print("searching for matching test targets") + matching_test_targets = _GetTargetsDependingOnMatchingTargets(test_targets) + matching_test_targets_contains_all = test_target_names_contains_all and set( + matching_test_targets + ) & set(self._root_targets) + if matching_test_targets_contains_all: + # Remove any of the targets for all that were not explicitly supplied, + # 'all' is subsequentely added to the matching names below. + matching_test_targets = [ + x for x in (set(matching_test_targets) & set(test_targets_no_all)) + ] + print("matched test_targets") + for target in matching_test_targets: + print("\t", target.name) + matching_target_names = [ + gyp.common.ParseQualifiedTarget(target.name)[1] + for target in matching_test_targets + ] + if matching_test_targets_contains_all: + matching_target_names.append("all") + print("\tall") + return matching_target_names + + def find_matching_compile_target_names(self): + """Returns the set of output compile targets.""" + assert self.is_build_impacted() + # Compile targets are found by searching up from changed targets. + # Reset the visited status for _GetBuildTargets. + for target in self._name_to_target.values(): + target.visited = False + + supplied_targets = _LookupTargets( + self._supplied_target_names_no_all(), self._unqualified_mapping + ) + if "all" in self._supplied_target_names(): + supplied_targets = [ + x for x in (set(supplied_targets) | set(self._root_targets)) + ] + print("Supplied test_targets & compile_targets") + for target in supplied_targets: + print("\t", target.name) + print("Finding compile targets") + compile_targets = _GetCompileTargets(self._changed_targets, supplied_targets) + return [ + gyp.common.ParseQualifiedTarget(target.name)[1] + for target in compile_targets + ] def GenerateOutput(target_list, target_dicts, data, params): - """Called by gyp as the final stage. Outputs results.""" - config = Config() - try: - config.Init(params) - - if not config.files: - raise Exception('Must specify files to analyze via config_path generator ' - 'flag') - - toplevel_dir = _ToGypPath(os.path.abspath(params['options'].toplevel_dir)) - if debug: - print('toplevel_dir', toplevel_dir) - - if _WasGypIncludeFileModified(params, config.files): - result_dict = { 'status': all_changed_string, - 'test_targets': list(config.test_target_names), - 'compile_targets': list( - config.additional_compile_target_names | - config.test_target_names) } - _WriteOutput(params, **result_dict) - return - - calculator = TargetCalculator(config.files, - config.additional_compile_target_names, - config.test_target_names, data, - target_list, target_dicts, toplevel_dir, - params['build_files']) - if not calculator.is_build_impacted(): - result_dict = { 'status': no_dependency_string, - 'test_targets': [], - 'compile_targets': [] } - if calculator.invalid_targets: - result_dict['invalid_targets'] = calculator.invalid_targets - _WriteOutput(params, **result_dict) - return - - test_target_names = calculator.find_matching_test_target_names() - compile_target_names = calculator.find_matching_compile_target_names() - found_at_least_one_target = compile_target_names or test_target_names - result_dict = { 'test_targets': test_target_names, - 'status': found_dependency_string if - found_at_least_one_target else no_dependency_string, - 'compile_targets': list( - set(compile_target_names) | - set(test_target_names)) } - if calculator.invalid_targets: - result_dict['invalid_targets'] = calculator.invalid_targets - _WriteOutput(params, **result_dict) - - except Exception as e: - _WriteOutput(params, error=str(e)) + """Called by gyp as the final stage. Outputs results.""" + config = Config() + try: + config.Init(params) + + if not config.files: + raise Exception( + "Must specify files to analyze via config_path generator " "flag" + ) + + toplevel_dir = _ToGypPath(os.path.abspath(params["options"].toplevel_dir)) + if debug: + print("toplevel_dir", toplevel_dir) + + if _WasGypIncludeFileModified(params, config.files): + result_dict = { + "status": all_changed_string, + "test_targets": list(config.test_target_names), + "compile_targets": list( + config.additional_compile_target_names | config.test_target_names + ), + } + _WriteOutput(params, **result_dict) + return + + calculator = TargetCalculator( + config.files, + config.additional_compile_target_names, + config.test_target_names, + data, + target_list, + target_dicts, + toplevel_dir, + params["build_files"], + ) + if not calculator.is_build_impacted(): + result_dict = { + "status": no_dependency_string, + "test_targets": [], + "compile_targets": [], + } + if calculator.invalid_targets: + result_dict["invalid_targets"] = calculator.invalid_targets + _WriteOutput(params, **result_dict) + return + + test_target_names = calculator.find_matching_test_target_names() + compile_target_names = calculator.find_matching_compile_target_names() + found_at_least_one_target = compile_target_names or test_target_names + result_dict = { + "test_targets": test_target_names, + "status": found_dependency_string + if found_at_least_one_target + else no_dependency_string, + "compile_targets": list(set(compile_target_names) | set(test_target_names)), + } + if calculator.invalid_targets: + result_dict["invalid_targets"] = calculator.invalid_targets + _WriteOutput(params, **result_dict) + + except Exception as e: + _WriteOutput(params, error=str(e)) diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py index cecb28c366..7dbbd579ca 100644 --- a/gyp/pylib/gyp/generator/android.py +++ b/gyp/pylib/gyp/generator/android.py @@ -24,24 +24,24 @@ import subprocess generator_default_variables = { - 'OS': 'android', - 'EXECUTABLE_PREFIX': '', - 'EXECUTABLE_SUFFIX': '', - 'STATIC_LIB_PREFIX': 'lib', - 'SHARED_LIB_PREFIX': 'lib', - 'STATIC_LIB_SUFFIX': '.a', - 'SHARED_LIB_SUFFIX': '.so', - 'INTERMEDIATE_DIR': '$(gyp_intermediate_dir)', - 'SHARED_INTERMEDIATE_DIR': '$(gyp_shared_intermediate_dir)', - 'PRODUCT_DIR': '$(gyp_shared_intermediate_dir)', - 'SHARED_LIB_DIR': '$(builddir)/lib.$(TOOLSET)', - 'LIB_DIR': '$(obj).$(TOOLSET)', - 'RULE_INPUT_ROOT': '%(INPUT_ROOT)s', # This gets expanded by Python. - 'RULE_INPUT_DIRNAME': '%(INPUT_DIRNAME)s', # This gets expanded by Python. - 'RULE_INPUT_PATH': '$(RULE_SOURCES)', - 'RULE_INPUT_EXT': '$(suffix $<)', - 'RULE_INPUT_NAME': '$(notdir $<)', - 'CONFIGURATION_NAME': '$(GYP_CONFIGURATION)', + "OS": "android", + "EXECUTABLE_PREFIX": "", + "EXECUTABLE_SUFFIX": "", + "STATIC_LIB_PREFIX": "lib", + "SHARED_LIB_PREFIX": "lib", + "STATIC_LIB_SUFFIX": ".a", + "SHARED_LIB_SUFFIX": ".so", + "INTERMEDIATE_DIR": "$(gyp_intermediate_dir)", + "SHARED_INTERMEDIATE_DIR": "$(gyp_shared_intermediate_dir)", + "PRODUCT_DIR": "$(gyp_shared_intermediate_dir)", + "SHARED_LIB_DIR": "$(builddir)/lib.$(TOOLSET)", + "LIB_DIR": "$(obj).$(TOOLSET)", + "RULE_INPUT_ROOT": "%(INPUT_ROOT)s", # This gets expanded by Python. + "RULE_INPUT_DIRNAME": "%(INPUT_DIRNAME)s", # This gets expanded by Python. + "RULE_INPUT_PATH": "$(RULE_SOURCES)", + "RULE_INPUT_EXT": "$(suffix $<)", + "RULE_INPUT_NAME": "$(notdir $<)", + "CONFIGURATION_NAME": "$(GYP_CONFIGURATION)", } # Make supports multiple toolsets @@ -51,9 +51,9 @@ # Generator-specific gyp specs. generator_additional_non_configuration_keys = [ # Boolean to declare that this target does not want its name mangled. - 'android_unmangled_name', + "android_unmangled_name", # Map of android build system variables to set. - 'aosp_build_settings', + "aosp_build_settings", ] generator_additional_path_sections = [] generator_extra_sources_for_rules = [] @@ -72,20 +72,20 @@ # Map gyp target types to Android module classes. MODULE_CLASSES = { - 'static_library': 'STATIC_LIBRARIES', - 'shared_library': 'SHARED_LIBRARIES', - 'executable': 'EXECUTABLES', + "static_library": "STATIC_LIBRARIES", + "shared_library": "SHARED_LIBRARIES", + "executable": "EXECUTABLES", } def IsCPPExtension(ext): - return make.COMPILABLE_EXTENSIONS.get(ext) == 'cxx' + return make.COMPILABLE_EXTENSIONS.get(ext) == "cxx" def Sourceify(path): - """Convert a path to its source directory form. The Android backend does not + """Convert a path to its source directory form. The Android backend does not support options.generator_output, so this function is a noop.""" - return path + return path # Map from qualified target to path to output. @@ -101,17 +101,27 @@ def Sourceify(path): class AndroidMkWriter(object): - """AndroidMkWriter packages up the writing of one target-specific Android.mk. + """AndroidMkWriter packages up the writing of one target-specific Android.mk. Its only real entry point is Write(), and is mostly used for namespacing. """ - def __init__(self, android_top_dir): - self.android_top_dir = android_top_dir - - def Write(self, qualified_target, relative_target, base_path, output_filename, - spec, configs, part_of_all, write_alias_target, sdk_version): - """The main entry point: writes a .mk file for a single target. + def __init__(self, android_top_dir): + self.android_top_dir = android_top_dir + + def Write( + self, + qualified_target, + relative_target, + base_path, + output_filename, + spec, + configs, + part_of_all, + write_alias_target, + sdk_version, + ): + """The main entry point: writes a .mk file for a single target. Arguments: qualified_target: target we're generating @@ -125,114 +135,124 @@ def Write(self, qualified_target, relative_target, base_path, output_filename, this target sdk_version: what to emit for LOCAL_SDK_VERSION in output """ - gyp.common.EnsureDirExists(output_filename) - - self.fp = open(output_filename, 'w') - - self.fp.write(header) - - self.qualified_target = qualified_target - self.relative_target = relative_target - self.path = base_path - self.target = spec['target_name'] - self.type = spec['type'] - self.toolset = spec['toolset'] - - deps, link_deps = self.ComputeDeps(spec) - - # Some of the generation below can add extra output, sources, or - # link dependencies. All of the out params of the functions that - # follow use names like extra_foo. - extra_outputs = [] - extra_sources = [] - - self.android_class = MODULE_CLASSES.get(self.type, 'GYP') - self.android_module = self.ComputeAndroidModule(spec) - (self.android_stem, self.android_suffix) = self.ComputeOutputParts(spec) - self.output = self.output_binary = self.ComputeOutput(spec) - - # Standard header. - self.WriteLn('include $(CLEAR_VARS)\n') - - # Module class and name. - self.WriteLn('LOCAL_MODULE_CLASS := ' + self.android_class) - self.WriteLn('LOCAL_MODULE := ' + self.android_module) - # Only emit LOCAL_MODULE_STEM if it's different to LOCAL_MODULE. - # The library module classes fail if the stem is set. ComputeOutputParts - # makes sure that stem == modulename in these cases. - if self.android_stem != self.android_module: - self.WriteLn('LOCAL_MODULE_STEM := ' + self.android_stem) - self.WriteLn('LOCAL_MODULE_SUFFIX := ' + self.android_suffix) - if self.toolset == 'host': - self.WriteLn('LOCAL_IS_HOST_MODULE := true') - self.WriteLn('LOCAL_MULTILIB := $(GYP_HOST_MULTILIB)') - elif sdk_version > 0: - self.WriteLn('LOCAL_MODULE_TARGET_ARCH := ' - '$(TARGET_$(GYP_VAR_PREFIX)ARCH)') - self.WriteLn('LOCAL_SDK_VERSION := %s' % sdk_version) - - # Grab output directories; needed for Actions and Rules. - if self.toolset == 'host': - self.WriteLn('gyp_intermediate_dir := ' - '$(call local-intermediates-dir,,$(GYP_HOST_VAR_PREFIX))') - else: - self.WriteLn('gyp_intermediate_dir := ' - '$(call local-intermediates-dir,,$(GYP_VAR_PREFIX))') - self.WriteLn('gyp_shared_intermediate_dir := ' - '$(call intermediates-dir-for,GYP,shared,,,$(GYP_VAR_PREFIX))') - self.WriteLn() - - # List files this target depends on so that actions/rules/copies/sources - # can depend on the list. - # TODO: doesn't pull in things through transitive link deps; needed? - target_dependencies = [x[1] for x in deps if x[0] == 'path'] - self.WriteLn('# Make sure our deps are built first.') - self.WriteList(target_dependencies, 'GYP_TARGET_DEPENDENCIES', - local_pathify=True) - - # Actions must come first, since they can generate more OBJs for use below. - if 'actions' in spec: - self.WriteActions(spec['actions'], extra_sources, extra_outputs) - - # Rules must be early like actions. - if 'rules' in spec: - self.WriteRules(spec['rules'], extra_sources, extra_outputs) - - if 'copies' in spec: - self.WriteCopies(spec['copies'], extra_outputs) - - # GYP generated outputs. - self.WriteList(extra_outputs, 'GYP_GENERATED_OUTPUTS', local_pathify=True) - - # Set LOCAL_ADDITIONAL_DEPENDENCIES so that Android's build rules depend - # on both our dependency targets and our generated files. - self.WriteLn('# Make sure our deps and generated files are built first.') - self.WriteLn('LOCAL_ADDITIONAL_DEPENDENCIES := $(GYP_TARGET_DEPENDENCIES) ' - '$(GYP_GENERATED_OUTPUTS)') - self.WriteLn() - - # Sources. - if spec.get('sources', []) or extra_sources: - self.WriteSources(spec, configs, extra_sources) - - self.WriteTarget(spec, configs, deps, link_deps, part_of_all, - write_alias_target) - - # Update global list of target outputs, used in dependency tracking. - target_outputs[qualified_target] = ('path', self.output_binary) - - # Update global list of link dependencies. - if self.type == 'static_library': - target_link_deps[qualified_target] = ('static', self.android_module) - elif self.type == 'shared_library': - target_link_deps[qualified_target] = ('shared', self.android_module) - - self.fp.close() - return self.android_module - - - def WriteActions(self, actions, extra_sources, extra_outputs): - """Write Makefile code for any 'actions' from the gyp input. + gyp.common.EnsureDirExists(output_filename) + + self.fp = open(output_filename, "w") + + self.fp.write(header) + + self.qualified_target = qualified_target + self.relative_target = relative_target + self.path = base_path + self.target = spec["target_name"] + self.type = spec["type"] + self.toolset = spec["toolset"] + + deps, link_deps = self.ComputeDeps(spec) + + # Some of the generation below can add extra output, sources, or + # link dependencies. All of the out params of the functions that + # follow use names like extra_foo. + extra_outputs = [] + extra_sources = [] + + self.android_class = MODULE_CLASSES.get(self.type, "GYP") + self.android_module = self.ComputeAndroidModule(spec) + (self.android_stem, self.android_suffix) = self.ComputeOutputParts(spec) + self.output = self.output_binary = self.ComputeOutput(spec) + + # Standard header. + self.WriteLn("include $(CLEAR_VARS)\n") + + # Module class and name. + self.WriteLn("LOCAL_MODULE_CLASS := " + self.android_class) + self.WriteLn("LOCAL_MODULE := " + self.android_module) + # Only emit LOCAL_MODULE_STEM if it's different to LOCAL_MODULE. + # The library module classes fail if the stem is set. ComputeOutputParts + # makes sure that stem == modulename in these cases. + if self.android_stem != self.android_module: + self.WriteLn("LOCAL_MODULE_STEM := " + self.android_stem) + self.WriteLn("LOCAL_MODULE_SUFFIX := " + self.android_suffix) + if self.toolset == "host": + self.WriteLn("LOCAL_IS_HOST_MODULE := true") + self.WriteLn("LOCAL_MULTILIB := $(GYP_HOST_MULTILIB)") + elif sdk_version > 0: + self.WriteLn( + "LOCAL_MODULE_TARGET_ARCH := " "$(TARGET_$(GYP_VAR_PREFIX)ARCH)" + ) + self.WriteLn("LOCAL_SDK_VERSION := %s" % sdk_version) + + # Grab output directories; needed for Actions and Rules. + if self.toolset == "host": + self.WriteLn( + "gyp_intermediate_dir := " + "$(call local-intermediates-dir,,$(GYP_HOST_VAR_PREFIX))" + ) + else: + self.WriteLn( + "gyp_intermediate_dir := " + "$(call local-intermediates-dir,,$(GYP_VAR_PREFIX))" + ) + self.WriteLn( + "gyp_shared_intermediate_dir := " + "$(call intermediates-dir-for,GYP,shared,,,$(GYP_VAR_PREFIX))" + ) + self.WriteLn() + + # List files this target depends on so that actions/rules/copies/sources + # can depend on the list. + # TODO: doesn't pull in things through transitive link deps; needed? + target_dependencies = [x[1] for x in deps if x[0] == "path"] + self.WriteLn("# Make sure our deps are built first.") + self.WriteList( + target_dependencies, "GYP_TARGET_DEPENDENCIES", local_pathify=True + ) + + # Actions must come first, since they can generate more OBJs for use below. + if "actions" in spec: + self.WriteActions(spec["actions"], extra_sources, extra_outputs) + + # Rules must be early like actions. + if "rules" in spec: + self.WriteRules(spec["rules"], extra_sources, extra_outputs) + + if "copies" in spec: + self.WriteCopies(spec["copies"], extra_outputs) + + # GYP generated outputs. + self.WriteList(extra_outputs, "GYP_GENERATED_OUTPUTS", local_pathify=True) + + # Set LOCAL_ADDITIONAL_DEPENDENCIES so that Android's build rules depend + # on both our dependency targets and our generated files. + self.WriteLn("# Make sure our deps and generated files are built first.") + self.WriteLn( + "LOCAL_ADDITIONAL_DEPENDENCIES := $(GYP_TARGET_DEPENDENCIES) " + "$(GYP_GENERATED_OUTPUTS)" + ) + self.WriteLn() + + # Sources. + if spec.get("sources", []) or extra_sources: + self.WriteSources(spec, configs, extra_sources) + + self.WriteTarget( + spec, configs, deps, link_deps, part_of_all, write_alias_target + ) + + # Update global list of target outputs, used in dependency tracking. + target_outputs[qualified_target] = ("path", self.output_binary) + + # Update global list of link dependencies. + if self.type == "static_library": + target_link_deps[qualified_target] = ("static", self.android_module) + elif self.type == "shared_library": + target_link_deps[qualified_target] = ("shared", self.android_module) + + self.fp.close() + return self.android_module + + def WriteActions(self, actions, extra_sources, extra_outputs): + """Write Makefile code for any 'actions' from the gyp input. extra_sources: a list that will be filled in with newly generated source files, if any @@ -240,266 +260,300 @@ def WriteActions(self, actions, extra_sources, extra_outputs): actions (used to make other pieces dependent on these actions) """ - for action in actions: - name = make.StringToMakefileVariable('%s_%s' % (self.relative_target, - action['action_name'])) - self.WriteLn('### Rules for action "%s":' % action['action_name']) - inputs = action['inputs'] - outputs = action['outputs'] - - # Build up a list of outputs. - # Collect the output dirs we'll need. - dirs = set() - for out in outputs: - if not out.startswith('$'): - print('WARNING: Action for target "%s" writes output to local path ' - '"%s".' % (self.target, out)) - dir = os.path.split(out)[0] - if dir: - dirs.add(dir) - if int(action.get('process_outputs_as_sources', False)): - extra_sources += outputs - - # Prepare the actual command. - command = gyp.common.EncodePOSIXShellList(action['action']) - if 'message' in action: - quiet_cmd = 'Gyp action: %s ($@)' % action['message'] - else: - quiet_cmd = 'Gyp action: %s ($@)' % name - if len(dirs) > 0: - command = 'mkdir -p %s' % ' '.join(dirs) + '; ' + command - - cd_action = 'cd $(gyp_local_path)/%s; ' % self.path - command = cd_action + command - - # The makefile rules are all relative to the top dir, but the gyp actions - # are defined relative to their containing dir. This replaces the gyp_* - # variables for the action rule with an absolute version so that the - # output goes in the right place. - # Only write the gyp_* rules for the "primary" output (:1); - # it's superfluous for the "extra outputs", and this avoids accidentally - # writing duplicate dummy rules for those outputs. - main_output = make.QuoteSpaces(self.LocalPathify(outputs[0])) - self.WriteLn('%s: gyp_local_path := $(LOCAL_PATH)' % main_output) - self.WriteLn('%s: gyp_var_prefix := $(GYP_VAR_PREFIX)' % main_output) - self.WriteLn('%s: gyp_intermediate_dir := ' - '$(abspath $(gyp_intermediate_dir))' % main_output) - self.WriteLn('%s: gyp_shared_intermediate_dir := ' - '$(abspath $(gyp_shared_intermediate_dir))' % main_output) - - # Android's envsetup.sh adds a number of directories to the path including - # the built host binary directory. This causes actions/rules invoked by - # gyp to sometimes use these instead of system versions, e.g. bison. - # The built host binaries may not be suitable, and can cause errors. - # So, we remove them from the PATH using the ANDROID_BUILD_PATHS variable - # set by envsetup. - self.WriteLn('%s: export PATH := $(subst $(ANDROID_BUILD_PATHS),,$(PATH))' - % main_output) - - # Don't allow spaces in input/output filenames, but make an exception for - # filenames which start with '$(' since it's okay for there to be spaces - # inside of make function/macro invocations. - for input in inputs: - if not input.startswith('$(') and ' ' in input: - raise gyp.common.GypError( - 'Action input filename "%s" in target %s contains a space' % - (input, self.target)) - for output in outputs: - if not output.startswith('$(') and ' ' in output: - raise gyp.common.GypError( - 'Action output filename "%s" in target %s contains a space' % - (output, self.target)) - - self.WriteLn('%s: %s $(GYP_TARGET_DEPENDENCIES)' % - (main_output, ' '.join(map(self.LocalPathify, inputs)))) - self.WriteLn('\t@echo "%s"' % quiet_cmd) - self.WriteLn('\t$(hide)%s\n' % command) - for output in outputs[1:]: - # Make each output depend on the main output, with an empty command - # to force make to notice that the mtime has changed. - self.WriteLn('%s: %s ;' % (self.LocalPathify(output), main_output)) - - extra_outputs += outputs - self.WriteLn() - - self.WriteLn() - - - def WriteRules(self, rules, extra_sources, extra_outputs): - """Write Makefile code for any 'rules' from the gyp input. + for action in actions: + name = make.StringToMakefileVariable( + "%s_%s" % (self.relative_target, action["action_name"]) + ) + self.WriteLn('### Rules for action "%s":' % action["action_name"]) + inputs = action["inputs"] + outputs = action["outputs"] + + # Build up a list of outputs. + # Collect the output dirs we'll need. + dirs = set() + for out in outputs: + if not out.startswith("$"): + print( + 'WARNING: Action for target "%s" writes output to local path ' + '"%s".' % (self.target, out) + ) + dir = os.path.split(out)[0] + if dir: + dirs.add(dir) + if int(action.get("process_outputs_as_sources", False)): + extra_sources += outputs + + # Prepare the actual command. + command = gyp.common.EncodePOSIXShellList(action["action"]) + if "message" in action: + quiet_cmd = "Gyp action: %s ($@)" % action["message"] + else: + quiet_cmd = "Gyp action: %s ($@)" % name + if len(dirs) > 0: + command = "mkdir -p %s" % " ".join(dirs) + "; " + command + + cd_action = "cd $(gyp_local_path)/%s; " % self.path + command = cd_action + command + + # The makefile rules are all relative to the top dir, but the gyp actions + # are defined relative to their containing dir. This replaces the gyp_* + # variables for the action rule with an absolute version so that the + # output goes in the right place. + # Only write the gyp_* rules for the "primary" output (:1); + # it's superfluous for the "extra outputs", and this avoids accidentally + # writing duplicate dummy rules for those outputs. + main_output = make.QuoteSpaces(self.LocalPathify(outputs[0])) + self.WriteLn("%s: gyp_local_path := $(LOCAL_PATH)" % main_output) + self.WriteLn("%s: gyp_var_prefix := $(GYP_VAR_PREFIX)" % main_output) + self.WriteLn( + "%s: gyp_intermediate_dir := " + "$(abspath $(gyp_intermediate_dir))" % main_output + ) + self.WriteLn( + "%s: gyp_shared_intermediate_dir := " + "$(abspath $(gyp_shared_intermediate_dir))" % main_output + ) + + # Android's envsetup.sh adds a number of directories to the path including + # the built host binary directory. This causes actions/rules invoked by + # gyp to sometimes use these instead of system versions, e.g. bison. + # The built host binaries may not be suitable, and can cause errors. + # So, we remove them from the PATH using the ANDROID_BUILD_PATHS variable + # set by envsetup. + self.WriteLn( + "%s: export PATH := $(subst $(ANDROID_BUILD_PATHS),,$(PATH))" + % main_output + ) + + # Don't allow spaces in input/output filenames, but make an exception for + # filenames which start with '$(' since it's okay for there to be spaces + # inside of make function/macro invocations. + for input in inputs: + if not input.startswith("$(") and " " in input: + raise gyp.common.GypError( + 'Action input filename "%s" in target %s contains a space' + % (input, self.target) + ) + for output in outputs: + if not output.startswith("$(") and " " in output: + raise gyp.common.GypError( + 'Action output filename "%s" in target %s contains a space' + % (output, self.target) + ) + + self.WriteLn( + "%s: %s $(GYP_TARGET_DEPENDENCIES)" + % (main_output, " ".join(map(self.LocalPathify, inputs))) + ) + self.WriteLn('\t@echo "%s"' % quiet_cmd) + self.WriteLn("\t$(hide)%s\n" % command) + for output in outputs[1:]: + # Make each output depend on the main output, with an empty command + # to force make to notice that the mtime has changed. + self.WriteLn("%s: %s ;" % (self.LocalPathify(output), main_output)) + + extra_outputs += outputs + self.WriteLn() + + self.WriteLn() + + def WriteRules(self, rules, extra_sources, extra_outputs): + """Write Makefile code for any 'rules' from the gyp input. extra_sources: a list that will be filled in with newly generated source files, if any extra_outputs: a list that will be filled in with any outputs of these rules (used to make other pieces dependent on these rules) """ - if len(rules) == 0: - return - - for rule in rules: - if len(rule.get('rule_sources', [])) == 0: - continue - name = make.StringToMakefileVariable('%s_%s' % (self.relative_target, - rule['rule_name'])) - self.WriteLn('\n### Generated for rule "%s":' % name) - self.WriteLn('# "%s":' % rule) - - inputs = rule.get('inputs') - for rule_source in rule.get('rule_sources', []): - (rule_source_dirname, rule_source_basename) = os.path.split(rule_source) - (rule_source_root, rule_source_ext) = \ - os.path.splitext(rule_source_basename) - - outputs = [self.ExpandInputRoot(out, rule_source_root, - rule_source_dirname) - for out in rule['outputs']] - - dirs = set() - for out in outputs: - if not out.startswith('$'): - print('WARNING: Rule for target %s writes output to local path %s' - % (self.target, out)) - dir = os.path.dirname(out) - if dir: - dirs.add(dir) - extra_outputs += outputs - if int(rule.get('process_outputs_as_sources', False)): - extra_sources.extend(outputs) - - components = [] - for component in rule['action']: - component = self.ExpandInputRoot(component, rule_source_root, - rule_source_dirname) - if '$(RULE_SOURCES)' in component: - component = component.replace('$(RULE_SOURCES)', - rule_source) - components.append(component) - - command = gyp.common.EncodePOSIXShellList(components) - cd_action = 'cd $(gyp_local_path)/%s; ' % self.path - command = cd_action + command - if dirs: - command = 'mkdir -p %s' % ' '.join(dirs) + '; ' + command - - # We set up a rule to build the first output, and then set up - # a rule for each additional output to depend on the first. - outputs = map(self.LocalPathify, outputs) - main_output = outputs[0] - self.WriteLn('%s: gyp_local_path := $(LOCAL_PATH)' % main_output) - self.WriteLn('%s: gyp_var_prefix := $(GYP_VAR_PREFIX)' % main_output) - self.WriteLn('%s: gyp_intermediate_dir := ' - '$(abspath $(gyp_intermediate_dir))' % main_output) - self.WriteLn('%s: gyp_shared_intermediate_dir := ' - '$(abspath $(gyp_shared_intermediate_dir))' % main_output) - - # See explanation in WriteActions. - self.WriteLn('%s: export PATH := ' - '$(subst $(ANDROID_BUILD_PATHS),,$(PATH))' % main_output) - - main_output_deps = self.LocalPathify(rule_source) - if inputs: - main_output_deps += ' ' - main_output_deps += ' '.join([self.LocalPathify(f) for f in inputs]) - - self.WriteLn('%s: %s $(GYP_TARGET_DEPENDENCIES)' % - (main_output, main_output_deps)) - self.WriteLn('\t%s\n' % command) - for output in outputs[1:]: - # Make each output depend on the main output, with an empty command - # to force make to notice that the mtime has changed. - self.WriteLn('%s: %s ;' % (output, main_output)) - self.WriteLn() - - self.WriteLn() + if len(rules) == 0: + return + + for rule in rules: + if len(rule.get("rule_sources", [])) == 0: + continue + name = make.StringToMakefileVariable( + "%s_%s" % (self.relative_target, rule["rule_name"]) + ) + self.WriteLn('\n### Generated for rule "%s":' % name) + self.WriteLn('# "%s":' % rule) + + inputs = rule.get("inputs") + for rule_source in rule.get("rule_sources", []): + (rule_source_dirname, rule_source_basename) = os.path.split(rule_source) + (rule_source_root, rule_source_ext) = os.path.splitext( + rule_source_basename + ) + + outputs = [ + self.ExpandInputRoot(out, rule_source_root, rule_source_dirname) + for out in rule["outputs"] + ] + + dirs = set() + for out in outputs: + if not out.startswith("$"): + print( + "WARNING: Rule for target %s writes output to local path %s" + % (self.target, out) + ) + dir = os.path.dirname(out) + if dir: + dirs.add(dir) + extra_outputs += outputs + if int(rule.get("process_outputs_as_sources", False)): + extra_sources.extend(outputs) + + components = [] + for component in rule["action"]: + component = self.ExpandInputRoot( + component, rule_source_root, rule_source_dirname + ) + if "$(RULE_SOURCES)" in component: + component = component.replace("$(RULE_SOURCES)", rule_source) + components.append(component) + + command = gyp.common.EncodePOSIXShellList(components) + cd_action = "cd $(gyp_local_path)/%s; " % self.path + command = cd_action + command + if dirs: + command = "mkdir -p %s" % " ".join(dirs) + "; " + command + + # We set up a rule to build the first output, and then set up + # a rule for each additional output to depend on the first. + outputs = map(self.LocalPathify, outputs) + main_output = outputs[0] + self.WriteLn("%s: gyp_local_path := $(LOCAL_PATH)" % main_output) + self.WriteLn("%s: gyp_var_prefix := $(GYP_VAR_PREFIX)" % main_output) + self.WriteLn( + "%s: gyp_intermediate_dir := " + "$(abspath $(gyp_intermediate_dir))" % main_output + ) + self.WriteLn( + "%s: gyp_shared_intermediate_dir := " + "$(abspath $(gyp_shared_intermediate_dir))" % main_output + ) + + # See explanation in WriteActions. + self.WriteLn( + "%s: export PATH := " + "$(subst $(ANDROID_BUILD_PATHS),,$(PATH))" % main_output + ) + + main_output_deps = self.LocalPathify(rule_source) + if inputs: + main_output_deps += " " + main_output_deps += " ".join([self.LocalPathify(f) for f in inputs]) + + self.WriteLn( + "%s: %s $(GYP_TARGET_DEPENDENCIES)" + % (main_output, main_output_deps) + ) + self.WriteLn("\t%s\n" % command) + for output in outputs[1:]: + # Make each output depend on the main output, with an empty command + # to force make to notice that the mtime has changed. + self.WriteLn("%s: %s ;" % (output, main_output)) + self.WriteLn() + self.WriteLn() - def WriteCopies(self, copies, extra_outputs): - """Write Makefile code for any 'copies' from the gyp input. + def WriteCopies(self, copies, extra_outputs): + """Write Makefile code for any 'copies' from the gyp input. extra_outputs: a list that will be filled in with any outputs of this action (used to make other pieces dependent on this action) """ - self.WriteLn('### Generated for copy rule.') - - variable = make.StringToMakefileVariable(self.relative_target + '_copies') - outputs = [] - for copy in copies: - for path in copy['files']: - # The Android build system does not allow generation of files into the - # source tree. The destination should start with a variable, which will - # typically be $(gyp_intermediate_dir) or - # $(gyp_shared_intermediate_dir). Note that we can't use an assertion - # because some of the gyp tests depend on this. - if not copy['destination'].startswith('$'): - print('WARNING: Copy rule for target %s writes output to ' - 'local path %s' % (self.target, copy['destination'])) - - # LocalPathify() calls normpath, stripping trailing slashes. - path = Sourceify(self.LocalPathify(path)) - filename = os.path.split(path)[1] - output = Sourceify(self.LocalPathify(os.path.join(copy['destination'], - filename))) - - self.WriteLn('%s: %s $(GYP_TARGET_DEPENDENCIES) | $(ACP)' % - (output, path)) - self.WriteLn('\t@echo Copying: $@') - self.WriteLn('\t$(hide) mkdir -p $(dir $@)') - self.WriteLn('\t$(hide) $(ACP) -rpf $< $@') + self.WriteLn("### Generated for copy rule.") + + variable = make.StringToMakefileVariable(self.relative_target + "_copies") + outputs = [] + for copy in copies: + for path in copy["files"]: + # The Android build system does not allow generation of files into the + # source tree. The destination should start with a variable, which will + # typically be $(gyp_intermediate_dir) or + # $(gyp_shared_intermediate_dir). Note that we can't use an assertion + # because some of the gyp tests depend on this. + if not copy["destination"].startswith("$"): + print( + "WARNING: Copy rule for target %s writes output to " + "local path %s" % (self.target, copy["destination"]) + ) + + # LocalPathify() calls normpath, stripping trailing slashes. + path = Sourceify(self.LocalPathify(path)) + filename = os.path.split(path)[1] + output = Sourceify( + self.LocalPathify(os.path.join(copy["destination"], filename)) + ) + + self.WriteLn( + "%s: %s $(GYP_TARGET_DEPENDENCIES) | $(ACP)" % (output, path) + ) + self.WriteLn("\t@echo Copying: $@") + self.WriteLn("\t$(hide) mkdir -p $(dir $@)") + self.WriteLn("\t$(hide) $(ACP) -rpf $< $@") + self.WriteLn() + outputs.append(output) + self.WriteLn("%s = %s" % (variable, " ".join(map(make.QuoteSpaces, outputs)))) + extra_outputs.append("$(%s)" % variable) self.WriteLn() - outputs.append(output) - self.WriteLn('%s = %s' % (variable, - ' '.join(map(make.QuoteSpaces, outputs)))) - extra_outputs.append('$(%s)' % variable) - self.WriteLn() - - def WriteSourceFlags(self, spec, configs): - """Write out the flags and include paths used to compile source files for + def WriteSourceFlags(self, spec, configs): + """Write out the flags and include paths used to compile source files for the current target. Args: spec, configs: input from gyp. """ - for configname, config in sorted(configs.items()): - extracted_includes = [] - - self.WriteLn('\n# Flags passed to both C and C++ files.') - cflags, includes_from_cflags = self.ExtractIncludesFromCFlags( - config.get('cflags', []) + config.get('cflags_c', [])) - extracted_includes.extend(includes_from_cflags) - self.WriteList(cflags, 'MY_CFLAGS_%s' % configname) - - self.WriteList(config.get('defines'), 'MY_DEFS_%s' % configname, - prefix='-D', quoter=make.EscapeCppDefine) - - self.WriteLn('\n# Include paths placed before CFLAGS/CPPFLAGS') - includes = list(config.get('include_dirs', [])) - includes.extend(extracted_includes) - includes = map(Sourceify, map(self.LocalPathify, includes)) - includes = self.NormalizeIncludePaths(includes) - self.WriteList(includes, 'LOCAL_C_INCLUDES_%s' % configname) - - self.WriteLn('\n# Flags passed to only C++ (and not C) files.') - self.WriteList(config.get('cflags_cc'), 'LOCAL_CPPFLAGS_%s' % configname) - - self.WriteLn('\nLOCAL_CFLAGS := $(MY_CFLAGS_$(GYP_CONFIGURATION)) ' - '$(MY_DEFS_$(GYP_CONFIGURATION))') - # Undefine ANDROID for host modules - # TODO: the source code should not use macro ANDROID to tell if it's host - # or target module. - if self.toolset == 'host': - self.WriteLn('# Undefine ANDROID for host modules') - self.WriteLn('LOCAL_CFLAGS += -UANDROID') - self.WriteLn('LOCAL_C_INCLUDES := $(GYP_COPIED_SOURCE_ORIGIN_DIRS) ' - '$(LOCAL_C_INCLUDES_$(GYP_CONFIGURATION))') - self.WriteLn('LOCAL_CPPFLAGS := $(LOCAL_CPPFLAGS_$(GYP_CONFIGURATION))') - # Android uses separate flags for assembly file invocations, but gyp expects - # the same CFLAGS to be applied: - self.WriteLn('LOCAL_ASFLAGS := $(LOCAL_CFLAGS)') - - - def WriteSources(self, spec, configs, extra_sources): - """Write Makefile code for any 'sources' from the gyp input. + for configname, config in sorted(configs.items()): + extracted_includes = [] + + self.WriteLn("\n# Flags passed to both C and C++ files.") + cflags, includes_from_cflags = self.ExtractIncludesFromCFlags( + config.get("cflags", []) + config.get("cflags_c", []) + ) + extracted_includes.extend(includes_from_cflags) + self.WriteList(cflags, "MY_CFLAGS_%s" % configname) + + self.WriteList( + config.get("defines"), + "MY_DEFS_%s" % configname, + prefix="-D", + quoter=make.EscapeCppDefine, + ) + + self.WriteLn("\n# Include paths placed before CFLAGS/CPPFLAGS") + includes = list(config.get("include_dirs", [])) + includes.extend(extracted_includes) + includes = map(Sourceify, map(self.LocalPathify, includes)) + includes = self.NormalizeIncludePaths(includes) + self.WriteList(includes, "LOCAL_C_INCLUDES_%s" % configname) + + self.WriteLn("\n# Flags passed to only C++ (and not C) files.") + self.WriteList(config.get("cflags_cc"), "LOCAL_CPPFLAGS_%s" % configname) + + self.WriteLn( + "\nLOCAL_CFLAGS := $(MY_CFLAGS_$(GYP_CONFIGURATION)) " + "$(MY_DEFS_$(GYP_CONFIGURATION))" + ) + # Undefine ANDROID for host modules + # TODO: the source code should not use macro ANDROID to tell if it's host + # or target module. + if self.toolset == "host": + self.WriteLn("# Undefine ANDROID for host modules") + self.WriteLn("LOCAL_CFLAGS += -UANDROID") + self.WriteLn( + "LOCAL_C_INCLUDES := $(GYP_COPIED_SOURCE_ORIGIN_DIRS) " + "$(LOCAL_C_INCLUDES_$(GYP_CONFIGURATION))" + ) + self.WriteLn("LOCAL_CPPFLAGS := $(LOCAL_CPPFLAGS_$(GYP_CONFIGURATION))") + # Android uses separate flags for assembly file invocations, but gyp expects + # the same CFLAGS to be applied: + self.WriteLn("LOCAL_ASFLAGS := $(LOCAL_CFLAGS)") + + def WriteSources(self, spec, configs, extra_sources): + """Write Makefile code for any 'sources' from the gyp input. These are source files necessary to build the current target. We need to handle shared_intermediate directory source files as a special case by copying them to the intermediate directory and @@ -510,187 +564,192 @@ def WriteSources(self, spec, configs, extra_sources): spec, configs: input from gyp. extra_sources: Sources generated from Actions or Rules. """ - sources = filter(make.Compilable, spec.get('sources', [])) - generated_not_sources = [x for x in extra_sources if not make.Compilable(x)] - extra_sources = filter(make.Compilable, extra_sources) - - # Determine and output the C++ extension used by these sources. - # We simply find the first C++ file and use that extension. - all_sources = sources + extra_sources - local_cpp_extension = '.cpp' - for source in all_sources: - (root, ext) = os.path.splitext(source) - if IsCPPExtension(ext): - local_cpp_extension = ext - break - if local_cpp_extension != '.cpp': - self.WriteLn('LOCAL_CPP_EXTENSION := %s' % local_cpp_extension) - - # We need to move any non-generated sources that are coming from the - # shared intermediate directory out of LOCAL_SRC_FILES and put them - # into LOCAL_GENERATED_SOURCES. We also need to move over any C++ files - # that don't match our local_cpp_extension, since Android will only - # generate Makefile rules for a single LOCAL_CPP_EXTENSION. - local_files = [] - for source in sources: - (root, ext) = os.path.splitext(source) - if '$(gyp_shared_intermediate_dir)' in source: - extra_sources.append(source) - elif '$(gyp_intermediate_dir)' in source: - extra_sources.append(source) - elif IsCPPExtension(ext) and ext != local_cpp_extension: - extra_sources.append(source) - else: - local_files.append(os.path.normpath(os.path.join(self.path, source))) - - # For any generated source, if it is coming from the shared intermediate - # directory then we add a Make rule to copy them to the local intermediate - # directory first. This is because the Android LOCAL_GENERATED_SOURCES - # must be in the local module intermediate directory for the compile rules - # to work properly. If the file has the wrong C++ extension, then we add - # a rule to copy that to intermediates and use the new version. - final_generated_sources = [] - # If a source file gets copied, we still need to add the original source - # directory as header search path, for GCC searches headers in the - # directory that contains the source file by default. - origin_src_dirs = [] - for source in extra_sources: - local_file = source - if not '$(gyp_intermediate_dir)/' in local_file: - basename = os.path.basename(local_file) - local_file = '$(gyp_intermediate_dir)/' + basename - (root, ext) = os.path.splitext(local_file) - if IsCPPExtension(ext) and ext != local_cpp_extension: - local_file = root + local_cpp_extension - if local_file != source: - self.WriteLn('%s: %s' % (local_file, self.LocalPathify(source))) - self.WriteLn('\tmkdir -p $(@D); cp $< $@') - origin_src_dirs.append(os.path.dirname(source)) - final_generated_sources.append(local_file) - - # We add back in all of the non-compilable stuff to make sure that the - # make rules have dependencies on them. - final_generated_sources.extend(generated_not_sources) - self.WriteList(final_generated_sources, 'LOCAL_GENERATED_SOURCES') - - origin_src_dirs = gyp.common.uniquer(origin_src_dirs) - origin_src_dirs = map(Sourceify, map(self.LocalPathify, origin_src_dirs)) - self.WriteList(origin_src_dirs, 'GYP_COPIED_SOURCE_ORIGIN_DIRS') - - self.WriteList(local_files, 'LOCAL_SRC_FILES') - - # Write out the flags used to compile the source; this must be done last - # so that GYP_COPIED_SOURCE_ORIGIN_DIRS can be used as an include path. - self.WriteSourceFlags(spec, configs) - - - def ComputeAndroidModule(self, spec): - """Return the Android module name used for a gyp spec. + sources = filter(make.Compilable, spec.get("sources", [])) + generated_not_sources = [x for x in extra_sources if not make.Compilable(x)] + extra_sources = filter(make.Compilable, extra_sources) + + # Determine and output the C++ extension used by these sources. + # We simply find the first C++ file and use that extension. + all_sources = sources + extra_sources + local_cpp_extension = ".cpp" + for source in all_sources: + (root, ext) = os.path.splitext(source) + if IsCPPExtension(ext): + local_cpp_extension = ext + break + if local_cpp_extension != ".cpp": + self.WriteLn("LOCAL_CPP_EXTENSION := %s" % local_cpp_extension) + + # We need to move any non-generated sources that are coming from the + # shared intermediate directory out of LOCAL_SRC_FILES and put them + # into LOCAL_GENERATED_SOURCES. We also need to move over any C++ files + # that don't match our local_cpp_extension, since Android will only + # generate Makefile rules for a single LOCAL_CPP_EXTENSION. + local_files = [] + for source in sources: + (root, ext) = os.path.splitext(source) + if "$(gyp_shared_intermediate_dir)" in source: + extra_sources.append(source) + elif "$(gyp_intermediate_dir)" in source: + extra_sources.append(source) + elif IsCPPExtension(ext) and ext != local_cpp_extension: + extra_sources.append(source) + else: + local_files.append(os.path.normpath(os.path.join(self.path, source))) + + # For any generated source, if it is coming from the shared intermediate + # directory then we add a Make rule to copy them to the local intermediate + # directory first. This is because the Android LOCAL_GENERATED_SOURCES + # must be in the local module intermediate directory for the compile rules + # to work properly. If the file has the wrong C++ extension, then we add + # a rule to copy that to intermediates and use the new version. + final_generated_sources = [] + # If a source file gets copied, we still need to add the original source + # directory as header search path, for GCC searches headers in the + # directory that contains the source file by default. + origin_src_dirs = [] + for source in extra_sources: + local_file = source + if "$(gyp_intermediate_dir)/" not in local_file: + basename = os.path.basename(local_file) + local_file = "$(gyp_intermediate_dir)/" + basename + (root, ext) = os.path.splitext(local_file) + if IsCPPExtension(ext) and ext != local_cpp_extension: + local_file = root + local_cpp_extension + if local_file != source: + self.WriteLn("%s: %s" % (local_file, self.LocalPathify(source))) + self.WriteLn("\tmkdir -p $(@D); cp $< $@") + origin_src_dirs.append(os.path.dirname(source)) + final_generated_sources.append(local_file) + + # We add back in all of the non-compilable stuff to make sure that the + # make rules have dependencies on them. + final_generated_sources.extend(generated_not_sources) + self.WriteList(final_generated_sources, "LOCAL_GENERATED_SOURCES") + + origin_src_dirs = gyp.common.uniquer(origin_src_dirs) + origin_src_dirs = map(Sourceify, map(self.LocalPathify, origin_src_dirs)) + self.WriteList(origin_src_dirs, "GYP_COPIED_SOURCE_ORIGIN_DIRS") + + self.WriteList(local_files, "LOCAL_SRC_FILES") + + # Write out the flags used to compile the source; this must be done last + # so that GYP_COPIED_SOURCE_ORIGIN_DIRS can be used as an include path. + self.WriteSourceFlags(spec, configs) + + def ComputeAndroidModule(self, spec): + """Return the Android module name used for a gyp spec. We use the complete qualified target name to avoid collisions between duplicate targets in different directories. We also add a suffix to distinguish gyp-generated module names. """ - if int(spec.get('android_unmangled_name', 0)): - assert self.type != 'shared_library' or self.target.startswith('lib') - return self.target - - if self.type == 'shared_library': - # For reasons of convention, the Android build system requires that all - # shared library modules are named 'libfoo' when generating -l flags. - prefix = 'lib_' - else: - prefix = '' + if int(spec.get("android_unmangled_name", 0)): + assert self.type != "shared_library" or self.target.startswith("lib") + return self.target - if spec['toolset'] == 'host': - suffix = '_$(TARGET_$(GYP_VAR_PREFIX)ARCH)_host_gyp' - else: - suffix = '_gyp' + if self.type == "shared_library": + # For reasons of convention, the Android build system requires that all + # shared library modules are named 'libfoo' when generating -l flags. + prefix = "lib_" + else: + prefix = "" - if self.path: - middle = make.StringToMakefileVariable('%s_%s' % (self.path, self.target)) - else: - middle = make.StringToMakefileVariable(self.target) + if spec["toolset"] == "host": + suffix = "_$(TARGET_$(GYP_VAR_PREFIX)ARCH)_host_gyp" + else: + suffix = "_gyp" - return ''.join([prefix, middle, suffix]) + if self.path: + middle = make.StringToMakefileVariable("%s_%s" % (self.path, self.target)) + else: + middle = make.StringToMakefileVariable(self.target) + return "".join([prefix, middle, suffix]) - def ComputeOutputParts(self, spec): - """Return the 'output basename' of a gyp spec, split into filename + ext. + def ComputeOutputParts(self, spec): + """Return the 'output basename' of a gyp spec, split into filename + ext. Android libraries must be named the same thing as their module name, otherwise the linker can't find them, so product_name and so on must be ignored if we are building a library, and the "lib" prepending is not done for Android. """ - assert self.type != 'loadable_module' # TODO: not supported? - - target = spec['target_name'] - target_prefix = '' - target_ext = '' - if self.type == 'static_library': - target = self.ComputeAndroidModule(spec) - target_ext = '.a' - elif self.type == 'shared_library': - target = self.ComputeAndroidModule(spec) - target_ext = '.so' - elif self.type == 'none': - target_ext = '.stamp' - elif self.type != 'executable': - print("ERROR: What output file should be generated?", - "type", self.type, "target", target) - - if self.type != 'static_library' and self.type != 'shared_library': - target_prefix = spec.get('product_prefix', target_prefix) - target = spec.get('product_name', target) - product_ext = spec.get('product_extension') - if product_ext: - target_ext = '.' + product_ext - - target_stem = target_prefix + target - return (target_stem, target_ext) - - - def ComputeOutputBasename(self, spec): - """Return the 'output basename' of a gyp spec. + assert self.type != "loadable_module" # TODO: not supported? + + target = spec["target_name"] + target_prefix = "" + target_ext = "" + if self.type == "static_library": + target = self.ComputeAndroidModule(spec) + target_ext = ".a" + elif self.type == "shared_library": + target = self.ComputeAndroidModule(spec) + target_ext = ".so" + elif self.type == "none": + target_ext = ".stamp" + elif self.type != "executable": + print( + "ERROR: What output file should be generated?", + "type", + self.type, + "target", + target, + ) + + if self.type != "static_library" and self.type != "shared_library": + target_prefix = spec.get("product_prefix", target_prefix) + target = spec.get("product_name", target) + product_ext = spec.get("product_extension") + if product_ext: + target_ext = "." + product_ext + + target_stem = target_prefix + target + return (target_stem, target_ext) + + def ComputeOutputBasename(self, spec): + """Return the 'output basename' of a gyp spec. E.g., the loadable module 'foobar' in directory 'baz' will produce 'libfoobar.so' """ - return ''.join(self.ComputeOutputParts(spec)) - + return "".join(self.ComputeOutputParts(spec)) - def ComputeOutput(self, spec): - """Return the 'output' (full output path) of a gyp spec. + def ComputeOutput(self, spec): + """Return the 'output' (full output path) of a gyp spec. E.g., the loadable module 'foobar' in directory 'baz' will produce '$(obj)/baz/libfoobar.so' """ - if self.type == 'executable': - # We install host executables into shared_intermediate_dir so they can be - # run by gyp rules that refer to PRODUCT_DIR. - path = '$(gyp_shared_intermediate_dir)' - elif self.type == 'shared_library': - if self.toolset == 'host': - path = '$($(GYP_HOST_VAR_PREFIX)HOST_OUT_INTERMEDIATE_LIBRARIES)' - else: - path = '$($(GYP_VAR_PREFIX)TARGET_OUT_INTERMEDIATE_LIBRARIES)' - else: - # Other targets just get built into their intermediate dir. - if self.toolset == 'host': - path = ('$(call intermediates-dir-for,%s,%s,true,,' - '$(GYP_HOST_VAR_PREFIX))' % (self.android_class, - self.android_module)) - else: - path = ('$(call intermediates-dir-for,%s,%s,,,$(GYP_VAR_PREFIX))' - % (self.android_class, self.android_module)) - - assert spec.get('product_dir') is None # TODO: not supported? - return os.path.join(path, self.ComputeOutputBasename(spec)) - - def NormalizeIncludePaths(self, include_paths): - """ Normalize include_paths. + if self.type == "executable": + # We install host executables into shared_intermediate_dir so they can be + # run by gyp rules that refer to PRODUCT_DIR. + path = "$(gyp_shared_intermediate_dir)" + elif self.type == "shared_library": + if self.toolset == "host": + path = "$($(GYP_HOST_VAR_PREFIX)HOST_OUT_INTERMEDIATE_LIBRARIES)" + else: + path = "$($(GYP_VAR_PREFIX)TARGET_OUT_INTERMEDIATE_LIBRARIES)" + else: + # Other targets just get built into their intermediate dir. + if self.toolset == "host": + path = ( + "$(call intermediates-dir-for,%s,%s,true,," + "$(GYP_HOST_VAR_PREFIX))" + % (self.android_class, self.android_module) + ) + else: + path = "$(call intermediates-dir-for,%s,%s,,,$(GYP_VAR_PREFIX))" % ( + self.android_class, + self.android_module, + ) + + assert spec.get("product_dir") is None # TODO: not supported? + return os.path.join(path, self.ComputeOutputBasename(spec)) + + def NormalizeIncludePaths(self, include_paths): + """ Normalize include_paths. Convert absolute paths to relative to the Android top directory. Args: @@ -698,33 +757,33 @@ def NormalizeIncludePaths(self, include_paths): Returns: A list of normalized include paths. """ - normalized = [] - for path in include_paths: - if path[0] == '/': - path = gyp.common.RelativePath(path, self.android_top_dir) - normalized.append(path) - return normalized + normalized = [] + for path in include_paths: + if path[0] == "/": + path = gyp.common.RelativePath(path, self.android_top_dir) + normalized.append(path) + return normalized - def ExtractIncludesFromCFlags(self, cflags): - """Extract includes "-I..." out from cflags + def ExtractIncludesFromCFlags(self, cflags): + """Extract includes "-I..." out from cflags Args: cflags: A list of compiler flags, which may be mixed with "-I.." Returns: A tuple of lists: (clean_clfags, include_paths). "-I.." is trimmed. """ - clean_cflags = [] - include_paths = [] - for flag in cflags: - if flag.startswith('-I'): - include_paths.append(flag[2:]) - else: - clean_cflags.append(flag) + clean_cflags = [] + include_paths = [] + for flag in cflags: + if flag.startswith("-I"): + include_paths.append(flag[2:]) + else: + clean_cflags.append(flag) - return (clean_cflags, include_paths) + return (clean_cflags, include_paths) - def FilterLibraries(self, libraries): - """Filter the 'libraries' key to separate things that shouldn't be ldflags. + def FilterLibraries(self, libraries): + """Filter the 'libraries' key to separate things that shouldn't be ldflags. Library entries that look like filenames should be converted to android module names instead of being passed to the linker as flags. @@ -734,96 +793,103 @@ def FilterLibraries(self, libraries): Returns: A tuple (static_lib_modules, dynamic_lib_modules, ldflags) """ - static_lib_modules = [] - dynamic_lib_modules = [] - ldflags = [] - for libs in libraries: - # Libs can have multiple words. - for lib in libs.split(): - # Filter the system libraries, which are added by default by the Android - # build system. - if (lib == '-lc' or lib == '-lstdc++' or lib == '-lm' or - lib.endswith('libgcc.a')): - continue - match = re.search(r'([^/]+)\.a$', lib) - if match: - static_lib_modules.append(match.group(1)) - continue - match = re.search(r'([^/]+)\.so$', lib) - if match: - dynamic_lib_modules.append(match.group(1)) - continue - if lib.startswith('-l'): - ldflags.append(lib) - return (static_lib_modules, dynamic_lib_modules, ldflags) - - - def ComputeDeps(self, spec): - """Compute the dependencies of a gyp spec. + static_lib_modules = [] + dynamic_lib_modules = [] + ldflags = [] + for libs in libraries: + # Libs can have multiple words. + for lib in libs.split(): + # Filter the system libraries, which are added by default by the Android + # build system. + if ( + lib == "-lc" + or lib == "-lstdc++" + or lib == "-lm" + or lib.endswith("libgcc.a") + ): + continue + match = re.search(r"([^/]+)\.a$", lib) + if match: + static_lib_modules.append(match.group(1)) + continue + match = re.search(r"([^/]+)\.so$", lib) + if match: + dynamic_lib_modules.append(match.group(1)) + continue + if lib.startswith("-l"): + ldflags.append(lib) + return (static_lib_modules, dynamic_lib_modules, ldflags) + + def ComputeDeps(self, spec): + """Compute the dependencies of a gyp spec. Returns a tuple (deps, link_deps), where each is a list of filenames that will need to be put in front of make for either building (deps) or linking (link_deps). """ - deps = [] - link_deps = [] - if 'dependencies' in spec: - deps.extend([target_outputs[dep] for dep in spec['dependencies'] - if target_outputs[dep]]) - for dep in spec['dependencies']: - if dep in target_link_deps: - link_deps.append(target_link_deps[dep]) - deps.extend(link_deps) - return (gyp.common.uniquer(deps), gyp.common.uniquer(link_deps)) - - - def WriteTargetFlags(self, spec, configs, link_deps): - """Write Makefile code to specify the link flags and library dependencies. + deps = [] + link_deps = [] + if "dependencies" in spec: + deps.extend( + [ + target_outputs[dep] + for dep in spec["dependencies"] + if target_outputs[dep] + ] + ) + for dep in spec["dependencies"]: + if dep in target_link_deps: + link_deps.append(target_link_deps[dep]) + deps.extend(link_deps) + return (gyp.common.uniquer(deps), gyp.common.uniquer(link_deps)) + + def WriteTargetFlags(self, spec, configs, link_deps): + """Write Makefile code to specify the link flags and library dependencies. spec, configs: input from gyp. link_deps: link dependency list; see ComputeDeps() """ - # Libraries (i.e. -lfoo) - # These must be included even for static libraries as some of them provide - # implicit include paths through the build system. - libraries = gyp.common.uniquer(spec.get('libraries', [])) - static_libs, dynamic_libs, ldflags_libs = self.FilterLibraries(libraries) - - if self.type != 'static_library': - for configname, config in sorted(configs.items()): - ldflags = list(config.get('ldflags', [])) - self.WriteLn('') - self.WriteList(ldflags, 'LOCAL_LDFLAGS_%s' % configname) - self.WriteList(ldflags_libs, 'LOCAL_GYP_LIBS') - self.WriteLn('LOCAL_LDFLAGS := $(LOCAL_LDFLAGS_$(GYP_CONFIGURATION)) ' - '$(LOCAL_GYP_LIBS)') - - # Link dependencies (i.e. other gyp targets this target depends on) - # These need not be included for static libraries as within the gyp build - # we do not use the implicit include path mechanism. - if self.type != 'static_library': - static_link_deps = [x[1] for x in link_deps if x[0] == 'static'] - shared_link_deps = [x[1] for x in link_deps if x[0] == 'shared'] - else: - static_link_deps = [] - shared_link_deps = [] - - # Only write the lists if they are non-empty. - if static_libs or static_link_deps: - self.WriteLn('') - self.WriteList(static_libs + static_link_deps, - 'LOCAL_STATIC_LIBRARIES') - self.WriteLn('# Enable grouping to fix circular references') - self.WriteLn('LOCAL_GROUP_STATIC_LIBRARIES := true') - if dynamic_libs or shared_link_deps: - self.WriteLn('') - self.WriteList(dynamic_libs + shared_link_deps, - 'LOCAL_SHARED_LIBRARIES') - - - def WriteTarget(self, spec, configs, deps, link_deps, part_of_all, - write_alias_target): - """Write Makefile code to produce the final target of the gyp spec. + # Libraries (i.e. -lfoo) + # These must be included even for static libraries as some of them provide + # implicit include paths through the build system. + libraries = gyp.common.uniquer(spec.get("libraries", [])) + static_libs, dynamic_libs, ldflags_libs = self.FilterLibraries(libraries) + + if self.type != "static_library": + for configname, config in sorted(configs.items()): + ldflags = list(config.get("ldflags", [])) + self.WriteLn("") + self.WriteList(ldflags, "LOCAL_LDFLAGS_%s" % configname) + self.WriteList(ldflags_libs, "LOCAL_GYP_LIBS") + self.WriteLn( + "LOCAL_LDFLAGS := $(LOCAL_LDFLAGS_$(GYP_CONFIGURATION)) " + "$(LOCAL_GYP_LIBS)" + ) + + # Link dependencies (i.e. other gyp targets this target depends on) + # These need not be included for static libraries as within the gyp build + # we do not use the implicit include path mechanism. + if self.type != "static_library": + static_link_deps = [x[1] for x in link_deps if x[0] == "static"] + shared_link_deps = [x[1] for x in link_deps if x[0] == "shared"] + else: + static_link_deps = [] + shared_link_deps = [] + + # Only write the lists if they are non-empty. + if static_libs or static_link_deps: + self.WriteLn("") + self.WriteList(static_libs + static_link_deps, "LOCAL_STATIC_LIBRARIES") + self.WriteLn("# Enable grouping to fix circular references") + self.WriteLn("LOCAL_GROUP_STATIC_LIBRARIES := true") + if dynamic_libs or shared_link_deps: + self.WriteLn("") + self.WriteList(dynamic_libs + shared_link_deps, "LOCAL_SHARED_LIBRARIES") + + def WriteTarget( + self, spec, configs, deps, link_deps, part_of_all, write_alias_target + ): + """Write Makefile code to produce the final target of the gyp spec. spec, configs: input from gyp. deps, link_deps: dependency lists; see ComputeDeps() @@ -831,267 +897,278 @@ def WriteTarget(self, spec, configs, deps, link_deps, part_of_all, write_alias_target: flag indicating whether to create short aliases for this target """ - self.WriteLn('### Rules for final target.') - - if self.type != 'none': - self.WriteTargetFlags(spec, configs, link_deps) - - settings = spec.get('aosp_build_settings', {}) - if settings: - self.WriteLn('### Set directly by aosp_build_settings.') - for k, v in settings.items(): - if isinstance(v, list): - self.WriteList(v, k) + self.WriteLn("### Rules for final target.") + + if self.type != "none": + self.WriteTargetFlags(spec, configs, link_deps) + + settings = spec.get("aosp_build_settings", {}) + if settings: + self.WriteLn("### Set directly by aosp_build_settings.") + for k, v in settings.items(): + if isinstance(v, list): + self.WriteList(v, k) + else: + self.WriteLn("%s := %s" % (k, make.QuoteIfNecessary(v))) + self.WriteLn("") + + # Add to the set of targets which represent the gyp 'all' target. We use the + # name 'gyp_all_modules' as the Android build system doesn't allow the use + # of the Make target 'all' and because 'all_modules' is the equivalent of + # the Make target 'all' on Android. + if part_of_all and write_alias_target: + self.WriteLn('# Add target alias to "gyp_all_modules" target.') + self.WriteLn(".PHONY: gyp_all_modules") + self.WriteLn("gyp_all_modules: %s" % self.android_module) + self.WriteLn("") + + # Add an alias from the gyp target name to the Android module name. This + # simplifies manual builds of the target, and is required by the test + # framework. + if self.target != self.android_module and write_alias_target: + self.WriteLn("# Alias gyp target name.") + self.WriteLn(".PHONY: %s" % self.target) + self.WriteLn("%s: %s" % (self.target, self.android_module)) + self.WriteLn("") + + # Add the command to trigger build of the target type depending + # on the toolset. Ex: BUILD_STATIC_LIBRARY vs. BUILD_HOST_STATIC_LIBRARY + # NOTE: This has to come last! + modifier = "" + if self.toolset == "host": + modifier = "HOST_" + if self.type == "static_library": + self.WriteLn("include $(BUILD_%sSTATIC_LIBRARY)" % modifier) + elif self.type == "shared_library": + self.WriteLn("LOCAL_PRELINK_MODULE := false") + self.WriteLn("include $(BUILD_%sSHARED_LIBRARY)" % modifier) + elif self.type == "executable": + self.WriteLn("LOCAL_CXX_STL := libc++_static") + # Executables are for build and test purposes only, so they're installed + # to a directory that doesn't get included in the system image. + self.WriteLn("LOCAL_MODULE_PATH := $(gyp_shared_intermediate_dir)") + self.WriteLn("include $(BUILD_%sEXECUTABLE)" % modifier) else: - self.WriteLn('%s := %s' % (k, make.QuoteIfNecessary(v))) - self.WriteLn('') - - # Add to the set of targets which represent the gyp 'all' target. We use the - # name 'gyp_all_modules' as the Android build system doesn't allow the use - # of the Make target 'all' and because 'all_modules' is the equivalent of - # the Make target 'all' on Android. - if part_of_all and write_alias_target: - self.WriteLn('# Add target alias to "gyp_all_modules" target.') - self.WriteLn('.PHONY: gyp_all_modules') - self.WriteLn('gyp_all_modules: %s' % self.android_module) - self.WriteLn('') - - # Add an alias from the gyp target name to the Android module name. This - # simplifies manual builds of the target, and is required by the test - # framework. - if self.target != self.android_module and write_alias_target: - self.WriteLn('# Alias gyp target name.') - self.WriteLn('.PHONY: %s' % self.target) - self.WriteLn('%s: %s' % (self.target, self.android_module)) - self.WriteLn('') - - # Add the command to trigger build of the target type depending - # on the toolset. Ex: BUILD_STATIC_LIBRARY vs. BUILD_HOST_STATIC_LIBRARY - # NOTE: This has to come last! - modifier = '' - if self.toolset == 'host': - modifier = 'HOST_' - if self.type == 'static_library': - self.WriteLn('include $(BUILD_%sSTATIC_LIBRARY)' % modifier) - elif self.type == 'shared_library': - self.WriteLn('LOCAL_PRELINK_MODULE := false') - self.WriteLn('include $(BUILD_%sSHARED_LIBRARY)' % modifier) - elif self.type == 'executable': - self.WriteLn('LOCAL_CXX_STL := libc++_static') - # Executables are for build and test purposes only, so they're installed - # to a directory that doesn't get included in the system image. - self.WriteLn('LOCAL_MODULE_PATH := $(gyp_shared_intermediate_dir)') - self.WriteLn('include $(BUILD_%sEXECUTABLE)' % modifier) - else: - self.WriteLn('LOCAL_MODULE_PATH := $(PRODUCT_OUT)/gyp_stamp') - self.WriteLn('LOCAL_UNINSTALLABLE_MODULE := true') - if self.toolset == 'target': - self.WriteLn('LOCAL_2ND_ARCH_VAR_PREFIX := $(GYP_VAR_PREFIX)') - else: - self.WriteLn('LOCAL_2ND_ARCH_VAR_PREFIX := $(GYP_HOST_VAR_PREFIX)') - self.WriteLn() - self.WriteLn('include $(BUILD_SYSTEM)/base_rules.mk') - self.WriteLn() - self.WriteLn('$(LOCAL_BUILT_MODULE): $(LOCAL_ADDITIONAL_DEPENDENCIES)') - self.WriteLn('\t$(hide) echo "Gyp timestamp: $@"') - self.WriteLn('\t$(hide) mkdir -p $(dir $@)') - self.WriteLn('\t$(hide) touch $@') - self.WriteLn() - self.WriteLn('LOCAL_2ND_ARCH_VAR_PREFIX :=') - - - def WriteList(self, value_list, variable=None, prefix='', - quoter=make.QuoteIfNecessary, local_pathify=False): - """Write a variable definition that is a list of values. + self.WriteLn("LOCAL_MODULE_PATH := $(PRODUCT_OUT)/gyp_stamp") + self.WriteLn("LOCAL_UNINSTALLABLE_MODULE := true") + if self.toolset == "target": + self.WriteLn("LOCAL_2ND_ARCH_VAR_PREFIX := $(GYP_VAR_PREFIX)") + else: + self.WriteLn("LOCAL_2ND_ARCH_VAR_PREFIX := $(GYP_HOST_VAR_PREFIX)") + self.WriteLn() + self.WriteLn("include $(BUILD_SYSTEM)/base_rules.mk") + self.WriteLn() + self.WriteLn("$(LOCAL_BUILT_MODULE): $(LOCAL_ADDITIONAL_DEPENDENCIES)") + self.WriteLn('\t$(hide) echo "Gyp timestamp: $@"') + self.WriteLn("\t$(hide) mkdir -p $(dir $@)") + self.WriteLn("\t$(hide) touch $@") + self.WriteLn() + self.WriteLn("LOCAL_2ND_ARCH_VAR_PREFIX :=") + + def WriteList( + self, + value_list, + variable=None, + prefix="", + quoter=make.QuoteIfNecessary, + local_pathify=False, + ): + """Write a variable definition that is a list of values. E.g. WriteList(['a','b'], 'foo', prefix='blah') writes out foo = blaha blahb but in a pretty-printed style. """ - values = '' - if value_list: - value_list = [quoter(prefix + l) for l in value_list] - if local_pathify: - value_list = [self.LocalPathify(l) for l in value_list] - values = ' \\\n\t' + ' \\\n\t'.join(value_list) - self.fp.write('%s :=%s\n\n' % (variable, values)) - - - def WriteLn(self, text=''): - self.fp.write(text + '\n') - - - def LocalPathify(self, path): - """Convert a subdirectory-relative path into a normalized path which starts + values = "" + if value_list: + value_list = [quoter(prefix + l) for l in value_list] + if local_pathify: + value_list = [self.LocalPathify(l) for l in value_list] + values = " \\\n\t" + " \\\n\t".join(value_list) + self.fp.write("%s :=%s\n\n" % (variable, values)) + + def WriteLn(self, text=""): + self.fp.write(text + "\n") + + def LocalPathify(self, path): + """Convert a subdirectory-relative path into a normalized path which starts with the make variable $(LOCAL_PATH) (i.e. the top of the project tree). Absolute paths, or paths that contain variables, are just normalized.""" - if '$(' in path or os.path.isabs(path): - # path is not a file in the project tree in this case, but calling - # normpath is still important for trimming trailing slashes. - return os.path.normpath(path) - local_path = os.path.join('$(LOCAL_PATH)', self.path, path) - local_path = os.path.normpath(local_path) - # Check that normalizing the path didn't ../ itself out of $(LOCAL_PATH) - # - i.e. that the resulting path is still inside the project tree. The - # path may legitimately have ended up containing just $(LOCAL_PATH), though, - # so we don't look for a slash. - assert local_path.startswith('$(LOCAL_PATH)'), ( - 'Path %s attempts to escape from gyp path %s !)' % (path, self.path)) - return local_path - - - def ExpandInputRoot(self, template, expansion, dirname): - if '%(INPUT_ROOT)s' not in template and '%(INPUT_DIRNAME)s' not in template: - return template - path = template % { - 'INPUT_ROOT': expansion, - 'INPUT_DIRNAME': dirname, + if "$(" in path or os.path.isabs(path): + # path is not a file in the project tree in this case, but calling + # normpath is still important for trimming trailing slashes. + return os.path.normpath(path) + local_path = os.path.join("$(LOCAL_PATH)", self.path, path) + local_path = os.path.normpath(local_path) + # Check that normalizing the path didn't ../ itself out of $(LOCAL_PATH) + # - i.e. that the resulting path is still inside the project tree. The + # path may legitimately have ended up containing just $(LOCAL_PATH), though, + # so we don't look for a slash. + assert local_path.startswith( + "$(LOCAL_PATH)" + ), "Path %s attempts to escape from gyp path %s !)" % (path, self.path) + return local_path + + def ExpandInputRoot(self, template, expansion, dirname): + if "%(INPUT_ROOT)s" not in template and "%(INPUT_DIRNAME)s" not in template: + return template + path = template % { + "INPUT_ROOT": expansion, + "INPUT_DIRNAME": dirname, } - return os.path.normpath(path) + return os.path.normpath(path) def PerformBuild(data, configurations, params): - # The android backend only supports the default configuration. - options = params['options'] - makefile = os.path.abspath(os.path.join(options.toplevel_dir, - 'GypAndroid.mk')) - env = dict(os.environ) - env['ONE_SHOT_MAKEFILE'] = makefile - arguments = ['make', '-C', os.environ['ANDROID_BUILD_TOP'], 'gyp_all_modules'] - print('Building: %s' % arguments) - subprocess.check_call(arguments, env=env) + # The android backend only supports the default configuration. + options = params["options"] + makefile = os.path.abspath(os.path.join(options.toplevel_dir, "GypAndroid.mk")) + env = dict(os.environ) + env["ONE_SHOT_MAKEFILE"] = makefile + arguments = ["make", "-C", os.environ["ANDROID_BUILD_TOP"], "gyp_all_modules"] + print("Building: %s" % arguments) + subprocess.check_call(arguments, env=env) def GenerateOutput(target_list, target_dicts, data, params): - options = params['options'] - generator_flags = params.get('generator_flags', {}) - builddir_name = generator_flags.get('output_dir', 'out') - limit_to_target_all = generator_flags.get('limit_to_target_all', False) - write_alias_targets = generator_flags.get('write_alias_targets', True) - sdk_version = generator_flags.get('aosp_sdk_version', 0) - android_top_dir = os.environ.get('ANDROID_BUILD_TOP') - assert android_top_dir, '$ANDROID_BUILD_TOP not set; you need to run lunch.' - - def CalculateMakefilePath(build_file, base_name): - """Determine where to write a Makefile for a given gyp file.""" - # Paths in gyp files are relative to the .gyp file, but we want - # paths relative to the source root for the master makefile. Grab - # the path of the .gyp file as the base to relativize against. - # E.g. "foo/bar" when we're constructing targets for "foo/bar/baz.gyp". - base_path = gyp.common.RelativePath(os.path.dirname(build_file), - options.depth) - # We write the file in the base_path directory. - output_file = os.path.join(options.depth, base_path, base_name) - assert not options.generator_output, ( - 'The Android backend does not support options.generator_output.') - base_path = gyp.common.RelativePath(os.path.dirname(build_file), - options.toplevel_dir) - return base_path, output_file - - # TODO: search for the first non-'Default' target. This can go - # away when we add verification that all targets have the - # necessary configurations. - default_configuration = None - toolsets = set([target_dicts[target]['toolset'] for target in target_list]) - for target in target_list: - spec = target_dicts[target] - if spec['default_configuration'] != 'Default': - default_configuration = spec['default_configuration'] - break - if not default_configuration: - default_configuration = 'Default' - - srcdir = '.' - makefile_name = 'GypAndroid' + options.suffix + '.mk' - makefile_path = os.path.join(options.toplevel_dir, makefile_name) - assert not options.generator_output, ( - 'The Android backend does not support options.generator_output.') - gyp.common.EnsureDirExists(makefile_path) - root_makefile = open(makefile_path, 'w') - - root_makefile.write(header) - - # We set LOCAL_PATH just once, here, to the top of the project tree. This - # allows all the other paths we use to be relative to the Android.mk file, - # as the Android build system expects. - root_makefile.write('\nLOCAL_PATH := $(call my-dir)\n') - - # Find the list of targets that derive from the gyp file(s) being built. - needed_targets = set() - for build_file in params['build_files']: - for target in gyp.common.AllTargets(target_list, target_dicts, build_file): - needed_targets.add(target) - - build_files = set() - include_list = set() - android_modules = {} - for qualified_target in target_list: - build_file, target, toolset = gyp.common.ParseQualifiedTarget( - qualified_target) - relative_build_file = gyp.common.RelativePath(build_file, - options.toplevel_dir) - build_files.add(relative_build_file) - included_files = data[build_file]['included_files'] - for included_file in included_files: - # The included_files entries are relative to the dir of the build file - # that included them, so we have to undo that and then make them relative - # to the root dir. - relative_include_file = gyp.common.RelativePath( - gyp.common.UnrelativePath(included_file, build_file), - options.toplevel_dir) - abs_include_file = os.path.abspath(relative_include_file) - # If the include file is from the ~/.gyp dir, we should use absolute path - # so that relocating the src dir doesn't break the path. - if (params['home_dot_gyp'] and - abs_include_file.startswith(params['home_dot_gyp'])): - build_files.add(abs_include_file) - else: - build_files.add(relative_include_file) - - base_path, output_file = CalculateMakefilePath(build_file, - target + '.' + toolset + options.suffix + '.mk') - - spec = target_dicts[qualified_target] - configs = spec['configurations'] - - part_of_all = qualified_target in needed_targets - if limit_to_target_all and not part_of_all: - continue - - relative_target = gyp.common.QualifiedTarget(relative_build_file, target, - toolset) - writer = AndroidMkWriter(android_top_dir) - android_module = writer.Write(qualified_target, relative_target, base_path, - output_file, spec, configs, - part_of_all=part_of_all, - write_alias_target=write_alias_targets, - sdk_version=sdk_version) - if android_module in android_modules: - print('ERROR: Android module names must be unique. The following ' - 'targets both generate Android module name %s.\n %s\n %s' % - (android_module, android_modules[android_module], - qualified_target)) - return - android_modules[android_module] = qualified_target - - # Our root_makefile lives at the source root. Compute the relative path - # from there to the output_file for including. - mkfile_rel_path = gyp.common.RelativePath(output_file, - os.path.dirname(makefile_path)) - include_list.add(mkfile_rel_path) - - root_makefile.write('GYP_CONFIGURATION ?= %s\n' % default_configuration) - root_makefile.write('GYP_VAR_PREFIX ?=\n') - root_makefile.write('GYP_HOST_VAR_PREFIX ?=\n') - root_makefile.write('GYP_HOST_MULTILIB ?= first\n') - - # Write out the sorted list of includes. - root_makefile.write('\n') - for include_file in sorted(include_list): - root_makefile.write('include $(LOCAL_PATH)/' + include_file + '\n') - root_makefile.write('\n') - - if write_alias_targets: - root_makefile.write(ALL_MODULES_FOOTER) - - root_makefile.close() + options = params["options"] + generator_flags = params.get("generator_flags", {}) + limit_to_target_all = generator_flags.get("limit_to_target_all", False) + write_alias_targets = generator_flags.get("write_alias_targets", True) + sdk_version = generator_flags.get("aosp_sdk_version", 0) + android_top_dir = os.environ.get("ANDROID_BUILD_TOP") + assert android_top_dir, "$ANDROID_BUILD_TOP not set; you need to run lunch." + + def CalculateMakefilePath(build_file, base_name): + """Determine where to write a Makefile for a given gyp file.""" + # Paths in gyp files are relative to the .gyp file, but we want + # paths relative to the source root for the master makefile. Grab + # the path of the .gyp file as the base to relativize against. + # E.g. "foo/bar" when we're constructing targets for "foo/bar/baz.gyp". + base_path = gyp.common.RelativePath(os.path.dirname(build_file), options.depth) + # We write the file in the base_path directory. + output_file = os.path.join(options.depth, base_path, base_name) + assert ( + not options.generator_output + ), "The Android backend does not support options.generator_output." + base_path = gyp.common.RelativePath( + os.path.dirname(build_file), options.toplevel_dir + ) + return base_path, output_file + + # TODO: search for the first non-'Default' target. This can go + # away when we add verification that all targets have the + # necessary configurations. + default_configuration = None + for target in target_list: + spec = target_dicts[target] + if spec["default_configuration"] != "Default": + default_configuration = spec["default_configuration"] + break + if not default_configuration: + default_configuration = "Default" + + makefile_name = "GypAndroid" + options.suffix + ".mk" + makefile_path = os.path.join(options.toplevel_dir, makefile_name) + assert ( + not options.generator_output + ), "The Android backend does not support options.generator_output." + gyp.common.EnsureDirExists(makefile_path) + root_makefile = open(makefile_path, "w") + + root_makefile.write(header) + + # We set LOCAL_PATH just once, here, to the top of the project tree. This + # allows all the other paths we use to be relative to the Android.mk file, + # as the Android build system expects. + root_makefile.write("\nLOCAL_PATH := $(call my-dir)\n") + + # Find the list of targets that derive from the gyp file(s) being built. + needed_targets = set() + for build_file in params["build_files"]: + for target in gyp.common.AllTargets(target_list, target_dicts, build_file): + needed_targets.add(target) + + build_files = set() + include_list = set() + android_modules = {} + for qualified_target in target_list: + build_file, target, toolset = gyp.common.ParseQualifiedTarget(qualified_target) + relative_build_file = gyp.common.RelativePath(build_file, options.toplevel_dir) + build_files.add(relative_build_file) + included_files = data[build_file]["included_files"] + for included_file in included_files: + # The included_files entries are relative to the dir of the build file + # that included them, so we have to undo that and then make them relative + # to the root dir. + relative_include_file = gyp.common.RelativePath( + gyp.common.UnrelativePath(included_file, build_file), + options.toplevel_dir, + ) + abs_include_file = os.path.abspath(relative_include_file) + # If the include file is from the ~/.gyp dir, we should use absolute path + # so that relocating the src dir doesn't break the path. + if params["home_dot_gyp"] and abs_include_file.startswith( + params["home_dot_gyp"] + ): + build_files.add(abs_include_file) + else: + build_files.add(relative_include_file) + + base_path, output_file = CalculateMakefilePath( + build_file, target + "." + toolset + options.suffix + ".mk" + ) + + spec = target_dicts[qualified_target] + configs = spec["configurations"] + + part_of_all = qualified_target in needed_targets + if limit_to_target_all and not part_of_all: + continue + + relative_target = gyp.common.QualifiedTarget( + relative_build_file, target, toolset + ) + writer = AndroidMkWriter(android_top_dir) + android_module = writer.Write( + qualified_target, + relative_target, + base_path, + output_file, + spec, + configs, + part_of_all=part_of_all, + write_alias_target=write_alias_targets, + sdk_version=sdk_version, + ) + if android_module in android_modules: + print( + "ERROR: Android module names must be unique. The following " + "targets both generate Android module name %s.\n %s\n %s" + % (android_module, android_modules[android_module], qualified_target) + ) + return + android_modules[android_module] = qualified_target + + # Our root_makefile lives at the source root. Compute the relative path + # from there to the output_file for including. + mkfile_rel_path = gyp.common.RelativePath( + output_file, os.path.dirname(makefile_path) + ) + include_list.add(mkfile_rel_path) + + root_makefile.write("GYP_CONFIGURATION ?= %s\n" % default_configuration) + root_makefile.write("GYP_VAR_PREFIX ?=\n") + root_makefile.write("GYP_HOST_VAR_PREFIX ?=\n") + root_makefile.write("GYP_HOST_MULTILIB ?= first\n") + + # Write out the sorted list of includes. + root_makefile.write("\n") + for include_file in sorted(include_list): + root_makefile.write("include $(LOCAL_PATH)/" + include_file + "\n") + root_makefile.write("\n") + + if write_alias_targets: + root_makefile.write(ALL_MODULES_FOOTER) + + root_makefile.close() diff --git a/gyp/pylib/gyp/generator/cmake.py b/gyp/pylib/gyp/generator/cmake.py index e966a8f23e..f5ceacfca3 100644 --- a/gyp/pylib/gyp/generator/cmake.py +++ b/gyp/pylib/gyp/generator/cmake.py @@ -38,91 +38,98 @@ import gyp.common import gyp.xcode_emulation +try: + # maketrans moved to str in python3. + _maketrans = string.maketrans +except NameError: + _maketrans = str.maketrans + generator_default_variables = { - 'EXECUTABLE_PREFIX': '', - 'EXECUTABLE_SUFFIX': '', - 'STATIC_LIB_PREFIX': 'lib', - 'STATIC_LIB_SUFFIX': '.a', - 'SHARED_LIB_PREFIX': 'lib', - 'SHARED_LIB_SUFFIX': '.so', - 'SHARED_LIB_DIR': '${builddir}/lib.${TOOLSET}', - 'LIB_DIR': '${obj}.${TOOLSET}', - 'INTERMEDIATE_DIR': '${obj}.${TOOLSET}/${TARGET}/geni', - 'SHARED_INTERMEDIATE_DIR': '${obj}/gen', - 'PRODUCT_DIR': '${builddir}', - 'RULE_INPUT_PATH': '${RULE_INPUT_PATH}', - 'RULE_INPUT_DIRNAME': '${RULE_INPUT_DIRNAME}', - 'RULE_INPUT_NAME': '${RULE_INPUT_NAME}', - 'RULE_INPUT_ROOT': '${RULE_INPUT_ROOT}', - 'RULE_INPUT_EXT': '${RULE_INPUT_EXT}', - 'CONFIGURATION_NAME': '${configuration}', + "EXECUTABLE_PREFIX": "", + "EXECUTABLE_SUFFIX": "", + "STATIC_LIB_PREFIX": "lib", + "STATIC_LIB_SUFFIX": ".a", + "SHARED_LIB_PREFIX": "lib", + "SHARED_LIB_SUFFIX": ".so", + "SHARED_LIB_DIR": "${builddir}/lib.${TOOLSET}", + "LIB_DIR": "${obj}.${TOOLSET}", + "INTERMEDIATE_DIR": "${obj}.${TOOLSET}/${TARGET}/geni", + "SHARED_INTERMEDIATE_DIR": "${obj}/gen", + "PRODUCT_DIR": "${builddir}", + "RULE_INPUT_PATH": "${RULE_INPUT_PATH}", + "RULE_INPUT_DIRNAME": "${RULE_INPUT_DIRNAME}", + "RULE_INPUT_NAME": "${RULE_INPUT_NAME}", + "RULE_INPUT_ROOT": "${RULE_INPUT_ROOT}", + "RULE_INPUT_EXT": "${RULE_INPUT_EXT}", + "CONFIGURATION_NAME": "${configuration}", } -FULL_PATH_VARS = ('${CMAKE_CURRENT_LIST_DIR}', '${builddir}', '${obj}') +FULL_PATH_VARS = ("${CMAKE_CURRENT_LIST_DIR}", "${builddir}", "${obj}") generator_supports_multiple_toolsets = True generator_wants_static_library_dependencies_adjusted = True COMPILABLE_EXTENSIONS = { - '.c': 'cc', - '.cc': 'cxx', - '.cpp': 'cxx', - '.cxx': 'cxx', - '.s': 's', # cc - '.S': 's', # cc + ".c": "cc", + ".cc": "cxx", + ".cpp": "cxx", + ".cxx": "cxx", + ".s": "s", # cc + ".S": "s", # cc } def RemovePrefix(a, prefix): - """Returns 'a' without 'prefix' if it starts with 'prefix'.""" - return a[len(prefix):] if a.startswith(prefix) else a + """Returns 'a' without 'prefix' if it starts with 'prefix'.""" + return a[len(prefix) :] if a.startswith(prefix) else a def CalculateVariables(default_variables, params): - """Calculate additional variables for use in the build (called by gyp).""" - default_variables.setdefault('OS', gyp.common.GetFlavor(params)) + """Calculate additional variables for use in the build (called by gyp).""" + default_variables.setdefault("OS", gyp.common.GetFlavor(params)) def Compilable(filename): - """Return true if the file is compilable (should be in OBJS).""" - return any(filename.endswith(e) for e in COMPILABLE_EXTENSIONS) + """Return true if the file is compilable (should be in OBJS).""" + return any(filename.endswith(e) for e in COMPILABLE_EXTENSIONS) def Linkable(filename): - """Return true if the file is linkable (should be on the link line).""" - return filename.endswith('.o') + """Return true if the file is linkable (should be on the link line).""" + return filename.endswith(".o") def NormjoinPathForceCMakeSource(base_path, rel_path): - """Resolves rel_path against base_path and returns the result. + """Resolves rel_path against base_path and returns the result. If rel_path is an absolute path it is returned unchanged. Otherwise it is resolved against base_path and normalized. If the result is a relative path, it is forced to be relative to the CMakeLists.txt. """ - if os.path.isabs(rel_path): - return rel_path - if any([rel_path.startswith(var) for var in FULL_PATH_VARS]): - return rel_path - # TODO: do we need to check base_path for absolute variables as well? - return os.path.join('${CMAKE_CURRENT_LIST_DIR}', - os.path.normpath(os.path.join(base_path, rel_path))) + if os.path.isabs(rel_path): + return rel_path + if any([rel_path.startswith(var) for var in FULL_PATH_VARS]): + return rel_path + # TODO: do we need to check base_path for absolute variables as well? + return os.path.join( + "${CMAKE_CURRENT_LIST_DIR}", os.path.normpath(os.path.join(base_path, rel_path)) + ) def NormjoinPath(base_path, rel_path): - """Resolves rel_path against base_path and returns the result. + """Resolves rel_path against base_path and returns the result. TODO: what is this really used for? If rel_path begins with '$' it is returned unchanged. Otherwise it is resolved against base_path if relative, then normalized. """ - if rel_path.startswith('$') and not rel_path.startswith('${configuration}'): - return rel_path - return os.path.normpath(os.path.join(base_path, rel_path)) + if rel_path.startswith("$") and not rel_path.startswith("${configuration}"): + return rel_path + return os.path.normpath(os.path.join(base_path, rel_path)) def CMakeStringEscape(a): - """Escapes the string 'a' for use inside a CMake string. + """Escapes the string 'a' for use inside a CMake string. This means escaping '\' otherwise it may be seen as modifying the next character @@ -137,118 +144,114 @@ def CMakeStringEscape(a): but text $ should be escaped what is wanted is to know which $ come from generator variables """ - return a.replace('\\', '\\\\').replace(';', '\\;').replace('"', '\\"') + return a.replace("\\", "\\\\").replace(";", "\\;").replace('"', '\\"') def SetFileProperty(output, source_name, property_name, values, sep): - """Given a set of source file, sets the given property on them.""" - output.write('set_source_files_properties(') - output.write(source_name) - output.write(' PROPERTIES ') - output.write(property_name) - output.write(' "') - for value in values: - output.write(CMakeStringEscape(value)) - output.write(sep) - output.write('")\n') + """Given a set of source file, sets the given property on them.""" + output.write("set_source_files_properties(") + output.write(source_name) + output.write(" PROPERTIES ") + output.write(property_name) + output.write(' "') + for value in values: + output.write(CMakeStringEscape(value)) + output.write(sep) + output.write('")\n') def SetFilesProperty(output, variable, property_name, values, sep): - """Given a set of source files, sets the given property on them.""" - output.write('set_source_files_properties(') - WriteVariable(output, variable) - output.write(' PROPERTIES ') - output.write(property_name) - output.write(' "') - for value in values: - output.write(CMakeStringEscape(value)) - output.write(sep) - output.write('")\n') - - -def SetTargetProperty(output, target_name, property_name, values, sep=''): - """Given a target, sets the given property.""" - output.write('set_target_properties(') - output.write(target_name) - output.write(' PROPERTIES ') - output.write(property_name) - output.write(' "') - for value in values: - output.write(CMakeStringEscape(value)) - output.write(sep) - output.write('")\n') + """Given a set of source files, sets the given property on them.""" + output.write("set_source_files_properties(") + WriteVariable(output, variable) + output.write(" PROPERTIES ") + output.write(property_name) + output.write(' "') + for value in values: + output.write(CMakeStringEscape(value)) + output.write(sep) + output.write('")\n') + + +def SetTargetProperty(output, target_name, property_name, values, sep=""): + """Given a target, sets the given property.""" + output.write("set_target_properties(") + output.write(target_name) + output.write(" PROPERTIES ") + output.write(property_name) + output.write(' "') + for value in values: + output.write(CMakeStringEscape(value)) + output.write(sep) + output.write('")\n') def SetVariable(output, variable_name, value): - """Sets a CMake variable.""" - output.write('set(') - output.write(variable_name) - output.write(' "') - output.write(CMakeStringEscape(value)) - output.write('")\n') + """Sets a CMake variable.""" + output.write("set(") + output.write(variable_name) + output.write(' "') + output.write(CMakeStringEscape(value)) + output.write('")\n') def SetVariableList(output, variable_name, values): - """Sets a CMake variable to a list.""" - if not values: - return SetVariable(output, variable_name, "") - if len(values) == 1: - return SetVariable(output, variable_name, values[0]) - output.write('list(APPEND ') - output.write(variable_name) - output.write('\n "') - output.write('"\n "'.join([CMakeStringEscape(value) for value in values])) - output.write('")\n') + """Sets a CMake variable to a list.""" + if not values: + return SetVariable(output, variable_name, "") + if len(values) == 1: + return SetVariable(output, variable_name, values[0]) + output.write("list(APPEND ") + output.write(variable_name) + output.write('\n "') + output.write('"\n "'.join([CMakeStringEscape(value) for value in values])) + output.write('")\n') def UnsetVariable(output, variable_name): - """Unsets a CMake variable.""" - output.write('unset(') - output.write(variable_name) - output.write(')\n') + """Unsets a CMake variable.""" + output.write("unset(") + output.write(variable_name) + output.write(")\n") def WriteVariable(output, variable_name, prepend=None): - if prepend: - output.write(prepend) - output.write('${') - output.write(variable_name) - output.write('}') + if prepend: + output.write(prepend) + output.write("${") + output.write(variable_name) + output.write("}") class CMakeTargetType(object): - def __init__(self, command, modifier, property_modifier): - self.command = command - self.modifier = modifier - self.property_modifier = property_modifier + def __init__(self, command, modifier, property_modifier): + self.command = command + self.modifier = modifier + self.property_modifier = property_modifier cmake_target_type_from_gyp_target_type = { - 'executable': CMakeTargetType('add_executable', None, 'RUNTIME'), - 'static_library': CMakeTargetType('add_library', 'STATIC', 'ARCHIVE'), - 'shared_library': CMakeTargetType('add_library', 'SHARED', 'LIBRARY'), - 'loadable_module': CMakeTargetType('add_library', 'MODULE', 'LIBRARY'), - 'none': CMakeTargetType('add_custom_target', 'SOURCES', None), + "executable": CMakeTargetType("add_executable", None, "RUNTIME"), + "static_library": CMakeTargetType("add_library", "STATIC", "ARCHIVE"), + "shared_library": CMakeTargetType("add_library", "SHARED", "LIBRARY"), + "loadable_module": CMakeTargetType("add_library", "MODULE", "LIBRARY"), + "none": CMakeTargetType("add_custom_target", "SOURCES", None), } def StringToCMakeTargetName(a): - """Converts the given string 'a' to a valid CMake target name. + """Converts the given string 'a' to a valid CMake target name. All invalid characters are replaced by '_'. Invalid for cmake: ' ', '/', '(', ')', '"' Invalid for make: ':' Invalid for unknown reasons but cause failures: '.' """ - try: - return a.translate(str.maketrans(' /():."', '_______')) - except AttributeError: - return a.translate(string.maketrans(' /():."', '_______')) + return a.translate(_maketrans(' /():."', "_______")) -def WriteActions(target_name, actions, extra_sources, extra_deps, - path_to_gyp, output): - """Write CMake for the 'actions' in the target. +def WriteActions(target_name, actions, extra_sources, extra_deps, path_to_gyp, output): + """Write CMake for the 'actions' in the target. Args: target_name: the name of the CMake target being generated. @@ -258,83 +261,86 @@ def WriteActions(target_name, actions, extra_sources, extra_deps, path_to_gyp: relative path from CMakeLists.txt being generated to the Gyp file in which the target being generated is defined. """ - for action in actions: - action_name = StringToCMakeTargetName(action['action_name']) - action_target_name = '%s__%s' % (target_name, action_name) - - inputs = action['inputs'] - inputs_name = action_target_name + '__input' - SetVariableList(output, inputs_name, - [NormjoinPathForceCMakeSource(path_to_gyp, dep) for dep in inputs]) - - outputs = action['outputs'] - cmake_outputs = [NormjoinPathForceCMakeSource(path_to_gyp, out) - for out in outputs] - outputs_name = action_target_name + '__output' - SetVariableList(output, outputs_name, cmake_outputs) - - # Build up a list of outputs. - # Collect the output dirs we'll need. - dirs = set(dir for dir in (os.path.dirname(o) for o in outputs) if dir) - - if int(action.get('process_outputs_as_sources', False)): - extra_sources.extend(zip(cmake_outputs, outputs)) - - # add_custom_command - output.write('add_custom_command(OUTPUT ') - WriteVariable(output, outputs_name) - output.write('\n') - - if len(dirs) > 0: - for directory in dirs: - output.write(' COMMAND ${CMAKE_COMMAND} -E make_directory ') - output.write(directory) - output.write('\n') - - output.write(' COMMAND ') - output.write(gyp.common.EncodePOSIXShellList(action['action'])) - output.write('\n') - - output.write(' DEPENDS ') - WriteVariable(output, inputs_name) - output.write('\n') - - output.write(' WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}/') - output.write(path_to_gyp) - output.write('\n') - - output.write(' COMMENT ') - if 'message' in action: - output.write(action['message']) - else: - output.write(action_target_name) - output.write('\n') - - output.write(' VERBATIM\n') - output.write(')\n') - - # add_custom_target - output.write('add_custom_target(') - output.write(action_target_name) - output.write('\n DEPENDS ') - WriteVariable(output, outputs_name) - output.write('\n SOURCES ') - WriteVariable(output, inputs_name) - output.write('\n)\n') - - extra_deps.append(action_target_name) + for action in actions: + action_name = StringToCMakeTargetName(action["action_name"]) + action_target_name = "%s__%s" % (target_name, action_name) + + inputs = action["inputs"] + inputs_name = action_target_name + "__input" + SetVariableList( + output, + inputs_name, + [NormjoinPathForceCMakeSource(path_to_gyp, dep) for dep in inputs], + ) + + outputs = action["outputs"] + cmake_outputs = [ + NormjoinPathForceCMakeSource(path_to_gyp, out) for out in outputs + ] + outputs_name = action_target_name + "__output" + SetVariableList(output, outputs_name, cmake_outputs) + + # Build up a list of outputs. + # Collect the output dirs we'll need. + dirs = set(dir for dir in (os.path.dirname(o) for o in outputs) if dir) + + if int(action.get("process_outputs_as_sources", False)): + extra_sources.extend(zip(cmake_outputs, outputs)) + + # add_custom_command + output.write("add_custom_command(OUTPUT ") + WriteVariable(output, outputs_name) + output.write("\n") + + if len(dirs) > 0: + for directory in dirs: + output.write(" COMMAND ${CMAKE_COMMAND} -E make_directory ") + output.write(directory) + output.write("\n") + + output.write(" COMMAND ") + output.write(gyp.common.EncodePOSIXShellList(action["action"])) + output.write("\n") + + output.write(" DEPENDS ") + WriteVariable(output, inputs_name) + output.write("\n") + + output.write(" WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}/") + output.write(path_to_gyp) + output.write("\n") + + output.write(" COMMENT ") + if "message" in action: + output.write(action["message"]) + else: + output.write(action_target_name) + output.write("\n") + + output.write(" VERBATIM\n") + output.write(")\n") + + # add_custom_target + output.write("add_custom_target(") + output.write(action_target_name) + output.write("\n DEPENDS ") + WriteVariable(output, outputs_name) + output.write("\n SOURCES ") + WriteVariable(output, inputs_name) + output.write("\n)\n") + + extra_deps.append(action_target_name) def NormjoinRulePathForceCMakeSource(base_path, rel_path, rule_source): - if rel_path.startswith(("${RULE_INPUT_PATH}","${RULE_INPUT_DIRNAME}")): - if any([rule_source.startswith(var) for var in FULL_PATH_VARS]): - return rel_path - return NormjoinPathForceCMakeSource(base_path, rel_path) + if rel_path.startswith(("${RULE_INPUT_PATH}", "${RULE_INPUT_DIRNAME}")): + if any([rule_source.startswith(var) for var in FULL_PATH_VARS]): + return rel_path + return NormjoinPathForceCMakeSource(base_path, rel_path) -def WriteRules(target_name, rules, extra_sources, extra_deps, - path_to_gyp, output): - """Write CMake for the 'rules' in the target. +def WriteRules(target_name, rules, extra_sources, extra_deps, path_to_gyp, output): + """Write CMake for the 'rules' in the target. Args: target_name: the name of the CMake target being generated. @@ -344,110 +350,115 @@ def WriteRules(target_name, rules, extra_sources, extra_deps, path_to_gyp: relative path from CMakeLists.txt being generated to the Gyp file in which the target being generated is defined. """ - for rule in rules: - rule_name = StringToCMakeTargetName(target_name + '__' + rule['rule_name']) - - inputs = rule.get('inputs', []) - inputs_name = rule_name + '__input' - SetVariableList(output, inputs_name, - [NormjoinPathForceCMakeSource(path_to_gyp, dep) for dep in inputs]) - outputs = rule['outputs'] - var_outputs = [] - - for count, rule_source in enumerate(rule.get('rule_sources', [])): - action_name = rule_name + '_' + str(count) - - rule_source_dirname, rule_source_basename = os.path.split(rule_source) - rule_source_root, rule_source_ext = os.path.splitext(rule_source_basename) - - SetVariable(output, 'RULE_INPUT_PATH', rule_source) - SetVariable(output, 'RULE_INPUT_DIRNAME', rule_source_dirname) - SetVariable(output, 'RULE_INPUT_NAME', rule_source_basename) - SetVariable(output, 'RULE_INPUT_ROOT', rule_source_root) - SetVariable(output, 'RULE_INPUT_EXT', rule_source_ext) - - # Build up a list of outputs. - # Collect the output dirs we'll need. - dirs = set(dir for dir in (os.path.dirname(o) for o in outputs) if dir) - - # Create variables for the output, as 'local' variable will be unset. - these_outputs = [] - for output_index, out in enumerate(outputs): - output_name = action_name + '_' + str(output_index) - SetVariable(output, output_name, - NormjoinRulePathForceCMakeSource(path_to_gyp, out, - rule_source)) - if int(rule.get('process_outputs_as_sources', False)): - extra_sources.append(('${' + output_name + '}', out)) - these_outputs.append('${' + output_name + '}') - var_outputs.append('${' + output_name + '}') - - # add_custom_command - output.write('add_custom_command(OUTPUT\n') - for out in these_outputs: - output.write(' ') - output.write(out) - output.write('\n') - - for directory in dirs: - output.write(' COMMAND ${CMAKE_COMMAND} -E make_directory ') - output.write(directory) - output.write('\n') - - output.write(' COMMAND ') - output.write(gyp.common.EncodePOSIXShellList(rule['action'])) - output.write('\n') - - output.write(' DEPENDS ') - WriteVariable(output, inputs_name) - output.write(' ') - output.write(NormjoinPath(path_to_gyp, rule_source)) - output.write('\n') - - # CMAKE_CURRENT_LIST_DIR is where the CMakeLists.txt lives. - # The cwd is the current build directory. - output.write(' WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}/') - output.write(path_to_gyp) - output.write('\n') - - output.write(' COMMENT ') - if 'message' in rule: - output.write(rule['message']) - else: - output.write(action_name) - output.write('\n') - - output.write(' VERBATIM\n') - output.write(')\n') - - UnsetVariable(output, 'RULE_INPUT_PATH') - UnsetVariable(output, 'RULE_INPUT_DIRNAME') - UnsetVariable(output, 'RULE_INPUT_NAME') - UnsetVariable(output, 'RULE_INPUT_ROOT') - UnsetVariable(output, 'RULE_INPUT_EXT') - - # add_custom_target - output.write('add_custom_target(') - output.write(rule_name) - output.write(' DEPENDS\n') - for out in var_outputs: - output.write(' ') - output.write(out) - output.write('\n') - output.write('SOURCES ') - WriteVariable(output, inputs_name) - output.write('\n') - for rule_source in rule.get('rule_sources', []): - output.write(' ') - output.write(NormjoinPath(path_to_gyp, rule_source)) - output.write('\n') - output.write(')\n') - - extra_deps.append(rule_name) + for rule in rules: + rule_name = StringToCMakeTargetName(target_name + "__" + rule["rule_name"]) + + inputs = rule.get("inputs", []) + inputs_name = rule_name + "__input" + SetVariableList( + output, + inputs_name, + [NormjoinPathForceCMakeSource(path_to_gyp, dep) for dep in inputs], + ) + outputs = rule["outputs"] + var_outputs = [] + + for count, rule_source in enumerate(rule.get("rule_sources", [])): + action_name = rule_name + "_" + str(count) + + rule_source_dirname, rule_source_basename = os.path.split(rule_source) + rule_source_root, rule_source_ext = os.path.splitext(rule_source_basename) + + SetVariable(output, "RULE_INPUT_PATH", rule_source) + SetVariable(output, "RULE_INPUT_DIRNAME", rule_source_dirname) + SetVariable(output, "RULE_INPUT_NAME", rule_source_basename) + SetVariable(output, "RULE_INPUT_ROOT", rule_source_root) + SetVariable(output, "RULE_INPUT_EXT", rule_source_ext) + + # Build up a list of outputs. + # Collect the output dirs we'll need. + dirs = set(dir for dir in (os.path.dirname(o) for o in outputs) if dir) + + # Create variables for the output, as 'local' variable will be unset. + these_outputs = [] + for output_index, out in enumerate(outputs): + output_name = action_name + "_" + str(output_index) + SetVariable( + output, + output_name, + NormjoinRulePathForceCMakeSource(path_to_gyp, out, rule_source), + ) + if int(rule.get("process_outputs_as_sources", False)): + extra_sources.append(("${" + output_name + "}", out)) + these_outputs.append("${" + output_name + "}") + var_outputs.append("${" + output_name + "}") + + # add_custom_command + output.write("add_custom_command(OUTPUT\n") + for out in these_outputs: + output.write(" ") + output.write(out) + output.write("\n") + + for directory in dirs: + output.write(" COMMAND ${CMAKE_COMMAND} -E make_directory ") + output.write(directory) + output.write("\n") + + output.write(" COMMAND ") + output.write(gyp.common.EncodePOSIXShellList(rule["action"])) + output.write("\n") + + output.write(" DEPENDS ") + WriteVariable(output, inputs_name) + output.write(" ") + output.write(NormjoinPath(path_to_gyp, rule_source)) + output.write("\n") + + # CMAKE_CURRENT_LIST_DIR is where the CMakeLists.txt lives. + # The cwd is the current build directory. + output.write(" WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}/") + output.write(path_to_gyp) + output.write("\n") + + output.write(" COMMENT ") + if "message" in rule: + output.write(rule["message"]) + else: + output.write(action_name) + output.write("\n") + + output.write(" VERBATIM\n") + output.write(")\n") + + UnsetVariable(output, "RULE_INPUT_PATH") + UnsetVariable(output, "RULE_INPUT_DIRNAME") + UnsetVariable(output, "RULE_INPUT_NAME") + UnsetVariable(output, "RULE_INPUT_ROOT") + UnsetVariable(output, "RULE_INPUT_EXT") + + # add_custom_target + output.write("add_custom_target(") + output.write(rule_name) + output.write(" DEPENDS\n") + for out in var_outputs: + output.write(" ") + output.write(out) + output.write("\n") + output.write("SOURCES ") + WriteVariable(output, inputs_name) + output.write("\n") + for rule_source in rule.get("rule_sources", []): + output.write(" ") + output.write(NormjoinPath(path_to_gyp, rule_source)) + output.write("\n") + output.write(")\n") + + extra_deps.append(rule_name) def WriteCopies(target_name, copies, extra_deps, path_to_gyp, output): - """Write CMake for the 'copies' in the target. + """Write CMake for the 'copies' in the target. Args: target_name: the name of the CMake target being generated. @@ -456,126 +467,128 @@ def WriteCopies(target_name, copies, extra_deps, path_to_gyp, output): path_to_gyp: relative path from CMakeLists.txt being generated to the Gyp file in which the target being generated is defined. """ - copy_name = target_name + '__copies' + copy_name = target_name + "__copies" + + # CMake gets upset with custom targets with OUTPUT which specify no output. + have_copies = any(copy["files"] for copy in copies) + if not have_copies: + output.write("add_custom_target(") + output.write(copy_name) + output.write(")\n") + extra_deps.append(copy_name) + return + + class Copy(object): + def __init__(self, ext, command): + self.cmake_inputs = [] + self.cmake_outputs = [] + self.gyp_inputs = [] + self.gyp_outputs = [] + self.ext = ext + self.inputs_name = None + self.outputs_name = None + self.command = command + + file_copy = Copy("", "copy") + dir_copy = Copy("_dirs", "copy_directory") + + for copy in copies: + files = copy["files"] + destination = copy["destination"] + for src in files: + path = os.path.normpath(src) + basename = os.path.split(path)[1] + dst = os.path.join(destination, basename) + + copy = file_copy if os.path.basename(src) else dir_copy + + copy.cmake_inputs.append(NormjoinPathForceCMakeSource(path_to_gyp, src)) + copy.cmake_outputs.append(NormjoinPathForceCMakeSource(path_to_gyp, dst)) + copy.gyp_inputs.append(src) + copy.gyp_outputs.append(dst) + + for copy in (file_copy, dir_copy): + if copy.cmake_inputs: + copy.inputs_name = copy_name + "__input" + copy.ext + SetVariableList(output, copy.inputs_name, copy.cmake_inputs) + + copy.outputs_name = copy_name + "__output" + copy.ext + SetVariableList(output, copy.outputs_name, copy.cmake_outputs) + + # add_custom_command + output.write("add_custom_command(\n") + + output.write("OUTPUT") + for copy in (file_copy, dir_copy): + if copy.outputs_name: + WriteVariable(output, copy.outputs_name, " ") + output.write("\n") + + for copy in (file_copy, dir_copy): + for src, dst in zip(copy.gyp_inputs, copy.gyp_outputs): + # 'cmake -E copy src dst' will create the 'dst' directory if needed. + output.write("COMMAND ${CMAKE_COMMAND} -E %s " % copy.command) + output.write(src) + output.write(" ") + output.write(dst) + output.write("\n") + + output.write("DEPENDS") + for copy in (file_copy, dir_copy): + if copy.inputs_name: + WriteVariable(output, copy.inputs_name, " ") + output.write("\n") + + output.write("WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}/") + output.write(path_to_gyp) + output.write("\n") + + output.write("COMMENT Copying for ") + output.write(target_name) + output.write("\n") - # CMake gets upset with custom targets with OUTPUT which specify no output. - have_copies = any(copy['files'] for copy in copies) - if not have_copies: - output.write('add_custom_target(') + output.write("VERBATIM\n") + output.write(")\n") + + # add_custom_target + output.write("add_custom_target(") output.write(copy_name) - output.write(')\n') + output.write("\n DEPENDS") + for copy in (file_copy, dir_copy): + if copy.outputs_name: + WriteVariable(output, copy.outputs_name, " ") + output.write("\n SOURCES") + if file_copy.inputs_name: + WriteVariable(output, file_copy.inputs_name, " ") + output.write("\n)\n") + extra_deps.append(copy_name) - return - - class Copy(object): - def __init__(self, ext, command): - self.cmake_inputs = [] - self.cmake_outputs = [] - self.gyp_inputs = [] - self.gyp_outputs = [] - self.ext = ext - self.inputs_name = None - self.outputs_name = None - self.command = command - - file_copy = Copy('', 'copy') - dir_copy = Copy('_dirs', 'copy_directory') - - for copy in copies: - files = copy['files'] - destination = copy['destination'] - for src in files: - path = os.path.normpath(src) - basename = os.path.split(path)[1] - dst = os.path.join(destination, basename) - - copy = file_copy if os.path.basename(src) else dir_copy - - copy.cmake_inputs.append(NormjoinPathForceCMakeSource(path_to_gyp, src)) - copy.cmake_outputs.append(NormjoinPathForceCMakeSource(path_to_gyp, dst)) - copy.gyp_inputs.append(src) - copy.gyp_outputs.append(dst) - - for copy in (file_copy, dir_copy): - if copy.cmake_inputs: - copy.inputs_name = copy_name + '__input' + copy.ext - SetVariableList(output, copy.inputs_name, copy.cmake_inputs) - - copy.outputs_name = copy_name + '__output' + copy.ext - SetVariableList(output, copy.outputs_name, copy.cmake_outputs) - - # add_custom_command - output.write('add_custom_command(\n') - - output.write('OUTPUT') - for copy in (file_copy, dir_copy): - if copy.outputs_name: - WriteVariable(output, copy.outputs_name, ' ') - output.write('\n') - - for copy in (file_copy, dir_copy): - for src, dst in zip(copy.gyp_inputs, copy.gyp_outputs): - # 'cmake -E copy src dst' will create the 'dst' directory if needed. - output.write('COMMAND ${CMAKE_COMMAND} -E %s ' % copy.command) - output.write(src) - output.write(' ') - output.write(dst) - output.write("\n") - - output.write('DEPENDS') - for copy in (file_copy, dir_copy): - if copy.inputs_name: - WriteVariable(output, copy.inputs_name, ' ') - output.write('\n') - - output.write('WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}/') - output.write(path_to_gyp) - output.write('\n') - - output.write('COMMENT Copying for ') - output.write(target_name) - output.write('\n') - - output.write('VERBATIM\n') - output.write(')\n') - - # add_custom_target - output.write('add_custom_target(') - output.write(copy_name) - output.write('\n DEPENDS') - for copy in (file_copy, dir_copy): - if copy.outputs_name: - WriteVariable(output, copy.outputs_name, ' ') - output.write('\n SOURCES') - if file_copy.inputs_name: - WriteVariable(output, file_copy.inputs_name, ' ') - output.write('\n)\n') - - extra_deps.append(copy_name) def CreateCMakeTargetBaseName(qualified_target): - """This is the name we would like the target to have.""" - _, gyp_target_name, gyp_target_toolset = ( - gyp.common.ParseQualifiedTarget(qualified_target)) - cmake_target_base_name = gyp_target_name - if gyp_target_toolset and gyp_target_toolset != 'target': - cmake_target_base_name += '_' + gyp_target_toolset - return StringToCMakeTargetName(cmake_target_base_name) + """This is the name we would like the target to have.""" + _, gyp_target_name, gyp_target_toolset = gyp.common.ParseQualifiedTarget( + qualified_target + ) + cmake_target_base_name = gyp_target_name + if gyp_target_toolset and gyp_target_toolset != "target": + cmake_target_base_name += "_" + gyp_target_toolset + return StringToCMakeTargetName(cmake_target_base_name) def CreateCMakeTargetFullName(qualified_target): - """An unambiguous name for the target.""" - gyp_file, gyp_target_name, gyp_target_toolset = ( - gyp.common.ParseQualifiedTarget(qualified_target)) - cmake_target_full_name = gyp_file + ':' + gyp_target_name - if gyp_target_toolset and gyp_target_toolset != 'target': - cmake_target_full_name += '_' + gyp_target_toolset - return StringToCMakeTargetName(cmake_target_full_name) + """An unambiguous name for the target.""" + gyp_file, gyp_target_name, gyp_target_toolset = gyp.common.ParseQualifiedTarget( + qualified_target + ) + cmake_target_full_name = gyp_file + ":" + gyp_target_name + if gyp_target_toolset and gyp_target_toolset != "target": + cmake_target_full_name += "_" + gyp_target_toolset + return StringToCMakeTargetName(cmake_target_full_name) class CMakeNamer(object): - """Converts Gyp target names into CMake target names. + """Converts Gyp target names into CMake target names. CMake requires that target names be globally unique. One way to ensure this is to fully qualify the names of the targets. Unfortunately, this @@ -594,660 +607,721 @@ class CMakeNamer(object): building. However, it also makes sense for an IDE, as it is possible for defines to be different. """ - def __init__(self, target_list): - self.cmake_target_base_names_conficting = set() - cmake_target_base_names_seen = set() - for qualified_target in target_list: - cmake_target_base_name = CreateCMakeTargetBaseName(qualified_target) - - if cmake_target_base_name not in cmake_target_base_names_seen: - cmake_target_base_names_seen.add(cmake_target_base_name) - else: - self.cmake_target_base_names_conficting.add(cmake_target_base_name) - - def CreateCMakeTargetName(self, qualified_target): - base_name = CreateCMakeTargetBaseName(qualified_target) - if base_name in self.cmake_target_base_names_conficting: - return CreateCMakeTargetFullName(qualified_target) - return base_name - - -def WriteTarget(namer, qualified_target, target_dicts, build_dir, config_to_use, - options, generator_flags, all_qualified_targets, flavor, - output): - # The make generator does this always. - # TODO: It would be nice to be able to tell CMake all dependencies. - circular_libs = generator_flags.get('circular', True) - - if not generator_flags.get('standalone', False): - output.write('\n#') - output.write(qualified_target) - output.write('\n') - - gyp_file, _, _ = gyp.common.ParseQualifiedTarget(qualified_target) - rel_gyp_file = gyp.common.RelativePath(gyp_file, options.toplevel_dir) - rel_gyp_dir = os.path.dirname(rel_gyp_file) - - # Relative path from build dir to top dir. - build_to_top = gyp.common.InvertRelativePath(build_dir, options.toplevel_dir) - # Relative path from build dir to gyp dir. - build_to_gyp = os.path.join(build_to_top, rel_gyp_dir) - - path_from_cmakelists_to_gyp = build_to_gyp - - spec = target_dicts.get(qualified_target, {}) - config = spec.get('configurations', {}).get(config_to_use, {}) - - xcode_settings = None - if flavor == 'mac': - xcode_settings = gyp.xcode_emulation.XcodeSettings(spec) - - target_name = spec.get('target_name', '') - target_type = spec.get('type', '') - target_toolset = spec.get('toolset') - - cmake_target_type = cmake_target_type_from_gyp_target_type.get(target_type) - if cmake_target_type is None: - print('Target %s has unknown target type %s, skipping.' % - ( target_name, target_type )) - return - - SetVariable(output, 'TARGET', target_name) - SetVariable(output, 'TOOLSET', target_toolset) - - cmake_target_name = namer.CreateCMakeTargetName(qualified_target) - - extra_sources = [] - extra_deps = [] - - # Actions must come first, since they can generate more OBJs for use below. - if 'actions' in spec: - WriteActions(cmake_target_name, spec['actions'], extra_sources, extra_deps, - path_from_cmakelists_to_gyp, output) - - # Rules must be early like actions. - if 'rules' in spec: - WriteRules(cmake_target_name, spec['rules'], extra_sources, extra_deps, - path_from_cmakelists_to_gyp, output) - - # Copies - if 'copies' in spec: - WriteCopies(cmake_target_name, spec['copies'], extra_deps, - path_from_cmakelists_to_gyp, output) - - # Target and sources - srcs = spec.get('sources', []) - - # Gyp separates the sheep from the goats based on file extensions. - # A full separation is done here because of flag handing (see below). - s_sources = [] - c_sources = [] - cxx_sources = [] - linkable_sources = [] - other_sources = [] - for src in srcs: - _, ext = os.path.splitext(src) - src_type = COMPILABLE_EXTENSIONS.get(ext, None) - src_norm_path = NormjoinPath(path_from_cmakelists_to_gyp, src) - - if src_type == 's': - s_sources.append(src_norm_path) - elif src_type == 'cc': - c_sources.append(src_norm_path) - elif src_type == 'cxx': - cxx_sources.append(src_norm_path) - elif Linkable(ext): - linkable_sources.append(src_norm_path) - else: - other_sources.append(src_norm_path) - - for extra_source in extra_sources: - src, real_source = extra_source - _, ext = os.path.splitext(real_source) - src_type = COMPILABLE_EXTENSIONS.get(ext, None) - - if src_type == 's': - s_sources.append(src) - elif src_type == 'cc': - c_sources.append(src) - elif src_type == 'cxx': - cxx_sources.append(src) - elif Linkable(ext): - linkable_sources.append(src) - else: - other_sources.append(src) - - s_sources_name = None - if s_sources: - s_sources_name = cmake_target_name + '__asm_srcs' - SetVariableList(output, s_sources_name, s_sources) - - c_sources_name = None - if c_sources: - c_sources_name = cmake_target_name + '__c_srcs' - SetVariableList(output, c_sources_name, c_sources) - - cxx_sources_name = None - if cxx_sources: - cxx_sources_name = cmake_target_name + '__cxx_srcs' - SetVariableList(output, cxx_sources_name, cxx_sources) - - linkable_sources_name = None - if linkable_sources: - linkable_sources_name = cmake_target_name + '__linkable_srcs' - SetVariableList(output, linkable_sources_name, linkable_sources) - - other_sources_name = None - if other_sources: - other_sources_name = cmake_target_name + '__other_srcs' - SetVariableList(output, other_sources_name, other_sources) - - # CMake gets upset when executable targets provide no sources. - # http://www.cmake.org/pipermail/cmake/2010-July/038461.html - dummy_sources_name = None - has_sources = (s_sources_name or - c_sources_name or - cxx_sources_name or - linkable_sources_name or - other_sources_name) - if target_type == 'executable' and not has_sources: - dummy_sources_name = cmake_target_name + '__dummy_srcs' - SetVariable(output, dummy_sources_name, - "${obj}.${TOOLSET}/${TARGET}/genc/dummy.c") - output.write('if(NOT EXISTS "') - WriteVariable(output, dummy_sources_name) - output.write('")\n') - output.write(' file(WRITE "') - WriteVariable(output, dummy_sources_name) - output.write('" "")\n') - output.write("endif()\n") - - - # CMake is opposed to setting linker directories and considers the practice - # of setting linker directories dangerous. Instead, it favors the use of - # find_library and passing absolute paths to target_link_libraries. - # However, CMake does provide the command link_directories, which adds - # link directories to targets defined after it is called. - # As a result, link_directories must come before the target definition. - # CMake unfortunately has no means of removing entries from LINK_DIRECTORIES. - library_dirs = config.get('library_dirs') - if library_dirs is not None: - output.write('link_directories(') - for library_dir in library_dirs: - output.write(' ') - output.write(NormjoinPath(path_from_cmakelists_to_gyp, library_dir)) - output.write('\n') - output.write(')\n') - - output.write(cmake_target_type.command) - output.write('(') - output.write(cmake_target_name) - - if cmake_target_type.modifier is not None: - output.write(' ') - output.write(cmake_target_type.modifier) - - if s_sources_name: - WriteVariable(output, s_sources_name, ' ') - if c_sources_name: - WriteVariable(output, c_sources_name, ' ') - if cxx_sources_name: - WriteVariable(output, cxx_sources_name, ' ') - if linkable_sources_name: - WriteVariable(output, linkable_sources_name, ' ') - if other_sources_name: - WriteVariable(output, other_sources_name, ' ') - if dummy_sources_name: - WriteVariable(output, dummy_sources_name, ' ') - - output.write(')\n') - - # Let CMake know if the 'all' target should depend on this target. - exclude_from_all = ('TRUE' if qualified_target not in all_qualified_targets - else 'FALSE') - SetTargetProperty(output, cmake_target_name, - 'EXCLUDE_FROM_ALL', exclude_from_all) - for extra_target_name in extra_deps: - SetTargetProperty(output, extra_target_name, - 'EXCLUDE_FROM_ALL', exclude_from_all) - - # Output name and location. - if target_type != 'none': - # Link as 'C' if there are no other files - if not c_sources and not cxx_sources: - SetTargetProperty(output, cmake_target_name, 'LINKER_LANGUAGE', ['C']) - - # Mark uncompiled sources as uncompiled. - if other_sources_name: - output.write('set_source_files_properties(') - WriteVariable(output, other_sources_name, '') - output.write(' PROPERTIES HEADER_FILE_ONLY "TRUE")\n') + def __init__(self, target_list): + self.cmake_target_base_names_conficting = set() + + cmake_target_base_names_seen = set() + for qualified_target in target_list: + cmake_target_base_name = CreateCMakeTargetBaseName(qualified_target) + + if cmake_target_base_name not in cmake_target_base_names_seen: + cmake_target_base_names_seen.add(cmake_target_base_name) + else: + self.cmake_target_base_names_conficting.add(cmake_target_base_name) + + def CreateCMakeTargetName(self, qualified_target): + base_name = CreateCMakeTargetBaseName(qualified_target) + if base_name in self.cmake_target_base_names_conficting: + return CreateCMakeTargetFullName(qualified_target) + return base_name + + +def WriteTarget( + namer, + qualified_target, + target_dicts, + build_dir, + config_to_use, + options, + generator_flags, + all_qualified_targets, + flavor, + output, +): + # The make generator does this always. + # TODO: It would be nice to be able to tell CMake all dependencies. + circular_libs = generator_flags.get("circular", True) + + if not generator_flags.get("standalone", False): + output.write("\n#") + output.write(qualified_target) + output.write("\n") + + gyp_file, _, _ = gyp.common.ParseQualifiedTarget(qualified_target) + rel_gyp_file = gyp.common.RelativePath(gyp_file, options.toplevel_dir) + rel_gyp_dir = os.path.dirname(rel_gyp_file) + + # Relative path from build dir to top dir. + build_to_top = gyp.common.InvertRelativePath(build_dir, options.toplevel_dir) + # Relative path from build dir to gyp dir. + build_to_gyp = os.path.join(build_to_top, rel_gyp_dir) + + path_from_cmakelists_to_gyp = build_to_gyp + + spec = target_dicts.get(qualified_target, {}) + config = spec.get("configurations", {}).get(config_to_use, {}) + + xcode_settings = None + if flavor == "mac": + xcode_settings = gyp.xcode_emulation.XcodeSettings(spec) + + target_name = spec.get("target_name", "") + target_type = spec.get("type", "") + target_toolset = spec.get("toolset") + + cmake_target_type = cmake_target_type_from_gyp_target_type.get(target_type) + if cmake_target_type is None: + print( + "Target %s has unknown target type %s, skipping." + % (target_name, target_type) + ) + return + + SetVariable(output, "TARGET", target_name) + SetVariable(output, "TOOLSET", target_toolset) + + cmake_target_name = namer.CreateCMakeTargetName(qualified_target) + + extra_sources = [] + extra_deps = [] + + # Actions must come first, since they can generate more OBJs for use below. + if "actions" in spec: + WriteActions( + cmake_target_name, + spec["actions"], + extra_sources, + extra_deps, + path_from_cmakelists_to_gyp, + output, + ) + + # Rules must be early like actions. + if "rules" in spec: + WriteRules( + cmake_target_name, + spec["rules"], + extra_sources, + extra_deps, + path_from_cmakelists_to_gyp, + output, + ) + + # Copies + if "copies" in spec: + WriteCopies( + cmake_target_name, + spec["copies"], + extra_deps, + path_from_cmakelists_to_gyp, + output, + ) + + # Target and sources + srcs = spec.get("sources", []) + + # Gyp separates the sheep from the goats based on file extensions. + # A full separation is done here because of flag handing (see below). + s_sources = [] + c_sources = [] + cxx_sources = [] + linkable_sources = [] + other_sources = [] + for src in srcs: + _, ext = os.path.splitext(src) + src_type = COMPILABLE_EXTENSIONS.get(ext, None) + src_norm_path = NormjoinPath(path_from_cmakelists_to_gyp, src) + + if src_type == "s": + s_sources.append(src_norm_path) + elif src_type == "cc": + c_sources.append(src_norm_path) + elif src_type == "cxx": + cxx_sources.append(src_norm_path) + elif Linkable(ext): + linkable_sources.append(src_norm_path) + else: + other_sources.append(src_norm_path) + + for extra_source in extra_sources: + src, real_source = extra_source + _, ext = os.path.splitext(real_source) + src_type = COMPILABLE_EXTENSIONS.get(ext, None) + + if src_type == "s": + s_sources.append(src) + elif src_type == "cc": + c_sources.append(src) + elif src_type == "cxx": + cxx_sources.append(src) + elif Linkable(ext): + linkable_sources.append(src) + else: + other_sources.append(src) + + s_sources_name = None + if s_sources: + s_sources_name = cmake_target_name + "__asm_srcs" + SetVariableList(output, s_sources_name, s_sources) + + c_sources_name = None + if c_sources: + c_sources_name = cmake_target_name + "__c_srcs" + SetVariableList(output, c_sources_name, c_sources) + + cxx_sources_name = None + if cxx_sources: + cxx_sources_name = cmake_target_name + "__cxx_srcs" + SetVariableList(output, cxx_sources_name, cxx_sources) + + linkable_sources_name = None + if linkable_sources: + linkable_sources_name = cmake_target_name + "__linkable_srcs" + SetVariableList(output, linkable_sources_name, linkable_sources) + + other_sources_name = None + if other_sources: + other_sources_name = cmake_target_name + "__other_srcs" + SetVariableList(output, other_sources_name, other_sources) + + # CMake gets upset when executable targets provide no sources. + # http://www.cmake.org/pipermail/cmake/2010-July/038461.html + dummy_sources_name = None + has_sources = ( + s_sources_name + or c_sources_name + or cxx_sources_name + or linkable_sources_name + or other_sources_name + ) + if target_type == "executable" and not has_sources: + dummy_sources_name = cmake_target_name + "__dummy_srcs" + SetVariable( + output, dummy_sources_name, "${obj}.${TOOLSET}/${TARGET}/genc/dummy.c" + ) + output.write('if(NOT EXISTS "') + WriteVariable(output, dummy_sources_name) + output.write('")\n') + output.write(' file(WRITE "') + WriteVariable(output, dummy_sources_name) + output.write('" "")\n') + output.write("endif()\n") + + # CMake is opposed to setting linker directories and considers the practice + # of setting linker directories dangerous. Instead, it favors the use of + # find_library and passing absolute paths to target_link_libraries. + # However, CMake does provide the command link_directories, which adds + # link directories to targets defined after it is called. + # As a result, link_directories must come before the target definition. + # CMake unfortunately has no means of removing entries from LINK_DIRECTORIES. + library_dirs = config.get("library_dirs") + if library_dirs is not None: + output.write("link_directories(") + for library_dir in library_dirs: + output.write(" ") + output.write(NormjoinPath(path_from_cmakelists_to_gyp, library_dir)) + output.write("\n") + output.write(")\n") + + output.write(cmake_target_type.command) + output.write("(") + output.write(cmake_target_name) - # Mark object sources as linkable. + if cmake_target_type.modifier is not None: + output.write(" ") + output.write(cmake_target_type.modifier) + + if s_sources_name: + WriteVariable(output, s_sources_name, " ") + if c_sources_name: + WriteVariable(output, c_sources_name, " ") + if cxx_sources_name: + WriteVariable(output, cxx_sources_name, " ") if linkable_sources_name: - output.write('set_source_files_properties(') - WriteVariable(output, other_sources_name, '') - output.write(' PROPERTIES EXTERNAL_OBJECT "TRUE")\n') - - # Output directory - target_output_directory = spec.get('product_dir') - if target_output_directory is None: - if target_type in ('executable', 'loadable_module'): - target_output_directory = generator_default_variables['PRODUCT_DIR'] - elif target_type == 'shared_library': - target_output_directory = '${builddir}/lib.${TOOLSET}' - elif spec.get('standalone_static_library', False): - target_output_directory = generator_default_variables['PRODUCT_DIR'] - else: - base_path = gyp.common.RelativePath(os.path.dirname(gyp_file), - options.toplevel_dir) - target_output_directory = '${obj}.${TOOLSET}' - target_output_directory = ( - os.path.join(target_output_directory, base_path)) - - cmake_target_output_directory = NormjoinPathForceCMakeSource( - path_from_cmakelists_to_gyp, - target_output_directory) - SetTargetProperty(output, - cmake_target_name, - cmake_target_type.property_modifier + '_OUTPUT_DIRECTORY', - cmake_target_output_directory) - - # Output name - default_product_prefix = '' - default_product_name = target_name - default_product_ext = '' - if target_type == 'static_library': - static_library_prefix = generator_default_variables['STATIC_LIB_PREFIX'] - default_product_name = RemovePrefix(default_product_name, - static_library_prefix) - default_product_prefix = static_library_prefix - default_product_ext = generator_default_variables['STATIC_LIB_SUFFIX'] - - elif target_type in ('loadable_module', 'shared_library'): - shared_library_prefix = generator_default_variables['SHARED_LIB_PREFIX'] - default_product_name = RemovePrefix(default_product_name, - shared_library_prefix) - default_product_prefix = shared_library_prefix - default_product_ext = generator_default_variables['SHARED_LIB_SUFFIX'] - - elif target_type != 'executable': - print('ERROR: What output file should be generated?', - 'type', target_type, 'target', target_name) - - product_prefix = spec.get('product_prefix', default_product_prefix) - product_name = spec.get('product_name', default_product_name) - product_ext = spec.get('product_extension') - if product_ext: - product_ext = '.' + product_ext - else: - product_ext = default_product_ext - - SetTargetProperty(output, cmake_target_name, 'PREFIX', product_prefix) - SetTargetProperty(output, cmake_target_name, - cmake_target_type.property_modifier + '_OUTPUT_NAME', - product_name) - SetTargetProperty(output, cmake_target_name, 'SUFFIX', product_ext) - - # Make the output of this target referenceable as a source. - cmake_target_output_basename = product_prefix + product_name + product_ext - cmake_target_output = os.path.join(cmake_target_output_directory, - cmake_target_output_basename) - SetFileProperty(output, cmake_target_output, 'GENERATED', ['TRUE'], '') - - # Includes - includes = config.get('include_dirs') - if includes: - # This (target include directories) is what requires CMake 2.8.8 - includes_name = cmake_target_name + '__include_dirs' - SetVariableList(output, includes_name, - [NormjoinPathForceCMakeSource(path_from_cmakelists_to_gyp, include) - for include in includes]) - output.write('set_property(TARGET ') - output.write(cmake_target_name) - output.write(' APPEND PROPERTY INCLUDE_DIRECTORIES ') - WriteVariable(output, includes_name, '') - output.write(')\n') - - # Defines - defines = config.get('defines') - if defines is not None: - SetTargetProperty(output, - cmake_target_name, - 'COMPILE_DEFINITIONS', - defines, - ';') - - # Compile Flags - http://www.cmake.org/Bug/view.php?id=6493 - # CMake currently does not have target C and CXX flags. - # So, instead of doing... - - # cflags_c = config.get('cflags_c') - # if cflags_c is not None: - # SetTargetProperty(output, cmake_target_name, - # 'C_COMPILE_FLAGS', cflags_c, ' ') - - # cflags_cc = config.get('cflags_cc') - # if cflags_cc is not None: - # SetTargetProperty(output, cmake_target_name, - # 'CXX_COMPILE_FLAGS', cflags_cc, ' ') - - # Instead we must... - cflags = config.get('cflags', []) - cflags_c = config.get('cflags_c', []) - cflags_cxx = config.get('cflags_cc', []) - if xcode_settings: - cflags = xcode_settings.GetCflags(config_to_use) - cflags_c = xcode_settings.GetCflagsC(config_to_use) - cflags_cxx = xcode_settings.GetCflagsCC(config_to_use) - #cflags_objc = xcode_settings.GetCflagsObjC(config_to_use) - #cflags_objcc = xcode_settings.GetCflagsObjCC(config_to_use) - - if (not cflags_c or not c_sources) and (not cflags_cxx or not cxx_sources): - SetTargetProperty(output, cmake_target_name, 'COMPILE_FLAGS', cflags, ' ') - - elif c_sources and not (s_sources or cxx_sources): - flags = [] - flags.extend(cflags) - flags.extend(cflags_c) - SetTargetProperty(output, cmake_target_name, 'COMPILE_FLAGS', flags, ' ') - - elif cxx_sources and not (s_sources or c_sources): - flags = [] - flags.extend(cflags) - flags.extend(cflags_cxx) - SetTargetProperty(output, cmake_target_name, 'COMPILE_FLAGS', flags, ' ') + WriteVariable(output, linkable_sources_name, " ") + if other_sources_name: + WriteVariable(output, other_sources_name, " ") + if dummy_sources_name: + WriteVariable(output, dummy_sources_name, " ") + + output.write(")\n") + + # Let CMake know if the 'all' target should depend on this target. + exclude_from_all = ( + "TRUE" if qualified_target not in all_qualified_targets else "FALSE" + ) + SetTargetProperty(output, cmake_target_name, "EXCLUDE_FROM_ALL", exclude_from_all) + for extra_target_name in extra_deps: + SetTargetProperty( + output, extra_target_name, "EXCLUDE_FROM_ALL", exclude_from_all + ) + + # Output name and location. + if target_type != "none": + # Link as 'C' if there are no other files + if not c_sources and not cxx_sources: + SetTargetProperty(output, cmake_target_name, "LINKER_LANGUAGE", ["C"]) + + # Mark uncompiled sources as uncompiled. + if other_sources_name: + output.write("set_source_files_properties(") + WriteVariable(output, other_sources_name, "") + output.write(' PROPERTIES HEADER_FILE_ONLY "TRUE")\n') + + # Mark object sources as linkable. + if linkable_sources_name: + output.write("set_source_files_properties(") + WriteVariable(output, other_sources_name, "") + output.write(' PROPERTIES EXTERNAL_OBJECT "TRUE")\n') + + # Output directory + target_output_directory = spec.get("product_dir") + if target_output_directory is None: + if target_type in ("executable", "loadable_module"): + target_output_directory = generator_default_variables["PRODUCT_DIR"] + elif target_type == "shared_library": + target_output_directory = "${builddir}/lib.${TOOLSET}" + elif spec.get("standalone_static_library", False): + target_output_directory = generator_default_variables["PRODUCT_DIR"] + else: + base_path = gyp.common.RelativePath( + os.path.dirname(gyp_file), options.toplevel_dir + ) + target_output_directory = "${obj}.${TOOLSET}" + target_output_directory = os.path.join( + target_output_directory, base_path + ) + + cmake_target_output_directory = NormjoinPathForceCMakeSource( + path_from_cmakelists_to_gyp, target_output_directory + ) + SetTargetProperty( + output, + cmake_target_name, + cmake_target_type.property_modifier + "_OUTPUT_DIRECTORY", + cmake_target_output_directory, + ) + + # Output name + default_product_prefix = "" + default_product_name = target_name + default_product_ext = "" + if target_type == "static_library": + static_library_prefix = generator_default_variables["STATIC_LIB_PREFIX"] + default_product_name = RemovePrefix( + default_product_name, static_library_prefix + ) + default_product_prefix = static_library_prefix + default_product_ext = generator_default_variables["STATIC_LIB_SUFFIX"] + + elif target_type in ("loadable_module", "shared_library"): + shared_library_prefix = generator_default_variables["SHARED_LIB_PREFIX"] + default_product_name = RemovePrefix( + default_product_name, shared_library_prefix + ) + default_product_prefix = shared_library_prefix + default_product_ext = generator_default_variables["SHARED_LIB_SUFFIX"] + + elif target_type != "executable": + print( + "ERROR: What output file should be generated?", + "type", + target_type, + "target", + target_name, + ) + + product_prefix = spec.get("product_prefix", default_product_prefix) + product_name = spec.get("product_name", default_product_name) + product_ext = spec.get("product_extension") + if product_ext: + product_ext = "." + product_ext + else: + product_ext = default_product_ext + + SetTargetProperty(output, cmake_target_name, "PREFIX", product_prefix) + SetTargetProperty( + output, + cmake_target_name, + cmake_target_type.property_modifier + "_OUTPUT_NAME", + product_name, + ) + SetTargetProperty(output, cmake_target_name, "SUFFIX", product_ext) + + # Make the output of this target referenceable as a source. + cmake_target_output_basename = product_prefix + product_name + product_ext + cmake_target_output = os.path.join( + cmake_target_output_directory, cmake_target_output_basename + ) + SetFileProperty(output, cmake_target_output, "GENERATED", ["TRUE"], "") + + # Includes + includes = config.get("include_dirs") + if includes: + # This (target include directories) is what requires CMake 2.8.8 + includes_name = cmake_target_name + "__include_dirs" + SetVariableList( + output, + includes_name, + [ + NormjoinPathForceCMakeSource(path_from_cmakelists_to_gyp, include) + for include in includes + ], + ) + output.write("set_property(TARGET ") + output.write(cmake_target_name) + output.write(" APPEND PROPERTY INCLUDE_DIRECTORIES ") + WriteVariable(output, includes_name, "") + output.write(")\n") + + # Defines + defines = config.get("defines") + if defines is not None: + SetTargetProperty( + output, cmake_target_name, "COMPILE_DEFINITIONS", defines, ";" + ) + + # Compile Flags - http://www.cmake.org/Bug/view.php?id=6493 + # CMake currently does not have target C and CXX flags. + # So, instead of doing... + + # cflags_c = config.get('cflags_c') + # if cflags_c is not None: + # SetTargetProperty(output, cmake_target_name, + # 'C_COMPILE_FLAGS', cflags_c, ' ') + + # cflags_cc = config.get('cflags_cc') + # if cflags_cc is not None: + # SetTargetProperty(output, cmake_target_name, + # 'CXX_COMPILE_FLAGS', cflags_cc, ' ') + + # Instead we must... + cflags = config.get("cflags", []) + cflags_c = config.get("cflags_c", []) + cflags_cxx = config.get("cflags_cc", []) + if xcode_settings: + cflags = xcode_settings.GetCflags(config_to_use) + cflags_c = xcode_settings.GetCflagsC(config_to_use) + cflags_cxx = xcode_settings.GetCflagsCC(config_to_use) + # cflags_objc = xcode_settings.GetCflagsObjC(config_to_use) + # cflags_objcc = xcode_settings.GetCflagsObjCC(config_to_use) + + if (not cflags_c or not c_sources) and (not cflags_cxx or not cxx_sources): + SetTargetProperty(output, cmake_target_name, "COMPILE_FLAGS", cflags, " ") + + elif c_sources and not (s_sources or cxx_sources): + flags = [] + flags.extend(cflags) + flags.extend(cflags_c) + SetTargetProperty(output, cmake_target_name, "COMPILE_FLAGS", flags, " ") + + elif cxx_sources and not (s_sources or c_sources): + flags = [] + flags.extend(cflags) + flags.extend(cflags_cxx) + SetTargetProperty(output, cmake_target_name, "COMPILE_FLAGS", flags, " ") + + else: + # TODO: This is broken, one cannot generally set properties on files, + # as other targets may require different properties on the same files. + if s_sources and cflags: + SetFilesProperty(output, s_sources_name, "COMPILE_FLAGS", cflags, " ") + + if c_sources and (cflags or cflags_c): + flags = [] + flags.extend(cflags) + flags.extend(cflags_c) + SetFilesProperty(output, c_sources_name, "COMPILE_FLAGS", flags, " ") + + if cxx_sources and (cflags or cflags_cxx): + flags = [] + flags.extend(cflags) + flags.extend(cflags_cxx) + SetFilesProperty(output, cxx_sources_name, "COMPILE_FLAGS", flags, " ") + + # Linker flags + ldflags = config.get("ldflags") + if ldflags is not None: + SetTargetProperty(output, cmake_target_name, "LINK_FLAGS", ldflags, " ") + + # XCode settings + xcode_settings = config.get("xcode_settings", {}) + for xcode_setting, xcode_value in xcode_settings.viewitems(): + SetTargetProperty( + output, + cmake_target_name, + "XCODE_ATTRIBUTE_%s" % xcode_setting, + xcode_value, + "" if isinstance(xcode_value, str) else " ", + ) + + # Note on Dependencies and Libraries: + # CMake wants to handle link order, resolving the link line up front. + # Gyp does not retain or enforce specifying enough information to do so. + # So do as other gyp generators and use --start-group and --end-group. + # Give CMake as little information as possible so that it doesn't mess it up. + + # Dependencies + rawDeps = spec.get("dependencies", []) + + static_deps = [] + shared_deps = [] + other_deps = [] + for rawDep in rawDeps: + dep_cmake_name = namer.CreateCMakeTargetName(rawDep) + dep_spec = target_dicts.get(rawDep, {}) + dep_target_type = dep_spec.get("type", None) + + if dep_target_type == "static_library": + static_deps.append(dep_cmake_name) + elif dep_target_type == "shared_library": + shared_deps.append(dep_cmake_name) + else: + other_deps.append(dep_cmake_name) + + # ensure all external dependencies are complete before internal dependencies + # extra_deps currently only depend on their own deps, so otherwise run early + if static_deps or shared_deps or other_deps: + for extra_dep in extra_deps: + output.write("add_dependencies(") + output.write(extra_dep) + output.write("\n") + for deps in (static_deps, shared_deps, other_deps): + for dep in gyp.common.uniquer(deps): + output.write(" ") + output.write(dep) + output.write("\n") + output.write(")\n") + + linkable = target_type in ("executable", "loadable_module", "shared_library") + other_deps.extend(extra_deps) + if other_deps or (not linkable and (static_deps or shared_deps)): + output.write("add_dependencies(") + output.write(cmake_target_name) + output.write("\n") + for dep in gyp.common.uniquer(other_deps): + output.write(" ") + output.write(dep) + output.write("\n") + if not linkable: + for deps in (static_deps, shared_deps): + for lib_dep in gyp.common.uniquer(deps): + output.write(" ") + output.write(lib_dep) + output.write("\n") + output.write(")\n") + + # Libraries + if linkable: + external_libs = [lib for lib in spec.get("libraries", []) if len(lib) > 0] + if external_libs or static_deps or shared_deps: + output.write("target_link_libraries(") + output.write(cmake_target_name) + output.write("\n") + if static_deps: + write_group = circular_libs and len(static_deps) > 1 and flavor != "mac" + if write_group: + output.write("-Wl,--start-group\n") + for dep in gyp.common.uniquer(static_deps): + output.write(" ") + output.write(dep) + output.write("\n") + if write_group: + output.write("-Wl,--end-group\n") + if shared_deps: + for dep in gyp.common.uniquer(shared_deps): + output.write(" ") + output.write(dep) + output.write("\n") + if external_libs: + for lib in gyp.common.uniquer(external_libs): + output.write(' "') + output.write(RemovePrefix(lib, "$(SDKROOT)")) + output.write('"\n') + + output.write(")\n") + + UnsetVariable(output, "TOOLSET") + UnsetVariable(output, "TARGET") + + +def GenerateOutputForConfig(target_list, target_dicts, data, params, config_to_use): + options = params["options"] + generator_flags = params["generator_flags"] + flavor = gyp.common.GetFlavor(params) + + # generator_dir: relative path from pwd to where make puts build files. + # Makes migrating from make to cmake easier, cmake doesn't put anything here. + # Each Gyp configuration creates a different CMakeLists.txt file + # to avoid incompatibilities between Gyp and CMake configurations. + generator_dir = os.path.relpath(options.generator_output or ".") + + # output_dir: relative path from generator_dir to the build directory. + output_dir = generator_flags.get("output_dir", "out") - else: - # TODO: This is broken, one cannot generally set properties on files, - # as other targets may require different properties on the same files. - if s_sources and cflags: - SetFilesProperty(output, s_sources_name, 'COMPILE_FLAGS', cflags, ' ') - - if c_sources and (cflags or cflags_c): - flags = [] - flags.extend(cflags) - flags.extend(cflags_c) - SetFilesProperty(output, c_sources_name, 'COMPILE_FLAGS', flags, ' ') - - if cxx_sources and (cflags or cflags_cxx): - flags = [] - flags.extend(cflags) - flags.extend(cflags_cxx) - SetFilesProperty(output, cxx_sources_name, 'COMPILE_FLAGS', flags, ' ') - - # Linker flags - ldflags = config.get('ldflags') - if ldflags is not None: - SetTargetProperty(output, cmake_target_name, 'LINK_FLAGS', ldflags, ' ') - - # XCode settings - xcode_settings = config.get('xcode_settings', {}) - for xcode_setting, xcode_value in xcode_settings.viewitems(): - SetTargetProperty(output, cmake_target_name, - "XCODE_ATTRIBUTE_%s" % xcode_setting, xcode_value, - '' if isinstance(xcode_value, str) else ' ') - - # Note on Dependencies and Libraries: - # CMake wants to handle link order, resolving the link line up front. - # Gyp does not retain or enforce specifying enough information to do so. - # So do as other gyp generators and use --start-group and --end-group. - # Give CMake as little information as possible so that it doesn't mess it up. - - # Dependencies - rawDeps = spec.get('dependencies', []) - - static_deps = [] - shared_deps = [] - other_deps = [] - for rawDep in rawDeps: - dep_cmake_name = namer.CreateCMakeTargetName(rawDep) - dep_spec = target_dicts.get(rawDep, {}) - dep_target_type = dep_spec.get('type', None) - - if dep_target_type == 'static_library': - static_deps.append(dep_cmake_name) - elif dep_target_type == 'shared_library': - shared_deps.append(dep_cmake_name) - else: - other_deps.append(dep_cmake_name) - - # ensure all external dependencies are complete before internal dependencies - # extra_deps currently only depend on their own deps, so otherwise run early - if static_deps or shared_deps or other_deps: - for extra_dep in extra_deps: - output.write('add_dependencies(') - output.write(extra_dep) - output.write('\n') - for deps in (static_deps, shared_deps, other_deps): - for dep in gyp.common.uniquer(deps): - output.write(' ') - output.write(dep) - output.write('\n') - output.write(')\n') - - linkable = target_type in ('executable', 'loadable_module', 'shared_library') - other_deps.extend(extra_deps) - if other_deps or (not linkable and (static_deps or shared_deps)): - output.write('add_dependencies(') - output.write(cmake_target_name) - output.write('\n') - for dep in gyp.common.uniquer(other_deps): - output.write(' ') - output.write(dep) - output.write('\n') - if not linkable: - for deps in (static_deps, shared_deps): - for lib_dep in gyp.common.uniquer(deps): - output.write(' ') - output.write(lib_dep) - output.write('\n') - output.write(')\n') - - # Libraries - if linkable: - external_libs = [lib for lib in spec.get('libraries', []) if len(lib) > 0] - if external_libs or static_deps or shared_deps: - output.write('target_link_libraries(') - output.write(cmake_target_name) - output.write('\n') - if static_deps: - write_group = circular_libs and len(static_deps) > 1 and flavor != 'mac' - if write_group: - output.write('-Wl,--start-group\n') - for dep in gyp.common.uniquer(static_deps): - output.write(' ') - output.write(dep) - output.write('\n') - if write_group: - output.write('-Wl,--end-group\n') - if shared_deps: - for dep in gyp.common.uniquer(shared_deps): - output.write(' ') - output.write(dep) - output.write('\n') - if external_libs: - for lib in gyp.common.uniquer(external_libs): - output.write(' "') - output.write(RemovePrefix(lib, "$(SDKROOT)")) - output.write('"\n') - - output.write(')\n') - - UnsetVariable(output, 'TOOLSET') - UnsetVariable(output, 'TARGET') - - -def GenerateOutputForConfig(target_list, target_dicts, data, - params, config_to_use): - options = params['options'] - generator_flags = params['generator_flags'] - flavor = gyp.common.GetFlavor(params) - - # generator_dir: relative path from pwd to where make puts build files. - # Makes migrating from make to cmake easier, cmake doesn't put anything here. - # Each Gyp configuration creates a different CMakeLists.txt file - # to avoid incompatibilities between Gyp and CMake configurations. - generator_dir = os.path.relpath(options.generator_output or '.') - - # output_dir: relative path from generator_dir to the build directory. - output_dir = generator_flags.get('output_dir', 'out') - - # build_dir: relative path from source root to our output files. - # e.g. "out/Debug" - build_dir = os.path.normpath(os.path.join(generator_dir, - output_dir, - config_to_use)) - - toplevel_build = os.path.join(options.toplevel_dir, build_dir) - - output_file = os.path.join(toplevel_build, 'CMakeLists.txt') - gyp.common.EnsureDirExists(output_file) - - output = open(output_file, 'w') - output.write('cmake_minimum_required(VERSION 2.8.8 FATAL_ERROR)\n') - output.write('cmake_policy(VERSION 2.8.8)\n') - - gyp_file, project_target, _ = gyp.common.ParseQualifiedTarget(target_list[-1]) - output.write('project(') - output.write(project_target) - output.write(')\n') - - SetVariable(output, 'configuration', config_to_use) - - ar = None - cc = None - cxx = None - - make_global_settings = data[gyp_file].get('make_global_settings', []) - build_to_top = gyp.common.InvertRelativePath(build_dir, - options.toplevel_dir) - for key, value in make_global_settings: - if key == 'AR': - ar = os.path.join(build_to_top, value) - if key == 'CC': - cc = os.path.join(build_to_top, value) - if key == 'CXX': - cxx = os.path.join(build_to_top, value) - - ar = gyp.common.GetEnvironFallback(['AR_target', 'AR'], ar) - cc = gyp.common.GetEnvironFallback(['CC_target', 'CC'], cc) - cxx = gyp.common.GetEnvironFallback(['CXX_target', 'CXX'], cxx) - - if ar: - SetVariable(output, 'CMAKE_AR', ar) - if cc: - SetVariable(output, 'CMAKE_C_COMPILER', cc) - if cxx: - SetVariable(output, 'CMAKE_CXX_COMPILER', cxx) - - # The following appears to be as-yet undocumented. - # http://public.kitware.com/Bug/view.php?id=8392 - output.write('enable_language(ASM)\n') - # ASM-ATT does not support .S files. - # output.write('enable_language(ASM-ATT)\n') - - if cc: - SetVariable(output, 'CMAKE_ASM_COMPILER', cc) - - SetVariable(output, 'builddir', '${CMAKE_CURRENT_BINARY_DIR}') - SetVariable(output, 'obj', '${builddir}/obj') - output.write('\n') - - # TODO: Undocumented/unsupported (the CMake Java generator depends on it). - # CMake by default names the object resulting from foo.c to be foo.c.o. - # Gyp traditionally names the object resulting from foo.c foo.o. - # This should be irrelevant, but some targets extract .o files from .a - # and depend on the name of the extracted .o files. - output.write('set(CMAKE_C_OUTPUT_EXTENSION_REPLACE 1)\n') - output.write('set(CMAKE_CXX_OUTPUT_EXTENSION_REPLACE 1)\n') - output.write('\n') - - # Force ninja to use rsp files. Otherwise link and ar lines can get too long, - # resulting in 'Argument list too long' errors. - # However, rsp files don't work correctly on Mac. - if flavor != 'mac': - output.write('set(CMAKE_NINJA_FORCE_RESPONSE_FILE 1)\n') - output.write('\n') - - namer = CMakeNamer(target_list) - - # The list of targets upon which the 'all' target should depend. - # CMake has it's own implicit 'all' target, one is not created explicitly. - all_qualified_targets = set() - for build_file in params['build_files']: - for qualified_target in gyp.common.AllTargets(target_list, - target_dicts, - os.path.normpath(build_file)): - all_qualified_targets.add(qualified_target) - - for qualified_target in target_list: - if flavor == 'mac': - gyp_file, _, _ = gyp.common.ParseQualifiedTarget(qualified_target) - spec = target_dicts[qualified_target] - gyp.xcode_emulation.MergeGlobalXcodeSettingsToSpec(data[gyp_file], spec) - - WriteTarget(namer, qualified_target, target_dicts, build_dir, config_to_use, - options, generator_flags, all_qualified_targets, flavor, output) - - output.close() + # build_dir: relative path from source root to our output files. + # e.g. "out/Debug" + build_dir = os.path.normpath(os.path.join(generator_dir, output_dir, config_to_use)) + + toplevel_build = os.path.join(options.toplevel_dir, build_dir) + + output_file = os.path.join(toplevel_build, "CMakeLists.txt") + gyp.common.EnsureDirExists(output_file) + + output = open(output_file, "w") + output.write("cmake_minimum_required(VERSION 2.8.8 FATAL_ERROR)\n") + output.write("cmake_policy(VERSION 2.8.8)\n") + + gyp_file, project_target, _ = gyp.common.ParseQualifiedTarget(target_list[-1]) + output.write("project(") + output.write(project_target) + output.write(")\n") + + SetVariable(output, "configuration", config_to_use) + + ar = None + cc = None + cxx = None + + make_global_settings = data[gyp_file].get("make_global_settings", []) + build_to_top = gyp.common.InvertRelativePath(build_dir, options.toplevel_dir) + for key, value in make_global_settings: + if key == "AR": + ar = os.path.join(build_to_top, value) + if key == "CC": + cc = os.path.join(build_to_top, value) + if key == "CXX": + cxx = os.path.join(build_to_top, value) + + ar = gyp.common.GetEnvironFallback(["AR_target", "AR"], ar) + cc = gyp.common.GetEnvironFallback(["CC_target", "CC"], cc) + cxx = gyp.common.GetEnvironFallback(["CXX_target", "CXX"], cxx) + + if ar: + SetVariable(output, "CMAKE_AR", ar) + if cc: + SetVariable(output, "CMAKE_C_COMPILER", cc) + if cxx: + SetVariable(output, "CMAKE_CXX_COMPILER", cxx) + + # The following appears to be as-yet undocumented. + # http://public.kitware.com/Bug/view.php?id=8392 + output.write("enable_language(ASM)\n") + # ASM-ATT does not support .S files. + # output.write('enable_language(ASM-ATT)\n') + + if cc: + SetVariable(output, "CMAKE_ASM_COMPILER", cc) + + SetVariable(output, "builddir", "${CMAKE_CURRENT_BINARY_DIR}") + SetVariable(output, "obj", "${builddir}/obj") + output.write("\n") + + # TODO: Undocumented/unsupported (the CMake Java generator depends on it). + # CMake by default names the object resulting from foo.c to be foo.c.o. + # Gyp traditionally names the object resulting from foo.c foo.o. + # This should be irrelevant, but some targets extract .o files from .a + # and depend on the name of the extracted .o files. + output.write("set(CMAKE_C_OUTPUT_EXTENSION_REPLACE 1)\n") + output.write("set(CMAKE_CXX_OUTPUT_EXTENSION_REPLACE 1)\n") + output.write("\n") + + # Force ninja to use rsp files. Otherwise link and ar lines can get too long, + # resulting in 'Argument list too long' errors. + # However, rsp files don't work correctly on Mac. + if flavor != "mac": + output.write("set(CMAKE_NINJA_FORCE_RESPONSE_FILE 1)\n") + output.write("\n") + + namer = CMakeNamer(target_list) + + # The list of targets upon which the 'all' target should depend. + # CMake has it's own implicit 'all' target, one is not created explicitly. + all_qualified_targets = set() + for build_file in params["build_files"]: + for qualified_target in gyp.common.AllTargets( + target_list, target_dicts, os.path.normpath(build_file) + ): + all_qualified_targets.add(qualified_target) + + for qualified_target in target_list: + if flavor == "mac": + gyp_file, _, _ = gyp.common.ParseQualifiedTarget(qualified_target) + spec = target_dicts[qualified_target] + gyp.xcode_emulation.MergeGlobalXcodeSettingsToSpec(data[gyp_file], spec) + + WriteTarget( + namer, + qualified_target, + target_dicts, + build_dir, + config_to_use, + options, + generator_flags, + all_qualified_targets, + flavor, + output, + ) + + output.close() def PerformBuild(data, configurations, params): - options = params['options'] - generator_flags = params['generator_flags'] + options = params["options"] + generator_flags = params["generator_flags"] - # generator_dir: relative path from pwd to where make puts build files. - # Makes migrating from make to cmake easier, cmake doesn't put anything here. - generator_dir = os.path.relpath(options.generator_output or '.') + # generator_dir: relative path from pwd to where make puts build files. + # Makes migrating from make to cmake easier, cmake doesn't put anything here. + generator_dir = os.path.relpath(options.generator_output or ".") - # output_dir: relative path from generator_dir to the build directory. - output_dir = generator_flags.get('output_dir', 'out') + # output_dir: relative path from generator_dir to the build directory. + output_dir = generator_flags.get("output_dir", "out") - for config_name in configurations: - # build_dir: relative path from source root to our output files. - # e.g. "out/Debug" - build_dir = os.path.normpath(os.path.join(generator_dir, - output_dir, - config_name)) - arguments = ['cmake', '-G', 'Ninja'] - print('Generating [%s]: %s' % (config_name, arguments)) - subprocess.check_call(arguments, cwd=build_dir) + for config_name in configurations: + # build_dir: relative path from source root to our output files. + # e.g. "out/Debug" + build_dir = os.path.normpath( + os.path.join(generator_dir, output_dir, config_name) + ) + arguments = ["cmake", "-G", "Ninja"] + print("Generating [%s]: %s" % (config_name, arguments)) + subprocess.check_call(arguments, cwd=build_dir) - arguments = ['ninja', '-C', build_dir] - print('Building [%s]: %s' % (config_name, arguments)) - subprocess.check_call(arguments) + arguments = ["ninja", "-C", build_dir] + print("Building [%s]: %s" % (config_name, arguments)) + subprocess.check_call(arguments) def CallGenerateOutputForConfig(arglist): - # Ignore the interrupt signal so that the parent process catches it and - # kills all multiprocessing children. - signal.signal(signal.SIGINT, signal.SIG_IGN) + # Ignore the interrupt signal so that the parent process catches it and + # kills all multiprocessing children. + signal.signal(signal.SIGINT, signal.SIG_IGN) - target_list, target_dicts, data, params, config_name = arglist - GenerateOutputForConfig(target_list, target_dicts, data, params, config_name) + target_list, target_dicts, data, params, config_name = arglist + GenerateOutputForConfig(target_list, target_dicts, data, params, config_name) def GenerateOutput(target_list, target_dicts, data, params): - user_config = params.get('generator_flags', {}).get('config', None) - if user_config: - GenerateOutputForConfig(target_list, target_dicts, data, - params, user_config) - else: - config_names = target_dicts[target_list[0]]['configurations'].keys() - if params['parallel']: - try: - pool = multiprocessing.Pool(len(config_names)) - arglists = [] - for config_name in config_names: - arglists.append((target_list, target_dicts, data, - params, config_name)) - pool.map(CallGenerateOutputForConfig, arglists) - except KeyboardInterrupt as e: - pool.terminate() - raise e + user_config = params.get("generator_flags", {}).get("config", None) + if user_config: + GenerateOutputForConfig(target_list, target_dicts, data, params, user_config) else: - for config_name in config_names: - GenerateOutputForConfig(target_list, target_dicts, data, - params, config_name) + config_names = target_dicts[target_list[0]]["configurations"] + if params["parallel"]: + try: + pool = multiprocessing.Pool(len(config_names)) + arglists = [] + for config_name in config_names: + arglists.append( + (target_list, target_dicts, data, params, config_name) + ) + pool.map(CallGenerateOutputForConfig, arglists) + except KeyboardInterrupt as e: + pool.terminate() + raise e + else: + for config_name in config_names: + GenerateOutputForConfig( + target_list, target_dicts, data, params, config_name + ) diff --git a/gyp/pylib/gyp/generator/compile_commands_json.py b/gyp/pylib/gyp/generator/compile_commands_json.py index 1b8490451f..9bc21ff834 100644 --- a/gyp/pylib/gyp/generator/compile_commands_json.py +++ b/gyp/pylib/gyp/generator/compile_commands_json.py @@ -16,100 +16,105 @@ # Lifted from make.py. The actual values don't matter much. generator_default_variables = { - 'CONFIGURATION_NAME': '$(BUILDTYPE)', - 'EXECUTABLE_PREFIX': '', - 'EXECUTABLE_SUFFIX': '', - 'INTERMEDIATE_DIR': '$(obj).$(TOOLSET)/$(TARGET)/geni', - 'PRODUCT_DIR': '$(builddir)', - 'RULE_INPUT_DIRNAME': '%(INPUT_DIRNAME)s', - 'RULE_INPUT_EXT': '$(suffix $<)', - 'RULE_INPUT_NAME': '$(notdir $<)', - 'RULE_INPUT_PATH': '$(abspath $<)', - 'RULE_INPUT_ROOT': '%(INPUT_ROOT)s', - 'SHARED_INTERMEDIATE_DIR': '$(obj)/gen', - 'SHARED_LIB_PREFIX': 'lib', - 'STATIC_LIB_PREFIX': 'lib', - 'STATIC_LIB_SUFFIX': '.a', + "CONFIGURATION_NAME": "$(BUILDTYPE)", + "EXECUTABLE_PREFIX": "", + "EXECUTABLE_SUFFIX": "", + "INTERMEDIATE_DIR": "$(obj).$(TOOLSET)/$(TARGET)/geni", + "PRODUCT_DIR": "$(builddir)", + "RULE_INPUT_DIRNAME": "%(INPUT_DIRNAME)s", + "RULE_INPUT_EXT": "$(suffix $<)", + "RULE_INPUT_NAME": "$(notdir $<)", + "RULE_INPUT_PATH": "$(abspath $<)", + "RULE_INPUT_ROOT": "%(INPUT_ROOT)s", + "SHARED_INTERMEDIATE_DIR": "$(obj)/gen", + "SHARED_LIB_PREFIX": "lib", + "STATIC_LIB_PREFIX": "lib", + "STATIC_LIB_SUFFIX": ".a", } def IsMac(params): - return 'mac' == gyp.common.GetFlavor(params) + return "mac" == gyp.common.GetFlavor(params) def CalculateVariables(default_variables, params): - default_variables.setdefault('OS', gyp.common.GetFlavor(params)) + default_variables.setdefault("OS", gyp.common.GetFlavor(params)) def AddCommandsForTarget(cwd, target, params, per_config_commands): - output_dir = params['generator_flags']['output_dir'] - for configuration_name, configuration in target['configurations'].items(): - builddir_name = os.path.join(output_dir, configuration_name) - - if IsMac(params): - xcode_settings = gyp.xcode_emulation.XcodeSettings(target) - cflags = xcode_settings.GetCflags(configuration_name) - cflags_c = xcode_settings.GetCflagsC(configuration_name) - cflags_cc = xcode_settings.GetCflagsCC(configuration_name) - else: - cflags = configuration.get('cflags', []) - cflags_c = configuration.get('cflags_c', []) - cflags_cc = configuration.get('cflags_cc', []) - - cflags_c = cflags + cflags_c - cflags_cc = cflags + cflags_cc - - defines = configuration.get('defines', []) - defines = ['-D' + s for s in defines] - - # TODO(bnoordhuis) Handle generated source files. - sources = target.get('sources', []) - sources = [s for s in sources if s.endswith('.c') or s.endswith('.cc')] - - def resolve(filename): - return os.path.abspath(os.path.join(cwd, filename)) - - # TODO(bnoordhuis) Handle generated header files. - include_dirs = configuration.get('include_dirs', []) - include_dirs = [s for s in include_dirs if not s.startswith('$(obj)')] - includes = ['-I' + resolve(s) for s in include_dirs] - - defines = gyp.common.EncodePOSIXShellList(defines) - includes = gyp.common.EncodePOSIXShellList(includes) - cflags_c = gyp.common.EncodePOSIXShellList(cflags_c) - cflags_cc = gyp.common.EncodePOSIXShellList(cflags_cc) - - commands = per_config_commands.setdefault(configuration_name, []) - for source in sources: - file = resolve(source) - isc = source.endswith('.c') - cc = 'cc' if isc else 'c++' - cflags = cflags_c if isc else cflags_cc - command = ' '.join((cc, defines, includes, cflags, - '-c', gyp.common.EncodePOSIXShellArgument(file))) - commands.append(dict(command=command, directory=output_dir, file=file)) + output_dir = params["generator_flags"].get("output_dir", "out") + for configuration_name, configuration in target["configurations"].items(): + if IsMac(params): + xcode_settings = gyp.xcode_emulation.XcodeSettings(target) + cflags = xcode_settings.GetCflags(configuration_name) + cflags_c = xcode_settings.GetCflagsC(configuration_name) + cflags_cc = xcode_settings.GetCflagsCC(configuration_name) + else: + cflags = configuration.get("cflags", []) + cflags_c = configuration.get("cflags_c", []) + cflags_cc = configuration.get("cflags_cc", []) + + cflags_c = cflags + cflags_c + cflags_cc = cflags + cflags_cc + + defines = configuration.get("defines", []) + defines = ["-D" + s for s in defines] + + # TODO(bnoordhuis) Handle generated source files. + sources = target.get("sources", []) + sources = [s for s in sources if s.endswith(".c") or s.endswith(".cc")] + + def resolve(filename): + return os.path.abspath(os.path.join(cwd, filename)) + + # TODO(bnoordhuis) Handle generated header files. + include_dirs = configuration.get("include_dirs", []) + include_dirs = [s for s in include_dirs if not s.startswith("$(obj)")] + includes = ["-I" + resolve(s) for s in include_dirs] + + defines = gyp.common.EncodePOSIXShellList(defines) + includes = gyp.common.EncodePOSIXShellList(includes) + cflags_c = gyp.common.EncodePOSIXShellList(cflags_c) + cflags_cc = gyp.common.EncodePOSIXShellList(cflags_cc) + + commands = per_config_commands.setdefault(configuration_name, []) + for source in sources: + file = resolve(source) + isc = source.endswith(".c") + cc = "cc" if isc else "c++" + cflags = cflags_c if isc else cflags_cc + command = " ".join( + ( + cc, + defines, + includes, + cflags, + "-c", + gyp.common.EncodePOSIXShellArgument(file), + ) + ) + commands.append(dict(command=command, directory=output_dir, file=file)) def GenerateOutput(target_list, target_dicts, data, params): - per_config_commands = {} - for qualified_target, target in target_dicts.items(): - build_file, target_name, toolset = ( - gyp.common.ParseQualifiedTarget(qualified_target)) - if IsMac(params): - settings = data[build_file] - gyp.xcode_emulation.MergeGlobalXcodeSettingsToSpec(settings, target) - cwd = os.path.dirname(build_file) - AddCommandsForTarget(cwd, target, params, per_config_commands) - - output_dir = params['generator_flags']['output_dir'] - for configuration_name, commands in per_config_commands.items(): - filename = os.path.join(output_dir, - configuration_name, - 'compile_commands.json') - gyp.common.EnsureDirExists(filename) - fp = open(filename, 'w') - json.dump(commands, fp=fp, indent=0, check_circular=False) + per_config_commands = {} + for qualified_target, target in target_dicts.items(): + build_file, target_name, toolset = gyp.common.ParseQualifiedTarget( + qualified_target + ) + if IsMac(params): + settings = data[build_file] + gyp.xcode_emulation.MergeGlobalXcodeSettingsToSpec(settings, target) + cwd = os.path.dirname(build_file) + AddCommandsForTarget(cwd, target, params, per_config_commands) + + output_dir = params["generator_flags"].get("output_dir", "out") + for configuration_name, commands in per_config_commands.items(): + filename = os.path.join(output_dir, configuration_name, "compile_commands.json") + gyp.common.EnsureDirExists(filename) + fp = open(filename, "w") + json.dump(commands, fp=fp, indent=0, check_circular=False) def PerformBuild(data, configurations, params): - pass + pass diff --git a/gyp/pylib/gyp/generator/dump_dependency_json.py b/gyp/pylib/gyp/generator/dump_dependency_json.py index 8e4f3168f3..46f68e0384 100644 --- a/gyp/pylib/gyp/generator/dump_dependency_json.py +++ b/gyp/pylib/gyp/generator/dump_dependency_json.py @@ -1,100 +1,104 @@ -from __future__ import print_function # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. -import collections +from __future__ import print_function + import os import gyp import gyp.common import gyp.msvs_emulation import json -import sys generator_supports_multiple_toolsets = True generator_wants_static_library_dependencies_adjusted = False -generator_filelist_paths = { -} - -generator_default_variables = { -} -for dirname in ['INTERMEDIATE_DIR', 'SHARED_INTERMEDIATE_DIR', 'PRODUCT_DIR', - 'LIB_DIR', 'SHARED_LIB_DIR']: - # Some gyp steps fail if these are empty(!). - generator_default_variables[dirname] = 'dir' -for unused in ['RULE_INPUT_PATH', 'RULE_INPUT_ROOT', 'RULE_INPUT_NAME', - 'RULE_INPUT_DIRNAME', 'RULE_INPUT_EXT', - 'EXECUTABLE_PREFIX', 'EXECUTABLE_SUFFIX', - 'STATIC_LIB_PREFIX', 'STATIC_LIB_SUFFIX', - 'SHARED_LIB_PREFIX', 'SHARED_LIB_SUFFIX', - 'CONFIGURATION_NAME']: - generator_default_variables[unused] = '' +generator_filelist_paths = {} + +generator_default_variables = {} +for dirname in [ + "INTERMEDIATE_DIR", + "SHARED_INTERMEDIATE_DIR", + "PRODUCT_DIR", + "LIB_DIR", + "SHARED_LIB_DIR", +]: + # Some gyp steps fail if these are empty(!). + generator_default_variables[dirname] = "dir" +for unused in [ + "RULE_INPUT_PATH", + "RULE_INPUT_ROOT", + "RULE_INPUT_NAME", + "RULE_INPUT_DIRNAME", + "RULE_INPUT_EXT", + "EXECUTABLE_PREFIX", + "EXECUTABLE_SUFFIX", + "STATIC_LIB_PREFIX", + "STATIC_LIB_SUFFIX", + "SHARED_LIB_PREFIX", + "SHARED_LIB_SUFFIX", + "CONFIGURATION_NAME", +]: + generator_default_variables[unused] = "" def CalculateVariables(default_variables, params): - generator_flags = params.get('generator_flags', {}) - for key, val in generator_flags.items(): - default_variables.setdefault(key, val) - default_variables.setdefault('OS', gyp.common.GetFlavor(params)) - - flavor = gyp.common.GetFlavor(params) - if flavor =='win': - # Copy additional generator configuration data from VS, which is shared - # by the Windows Ninja generator. - import gyp.generator.msvs as msvs_generator - generator_additional_non_configuration_keys = getattr(msvs_generator, - 'generator_additional_non_configuration_keys', []) - generator_additional_path_sections = getattr(msvs_generator, - 'generator_additional_path_sections', []) + generator_flags = params.get("generator_flags", {}) + for key, val in generator_flags.items(): + default_variables.setdefault(key, val) + default_variables.setdefault("OS", gyp.common.GetFlavor(params)) - gyp.msvs_emulation.CalculateCommonVariables(default_variables, params) + flavor = gyp.common.GetFlavor(params) + if flavor == "win": + gyp.msvs_emulation.CalculateCommonVariables(default_variables, params) def CalculateGeneratorInputInfo(params): - """Calculate the generator specific info that gets fed to input (called by + """Calculate the generator specific info that gets fed to input (called by gyp).""" - generator_flags = params.get('generator_flags', {}) - if generator_flags.get('adjust_static_libraries', False): - global generator_wants_static_library_dependencies_adjusted - generator_wants_static_library_dependencies_adjusted = True - - toplevel = params['options'].toplevel_dir - generator_dir = os.path.relpath(params['options'].generator_output or '.') - # output_dir: relative path from generator_dir to the build directory. - output_dir = generator_flags.get('output_dir', 'out') - qualified_out_dir = os.path.normpath(os.path.join( - toplevel, generator_dir, output_dir, 'gypfiles')) - global generator_filelist_paths - generator_filelist_paths = { - 'toplevel': toplevel, - 'qualified_out_dir': qualified_out_dir, - } + generator_flags = params.get("generator_flags", {}) + if generator_flags.get("adjust_static_libraries", False): + global generator_wants_static_library_dependencies_adjusted + generator_wants_static_library_dependencies_adjusted = True + + toplevel = params["options"].toplevel_dir + generator_dir = os.path.relpath(params["options"].generator_output or ".") + # output_dir: relative path from generator_dir to the build directory. + output_dir = generator_flags.get("output_dir", "out") + qualified_out_dir = os.path.normpath( + os.path.join(toplevel, generator_dir, output_dir, "gypfiles") + ) + global generator_filelist_paths + generator_filelist_paths = { + "toplevel": toplevel, + "qualified_out_dir": qualified_out_dir, + } + def GenerateOutput(target_list, target_dicts, data, params): - # Map of target -> list of targets it depends on. - edges = {} - - # Queue of targets to visit. - targets_to_visit = target_list[:] - - while len(targets_to_visit) > 0: - target = targets_to_visit.pop() - if target in edges: - continue - edges[target] = [] - - for dep in target_dicts[target].get('dependencies', []): - edges[target].append(dep) - targets_to_visit.append(dep) - - try: - filepath = params['generator_flags']['output_dir'] - except KeyError: - filepath = '.' - filename = os.path.join(filepath, 'dump.json') - f = open(filename, 'w') - json.dump(edges, f) - f.close() - print('Wrote json to %s.' % filename) + # Map of target -> list of targets it depends on. + edges = {} + + # Queue of targets to visit. + targets_to_visit = target_list[:] + + while len(targets_to_visit) > 0: + target = targets_to_visit.pop() + if target in edges: + continue + edges[target] = [] + + for dep in target_dicts[target].get("dependencies", []): + edges[target].append(dep) + targets_to_visit.append(dep) + + try: + filepath = params["generator_flags"]["output_dir"] + except KeyError: + filepath = "." + filename = os.path.join(filepath, "dump.json") + f = open(filename, "w") + json.dump(edges, f) + f.close() + print("Wrote json to %s." % filename) diff --git a/gyp/pylib/gyp/generator/eclipse.py b/gyp/pylib/gyp/generator/eclipse.py index 80e5fb6302..4bd49725dc 100644 --- a/gyp/pylib/gyp/generator/eclipse.py +++ b/gyp/pylib/gyp/generator/eclipse.py @@ -30,402 +30,441 @@ generator_wants_static_library_dependencies_adjusted = False -generator_default_variables = { -} - -for dirname in ['INTERMEDIATE_DIR', 'PRODUCT_DIR', 'LIB_DIR', 'SHARED_LIB_DIR']: - # Some gyp steps fail if these are empty(!), so we convert them to variables - generator_default_variables[dirname] = '$' + dirname - -for unused in ['RULE_INPUT_PATH', 'RULE_INPUT_ROOT', 'RULE_INPUT_NAME', - 'RULE_INPUT_DIRNAME', 'RULE_INPUT_EXT', - 'EXECUTABLE_PREFIX', 'EXECUTABLE_SUFFIX', - 'STATIC_LIB_PREFIX', 'STATIC_LIB_SUFFIX', - 'SHARED_LIB_PREFIX', 'SHARED_LIB_SUFFIX', - 'CONFIGURATION_NAME']: - generator_default_variables[unused] = '' +generator_default_variables = {} + +for dirname in ["INTERMEDIATE_DIR", "PRODUCT_DIR", "LIB_DIR", "SHARED_LIB_DIR"]: + # Some gyp steps fail if these are empty(!), so we convert them to variables + generator_default_variables[dirname] = "$" + dirname + +for unused in [ + "RULE_INPUT_PATH", + "RULE_INPUT_ROOT", + "RULE_INPUT_NAME", + "RULE_INPUT_DIRNAME", + "RULE_INPUT_EXT", + "EXECUTABLE_PREFIX", + "EXECUTABLE_SUFFIX", + "STATIC_LIB_PREFIX", + "STATIC_LIB_SUFFIX", + "SHARED_LIB_PREFIX", + "SHARED_LIB_SUFFIX", + "CONFIGURATION_NAME", +]: + generator_default_variables[unused] = "" # Include dirs will occasionally use the SHARED_INTERMEDIATE_DIR variable as # part of the path when dealing with generated headers. This value will be # replaced dynamically for each configuration. -generator_default_variables['SHARED_INTERMEDIATE_DIR'] = \ - '$SHARED_INTERMEDIATE_DIR' +generator_default_variables["SHARED_INTERMEDIATE_DIR"] = "$SHARED_INTERMEDIATE_DIR" def CalculateVariables(default_variables, params): - generator_flags = params.get('generator_flags', {}) - for key, val in generator_flags.items(): - default_variables.setdefault(key, val) - flavor = gyp.common.GetFlavor(params) - default_variables.setdefault('OS', flavor) - if flavor == 'win': - # Copy additional generator configuration data from VS, which is shared - # by the Eclipse generator. - import gyp.generator.msvs as msvs_generator - generator_additional_non_configuration_keys = getattr(msvs_generator, - 'generator_additional_non_configuration_keys', []) - generator_additional_path_sections = getattr(msvs_generator, - 'generator_additional_path_sections', []) - - gyp.msvs_emulation.CalculateCommonVariables(default_variables, params) + generator_flags = params.get("generator_flags", {}) + for key, val in generator_flags.items(): + default_variables.setdefault(key, val) + flavor = gyp.common.GetFlavor(params) + default_variables.setdefault("OS", flavor) + if flavor == "win": + gyp.msvs_emulation.CalculateCommonVariables(default_variables, params) def CalculateGeneratorInputInfo(params): - """Calculate the generator specific info that gets fed to input (called by + """Calculate the generator specific info that gets fed to input (called by gyp).""" - generator_flags = params.get('generator_flags', {}) - if generator_flags.get('adjust_static_libraries', False): - global generator_wants_static_library_dependencies_adjusted - generator_wants_static_library_dependencies_adjusted = True - - -def GetAllIncludeDirectories(target_list, target_dicts, - shared_intermediate_dirs, config_name, params, - compiler_path): - """Calculate the set of include directories to be used. + generator_flags = params.get("generator_flags", {}) + if generator_flags.get("adjust_static_libraries", False): + global generator_wants_static_library_dependencies_adjusted + generator_wants_static_library_dependencies_adjusted = True + + +def GetAllIncludeDirectories( + target_list, + target_dicts, + shared_intermediate_dirs, + config_name, + params, + compiler_path, +): + """Calculate the set of include directories to be used. Returns: A list including all the include_dir's specified for every target followed by any include directories that were added as cflag compiler options. """ - gyp_includes_set = set() - compiler_includes_list = [] - - # Find compiler's default include dirs. - if compiler_path: - command = shlex.split(compiler_path) - command.extend(['-E', '-xc++', '-v', '-']) - proc = subprocess.Popen(args=command, stdin=subprocess.PIPE, - stdout=subprocess.PIPE, stderr=subprocess.PIPE) - output = proc.communicate()[1] - if PY3: - output = output.decode('utf-8') - # Extract the list of include dirs from the output, which has this format: - # ... - # #include "..." search starts here: - # #include <...> search starts here: - # /usr/include/c++/4.6 - # /usr/local/include - # End of search list. - # ... - in_include_list = False - for line in output.splitlines(): - if line.startswith('#include'): - in_include_list = True - continue - if line.startswith('End of search list.'): - break - if in_include_list: - include_dir = line.strip() - if include_dir not in compiler_includes_list: - compiler_includes_list.append(include_dir) - - flavor = gyp.common.GetFlavor(params) - if flavor == 'win': - generator_flags = params.get('generator_flags', {}) - for target_name in target_list: - target = target_dicts[target_name] - if config_name in target['configurations']: - config = target['configurations'][config_name] - - # Look for any include dirs that were explicitly added via cflags. This - # may be done in gyp files to force certain includes to come at the end. - # TODO(jgreenwald): Change the gyp files to not abuse cflags for this, and - # remove this. - if flavor == 'win': - msvs_settings = gyp.msvs_emulation.MsvsSettings(target, generator_flags) - cflags = msvs_settings.GetCflags(config_name) - else: - cflags = config['cflags'] - for cflag in cflags: - if cflag.startswith('-I'): - include_dir = cflag[2:] - if include_dir not in compiler_includes_list: - compiler_includes_list.append(include_dir) - - # Find standard gyp include dirs. - if 'include_dirs' in config: - include_dirs = config['include_dirs'] - for shared_intermediate_dir in shared_intermediate_dirs: - for include_dir in include_dirs: - include_dir = include_dir.replace('$SHARED_INTERMEDIATE_DIR', - shared_intermediate_dir) - if not os.path.isabs(include_dir): - base_dir = os.path.dirname(target_name) - - include_dir = base_dir + '/' + include_dir - include_dir = os.path.abspath(include_dir) - - gyp_includes_set.add(include_dir) - - # Generate a list that has all the include dirs. - all_includes_list = list(gyp_includes_set) - all_includes_list.sort() - for compiler_include in compiler_includes_list: - if not compiler_include in gyp_includes_set: - all_includes_list.append(compiler_include) - - # All done. - return all_includes_list + gyp_includes_set = set() + compiler_includes_list = [] + + # Find compiler's default include dirs. + if compiler_path: + command = shlex.split(compiler_path) + command.extend(["-E", "-xc++", "-v", "-"]) + proc = subprocess.Popen( + args=command, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + output = proc.communicate()[1] + if PY3: + output = output.decode("utf-8") + # Extract the list of include dirs from the output, which has this format: + # ... + # #include "..." search starts here: + # #include <...> search starts here: + # /usr/include/c++/4.6 + # /usr/local/include + # End of search list. + # ... + in_include_list = False + for line in output.splitlines(): + if line.startswith("#include"): + in_include_list = True + continue + if line.startswith("End of search list."): + break + if in_include_list: + include_dir = line.strip() + if include_dir not in compiler_includes_list: + compiler_includes_list.append(include_dir) + + flavor = gyp.common.GetFlavor(params) + if flavor == "win": + generator_flags = params.get("generator_flags", {}) + for target_name in target_list: + target = target_dicts[target_name] + if config_name in target["configurations"]: + config = target["configurations"][config_name] + + # Look for any include dirs that were explicitly added via cflags. This + # may be done in gyp files to force certain includes to come at the end. + # TODO(jgreenwald): Change the gyp files to not abuse cflags for this, and + # remove this. + if flavor == "win": + msvs_settings = gyp.msvs_emulation.MsvsSettings(target, generator_flags) + cflags = msvs_settings.GetCflags(config_name) + else: + cflags = config["cflags"] + for cflag in cflags: + if cflag.startswith("-I"): + include_dir = cflag[2:] + if include_dir not in compiler_includes_list: + compiler_includes_list.append(include_dir) + + # Find standard gyp include dirs. + if "include_dirs" in config: + include_dirs = config["include_dirs"] + for shared_intermediate_dir in shared_intermediate_dirs: + for include_dir in include_dirs: + include_dir = include_dir.replace( + "$SHARED_INTERMEDIATE_DIR", shared_intermediate_dir + ) + if not os.path.isabs(include_dir): + base_dir = os.path.dirname(target_name) + + include_dir = base_dir + "/" + include_dir + include_dir = os.path.abspath(include_dir) + + gyp_includes_set.add(include_dir) + + # Generate a list that has all the include dirs. + all_includes_list = list(gyp_includes_set) + all_includes_list.sort() + for compiler_include in compiler_includes_list: + if compiler_include not in gyp_includes_set: + all_includes_list.append(compiler_include) + + # All done. + return all_includes_list def GetCompilerPath(target_list, data, options): - """Determine a command that can be used to invoke the compiler. + """Determine a command that can be used to invoke the compiler. Returns: If this is a gyp project that has explicit make settings, try to determine the compiler from that. Otherwise, see if a compiler was specified via the CC_target environment variable. """ - # First, see if the compiler is configured in make's settings. - build_file, _, _ = gyp.common.ParseQualifiedTarget(target_list[0]) - make_global_settings_dict = data[build_file].get('make_global_settings', {}) - for key, value in make_global_settings_dict: - if key in ['CC', 'CXX']: - return os.path.join(options.toplevel_dir, value) + # First, see if the compiler is configured in make's settings. + build_file, _, _ = gyp.common.ParseQualifiedTarget(target_list[0]) + make_global_settings_dict = data[build_file].get("make_global_settings", {}) + for key, value in make_global_settings_dict: + if key in ["CC", "CXX"]: + return os.path.join(options.toplevel_dir, value) - # Check to see if the compiler was specified as an environment variable. - for key in ['CC_target', 'CC', 'CXX']: - compiler = os.environ.get(key) - if compiler: - return compiler + # Check to see if the compiler was specified as an environment variable. + for key in ["CC_target", "CC", "CXX"]: + compiler = os.environ.get(key) + if compiler: + return compiler - return 'gcc' + return "gcc" -def GetAllDefines(target_list, target_dicts, data, config_name, params, - compiler_path): - """Calculate the defines for a project. +def GetAllDefines(target_list, target_dicts, data, config_name, params, compiler_path): + """Calculate the defines for a project. Returns: A dict that includes explicit defines declared in gyp files along with all of the default defines that the compiler uses. """ - # Get defines declared in the gyp files. - all_defines = {} - flavor = gyp.common.GetFlavor(params) - if flavor == 'win': - generator_flags = params.get('generator_flags', {}) - for target_name in target_list: - target = target_dicts[target_name] - - if flavor == 'win': - msvs_settings = gyp.msvs_emulation.MsvsSettings(target, generator_flags) - extra_defines = msvs_settings.GetComputedDefines(config_name) - else: - extra_defines = [] - if config_name in target['configurations']: - config = target['configurations'][config_name] - target_defines = config['defines'] - else: - target_defines = [] - for define in target_defines + extra_defines: - split_define = define.split('=', 1) - if len(split_define) == 1: - split_define.append('1') - if split_define[0].strip() in all_defines: - # Already defined - continue - all_defines[split_define[0].strip()] = split_define[1].strip() - # Get default compiler defines (if possible). - if flavor == 'win': - return all_defines # Default defines already processed in the loop above. - if compiler_path: - command = shlex.split(compiler_path) - command.extend(['-E', '-dM', '-']) - cpp_proc = subprocess.Popen(args=command, cwd='.', - stdin=subprocess.PIPE, stdout=subprocess.PIPE) - cpp_output = cpp_proc.communicate()[0] - if PY3: - cpp_output = cpp_output.decode('utf-8') - cpp_lines = cpp_output.split('\n') - for cpp_line in cpp_lines: - if not cpp_line.strip(): - continue - cpp_line_parts = cpp_line.split(' ', 2) - key = cpp_line_parts[1] - if len(cpp_line_parts) >= 3: - val = cpp_line_parts[2] - else: - val = '1' - all_defines[key] = val - - return all_defines + # Get defines declared in the gyp files. + all_defines = {} + flavor = gyp.common.GetFlavor(params) + if flavor == "win": + generator_flags = params.get("generator_flags", {}) + for target_name in target_list: + target = target_dicts[target_name] + + if flavor == "win": + msvs_settings = gyp.msvs_emulation.MsvsSettings(target, generator_flags) + extra_defines = msvs_settings.GetComputedDefines(config_name) + else: + extra_defines = [] + if config_name in target["configurations"]: + config = target["configurations"][config_name] + target_defines = config["defines"] + else: + target_defines = [] + for define in target_defines + extra_defines: + split_define = define.split("=", 1) + if len(split_define) == 1: + split_define.append("1") + if split_define[0].strip() in all_defines: + # Already defined + continue + all_defines[split_define[0].strip()] = split_define[1].strip() + # Get default compiler defines (if possible). + if flavor == "win": + return all_defines # Default defines already processed in the loop above. + if compiler_path: + command = shlex.split(compiler_path) + command.extend(["-E", "-dM", "-"]) + cpp_proc = subprocess.Popen( + args=command, cwd=".", stdin=subprocess.PIPE, stdout=subprocess.PIPE + ) + cpp_output = cpp_proc.communicate()[0] + if PY3: + cpp_output = cpp_output.decode("utf-8") + cpp_lines = cpp_output.split("\n") + for cpp_line in cpp_lines: + if not cpp_line.strip(): + continue + cpp_line_parts = cpp_line.split(" ", 2) + key = cpp_line_parts[1] + if len(cpp_line_parts) >= 3: + val = cpp_line_parts[2] + else: + val = "1" + all_defines[key] = val + + return all_defines def WriteIncludePaths(out, eclipse_langs, include_dirs): - """Write the includes section of a CDT settings export file.""" - - out.write('
\n') - out.write(' \n') - for lang in eclipse_langs: - out.write(' \n' % lang) - for include_dir in include_dirs: - out.write(' %s\n' % - include_dir) - out.write(' \n') - out.write('
\n') + """Write the includes section of a CDT settings export file.""" + + out.write( + '
\n' + ) + out.write(' \n') + for lang in eclipse_langs: + out.write(' \n' % lang) + for include_dir in include_dirs: + out.write( + ' %s\n' + % include_dir + ) + out.write(" \n") + out.write("
\n") def WriteMacros(out, eclipse_langs, defines): - """Write the macros section of a CDT settings export file.""" - - out.write('
\n') - out.write(' \n') - for lang in eclipse_langs: - out.write(' \n' % lang) - for key in sorted(defines): - out.write(' %s%s\n' % - (escape(key), escape(defines[key]))) - out.write(' \n') - out.write('
\n') - - -def GenerateOutputForConfig(target_list, target_dicts, data, params, - config_name): - options = params['options'] - generator_flags = params.get('generator_flags', {}) - - # build_dir: relative path from source root to our output files. - # e.g. "out/Debug" - build_dir = os.path.join(generator_flags.get('output_dir', 'out'), - config_name) - - toplevel_build = os.path.join(options.toplevel_dir, build_dir) - # Ninja uses out/Debug/gen while make uses out/Debug/obj/gen as the - # SHARED_INTERMEDIATE_DIR. Include both possible locations. - shared_intermediate_dirs = [os.path.join(toplevel_build, 'obj', 'gen'), - os.path.join(toplevel_build, 'gen')] - - GenerateCdtSettingsFile(target_list, - target_dicts, - data, - params, - config_name, - os.path.join(toplevel_build, - 'eclipse-cdt-settings.xml'), - options, - shared_intermediate_dirs) - GenerateClasspathFile(target_list, - target_dicts, - options.toplevel_dir, - toplevel_build, - os.path.join(toplevel_build, - 'eclipse-classpath.xml')) - - -def GenerateCdtSettingsFile(target_list, target_dicts, data, params, - config_name, out_name, options, - shared_intermediate_dirs): - gyp.common.EnsureDirExists(out_name) - with open(out_name, 'w') as out: - out.write('\n') - out.write('\n') - - eclipse_langs = ['C++ Source File', 'C Source File', 'Assembly Source File', - 'GNU C++', 'GNU C', 'Assembly'] - compiler_path = GetCompilerPath(target_list, data, options) - include_dirs = GetAllIncludeDirectories(target_list, target_dicts, - shared_intermediate_dirs, - config_name, params, compiler_path) - WriteIncludePaths(out, eclipse_langs, include_dirs) - defines = GetAllDefines(target_list, target_dicts, data, config_name, - params, compiler_path) - WriteMacros(out, eclipse_langs, defines) - - out.write('\n') - - -def GenerateClasspathFile(target_list, target_dicts, toplevel_dir, - toplevel_build, out_name): - '''Generates a classpath file suitable for symbol navigation and code + """Write the macros section of a CDT settings export file.""" + + out.write( + '
\n' + ) + out.write(' \n') + for lang in eclipse_langs: + out.write(' \n' % lang) + for key in sorted(defines): + out.write( + " %s%s\n" + % (escape(key), escape(defines[key])) + ) + out.write(" \n") + out.write("
\n") + + +def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name): + options = params["options"] + generator_flags = params.get("generator_flags", {}) + + # build_dir: relative path from source root to our output files. + # e.g. "out/Debug" + build_dir = os.path.join(generator_flags.get("output_dir", "out"), config_name) + + toplevel_build = os.path.join(options.toplevel_dir, build_dir) + # Ninja uses out/Debug/gen while make uses out/Debug/obj/gen as the + # SHARED_INTERMEDIATE_DIR. Include both possible locations. + shared_intermediate_dirs = [ + os.path.join(toplevel_build, "obj", "gen"), + os.path.join(toplevel_build, "gen"), + ] + + GenerateCdtSettingsFile( + target_list, + target_dicts, + data, + params, + config_name, + os.path.join(toplevel_build, "eclipse-cdt-settings.xml"), + options, + shared_intermediate_dirs, + ) + GenerateClasspathFile( + target_list, + target_dicts, + options.toplevel_dir, + toplevel_build, + os.path.join(toplevel_build, "eclipse-classpath.xml"), + ) + + +def GenerateCdtSettingsFile( + target_list, + target_dicts, + data, + params, + config_name, + out_name, + options, + shared_intermediate_dirs, +): + gyp.common.EnsureDirExists(out_name) + with open(out_name, "w") as out: + out.write('\n') + out.write("\n") + + eclipse_langs = [ + "C++ Source File", + "C Source File", + "Assembly Source File", + "GNU C++", + "GNU C", + "Assembly", + ] + compiler_path = GetCompilerPath(target_list, data, options) + include_dirs = GetAllIncludeDirectories( + target_list, + target_dicts, + shared_intermediate_dirs, + config_name, + params, + compiler_path, + ) + WriteIncludePaths(out, eclipse_langs, include_dirs) + defines = GetAllDefines( + target_list, target_dicts, data, config_name, params, compiler_path + ) + WriteMacros(out, eclipse_langs, defines) + + out.write("\n") + + +def GenerateClasspathFile( + target_list, target_dicts, toplevel_dir, toplevel_build, out_name +): + """Generates a classpath file suitable for symbol navigation and code completion of Java code (such as in Android projects) by finding all - .java and .jar files used as action inputs.''' - gyp.common.EnsureDirExists(out_name) - result = ET.Element('classpath') - - def AddElements(kind, paths): - # First, we need to normalize the paths so they are all relative to the - # toplevel dir. - rel_paths = set() - for path in paths: - if os.path.isabs(path): - rel_paths.add(os.path.relpath(path, toplevel_dir)) - else: - rel_paths.add(path) - - for path in sorted(rel_paths): - entry_element = ET.SubElement(result, 'classpathentry') - entry_element.set('kind', kind) - entry_element.set('path', path) - - AddElements('lib', GetJavaJars(target_list, target_dicts, toplevel_dir)) - AddElements('src', GetJavaSourceDirs(target_list, target_dicts, toplevel_dir)) - # Include the standard JRE container and a dummy out folder - AddElements('con', ['org.eclipse.jdt.launching.JRE_CONTAINER']) - # Include a dummy out folder so that Eclipse doesn't use the default /bin - # folder in the root of the project. - AddElements('output', [os.path.join(toplevel_build, '.eclipse-java-build')]) - - ET.ElementTree(result).write(out_name) + .java and .jar files used as action inputs.""" + gyp.common.EnsureDirExists(out_name) + result = ET.Element("classpath") + + def AddElements(kind, paths): + # First, we need to normalize the paths so they are all relative to the + # toplevel dir. + rel_paths = set() + for path in paths: + if os.path.isabs(path): + rel_paths.add(os.path.relpath(path, toplevel_dir)) + else: + rel_paths.add(path) + + for path in sorted(rel_paths): + entry_element = ET.SubElement(result, "classpathentry") + entry_element.set("kind", kind) + entry_element.set("path", path) + + AddElements("lib", GetJavaJars(target_list, target_dicts, toplevel_dir)) + AddElements("src", GetJavaSourceDirs(target_list, target_dicts, toplevel_dir)) + # Include the standard JRE container and a dummy out folder + AddElements("con", ["org.eclipse.jdt.launching.JRE_CONTAINER"]) + # Include a dummy out folder so that Eclipse doesn't use the default /bin + # folder in the root of the project. + AddElements("output", [os.path.join(toplevel_build, ".eclipse-java-build")]) + + ET.ElementTree(result).write(out_name) def GetJavaJars(target_list, target_dicts, toplevel_dir): - '''Generates a sequence of all .jars used as inputs.''' - for target_name in target_list: - target = target_dicts[target_name] - for action in target.get('actions', []): - for input_ in action['inputs']: - if os.path.splitext(input_)[1] == '.jar' and not input_.startswith('$'): - if os.path.isabs(input_): - yield input_ - else: - yield os.path.join(os.path.dirname(target_name), input_) + """Generates a sequence of all .jars used as inputs.""" + for target_name in target_list: + target = target_dicts[target_name] + for action in target.get("actions", []): + for input_ in action["inputs"]: + if os.path.splitext(input_)[1] == ".jar" and not input_.startswith("$"): + if os.path.isabs(input_): + yield input_ + else: + yield os.path.join(os.path.dirname(target_name), input_) def GetJavaSourceDirs(target_list, target_dicts, toplevel_dir): - '''Generates a sequence of all likely java package root directories.''' - for target_name in target_list: - target = target_dicts[target_name] - for action in target.get('actions', []): - for input_ in action['inputs']: - if (os.path.splitext(input_)[1] == '.java' and - not input_.startswith('$')): - dir_ = os.path.dirname(os.path.join(os.path.dirname(target_name), - input_)) - # If there is a parent 'src' or 'java' folder, navigate up to it - - # these are canonical package root names in Chromium. This will - # break if 'src' or 'java' exists in the package structure. This - # could be further improved by inspecting the java file for the - # package name if this proves to be too fragile in practice. - parent_search = dir_ - while os.path.basename(parent_search) not in ['src', 'java']: - parent_search, _ = os.path.split(parent_search) - if not parent_search or parent_search == toplevel_dir: - # Didn't find a known root, just return the original path - yield dir_ - break - else: - yield parent_search + """Generates a sequence of all likely java package root directories.""" + for target_name in target_list: + target = target_dicts[target_name] + for action in target.get("actions", []): + for input_ in action["inputs"]: + if os.path.splitext(input_)[1] == ".java" and not input_.startswith( + "$" + ): + dir_ = os.path.dirname( + os.path.join(os.path.dirname(target_name), input_) + ) + # If there is a parent 'src' or 'java' folder, navigate up to it - + # these are canonical package root names in Chromium. This will + # break if 'src' or 'java' exists in the package structure. This + # could be further improved by inspecting the java file for the + # package name if this proves to be too fragile in practice. + parent_search = dir_ + while os.path.basename(parent_search) not in ["src", "java"]: + parent_search, _ = os.path.split(parent_search) + if not parent_search or parent_search == toplevel_dir: + # Didn't find a known root, just return the original path + yield dir_ + break + else: + yield parent_search def GenerateOutput(target_list, target_dicts, data, params): - """Generate an XML settings file that can be imported into a CDT project.""" + """Generate an XML settings file that can be imported into a CDT project.""" - if params['options'].generator_output: - raise NotImplementedError("--generator_output not implemented for eclipse") - - user_config = params.get('generator_flags', {}).get('config', None) - if user_config: - GenerateOutputForConfig(target_list, target_dicts, data, params, - user_config) - else: - config_names = target_dicts[target_list[0]]['configurations'].keys() - for config_name in config_names: - GenerateOutputForConfig(target_list, target_dicts, data, params, - config_name) + if params["options"].generator_output: + raise NotImplementedError("--generator_output not implemented for eclipse") + user_config = params.get("generator_flags", {}).get("config", None) + if user_config: + GenerateOutputForConfig(target_list, target_dicts, data, params, user_config) + else: + config_names = target_dicts[target_list[0]]["configurations"] + for config_name in config_names: + GenerateOutputForConfig( + target_list, target_dicts, data, params, config_name + ) diff --git a/gyp/pylib/gyp/generator/gypd.py b/gyp/pylib/gyp/generator/gypd.py index 78eeaa61b2..4171704c47 100644 --- a/gyp/pylib/gyp/generator/gypd.py +++ b/gyp/pylib/gyp/generator/gypd.py @@ -32,36 +32,33 @@ import gyp.common -import errno -import os import pprint # These variables should just be spit back out as variable references. _generator_identity_variables = [ - 'CONFIGURATION_NAME', - 'EXECUTABLE_PREFIX', - 'EXECUTABLE_SUFFIX', - 'INTERMEDIATE_DIR', - 'LIB_DIR', - 'PRODUCT_DIR', - 'RULE_INPUT_ROOT', - 'RULE_INPUT_DIRNAME', - 'RULE_INPUT_EXT', - 'RULE_INPUT_NAME', - 'RULE_INPUT_PATH', - 'SHARED_INTERMEDIATE_DIR', - 'SHARED_LIB_DIR', - 'SHARED_LIB_PREFIX', - 'SHARED_LIB_SUFFIX', - 'STATIC_LIB_PREFIX', - 'STATIC_LIB_SUFFIX', + "CONFIGURATION_NAME", + "EXECUTABLE_PREFIX", + "EXECUTABLE_SUFFIX", + "INTERMEDIATE_DIR", + "LIB_DIR", + "PRODUCT_DIR", + "RULE_INPUT_ROOT", + "RULE_INPUT_DIRNAME", + "RULE_INPUT_EXT", + "RULE_INPUT_NAME", + "RULE_INPUT_PATH", + "SHARED_INTERMEDIATE_DIR", + "SHARED_LIB_DIR", + "SHARED_LIB_PREFIX", + "SHARED_LIB_SUFFIX", + "STATIC_LIB_PREFIX", + "STATIC_LIB_SUFFIX", ] # gypd doesn't define a default value for OS like many other generator # modules. Specify "-D OS=whatever" on the command line to provide a value. -generator_default_variables = { -} +generator_default_variables = {} # gypd supports multiple toolsets generator_supports_multiple_toolsets = True @@ -71,24 +68,22 @@ # module should use < for the early phase and then switch to > for the late # phase. Bonus points for carrying @ back into the output too. for v in _generator_identity_variables: - generator_default_variables[v] = '<(%s)' % v + generator_default_variables[v] = "<(%s)" % v def GenerateOutput(target_list, target_dicts, data, params): - output_files = {} - for qualified_target in target_list: - [input_file, target] = \ - gyp.common.ParseQualifiedTarget(qualified_target)[0:2] - - if input_file[-4:] != '.gyp': - continue - input_file_stem = input_file[:-4] - output_file = input_file_stem + params['options'].suffix + '.gypd' - - if not output_file in output_files: - output_files[output_file] = input_file - - for output_file, input_file in output_files.items(): - output = open(output_file, 'w') - pprint.pprint(data[input_file], output) - output.close() + output_files = {} + for qualified_target in target_list: + [input_file, target] = gyp.common.ParseQualifiedTarget(qualified_target)[0:2] + + if input_file[-4:] != ".gyp": + continue + input_file_stem = input_file[:-4] + output_file = input_file_stem + params["options"].suffix + ".gypd" + + output_files[output_file] = output_files.get(output_file, input_file) + + for output_file, input_file in output_files.items(): + output = open(output_file, "w") + pprint.pprint(data[input_file], output) + output.close() diff --git a/gyp/pylib/gyp/generator/gypsh.py b/gyp/pylib/gyp/generator/gypsh.py index bd405f43a9..2d8aba5d1c 100644 --- a/gyp/pylib/gyp/generator/gypsh.py +++ b/gyp/pylib/gyp/generator/gypsh.py @@ -21,36 +21,38 @@ # All of this stuff about generator variables was lovingly ripped from gypd.py. # That module has a much better description of what's going on and why. _generator_identity_variables = [ - 'EXECUTABLE_PREFIX', - 'EXECUTABLE_SUFFIX', - 'INTERMEDIATE_DIR', - 'PRODUCT_DIR', - 'RULE_INPUT_ROOT', - 'RULE_INPUT_DIRNAME', - 'RULE_INPUT_EXT', - 'RULE_INPUT_NAME', - 'RULE_INPUT_PATH', - 'SHARED_INTERMEDIATE_DIR', + "EXECUTABLE_PREFIX", + "EXECUTABLE_SUFFIX", + "INTERMEDIATE_DIR", + "PRODUCT_DIR", + "RULE_INPUT_ROOT", + "RULE_INPUT_DIRNAME", + "RULE_INPUT_EXT", + "RULE_INPUT_NAME", + "RULE_INPUT_PATH", + "SHARED_INTERMEDIATE_DIR", ] -generator_default_variables = { -} +generator_default_variables = {} for v in _generator_identity_variables: - generator_default_variables[v] = '<(%s)' % v + generator_default_variables[v] = "<(%s)" % v def GenerateOutput(target_list, target_dicts, data, params): - locals = { - 'target_list': target_list, - 'target_dicts': target_dicts, - 'data': data, - } - - # Use a banner that looks like the stock Python one and like what - # code.interact uses by default, but tack on something to indicate what - # locals are available, and identify gypsh. - banner='Python %s on %s\nlocals.keys() = %s\ngypsh' % \ - (sys.version, sys.platform, repr(sorted(locals.keys()))) - - code.interact(banner, local=locals) + locals = { + "target_list": target_list, + "target_dicts": target_dicts, + "data": data, + } + + # Use a banner that looks like the stock Python one and like what + # code.interact uses by default, but tack on something to indicate what + # locals are available, and identify gypsh. + banner = "Python %s on %s\nlocals.keys() = %s\ngypsh" % ( + sys.version, + sys.platform, + repr(sorted(locals.keys())), + ) + + code.interact(banner, local=locals) diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index 26cf88cccf..4a50b04339 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -25,7 +25,6 @@ import os import re -import sys import subprocess import gyp import gyp.common @@ -36,20 +35,20 @@ import hashlib generator_default_variables = { - 'EXECUTABLE_PREFIX': '', - 'EXECUTABLE_SUFFIX': '', - 'STATIC_LIB_PREFIX': 'lib', - 'SHARED_LIB_PREFIX': 'lib', - 'STATIC_LIB_SUFFIX': '.a', - 'INTERMEDIATE_DIR': '$(obj).$(TOOLSET)/$(TARGET)/geni', - 'SHARED_INTERMEDIATE_DIR': '$(obj)/gen', - 'PRODUCT_DIR': '$(builddir)', - 'RULE_INPUT_ROOT': '%(INPUT_ROOT)s', # This gets expanded by Python. - 'RULE_INPUT_DIRNAME': '%(INPUT_DIRNAME)s', # This gets expanded by Python. - 'RULE_INPUT_PATH': '$(abspath $<)', - 'RULE_INPUT_EXT': '$(suffix $<)', - 'RULE_INPUT_NAME': '$(notdir $<)', - 'CONFIGURATION_NAME': '$(BUILDTYPE)', + "EXECUTABLE_PREFIX": "", + "EXECUTABLE_SUFFIX": "", + "STATIC_LIB_PREFIX": "lib", + "SHARED_LIB_PREFIX": "lib", + "STATIC_LIB_SUFFIX": ".a", + "INTERMEDIATE_DIR": "$(obj).$(TOOLSET)/$(TARGET)/geni", + "SHARED_INTERMEDIATE_DIR": "$(obj)/gen", + "PRODUCT_DIR": "$(builddir)", + "RULE_INPUT_ROOT": "%(INPUT_ROOT)s", # This gets expanded by Python. + "RULE_INPUT_DIRNAME": "%(INPUT_DIRNAME)s", # This gets expanded by Python. + "RULE_INPUT_PATH": "$(abspath $<)", + "RULE_INPUT_EXT": "$(suffix $<)", + "RULE_INPUT_NAME": "$(notdir $<)", + "CONFIGURATION_NAME": "$(BUILDTYPE)", } # Make supports multiple toolsets @@ -66,63 +65,69 @@ def CalculateVariables(default_variables, params): - """Calculate additional variables for use in the build (called by gyp).""" - flavor = gyp.common.GetFlavor(params) - if flavor == 'mac': - default_variables.setdefault('OS', 'mac') - default_variables.setdefault('SHARED_LIB_SUFFIX', '.dylib') - default_variables.setdefault('SHARED_LIB_DIR', - generator_default_variables['PRODUCT_DIR']) - default_variables.setdefault('LIB_DIR', - generator_default_variables['PRODUCT_DIR']) - - # Copy additional generator configuration data from Xcode, which is shared - # by the Mac Make generator. - import gyp.generator.xcode as xcode_generator - global generator_additional_non_configuration_keys - generator_additional_non_configuration_keys = getattr(xcode_generator, - 'generator_additional_non_configuration_keys', []) - global generator_additional_path_sections - generator_additional_path_sections = getattr(xcode_generator, - 'generator_additional_path_sections', []) - global generator_extra_sources_for_rules - generator_extra_sources_for_rules = getattr(xcode_generator, - 'generator_extra_sources_for_rules', []) - COMPILABLE_EXTENSIONS.update({'.m': 'objc', '.mm' : 'objcxx'}) - else: - operating_system = flavor - if flavor == 'android': - operating_system = 'linux' # Keep this legacy behavior for now. - default_variables.setdefault('OS', operating_system) - if flavor == 'aix': - default_variables.setdefault('SHARED_LIB_SUFFIX', '.a') + """Calculate additional variables for use in the build (called by gyp).""" + flavor = gyp.common.GetFlavor(params) + if flavor == "mac": + default_variables.setdefault("OS", "mac") + default_variables.setdefault("SHARED_LIB_SUFFIX", ".dylib") + default_variables.setdefault( + "SHARED_LIB_DIR", generator_default_variables["PRODUCT_DIR"] + ) + default_variables.setdefault( + "LIB_DIR", generator_default_variables["PRODUCT_DIR"] + ) + + # Copy additional generator configuration data from Xcode, which is shared + # by the Mac Make generator. + import gyp.generator.xcode as xcode_generator + + global generator_additional_non_configuration_keys + generator_additional_non_configuration_keys = getattr( + xcode_generator, "generator_additional_non_configuration_keys", [] + ) + global generator_additional_path_sections + generator_additional_path_sections = getattr( + xcode_generator, "generator_additional_path_sections", [] + ) + global generator_extra_sources_for_rules + generator_extra_sources_for_rules = getattr( + xcode_generator, "generator_extra_sources_for_rules", [] + ) + COMPILABLE_EXTENSIONS.update({".m": "objc", ".mm": "objcxx"}) else: - default_variables.setdefault('SHARED_LIB_SUFFIX', '.so') - default_variables.setdefault('SHARED_LIB_DIR','$(builddir)/lib.$(TOOLSET)') - default_variables.setdefault('LIB_DIR', '$(obj).$(TOOLSET)') + operating_system = flavor + if flavor == "android": + operating_system = "linux" # Keep this legacy behavior for now. + default_variables.setdefault("OS", operating_system) + if flavor == "aix": + default_variables.setdefault("SHARED_LIB_SUFFIX", ".a") + else: + default_variables.setdefault("SHARED_LIB_SUFFIX", ".so") + default_variables.setdefault("SHARED_LIB_DIR", "$(builddir)/lib.$(TOOLSET)") + default_variables.setdefault("LIB_DIR", "$(obj).$(TOOLSET)") def CalculateGeneratorInputInfo(params): - """Calculate the generator specific info that gets fed to input (called by + """Calculate the generator specific info that gets fed to input (called by gyp).""" - generator_flags = params.get('generator_flags', {}) - android_ndk_version = generator_flags.get('android_ndk_version', None) - # Android NDK requires a strict link order. - if android_ndk_version: - global generator_wants_sorted_dependencies - generator_wants_sorted_dependencies = True - - output_dir = params['options'].generator_output or \ - params['options'].toplevel_dir - builddir_name = generator_flags.get('output_dir', 'out') - qualified_out_dir = os.path.normpath(os.path.join( - output_dir, builddir_name, 'gypfiles')) - - global generator_filelist_paths - generator_filelist_paths = { - 'toplevel': params['options'].toplevel_dir, - 'qualified_out_dir': qualified_out_dir, - } + generator_flags = params.get("generator_flags", {}) + android_ndk_version = generator_flags.get("android_ndk_version", None) + # Android NDK requires a strict link order. + if android_ndk_version: + global generator_wants_sorted_dependencies + generator_wants_sorted_dependencies = True + + output_dir = params["options"].generator_output or params["options"].toplevel_dir + builddir_name = generator_flags.get("output_dir", "out") + qualified_out_dir = os.path.normpath( + os.path.join(output_dir, builddir_name, "gypfiles") + ) + + global generator_filelist_paths + generator_filelist_paths = { + "toplevel": params["options"].toplevel_dir, + "qualified_out_dir": qualified_out_dir, + } # The .d checking code below uses these functions: @@ -135,7 +140,7 @@ def CalculateGeneratorInputInfo(params): # is for example # out/Release/.deps/out/Release/Chromium?Framework.framework/foo # This is the replacement character. -SPACE_REPLACEMENT = '?' +SPACE_REPLACEMENT = "?" LINK_COMMANDS_LINUX = """\ @@ -201,7 +206,7 @@ def CalculateGeneratorInputInfo(params): quiet_cmd_link = LINK($(TOOLSET)) $@ quiet_cmd_link_host = LINK($(TOOLSET)) $@ cmd_link = $(LINK.$(TOOLSET)) $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ -Wl,--start-group $(LD_INPUTS) -Wl,--end-group $(LIBS) -cmd_link_host = $(LINK.$(TOOLSET)) $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ $(LD_INPUTS) $(LIBS) +cmd_link_host = $(LINK.$(TOOLSET)) $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ -Wl,--start-group $(LD_INPUTS) -Wl,--end-group $(LIBS) # Other shared-object link notes: # - Set SONAME to the library filename so our binaries don't reference @@ -255,7 +260,8 @@ def CalculateGeneratorInputInfo(params): # Header of toplevel Makefile. # This should go into the build tree, but it's easier to keep it here for now. -SHARED_HEADER = ("""\ +SHARED_HEADER = ( + """\ # We borrow heavily from the kernel build setup, though we are simpler since # we don't have Kconfig tweaking settings on us. @@ -327,8 +333,12 @@ def CalculateGeneratorInputInfo(params): space := $(empty) $(empty) # http://stackoverflow.com/questions/1189781/using-make-dir-or-notdir-on-a-path-with-spaces -replace_spaces = $(subst $(space),""" + SPACE_REPLACEMENT + """,$1) -unreplace_spaces = $(subst """ + SPACE_REPLACEMENT + """,$(space),$1) +replace_spaces = $(subst $(space),""" + + SPACE_REPLACEMENT + + """,$1) +unreplace_spaces = $(subst """ + + SPACE_REPLACEMENT + + """,$(space),$1) dirx = $(call unreplace_spaces,$(dir $(call replace_spaces,$1))) # Flags to make gcc output dependency info. Note that you need to be @@ -358,7 +368,7 @@ def CalculateGeneratorInputInfo(params): # and dollar signs past make, the shell, and sed at the same time. # Doesn't work with spaces, but that's fine: .d files have spaces in # their names replaced with other characters.""" -r""" + r""" define fixup_dep # The depfile may not exist if the input file didn't have any #includes. touch $(depfile).raw @@ -375,7 +385,7 @@ def CalculateGeneratorInputInfo(params): rm $(depfile).raw endef """ -""" + """ # Command definitions: # - cmd_foo is the actual command to run; # - quiet_cmd_foo is the brief-output summary of the command. @@ -395,8 +405,7 @@ def CalculateGeneratorInputInfo(params): %(link_commands)s """ - -r""" + r""" # Define an escape_quotes function to escape single quotes. # This allows us to handle quotes properly as long as we always use # use single quotes and escape_quotes. @@ -412,7 +421,7 @@ def CalculateGeneratorInputInfo(params): # (e.g., dash, bash). exact_echo = printf '%%s\n' '$(call escape_quotes,$(1))' """ -""" + """ # Helper to compare the command we're about to run against the command # we logged the last time we ran the command. Produces an empty # string (false) when the commands match. @@ -423,8 +432,9 @@ def CalculateGeneratorInputInfo(params): # $(filter-out $(cmd_$@), $(cmd_$(1)))) # We instead substitute each for the empty string into the other, and # say they're equal if both substitutions produce the empty string. -# .d files contain """ + SPACE_REPLACEMENT + \ - """ instead of spaces, take that into account. +# .d files contain """ + + SPACE_REPLACEMENT + + """ instead of spaces, take that into account. command_changed = $(or $(subst $(cmd_$(1)),,$(cmd_$(call replace_spaces,$@))),\\ $(subst $(cmd_$(call replace_spaces,$@)),,$(cmd_$(1)))) @@ -455,10 +465,12 @@ def CalculateGeneratorInputInfo(params): # Should always run for a given target to handle command-line changes. # Second argument, if non-zero, makes it do asm/C/C++ dependency munging. # Third argument, if non-zero, makes it do POSTBUILDS processing. -# Note: We intentionally do NOT call dirx for depfile, since it contains """ + \ - SPACE_REPLACEMENT + """ for -# spaces already and dirx strips the """ + SPACE_REPLACEMENT + \ - """ characters. +# Note: We intentionally do NOT call dirx for depfile, since it contains """ + + SPACE_REPLACEMENT + + """ for +# spaces already and dirx strips the """ + + SPACE_REPLACEMENT + + """ characters. define do_cmd $(if $(or $(command_changed),$(prereq_changed)), @$(call exact_echo, $($(quiet)cmd_$(1))) @@ -491,7 +503,8 @@ def CalculateGeneratorInputInfo(params): .PHONY: FORCE_DO_CMD FORCE_DO_CMD: -""") +""" +) SHARED_HEADER_MAC_COMMANDS = """ quiet_cmd_objc = CXX($(TOOLSET)) $@ @@ -525,33 +538,34 @@ def CalculateGeneratorInputInfo(params): def WriteRootHeaderSuffixRules(writer): - extensions = sorted(COMPILABLE_EXTENSIONS.keys(), key=str.lower) - - writer.write('# Suffix rules, putting all outputs into $(obj).\n') - for ext in extensions: - writer.write('$(obj).$(TOOLSET)/%%.o: $(srcdir)/%%%s FORCE_DO_CMD\n' % ext) - writer.write('\t@$(call do_cmd,%s,1)\n' % COMPILABLE_EXTENSIONS[ext]) - - writer.write('\n# Try building from generated source, too.\n') - for ext in extensions: - writer.write( - '$(obj).$(TOOLSET)/%%.o: $(obj).$(TOOLSET)/%%%s FORCE_DO_CMD\n' % ext) - writer.write('\t@$(call do_cmd,%s,1)\n' % COMPILABLE_EXTENSIONS[ext]) - writer.write('\n') - for ext in extensions: - writer.write('$(obj).$(TOOLSET)/%%.o: $(obj)/%%%s FORCE_DO_CMD\n' % ext) - writer.write('\t@$(call do_cmd,%s,1)\n' % COMPILABLE_EXTENSIONS[ext]) - writer.write('\n') - - -SHARED_HEADER_SUFFIX_RULES_COMMENT1 = ("""\ + extensions = sorted(COMPILABLE_EXTENSIONS.keys(), key=str.lower) + + writer.write("# Suffix rules, putting all outputs into $(obj).\n") + for ext in extensions: + writer.write("$(obj).$(TOOLSET)/%%.o: $(srcdir)/%%%s FORCE_DO_CMD\n" % ext) + writer.write("\t@$(call do_cmd,%s,1)\n" % COMPILABLE_EXTENSIONS[ext]) + + writer.write("\n# Try building from generated source, too.\n") + for ext in extensions: + writer.write( + "$(obj).$(TOOLSET)/%%.o: $(obj).$(TOOLSET)/%%%s FORCE_DO_CMD\n" % ext + ) + writer.write("\t@$(call do_cmd,%s,1)\n" % COMPILABLE_EXTENSIONS[ext]) + writer.write("\n") + for ext in extensions: + writer.write("$(obj).$(TOOLSET)/%%.o: $(obj)/%%%s FORCE_DO_CMD\n" % ext) + writer.write("\t@$(call do_cmd,%s,1)\n" % COMPILABLE_EXTENSIONS[ext]) + writer.write("\n") + + +SHARED_HEADER_SUFFIX_RULES_COMMENT1 = """\ # Suffix rules, putting all outputs into $(obj). -""") +""" -SHARED_HEADER_SUFFIX_RULES_COMMENT2 = ("""\ +SHARED_HEADER_SUFFIX_RULES_COMMENT2 = """\ # Try building from generated source, too. -""") +""" SHARED_FOOTER = """\ @@ -574,114 +588,125 @@ def WriteRootHeaderSuffixRules(writer): # Maps every compilable file extension to the do_cmd that compiles it. COMPILABLE_EXTENSIONS = { - '.c': 'cc', - '.cc': 'cxx', - '.cpp': 'cxx', - '.cxx': 'cxx', - '.s': 'cc', - '.S': 'cc', + ".c": "cc", + ".cc": "cxx", + ".cpp": "cxx", + ".cxx": "cxx", + ".s": "cc", + ".S": "cc", } + def Compilable(filename): - """Return true if the file is compilable (should be in OBJS).""" - for res in (filename.endswith(e) for e in COMPILABLE_EXTENSIONS): - if res: - return True - return False + """Return true if the file is compilable (should be in OBJS).""" + for res in (filename.endswith(e) for e in COMPILABLE_EXTENSIONS): + if res: + return True + return False def Linkable(filename): - """Return true if the file is linkable (should be on the link line).""" - return filename.endswith('.o') + """Return true if the file is linkable (should be on the link line).""" + return filename.endswith(".o") def Target(filename): - """Translate a compilable filename to its .o target.""" - return os.path.splitext(filename)[0] + '.o' + """Translate a compilable filename to its .o target.""" + return os.path.splitext(filename)[0] + ".o" def EscapeShellArgument(s): - """Quotes an argument so that it will be interpreted literally by a POSIX + """Quotes an argument so that it will be interpreted literally by a POSIX shell. Taken from http://stackoverflow.com/questions/35817/whats-the-best-way-to-escape-ossystem-calls-in-python """ - return "'" + s.replace("'", "'\\''") + "'" + return "'" + s.replace("'", "'\\''") + "'" def EscapeMakeVariableExpansion(s): - """Make has its own variable expansion syntax using $. We must escape it for + """Make has its own variable expansion syntax using $. We must escape it for string to be interpreted literally.""" - return s.replace('$', '$$') + return s.replace("$", "$$") def EscapeCppDefine(s): - """Escapes a CPP define so that it will reach the compiler unaltered.""" - s = EscapeShellArgument(s) - s = EscapeMakeVariableExpansion(s) - # '#' characters must be escaped even embedded in a string, else Make will - # treat it as the start of a comment. - return s.replace('#', r'\#') + """Escapes a CPP define so that it will reach the compiler unaltered.""" + s = EscapeShellArgument(s) + s = EscapeMakeVariableExpansion(s) + # '#' characters must be escaped even embedded in a string, else Make will + # treat it as the start of a comment. + return s.replace("#", r"\#") def QuoteIfNecessary(string): - """TODO: Should this ideally be replaced with one or more of the above + """TODO: Should this ideally be replaced with one or more of the above functions?""" - if '"' in string: - string = '"' + string.replace('"', '\\"') + '"' - return string + if '"' in string: + string = '"' + string.replace('"', '\\"') + '"' + return string def StringToMakefileVariable(string): - """Convert a string to a value that is acceptable as a make variable name.""" - return re.sub('[^a-zA-Z0-9_]', '_', string) + """Convert a string to a value that is acceptable as a make variable name.""" + return re.sub("[^a-zA-Z0-9_]", "_", string) + + +srcdir_prefix = "" -srcdir_prefix = '' def Sourceify(path): - """Convert a path to its source directory form.""" - if '$(' in path: - return path - if os.path.isabs(path): - return path - return srcdir_prefix + path + """Convert a path to its source directory form.""" + if "$(" in path: + return path + if os.path.isabs(path): + return path + return srcdir_prefix + path -def QuoteSpaces(s, quote=r'\ '): - return s.replace(' ', quote) +def QuoteSpaces(s, quote=r"\ "): + return s.replace(" ", quote) + def SourceifyAndQuoteSpaces(path): - """Convert a path to its source directory form and quote spaces.""" - return QuoteSpaces(Sourceify(path)) + """Convert a path to its source directory form and quote spaces.""" + return QuoteSpaces(Sourceify(path)) + # TODO: Avoid code duplication with _ValidateSourcesForMSVSProject in msvs.py. def _ValidateSourcesForOSX(spec, all_sources): - """Makes sure if duplicate basenames are not specified in the source list. + """Makes sure if duplicate basenames are not specified in the source list. Arguments: spec: The target dictionary containing the properties of the target. """ - if spec.get('type', None) != 'static_library': - return - - basenames = {} - for source in all_sources: - name, ext = os.path.splitext(source) - is_compiled_file = ext in [ - '.c', '.cc', '.cpp', '.cxx', '.m', '.mm', '.s', '.S'] - if not is_compiled_file: - continue - basename = os.path.basename(name) # Don't include extension. - basenames.setdefault(basename, []).append(source) - - error = '' - for basename, files in basenames.items(): - if len(files) > 1: - error += ' %s: %s\n' % (basename, ' '.join(files)) - - if error: - print(('static library %s has several files with the same basename:\n' % spec['target_name']) - + error + 'libtool on OS X will generate' + ' warnings for them.') - raise GypError('Duplicate basenames in sources section, see list above') + if spec.get("type", None) != "static_library": + return + + basenames = {} + for source in all_sources: + name, ext = os.path.splitext(source) + is_compiled_file = ext in [".c", ".cc", ".cpp", ".cxx", ".m", ".mm", ".s", ".S"] + if not is_compiled_file: + continue + basename = os.path.basename(name) # Don't include extension. + basenames.setdefault(basename, []).append(source) + + error = "" + for basename, files in basenames.items(): + if len(files) > 1: + error += " %s: %s\n" % (basename, " ".join(files)) + + if error: + print( + ( + "static library %s has several files with the same basename:\n" + % spec["target_name"] + ) + + error + + "libtool on OS X will generate" + + " warnings for them." + ) + raise GypError("Duplicate basenames in sources section, see list above") # Map from qualified target to path to output. @@ -694,41 +719,62 @@ def _ValidateSourcesForOSX(spec, all_sources): class MakefileWriter(object): - """MakefileWriter packages up the writing of one target-specific foobar.mk. + """MakefileWriter packages up the writing of one target-specific foobar.mk. Its only real entry point is Write(), and is mostly used for namespacing. """ - def __init__(self, generator_flags, flavor): - self.generator_flags = generator_flags - self.flavor = flavor - - self.suffix_rules_srcdir = {} - self.suffix_rules_objdir1 = {} - self.suffix_rules_objdir2 = {} - - # Generate suffix rules for all compilable extensions. - for ext in COMPILABLE_EXTENSIONS.keys(): - # Suffix rules for source folder. - self.suffix_rules_srcdir.update({ext: ("""\ + def __init__(self, generator_flags, flavor): + self.generator_flags = generator_flags + self.flavor = flavor + + self.suffix_rules_srcdir = {} + self.suffix_rules_objdir1 = {} + self.suffix_rules_objdir2 = {} + + # Generate suffix rules for all compilable extensions. + for ext in COMPILABLE_EXTENSIONS.keys(): + # Suffix rules for source folder. + self.suffix_rules_srcdir.update( + { + ext: ( + """\ $(obj).$(TOOLSET)/$(TARGET)/%%.o: $(srcdir)/%%%s FORCE_DO_CMD - @$(call do_cmd,%s,1) -""" % (ext, COMPILABLE_EXTENSIONS[ext]))}) - - # Suffix rules for generated source files. - self.suffix_rules_objdir1.update({ext: ("""\ +\t@$(call do_cmd,%s,1) +""" + % (ext, COMPILABLE_EXTENSIONS[ext]) + ) + } + ) + + # Suffix rules for generated source files. + self.suffix_rules_objdir1.update( + { + ext: ( + """\ $(obj).$(TOOLSET)/$(TARGET)/%%.o: $(obj).$(TOOLSET)/%%%s FORCE_DO_CMD - @$(call do_cmd,%s,1) -""" % (ext, COMPILABLE_EXTENSIONS[ext]))}) - self.suffix_rules_objdir2.update({ext: ("""\ +\t@$(call do_cmd,%s,1) +""" + % (ext, COMPILABLE_EXTENSIONS[ext]) + ) + } + ) + self.suffix_rules_objdir2.update( + { + ext: ( + """\ $(obj).$(TOOLSET)/$(TARGET)/%%.o: $(obj)/%%%s FORCE_DO_CMD - @$(call do_cmd,%s,1) -""" % (ext, COMPILABLE_EXTENSIONS[ext]))}) - +\t@$(call do_cmd,%s,1) +""" + % (ext, COMPILABLE_EXTENSIONS[ext]) + ) + } + ) - def Write(self, qualified_target, base_path, output_filename, spec, configs, - part_of_all): - """The main entry point: writes a .mk file for a single target. + def Write( + self, qualified_target, base_path, output_filename, spec, configs, part_of_all + ): + """The main entry point: writes a .mk file for a single target. Arguments: qualified_target: target we're generating @@ -738,129 +784,152 @@ def Write(self, qualified_target, base_path, output_filename, spec, configs, spec, configs: gyp info part_of_all: flag indicating this target is part of 'all' """ - gyp.common.EnsureDirExists(output_filename) - - self.fp = open(output_filename, 'w') + gyp.common.EnsureDirExists(output_filename) - self.fp.write(header) + self.fp = open(output_filename, "w") - self.qualified_target = qualified_target - self.path = base_path - self.target = spec['target_name'] - self.type = spec['type'] - self.toolset = spec['toolset'] + self.fp.write(header) - self.is_mac_bundle = gyp.xcode_emulation.IsMacBundle(self.flavor, spec) - if self.flavor == 'mac': - self.xcode_settings = gyp.xcode_emulation.XcodeSettings(spec) - else: - self.xcode_settings = None - - deps, link_deps = self.ComputeDeps(spec) - - # Some of the generation below can add extra output, sources, or - # link dependencies. All of the out params of the functions that - # follow use names like extra_foo. - extra_outputs = [] - extra_sources = [] - extra_link_deps = [] - extra_mac_bundle_resources = [] - mac_bundle_deps = [] - - if self.is_mac_bundle: - self.output = self.ComputeMacBundleOutput(spec) - self.output_binary = self.ComputeMacBundleBinaryOutput(spec) - else: - self.output = self.output_binary = self.ComputeOutput(spec) - - self.is_standalone_static_library = bool( - spec.get('standalone_static_library', 0)) - self._INSTALLABLE_TARGETS = ('executable', 'loadable_module', - 'shared_library') - if (self.is_standalone_static_library or - self.type in self._INSTALLABLE_TARGETS): - self.alias = os.path.basename(self.output) - install_path = self._InstallableTargetInstallPath() - else: - self.alias = self.output - install_path = self.output - - self.WriteLn("TOOLSET := " + self.toolset) - self.WriteLn("TARGET := " + self.target) - - # Actions must come first, since they can generate more OBJs for use below. - if 'actions' in spec: - self.WriteActions(spec['actions'], extra_sources, extra_outputs, - extra_mac_bundle_resources, part_of_all) - - # Rules must be early like actions. - if 'rules' in spec: - self.WriteRules(spec['rules'], extra_sources, extra_outputs, - extra_mac_bundle_resources, part_of_all) - - if 'copies' in spec: - self.WriteCopies(spec['copies'], extra_outputs, part_of_all) - - # Bundle resources. - if self.is_mac_bundle: - all_mac_bundle_resources = ( - spec.get('mac_bundle_resources', []) + extra_mac_bundle_resources) - self.WriteMacBundleResources(all_mac_bundle_resources, mac_bundle_deps) - self.WriteMacInfoPlist(mac_bundle_deps) - - # Sources. - all_sources = spec.get('sources', []) + extra_sources - if all_sources: - if self.flavor == 'mac': - # libtool on OS X generates warnings for duplicate basenames in the same - # target. - _ValidateSourcesForOSX(spec, all_sources) - self.WriteSources( - configs, deps, all_sources, extra_outputs, - extra_link_deps, part_of_all, - gyp.xcode_emulation.MacPrefixHeader( - self.xcode_settings, lambda p: Sourceify(self.Absolutify(p)), - self.Pchify)) - sources = list(filter(Compilable, all_sources)) - if sources: - self.WriteLn(SHARED_HEADER_SUFFIX_RULES_COMMENT1) - extensions = set([os.path.splitext(s)[1] for s in sources]) - for ext in extensions: - if ext in self.suffix_rules_srcdir: - self.WriteLn(self.suffix_rules_srcdir[ext]) - self.WriteLn(SHARED_HEADER_SUFFIX_RULES_COMMENT2) - for ext in extensions: - if ext in self.suffix_rules_objdir1: - self.WriteLn(self.suffix_rules_objdir1[ext]) - for ext in extensions: - if ext in self.suffix_rules_objdir2: - self.WriteLn(self.suffix_rules_objdir2[ext]) - self.WriteLn('# End of this set of suffix rules') - - # Add dependency from bundle to bundle binary. - if self.is_mac_bundle: - mac_bundle_deps.append(self.output_binary) + self.qualified_target = qualified_target + self.path = base_path + self.target = spec["target_name"] + self.type = spec["type"] + self.toolset = spec["toolset"] - self.WriteTarget(spec, configs, deps, extra_link_deps + link_deps, - mac_bundle_deps, extra_outputs, part_of_all) - - # Update global list of target outputs, used in dependency tracking. - target_outputs[qualified_target] = install_path - - # Update global list of link dependencies. - if self.type in ('static_library', 'shared_library'): - target_link_deps[qualified_target] = self.output_binary - - # Currently any versions have the same effect, but in future the behavior - # could be different. - if self.generator_flags.get('android_ndk_version', None): - self.WriteAndroidNdkModuleRule(self.target, all_sources, link_deps) + self.is_mac_bundle = gyp.xcode_emulation.IsMacBundle(self.flavor, spec) + if self.flavor == "mac": + self.xcode_settings = gyp.xcode_emulation.XcodeSettings(spec) + else: + self.xcode_settings = None - self.fp.close() + deps, link_deps = self.ComputeDeps(spec) + # Some of the generation below can add extra output, sources, or + # link dependencies. All of the out params of the functions that + # follow use names like extra_foo. + extra_outputs = [] + extra_sources = [] + extra_link_deps = [] + extra_mac_bundle_resources = [] + mac_bundle_deps = [] - def WriteSubMake(self, output_filename, makefile_path, targets, build_dir): - """Write a "sub-project" Makefile. + if self.is_mac_bundle: + self.output = self.ComputeMacBundleOutput(spec) + self.output_binary = self.ComputeMacBundleBinaryOutput(spec) + else: + self.output = self.output_binary = self.ComputeOutput(spec) + + self.is_standalone_static_library = bool( + spec.get("standalone_static_library", 0) + ) + self._INSTALLABLE_TARGETS = ("executable", "loadable_module", "shared_library") + if self.is_standalone_static_library or self.type in self._INSTALLABLE_TARGETS: + self.alias = os.path.basename(self.output) + install_path = self._InstallableTargetInstallPath() + else: + self.alias = self.output + install_path = self.output + + self.WriteLn("TOOLSET := " + self.toolset) + self.WriteLn("TARGET := " + self.target) + + # Actions must come first, since they can generate more OBJs for use below. + if "actions" in spec: + self.WriteActions( + spec["actions"], + extra_sources, + extra_outputs, + extra_mac_bundle_resources, + part_of_all, + ) + + # Rules must be early like actions. + if "rules" in spec: + self.WriteRules( + spec["rules"], + extra_sources, + extra_outputs, + extra_mac_bundle_resources, + part_of_all, + ) + + if "copies" in spec: + self.WriteCopies(spec["copies"], extra_outputs, part_of_all) + + # Bundle resources. + if self.is_mac_bundle: + all_mac_bundle_resources = ( + spec.get("mac_bundle_resources", []) + extra_mac_bundle_resources + ) + self.WriteMacBundleResources(all_mac_bundle_resources, mac_bundle_deps) + self.WriteMacInfoPlist(mac_bundle_deps) + + # Sources. + all_sources = spec.get("sources", []) + extra_sources + if all_sources: + if self.flavor == "mac": + # libtool on OS X generates warnings for duplicate basenames in the same + # target. + _ValidateSourcesForOSX(spec, all_sources) + self.WriteSources( + configs, + deps, + all_sources, + extra_outputs, + extra_link_deps, + part_of_all, + gyp.xcode_emulation.MacPrefixHeader( + self.xcode_settings, + lambda p: Sourceify(self.Absolutify(p)), + self.Pchify, + ), + ) + sources = [x for x in all_sources if Compilable(x)] + if sources: + self.WriteLn(SHARED_HEADER_SUFFIX_RULES_COMMENT1) + extensions = set([os.path.splitext(s)[1] for s in sources]) + for ext in extensions: + if ext in self.suffix_rules_srcdir: + self.WriteLn(self.suffix_rules_srcdir[ext]) + self.WriteLn(SHARED_HEADER_SUFFIX_RULES_COMMENT2) + for ext in extensions: + if ext in self.suffix_rules_objdir1: + self.WriteLn(self.suffix_rules_objdir1[ext]) + for ext in extensions: + if ext in self.suffix_rules_objdir2: + self.WriteLn(self.suffix_rules_objdir2[ext]) + self.WriteLn("# End of this set of suffix rules") + + # Add dependency from bundle to bundle binary. + if self.is_mac_bundle: + mac_bundle_deps.append(self.output_binary) + + self.WriteTarget( + spec, + configs, + deps, + extra_link_deps + link_deps, + mac_bundle_deps, + extra_outputs, + part_of_all, + ) + + # Update global list of target outputs, used in dependency tracking. + target_outputs[qualified_target] = install_path + + # Update global list of link dependencies. + if self.type in ("static_library", "shared_library"): + target_link_deps[qualified_target] = self.output_binary + + # Currently any versions have the same effect, but in future the behavior + # could be different. + if self.generator_flags.get("android_ndk_version", None): + self.WriteAndroidNdkModuleRule(self.target, all_sources, link_deps) + + self.fp.close() + + def WriteSubMake(self, output_filename, makefile_path, targets, build_dir): + """Write a "sub-project" Makefile. This is a small, wrapper Makefile that calls the top-level Makefile to build the targets from a single gyp file (i.e. a sub-project). @@ -871,24 +940,31 @@ def WriteSubMake(self, output_filename, makefile_path, targets, build_dir): targets: list of "all" targets for this sub-project build_dir: build output directory, relative to the sub-project """ - gyp.common.EnsureDirExists(output_filename) - self.fp = open(output_filename, 'w') - self.fp.write(header) - # For consistency with other builders, put sub-project build output in the - # sub-project dir (see test/subdirectory/gyptest-subdir-all.py). - self.WriteLn('export builddir_name ?= %s' % - os.path.join(os.path.dirname(output_filename), build_dir)) - self.WriteLn('.PHONY: all') - self.WriteLn('all:') - if makefile_path: - makefile_path = ' -C ' + makefile_path - self.WriteLn('\t$(MAKE)%s %s' % (makefile_path, ' '.join(targets))) - self.fp.close() - - - def WriteActions(self, actions, extra_sources, extra_outputs, - extra_mac_bundle_resources, part_of_all): - """Write Makefile code for any 'actions' from the gyp input. + gyp.common.EnsureDirExists(output_filename) + self.fp = open(output_filename, "w") + self.fp.write(header) + # For consistency with other builders, put sub-project build output in the + # sub-project dir (see test/subdirectory/gyptest-subdir-all.py). + self.WriteLn( + "export builddir_name ?= %s" + % os.path.join(os.path.dirname(output_filename), build_dir) + ) + self.WriteLn(".PHONY: all") + self.WriteLn("all:") + if makefile_path: + makefile_path = " -C " + makefile_path + self.WriteLn("\t$(MAKE)%s %s" % (makefile_path, " ".join(targets))) + self.fp.close() + + def WriteActions( + self, + actions, + extra_sources, + extra_outputs, + extra_mac_bundle_resources, + part_of_all, + ): + """Write Makefile code for any 'actions' from the gyp input. extra_sources: a list that will be filled in with newly generated source files, if any @@ -897,98 +973,113 @@ def WriteActions(self, actions, extra_sources, extra_outputs, actions) part_of_all: flag indicating this target is part of 'all' """ - env = self.GetSortedXcodeEnv() - for action in actions: - name = StringToMakefileVariable('%s_%s' % (self.qualified_target, - action['action_name'])) - self.WriteLn('### Rules for action "%s":' % action['action_name']) - inputs = action['inputs'] - outputs = action['outputs'] - - # Build up a list of outputs. - # Collect the output dirs we'll need. - dirs = set() - for out in outputs: - dir = os.path.split(out)[0] - if dir: - dirs.add(dir) - if int(action.get('process_outputs_as_sources', False)): - extra_sources += outputs - if int(action.get('process_outputs_as_mac_bundle_resources', False)): - extra_mac_bundle_resources += outputs - - # Write the actual command. - action_commands = action['action'] - if self.flavor == 'mac': - action_commands = [gyp.xcode_emulation.ExpandEnvVars(command, env) - for command in action_commands] - command = gyp.common.EncodePOSIXShellList(action_commands) - if 'message' in action: - self.WriteLn('quiet_cmd_%s = ACTION %s $@' % (name, action['message'])) - else: - self.WriteLn('quiet_cmd_%s = ACTION %s $@' % (name, name)) - if len(dirs) > 0: - command = 'mkdir -p %s' % ' '.join(dirs) + '; ' + command - - cd_action = 'cd %s; ' % Sourceify(self.path or '.') - - # command and cd_action get written to a toplevel variable called - # cmd_foo. Toplevel variables can't handle things that change per - # makefile like $(TARGET), so hardcode the target. - command = command.replace('$(TARGET)', self.target) - cd_action = cd_action.replace('$(TARGET)', self.target) - - # Set LD_LIBRARY_PATH in case the action runs an executable from this - # build which links to shared libs from this build. - # actions run on the host, so they should in theory only use host - # libraries, but until everything is made cross-compile safe, also use - # target libraries. - # TODO(piman): when everything is cross-compile safe, remove lib.target - self.WriteLn('cmd_%s = LD_LIBRARY_PATH=$(builddir)/lib.host:' - '$(builddir)/lib.target:$$LD_LIBRARY_PATH; ' - 'export LD_LIBRARY_PATH; ' - '%s%s' - % (name, cd_action, command)) - self.WriteLn() - outputs = [self.Absolutify(o) for o in outputs] - # The makefile rules are all relative to the top dir, but the gyp actions - # are defined relative to their containing dir. This replaces the obj - # variable for the action rule with an absolute version so that the output - # goes in the right place. - # Only write the 'obj' and 'builddir' rules for the "primary" output (:1); - # it's superfluous for the "extra outputs", and this avoids accidentally - # writing duplicate dummy rules for those outputs. - # Same for environment. - self.WriteLn("%s: obj := $(abs_obj)" % QuoteSpaces(outputs[0])) - self.WriteLn("%s: builddir := $(abs_builddir)" % QuoteSpaces(outputs[0])) - self.WriteSortedXcodeEnv(outputs[0], self.GetSortedXcodeEnv()) - - for input in inputs: - assert ' ' not in input, ( - "Spaces in action input filenames not supported (%s)" % input) - for output in outputs: - assert ' ' not in output, ( - "Spaces in action output filenames not supported (%s)" % output) - - # See the comment in WriteCopies about expanding env vars. - outputs = [gyp.xcode_emulation.ExpandEnvVars(o, env) for o in outputs] - inputs = [gyp.xcode_emulation.ExpandEnvVars(i, env) for i in inputs] - - self.WriteDoCmd(outputs, [Sourceify(self.Absolutify(i)) for i in inputs], - part_of_all=part_of_all, command=name) - - # Stuff the outputs in a variable so we can refer to them later. - outputs_variable = 'action_%s_outputs' % name - self.WriteLn('%s := %s' % (outputs_variable, ' '.join(outputs))) - extra_outputs.append('$(%s)' % outputs_variable) - self.WriteLn() - - self.WriteLn() - - - def WriteRules(self, rules, extra_sources, extra_outputs, - extra_mac_bundle_resources, part_of_all): - """Write Makefile code for any 'rules' from the gyp input. + env = self.GetSortedXcodeEnv() + for action in actions: + name = StringToMakefileVariable( + "%s_%s" % (self.qualified_target, action["action_name"]) + ) + self.WriteLn('### Rules for action "%s":' % action["action_name"]) + inputs = action["inputs"] + outputs = action["outputs"] + + # Build up a list of outputs. + # Collect the output dirs we'll need. + dirs = set() + for out in outputs: + dir = os.path.split(out)[0] + if dir: + dirs.add(dir) + if int(action.get("process_outputs_as_sources", False)): + extra_sources += outputs + if int(action.get("process_outputs_as_mac_bundle_resources", False)): + extra_mac_bundle_resources += outputs + + # Write the actual command. + action_commands = action["action"] + if self.flavor == "mac": + action_commands = [ + gyp.xcode_emulation.ExpandEnvVars(command, env) + for command in action_commands + ] + command = gyp.common.EncodePOSIXShellList(action_commands) + if "message" in action: + self.WriteLn("quiet_cmd_%s = ACTION %s $@" % (name, action["message"])) + else: + self.WriteLn("quiet_cmd_%s = ACTION %s $@" % (name, name)) + if len(dirs) > 0: + command = "mkdir -p %s" % " ".join(dirs) + "; " + command + + cd_action = "cd %s; " % Sourceify(self.path or ".") + + # command and cd_action get written to a toplevel variable called + # cmd_foo. Toplevel variables can't handle things that change per + # makefile like $(TARGET), so hardcode the target. + command = command.replace("$(TARGET)", self.target) + cd_action = cd_action.replace("$(TARGET)", self.target) + + # Set LD_LIBRARY_PATH in case the action runs an executable from this + # build which links to shared libs from this build. + # actions run on the host, so they should in theory only use host + # libraries, but until everything is made cross-compile safe, also use + # target libraries. + # TODO(piman): when everything is cross-compile safe, remove lib.target + self.WriteLn( + "cmd_%s = LD_LIBRARY_PATH=$(builddir)/lib.host:" + "$(builddir)/lib.target:$$LD_LIBRARY_PATH; " + "export LD_LIBRARY_PATH; " + "%s%s" % (name, cd_action, command) + ) + self.WriteLn() + outputs = [self.Absolutify(o) for o in outputs] + # The makefile rules are all relative to the top dir, but the gyp actions + # are defined relative to their containing dir. This replaces the obj + # variable for the action rule with an absolute version so that the output + # goes in the right place. + # Only write the 'obj' and 'builddir' rules for the "primary" output (:1); + # it's superfluous for the "extra outputs", and this avoids accidentally + # writing duplicate dummy rules for those outputs. + # Same for environment. + self.WriteLn("%s: obj := $(abs_obj)" % QuoteSpaces(outputs[0])) + self.WriteLn("%s: builddir := $(abs_builddir)" % QuoteSpaces(outputs[0])) + self.WriteSortedXcodeEnv(outputs[0], self.GetSortedXcodeEnv()) + + for input in inputs: + assert " " not in input, ( + "Spaces in action input filenames not supported (%s)" % input + ) + for output in outputs: + assert " " not in output, ( + "Spaces in action output filenames not supported (%s)" % output + ) + + # See the comment in WriteCopies about expanding env vars. + outputs = [gyp.xcode_emulation.ExpandEnvVars(o, env) for o in outputs] + inputs = [gyp.xcode_emulation.ExpandEnvVars(i, env) for i in inputs] + + self.WriteDoCmd( + outputs, + [Sourceify(self.Absolutify(i)) for i in inputs], + part_of_all=part_of_all, + command=name, + ) + + # Stuff the outputs in a variable so we can refer to them later. + outputs_variable = "action_%s_outputs" % name + self.WriteLn("%s := %s" % (outputs_variable, " ".join(outputs))) + extra_outputs.append("$(%s)" % outputs_variable) + self.WriteLn() + + self.WriteLn() + + def WriteRules( + self, + rules, + extra_sources, + extra_outputs, + extra_mac_bundle_resources, + part_of_all, + ): + """Write Makefile code for any 'rules' from the gyp input. extra_sources: a list that will be filled in with newly generated source files, if any @@ -996,208 +1087,240 @@ def WriteRules(self, rules, extra_sources, extra_outputs, rules (used to make other pieces dependent on these rules) part_of_all: flag indicating this target is part of 'all' """ - env = self.GetSortedXcodeEnv() - for rule in rules: - name = StringToMakefileVariable('%s_%s' % (self.qualified_target, - rule['rule_name'])) - count = 0 - self.WriteLn('### Generated for rule %s:' % name) - - all_outputs = [] - - for rule_source in rule.get('rule_sources', []): - dirs = set() - (rule_source_dirname, rule_source_basename) = os.path.split(rule_source) - (rule_source_root, rule_source_ext) = \ - os.path.splitext(rule_source_basename) - - outputs = [self.ExpandInputRoot(out, rule_source_root, - rule_source_dirname) - for out in rule['outputs']] - - for out in outputs: - dir = os.path.dirname(out) - if dir: - dirs.add(dir) - if int(rule.get('process_outputs_as_sources', False)): - extra_sources += outputs - if int(rule.get('process_outputs_as_mac_bundle_resources', False)): - extra_mac_bundle_resources += outputs - inputs = [Sourceify(self.Absolutify(i)) for i - in [rule_source] + rule.get('inputs', [])] - actions = ['$(call do_cmd,%s_%d)' % (name, count)] - - if name == 'resources_grit': - # HACK: This is ugly. Grit intentionally doesn't touch the - # timestamp of its output file when the file doesn't change, - # which is fine in hash-based dependency systems like scons - # and forge, but not kosher in the make world. After some - # discussion, hacking around it here seems like the least - # amount of pain. - actions += ['@touch --no-create $@'] - - # See the comment in WriteCopies about expanding env vars. - outputs = [gyp.xcode_emulation.ExpandEnvVars(o, env) for o in outputs] - inputs = [gyp.xcode_emulation.ExpandEnvVars(i, env) for i in inputs] - - outputs = [self.Absolutify(o) for o in outputs] - all_outputs += outputs - # Only write the 'obj' and 'builddir' rules for the "primary" output - # (:1); it's superfluous for the "extra outputs", and this avoids - # accidentally writing duplicate dummy rules for those outputs. - self.WriteLn('%s: obj := $(abs_obj)' % outputs[0]) - self.WriteLn('%s: builddir := $(abs_builddir)' % outputs[0]) - self.WriteMakeRule(outputs, inputs, actions, - command="%s_%d" % (name, count)) - # Spaces in rule filenames are not supported, but rule variables have - # spaces in them (e.g. RULE_INPUT_PATH expands to '$(abspath $<)'). - # The spaces within the variables are valid, so remove the variables - # before checking. - variables_with_spaces = re.compile(r'\$\([^ ]* \$<\)') - for output in outputs: - output = re.sub(variables_with_spaces, '', output) - assert ' ' not in output, ( - "Spaces in rule filenames not yet supported (%s)" % output) - self.WriteLn('all_deps += %s' % ' '.join(outputs)) - - action = [self.ExpandInputRoot(ac, rule_source_root, - rule_source_dirname) - for ac in rule['action']] - mkdirs = '' - if len(dirs) > 0: - mkdirs = 'mkdir -p %s; ' % ' '.join(dirs) - cd_action = 'cd %s; ' % Sourceify(self.path or '.') - - # action, cd_action, and mkdirs get written to a toplevel variable - # called cmd_foo. Toplevel variables can't handle things that change - # per makefile like $(TARGET), so hardcode the target. - if self.flavor == 'mac': - action = [gyp.xcode_emulation.ExpandEnvVars(command, env) - for command in action] - action = gyp.common.EncodePOSIXShellList(action) - action = action.replace('$(TARGET)', self.target) - cd_action = cd_action.replace('$(TARGET)', self.target) - mkdirs = mkdirs.replace('$(TARGET)', self.target) - - # Set LD_LIBRARY_PATH in case the rule runs an executable from this - # build which links to shared libs from this build. - # rules run on the host, so they should in theory only use host - # libraries, but until everything is made cross-compile safe, also use - # target libraries. - # TODO(piman): when everything is cross-compile safe, remove lib.target - self.WriteLn( - "cmd_%(name)s_%(count)d = LD_LIBRARY_PATH=" - "$(builddir)/lib.host:$(builddir)/lib.target:$$LD_LIBRARY_PATH; " - "export LD_LIBRARY_PATH; " - "%(cd_action)s%(mkdirs)s%(action)s" % { - 'action': action, - 'cd_action': cd_action, - 'count': count, - 'mkdirs': mkdirs, - 'name': name, - }) - self.WriteLn( - 'quiet_cmd_%(name)s_%(count)d = RULE %(name)s_%(count)d $@' % { - 'count': count, - 'name': name, - }) - self.WriteLn() - count += 1 - - outputs_variable = 'rule_%s_outputs' % name - self.WriteList(all_outputs, outputs_variable) - extra_outputs.append('$(%s)' % outputs_variable) - - self.WriteLn('### Finished generating for rule: %s' % name) - self.WriteLn() - self.WriteLn('### Finished generating for all rules') - self.WriteLn('') - - - def WriteCopies(self, copies, extra_outputs, part_of_all): - """Write Makefile code for any 'copies' from the gyp input. + env = self.GetSortedXcodeEnv() + for rule in rules: + name = StringToMakefileVariable( + "%s_%s" % (self.qualified_target, rule["rule_name"]) + ) + count = 0 + self.WriteLn("### Generated for rule %s:" % name) + + all_outputs = [] + + for rule_source in rule.get("rule_sources", []): + dirs = set() + (rule_source_dirname, rule_source_basename) = os.path.split(rule_source) + (rule_source_root, rule_source_ext) = os.path.splitext( + rule_source_basename + ) + + outputs = [ + self.ExpandInputRoot(out, rule_source_root, rule_source_dirname) + for out in rule["outputs"] + ] + + for out in outputs: + dir = os.path.dirname(out) + if dir: + dirs.add(dir) + if int(rule.get("process_outputs_as_sources", False)): + extra_sources += outputs + if int(rule.get("process_outputs_as_mac_bundle_resources", False)): + extra_mac_bundle_resources += outputs + inputs = [ + Sourceify(self.Absolutify(i)) + for i in [rule_source] + rule.get("inputs", []) + ] + actions = ["$(call do_cmd,%s_%d)" % (name, count)] + + if name == "resources_grit": + # HACK: This is ugly. Grit intentionally doesn't touch the + # timestamp of its output file when the file doesn't change, + # which is fine in hash-based dependency systems like scons + # and forge, but not kosher in the make world. After some + # discussion, hacking around it here seems like the least + # amount of pain. + actions += ["@touch --no-create $@"] + + # See the comment in WriteCopies about expanding env vars. + outputs = [gyp.xcode_emulation.ExpandEnvVars(o, env) for o in outputs] + inputs = [gyp.xcode_emulation.ExpandEnvVars(i, env) for i in inputs] + + outputs = [self.Absolutify(o) for o in outputs] + all_outputs += outputs + # Only write the 'obj' and 'builddir' rules for the "primary" output + # (:1); it's superfluous for the "extra outputs", and this avoids + # accidentally writing duplicate dummy rules for those outputs. + self.WriteLn("%s: obj := $(abs_obj)" % outputs[0]) + self.WriteLn("%s: builddir := $(abs_builddir)" % outputs[0]) + self.WriteMakeRule( + outputs, inputs, actions, command="%s_%d" % (name, count) + ) + # Spaces in rule filenames are not supported, but rule variables have + # spaces in them (e.g. RULE_INPUT_PATH expands to '$(abspath $<)'). + # The spaces within the variables are valid, so remove the variables + # before checking. + variables_with_spaces = re.compile(r"\$\([^ ]* \$<\)") + for output in outputs: + output = re.sub(variables_with_spaces, "", output) + assert " " not in output, ( + "Spaces in rule filenames not yet supported (%s)" % output + ) + self.WriteLn("all_deps += %s" % " ".join(outputs)) + + action = [ + self.ExpandInputRoot(ac, rule_source_root, rule_source_dirname) + for ac in rule["action"] + ] + mkdirs = "" + if len(dirs) > 0: + mkdirs = "mkdir -p %s; " % " ".join(dirs) + cd_action = "cd %s; " % Sourceify(self.path or ".") + + # action, cd_action, and mkdirs get written to a toplevel variable + # called cmd_foo. Toplevel variables can't handle things that change + # per makefile like $(TARGET), so hardcode the target. + if self.flavor == "mac": + action = [ + gyp.xcode_emulation.ExpandEnvVars(command, env) + for command in action + ] + action = gyp.common.EncodePOSIXShellList(action) + action = action.replace("$(TARGET)", self.target) + cd_action = cd_action.replace("$(TARGET)", self.target) + mkdirs = mkdirs.replace("$(TARGET)", self.target) + + # Set LD_LIBRARY_PATH in case the rule runs an executable from this + # build which links to shared libs from this build. + # rules run on the host, so they should in theory only use host + # libraries, but until everything is made cross-compile safe, also use + # target libraries. + # TODO(piman): when everything is cross-compile safe, remove lib.target + self.WriteLn( + "cmd_%(name)s_%(count)d = LD_LIBRARY_PATH=" + "$(builddir)/lib.host:$(builddir)/lib.target:$$LD_LIBRARY_PATH; " + "export LD_LIBRARY_PATH; " + "%(cd_action)s%(mkdirs)s%(action)s" + % { + "action": action, + "cd_action": cd_action, + "count": count, + "mkdirs": mkdirs, + "name": name, + } + ) + self.WriteLn( + "quiet_cmd_%(name)s_%(count)d = RULE %(name)s_%(count)d $@" + % {"count": count, "name": name} + ) + self.WriteLn() + count += 1 + + outputs_variable = "rule_%s_outputs" % name + self.WriteList(all_outputs, outputs_variable) + extra_outputs.append("$(%s)" % outputs_variable) + + self.WriteLn("### Finished generating for rule: %s" % name) + self.WriteLn() + self.WriteLn("### Finished generating for all rules") + self.WriteLn("") + + def WriteCopies(self, copies, extra_outputs, part_of_all): + """Write Makefile code for any 'copies' from the gyp input. extra_outputs: a list that will be filled in with any outputs of this action (used to make other pieces dependent on this action) part_of_all: flag indicating this target is part of 'all' """ - self.WriteLn('### Generated for copy rule.') - - variable = StringToMakefileVariable(self.qualified_target + '_copies') - outputs = [] - for copy in copies: - for path in copy['files']: - # Absolutify() may call normpath, and will strip trailing slashes. - path = Sourceify(self.Absolutify(path)) - filename = os.path.split(path)[1] - output = Sourceify(self.Absolutify(os.path.join(copy['destination'], - filename))) - - # If the output path has variables in it, which happens in practice for - # 'copies', writing the environment as target-local doesn't work, - # because the variables are already needed for the target name. - # Copying the environment variables into global make variables doesn't - # work either, because then the .d files will potentially contain spaces - # after variable expansion, and .d file handling cannot handle spaces. - # As a workaround, manually expand variables at gyp time. Since 'copies' - # can't run scripts, there's no need to write the env then. - # WriteDoCmd() will escape spaces for .d files. - env = self.GetSortedXcodeEnv() - output = gyp.xcode_emulation.ExpandEnvVars(output, env) - path = gyp.xcode_emulation.ExpandEnvVars(path, env) - self.WriteDoCmd([output], [path], 'copy', part_of_all) - outputs.append(output) - self.WriteLn('%s = %s' % (variable, ' '.join(QuoteSpaces(o) for o in outputs))) - extra_outputs.append('$(%s)' % variable) - self.WriteLn() - - - def WriteMacBundleResources(self, resources, bundle_deps): - """Writes Makefile code for 'mac_bundle_resources'.""" - self.WriteLn('### Generated for mac_bundle_resources') - - for output, res in gyp.xcode_emulation.GetMacBundleResources( - generator_default_variables['PRODUCT_DIR'], self.xcode_settings, - [Sourceify(self.Absolutify(r)) for r in resources]): - _, ext = os.path.splitext(output) - if ext != '.xcassets': - # Make does not supports '.xcassets' emulation. - self.WriteDoCmd([output], [res], 'mac_tool,,,copy-bundle-resource', - part_of_all=True) - bundle_deps.append(output) - - - def WriteMacInfoPlist(self, bundle_deps): - """Write Makefile code for bundle Info.plist files.""" - info_plist, out, defines, extra_env = gyp.xcode_emulation.GetMacInfoPlist( - generator_default_variables['PRODUCT_DIR'], self.xcode_settings, - lambda p: Sourceify(self.Absolutify(p))) - if not info_plist: - return - if defines: - # Create an intermediate file to store preprocessed results. - intermediate_plist = ('$(obj).$(TOOLSET)/$(TARGET)/' + - os.path.basename(info_plist)) - self.WriteList(defines, intermediate_plist + ': INFOPLIST_DEFINES', '-D', - quoter=EscapeCppDefine) - self.WriteMakeRule([intermediate_plist], [info_plist], - ['$(call do_cmd,infoplist)', - # "Convert" the plist so that any weird whitespace changes from the - # preprocessor do not affect the XML parser in mac_tool. - '@plutil -convert xml1 $@ $@']) - info_plist = intermediate_plist - # plists can contain envvars and substitute them into the file. - self.WriteSortedXcodeEnv( - out, self.GetSortedXcodeEnv(additional_settings=extra_env)) - self.WriteDoCmd([out], [info_plist], 'mac_tool,,,copy-info-plist', - part_of_all=True) - bundle_deps.append(out) - - - def WriteSources(self, configs, deps, sources, - extra_outputs, extra_link_deps, - part_of_all, precompiled_header): - """Write Makefile code for any 'sources' from the gyp input. + self.WriteLn("### Generated for copy rule.") + + variable = StringToMakefileVariable(self.qualified_target + "_copies") + outputs = [] + for copy in copies: + for path in copy["files"]: + # Absolutify() may call normpath, and will strip trailing slashes. + path = Sourceify(self.Absolutify(path)) + filename = os.path.split(path)[1] + output = Sourceify( + self.Absolutify(os.path.join(copy["destination"], filename)) + ) + + # If the output path has variables in it, which happens in practice for + # 'copies', writing the environment as target-local doesn't work, + # because the variables are already needed for the target name. + # Copying the environment variables into global make variables doesn't + # work either, because then the .d files will potentially contain spaces + # after variable expansion, and .d file handling cannot handle spaces. + # As a workaround, manually expand variables at gyp time. Since 'copies' + # can't run scripts, there's no need to write the env then. + # WriteDoCmd() will escape spaces for .d files. + env = self.GetSortedXcodeEnv() + output = gyp.xcode_emulation.ExpandEnvVars(output, env) + path = gyp.xcode_emulation.ExpandEnvVars(path, env) + self.WriteDoCmd([output], [path], "copy", part_of_all) + outputs.append(output) + self.WriteLn("%s = %s" % (variable, " ".join(QuoteSpaces(o) for o in outputs))) + extra_outputs.append("$(%s)" % variable) + self.WriteLn() + + def WriteMacBundleResources(self, resources, bundle_deps): + """Writes Makefile code for 'mac_bundle_resources'.""" + self.WriteLn("### Generated for mac_bundle_resources") + + for output, res in gyp.xcode_emulation.GetMacBundleResources( + generator_default_variables["PRODUCT_DIR"], + self.xcode_settings, + [Sourceify(self.Absolutify(r)) for r in resources], + ): + _, ext = os.path.splitext(output) + if ext != ".xcassets": + # Make does not supports '.xcassets' emulation. + self.WriteDoCmd( + [output], [res], "mac_tool,,,copy-bundle-resource", part_of_all=True + ) + bundle_deps.append(output) + + def WriteMacInfoPlist(self, bundle_deps): + """Write Makefile code for bundle Info.plist files.""" + info_plist, out, defines, extra_env = gyp.xcode_emulation.GetMacInfoPlist( + generator_default_variables["PRODUCT_DIR"], + self.xcode_settings, + lambda p: Sourceify(self.Absolutify(p)), + ) + if not info_plist: + return + if defines: + # Create an intermediate file to store preprocessed results. + intermediate_plist = "$(obj).$(TOOLSET)/$(TARGET)/" + os.path.basename( + info_plist + ) + self.WriteList( + defines, + intermediate_plist + ": INFOPLIST_DEFINES", + "-D", + quoter=EscapeCppDefine, + ) + self.WriteMakeRule( + [intermediate_plist], + [info_plist], + [ + "$(call do_cmd,infoplist)", + # "Convert" the plist so that any weird whitespace changes from the + # preprocessor do not affect the XML parser in mac_tool. + "@plutil -convert xml1 $@ $@", + ], + ) + info_plist = intermediate_plist + # plists can contain envvars and substitute them into the file. + self.WriteSortedXcodeEnv( + out, self.GetSortedXcodeEnv(additional_settings=extra_env) + ) + self.WriteDoCmd( + [out], [info_plist], "mac_tool,,,copy-info-plist", part_of_all=True + ) + bundle_deps.append(out) + + def WriteSources( + self, + configs, + deps, + sources, + extra_outputs, + extra_link_deps, + part_of_all, + precompiled_header, + ): + """Write Makefile code for any 'sources' from the gyp input. These are source files necessary to build the current target. configs, deps, sources: input from gyp. @@ -1208,256 +1331,285 @@ def WriteSources(self, configs, deps, sources, part_of_all: flag indicating this target is part of 'all' """ - # Write configuration-specific variables for CFLAGS, etc. - for configname in sorted(configs.keys()): - config = configs[configname] - self.WriteList(config.get('defines'), 'DEFS_%s' % configname, prefix='-D', - quoter=EscapeCppDefine) - - if self.flavor == 'mac': - cflags = self.xcode_settings.GetCflags(configname) - cflags_c = self.xcode_settings.GetCflagsC(configname) - cflags_cc = self.xcode_settings.GetCflagsCC(configname) - cflags_objc = self.xcode_settings.GetCflagsObjC(configname) - cflags_objcc = self.xcode_settings.GetCflagsObjCC(configname) - else: - cflags = config.get('cflags') - cflags_c = config.get('cflags_c') - cflags_cc = config.get('cflags_cc') - - self.WriteLn("# Flags passed to all source files.") - self.WriteList(cflags, 'CFLAGS_%s' % configname) - self.WriteLn("# Flags passed to only C files.") - self.WriteList(cflags_c, 'CFLAGS_C_%s' % configname) - self.WriteLn("# Flags passed to only C++ files.") - self.WriteList(cflags_cc, 'CFLAGS_CC_%s' % configname) - if self.flavor == 'mac': - self.WriteLn("# Flags passed to only ObjC files.") - self.WriteList(cflags_objc, 'CFLAGS_OBJC_%s' % configname) - self.WriteLn("# Flags passed to only ObjC++ files.") - self.WriteList(cflags_objcc, 'CFLAGS_OBJCC_%s' % configname) - includes = config.get('include_dirs') - if includes: - includes = [Sourceify(self.Absolutify(i)) for i in includes] - self.WriteList(includes, 'INCS_%s' % configname, prefix='-I') - - compilable = list(filter(Compilable, sources)) - objs = [self.Objectify(self.Absolutify(Target(c))) for c in compilable] - self.WriteList(objs, 'OBJS') - - for obj in objs: - assert ' ' not in obj, ( - "Spaces in object filenames not supported (%s)" % obj) - self.WriteLn('# Add to the list of files we specially track ' - 'dependencies for.') - self.WriteLn('all_deps += $(OBJS)') - self.WriteLn() - - # Make sure our dependencies are built first. - if deps: - self.WriteMakeRule(['$(OBJS)'], deps, - comment = 'Make sure our dependencies are built ' - 'before any of us.', - order_only = True) - - # Make sure the actions and rules run first. - # If they generate any extra headers etc., the per-.o file dep tracking - # will catch the proper rebuilds, so order only is still ok here. - if extra_outputs: - self.WriteMakeRule(['$(OBJS)'], extra_outputs, - comment = 'Make sure our actions/rules run ' - 'before any of us.', - order_only = True) - - pchdeps = precompiled_header.GetObjDependencies(compilable, objs ) - if pchdeps: - self.WriteLn('# Dependencies from obj files to their precompiled headers') - for source, obj, gch in pchdeps: - self.WriteLn('%s: %s' % (obj, gch)) - self.WriteLn('# End precompiled header dependencies') - - if objs: - extra_link_deps.append('$(OBJS)') - self.WriteLn("""\ + # Write configuration-specific variables for CFLAGS, etc. + for configname in sorted(configs.keys()): + config = configs[configname] + self.WriteList( + config.get("defines"), + "DEFS_%s" % configname, + prefix="-D", + quoter=EscapeCppDefine, + ) + + if self.flavor == "mac": + cflags = self.xcode_settings.GetCflags(configname) + cflags_c = self.xcode_settings.GetCflagsC(configname) + cflags_cc = self.xcode_settings.GetCflagsCC(configname) + cflags_objc = self.xcode_settings.GetCflagsObjC(configname) + cflags_objcc = self.xcode_settings.GetCflagsObjCC(configname) + else: + cflags = config.get("cflags") + cflags_c = config.get("cflags_c") + cflags_cc = config.get("cflags_cc") + + self.WriteLn("# Flags passed to all source files.") + self.WriteList(cflags, "CFLAGS_%s" % configname) + self.WriteLn("# Flags passed to only C files.") + self.WriteList(cflags_c, "CFLAGS_C_%s" % configname) + self.WriteLn("# Flags passed to only C++ files.") + self.WriteList(cflags_cc, "CFLAGS_CC_%s" % configname) + if self.flavor == "mac": + self.WriteLn("# Flags passed to only ObjC files.") + self.WriteList(cflags_objc, "CFLAGS_OBJC_%s" % configname) + self.WriteLn("# Flags passed to only ObjC++ files.") + self.WriteList(cflags_objcc, "CFLAGS_OBJCC_%s" % configname) + includes = config.get("include_dirs") + if includes: + includes = [Sourceify(self.Absolutify(i)) for i in includes] + self.WriteList(includes, "INCS_%s" % configname, prefix="-I") + + compilable = list(filter(Compilable, sources)) + objs = [self.Objectify(self.Absolutify(Target(c))) for c in compilable] + self.WriteList(objs, "OBJS") + + for obj in objs: + assert " " not in obj, "Spaces in object filenames not supported (%s)" % obj + self.WriteLn( + "# Add to the list of files we specially track " "dependencies for." + ) + self.WriteLn("all_deps += $(OBJS)") + self.WriteLn() + + # Make sure our dependencies are built first. + if deps: + self.WriteMakeRule( + ["$(OBJS)"], + deps, + comment="Make sure our dependencies are built " "before any of us.", + order_only=True, + ) + + # Make sure the actions and rules run first. + # If they generate any extra headers etc., the per-.o file dep tracking + # will catch the proper rebuilds, so order only is still ok here. + if extra_outputs: + self.WriteMakeRule( + ["$(OBJS)"], + extra_outputs, + comment="Make sure our actions/rules run " "before any of us.", + order_only=True, + ) + + pchdeps = precompiled_header.GetObjDependencies(compilable, objs) + if pchdeps: + self.WriteLn("# Dependencies from obj files to their precompiled headers") + for source, obj, gch in pchdeps: + self.WriteLn("%s: %s" % (obj, gch)) + self.WriteLn("# End precompiled header dependencies") + + if objs: + extra_link_deps.append("$(OBJS)") + self.WriteLn( + """\ # CFLAGS et al overrides must be target-local. -# See "Target-specific Variable Values" in the GNU Make manual.""") - self.WriteLn("$(OBJS): TOOLSET := $(TOOLSET)") - self.WriteLn("$(OBJS): GYP_CFLAGS := " - "$(DEFS_$(BUILDTYPE)) " - "$(INCS_$(BUILDTYPE)) " - "%s " % precompiled_header.GetInclude('c') + - "$(CFLAGS_$(BUILDTYPE)) " - "$(CFLAGS_C_$(BUILDTYPE))") - self.WriteLn("$(OBJS): GYP_CXXFLAGS := " - "$(DEFS_$(BUILDTYPE)) " - "$(INCS_$(BUILDTYPE)) " - "%s " % precompiled_header.GetInclude('cc') + - "$(CFLAGS_$(BUILDTYPE)) " - "$(CFLAGS_CC_$(BUILDTYPE))") - if self.flavor == 'mac': - self.WriteLn("$(OBJS): GYP_OBJCFLAGS := " - "$(DEFS_$(BUILDTYPE)) " - "$(INCS_$(BUILDTYPE)) " - "%s " % precompiled_header.GetInclude('m') + - "$(CFLAGS_$(BUILDTYPE)) " - "$(CFLAGS_C_$(BUILDTYPE)) " - "$(CFLAGS_OBJC_$(BUILDTYPE))") - self.WriteLn("$(OBJS): GYP_OBJCXXFLAGS := " - "$(DEFS_$(BUILDTYPE)) " - "$(INCS_$(BUILDTYPE)) " - "%s " % precompiled_header.GetInclude('mm') + - "$(CFLAGS_$(BUILDTYPE)) " - "$(CFLAGS_CC_$(BUILDTYPE)) " - "$(CFLAGS_OBJCC_$(BUILDTYPE))") - - self.WritePchTargets(precompiled_header.GetPchBuildCommands()) - - # If there are any object files in our input file list, link them into our - # output. - extra_link_deps += list(filter(Linkable, sources)) - - self.WriteLn() - - def WritePchTargets(self, pch_commands): - """Writes make rules to compile prefix headers.""" - if not pch_commands: - return - - for gch, lang_flag, lang, input in pch_commands: - extra_flags = { - 'c': '$(CFLAGS_C_$(BUILDTYPE))', - 'cc': '$(CFLAGS_CC_$(BUILDTYPE))', - 'm': '$(CFLAGS_C_$(BUILDTYPE)) $(CFLAGS_OBJC_$(BUILDTYPE))', - 'mm': '$(CFLAGS_CC_$(BUILDTYPE)) $(CFLAGS_OBJCC_$(BUILDTYPE))', - }[lang] - var_name = { - 'c': 'GYP_PCH_CFLAGS', - 'cc': 'GYP_PCH_CXXFLAGS', - 'm': 'GYP_PCH_OBJCFLAGS', - 'mm': 'GYP_PCH_OBJCXXFLAGS', - }[lang] - self.WriteLn("%s: %s := %s " % (gch, var_name, lang_flag) + - "$(DEFS_$(BUILDTYPE)) " - "$(INCS_$(BUILDTYPE)) " - "$(CFLAGS_$(BUILDTYPE)) " + - extra_flags) - - self.WriteLn('%s: %s FORCE_DO_CMD' % (gch, input)) - self.WriteLn('\t@$(call do_cmd,pch_%s,1)' % lang) - self.WriteLn('') - assert ' ' not in gch, ( - "Spaces in gch filenames not supported (%s)" % gch) - self.WriteLn('all_deps += %s' % gch) - self.WriteLn('') - - - def ComputeOutputBasename(self, spec): - """Return the 'output basename' of a gyp spec. +# See "Target-specific Variable Values" in the GNU Make manual.""" + ) + self.WriteLn("$(OBJS): TOOLSET := $(TOOLSET)") + self.WriteLn( + "$(OBJS): GYP_CFLAGS := " + "$(DEFS_$(BUILDTYPE)) " + "$(INCS_$(BUILDTYPE)) " + "%s " % precompiled_header.GetInclude("c") + "$(CFLAGS_$(BUILDTYPE)) " + "$(CFLAGS_C_$(BUILDTYPE))" + ) + self.WriteLn( + "$(OBJS): GYP_CXXFLAGS := " + "$(DEFS_$(BUILDTYPE)) " + "$(INCS_$(BUILDTYPE)) " + "%s " % precompiled_header.GetInclude("cc") + "$(CFLAGS_$(BUILDTYPE)) " + "$(CFLAGS_CC_$(BUILDTYPE))" + ) + if self.flavor == "mac": + self.WriteLn( + "$(OBJS): GYP_OBJCFLAGS := " + "$(DEFS_$(BUILDTYPE)) " + "$(INCS_$(BUILDTYPE)) " + "%s " % precompiled_header.GetInclude("m") + + "$(CFLAGS_$(BUILDTYPE)) " + "$(CFLAGS_C_$(BUILDTYPE)) " + "$(CFLAGS_OBJC_$(BUILDTYPE))" + ) + self.WriteLn( + "$(OBJS): GYP_OBJCXXFLAGS := " + "$(DEFS_$(BUILDTYPE)) " + "$(INCS_$(BUILDTYPE)) " + "%s " % precompiled_header.GetInclude("mm") + + "$(CFLAGS_$(BUILDTYPE)) " + "$(CFLAGS_CC_$(BUILDTYPE)) " + "$(CFLAGS_OBJCC_$(BUILDTYPE))" + ) + + self.WritePchTargets(precompiled_header.GetPchBuildCommands()) + + # If there are any object files in our input file list, link them into our + # output. + extra_link_deps += [source for source in sources if Linkable(source)] + + self.WriteLn() + + def WritePchTargets(self, pch_commands): + """Writes make rules to compile prefix headers.""" + if not pch_commands: + return + + for gch, lang_flag, lang, input in pch_commands: + extra_flags = { + "c": "$(CFLAGS_C_$(BUILDTYPE))", + "cc": "$(CFLAGS_CC_$(BUILDTYPE))", + "m": "$(CFLAGS_C_$(BUILDTYPE)) $(CFLAGS_OBJC_$(BUILDTYPE))", + "mm": "$(CFLAGS_CC_$(BUILDTYPE)) $(CFLAGS_OBJCC_$(BUILDTYPE))", + }[lang] + var_name = { + "c": "GYP_PCH_CFLAGS", + "cc": "GYP_PCH_CXXFLAGS", + "m": "GYP_PCH_OBJCFLAGS", + "mm": "GYP_PCH_OBJCXXFLAGS", + }[lang] + self.WriteLn( + "%s: %s := %s " % (gch, var_name, lang_flag) + "$(DEFS_$(BUILDTYPE)) " + "$(INCS_$(BUILDTYPE)) " + "$(CFLAGS_$(BUILDTYPE)) " + extra_flags + ) + + self.WriteLn("%s: %s FORCE_DO_CMD" % (gch, input)) + self.WriteLn("\t@$(call do_cmd,pch_%s,1)" % lang) + self.WriteLn("") + assert " " not in gch, "Spaces in gch filenames not supported (%s)" % gch + self.WriteLn("all_deps += %s" % gch) + self.WriteLn("") + + def ComputeOutputBasename(self, spec): + """Return the 'output basename' of a gyp spec. E.g., the loadable module 'foobar' in directory 'baz' will produce 'libfoobar.so' """ - assert not self.is_mac_bundle - - if self.flavor == 'mac' and self.type in ( - 'static_library', 'executable', 'shared_library', 'loadable_module'): - return self.xcode_settings.GetExecutablePath() - - target = spec['target_name'] - target_prefix = '' - target_ext = '' - if self.type == 'static_library': - if target[:3] == 'lib': - target = target[3:] - target_prefix = 'lib' - target_ext = '.a' - elif self.type in ('loadable_module', 'shared_library'): - if target[:3] == 'lib': - target = target[3:] - target_prefix = 'lib' - if self.flavor == 'aix': - target_ext = '.a' - else: - target_ext = '.so' - elif self.type == 'none': - target = '%s.stamp' % target - elif self.type != 'executable': - print("ERROR: What output file should be generated?", - "type", self.type, "target", target) - - target_prefix = spec.get('product_prefix', target_prefix) - target = spec.get('product_name', target) - product_ext = spec.get('product_extension') - if product_ext: - target_ext = '.' + product_ext - - return target_prefix + target + target_ext - - - def _InstallImmediately(self): - return self.toolset == 'target' and self.flavor == 'mac' and self.type in ( - 'static_library', 'executable', 'shared_library', 'loadable_module') - - - def ComputeOutput(self, spec): - """Return the 'output' (full output path) of a gyp spec. + assert not self.is_mac_bundle + + if self.flavor == "mac" and self.type in ( + "static_library", + "executable", + "shared_library", + "loadable_module", + ): + return self.xcode_settings.GetExecutablePath() + + target = spec["target_name"] + target_prefix = "" + target_ext = "" + if self.type == "static_library": + if target[:3] == "lib": + target = target[3:] + target_prefix = "lib" + target_ext = ".a" + elif self.type in ("loadable_module", "shared_library"): + if target[:3] == "lib": + target = target[3:] + target_prefix = "lib" + if self.flavor == "aix": + target_ext = ".a" + else: + target_ext = ".so" + elif self.type == "none": + target = "%s.stamp" % target + elif self.type != "executable": + print( + "ERROR: What output file should be generated?", + "type", + self.type, + "target", + target, + ) + + target_prefix = spec.get("product_prefix", target_prefix) + target = spec.get("product_name", target) + product_ext = spec.get("product_extension") + if product_ext: + target_ext = "." + product_ext + + return target_prefix + target + target_ext + + def _InstallImmediately(self): + return ( + self.toolset == "target" + and self.flavor == "mac" + and self.type + in ("static_library", "executable", "shared_library", "loadable_module") + ) + + def ComputeOutput(self, spec): + """Return the 'output' (full output path) of a gyp spec. E.g., the loadable module 'foobar' in directory 'baz' will produce '$(obj)/baz/libfoobar.so' """ - assert not self.is_mac_bundle - - path = os.path.join('$(obj).' + self.toolset, self.path) - if self.type == 'executable' or self._InstallImmediately(): - path = '$(builddir)' - path = spec.get('product_dir', path) - return os.path.join(path, self.ComputeOutputBasename(spec)) - + assert not self.is_mac_bundle - def ComputeMacBundleOutput(self, spec): - """Return the 'output' (full output path) to a bundle output directory.""" - assert self.is_mac_bundle - path = generator_default_variables['PRODUCT_DIR'] - return os.path.join(path, self.xcode_settings.GetWrapperName()) + path = os.path.join("$(obj)." + self.toolset, self.path) + if self.type == "executable" or self._InstallImmediately(): + path = "$(builddir)" + path = spec.get("product_dir", path) + return os.path.join(path, self.ComputeOutputBasename(spec)) + def ComputeMacBundleOutput(self, spec): + """Return the 'output' (full output path) to a bundle output directory.""" + assert self.is_mac_bundle + path = generator_default_variables["PRODUCT_DIR"] + return os.path.join(path, self.xcode_settings.GetWrapperName()) - def ComputeMacBundleBinaryOutput(self, spec): - """Return the 'output' (full output path) to the binary in a bundle.""" - path = generator_default_variables['PRODUCT_DIR'] - return os.path.join(path, self.xcode_settings.GetExecutablePath()) + def ComputeMacBundleBinaryOutput(self, spec): + """Return the 'output' (full output path) to the binary in a bundle.""" + path = generator_default_variables["PRODUCT_DIR"] + return os.path.join(path, self.xcode_settings.GetExecutablePath()) - - def ComputeDeps(self, spec): - """Compute the dependencies of a gyp spec. + def ComputeDeps(self, spec): + """Compute the dependencies of a gyp spec. Returns a tuple (deps, link_deps), where each is a list of filenames that will need to be put in front of make for either building (deps) or linking (link_deps). """ - deps = [] - link_deps = [] - if 'dependencies' in spec: - deps.extend([target_outputs[dep] for dep in spec['dependencies'] - if target_outputs[dep]]) - for dep in spec['dependencies']: - if dep in target_link_deps: - link_deps.append(target_link_deps[dep]) - deps.extend(link_deps) - # TODO: It seems we need to transitively link in libraries (e.g. -lfoo)? - # This hack makes it work: - # link_deps.extend(spec.get('libraries', [])) - return (gyp.common.uniquer(deps), gyp.common.uniquer(link_deps)) - - - def WriteDependencyOnExtraOutputs(self, target, extra_outputs): - self.WriteMakeRule([self.output_binary], extra_outputs, - comment = 'Build our special outputs first.', - order_only = True) - - - def WriteTarget(self, spec, configs, deps, link_deps, bundle_deps, - extra_outputs, part_of_all): - """Write Makefile code to produce the final target of the gyp spec. + deps = [] + link_deps = [] + if "dependencies" in spec: + deps.extend( + [ + target_outputs[dep] + for dep in spec["dependencies"] + if target_outputs[dep] + ] + ) + for dep in spec["dependencies"]: + if dep in target_link_deps: + link_deps.append(target_link_deps[dep]) + deps.extend(link_deps) + # TODO: It seems we need to transitively link in libraries (e.g. -lfoo)? + # This hack makes it work: + # link_deps.extend(spec.get('libraries', [])) + return (gyp.common.uniquer(deps), gyp.common.uniquer(link_deps)) + + def WriteDependencyOnExtraOutputs(self, target, extra_outputs): + self.WriteMakeRule( + [self.output_binary], + extra_outputs, + comment="Build our special outputs first.", + order_only=True, + ) + + def WriteTarget( + self, spec, configs, deps, link_deps, bundle_deps, extra_outputs, part_of_all + ): + """Write Makefile code to produce the final target of the gyp spec. spec, configs: input from gyp. deps, link_deps: dependency lists; see ComputeDeps() @@ -1465,274 +1617,376 @@ def WriteTarget(self, spec, configs, deps, link_deps, bundle_deps, part_of_all: flag indicating this target is part of 'all' """ - self.WriteLn('### Rules for final target.') - - if extra_outputs: - self.WriteDependencyOnExtraOutputs(self.output_binary, extra_outputs) - self.WriteMakeRule(extra_outputs, deps, - comment=('Preserve order dependency of ' - 'special output on deps.'), - order_only = True) - - target_postbuilds = {} - if self.type != 'none': - for configname in sorted(configs.keys()): - config = configs[configname] - if self.flavor == 'mac': - ldflags = self.xcode_settings.GetLdflags(configname, - generator_default_variables['PRODUCT_DIR'], - lambda p: Sourceify(self.Absolutify(p))) - - # TARGET_POSTBUILDS_$(BUILDTYPE) is added to postbuilds later on. - gyp_to_build = gyp.common.InvertRelativePath(self.path) - target_postbuild = self.xcode_settings.AddImplicitPostbuilds( - configname, - QuoteSpaces(os.path.normpath(os.path.join(gyp_to_build, - self.output))), - QuoteSpaces(os.path.normpath(os.path.join(gyp_to_build, - self.output_binary)))) - if target_postbuild: - target_postbuilds[configname] = target_postbuild + self.WriteLn("### Rules for final target.") + + if extra_outputs: + self.WriteDependencyOnExtraOutputs(self.output_binary, extra_outputs) + self.WriteMakeRule( + extra_outputs, + deps, + comment=("Preserve order dependency of " "special output on deps."), + order_only=True, + ) + + target_postbuilds = {} + if self.type != "none": + for configname in sorted(configs.keys()): + config = configs[configname] + if self.flavor == "mac": + ldflags = self.xcode_settings.GetLdflags( + configname, + generator_default_variables["PRODUCT_DIR"], + lambda p: Sourceify(self.Absolutify(p)), + ) + + # TARGET_POSTBUILDS_$(BUILDTYPE) is added to postbuilds later on. + gyp_to_build = gyp.common.InvertRelativePath(self.path) + target_postbuild = self.xcode_settings.AddImplicitPostbuilds( + configname, + QuoteSpaces( + os.path.normpath(os.path.join(gyp_to_build, self.output)) + ), + QuoteSpaces( + os.path.normpath( + os.path.join(gyp_to_build, self.output_binary) + ) + ), + ) + if target_postbuild: + target_postbuilds[configname] = target_postbuild + else: + ldflags = config.get("ldflags", []) + # Compute an rpath for this output if needed. + if any(dep.endswith(".so") or ".so." in dep for dep in deps): + # We want to get the literal string "$ORIGIN" into the link command, + # so we need lots of escaping. + ldflags.append(r"-Wl,-rpath=\$$ORIGIN/lib.%s/" % self.toolset) + ldflags.append( + r"-Wl,-rpath-link=\$(builddir)/lib.%s/" % self.toolset + ) + library_dirs = config.get("library_dirs", []) + ldflags += [("-L%s" % library_dir) for library_dir in library_dirs] + self.WriteList(ldflags, "LDFLAGS_%s" % configname) + if self.flavor == "mac": + self.WriteList( + self.xcode_settings.GetLibtoolflags(configname), + "LIBTOOLFLAGS_%s" % configname, + ) + libraries = spec.get("libraries") + if libraries: + # Remove duplicate entries + libraries = gyp.common.uniquer(libraries) + if self.flavor == "mac": + libraries = self.xcode_settings.AdjustLibraries(libraries) + self.WriteList(libraries, "LIBS") + self.WriteLn( + "%s: GYP_LDFLAGS := $(LDFLAGS_$(BUILDTYPE))" + % QuoteSpaces(self.output_binary) + ) + self.WriteLn("%s: LIBS := $(LIBS)" % QuoteSpaces(self.output_binary)) + + if self.flavor == "mac": + self.WriteLn( + "%s: GYP_LIBTOOLFLAGS := $(LIBTOOLFLAGS_$(BUILDTYPE))" + % QuoteSpaces(self.output_binary) + ) + + # Postbuild actions. Like actions, but implicitly depend on the target's + # output. + postbuilds = [] + if self.flavor == "mac": + if target_postbuilds: + postbuilds.append("$(TARGET_POSTBUILDS_$(BUILDTYPE))") + postbuilds.extend(gyp.xcode_emulation.GetSpecPostbuildCommands(spec)) + + if postbuilds: + # Envvars may be referenced by TARGET_POSTBUILDS_$(BUILDTYPE), + # so we must output its definition first, since we declare variables + # using ":=". + self.WriteSortedXcodeEnv(self.output, self.GetSortedXcodePostbuildEnv()) + + for configname in target_postbuilds: + self.WriteLn( + "%s: TARGET_POSTBUILDS_%s := %s" + % ( + QuoteSpaces(self.output), + configname, + gyp.common.EncodePOSIXShellList(target_postbuilds[configname]), + ) + ) + + # Postbuilds expect to be run in the gyp file's directory, so insert an + # implicit postbuild to cd to there. + postbuilds.insert(0, gyp.common.EncodePOSIXShellList(["cd", self.path])) + for i, postbuild in enumerate(postbuilds): + if not postbuild.startswith("$"): + postbuilds[i] = EscapeShellArgument(postbuild) + self.WriteLn("%s: builddir := $(abs_builddir)" % QuoteSpaces(self.output)) + self.WriteLn( + "%s: POSTBUILDS := %s" + % (QuoteSpaces(self.output), " ".join(postbuilds)) + ) + + # A bundle directory depends on its dependencies such as bundle resources + # and bundle binary. When all dependencies have been built, the bundle + # needs to be packaged. + if self.is_mac_bundle: + # If the framework doesn't contain a binary, then nothing depends + # on the actions -- make the framework depend on them directly too. + self.WriteDependencyOnExtraOutputs(self.output, extra_outputs) + + # Bundle dependencies. Note that the code below adds actions to this + # target, so if you move these two lines, move the lines below as well. + self.WriteList([QuoteSpaces(dep) for dep in bundle_deps], "BUNDLE_DEPS") + self.WriteLn("%s: $(BUNDLE_DEPS)" % QuoteSpaces(self.output)) + + # After the framework is built, package it. Needs to happen before + # postbuilds, since postbuilds depend on this. + if self.type in ("shared_library", "loadable_module"): + self.WriteLn( + "\t@$(call do_cmd,mac_package_framework,,,%s)" + % self.xcode_settings.GetFrameworkVersion() + ) + + # Bundle postbuilds can depend on the whole bundle, so run them after + # the bundle is packaged, not already after the bundle binary is done. + if postbuilds: + self.WriteLn("\t@$(call do_postbuilds)") + postbuilds = [] # Don't write postbuilds for target's output. + + # Needed by test/mac/gyptest-rebuild.py. + self.WriteLn("\t@true # No-op, used by tests") + + # Since this target depends on binary and resources which are in + # nested subfolders, the framework directory will be older than + # its dependencies usually. To prevent this rule from executing + # on every build (expensive, especially with postbuilds), expliclity + # update the time on the framework directory. + self.WriteLn("\t@touch -c %s" % QuoteSpaces(self.output)) + + if postbuilds: + assert not self.is_mac_bundle, ( + "Postbuilds for bundles should be done " + "on the bundle, not the binary (target '%s')" % self.target + ) + assert "product_dir" not in spec, ( + "Postbuilds do not work with " "custom product_dir" + ) + + if self.type == "executable": + self.WriteLn( + "%s: LD_INPUTS := %s" + % ( + QuoteSpaces(self.output_binary), + " ".join(QuoteSpaces(dep) for dep in link_deps), + ) + ) + if self.toolset == "host" and self.flavor == "android": + self.WriteDoCmd( + [self.output_binary], + link_deps, + "link_host", + part_of_all, + postbuilds=postbuilds, + ) + else: + self.WriteDoCmd( + [self.output_binary], + link_deps, + "link", + part_of_all, + postbuilds=postbuilds, + ) + + elif self.type == "static_library": + for link_dep in link_deps: + assert " " not in link_dep, ( + "Spaces in alink input filenames not supported (%s)" % link_dep + ) + if ( + self.flavor not in ("mac", "openbsd", "netbsd", "win") + and not self.is_standalone_static_library + ): + self.WriteDoCmd( + [self.output_binary], + link_deps, + "alink_thin", + part_of_all, + postbuilds=postbuilds, + ) + else: + self.WriteDoCmd( + [self.output_binary], + link_deps, + "alink", + part_of_all, + postbuilds=postbuilds, + ) + elif self.type == "shared_library": + self.WriteLn( + "%s: LD_INPUTS := %s" + % ( + QuoteSpaces(self.output_binary), + " ".join(QuoteSpaces(dep) for dep in link_deps), + ) + ) + self.WriteDoCmd( + [self.output_binary], + link_deps, + "solink", + part_of_all, + postbuilds=postbuilds, + ) + elif self.type == "loadable_module": + for link_dep in link_deps: + assert " " not in link_dep, ( + "Spaces in module input filenames not supported (%s)" % link_dep + ) + if self.toolset == "host" and self.flavor == "android": + self.WriteDoCmd( + [self.output_binary], + link_deps, + "solink_module_host", + part_of_all, + postbuilds=postbuilds, + ) + else: + self.WriteDoCmd( + [self.output_binary], + link_deps, + "solink_module", + part_of_all, + postbuilds=postbuilds, + ) + elif self.type == "none": + # Write a stamp line. + self.WriteDoCmd( + [self.output_binary], deps, "touch", part_of_all, postbuilds=postbuilds + ) else: - ldflags = config.get('ldflags', []) - # Compute an rpath for this output if needed. - if any(dep.endswith('.so') or '.so.' in dep for dep in deps): - # We want to get the literal string "$ORIGIN" into the link command, - # so we need lots of escaping. - ldflags.append(r'-Wl,-rpath=\$$ORIGIN/lib.%s/' % self.toolset) - ldflags.append(r'-Wl,-rpath-link=\$(builddir)/lib.%s/' % - self.toolset) - library_dirs = config.get('library_dirs', []) - ldflags += [('-L%s' % library_dir) for library_dir in library_dirs] - self.WriteList(ldflags, 'LDFLAGS_%s' % configname) - if self.flavor == 'mac': - self.WriteList(self.xcode_settings.GetLibtoolflags(configname), - 'LIBTOOLFLAGS_%s' % configname) - libraries = spec.get('libraries') - if libraries: - # Remove duplicate entries - libraries = gyp.common.uniquer(libraries) - if self.flavor == 'mac': - libraries = self.xcode_settings.AdjustLibraries(libraries) - self.WriteList(libraries, 'LIBS') - self.WriteLn('%s: GYP_LDFLAGS := $(LDFLAGS_$(BUILDTYPE))' % - QuoteSpaces(self.output_binary)) - self.WriteLn('%s: LIBS := $(LIBS)' % QuoteSpaces(self.output_binary)) - - if self.flavor == 'mac': - self.WriteLn('%s: GYP_LIBTOOLFLAGS := $(LIBTOOLFLAGS_$(BUILDTYPE))' % - QuoteSpaces(self.output_binary)) - - # Postbuild actions. Like actions, but implicitly depend on the target's - # output. - postbuilds = [] - if self.flavor == 'mac': - if target_postbuilds: - postbuilds.append('$(TARGET_POSTBUILDS_$(BUILDTYPE))') - postbuilds.extend( - gyp.xcode_emulation.GetSpecPostbuildCommands(spec)) - - if postbuilds: - # Envvars may be referenced by TARGET_POSTBUILDS_$(BUILDTYPE), - # so we must output its definition first, since we declare variables - # using ":=". - self.WriteSortedXcodeEnv(self.output, self.GetSortedXcodePostbuildEnv()) - - for configname in target_postbuilds: - self.WriteLn('%s: TARGET_POSTBUILDS_%s := %s' % - (QuoteSpaces(self.output), - configname, - gyp.common.EncodePOSIXShellList(target_postbuilds[configname]))) - - # Postbuilds expect to be run in the gyp file's directory, so insert an - # implicit postbuild to cd to there. - postbuilds.insert(0, gyp.common.EncodePOSIXShellList(['cd', self.path])) - for i in range(len(postbuilds)): - if not postbuilds[i].startswith('$'): - postbuilds[i] = EscapeShellArgument(postbuilds[i]) - self.WriteLn('%s: builddir := $(abs_builddir)' % QuoteSpaces(self.output)) - self.WriteLn('%s: POSTBUILDS := %s' % ( - QuoteSpaces(self.output), ' '.join(postbuilds))) - - # A bundle directory depends on its dependencies such as bundle resources - # and bundle binary. When all dependencies have been built, the bundle - # needs to be packaged. - if self.is_mac_bundle: - # If the framework doesn't contain a binary, then nothing depends - # on the actions -- make the framework depend on them directly too. - self.WriteDependencyOnExtraOutputs(self.output, extra_outputs) - - # Bundle dependencies. Note that the code below adds actions to this - # target, so if you move these two lines, move the lines below as well. - self.WriteList([QuoteSpaces(dep) for dep in bundle_deps], 'BUNDLE_DEPS') - self.WriteLn('%s: $(BUNDLE_DEPS)' % QuoteSpaces(self.output)) - - # After the framework is built, package it. Needs to happen before - # postbuilds, since postbuilds depend on this. - if self.type in ('shared_library', 'loadable_module'): - self.WriteLn('\t@$(call do_cmd,mac_package_framework,,,%s)' % - self.xcode_settings.GetFrameworkVersion()) - - # Bundle postbuilds can depend on the whole bundle, so run them after - # the bundle is packaged, not already after the bundle binary is done. - if postbuilds: - self.WriteLn('\t@$(call do_postbuilds)') - postbuilds = [] # Don't write postbuilds for target's output. - - # Needed by test/mac/gyptest-rebuild.py. - self.WriteLn('\t@true # No-op, used by tests') - - # Since this target depends on binary and resources which are in - # nested subfolders, the framework directory will be older than - # its dependencies usually. To prevent this rule from executing - # on every build (expensive, especially with postbuilds), expliclity - # update the time on the framework directory. - self.WriteLn('\t@touch -c %s' % QuoteSpaces(self.output)) - - if postbuilds: - assert not self.is_mac_bundle, ('Postbuilds for bundles should be done ' - 'on the bundle, not the binary (target \'%s\')' % self.target) - assert 'product_dir' not in spec, ('Postbuilds do not work with ' - 'custom product_dir') - - if self.type == 'executable': - self.WriteLn('%s: LD_INPUTS := %s' % ( - QuoteSpaces(self.output_binary), - ' '.join(QuoteSpaces(dep) for dep in link_deps))) - if self.toolset == 'host' and self.flavor == 'android': - self.WriteDoCmd([self.output_binary], link_deps, 'link_host', - part_of_all, postbuilds=postbuilds) - else: - self.WriteDoCmd([self.output_binary], link_deps, 'link', part_of_all, - postbuilds=postbuilds) - - elif self.type == 'static_library': - for link_dep in link_deps: - assert ' ' not in link_dep, ( - "Spaces in alink input filenames not supported (%s)" % link_dep) - if (self.flavor not in ('mac', 'openbsd', 'netbsd', 'win') and not - self.is_standalone_static_library): - self.WriteDoCmd([self.output_binary], link_deps, 'alink_thin', - part_of_all, postbuilds=postbuilds) - else: - self.WriteDoCmd([self.output_binary], link_deps, 'alink', part_of_all, - postbuilds=postbuilds) - elif self.type == 'shared_library': - self.WriteLn('%s: LD_INPUTS := %s' % ( - QuoteSpaces(self.output_binary), - ' '.join(QuoteSpaces(dep) for dep in link_deps))) - self.WriteDoCmd([self.output_binary], link_deps, 'solink', part_of_all, - postbuilds=postbuilds) - elif self.type == 'loadable_module': - for link_dep in link_deps: - assert ' ' not in link_dep, ( - "Spaces in module input filenames not supported (%s)" % link_dep) - if self.toolset == 'host' and self.flavor == 'android': - self.WriteDoCmd([self.output_binary], link_deps, 'solink_module_host', - part_of_all, postbuilds=postbuilds) - else: - self.WriteDoCmd( - [self.output_binary], link_deps, 'solink_module', part_of_all, - postbuilds=postbuilds) - elif self.type == 'none': - # Write a stamp line. - self.WriteDoCmd([self.output_binary], deps, 'touch', part_of_all, - postbuilds=postbuilds) - else: - print("WARNING: no output for", self.type, self.target) - - # Add an alias for each target (if there are any outputs). - # Installable target aliases are created below. - if ((self.output and self.output != self.target) and - (self.type not in self._INSTALLABLE_TARGETS)): - self.WriteMakeRule([self.target], [self.output], - comment='Add target alias', phony = True) - if part_of_all: - self.WriteMakeRule(['all'], [self.target], - comment = 'Add target alias to "all" target.', - phony = True) - - # Add special-case rules for our installable targets. - # 1) They need to install to the build dir or "product" dir. - # 2) They get shortcuts for building (e.g. "make chrome"). - # 3) They are part of "make all". - if (self.type in self._INSTALLABLE_TARGETS or - self.is_standalone_static_library): - if self.type == 'shared_library': - file_desc = 'shared library' - elif self.type == 'static_library': - file_desc = 'static library' - else: - file_desc = 'executable' - install_path = self._InstallableTargetInstallPath() - installable_deps = [self.output] - if (self.flavor == 'mac' and not 'product_dir' in spec and - self.toolset == 'target'): - # On mac, products are created in install_path immediately. - assert install_path == self.output, '%s != %s' % ( - install_path, self.output) - - # Point the target alias to the final binary output. - self.WriteMakeRule([self.target], [install_path], - comment='Add target alias', phony = True) - if install_path != self.output: - assert not self.is_mac_bundle # See comment a few lines above. - self.WriteDoCmd([install_path], [self.output], 'copy', - comment = 'Copy this to the %s output path.' % - file_desc, part_of_all=part_of_all) - installable_deps.append(install_path) - if self.output != self.alias and self.alias != self.target: - self.WriteMakeRule([self.alias], installable_deps, - comment = 'Short alias for building this %s.' % - file_desc, phony = True) - if part_of_all: - self.WriteMakeRule(['all'], [install_path], - comment = 'Add %s to "all" target.' % file_desc, - phony = True) - - - def WriteList(self, value_list, variable=None, prefix='', - quoter=QuoteIfNecessary): - """Write a variable definition that is a list of values. + print("WARNING: no output for", self.type, self.target) + + # Add an alias for each target (if there are any outputs). + # Installable target aliases are created below. + if (self.output and self.output != self.target) and ( + self.type not in self._INSTALLABLE_TARGETS + ): + self.WriteMakeRule( + [self.target], [self.output], comment="Add target alias", phony=True + ) + if part_of_all: + self.WriteMakeRule( + ["all"], + [self.target], + comment='Add target alias to "all" target.', + phony=True, + ) + + # Add special-case rules for our installable targets. + # 1) They need to install to the build dir or "product" dir. + # 2) They get shortcuts for building (e.g. "make chrome"). + # 3) They are part of "make all". + if self.type in self._INSTALLABLE_TARGETS or self.is_standalone_static_library: + if self.type == "shared_library": + file_desc = "shared library" + elif self.type == "static_library": + file_desc = "static library" + else: + file_desc = "executable" + install_path = self._InstallableTargetInstallPath() + installable_deps = [self.output] + if ( + self.flavor == "mac" + and "product_dir" not in spec + and self.toolset == "target" + ): + # On mac, products are created in install_path immediately. + assert install_path == self.output, "%s != %s" % ( + install_path, + self.output, + ) + + # Point the target alias to the final binary output. + self.WriteMakeRule( + [self.target], [install_path], comment="Add target alias", phony=True + ) + if install_path != self.output: + assert not self.is_mac_bundle # See comment a few lines above. + self.WriteDoCmd( + [install_path], + [self.output], + "copy", + comment="Copy this to the %s output path." % file_desc, + part_of_all=part_of_all, + ) + installable_deps.append(install_path) + if self.output != self.alias and self.alias != self.target: + self.WriteMakeRule( + [self.alias], + installable_deps, + comment="Short alias for building this %s." % file_desc, + phony=True, + ) + if part_of_all: + self.WriteMakeRule( + ["all"], + [install_path], + comment='Add %s to "all" target.' % file_desc, + phony=True, + ) + + def WriteList(self, value_list, variable=None, prefix="", quoter=QuoteIfNecessary): + """Write a variable definition that is a list of values. E.g. WriteList(['a','b'], 'foo', prefix='blah') writes out foo = blaha blahb but in a pretty-printed style. """ - values = '' - if value_list: - value_list = [quoter(prefix + l) for l in value_list] - values = ' \\\n\t' + ' \\\n\t'.join(value_list) - self.fp.write('%s :=%s\n\n' % (variable, values)) - + values = "" + if value_list: + value_list = [quoter(prefix + l) for l in value_list] + values = " \\\n\t" + " \\\n\t".join(value_list) + self.fp.write("%s :=%s\n\n" % (variable, values)) - def WriteDoCmd(self, outputs, inputs, command, part_of_all, comment=None, - postbuilds=False): - """Write a Makefile rule that uses do_cmd. + def WriteDoCmd( + self, outputs, inputs, command, part_of_all, comment=None, postbuilds=False + ): + """Write a Makefile rule that uses do_cmd. This makes the outputs dependent on the command line that was run, as well as support the V= make command line flag. """ - suffix = '' - if postbuilds: - assert ',' not in command - suffix = ',,1' # Tell do_cmd to honor $POSTBUILDS - self.WriteMakeRule(outputs, inputs, - actions = ['$(call do_cmd,%s%s)' % (command, suffix)], - comment = comment, - command = command, - force = True) - # Add our outputs to the list of targets we read depfiles from. - # all_deps is only used for deps file reading, and for deps files we replace - # spaces with ? because escaping doesn't work with make's $(sort) and - # other functions. - outputs = [QuoteSpaces(o, SPACE_REPLACEMENT) for o in outputs] - self.WriteLn('all_deps += %s' % ' '.join(outputs)) - - - def WriteMakeRule(self, outputs, inputs, actions=None, comment=None, - order_only=False, force=False, phony=False, command=None): - """Write a Makefile rule, with some extra tricks. + suffix = "" + if postbuilds: + assert "," not in command + suffix = ",,1" # Tell do_cmd to honor $POSTBUILDS + self.WriteMakeRule( + outputs, + inputs, + actions=["$(call do_cmd,%s%s)" % (command, suffix)], + comment=comment, + command=command, + force=True, + ) + # Add our outputs to the list of targets we read depfiles from. + # all_deps is only used for deps file reading, and for deps files we replace + # spaces with ? because escaping doesn't work with make's $(sort) and + # other functions. + outputs = [QuoteSpaces(o, SPACE_REPLACEMENT) for o in outputs] + self.WriteLn("all_deps += %s" % " ".join(outputs)) + + def WriteMakeRule( + self, + outputs, + inputs, + actions=None, + comment=None, + order_only=False, + force=False, + phony=False, + command=None, + ): + """Write a Makefile rule, with some extra tricks. outputs: a list of outputs for the rule (note: this is not directly supported by make; see comments below) @@ -1746,53 +2000,54 @@ def WriteMakeRule(self, outputs, inputs, actions=None, comment=None, output is just a name to run the rule command: (optional) command name to generate unambiguous labels """ - outputs = [QuoteSpaces(o) for o in outputs] - inputs = [QuoteSpaces(i) for i in inputs] - - if comment: - self.WriteLn('# ' + comment) - if phony: - self.WriteLn('.PHONY: ' + ' '.join(outputs)) - if actions: - self.WriteLn("%s: TOOLSET := $(TOOLSET)" % outputs[0]) - force_append = ' FORCE_DO_CMD' if force else '' - - if order_only: - # Order only rule: Just write a simple rule. - # TODO(evanm): just make order_only a list of deps instead of this hack. - self.WriteLn('%s: | %s%s' % - (' '.join(outputs), ' '.join(inputs), force_append)) - elif len(outputs) == 1: - # Regular rule, one output: Just write a simple rule. - self.WriteLn('%s: %s%s' % (outputs[0], ' '.join(inputs), force_append)) - else: - # Regular rule, more than one output: Multiple outputs are tricky in - # make. We will write three rules: - # - All outputs depend on an intermediate file. - # - Make .INTERMEDIATE depend on the intermediate. - # - The intermediate file depends on the inputs and executes the - # actual command. - # - The intermediate recipe will 'touch' the intermediate file. - # - The multi-output rule will have an do-nothing recipe. - - # Hash the target name to avoid generating overlong filenames. - cmddigest = hashlib.sha1((command or self.target).encode('utf-8')).hexdigest() - intermediate = "%s.intermediate" % cmddigest - self.WriteLn('%s: %s' % (' '.join(outputs), intermediate)) - self.WriteLn('\t%s' % '@:') - self.WriteLn('%s: %s' % ('.INTERMEDIATE', intermediate)) - self.WriteLn('%s: %s%s' % - (intermediate, ' '.join(inputs), force_append)) - actions.insert(0, '$(call do_cmd,touch)') - - if actions: - for action in actions: - self.WriteLn('\t%s' % action) - self.WriteLn() - - - def WriteAndroidNdkModuleRule(self, module_name, all_sources, link_deps): - """Write a set of LOCAL_XXX definitions for Android NDK. + outputs = [QuoteSpaces(o) for o in outputs] + inputs = [QuoteSpaces(i) for i in inputs] + + if comment: + self.WriteLn("# " + comment) + if phony: + self.WriteLn(".PHONY: " + " ".join(outputs)) + if actions: + self.WriteLn("%s: TOOLSET := $(TOOLSET)" % outputs[0]) + force_append = " FORCE_DO_CMD" if force else "" + + if order_only: + # Order only rule: Just write a simple rule. + # TODO(evanm): just make order_only a list of deps instead of this hack. + self.WriteLn( + "%s: | %s%s" % (" ".join(outputs), " ".join(inputs), force_append) + ) + elif len(outputs) == 1: + # Regular rule, one output: Just write a simple rule. + self.WriteLn("%s: %s%s" % (outputs[0], " ".join(inputs), force_append)) + else: + # Regular rule, more than one output: Multiple outputs are tricky in + # make. We will write three rules: + # - All outputs depend on an intermediate file. + # - Make .INTERMEDIATE depend on the intermediate. + # - The intermediate file depends on the inputs and executes the + # actual command. + # - The intermediate recipe will 'touch' the intermediate file. + # - The multi-output rule will have an do-nothing recipe. + + # Hash the target name to avoid generating overlong filenames. + cmddigest = hashlib.sha1( + (command or self.target).encode("utf-8") + ).hexdigest() + intermediate = "%s.intermediate" % cmddigest + self.WriteLn("%s: %s" % (" ".join(outputs), intermediate)) + self.WriteLn("\t%s" % "@:") + self.WriteLn("%s: %s" % (".INTERMEDIATE", intermediate)) + self.WriteLn("%s: %s%s" % (intermediate, " ".join(inputs), force_append)) + actions.insert(0, "$(call do_cmd,touch)") + + if actions: + for action in actions: + self.WriteLn("\t%s" % action) + self.WriteLn() + + def WriteAndroidNdkModuleRule(self, module_name, all_sources, link_deps): + """Write a set of LOCAL_XXX definitions for Android NDK. These variable definitions will be used by Android NDK but do nothing for non-Android applications. @@ -1804,460 +2059,488 @@ def WriteAndroidNdkModuleRule(self, module_name, all_sources, link_deps): link_deps: A list of link dependencies, which must be sorted in the order from dependencies to dependents. """ - if self.type not in ('executable', 'shared_library', 'static_library'): - return - - self.WriteLn('# Variable definitions for Android applications') - self.WriteLn('include $(CLEAR_VARS)') - self.WriteLn('LOCAL_MODULE := ' + module_name) - self.WriteLn('LOCAL_CFLAGS := $(CFLAGS_$(BUILDTYPE)) ' - '$(DEFS_$(BUILDTYPE)) ' - # LOCAL_CFLAGS is applied to both of C and C++. There is - # no way to specify $(CFLAGS_C_$(BUILDTYPE)) only for C - # sources. - '$(CFLAGS_C_$(BUILDTYPE)) ' - # $(INCS_$(BUILDTYPE)) includes the prefix '-I' while - # LOCAL_C_INCLUDES does not expect it. So put it in - # LOCAL_CFLAGS. - '$(INCS_$(BUILDTYPE))') - # LOCAL_CXXFLAGS is obsolete and LOCAL_CPPFLAGS is preferred. - self.WriteLn('LOCAL_CPPFLAGS := $(CFLAGS_CC_$(BUILDTYPE))') - self.WriteLn('LOCAL_C_INCLUDES :=') - self.WriteLn('LOCAL_LDLIBS := $(LDFLAGS_$(BUILDTYPE)) $(LIBS)') - - # Detect the C++ extension. - cpp_ext = {'.cc': 0, '.cpp': 0, '.cxx': 0} - default_cpp_ext = '.cpp' - for filename in all_sources: - ext = os.path.splitext(filename)[1] - if ext in cpp_ext: - cpp_ext[ext] += 1 - if cpp_ext[ext] > cpp_ext[default_cpp_ext]: - default_cpp_ext = ext - self.WriteLn('LOCAL_CPP_EXTENSION := ' + default_cpp_ext) - - self.WriteList(list(map(self.Absolutify, filter(Compilable, all_sources))), - 'LOCAL_SRC_FILES') - - # Filter out those which do not match prefix and suffix and produce - # the resulting list without prefix and suffix. - def DepsToModules(deps, prefix, suffix): - modules = [] - for filepath in deps: - filename = os.path.basename(filepath) - if filename.startswith(prefix) and filename.endswith(suffix): - modules.append(filename[len(prefix):-len(suffix)]) - return modules - - # Retrieve the default value of 'SHARED_LIB_SUFFIX' - params = {'flavor': 'linux'} - default_variables = {} - CalculateVariables(default_variables, params) - - self.WriteList( - DepsToModules(link_deps, - generator_default_variables['SHARED_LIB_PREFIX'], - default_variables['SHARED_LIB_SUFFIX']), - 'LOCAL_SHARED_LIBRARIES') - self.WriteList( - DepsToModules(link_deps, - generator_default_variables['STATIC_LIB_PREFIX'], - generator_default_variables['STATIC_LIB_SUFFIX']), - 'LOCAL_STATIC_LIBRARIES') - - if self.type == 'executable': - self.WriteLn('include $(BUILD_EXECUTABLE)') - elif self.type == 'shared_library': - self.WriteLn('include $(BUILD_SHARED_LIBRARY)') - elif self.type == 'static_library': - self.WriteLn('include $(BUILD_STATIC_LIBRARY)') - self.WriteLn() - - - def WriteLn(self, text=''): - self.fp.write(text + '\n') - - - def GetSortedXcodeEnv(self, additional_settings=None): - return gyp.xcode_emulation.GetSortedXcodeEnv( - self.xcode_settings, "$(abs_builddir)", - os.path.join("$(abs_srcdir)", self.path), "$(BUILDTYPE)", - additional_settings) - - - def GetSortedXcodePostbuildEnv(self): - # CHROMIUM_STRIP_SAVE_FILE is a chromium-specific hack. - # TODO(thakis): It would be nice to have some general mechanism instead. - strip_save_file = self.xcode_settings.GetPerTargetSetting( - 'CHROMIUM_STRIP_SAVE_FILE', '') - # Even if strip_save_file is empty, explicitly write it. Else a postbuild - # might pick up an export from an earlier target. - return self.GetSortedXcodeEnv( - additional_settings={'CHROMIUM_STRIP_SAVE_FILE': strip_save_file}) - - - def WriteSortedXcodeEnv(self, target, env): - for k, v in env: - # For - # foo := a\ b - # the escaped space does the right thing. For - # export foo := a\ b - # it does not -- the backslash is written to the env as literal character. - # So don't escape spaces in |env[k]|. - self.WriteLn('%s: export %s := %s' % (QuoteSpaces(target), k, v)) - - - def Objectify(self, path): - """Convert a path to its output directory form.""" - if '$(' in path: - path = path.replace('$(obj)/', '$(obj).%s/$(TARGET)/' % self.toolset) - if not '$(obj)' in path: - path = '$(obj).%s/$(TARGET)/%s' % (self.toolset, path) - return path - - - def Pchify(self, path, lang): - """Convert a prefix header path to its output directory form.""" - path = self.Absolutify(path) - if '$(' in path: - path = path.replace('$(obj)/', '$(obj).%s/$(TARGET)/pch-%s' % - (self.toolset, lang)) - return path - return '$(obj).%s/$(TARGET)/pch-%s/%s' % (self.toolset, lang, path) - - - def Absolutify(self, path): - """Convert a subdirectory-relative path into a base-relative path. + if self.type not in ("executable", "shared_library", "static_library"): + return + + self.WriteLn("# Variable definitions for Android applications") + self.WriteLn("include $(CLEAR_VARS)") + self.WriteLn("LOCAL_MODULE := " + module_name) + self.WriteLn( + "LOCAL_CFLAGS := $(CFLAGS_$(BUILDTYPE)) " + "$(DEFS_$(BUILDTYPE)) " + # LOCAL_CFLAGS is applied to both of C and C++. There is + # no way to specify $(CFLAGS_C_$(BUILDTYPE)) only for C + # sources. + "$(CFLAGS_C_$(BUILDTYPE)) " + # $(INCS_$(BUILDTYPE)) includes the prefix '-I' while + # LOCAL_C_INCLUDES does not expect it. So put it in + # LOCAL_CFLAGS. + "$(INCS_$(BUILDTYPE))" + ) + # LOCAL_CXXFLAGS is obsolete and LOCAL_CPPFLAGS is preferred. + self.WriteLn("LOCAL_CPPFLAGS := $(CFLAGS_CC_$(BUILDTYPE))") + self.WriteLn("LOCAL_C_INCLUDES :=") + self.WriteLn("LOCAL_LDLIBS := $(LDFLAGS_$(BUILDTYPE)) $(LIBS)") + + # Detect the C++ extension. + cpp_ext = {".cc": 0, ".cpp": 0, ".cxx": 0} + default_cpp_ext = ".cpp" + for filename in all_sources: + ext = os.path.splitext(filename)[1] + if ext in cpp_ext: + cpp_ext[ext] += 1 + if cpp_ext[ext] > cpp_ext[default_cpp_ext]: + default_cpp_ext = ext + self.WriteLn("LOCAL_CPP_EXTENSION := " + default_cpp_ext) + + self.WriteList( + list(map(self.Absolutify, filter(Compilable, all_sources))), + "LOCAL_SRC_FILES", + ) + + # Filter out those which do not match prefix and suffix and produce + # the resulting list without prefix and suffix. + def DepsToModules(deps, prefix, suffix): + modules = [] + for filepath in deps: + filename = os.path.basename(filepath) + if filename.startswith(prefix) and filename.endswith(suffix): + modules.append(filename[len(prefix) : -len(suffix)]) + return modules + + # Retrieve the default value of 'SHARED_LIB_SUFFIX' + params = {"flavor": "linux"} + default_variables = {} + CalculateVariables(default_variables, params) + + self.WriteList( + DepsToModules( + link_deps, + generator_default_variables["SHARED_LIB_PREFIX"], + default_variables["SHARED_LIB_SUFFIX"], + ), + "LOCAL_SHARED_LIBRARIES", + ) + self.WriteList( + DepsToModules( + link_deps, + generator_default_variables["STATIC_LIB_PREFIX"], + generator_default_variables["STATIC_LIB_SUFFIX"], + ), + "LOCAL_STATIC_LIBRARIES", + ) + + if self.type == "executable": + self.WriteLn("include $(BUILD_EXECUTABLE)") + elif self.type == "shared_library": + self.WriteLn("include $(BUILD_SHARED_LIBRARY)") + elif self.type == "static_library": + self.WriteLn("include $(BUILD_STATIC_LIBRARY)") + self.WriteLn() + + def WriteLn(self, text=""): + self.fp.write(text + "\n") + + def GetSortedXcodeEnv(self, additional_settings=None): + return gyp.xcode_emulation.GetSortedXcodeEnv( + self.xcode_settings, + "$(abs_builddir)", + os.path.join("$(abs_srcdir)", self.path), + "$(BUILDTYPE)", + additional_settings, + ) + + def GetSortedXcodePostbuildEnv(self): + # CHROMIUM_STRIP_SAVE_FILE is a chromium-specific hack. + # TODO(thakis): It would be nice to have some general mechanism instead. + strip_save_file = self.xcode_settings.GetPerTargetSetting( + "CHROMIUM_STRIP_SAVE_FILE", "" + ) + # Even if strip_save_file is empty, explicitly write it. Else a postbuild + # might pick up an export from an earlier target. + return self.GetSortedXcodeEnv( + additional_settings={"CHROMIUM_STRIP_SAVE_FILE": strip_save_file} + ) + + def WriteSortedXcodeEnv(self, target, env): + for k, v in env: + # For + # foo := a\ b + # the escaped space does the right thing. For + # export foo := a\ b + # it does not -- the backslash is written to the env as literal character. + # So don't escape spaces in |env[k]|. + self.WriteLn("%s: export %s := %s" % (QuoteSpaces(target), k, v)) + + def Objectify(self, path): + """Convert a path to its output directory form.""" + if "$(" in path: + path = path.replace("$(obj)/", "$(obj).%s/$(TARGET)/" % self.toolset) + if "$(obj)" not in path: + path = "$(obj).%s/$(TARGET)/%s" % (self.toolset, path) + return path + + def Pchify(self, path, lang): + """Convert a prefix header path to its output directory form.""" + path = self.Absolutify(path) + if "$(" in path: + path = path.replace( + "$(obj)/", "$(obj).%s/$(TARGET)/pch-%s" % (self.toolset, lang) + ) + return path + return "$(obj).%s/$(TARGET)/pch-%s/%s" % (self.toolset, lang, path) + + def Absolutify(self, path): + """Convert a subdirectory-relative path into a base-relative path. Skips over paths that contain variables.""" - if '$(' in path: - # Don't call normpath in this case, as it might collapse the - # path too aggressively if it features '..'. However it's still - # important to strip trailing slashes. - return path.rstrip('/') - return os.path.normpath(os.path.join(self.path, path)) - - - def ExpandInputRoot(self, template, expansion, dirname): - if '%(INPUT_ROOT)s' not in template and '%(INPUT_DIRNAME)s' not in template: - return template - path = template % { - 'INPUT_ROOT': expansion, - 'INPUT_DIRNAME': dirname, + if "$(" in path: + # Don't call normpath in this case, as it might collapse the + # path too aggressively if it features '..'. However it's still + # important to strip trailing slashes. + return path.rstrip("/") + return os.path.normpath(os.path.join(self.path, path)) + + def ExpandInputRoot(self, template, expansion, dirname): + if "%(INPUT_ROOT)s" not in template and "%(INPUT_DIRNAME)s" not in template: + return template + path = template % { + "INPUT_ROOT": expansion, + "INPUT_DIRNAME": dirname, } - return path - - - def _InstallableTargetInstallPath(self): - """Returns the location of the final output for an installable target.""" - # Xcode puts shared_library results into PRODUCT_DIR, and some gyp files - # rely on this. Emulate this behavior for mac. - if (self.type == 'shared_library' and - (self.flavor != 'mac' or self.toolset != 'target')): - # Install all shared libs into a common directory (per toolset) for - # convenient access with LD_LIBRARY_PATH. - return '$(builddir)/lib.%s/%s' % (self.toolset, self.alias) - return '$(builddir)/' + self.alias - - -def WriteAutoRegenerationRule(params, root_makefile, makefile_name, - build_files): - """Write the target to regenerate the Makefile.""" - options = params['options'] - build_files_args = [gyp.common.RelativePath(filename, options.toplevel_dir) - for filename in params['build_files_arg']] - - gyp_binary = gyp.common.FixIfRelativePath(params['gyp_binary'], - options.toplevel_dir) - if not gyp_binary.startswith(os.sep): - gyp_binary = os.path.join('.', gyp_binary) - - root_makefile.write( - "quiet_cmd_regen_makefile = ACTION Regenerating $@\n" - "cmd_regen_makefile = cd $(srcdir); %(cmd)s\n" - "%(makefile_name)s: %(deps)s\n" - "\t$(call do_cmd,regen_makefile)\n\n" % { - 'makefile_name': makefile_name, - 'deps': ' '.join(SourceifyAndQuoteSpaces(bf) for bf in build_files), - 'cmd': gyp.common.EncodePOSIXShellList( - [gyp_binary, '-fmake'] + - gyp.RegenerateFlags(options) + - build_files_args)}) + return path + + def _InstallableTargetInstallPath(self): + """Returns the location of the final output for an installable target.""" + # Xcode puts shared_library results into PRODUCT_DIR, and some gyp files + # rely on this. Emulate this behavior for mac. + if self.type == "shared_library" and ( + self.flavor != "mac" or self.toolset != "target" + ): + # Install all shared libs into a common directory (per toolset) for + # convenient access with LD_LIBRARY_PATH. + return "$(builddir)/lib.%s/%s" % (self.toolset, self.alias) + return "$(builddir)/" + self.alias + + +def WriteAutoRegenerationRule(params, root_makefile, makefile_name, build_files): + """Write the target to regenerate the Makefile.""" + options = params["options"] + build_files_args = [ + gyp.common.RelativePath(filename, options.toplevel_dir) + for filename in params["build_files_arg"] + ] + + gyp_binary = gyp.common.FixIfRelativePath( + params["gyp_binary"], options.toplevel_dir + ) + if not gyp_binary.startswith(os.sep): + gyp_binary = os.path.join(".", gyp_binary) + + root_makefile.write( + "quiet_cmd_regen_makefile = ACTION Regenerating $@\n" + "cmd_regen_makefile = cd $(srcdir); %(cmd)s\n" + "%(makefile_name)s: %(deps)s\n" + "\t$(call do_cmd,regen_makefile)\n\n" + % { + "makefile_name": makefile_name, + "deps": " ".join(SourceifyAndQuoteSpaces(bf) for bf in build_files), + "cmd": gyp.common.EncodePOSIXShellList( + [gyp_binary, "-fmake"] + gyp.RegenerateFlags(options) + build_files_args + ), + } + ) def PerformBuild(data, configurations, params): - options = params['options'] - for config in configurations: - arguments = ['make'] - if options.toplevel_dir and options.toplevel_dir != '.': - arguments += '-C', options.toplevel_dir - arguments.append('BUILDTYPE=' + config) - print('Building [%s]: %s' % (config, arguments)) - subprocess.check_call(arguments) + options = params["options"] + for config in configurations: + arguments = ["make"] + if options.toplevel_dir and options.toplevel_dir != ".": + arguments += "-C", options.toplevel_dir + arguments.append("BUILDTYPE=" + config) + print("Building [%s]: %s" % (config, arguments)) + subprocess.check_call(arguments) def GenerateOutput(target_list, target_dicts, data, params): - options = params['options'] - flavor = gyp.common.GetFlavor(params) - generator_flags = params.get('generator_flags', {}) - builddir_name = generator_flags.get('output_dir', 'out') - android_ndk_version = generator_flags.get('android_ndk_version', None) - default_target = generator_flags.get('default_target', 'all') - - def CalculateMakefilePath(build_file, base_name): - """Determine where to write a Makefile for a given gyp file.""" - # Paths in gyp files are relative to the .gyp file, but we want - # paths relative to the source root for the master makefile. Grab - # the path of the .gyp file as the base to relativize against. - # E.g. "foo/bar" when we're constructing targets for "foo/bar/baz.gyp". - base_path = gyp.common.RelativePath(os.path.dirname(build_file), - options.depth) - # We write the file in the base_path directory. - output_file = os.path.join(options.depth, base_path, base_name) + options = params["options"] + flavor = gyp.common.GetFlavor(params) + generator_flags = params.get("generator_flags", {}) + builddir_name = generator_flags.get("output_dir", "out") + android_ndk_version = generator_flags.get("android_ndk_version", None) + default_target = generator_flags.get("default_target", "all") + + def CalculateMakefilePath(build_file, base_name): + """Determine where to write a Makefile for a given gyp file.""" + # Paths in gyp files are relative to the .gyp file, but we want + # paths relative to the source root for the master makefile. Grab + # the path of the .gyp file as the base to relativize against. + # E.g. "foo/bar" when we're constructing targets for "foo/bar/baz.gyp". + base_path = gyp.common.RelativePath(os.path.dirname(build_file), options.depth) + # We write the file in the base_path directory. + output_file = os.path.join(options.depth, base_path, base_name) + if options.generator_output: + output_file = os.path.join( + options.depth, options.generator_output, base_path, base_name + ) + base_path = gyp.common.RelativePath( + os.path.dirname(build_file), options.toplevel_dir + ) + return base_path, output_file + + # TODO: search for the first non-'Default' target. This can go + # away when we add verification that all targets have the + # necessary configurations. + default_configuration = None + toolsets = set([target_dicts[target]["toolset"] for target in target_list]) + for target in target_list: + spec = target_dicts[target] + if spec["default_configuration"] != "Default": + default_configuration = spec["default_configuration"] + break + if not default_configuration: + default_configuration = "Default" + + srcdir = "." + makefile_name = "Makefile" + options.suffix + makefile_path = os.path.join(options.toplevel_dir, makefile_name) if options.generator_output: - output_file = os.path.join( - options.depth, options.generator_output, base_path, base_name) - base_path = gyp.common.RelativePath(os.path.dirname(build_file), - options.toplevel_dir) - return base_path, output_file - - # TODO: search for the first non-'Default' target. This can go - # away when we add verification that all targets have the - # necessary configurations. - default_configuration = None - toolsets = set([target_dicts[target]['toolset'] for target in target_list]) - for target in target_list: - spec = target_dicts[target] - if spec['default_configuration'] != 'Default': - default_configuration = spec['default_configuration'] - break - if not default_configuration: - default_configuration = 'Default' - - srcdir = '.' - makefile_name = 'Makefile' + options.suffix - makefile_path = os.path.join(options.toplevel_dir, makefile_name) - if options.generator_output: - global srcdir_prefix - makefile_path = os.path.join( - options.toplevel_dir, options.generator_output, makefile_name) - srcdir = gyp.common.RelativePath(srcdir, options.generator_output) - srcdir_prefix = '$(srcdir)/' - - flock_command= 'flock' - copy_archive_arguments = '-af' - makedep_arguments = '-MMD' - header_params = { - 'default_target': default_target, - 'builddir': builddir_name, - 'default_configuration': default_configuration, - 'flock': flock_command, - 'flock_index': 1, - 'link_commands': LINK_COMMANDS_LINUX, - 'extra_commands': '', - 'srcdir': srcdir, - 'copy_archive_args': copy_archive_arguments, - 'makedep_args': makedep_arguments, - 'CC.target': GetEnvironFallback(('CC_target', 'CC'), '$(CC)'), - 'AR.target': GetEnvironFallback(('AR_target', 'AR'), '$(AR)'), - 'CXX.target': GetEnvironFallback(('CXX_target', 'CXX'), '$(CXX)'), - 'LINK.target': GetEnvironFallback(('LINK_target', 'LINK'), '$(LINK)'), - 'CC.host': GetEnvironFallback(('CC_host', 'CC'), 'gcc'), - 'AR.host': GetEnvironFallback(('AR_host', 'AR'), 'ar'), - 'CXX.host': GetEnvironFallback(('CXX_host', 'CXX'), 'g++'), - 'LINK.host': GetEnvironFallback(('LINK_host', 'LINK'), '$(CXX.host)'), + global srcdir_prefix + makefile_path = os.path.join( + options.toplevel_dir, options.generator_output, makefile_name + ) + srcdir = gyp.common.RelativePath(srcdir, options.generator_output) + srcdir_prefix = "$(srcdir)/" + + flock_command = "flock" + copy_archive_arguments = "-af" + makedep_arguments = "-MMD" + header_params = { + "default_target": default_target, + "builddir": builddir_name, + "default_configuration": default_configuration, + "flock": flock_command, + "flock_index": 1, + "link_commands": LINK_COMMANDS_LINUX, + "extra_commands": "", + "srcdir": srcdir, + "copy_archive_args": copy_archive_arguments, + "makedep_args": makedep_arguments, + "CC.target": GetEnvironFallback(("CC_target", "CC"), "$(CC)"), + "AR.target": GetEnvironFallback(("AR_target", "AR"), "$(AR)"), + "CXX.target": GetEnvironFallback(("CXX_target", "CXX"), "$(CXX)"), + "LINK.target": GetEnvironFallback(("LINK_target", "LINK"), "$(LINK)"), + "CC.host": GetEnvironFallback(("CC_host", "CC"), "gcc"), + "AR.host": GetEnvironFallback(("AR_host", "AR"), "ar"), + "CXX.host": GetEnvironFallback(("CXX_host", "CXX"), "g++"), + "LINK.host": GetEnvironFallback(("LINK_host", "LINK"), "$(CXX.host)"), } - if flavor == 'mac': - flock_command = './gyp-mac-tool flock' - header_params.update({ - 'flock': flock_command, - 'flock_index': 2, - 'link_commands': LINK_COMMANDS_MAC, - 'extra_commands': SHARED_HEADER_MAC_COMMANDS, - }) - elif flavor == 'android': - header_params.update({ - 'link_commands': LINK_COMMANDS_ANDROID, - }) - elif flavor == 'zos': - copy_archive_arguments = '-fPR' - makedep_arguments = '-qmakedep=gcc' - header_params.update({ - 'copy_archive_args': copy_archive_arguments, - 'makedep_args': makedep_arguments, - 'link_commands': LINK_COMMANDS_OS390, - 'CC.target': GetEnvironFallback(('CC_target', 'CC'), 'njsc'), - 'CXX.target': GetEnvironFallback(('CXX_target', 'CXX'), 'njsc++'), - 'CC.host': GetEnvironFallback(('CC_host', 'CC'), 'njsc'), - 'CXX.host': GetEnvironFallback(('CXX_host', 'CXX'), 'njsc++'), - }) - elif flavor == 'solaris': - header_params.update({ - 'flock': './gyp-flock-tool flock', - 'flock_index': 2, - }) - elif flavor == 'freebsd': - # Note: OpenBSD has sysutils/flock. lockf seems to be FreeBSD specific. - header_params.update({ - 'flock': 'lockf', - }) - elif flavor == 'openbsd': - copy_archive_arguments = '-pPRf' - header_params.update({ - 'copy_archive_args': copy_archive_arguments, - }) - elif flavor == 'aix': - copy_archive_arguments = '-pPRf' - header_params.update({ - 'copy_archive_args': copy_archive_arguments, - 'link_commands': LINK_COMMANDS_AIX, - 'flock': './gyp-flock-tool flock', - 'flock_index': 2, - }) - - build_file, _, _ = gyp.common.ParseQualifiedTarget(target_list[0]) - make_global_settings_array = data[build_file].get('make_global_settings', []) - wrappers = {} - for key, value in make_global_settings_array: - if key.endswith('_wrapper'): - wrappers[key[:-len('_wrapper')]] = '$(abspath %s)' % value - make_global_settings = '' - for key, value in make_global_settings_array: - if re.match('.*_wrapper', key): - continue - if value[0] != '$': - value = '$(abspath %s)' % value - wrapper = wrappers.get(key) - if wrapper: - value = '%s %s' % (wrapper, value) - del wrappers[key] - if key in ('CC', 'CC.host', 'CXX', 'CXX.host'): - make_global_settings += ( - 'ifneq (,$(filter $(origin %s), undefined default))\n' % key) - # Let gyp-time envvars win over global settings. - env_key = key.replace('.', '_') # CC.host -> CC_host - if env_key in os.environ: - value = os.environ[env_key] - make_global_settings += ' %s = %s\n' % (key, value) - make_global_settings += 'endif\n' - else: - make_global_settings += '%s ?= %s\n' % (key, value) - # TODO(ukai): define cmd when only wrapper is specified in - # make_global_settings. - - header_params['make_global_settings'] = make_global_settings - - gyp.common.EnsureDirExists(makefile_path) - root_makefile = open(makefile_path, 'w') - root_makefile.write(SHARED_HEADER % header_params) - # Currently any versions have the same effect, but in future the behavior - # could be different. - if android_ndk_version: - root_makefile.write( - '# Define LOCAL_PATH for build of Android applications.\n' - 'LOCAL_PATH := $(call my-dir)\n' - '\n') - for toolset in toolsets: - root_makefile.write('TOOLSET := %s\n' % toolset) - WriteRootHeaderSuffixRules(root_makefile) - - # Put build-time support tools next to the root Makefile. - dest_path = os.path.dirname(makefile_path) - gyp.common.CopyTool(flavor, dest_path) - - # Find the list of targets that derive from the gyp file(s) being built. - needed_targets = set() - for build_file in params['build_files']: - for target in gyp.common.AllTargets(target_list, target_dicts, build_file): - needed_targets.add(target) - - build_files = set() - include_list = set() - for qualified_target in target_list: - build_file, target, toolset = gyp.common.ParseQualifiedTarget( - qualified_target) - - this_make_global_settings = data[build_file].get('make_global_settings', []) - assert make_global_settings_array == this_make_global_settings, ( - "make_global_settings needs to be the same for all targets. %s vs. %s" % - (this_make_global_settings, make_global_settings)) - - build_files.add(gyp.common.RelativePath(build_file, options.toplevel_dir)) - included_files = data[build_file]['included_files'] - for included_file in included_files: - # The included_files entries are relative to the dir of the build file - # that included them, so we have to undo that and then make them relative - # to the root dir. - relative_include_file = gyp.common.RelativePath( - gyp.common.UnrelativePath(included_file, build_file), - options.toplevel_dir) - abs_include_file = os.path.abspath(relative_include_file) - # If the include file is from the ~/.gyp dir, we should use absolute path - # so that relocating the src dir doesn't break the path. - if (params['home_dot_gyp'] and - abs_include_file.startswith(params['home_dot_gyp'])): - build_files.add(abs_include_file) - else: - build_files.add(relative_include_file) - - base_path, output_file = CalculateMakefilePath(build_file, - target + '.' + toolset + options.suffix + '.mk') - - spec = target_dicts[qualified_target] - configs = spec['configurations'] - - if flavor == 'mac': - gyp.xcode_emulation.MergeGlobalXcodeSettingsToSpec(data[build_file], spec) - - writer = MakefileWriter(generator_flags, flavor) - writer.Write(qualified_target, base_path, output_file, spec, configs, - part_of_all=qualified_target in needed_targets) - - # Our root_makefile lives at the source root. Compute the relative path - # from there to the output_file for including. - mkfile_rel_path = gyp.common.RelativePath(output_file, - os.path.dirname(makefile_path)) - include_list.add(mkfile_rel_path) - - # Write out per-gyp (sub-project) Makefiles. - depth_rel_path = gyp.common.RelativePath(options.depth, os.getcwd()) - for build_file in build_files: - # The paths in build_files were relativized above, so undo that before - # testing against the non-relativized items in target_list and before - # calculating the Makefile path. - build_file = os.path.join(depth_rel_path, build_file) - gyp_targets = [target_dicts[target]['target_name'] for target in target_list - if target.startswith(build_file) and - target in needed_targets] - # Only generate Makefiles for gyp files with targets. - if not gyp_targets: - continue - base_path, output_file = CalculateMakefilePath(build_file, - os.path.splitext(os.path.basename(build_file))[0] + '.Makefile') - makefile_rel_path = gyp.common.RelativePath(os.path.dirname(makefile_path), - os.path.dirname(output_file)) - writer.WriteSubMake(output_file, makefile_rel_path, gyp_targets, - builddir_name) - - - # Write out the sorted list of includes. - root_makefile.write('\n') - for include_file in sorted(include_list): - # We wrap each .mk include in an if statement so users can tell make to - # not load a file by setting NO_LOAD. The below make code says, only - # load the .mk file if the .mk filename doesn't start with a token in - # NO_LOAD. - root_makefile.write( - "ifeq ($(strip $(foreach prefix,$(NO_LOAD),\\\n" - " $(findstring $(join ^,$(prefix)),\\\n" - " $(join ^," + include_file + ")))),)\n") - root_makefile.write(" include " + include_file + "\n") - root_makefile.write("endif\n") - root_makefile.write('\n') - - if (not generator_flags.get('standalone') - and generator_flags.get('auto_regeneration', True)): - WriteAutoRegenerationRule(params, root_makefile, makefile_name, build_files) + if flavor == "mac": + flock_command = "./gyp-mac-tool flock" + header_params.update( + { + "flock": flock_command, + "flock_index": 2, + "link_commands": LINK_COMMANDS_MAC, + "extra_commands": SHARED_HEADER_MAC_COMMANDS, + } + ) + elif flavor == "android": + header_params.update({"link_commands": LINK_COMMANDS_ANDROID}) + elif flavor == "zos": + copy_archive_arguments = "-fPR" + makedep_arguments = "-qmakedep=gcc" + header_params.update( + { + "copy_archive_args": copy_archive_arguments, + "makedep_args": makedep_arguments, + "link_commands": LINK_COMMANDS_OS390, + "CC.target": GetEnvironFallback(("CC_target", "CC"), "njsc"), + "CXX.target": GetEnvironFallback(("CXX_target", "CXX"), "njsc++"), + "CC.host": GetEnvironFallback(("CC_host", "CC"), "njsc"), + "CXX.host": GetEnvironFallback(("CXX_host", "CXX"), "njsc++"), + } + ) + elif flavor == "solaris": + header_params.update({"flock": "./gyp-flock-tool flock", "flock_index": 2}) + elif flavor == "freebsd": + # Note: OpenBSD has sysutils/flock. lockf seems to be FreeBSD specific. + header_params.update({"flock": "lockf"}) + elif flavor == "openbsd": + copy_archive_arguments = "-pPRf" + header_params.update({"copy_archive_args": copy_archive_arguments}) + elif flavor == "aix": + copy_archive_arguments = "-pPRf" + header_params.update( + { + "copy_archive_args": copy_archive_arguments, + "link_commands": LINK_COMMANDS_AIX, + "flock": "./gyp-flock-tool flock", + "flock_index": 2, + } + ) + + build_file, _, _ = gyp.common.ParseQualifiedTarget(target_list[0]) + make_global_settings_array = data[build_file].get("make_global_settings", []) + wrappers = {} + for key, value in make_global_settings_array: + if key.endswith("_wrapper"): + wrappers[key[: -len("_wrapper")]] = "$(abspath %s)" % value + make_global_settings = "" + for key, value in make_global_settings_array: + if re.match(".*_wrapper", key): + continue + if value[0] != "$": + value = "$(abspath %s)" % value + wrapper = wrappers.get(key) + if wrapper: + value = "%s %s" % (wrapper, value) + del wrappers[key] + if key in ("CC", "CC.host", "CXX", "CXX.host"): + make_global_settings += ( + "ifneq (,$(filter $(origin %s), undefined default))\n" % key + ) + # Let gyp-time envvars win over global settings. + env_key = key.replace(".", "_") # CC.host -> CC_host + if env_key in os.environ: + value = os.environ[env_key] + make_global_settings += " %s = %s\n" % (key, value) + make_global_settings += "endif\n" + else: + make_global_settings += "%s ?= %s\n" % (key, value) + # TODO(ukai): define cmd when only wrapper is specified in + # make_global_settings. - root_makefile.write(SHARED_FOOTER) + header_params["make_global_settings"] = make_global_settings - root_makefile.close() + gyp.common.EnsureDirExists(makefile_path) + root_makefile = open(makefile_path, "w") + root_makefile.write(SHARED_HEADER % header_params) + # Currently any versions have the same effect, but in future the behavior + # could be different. + if android_ndk_version: + root_makefile.write( + "# Define LOCAL_PATH for build of Android applications.\n" + "LOCAL_PATH := $(call my-dir)\n" + "\n" + ) + for toolset in toolsets: + root_makefile.write("TOOLSET := %s\n" % toolset) + WriteRootHeaderSuffixRules(root_makefile) + + # Put build-time support tools next to the root Makefile. + dest_path = os.path.dirname(makefile_path) + gyp.common.CopyTool(flavor, dest_path) + + # Find the list of targets that derive from the gyp file(s) being built. + needed_targets = set() + for build_file in params["build_files"]: + for target in gyp.common.AllTargets(target_list, target_dicts, build_file): + needed_targets.add(target) + + build_files = set() + include_list = set() + for qualified_target in target_list: + build_file, target, toolset = gyp.common.ParseQualifiedTarget(qualified_target) + + this_make_global_settings = data[build_file].get("make_global_settings", []) + assert make_global_settings_array == this_make_global_settings, ( + "make_global_settings needs to be the same for all targets. %s vs. %s" + % (this_make_global_settings, make_global_settings) + ) + + build_files.add(gyp.common.RelativePath(build_file, options.toplevel_dir)) + included_files = data[build_file]["included_files"] + for included_file in included_files: + # The included_files entries are relative to the dir of the build file + # that included them, so we have to undo that and then make them relative + # to the root dir. + relative_include_file = gyp.common.RelativePath( + gyp.common.UnrelativePath(included_file, build_file), + options.toplevel_dir, + ) + abs_include_file = os.path.abspath(relative_include_file) + # If the include file is from the ~/.gyp dir, we should use absolute path + # so that relocating the src dir doesn't break the path. + if params["home_dot_gyp"] and abs_include_file.startswith( + params["home_dot_gyp"] + ): + build_files.add(abs_include_file) + else: + build_files.add(relative_include_file) + + base_path, output_file = CalculateMakefilePath( + build_file, target + "." + toolset + options.suffix + ".mk" + ) + + spec = target_dicts[qualified_target] + configs = spec["configurations"] + + if flavor == "mac": + gyp.xcode_emulation.MergeGlobalXcodeSettingsToSpec(data[build_file], spec) + + writer = MakefileWriter(generator_flags, flavor) + writer.Write( + qualified_target, + base_path, + output_file, + spec, + configs, + part_of_all=qualified_target in needed_targets, + ) + + # Our root_makefile lives at the source root. Compute the relative path + # from there to the output_file for including. + mkfile_rel_path = gyp.common.RelativePath( + output_file, os.path.dirname(makefile_path) + ) + include_list.add(mkfile_rel_path) + + # Write out per-gyp (sub-project) Makefiles. + depth_rel_path = gyp.common.RelativePath(options.depth, os.getcwd()) + for build_file in build_files: + # The paths in build_files were relativized above, so undo that before + # testing against the non-relativized items in target_list and before + # calculating the Makefile path. + build_file = os.path.join(depth_rel_path, build_file) + gyp_targets = [ + target_dicts[qualified_target]["target_name"] + for qualified_target in target_list + if qualified_target.startswith(build_file) + and qualified_target in needed_targets + ] + # Only generate Makefiles for gyp files with targets. + if not gyp_targets: + continue + base_path, output_file = CalculateMakefilePath( + build_file, os.path.splitext(os.path.basename(build_file))[0] + ".Makefile" + ) + makefile_rel_path = gyp.common.RelativePath( + os.path.dirname(makefile_path), os.path.dirname(output_file) + ) + writer.WriteSubMake(output_file, makefile_rel_path, gyp_targets, builddir_name) + + # Write out the sorted list of includes. + root_makefile.write("\n") + for include_file in sorted(include_list): + # We wrap each .mk include in an if statement so users can tell make to + # not load a file by setting NO_LOAD. The below make code says, only + # load the .mk file if the .mk filename doesn't start with a token in + # NO_LOAD. + root_makefile.write( + "ifeq ($(strip $(foreach prefix,$(NO_LOAD),\\\n" + " $(findstring $(join ^,$(prefix)),\\\n" + " $(join ^," + include_file + ")))),)\n" + ) + root_makefile.write(" include " + include_file + "\n") + root_makefile.write("endif\n") + root_makefile.write("\n") + + if not generator_flags.get("standalone") and generator_flags.get( + "auto_regeneration", True + ): + WriteAutoRegenerationRule(params, root_makefile, makefile_name, build_files) + + root_makefile.write(SHARED_FOOTER) + + root_makefile.close() diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index 933042c711..3a8aac0eb5 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -4,7 +4,6 @@ from __future__ import print_function -import copy import ntpath import os import posixpath @@ -38,64 +37,64 @@ # if IncrediBuild is executed from inside Visual Studio. This regex # validates that the string looks like a GUID with all uppercase hex # letters. -VALID_MSVS_GUID_CHARS = re.compile(r'^[A-F0-9\-]+$') +VALID_MSVS_GUID_CHARS = re.compile(r"^[A-F0-9\-]+$") generator_default_variables = { - 'DRIVER_PREFIX': '', - 'DRIVER_SUFFIX': '.sys', - 'EXECUTABLE_PREFIX': '', - 'EXECUTABLE_SUFFIX': '.exe', - 'STATIC_LIB_PREFIX': '', - 'SHARED_LIB_PREFIX': '', - 'STATIC_LIB_SUFFIX': '.lib', - 'SHARED_LIB_SUFFIX': '.dll', - 'INTERMEDIATE_DIR': '$(IntDir)', - 'SHARED_INTERMEDIATE_DIR': '$(OutDir)obj/global_intermediate', - 'OS': 'win', - 'PRODUCT_DIR': '$(OutDir)', - 'LIB_DIR': '$(OutDir)lib', - 'RULE_INPUT_ROOT': '$(InputName)', - 'RULE_INPUT_DIRNAME': '$(InputDir)', - 'RULE_INPUT_EXT': '$(InputExt)', - 'RULE_INPUT_NAME': '$(InputFileName)', - 'RULE_INPUT_PATH': '$(InputPath)', - 'CONFIGURATION_NAME': '$(ConfigurationName)', + "DRIVER_PREFIX": "", + "DRIVER_SUFFIX": ".sys", + "EXECUTABLE_PREFIX": "", + "EXECUTABLE_SUFFIX": ".exe", + "STATIC_LIB_PREFIX": "", + "SHARED_LIB_PREFIX": "", + "STATIC_LIB_SUFFIX": ".lib", + "SHARED_LIB_SUFFIX": ".dll", + "INTERMEDIATE_DIR": "$(IntDir)", + "SHARED_INTERMEDIATE_DIR": "$(OutDir)obj/global_intermediate", + "OS": "win", + "PRODUCT_DIR": "$(OutDir)", + "LIB_DIR": "$(OutDir)lib", + "RULE_INPUT_ROOT": "$(InputName)", + "RULE_INPUT_DIRNAME": "$(InputDir)", + "RULE_INPUT_EXT": "$(InputExt)", + "RULE_INPUT_NAME": "$(InputFileName)", + "RULE_INPUT_PATH": "$(InputPath)", + "CONFIGURATION_NAME": "$(ConfigurationName)", } # The msvs specific sections that hold paths generator_additional_path_sections = [ - 'msvs_cygwin_dirs', - 'msvs_props', + "msvs_cygwin_dirs", + "msvs_props", ] generator_additional_non_configuration_keys = [ - 'msvs_cygwin_dirs', - 'msvs_cygwin_shell', - 'msvs_large_pdb', - 'msvs_shard', - 'msvs_external_builder', - 'msvs_external_builder_out_dir', - 'msvs_external_builder_build_cmd', - 'msvs_external_builder_clean_cmd', - 'msvs_external_builder_clcompile_cmd', - 'msvs_enable_winrt', - 'msvs_requires_importlibrary', - 'msvs_enable_winphone', - 'msvs_enable_marmasm', - 'msvs_application_type_revision', - 'msvs_target_platform_version', - 'msvs_target_platform_minversion', + "msvs_cygwin_dirs", + "msvs_cygwin_shell", + "msvs_large_pdb", + "msvs_shard", + "msvs_external_builder", + "msvs_external_builder_out_dir", + "msvs_external_builder_build_cmd", + "msvs_external_builder_clean_cmd", + "msvs_external_builder_clcompile_cmd", + "msvs_enable_winrt", + "msvs_requires_importlibrary", + "msvs_enable_winphone", + "msvs_enable_marmasm", + "msvs_application_type_revision", + "msvs_target_platform_version", + "msvs_target_platform_minversion", ] generator_filelist_paths = None # List of precompiled header related keys. precomp_keys = [ - 'msvs_precompiled_header', - 'msvs_precompiled_source', + "msvs_precompiled_header", + "msvs_precompiled_source", ] @@ -110,36 +109,38 @@ # python version in depot_tools has been updated to work on Vista # 64-bit. def _GetDomainAndUserName(): - if sys.platform not in ('win32', 'cygwin'): - return ('DOMAIN', 'USERNAME') - global cached_username - global cached_domain - if not cached_domain or not cached_username: - domain = os.environ.get('USERDOMAIN') - username = os.environ.get('USERNAME') - if not domain or not username: - call = subprocess.Popen(['net', 'config', 'Workstation'], - stdout=subprocess.PIPE) - config = call.communicate()[0] - if PY3: - config = config.decode('utf-8') - username_re = re.compile(r'^User name\s+(\S+)', re.MULTILINE) - username_match = username_re.search(config) - if username_match: - username = username_match.group(1) - domain_re = re.compile(r'^Logon domain\s+(\S+)', re.MULTILINE) - domain_match = domain_re.search(config) - if domain_match: - domain = domain_match.group(1) - cached_domain = domain - cached_username = username - return (cached_domain, cached_username) + if sys.platform not in ("win32", "cygwin"): + return ("DOMAIN", "USERNAME") + global cached_username + global cached_domain + if not cached_domain or not cached_username: + domain = os.environ.get("USERDOMAIN") + username = os.environ.get("USERNAME") + if not domain or not username: + call = subprocess.Popen( + ["net", "config", "Workstation"], stdout=subprocess.PIPE + ) + config = call.communicate()[0] + if PY3: + config = config.decode("utf-8") + username_re = re.compile(r"^User name\s+(\S+)", re.MULTILINE) + username_match = username_re.search(config) + if username_match: + username = username_match.group(1) + domain_re = re.compile(r"^Logon domain\s+(\S+)", re.MULTILINE) + domain_match = domain_re.search(config) + if domain_match: + domain = domain_match.group(1) + cached_domain = domain + cached_username = username + return (cached_domain, cached_username) + fixpath_prefix = None def _NormalizedSource(source): - """Normalize the path. + """Normalize the path. But not if that gets rid of a variable, as this may expand to something larger than one directory. @@ -150,46 +151,53 @@ def _NormalizedSource(source): Returns: The normalized path. """ - normalized = os.path.normpath(source) - if source.count('$') == normalized.count('$'): - source = normalized - return source + normalized = os.path.normpath(source) + if source.count("$") == normalized.count("$"): + source = normalized + return source def _FixPath(path): - """Convert paths to a form that will make sense in a vcproj file. + """Convert paths to a form that will make sense in a vcproj file. Arguments: path: The path to convert, may contain / etc. Returns: The path with all slashes made into backslashes. """ - if fixpath_prefix and path and not os.path.isabs(path) and not path[0] == '$' and not _IsWindowsAbsPath(path): - path = os.path.join(fixpath_prefix, path) - path = path.replace('/', '\\') - path = _NormalizedSource(path) - if path and path[-1] == '\\': - path = path[:-1] - return path + if ( + fixpath_prefix + and path + and not os.path.isabs(path) + and not path[0] == "$" + and not _IsWindowsAbsPath(path) + ): + path = os.path.join(fixpath_prefix, path) + path = path.replace("/", "\\") + path = _NormalizedSource(path) + if path and path[-1] == "\\": + path = path[:-1] + return path def _IsWindowsAbsPath(path): - """ + """ On Cygwin systems Python needs a little help determining if a path is an absolute Windows path or not, so that it does not treat those as relative, which results in bad paths like: - '..\C:\\some_source_code_file.cc' + '..\\C:\\\\some_source_code_file.cc' """ - return path.startswith('c:') or path.startswith('C:') + return path.startswith("c:") or path.startswith("C:") def _FixPaths(paths): - """Fix each of the paths of the list.""" - return [_FixPath(i) for i in paths] + """Fix each of the paths of the list.""" + return [_FixPath(i) for i in paths] -def _ConvertSourcesToFilterHierarchy(sources, prefix=None, excluded=None, - list_excluded=True, msvs_version=None): - """Converts a list split source file paths into a vcproj folder hierarchy. +def _ConvertSourcesToFilterHierarchy( + sources, prefix=None, excluded=None, list_excluded=True, msvs_version=None +): + """Converts a list split source file paths into a vcproj folder hierarchy. Arguments: sources: A list of source file paths split. @@ -207,221 +215,248 @@ def _ConvertSourcesToFilterHierarchy(sources, prefix=None, excluded=None, [MSVSProject.Filter('a', contents=['joe\\a\\bob1.c']), MSVSProject.Filter('b', contents=['joe\\b\\bob2.c'])] """ - if not prefix: prefix = [] - result = [] - excluded_result = [] - folders = OrderedDict() - # Gather files into the final result, excluded, or folders. - for s in sources: - if len(s) == 1: - filename = _NormalizedSource('\\'.join(prefix + s)) - if filename in excluded: - excluded_result.append(filename) - else: - result.append(filename) - elif msvs_version and not msvs_version.UsesVcxproj(): - # For MSVS 2008 and earlier, we need to process all files before walking - # the sub folders. - if not folders.get(s[0]): - folders[s[0]] = [] - folders[s[0]].append(s[1:]) - else: - contents = _ConvertSourcesToFilterHierarchy([s[1:]], prefix + [s[0]], - excluded=excluded, - list_excluded=list_excluded, - msvs_version=msvs_version) - contents = MSVSProject.Filter(s[0], contents=contents) - result.append(contents) - # Add a folder for excluded files. - if excluded_result and list_excluded: - excluded_folder = MSVSProject.Filter('_excluded_files', - contents=excluded_result) - result.append(excluded_folder) - - if msvs_version and msvs_version.UsesVcxproj(): + if not prefix: + prefix = [] + result = [] + excluded_result = [] + folders = OrderedDict() + # Gather files into the final result, excluded, or folders. + for s in sources: + if len(s) == 1: + filename = _NormalizedSource("\\".join(prefix + s)) + if filename in excluded: + excluded_result.append(filename) + else: + result.append(filename) + elif msvs_version and not msvs_version.UsesVcxproj(): + # For MSVS 2008 and earlier, we need to process all files before walking + # the sub folders. + if not folders.get(s[0]): + folders[s[0]] = [] + folders[s[0]].append(s[1:]) + else: + contents = _ConvertSourcesToFilterHierarchy( + [s[1:]], + prefix + [s[0]], + excluded=excluded, + list_excluded=list_excluded, + msvs_version=msvs_version, + ) + contents = MSVSProject.Filter(s[0], contents=contents) + result.append(contents) + # Add a folder for excluded files. + if excluded_result and list_excluded: + excluded_folder = MSVSProject.Filter( + "_excluded_files", contents=excluded_result + ) + result.append(excluded_folder) + + if msvs_version and msvs_version.UsesVcxproj(): + return result + + # Populate all the folders. + for f in folders: + contents = _ConvertSourcesToFilterHierarchy( + folders[f], + prefix=prefix + [f], + excluded=excluded, + list_excluded=list_excluded, + msvs_version=msvs_version, + ) + contents = MSVSProject.Filter(f, contents=contents) + result.append(contents) return result - # Populate all the folders. - for f in folders: - contents = _ConvertSourcesToFilterHierarchy(folders[f], prefix=prefix + [f], - excluded=excluded, - list_excluded=list_excluded, - msvs_version=msvs_version) - contents = MSVSProject.Filter(f, contents=contents) - result.append(contents) - return result - def _ToolAppend(tools, tool_name, setting, value, only_if_unset=False): - if not value: return - _ToolSetOrAppend(tools, tool_name, setting, value, only_if_unset) + if not value: + return + _ToolSetOrAppend(tools, tool_name, setting, value, only_if_unset) def _ToolSetOrAppend(tools, tool_name, setting, value, only_if_unset=False): - # TODO(bradnelson): ugly hack, fix this more generally!!! - if 'Directories' in setting or 'Dependencies' in setting: - if type(value) == str: - value = value.replace('/', '\\') - else: - value = [i.replace('/', '\\') for i in value] - if not tools.get(tool_name): - tools[tool_name] = dict() - tool = tools[tool_name] - if 'CompileAsWinRT' == setting: - return - if tool.get(setting): - if only_if_unset: return - if type(tool[setting]) == list and type(value) == list: - tool[setting] += value + # TODO(bradnelson): ugly hack, fix this more generally!!! + if "Directories" in setting or "Dependencies" in setting: + if type(value) == str: + value = value.replace("/", "\\") + else: + value = [i.replace("/", "\\") for i in value] + if not tools.get(tool_name): + tools[tool_name] = dict() + tool = tools[tool_name] + if "CompileAsWinRT" == setting: + return + if tool.get(setting): + if only_if_unset: + return + if type(tool[setting]) == list and type(value) == list: + tool[setting] += value + else: + raise TypeError( + 'Appending "%s" to a non-list setting "%s" for tool "%s" is ' + "not allowed, previous value: %s" + % (value, setting, tool_name, str(tool[setting])) + ) else: - raise TypeError( - 'Appending "%s" to a non-list setting "%s" for tool "%s" is ' - 'not allowed, previous value: %s' % ( - value, setting, tool_name, str(tool[setting]))) - else: - tool[setting] = value + tool[setting] = value def _ConfigTargetVersion(config_data): - return config_data.get('msvs_target_version', 'Windows7') + return config_data.get("msvs_target_version", "Windows7") def _ConfigPlatform(config_data): - return config_data.get('msvs_configuration_platform', 'Win32') + return config_data.get("msvs_configuration_platform", "Win32") def _ConfigBaseName(config_name, platform_name): - if config_name.endswith('_' + platform_name): - return config_name[0:-len(platform_name) - 1] - else: - return config_name + if config_name.endswith("_" + platform_name): + return config_name[0 : -len(platform_name) - 1] + else: + return config_name def _ConfigFullName(config_name, config_data): - platform_name = _ConfigPlatform(config_data) - return '%s|%s' % (_ConfigBaseName(config_name, platform_name), platform_name) + platform_name = _ConfigPlatform(config_data) + return "%s|%s" % (_ConfigBaseName(config_name, platform_name), platform_name) def _ConfigWindowsTargetPlatformVersion(config_data, version): - target_ver = config_data.get('msvs_windows_target_platform_version') - if target_ver and re.match(r'^\d+', target_ver): - return target_ver - config_ver = config_data.get('msvs_windows_sdk_version') - vers = [config_ver] if config_ver else version.compatible_sdks - for ver in vers: - for key in [ - r'HKLM\Software\Microsoft\Microsoft SDKs\Windows\%s', - r'HKLM\Software\Wow6432Node\Microsoft\Microsoft SDKs\Windows\%s']: - sdk_dir = MSVSVersion._RegistryGetValue(key % ver, 'InstallationFolder') - if not sdk_dir: - continue - version = MSVSVersion._RegistryGetValue(key % ver, 'ProductVersion') or '' - # Find a matching entry in sdk_dir\include. - expected_sdk_dir=r'%s\include' % sdk_dir - names = sorted([x for x in (os.listdir(expected_sdk_dir) - if os.path.isdir(expected_sdk_dir) - else [] - ) - if x.startswith(version)], reverse=True) - if names: - return names[0] - else: - print('Warning: No include files found for detected ' - 'Windows SDK version %s' % (version), file=sys.stdout) - - -def _BuildCommandLineForRuleRaw(spec, cmd, cygwin_shell, has_input_path, - quote_cmd, do_setup_env): - - if [x for x in cmd if '$(InputDir)' in x]: - input_dir_preamble = ( - 'set INPUTDIR=$(InputDir)\n' - 'if NOT DEFINED INPUTDIR set INPUTDIR=.\\\n' - 'set INPUTDIR=%INPUTDIR:~0,-1%\n' - ) - else: - input_dir_preamble = '' - - if cygwin_shell: - # Find path to cygwin. - cygwin_dir = _FixPath(spec.get('msvs_cygwin_dirs', ['.'])[0]) - # Prepare command. - direct_cmd = cmd - direct_cmd = [i.replace('$(IntDir)', - '`cygpath -m "${INTDIR}"`') for i in direct_cmd] - direct_cmd = [i.replace('$(OutDir)', - '`cygpath -m "${OUTDIR}"`') for i in direct_cmd] - direct_cmd = [i.replace('$(InputDir)', - '`cygpath -m "${INPUTDIR}"`') for i in direct_cmd] - if has_input_path: - direct_cmd = [i.replace('$(InputPath)', - '`cygpath -m "${INPUTPATH}"`') - for i in direct_cmd] - direct_cmd = ['\\"%s\\"' % i.replace('"', '\\\\\\"') for i in direct_cmd] - # direct_cmd = gyp.common.EncodePOSIXShellList(direct_cmd) - direct_cmd = ' '.join(direct_cmd) - # TODO(quote): regularize quoting path names throughout the module - cmd = '' - if do_setup_env: - cmd += 'call "$(ProjectDir)%(cygwin_dir)s\\setup_env.bat" && ' - cmd += 'set CYGWIN=nontsec&& ' - if direct_cmd.find('NUMBER_OF_PROCESSORS') >= 0: - cmd += 'set /a NUMBER_OF_PROCESSORS_PLUS_1=%%NUMBER_OF_PROCESSORS%%+1&& ' - if direct_cmd.find('INTDIR') >= 0: - cmd += 'set INTDIR=$(IntDir)&& ' - if direct_cmd.find('OUTDIR') >= 0: - cmd += 'set OUTDIR=$(OutDir)&& ' - if has_input_path and direct_cmd.find('INPUTPATH') >= 0: - cmd += 'set INPUTPATH=$(InputPath) && ' - cmd += 'bash -c "%(cmd)s"' - cmd = cmd % {'cygwin_dir': cygwin_dir, - 'cmd': direct_cmd} - return input_dir_preamble + cmd - else: - # Convert cat --> type to mimic unix. - if cmd[0] == 'cat': - command = ['type'] + target_ver = config_data.get("msvs_windows_target_platform_version") + if target_ver and re.match(r"^\d+", target_ver): + return target_ver + config_ver = config_data.get("msvs_windows_sdk_version") + vers = [config_ver] if config_ver else version.compatible_sdks + for ver in vers: + for key in [ + r"HKLM\Software\Microsoft\Microsoft SDKs\Windows\%s", + r"HKLM\Software\Wow6432Node\Microsoft\Microsoft SDKs\Windows\%s", + ]: + sdk_dir = MSVSVersion._RegistryGetValue(key % ver, "InstallationFolder") + if not sdk_dir: + continue + version = MSVSVersion._RegistryGetValue(key % ver, "ProductVersion") or "" + # Find a matching entry in sdk_dir\include. + expected_sdk_dir = r"%s\include" % sdk_dir + names = sorted( + [ + x + for x in ( + os.listdir(expected_sdk_dir) + if os.path.isdir(expected_sdk_dir) + else [] + ) + if x.startswith(version) + ], + reverse=True, + ) + if names: + return names[0] + else: + print( + "Warning: No include files found for detected " + "Windows SDK version %s" % (version), + file=sys.stdout, + ) + + +def _BuildCommandLineForRuleRaw( + spec, cmd, cygwin_shell, has_input_path, quote_cmd, do_setup_env +): + + if [x for x in cmd if "$(InputDir)" in x]: + input_dir_preamble = ( + "set INPUTDIR=$(InputDir)\n" + "if NOT DEFINED INPUTDIR set INPUTDIR=.\\\n" + "set INPUTDIR=%INPUTDIR:~0,-1%\n" + ) + else: + input_dir_preamble = "" + + if cygwin_shell: + # Find path to cygwin. + cygwin_dir = _FixPath(spec.get("msvs_cygwin_dirs", ["."])[0]) + # Prepare command. + direct_cmd = cmd + direct_cmd = [ + i.replace("$(IntDir)", '`cygpath -m "${INTDIR}"`') for i in direct_cmd + ] + direct_cmd = [ + i.replace("$(OutDir)", '`cygpath -m "${OUTDIR}"`') for i in direct_cmd + ] + direct_cmd = [ + i.replace("$(InputDir)", '`cygpath -m "${INPUTDIR}"`') for i in direct_cmd + ] + if has_input_path: + direct_cmd = [ + i.replace("$(InputPath)", '`cygpath -m "${INPUTPATH}"`') + for i in direct_cmd + ] + direct_cmd = ['\\"%s\\"' % i.replace('"', '\\\\\\"') for i in direct_cmd] + # direct_cmd = gyp.common.EncodePOSIXShellList(direct_cmd) + direct_cmd = " ".join(direct_cmd) + # TODO(quote): regularize quoting path names throughout the module + cmd = "" + if do_setup_env: + cmd += 'call "$(ProjectDir)%(cygwin_dir)s\\setup_env.bat" && ' + cmd += "set CYGWIN=nontsec&& " + if direct_cmd.find("NUMBER_OF_PROCESSORS") >= 0: + cmd += "set /a NUMBER_OF_PROCESSORS_PLUS_1=%%NUMBER_OF_PROCESSORS%%+1&& " + if direct_cmd.find("INTDIR") >= 0: + cmd += "set INTDIR=$(IntDir)&& " + if direct_cmd.find("OUTDIR") >= 0: + cmd += "set OUTDIR=$(OutDir)&& " + if has_input_path and direct_cmd.find("INPUTPATH") >= 0: + cmd += "set INPUTPATH=$(InputPath) && " + cmd += 'bash -c "%(cmd)s"' + cmd = cmd % {"cygwin_dir": cygwin_dir, "cmd": direct_cmd} + return input_dir_preamble + cmd else: - command = [cmd[0].replace('/', '\\')] - # Add call before command to ensure that commands can be tied together one - # after the other without aborting in Incredibuild, since IB makes a bat - # file out of the raw command string, and some commands (like python) are - # actually batch files themselves. - command.insert(0, 'call') - # Fix the paths - # TODO(quote): This is a really ugly heuristic, and will miss path fixing - # for arguments like "--arg=path" or "/opt:path". - # If the argument starts with a slash or dash, it's probably a command line - # switch - arguments = [i if (i[:1] in "/-") else _FixPath(i) for i in cmd[1:]] - arguments = [i.replace('$(InputDir)', '%INPUTDIR%') for i in arguments] - arguments = [MSVSSettings.FixVCMacroSlashes(i) for i in arguments] - if quote_cmd: - # Support a mode for using cmd directly. - # Convert any paths to native form (first element is used directly). - # TODO(quote): regularize quoting path names throughout the module - arguments = ['"%s"' % i for i in arguments] - # Collapse into a single command. - return input_dir_preamble + ' '.join(command + arguments) + # Convert cat --> type to mimic unix. + if cmd[0] == "cat": + command = ["type"] + else: + command = [cmd[0].replace("/", "\\")] + # Add call before command to ensure that commands can be tied together one + # after the other without aborting in Incredibuild, since IB makes a bat + # file out of the raw command string, and some commands (like python) are + # actually batch files themselves. + command.insert(0, "call") + # Fix the paths + # TODO(quote): This is a really ugly heuristic, and will miss path fixing + # for arguments like "--arg=path" or "/opt:path". + # If the argument starts with a slash or dash, it's probably a command line + # switch + arguments = [i if (i[:1] in "/-") else _FixPath(i) for i in cmd[1:]] + arguments = [i.replace("$(InputDir)", "%INPUTDIR%") for i in arguments] + arguments = [MSVSSettings.FixVCMacroSlashes(i) for i in arguments] + if quote_cmd: + # Support a mode for using cmd directly. + # Convert any paths to native form (first element is used directly). + # TODO(quote): regularize quoting path names throughout the module + arguments = ['"%s"' % i for i in arguments] + # Collapse into a single command. + return input_dir_preamble + " ".join(command + arguments) def _BuildCommandLineForRule(spec, rule, has_input_path, do_setup_env): - # Currently this weird argument munging is used to duplicate the way a - # python script would need to be run as part of the chrome tree. - # Eventually we should add some sort of rule_default option to set this - # per project. For now the behavior chrome needs is the default. - mcs = rule.get('msvs_cygwin_shell') - if mcs is None: - mcs = int(spec.get('msvs_cygwin_shell', 1)) - elif isinstance(mcs, str): - mcs = int(mcs) - quote_cmd = int(rule.get('msvs_quote_cmd', 1)) - return _BuildCommandLineForRuleRaw(spec, rule['action'], mcs, has_input_path, - quote_cmd, do_setup_env=do_setup_env) + # Currently this weird argument munging is used to duplicate the way a + # python script would need to be run as part of the chrome tree. + # Eventually we should add some sort of rule_default option to set this + # per project. For now the behavior chrome needs is the default. + mcs = rule.get("msvs_cygwin_shell") + if mcs is None: + mcs = int(spec.get("msvs_cygwin_shell", 1)) + elif isinstance(mcs, str): + mcs = int(mcs) + quote_cmd = int(rule.get("msvs_quote_cmd", 1)) + return _BuildCommandLineForRuleRaw( + spec, rule["action"], mcs, has_input_path, quote_cmd, do_setup_env=do_setup_env + ) def _AddActionStep(actions_dict, inputs, outputs, description, command): - """Merge action into an existing list of actions. + """Merge action into an existing list of actions. Care must be taken so that actions which have overlapping inputs either don't get assigned to the same input, or get collapsed into one. @@ -434,30 +469,31 @@ def _AddActionStep(actions_dict, inputs, outputs, description, command): description: description of the action command: command line to execute """ - # Require there to be at least one input (call sites will ensure this). - assert inputs - - action = { - 'inputs': inputs, - 'outputs': outputs, - 'description': description, - 'command': command, - } + # Require there to be at least one input (call sites will ensure this). + assert inputs + + action = { + "inputs": inputs, + "outputs": outputs, + "description": description, + "command": command, + } - # Pick where to stick this action. - # While less than optimal in terms of build time, attach them to the first - # input for now. - chosen_input = inputs[0] + # Pick where to stick this action. + # While less than optimal in terms of build time, attach them to the first + # input for now. + chosen_input = inputs[0] - # Add it there. - if chosen_input not in actions_dict: - actions_dict[chosen_input] = [] - actions_dict[chosen_input].append(action) + # Add it there. + if chosen_input not in actions_dict: + actions_dict[chosen_input] = [] + actions_dict[chosen_input].append(action) -def _AddCustomBuildToolForMSVS(p, spec, primary_input, - inputs, outputs, description, cmd): - """Add a custom build tool to execute something. +def _AddCustomBuildToolForMSVS( + p, spec, primary_input, inputs, outputs, description, cmd +): + """Add a custom build tool to execute something. Arguments: p: the target project @@ -468,23 +504,26 @@ def _AddCustomBuildToolForMSVS(p, spec, primary_input, description: description of the action cmd: command line to execute """ - inputs = _FixPaths(inputs) - outputs = _FixPaths(outputs) - tool = MSVSProject.Tool( - 'VCCustomBuildTool', - {'Description': description, - 'AdditionalDependencies': ';'.join(inputs), - 'Outputs': ';'.join(outputs), - 'CommandLine': cmd, - }) - # Add to the properties of primary input for each config. - for config_name, c_data in spec['configurations'].items(): - p.AddFileConfig(_FixPath(primary_input), - _ConfigFullName(config_name, c_data), tools=[tool]) + inputs = _FixPaths(inputs) + outputs = _FixPaths(outputs) + tool = MSVSProject.Tool( + "VCCustomBuildTool", + { + "Description": description, + "AdditionalDependencies": ";".join(inputs), + "Outputs": ";".join(outputs), + "CommandLine": cmd, + }, + ) + # Add to the properties of primary input for each config. + for config_name, c_data in spec["configurations"].items(): + p.AddFileConfig( + _FixPath(primary_input), _ConfigFullName(config_name, c_data), tools=[tool] + ) def _AddAccumulatedActionsToMSVS(p, spec, actions_dict): - """Add actions accumulated into an actions_dict, merging as needed. + """Add actions accumulated into an actions_dict, merging as needed. Arguments: p: the target project @@ -492,29 +531,32 @@ def _AddAccumulatedActionsToMSVS(p, spec, actions_dict): actions_dict: dictionary keyed on input name, which maps to a list of dicts describing the actions attached to that input file. """ - for primary_input in actions_dict: - inputs = OrderedSet() - outputs = OrderedSet() - descriptions = [] - commands = [] - for action in actions_dict[primary_input]: - inputs.update(OrderedSet(action['inputs'])) - outputs.update(OrderedSet(action['outputs'])) - descriptions.append(action['description']) - commands.append(action['command']) - # Add the custom build step for one input file. - description = ', and also '.join(descriptions) - command = '\r\n'.join(commands) - _AddCustomBuildToolForMSVS(p, spec, - primary_input=primary_input, - inputs=inputs, - outputs=outputs, - description=description, - cmd=command) + for primary_input in actions_dict: + inputs = OrderedSet() + outputs = OrderedSet() + descriptions = [] + commands = [] + for action in actions_dict[primary_input]: + inputs.update(OrderedSet(action["inputs"])) + outputs.update(OrderedSet(action["outputs"])) + descriptions.append(action["description"]) + commands.append(action["command"]) + # Add the custom build step for one input file. + description = ", and also ".join(descriptions) + command = "\r\n".join(commands) + _AddCustomBuildToolForMSVS( + p, + spec, + primary_input=primary_input, + inputs=inputs, + outputs=outputs, + description=description, + cmd=command, + ) def _RuleExpandPath(path, input_file): - """Given the input file to which a rule applied, string substitute a path. + """Given the input file to which a rule applied, string substitute a path. Arguments: path: a path to string expand @@ -522,18 +564,20 @@ def _RuleExpandPath(path, input_file): Returns: The string substituted path. """ - path = path.replace('$(InputName)', - os.path.splitext(os.path.split(input_file)[1])[0]) - path = path.replace('$(InputDir)', os.path.dirname(input_file)) - path = path.replace('$(InputExt)', - os.path.splitext(os.path.split(input_file)[1])[1]) - path = path.replace('$(InputFileName)', os.path.split(input_file)[1]) - path = path.replace('$(InputPath)', input_file) - return path + path = path.replace( + "$(InputName)", os.path.splitext(os.path.split(input_file)[1])[0] + ) + path = path.replace("$(InputDir)", os.path.dirname(input_file)) + path = path.replace( + "$(InputExt)", os.path.splitext(os.path.split(input_file)[1])[1] + ) + path = path.replace("$(InputFileName)", os.path.split(input_file)[1]) + path = path.replace("$(InputPath)", input_file) + return path def _FindRuleTriggerFiles(rule, sources): - """Find the list of files which a particular rule applies to. + """Find the list of files which a particular rule applies to. Arguments: rule: the rule in question @@ -541,11 +585,11 @@ def _FindRuleTriggerFiles(rule, sources): Returns: The list of sources that trigger a particular rule. """ - return rule.get('rule_sources', []) + return rule.get("rule_sources", []) def _RuleInputsAndOutputs(rule, trigger_file): - """Find the inputs and outputs generated by a rule. + """Find the inputs and outputs generated by a rule. Arguments: rule: the rule in question. @@ -553,20 +597,20 @@ def _RuleInputsAndOutputs(rule, trigger_file): Returns: The pair of (inputs, outputs) involved in this rule. """ - raw_inputs = _FixPaths(rule.get('inputs', [])) - raw_outputs = _FixPaths(rule.get('outputs', [])) - inputs = OrderedSet() - outputs = OrderedSet() - inputs.add(trigger_file) - for i in raw_inputs: - inputs.add(_RuleExpandPath(i, trigger_file)) - for o in raw_outputs: - outputs.add(_RuleExpandPath(o, trigger_file)) - return (inputs, outputs) + raw_inputs = _FixPaths(rule.get("inputs", [])) + raw_outputs = _FixPaths(rule.get("outputs", [])) + inputs = OrderedSet() + outputs = OrderedSet() + inputs.add(trigger_file) + for i in raw_inputs: + inputs.add(_RuleExpandPath(i, trigger_file)) + for o in raw_outputs: + outputs.add(_RuleExpandPath(o, trigger_file)) + return (inputs, outputs) def _GenerateNativeRulesForMSVS(p, rules, output_dir, spec, options): - """Generate a native rules file. + """Generate a native rules file. Arguments: p: the target project @@ -575,43 +619,43 @@ def _GenerateNativeRulesForMSVS(p, rules, output_dir, spec, options): spec: the project dict options: global generator options """ - rules_filename = '%s%s.rules' % (spec['target_name'], - options.suffix) - rules_file = MSVSToolFile.Writer(os.path.join(output_dir, rules_filename), - spec['target_name']) - # Add each rule. - for r in rules: - rule_name = r['rule_name'] - rule_ext = r['extension'] - inputs = _FixPaths(r.get('inputs', [])) - outputs = _FixPaths(r.get('outputs', [])) - # Skip a rule with no action and no inputs. - if 'action' not in r and not r.get('rule_sources', []): - continue - cmd = _BuildCommandLineForRule(spec, r, has_input_path=True, - do_setup_env=True) - rules_file.AddCustomBuildRule(name=rule_name, - description=r.get('message', rule_name), - extensions=[rule_ext], - additional_dependencies=inputs, - outputs=outputs, - cmd=cmd) - # Write out rules file. - rules_file.WriteIfChanged() - - # Add rules file to project. - p.AddToolFile(rules_filename) + rules_filename = "%s%s.rules" % (spec["target_name"], options.suffix) + rules_file = MSVSToolFile.Writer( + os.path.join(output_dir, rules_filename), spec["target_name"] + ) + # Add each rule. + for r in rules: + rule_name = r["rule_name"] + rule_ext = r["extension"] + inputs = _FixPaths(r.get("inputs", [])) + outputs = _FixPaths(r.get("outputs", [])) + # Skip a rule with no action and no inputs. + if "action" not in r and not r.get("rule_sources", []): + continue + cmd = _BuildCommandLineForRule(spec, r, has_input_path=True, do_setup_env=True) + rules_file.AddCustomBuildRule( + name=rule_name, + description=r.get("message", rule_name), + extensions=[rule_ext], + additional_dependencies=inputs, + outputs=outputs, + cmd=cmd, + ) + # Write out rules file. + rules_file.WriteIfChanged() + + # Add rules file to project. + p.AddToolFile(rules_filename) def _Cygwinify(path): - path = path.replace('$(OutDir)', '$(OutDirCygwin)') - path = path.replace('$(IntDir)', '$(IntDirCygwin)') - return path + path = path.replace("$(OutDir)", "$(OutDirCygwin)") + path = path.replace("$(IntDir)", "$(IntDirCygwin)") + return path -def _GenerateExternalRules(rules, output_dir, spec, - sources, options, actions_to_add): - """Generate an external makefile to do a set of rules. +def _GenerateExternalRules(rules, output_dir, spec, sources, options, actions_to_add): + """Generate an external makefile to do a set of rules. Arguments: rules: the list of rules to include @@ -621,77 +665,82 @@ def _GenerateExternalRules(rules, output_dir, spec, options: global generator options actions_to_add: The list of actions we will add to. """ - filename = '%s_rules%s.mk' % (spec['target_name'], options.suffix) - mk_file = gyp.common.WriteOnDiff(os.path.join(output_dir, filename)) - # Find cygwin style versions of some paths. - mk_file.write('OutDirCygwin:=$(shell cygpath -u "$(OutDir)")\n') - mk_file.write('IntDirCygwin:=$(shell cygpath -u "$(IntDir)")\n') - # Gather stuff needed to emit all: target. - all_inputs = OrderedSet() - all_outputs = OrderedSet() - all_output_dirs = OrderedSet() - first_outputs = [] - for rule in rules: - trigger_files = _FindRuleTriggerFiles(rule, sources) - for tf in trigger_files: - inputs, outputs = _RuleInputsAndOutputs(rule, tf) - all_inputs.update(OrderedSet(inputs)) - all_outputs.update(OrderedSet(outputs)) - # Only use one target from each rule as the dependency for - # 'all' so we don't try to build each rule multiple times. - first_outputs.append(list(outputs)[0]) - # Get the unique output directories for this rule. - output_dirs = [os.path.split(i)[0] for i in outputs] - for od in output_dirs: - all_output_dirs.add(od) - first_outputs_cyg = [_Cygwinify(i) for i in first_outputs] - # Write out all: target, including mkdir for each output directory. - mk_file.write('all: %s\n' % ' '.join(first_outputs_cyg)) - for od in all_output_dirs: - if od: - mk_file.write('\tmkdir -p `cygpath -u "%s"`\n' % od) - mk_file.write('\n') - # Define how each output is generated. - for rule in rules: - trigger_files = _FindRuleTriggerFiles(rule, sources) - for tf in trigger_files: - # Get all the inputs and outputs for this rule for this trigger file. - inputs, outputs = _RuleInputsAndOutputs(rule, tf) - inputs = [_Cygwinify(i) for i in inputs] - outputs = [_Cygwinify(i) for i in outputs] - # Prepare the command line for this rule. - cmd = [_RuleExpandPath(c, tf) for c in rule['action']] - cmd = ['"%s"' % i for i in cmd] - cmd = ' '.join(cmd) - # Add it to the makefile. - mk_file.write('%s: %s\n' % (' '.join(outputs), ' '.join(inputs))) - mk_file.write('\t%s\n\n' % cmd) - # Close up the file. - mk_file.close() - - # Add makefile to list of sources. - sources.add(filename) - # Add a build action to call makefile. - cmd = ['make', - 'OutDir=$(OutDir)', - 'IntDir=$(IntDir)', - '-j', '${NUMBER_OF_PROCESSORS_PLUS_1}', - '-f', filename] - cmd = _BuildCommandLineForRuleRaw(spec, cmd, True, False, True, True) - # Insert makefile as 0'th input, so it gets the action attached there, - # as this is easier to understand from in the IDE. - all_inputs = list(all_inputs) - all_inputs.insert(0, filename) - _AddActionStep(actions_to_add, - inputs=_FixPaths(all_inputs), - outputs=_FixPaths(all_outputs), - description='Running external rules for %s' % - spec['target_name'], - command=cmd) + filename = "%s_rules%s.mk" % (spec["target_name"], options.suffix) + mk_file = gyp.common.WriteOnDiff(os.path.join(output_dir, filename)) + # Find cygwin style versions of some paths. + mk_file.write('OutDirCygwin:=$(shell cygpath -u "$(OutDir)")\n') + mk_file.write('IntDirCygwin:=$(shell cygpath -u "$(IntDir)")\n') + # Gather stuff needed to emit all: target. + all_inputs = OrderedSet() + all_outputs = OrderedSet() + all_output_dirs = OrderedSet() + first_outputs = [] + for rule in rules: + trigger_files = _FindRuleTriggerFiles(rule, sources) + for tf in trigger_files: + inputs, outputs = _RuleInputsAndOutputs(rule, tf) + all_inputs.update(OrderedSet(inputs)) + all_outputs.update(OrderedSet(outputs)) + # Only use one target from each rule as the dependency for + # 'all' so we don't try to build each rule multiple times. + first_outputs.append(list(outputs)[0]) + # Get the unique output directories for this rule. + output_dirs = [os.path.split(i)[0] for i in outputs] + for od in output_dirs: + all_output_dirs.add(od) + first_outputs_cyg = [_Cygwinify(i) for i in first_outputs] + # Write out all: target, including mkdir for each output directory. + mk_file.write("all: %s\n" % " ".join(first_outputs_cyg)) + for od in all_output_dirs: + if od: + mk_file.write('\tmkdir -p `cygpath -u "%s"`\n' % od) + mk_file.write("\n") + # Define how each output is generated. + for rule in rules: + trigger_files = _FindRuleTriggerFiles(rule, sources) + for tf in trigger_files: + # Get all the inputs and outputs for this rule for this trigger file. + inputs, outputs = _RuleInputsAndOutputs(rule, tf) + inputs = [_Cygwinify(i) for i in inputs] + outputs = [_Cygwinify(i) for i in outputs] + # Prepare the command line for this rule. + cmd = [_RuleExpandPath(c, tf) for c in rule["action"]] + cmd = ['"%s"' % i for i in cmd] + cmd = " ".join(cmd) + # Add it to the makefile. + mk_file.write("%s: %s\n" % (" ".join(outputs), " ".join(inputs))) + mk_file.write("\t%s\n\n" % cmd) + # Close up the file. + mk_file.close() + + # Add makefile to list of sources. + sources.add(filename) + # Add a build action to call makefile. + cmd = [ + "make", + "OutDir=$(OutDir)", + "IntDir=$(IntDir)", + "-j", + "${NUMBER_OF_PROCESSORS_PLUS_1}", + "-f", + filename, + ] + cmd = _BuildCommandLineForRuleRaw(spec, cmd, True, False, True, True) + # Insert makefile as 0'th input, so it gets the action attached there, + # as this is easier to understand from in the IDE. + all_inputs = list(all_inputs) + all_inputs.insert(0, filename) + _AddActionStep( + actions_to_add, + inputs=_FixPaths(all_inputs), + outputs=_FixPaths(all_outputs), + description="Running external rules for %s" % spec["target_name"], + command=cmd, + ) def _EscapeEnvironmentVariableExpansion(s): - """Escapes % characters. + """Escapes % characters. Escapes any % characters so that Windows-style environment variable expansions will leave them alone. @@ -704,15 +753,15 @@ def _EscapeEnvironmentVariableExpansion(s): Returns: The escaped string. """ - s = s.replace('%', '%%') - return s + s = s.replace("%", "%%") + return s quote_replacer_regex = re.compile(r'(\\*)"') def _EscapeCommandLineArgumentForMSVS(s): - """Escapes a Windows command-line argument. + """Escapes a Windows command-line argument. So that the Win32 CommandLineToArgv function will turn the escaped result back into the original string. @@ -726,24 +775,24 @@ def _EscapeCommandLineArgumentForMSVS(s): the escaped string. """ - def _Replace(match): - # For a literal quote, CommandLineToArgv requires an odd number of - # backslashes preceding it, and it produces half as many literal backslashes - # (rounded down). So we need to produce 2n+1 backslashes. - return 2 * match.group(1) + '\\"' + def _Replace(match): + # For a literal quote, CommandLineToArgv requires an odd number of + # backslashes preceding it, and it produces half as many literal backslashes + # (rounded down). So we need to produce 2n+1 backslashes. + return 2 * match.group(1) + '\\"' - # Escape all quotes so that they are interpreted literally. - s = quote_replacer_regex.sub(_Replace, s) - # Now add unescaped quotes so that any whitespace is interpreted literally. - s = '"' + s + '"' - return s + # Escape all quotes so that they are interpreted literally. + s = quote_replacer_regex.sub(_Replace, s) + # Now add unescaped quotes so that any whitespace is interpreted literally. + s = '"' + s + '"' + return s -delimiters_replacer_regex = re.compile(r'(\\*)([,;]+)') +delimiters_replacer_regex = re.compile(r"(\\*)([,;]+)") def _EscapeVCProjCommandLineArgListItem(s): - """Escapes command line arguments for MSVS. + """Escapes command line arguments for MSVS. The VCProj format stores string lists in a single string using commas and semi-colons as separators, which must be quoted if they are to be @@ -764,85 +813,87 @@ def _EscapeVCProjCommandLineArgListItem(s): the escaped string. """ - def _Replace(match): - # For a non-literal quote, CommandLineToArgv requires an even number of - # backslashes preceding it, and it produces half as many literal - # backslashes. So we need to produce 2n backslashes. - return 2 * match.group(1) + '"' + match.group(2) + '"' - - segments = s.split('"') - # The unquoted segments are at the even-numbered indices. - for i in range(0, len(segments), 2): - segments[i] = delimiters_replacer_regex.sub(_Replace, segments[i]) - # Concatenate back into a single string - s = '"'.join(segments) - if len(segments) % 2 == 0: - # String ends while still quoted according to VCProj's convention. This - # means the delimiter and the next list item that follow this one in the - # .vcproj file will be misinterpreted as part of this item. There is nothing - # we can do about this. Adding an extra quote would correct the problem in - # the VCProj but cause the same problem on the final command-line. Moving - # the item to the end of the list does works, but that's only possible if - # there's only one such item. Let's just warn the user. - print('Warning: MSVS may misinterpret the odd number of ' + - 'quotes in ' + s, file=sys.stderr) - return s + def _Replace(match): + # For a non-literal quote, CommandLineToArgv requires an even number of + # backslashes preceding it, and it produces half as many literal + # backslashes. So we need to produce 2n backslashes. + return 2 * match.group(1) + '"' + match.group(2) + '"' + + segments = s.split('"') + # The unquoted segments are at the even-numbered indices. + for i in range(0, len(segments), 2): + segments[i] = delimiters_replacer_regex.sub(_Replace, segments[i]) + # Concatenate back into a single string + s = '"'.join(segments) + if len(segments) % 2 == 0: + # String ends while still quoted according to VCProj's convention. This + # means the delimiter and the next list item that follow this one in the + # .vcproj file will be misinterpreted as part of this item. There is nothing + # we can do about this. Adding an extra quote would correct the problem in + # the VCProj but cause the same problem on the final command-line. Moving + # the item to the end of the list does works, but that's only possible if + # there's only one such item. Let's just warn the user. + print( + "Warning: MSVS may misinterpret the odd number of " + "quotes in " + s, + file=sys.stderr, + ) + return s def _EscapeCppDefineForMSVS(s): - """Escapes a CPP define so that it will reach the compiler unaltered.""" - s = _EscapeEnvironmentVariableExpansion(s) - s = _EscapeCommandLineArgumentForMSVS(s) - s = _EscapeVCProjCommandLineArgListItem(s) - # cl.exe replaces literal # characters with = in preprocesor definitions for - # some reason. Octal-encode to work around that. - s = s.replace('#', '\\%03o' % ord('#')) - return s + """Escapes a CPP define so that it will reach the compiler unaltered.""" + s = _EscapeEnvironmentVariableExpansion(s) + s = _EscapeCommandLineArgumentForMSVS(s) + s = _EscapeVCProjCommandLineArgListItem(s) + # cl.exe replaces literal # characters with = in preprocesor definitions for + # some reason. Octal-encode to work around that. + s = s.replace("#", "\\%03o" % ord("#")) + return s quote_replacer_regex2 = re.compile(r'(\\+)"') def _EscapeCommandLineArgumentForMSBuild(s): - """Escapes a Windows command-line argument for use by MSBuild.""" + """Escapes a Windows command-line argument for use by MSBuild.""" - def _Replace(match): - return (len(match.group(1)) / 2 * 4) * '\\' + '\\"' + def _Replace(match): + return (len(match.group(1)) / 2 * 4) * "\\" + '\\"' - # Escape all quotes so that they are interpreted literally. - s = quote_replacer_regex2.sub(_Replace, s) - return s + # Escape all quotes so that they are interpreted literally. + s = quote_replacer_regex2.sub(_Replace, s) + return s def _EscapeMSBuildSpecialCharacters(s): - escape_dictionary = { - '%': '%25', - '$': '%24', - '@': '%40', - "'": '%27', - ';': '%3B', - '?': '%3F', - '*': '%2A' - } - result = ''.join([escape_dictionary.get(c, c) for c in s]) - return result + escape_dictionary = { + "%": "%25", + "$": "%24", + "@": "%40", + "'": "%27", + ";": "%3B", + "?": "%3F", + "*": "%2A", + } + result = "".join([escape_dictionary.get(c, c) for c in s]) + return result def _EscapeCppDefineForMSBuild(s): - """Escapes a CPP define so that it will reach the compiler unaltered.""" - s = _EscapeEnvironmentVariableExpansion(s) - s = _EscapeCommandLineArgumentForMSBuild(s) - s = _EscapeMSBuildSpecialCharacters(s) - # cl.exe replaces literal # characters with = in preprocesor definitions for - # some reason. Octal-encode to work around that. - s = s.replace('#', '\\%03o' % ord('#')) - return s + """Escapes a CPP define so that it will reach the compiler unaltered.""" + s = _EscapeEnvironmentVariableExpansion(s) + s = _EscapeCommandLineArgumentForMSBuild(s) + s = _EscapeMSBuildSpecialCharacters(s) + # cl.exe replaces literal # characters with = in preprocesor definitions for + # some reason. Octal-encode to work around that. + s = s.replace("#", "\\%03o" % ord("#")) + return s -def _GenerateRulesForMSVS(p, output_dir, options, spec, - sources, excluded_sources, - actions_to_add): - """Generate all the rules for a particular project. +def _GenerateRulesForMSVS( + p, output_dir, options, spec, sources, excluded_sources, actions_to_add +): + """Generate all the rules for a particular project. Arguments: p: the project @@ -853,45 +904,46 @@ def _GenerateRulesForMSVS(p, output_dir, options, spec, excluded_sources: the set of sources excluded from normal processing actions_to_add: deferred list of actions to add in """ - rules = spec.get('rules', []) - rules_native = [r for r in rules if not int(r.get('msvs_external_rule', 0))] - rules_external = [r for r in rules if int(r.get('msvs_external_rule', 0))] + rules = spec.get("rules", []) + rules_native = [r for r in rules if not int(r.get("msvs_external_rule", 0))] + rules_external = [r for r in rules if int(r.get("msvs_external_rule", 0))] - # Handle rules that use a native rules file. - if rules_native: - _GenerateNativeRulesForMSVS(p, rules_native, output_dir, spec, options) + # Handle rules that use a native rules file. + if rules_native: + _GenerateNativeRulesForMSVS(p, rules_native, output_dir, spec, options) - # Handle external rules (non-native rules). - if rules_external: - _GenerateExternalRules(rules_external, output_dir, spec, - sources, options, actions_to_add) - _AdjustSourcesForRules(rules, sources, excluded_sources, False) + # Handle external rules (non-native rules). + if rules_external: + _GenerateExternalRules( + rules_external, output_dir, spec, sources, options, actions_to_add + ) + _AdjustSourcesForRules(rules, sources, excluded_sources, False) def _AdjustSourcesForRules(rules, sources, excluded_sources, is_msbuild): - # Add outputs generated by each rule (if applicable). - for rule in rules: - # Add in the outputs from this rule. - trigger_files = _FindRuleTriggerFiles(rule, sources) - for trigger_file in trigger_files: - # Remove trigger_file from excluded_sources to let the rule be triggered - # (e.g. rule trigger ax_enums.idl is added to excluded_sources - # because it's also in an action's inputs in the same project) - excluded_sources.discard(_FixPath(trigger_file)) - # Done if not processing outputs as sources. - if int(rule.get('process_outputs_as_sources', False)): - inputs, outputs = _RuleInputsAndOutputs(rule, trigger_file) - inputs = OrderedSet(_FixPaths(inputs)) - outputs = OrderedSet(_FixPaths(outputs)) - inputs.remove(_FixPath(trigger_file)) - sources.update(inputs) - if not is_msbuild: - excluded_sources.update(inputs) - sources.update(outputs) + # Add outputs generated by each rule (if applicable). + for rule in rules: + # Add in the outputs from this rule. + trigger_files = _FindRuleTriggerFiles(rule, sources) + for trigger_file in trigger_files: + # Remove trigger_file from excluded_sources to let the rule be triggered + # (e.g. rule trigger ax_enums.idl is added to excluded_sources + # because it's also in an action's inputs in the same project) + excluded_sources.discard(_FixPath(trigger_file)) + # Done if not processing outputs as sources. + if int(rule.get("process_outputs_as_sources", False)): + inputs, outputs = _RuleInputsAndOutputs(rule, trigger_file) + inputs = OrderedSet(_FixPaths(inputs)) + outputs = OrderedSet(_FixPaths(outputs)) + inputs.remove(_FixPath(trigger_file)) + sources.update(inputs) + if not is_msbuild: + excluded_sources.update(inputs) + sources.update(outputs) def _FilterActionsFromExcluded(excluded_sources, actions_to_add): - """Take inputs with actions attached out of the list of exclusions. + """Take inputs with actions attached out of the list of exclusions. Arguments: excluded_sources: list of source files not to be built. @@ -899,16 +951,16 @@ def _FilterActionsFromExcluded(excluded_sources, actions_to_add): Returns: excluded_sources with files that have actions attached removed. """ - must_keep = OrderedSet(_FixPaths(actions_to_add.keys())) - return [s for s in excluded_sources if s not in must_keep] + must_keep = OrderedSet(_FixPaths(actions_to_add.keys())) + return [s for s in excluded_sources if s not in must_keep] def _GetDefaultConfiguration(spec): - return spec['configurations'][spec['default_configuration']] + return spec["configurations"][spec["default_configuration"]] def _GetGuidOfProject(proj_path, spec): - """Get the guid for the project. + """Get the guid for the project. Arguments: proj_path: Path of the vcproj or vcxproj file to generate. @@ -918,21 +970,23 @@ def _GetGuidOfProject(proj_path, spec): Raises: ValueError: if the specified GUID is invalid. """ - # Pluck out the default configuration. - default_config = _GetDefaultConfiguration(spec) - # Decide the guid of the project. - guid = default_config.get('msvs_guid') - if guid: - if VALID_MSVS_GUID_CHARS.match(guid) is None: - raise ValueError('Invalid MSVS guid: "%s". Must match regex: "%s".' % - (guid, VALID_MSVS_GUID_CHARS.pattern)) - guid = '{%s}' % guid - guid = guid or MSVSNew.MakeGuid(proj_path) - return guid + # Pluck out the default configuration. + default_config = _GetDefaultConfiguration(spec) + # Decide the guid of the project. + guid = default_config.get("msvs_guid") + if guid: + if VALID_MSVS_GUID_CHARS.match(guid) is None: + raise ValueError( + 'Invalid MSVS guid: "%s". Must match regex: "%s".' + % (guid, VALID_MSVS_GUID_CHARS.pattern) + ) + guid = "{%s}" % guid + guid = guid or MSVSNew.MakeGuid(proj_path) + return guid def _GetMsbuildToolsetOfProject(proj_path, spec, version): - """Get the platform toolset for the project. + """Get the platform toolset for the project. Arguments: proj_path: Path of the vcproj or vcxproj file to generate. @@ -941,18 +995,18 @@ def _GetMsbuildToolsetOfProject(proj_path, spec, version): Returns: the platform toolset string or None. """ - # Pluck out the default configuration. - default_config = _GetDefaultConfiguration(spec) - toolset = default_config.get('msbuild_toolset') - if not toolset and version.DefaultToolset(): - toolset = version.DefaultToolset() - if spec['type'] == 'windows_driver': - toolset = 'WindowsKernelModeDriver10.0' - return toolset + # Pluck out the default configuration. + default_config = _GetDefaultConfiguration(spec) + toolset = default_config.get("msbuild_toolset") + if not toolset and version.DefaultToolset(): + toolset = version.DefaultToolset() + if spec["type"] == "windows_driver": + toolset = "WindowsKernelModeDriver10.0" + return toolset def _GenerateProject(project, options, version, generator_flags): - """Generates a vcproj file. + """Generates a vcproj file. Arguments: project: the MSVSProject object. @@ -962,56 +1016,59 @@ def _GenerateProject(project, options, version, generator_flags): Returns: A list of source files that cannot be found on disk. """ - default_config = _GetDefaultConfiguration(project.spec) + default_config = _GetDefaultConfiguration(project.spec) - # Skip emitting anything if told to with msvs_existing_vcproj option. - if default_config.get('msvs_existing_vcproj'): - return [] + # Skip emitting anything if told to with msvs_existing_vcproj option. + if default_config.get("msvs_existing_vcproj"): + return [] - if version.UsesVcxproj(): - return _GenerateMSBuildProject(project, options, version, generator_flags) - else: - return _GenerateMSVSProject(project, options, version, generator_flags) + if version.UsesVcxproj(): + return _GenerateMSBuildProject(project, options, version, generator_flags) + else: + return _GenerateMSVSProject(project, options, version, generator_flags) # TODO: Avoid code duplication with _ValidateSourcesForOSX in make.py. def _ValidateSourcesForMSVSProject(spec, version): - """Makes sure if duplicate basenames are not specified in the source list. + """Makes sure if duplicate basenames are not specified in the source list. Arguments: spec: The target dictionary containing the properties of the target. version: The VisualStudioVersion object. """ - # This validation should not be applied to MSVC2010 and later. - assert not version.UsesVcxproj() - - # TODO: Check if MSVC allows this for loadable_module targets. - if spec.get('type', None) not in ('static_library', 'shared_library'): - return - sources = spec.get('sources', []) - basenames = {} - for source in sources: - name, ext = os.path.splitext(source) - is_compiled_file = ext in [ - '.c', '.cc', '.cpp', '.cxx', '.m', '.mm', '.s', '.S'] - if not is_compiled_file: - continue - basename = os.path.basename(name) # Don't include extension. - basenames.setdefault(basename, []).append(source) - - error = '' - for basename, files in basenames.items(): - if len(files) > 1: - error += ' %s: %s\n' % (basename, ' '.join(files)) - - if error: - print('static library %s has several files with the same basename:\n' % spec['target_name'] - + error + 'MSVC08 cannot handle that.') - raise GypError('Duplicate basenames in sources section, see list above') + # This validation should not be applied to MSVC2010 and later. + assert not version.UsesVcxproj() + + # TODO: Check if MSVC allows this for loadable_module targets. + if spec.get("type", None) not in ("static_library", "shared_library"): + return + sources = spec.get("sources", []) + basenames = {} + for source in sources: + name, ext = os.path.splitext(source) + is_compiled_file = ext in [".c", ".cc", ".cpp", ".cxx", ".m", ".mm", ".s", ".S"] + if not is_compiled_file: + continue + basename = os.path.basename(name) # Don't include extension. + basenames.setdefault(basename, []).append(source) + + error = "" + for basename, files in basenames.items(): + if len(files) > 1: + error += " %s: %s\n" % (basename, " ".join(files)) + + if error: + print( + "static library %s has several files with the same basename:\n" + % spec["target_name"] + + error + + "MSVC08 cannot handle that." + ) + raise GypError("Duplicate basenames in sources section, see list above") def _GenerateMSVSProject(project, options, version, generator_flags): - """Generates a .vcproj file. It may create .rules and .user files too. + """Generates a .vcproj file. It may create .rules and .user files too. Arguments: project: The project object we will generate the file for. @@ -1019,85 +1076,82 @@ def _GenerateMSVSProject(project, options, version, generator_flags): version: The VisualStudioVersion object. generator_flags: dict of generator-specific flags. """ - spec = project.spec - gyp.common.EnsureDirExists(project.path) - - platforms = _GetUniquePlatforms(spec) - p = MSVSProject.Writer(project.path, version, spec['target_name'], - project.guid, platforms) - - # Get directory project file is in. - project_dir = os.path.split(project.path)[0] - gyp_path = _NormalizedSource(project.build_file) - relative_path_of_gyp_file = gyp.common.RelativePath(gyp_path, project_dir) - - config_type = _GetMSVSConfigurationType(spec, project.build_file) - for config_name, config in spec['configurations'].items(): - _AddConfigurationToMSVSProject(p, spec, config_type, config_name, config) - - # MSVC08 and prior version cannot handle duplicate basenames in the same - # target. - # TODO: Take excluded sources into consideration if possible. - _ValidateSourcesForMSVSProject(spec, version) - - # Prepare list of sources and excluded sources. - gyp_file = os.path.split(project.build_file)[1] - sources, excluded_sources = _PrepareListOfSources(spec, generator_flags, - gyp_file) - - # Add rules. - actions_to_add = {} - _GenerateRulesForMSVS(p, project_dir, options, spec, - sources, excluded_sources, - actions_to_add) - list_excluded = generator_flags.get('msvs_list_excluded_files', True) - sources, excluded_sources, excluded_idl = ( - _AdjustSourcesAndConvertToFilterHierarchy(spec, options, project_dir, - sources, excluded_sources, - list_excluded, version)) - - # Add in files. - missing_sources = _VerifySourcesExist(sources, project_dir) - p.AddFiles(sources) - - _AddToolFilesToMSVS(p, spec) - _HandlePreCompiledHeaders(p, sources, spec) - _AddActions(actions_to_add, spec, relative_path_of_gyp_file) - _AddCopies(actions_to_add, spec) - _WriteMSVSUserFile(project.path, version, spec) - - # NOTE: this stanza must appear after all actions have been decided. - # Don't excluded sources with actions attached, or they won't run. - excluded_sources = _FilterActionsFromExcluded( - excluded_sources, actions_to_add) - _ExcludeFilesFromBeingBuilt(p, spec, excluded_sources, excluded_idl, - list_excluded) - _AddAccumulatedActionsToMSVS(p, spec, actions_to_add) - - # Write it out. - p.WriteIfChanged() - - return missing_sources + spec = project.spec + gyp.common.EnsureDirExists(project.path) + + platforms = _GetUniquePlatforms(spec) + p = MSVSProject.Writer( + project.path, version, spec["target_name"], project.guid, platforms + ) + + # Get directory project file is in. + project_dir = os.path.split(project.path)[0] + gyp_path = _NormalizedSource(project.build_file) + relative_path_of_gyp_file = gyp.common.RelativePath(gyp_path, project_dir) + + config_type = _GetMSVSConfigurationType(spec, project.build_file) + for config_name, config in spec["configurations"].items(): + _AddConfigurationToMSVSProject(p, spec, config_type, config_name, config) + + # MSVC08 and prior version cannot handle duplicate basenames in the same + # target. + # TODO: Take excluded sources into consideration if possible. + _ValidateSourcesForMSVSProject(spec, version) + + # Prepare list of sources and excluded sources. + gyp_file = os.path.split(project.build_file)[1] + sources, excluded_sources = _PrepareListOfSources(spec, generator_flags, gyp_file) + + # Add rules. + actions_to_add = {} + _GenerateRulesForMSVS( + p, project_dir, options, spec, sources, excluded_sources, actions_to_add + ) + list_excluded = generator_flags.get("msvs_list_excluded_files", True) + sources, excluded_sources, excluded_idl = _AdjustSourcesAndConvertToFilterHierarchy( + spec, options, project_dir, sources, excluded_sources, list_excluded, version + ) + + # Add in files. + missing_sources = _VerifySourcesExist(sources, project_dir) + p.AddFiles(sources) + + _AddToolFilesToMSVS(p, spec) + _HandlePreCompiledHeaders(p, sources, spec) + _AddActions(actions_to_add, spec, relative_path_of_gyp_file) + _AddCopies(actions_to_add, spec) + _WriteMSVSUserFile(project.path, version, spec) + + # NOTE: this stanza must appear after all actions have been decided. + # Don't excluded sources with actions attached, or they won't run. + excluded_sources = _FilterActionsFromExcluded(excluded_sources, actions_to_add) + _ExcludeFilesFromBeingBuilt(p, spec, excluded_sources, excluded_idl, list_excluded) + _AddAccumulatedActionsToMSVS(p, spec, actions_to_add) + + # Write it out. + p.WriteIfChanged() + + return missing_sources def _GetUniquePlatforms(spec): - """Returns the list of unique platforms for this spec, e.g ['win32', ...]. + """Returns the list of unique platforms for this spec, e.g ['win32', ...]. Arguments: spec: The target dictionary containing the properties of the target. Returns: The MSVSUserFile object created. """ - # Gather list of unique platforms. - platforms = OrderedSet() - for configuration in spec['configurations']: - platforms.add(_ConfigPlatform(spec['configurations'][configuration])) - platforms = list(platforms) - return platforms + # Gather list of unique platforms. + platforms = OrderedSet() + for configuration in spec["configurations"]: + platforms.add(_ConfigPlatform(spec["configurations"][configuration])) + platforms = list(platforms) + return platforms def _CreateMSVSUserFile(proj_path, version, spec): - """Generates a .user file for the user running this Gyp program. + """Generates a .user file for the user running this Gyp program. Arguments: proj_path: The path of the project file being created. The .user file @@ -1107,15 +1161,14 @@ def _CreateMSVSUserFile(proj_path, version, spec): Returns: The MSVSUserFile object created. """ - (domain, username) = _GetDomainAndUserName() - vcuser_filename = '.'.join([proj_path, domain, username, 'user']) - user_file = MSVSUserFile.Writer(vcuser_filename, version, - spec['target_name']) - return user_file + (domain, username) = _GetDomainAndUserName() + vcuser_filename = ".".join([proj_path, domain, username, "user"]) + user_file = MSVSUserFile.Writer(vcuser_filename, version, spec["target_name"]) + return user_file def _GetMSVSConfigurationType(spec, build_file): - """Returns the configuration type for this project. + """Returns the configuration type for this project. It's a number defined by Microsoft. May raise an exception. @@ -1125,28 +1178,31 @@ def _GetMSVSConfigurationType(spec, build_file): Returns: An integer, the configuration type. """ - try: - config_type = { - 'executable': '1', # .exe - 'shared_library': '2', # .dll - 'loadable_module': '2', # .dll - 'static_library': '4', # .lib - 'windows_driver': '5', # .sys - 'none': '10', # Utility type - }[spec['type']] - except KeyError: - if spec.get('type'): - raise GypError('Target type %s is not a valid target type for ' - 'target %s in %s.' % - (spec['type'], spec['target_name'], build_file)) - else: - raise GypError('Missing type field for target %s in %s.' % - (spec['target_name'], build_file)) - return config_type + try: + config_type = { + "executable": "1", # .exe + "shared_library": "2", # .dll + "loadable_module": "2", # .dll + "static_library": "4", # .lib + "windows_driver": "5", # .sys + "none": "10", # Utility type + }[spec["type"]] + except KeyError: + if spec.get("type"): + raise GypError( + "Target type %s is not a valid target type for " + "target %s in %s." % (spec["type"], spec["target_name"], build_file) + ) + else: + raise GypError( + "Missing type field for target %s in %s." + % (spec["target_name"], build_file) + ) + return config_type def _AddConfigurationToMSVSProject(p, spec, config_type, config_name, config): - """Adds a configuration to the MSVS project. + """Adds a configuration to the MSVS project. Many settings in a vcproj file are specific to a configuration. This function the main part of the vcproj file that's configuration specific. @@ -1159,81 +1215,84 @@ def _AddConfigurationToMSVSProject(p, spec, config_type, config_name, config): config: The dictionary that defines the special processing to be done for this configuration. """ - # Get the information for this configuration - include_dirs, midl_include_dirs, resource_include_dirs = \ - _GetIncludeDirs(config) - libraries = _GetLibraries(spec) - library_dirs = _GetLibraryDirs(config) - out_file, vc_tool, _ = _GetOutputFilePathAndTool(spec, msbuild=False) - defines = _GetDefines(config) - defines = [_EscapeCppDefineForMSVS(d) for d in defines] - disabled_warnings = _GetDisabledWarnings(config) - prebuild = config.get('msvs_prebuild') - postbuild = config.get('msvs_postbuild') - def_file = _GetModuleDefinition(spec) - precompiled_header = config.get('msvs_precompiled_header') - - # Prepare the list of tools as a dictionary. - tools = dict() - # Add in user specified msvs_settings. - msvs_settings = config.get('msvs_settings', {}) - MSVSSettings.ValidateMSVSSettings(msvs_settings) - - # Prevent default library inheritance from the environment. - _ToolAppend(tools, 'VCLinkerTool', 'AdditionalDependencies', ['$(NOINHERIT)']) - - for tool in msvs_settings: - settings = config['msvs_settings'][tool] - for setting in settings: - _ToolAppend(tools, tool, setting, settings[setting]) - # Add the information to the appropriate tool - _ToolAppend(tools, 'VCCLCompilerTool', - 'AdditionalIncludeDirectories', include_dirs) - _ToolAppend(tools, 'VCMIDLTool', - 'AdditionalIncludeDirectories', midl_include_dirs) - _ToolAppend(tools, 'VCResourceCompilerTool', - 'AdditionalIncludeDirectories', resource_include_dirs) - # Add in libraries. - _ToolAppend(tools, 'VCLinkerTool', 'AdditionalDependencies', libraries) - _ToolAppend(tools, 'VCLinkerTool', 'AdditionalLibraryDirectories', - library_dirs) - if out_file: - _ToolAppend(tools, vc_tool, 'OutputFile', out_file, only_if_unset=True) - # Add defines. - _ToolAppend(tools, 'VCCLCompilerTool', 'PreprocessorDefinitions', defines) - _ToolAppend(tools, 'VCResourceCompilerTool', 'PreprocessorDefinitions', - defines) - # Change program database directory to prevent collisions. - _ToolAppend(tools, 'VCCLCompilerTool', 'ProgramDataBaseFileName', - '$(IntDir)$(ProjectName)\\vc80.pdb', only_if_unset=True) - # Add disabled warnings. - _ToolAppend(tools, 'VCCLCompilerTool', - 'DisableSpecificWarnings', disabled_warnings) - # Add Pre-build. - _ToolAppend(tools, 'VCPreBuildEventTool', 'CommandLine', prebuild) - # Add Post-build. - _ToolAppend(tools, 'VCPostBuildEventTool', 'CommandLine', postbuild) - # Turn on precompiled headers if appropriate. - if precompiled_header: - precompiled_header = os.path.split(precompiled_header)[1] - _ToolAppend(tools, 'VCCLCompilerTool', 'UsePrecompiledHeader', '2') - _ToolAppend(tools, 'VCCLCompilerTool', - 'PrecompiledHeaderThrough', precompiled_header) - _ToolAppend(tools, 'VCCLCompilerTool', - 'ForcedIncludeFiles', precompiled_header) - # Loadable modules don't generate import libraries; - # tell dependent projects to not expect one. - if spec['type'] == 'loadable_module': - _ToolAppend(tools, 'VCLinkerTool', 'IgnoreImportLibrary', 'true') - # Set the module definition file if any. - if def_file: - _ToolAppend(tools, 'VCLinkerTool', 'ModuleDefinitionFile', def_file) - - _AddConfigurationToMSVS(p, spec, tools, config, config_type, config_name) + # Get the information for this configuration + include_dirs, midl_include_dirs, resource_include_dirs = _GetIncludeDirs(config) + libraries = _GetLibraries(spec) + library_dirs = _GetLibraryDirs(config) + out_file, vc_tool, _ = _GetOutputFilePathAndTool(spec, msbuild=False) + defines = _GetDefines(config) + defines = [_EscapeCppDefineForMSVS(d) for d in defines] + disabled_warnings = _GetDisabledWarnings(config) + prebuild = config.get("msvs_prebuild") + postbuild = config.get("msvs_postbuild") + def_file = _GetModuleDefinition(spec) + precompiled_header = config.get("msvs_precompiled_header") + + # Prepare the list of tools as a dictionary. + tools = dict() + # Add in user specified msvs_settings. + msvs_settings = config.get("msvs_settings", {}) + MSVSSettings.ValidateMSVSSettings(msvs_settings) + + # Prevent default library inheritance from the environment. + _ToolAppend(tools, "VCLinkerTool", "AdditionalDependencies", ["$(NOINHERIT)"]) + + for tool in msvs_settings: + settings = config["msvs_settings"][tool] + for setting in settings: + _ToolAppend(tools, tool, setting, settings[setting]) + # Add the information to the appropriate tool + _ToolAppend(tools, "VCCLCompilerTool", "AdditionalIncludeDirectories", include_dirs) + _ToolAppend(tools, "VCMIDLTool", "AdditionalIncludeDirectories", midl_include_dirs) + _ToolAppend( + tools, + "VCResourceCompilerTool", + "AdditionalIncludeDirectories", + resource_include_dirs, + ) + # Add in libraries. + _ToolAppend(tools, "VCLinkerTool", "AdditionalDependencies", libraries) + _ToolAppend(tools, "VCLinkerTool", "AdditionalLibraryDirectories", library_dirs) + if out_file: + _ToolAppend(tools, vc_tool, "OutputFile", out_file, only_if_unset=True) + # Add defines. + _ToolAppend(tools, "VCCLCompilerTool", "PreprocessorDefinitions", defines) + _ToolAppend(tools, "VCResourceCompilerTool", "PreprocessorDefinitions", defines) + # Change program database directory to prevent collisions. + _ToolAppend( + tools, + "VCCLCompilerTool", + "ProgramDataBaseFileName", + "$(IntDir)$(ProjectName)\\vc80.pdb", + only_if_unset=True, + ) + # Add disabled warnings. + _ToolAppend(tools, "VCCLCompilerTool", "DisableSpecificWarnings", disabled_warnings) + # Add Pre-build. + _ToolAppend(tools, "VCPreBuildEventTool", "CommandLine", prebuild) + # Add Post-build. + _ToolAppend(tools, "VCPostBuildEventTool", "CommandLine", postbuild) + # Turn on precompiled headers if appropriate. + if precompiled_header: + precompiled_header = os.path.split(precompiled_header)[1] + _ToolAppend(tools, "VCCLCompilerTool", "UsePrecompiledHeader", "2") + _ToolAppend( + tools, "VCCLCompilerTool", "PrecompiledHeaderThrough", precompiled_header + ) + _ToolAppend(tools, "VCCLCompilerTool", "ForcedIncludeFiles", precompiled_header) + # Loadable modules don't generate import libraries; + # tell dependent projects to not expect one. + if spec["type"] == "loadable_module": + _ToolAppend(tools, "VCLinkerTool", "IgnoreImportLibrary", "true") + # Set the module definition file if any. + if def_file: + _ToolAppend(tools, "VCLinkerTool", "ModuleDefinitionFile", def_file) + + _AddConfigurationToMSVS(p, spec, tools, config, config_type, config_name) def _GetIncludeDirs(config): - """Returns the list of directories to be used for #include directives. + """Returns the list of directories to be used for #include directives. Arguments: config: The dictionary that defines the special processing to be done @@ -1241,23 +1300,23 @@ def _GetIncludeDirs(config): Returns: The list of directory paths. """ - # TODO(bradnelson): include_dirs should really be flexible enough not to - # require this sort of thing. - include_dirs = ( - config.get('include_dirs', []) + - config.get('msvs_system_include_dirs', [])) - midl_include_dirs = ( - config.get('midl_include_dirs', []) + - config.get('msvs_system_include_dirs', [])) - resource_include_dirs = config.get('resource_include_dirs', include_dirs) - include_dirs = _FixPaths(include_dirs) - midl_include_dirs = _FixPaths(midl_include_dirs) - resource_include_dirs = _FixPaths(resource_include_dirs) - return include_dirs, midl_include_dirs, resource_include_dirs + # TODO(bradnelson): include_dirs should really be flexible enough not to + # require this sort of thing. + include_dirs = config.get("include_dirs", []) + config.get( + "msvs_system_include_dirs", [] + ) + midl_include_dirs = config.get("midl_include_dirs", []) + config.get( + "msvs_system_include_dirs", [] + ) + resource_include_dirs = config.get("resource_include_dirs", include_dirs) + include_dirs = _FixPaths(include_dirs) + midl_include_dirs = _FixPaths(midl_include_dirs) + resource_include_dirs = _FixPaths(resource_include_dirs) + return include_dirs, midl_include_dirs, resource_include_dirs def _GetLibraryDirs(config): - """Returns the list of directories to be used for library search paths. + """Returns the list of directories to be used for library search paths. Arguments: config: The dictionary that defines the special processing to be done @@ -1266,39 +1325,39 @@ def _GetLibraryDirs(config): The list of directory paths. """ - library_dirs = config.get('library_dirs', []) - library_dirs = _FixPaths(library_dirs) - return library_dirs + library_dirs = config.get("library_dirs", []) + library_dirs = _FixPaths(library_dirs) + return library_dirs def _GetLibraries(spec): - """Returns the list of libraries for this configuration. + """Returns the list of libraries for this configuration. Arguments: spec: The target dictionary containing the properties of the target. Returns: The list of directory paths. """ - libraries = spec.get('libraries', []) - # Strip out -l, as it is not used on windows (but is needed so we can pass - # in libraries that are assumed to be in the default library path). - # Also remove duplicate entries, leaving only the last duplicate, while - # preserving order. - found = OrderedSet() - unique_libraries_list = [] - for entry in reversed(libraries): - library = re.sub(r'^\-l', '', entry) - if not os.path.splitext(library)[1]: - library += '.lib' - if library not in found: - found.add(library) - unique_libraries_list.append(library) - unique_libraries_list.reverse() - return unique_libraries_list + libraries = spec.get("libraries", []) + # Strip out -l, as it is not used on windows (but is needed so we can pass + # in libraries that are assumed to be in the default library path). + # Also remove duplicate entries, leaving only the last duplicate, while + # preserving order. + found = OrderedSet() + unique_libraries_list = [] + for entry in reversed(libraries): + library = re.sub(r"^\-l", "", entry) + if not os.path.splitext(library)[1]: + library += ".lib" + if library not in found: + found.add(library) + unique_libraries_list.append(library) + unique_libraries_list.reverse() + return unique_libraries_list def _GetOutputFilePathAndTool(spec, msbuild): - """Returns the path and tool to use for this target. + """Returns the path and tool to use for this target. Figures out the path of the file this spec will create and the name of the VC tool that will create it. @@ -1308,36 +1367,36 @@ def _GetOutputFilePathAndTool(spec, msbuild): Returns: A triple of (file path, name of the vc tool, name of the msbuild tool) """ - # Select a name for the output file. - out_file = '' - vc_tool = '' - msbuild_tool = '' - output_file_map = { - 'executable': ('VCLinkerTool', 'Link', '$(OutDir)', '.exe'), - 'shared_library': ('VCLinkerTool', 'Link', '$(OutDir)', '.dll'), - 'loadable_module': ('VCLinkerTool', 'Link', '$(OutDir)', '.dll'), - 'windows_driver': ('VCLinkerTool', 'Link', '$(OutDir)', '.sys'), - 'static_library': ('VCLibrarianTool', 'Lib', '$(OutDir)lib\\', '.lib'), - } - output_file_props = output_file_map.get(spec['type']) - if output_file_props and int(spec.get('msvs_auto_output_file', 1)): - vc_tool, msbuild_tool, out_dir, suffix = output_file_props - if spec.get('standalone_static_library', 0): - out_dir = '$(OutDir)' - out_dir = spec.get('product_dir', out_dir) - product_extension = spec.get('product_extension') - if product_extension: - suffix = '.' + product_extension - elif msbuild: - suffix = '$(TargetExt)' - prefix = spec.get('product_prefix', '') - product_name = spec.get('product_name', '$(ProjectName)') - out_file = ntpath.join(out_dir, prefix + product_name + suffix) - return out_file, vc_tool, msbuild_tool + # Select a name for the output file. + out_file = "" + vc_tool = "" + msbuild_tool = "" + output_file_map = { + "executable": ("VCLinkerTool", "Link", "$(OutDir)", ".exe"), + "shared_library": ("VCLinkerTool", "Link", "$(OutDir)", ".dll"), + "loadable_module": ("VCLinkerTool", "Link", "$(OutDir)", ".dll"), + "windows_driver": ("VCLinkerTool", "Link", "$(OutDir)", ".sys"), + "static_library": ("VCLibrarianTool", "Lib", "$(OutDir)lib\\", ".lib"), + } + output_file_props = output_file_map.get(spec["type"]) + if output_file_props and int(spec.get("msvs_auto_output_file", 1)): + vc_tool, msbuild_tool, out_dir, suffix = output_file_props + if spec.get("standalone_static_library", 0): + out_dir = "$(OutDir)" + out_dir = spec.get("product_dir", out_dir) + product_extension = spec.get("product_extension") + if product_extension: + suffix = "." + product_extension + elif msbuild: + suffix = "$(TargetExt)" + prefix = spec.get("product_prefix", "") + product_name = spec.get("product_name", "$(ProjectName)") + out_file = ntpath.join(out_dir, prefix + product_name + suffix) + return out_file, vc_tool, msbuild_tool def _GetOutputTargetExt(spec): - """Returns the extension for this target, including the dot + """Returns the extension for this target, including the dot If product_extension is specified, set target_extension to this to avoid MSB8012, returns None otherwise. Ignores any target_extension settings in @@ -1348,14 +1407,14 @@ def _GetOutputTargetExt(spec): Returns: A string with the extension, or None """ - target_extension = spec.get('product_extension') - if target_extension: - return '.' + target_extension - return None + target_extension = spec.get("product_extension") + if target_extension: + return "." + target_extension + return None def _GetDefines(config): - """Returns the list of preprocessor definitions for this configuation. + """Returns the list of preprocessor definitions for this configuation. Arguments: config: The dictionary that defines the special processing to be done @@ -1363,64 +1422,68 @@ def _GetDefines(config): Returns: The list of preprocessor definitions. """ - defines = [] - for d in config.get('defines', []): - if type(d) == list: - fd = '='.join([str(dpart) for dpart in d]) - else: - fd = str(d) - defines.append(fd) - return defines + defines = [] + for d in config.get("defines", []): + if type(d) == list: + fd = "=".join([str(dpart) for dpart in d]) + else: + fd = str(d) + defines.append(fd) + return defines def _GetDisabledWarnings(config): - return [str(i) for i in config.get('msvs_disabled_warnings', [])] + return [str(i) for i in config.get("msvs_disabled_warnings", [])] def _GetModuleDefinition(spec): - def_file = '' - if spec['type'] in ['shared_library', 'loadable_module', 'executable', - 'windows_driver']: - def_files = [s for s in spec.get('sources', []) if s.endswith('.def')] - if len(def_files) == 1: - def_file = _FixPath(def_files[0]) - elif def_files: - raise ValueError( - 'Multiple module definition files in one target, target %s lists ' - 'multiple .def files: %s' % ( - spec['target_name'], ' '.join(def_files))) - return def_file + def_file = "" + if spec["type"] in [ + "shared_library", + "loadable_module", + "executable", + "windows_driver", + ]: + def_files = [s for s in spec.get("sources", []) if s.endswith(".def")] + if len(def_files) == 1: + def_file = _FixPath(def_files[0]) + elif def_files: + raise ValueError( + "Multiple module definition files in one target, target %s lists " + "multiple .def files: %s" % (spec["target_name"], " ".join(def_files)) + ) + return def_file def _ConvertToolsToExpectedForm(tools): - """Convert tools to a form expected by Visual Studio. + """Convert tools to a form expected by Visual Studio. Arguments: tools: A dictionary of settings; the tool name is the key. Returns: A list of Tool objects. """ - tool_list = [] - for tool, settings in tools.items(): - # Collapse settings with lists. - settings_fixed = {} - for setting, value in settings.items(): - if type(value) == list: - if ((tool == 'VCLinkerTool' and - setting == 'AdditionalDependencies') or - setting == 'AdditionalOptions'): - settings_fixed[setting] = ' '.join(value) - else: - settings_fixed[setting] = ';'.join(value) - else: - settings_fixed[setting] = value - # Add in this tool. - tool_list.append(MSVSProject.Tool(tool, settings_fixed)) - return tool_list + tool_list = [] + for tool, settings in tools.items(): + # Collapse settings with lists. + settings_fixed = {} + for setting, value in settings.items(): + if type(value) == list: + if ( + tool == "VCLinkerTool" and setting == "AdditionalDependencies" + ) or setting == "AdditionalOptions": + settings_fixed[setting] = " ".join(value) + else: + settings_fixed[setting] = ";".join(value) + else: + settings_fixed[setting] = value + # Add in this tool. + tool_list.append(MSVSProject.Tool(tool, settings_fixed)) + return tool_list def _AddConfigurationToMSVS(p, spec, tools, config, config_type, config_name): - """Add to the project file the configuration specified by config. + """Add to the project file the configuration specified by config. Arguments: p: The target project being generated. @@ -1431,45 +1494,45 @@ def _AddConfigurationToMSVS(p, spec, tools, config, config_type, config_name): config_type: The configuration type, a number as defined by Microsoft. config_name: The name of the configuration. """ - attributes = _GetMSVSAttributes(spec, config, config_type) - # Add in this configuration. - tool_list = _ConvertToolsToExpectedForm(tools) - p.AddConfig(_ConfigFullName(config_name, config), - attrs=attributes, tools=tool_list) + attributes = _GetMSVSAttributes(spec, config, config_type) + # Add in this configuration. + tool_list = _ConvertToolsToExpectedForm(tools) + p.AddConfig(_ConfigFullName(config_name, config), attrs=attributes, tools=tool_list) def _GetMSVSAttributes(spec, config, config_type): - # Prepare configuration attributes. - prepared_attrs = {} - source_attrs = config.get('msvs_configuration_attributes', {}) - for a in source_attrs: - prepared_attrs[a] = source_attrs[a] - # Add props files. - vsprops_dirs = config.get('msvs_props', []) - vsprops_dirs = _FixPaths(vsprops_dirs) - if vsprops_dirs: - prepared_attrs['InheritedPropertySheets'] = ';'.join(vsprops_dirs) - # Set configuration type. - prepared_attrs['ConfigurationType'] = config_type - output_dir = prepared_attrs.get('OutputDirectory', - '$(SolutionDir)$(ConfigurationName)') - prepared_attrs['OutputDirectory'] = _FixPath(output_dir) + '\\' - if 'IntermediateDirectory' not in prepared_attrs: - intermediate = '$(ConfigurationName)\\obj\\$(ProjectName)' - prepared_attrs['IntermediateDirectory'] = _FixPath(intermediate) + '\\' - else: - intermediate = _FixPath(prepared_attrs['IntermediateDirectory']) + '\\' - intermediate = MSVSSettings.FixVCMacroSlashes(intermediate) - prepared_attrs['IntermediateDirectory'] = intermediate - return prepared_attrs + # Prepare configuration attributes. + prepared_attrs = {} + source_attrs = config.get("msvs_configuration_attributes", {}) + for a in source_attrs: + prepared_attrs[a] = source_attrs[a] + # Add props files. + vsprops_dirs = config.get("msvs_props", []) + vsprops_dirs = _FixPaths(vsprops_dirs) + if vsprops_dirs: + prepared_attrs["InheritedPropertySheets"] = ";".join(vsprops_dirs) + # Set configuration type. + prepared_attrs["ConfigurationType"] = config_type + output_dir = prepared_attrs.get( + "OutputDirectory", "$(SolutionDir)$(ConfigurationName)" + ) + prepared_attrs["OutputDirectory"] = _FixPath(output_dir) + "\\" + if "IntermediateDirectory" not in prepared_attrs: + intermediate = "$(ConfigurationName)\\obj\\$(ProjectName)" + prepared_attrs["IntermediateDirectory"] = _FixPath(intermediate) + "\\" + else: + intermediate = _FixPath(prepared_attrs["IntermediateDirectory"]) + "\\" + intermediate = MSVSSettings.FixVCMacroSlashes(intermediate) + prepared_attrs["IntermediateDirectory"] = intermediate + return prepared_attrs def _AddNormalizedSources(sources_set, sources_array): - sources_set.update(_NormalizedSource(s) for s in sources_array) + sources_set.update(_NormalizedSource(s) for s in sources_array) def _PrepareListOfSources(spec, generator_flags, gyp_file): - """Prepare list of sources and excluded sources. + """Prepare list of sources and excluded sources. Besides the sources specified directly in the spec, adds the gyp file so that a change to it will cause a re-compile. Also adds appropriate sources @@ -1483,33 +1546,34 @@ def _PrepareListOfSources(spec, generator_flags, gyp_file): A pair of (list of sources, list of excluded sources). The sources will be relative to the gyp file. """ - sources = OrderedSet() - _AddNormalizedSources(sources, spec.get('sources', [])) - excluded_sources = OrderedSet() - # Add in the gyp file. - if not generator_flags.get('standalone'): - sources.add(gyp_file) - - # Add in 'action' inputs and outputs. - for a in spec.get('actions', []): - inputs = a['inputs'] - inputs = [_NormalizedSource(i) for i in inputs] - # Add all inputs to sources and excluded sources. - inputs = OrderedSet(inputs) - sources.update(inputs) - if not spec.get('msvs_external_builder'): - excluded_sources.update(inputs) - if int(a.get('process_outputs_as_sources', False)): - _AddNormalizedSources(sources, a.get('outputs', [])) - # Add in 'copies' inputs and outputs. - for cpy in spec.get('copies', []): - _AddNormalizedSources(sources, cpy.get('files', [])) - return (sources, excluded_sources) + sources = OrderedSet() + _AddNormalizedSources(sources, spec.get("sources", [])) + excluded_sources = OrderedSet() + # Add in the gyp file. + if not generator_flags.get("standalone"): + sources.add(gyp_file) + + # Add in 'action' inputs and outputs. + for a in spec.get("actions", []): + inputs = a["inputs"] + inputs = [_NormalizedSource(i) for i in inputs] + # Add all inputs to sources and excluded sources. + inputs = OrderedSet(inputs) + sources.update(inputs) + if not spec.get("msvs_external_builder"): + excluded_sources.update(inputs) + if int(a.get("process_outputs_as_sources", False)): + _AddNormalizedSources(sources, a.get("outputs", [])) + # Add in 'copies' inputs and outputs. + for cpy in spec.get("copies", []): + _AddNormalizedSources(sources, cpy.get("files", [])) + return (sources, excluded_sources) def _AdjustSourcesAndConvertToFilterHierarchy( - spec, options, gyp_dir, sources, excluded_sources, list_excluded, version): - """Adjusts the list of sources and excluded sources. + spec, options, gyp_dir, sources, excluded_sources, list_excluded, version +): + """Adjusts the list of sources and excluded sources. Also converts the sets to lists. @@ -1524,330 +1588,372 @@ def _AdjustSourcesAndConvertToFilterHierarchy( A trio of (list of sources, list of excluded sources, path of excluded IDL file) """ - # Exclude excluded sources coming into the generator. - excluded_sources.update(OrderedSet(spec.get('sources_excluded', []))) - # Add excluded sources into sources for good measure. - sources.update(excluded_sources) - # Convert to proper windows form. - # NOTE: sources goes from being a set to a list here. - # NOTE: excluded_sources goes from being a set to a list here. - sources = _FixPaths(sources) - # Convert to proper windows form. - excluded_sources = _FixPaths(excluded_sources) - - excluded_idl = _IdlFilesHandledNonNatively(spec, sources) - - precompiled_related = _GetPrecompileRelatedFiles(spec) - # Find the excluded ones, minus the precompiled header related ones. - fully_excluded = [i for i in excluded_sources if i not in precompiled_related] - - # Convert to folders and the right slashes. - sources = [i.split('\\') for i in sources] - sources = _ConvertSourcesToFilterHierarchy(sources, excluded=fully_excluded, - list_excluded=list_excluded, - msvs_version=version) - - # Prune filters with a single child to flatten ugly directory structures - # such as ../../src/modules/module1 etc. - if version.UsesVcxproj(): - while all([isinstance(s, MSVSProject.Filter) for s in sources]) \ - and len(set([s.name for s in sources])) == 1: - assert all([len(s.contents) == 1 for s in sources]) - sources = [s.contents[0] for s in sources] - else: - while len(sources) == 1 and isinstance(sources[0], MSVSProject.Filter): - sources = sources[0].contents - - return sources, excluded_sources, excluded_idl + # Exclude excluded sources coming into the generator. + excluded_sources.update(OrderedSet(spec.get("sources_excluded", []))) + # Add excluded sources into sources for good measure. + sources.update(excluded_sources) + # Convert to proper windows form. + # NOTE: sources goes from being a set to a list here. + # NOTE: excluded_sources goes from being a set to a list here. + sources = _FixPaths(sources) + # Convert to proper windows form. + excluded_sources = _FixPaths(excluded_sources) + + excluded_idl = _IdlFilesHandledNonNatively(spec, sources) + + precompiled_related = _GetPrecompileRelatedFiles(spec) + # Find the excluded ones, minus the precompiled header related ones. + fully_excluded = [i for i in excluded_sources if i not in precompiled_related] + + # Convert to folders and the right slashes. + sources = [i.split("\\") for i in sources] + sources = _ConvertSourcesToFilterHierarchy( + sources, + excluded=fully_excluded, + list_excluded=list_excluded, + msvs_version=version, + ) + + # Prune filters with a single child to flatten ugly directory structures + # such as ../../src/modules/module1 etc. + if version.UsesVcxproj(): + while ( + all([isinstance(s, MSVSProject.Filter) for s in sources]) + and len(set([s.name for s in sources])) == 1 + ): + assert all([len(s.contents) == 1 for s in sources]) + sources = [s.contents[0] for s in sources] + else: + while len(sources) == 1 and isinstance(sources[0], MSVSProject.Filter): + sources = sources[0].contents + + return sources, excluded_sources, excluded_idl def _IdlFilesHandledNonNatively(spec, sources): - # If any non-native rules use 'idl' as an extension exclude idl files. - # Gather a list here to use later. - using_idl = False - for rule in spec.get('rules', []): - if rule['extension'] == 'idl' and int(rule.get('msvs_external_rule', 0)): - using_idl = True - break - if using_idl: - excluded_idl = [i for i in sources if i.endswith('.idl')] - else: - excluded_idl = [] - return excluded_idl + # If any non-native rules use 'idl' as an extension exclude idl files. + # Gather a list here to use later. + using_idl = False + for rule in spec.get("rules", []): + if rule["extension"] == "idl" and int(rule.get("msvs_external_rule", 0)): + using_idl = True + break + if using_idl: + excluded_idl = [i for i in sources if i.endswith(".idl")] + else: + excluded_idl = [] + return excluded_idl def _GetPrecompileRelatedFiles(spec): - # Gather a list of precompiled header related sources. - precompiled_related = [] - for _, config in spec['configurations'].items(): - for k in precomp_keys: - f = config.get(k) - if f: - precompiled_related.append(_FixPath(f)) - return precompiled_related - - -def _ExcludeFilesFromBeingBuilt(p, spec, excluded_sources, excluded_idl, - list_excluded): - exclusions = _GetExcludedFilesFromBuild(spec, excluded_sources, excluded_idl) - for file_name, excluded_configs in exclusions.items(): - if (not list_excluded and - len(excluded_configs) == len(spec['configurations'])): - # If we're not listing excluded files, then they won't appear in the - # project, so don't try to configure them to be excluded. - pass - else: - for config_name, config in excluded_configs: - p.AddFileConfig(file_name, _ConfigFullName(config_name, config), - {'ExcludedFromBuild': 'true'}) + # Gather a list of precompiled header related sources. + precompiled_related = [] + for _, config in spec["configurations"].items(): + for k in precomp_keys: + f = config.get(k) + if f: + precompiled_related.append(_FixPath(f)) + return precompiled_related + + +def _ExcludeFilesFromBeingBuilt(p, spec, excluded_sources, excluded_idl, list_excluded): + exclusions = _GetExcludedFilesFromBuild(spec, excluded_sources, excluded_idl) + for file_name, excluded_configs in exclusions.items(): + if not list_excluded and len(excluded_configs) == len(spec["configurations"]): + # If we're not listing excluded files, then they won't appear in the + # project, so don't try to configure them to be excluded. + pass + else: + for config_name, config in excluded_configs: + p.AddFileConfig( + file_name, + _ConfigFullName(config_name, config), + {"ExcludedFromBuild": "true"}, + ) def _GetExcludedFilesFromBuild(spec, excluded_sources, excluded_idl): - exclusions = {} - # Exclude excluded sources from being built. - for f in excluded_sources: - excluded_configs = [] - for config_name, config in spec['configurations'].items(): - precomped = [_FixPath(config.get(i, '')) for i in precomp_keys] - # Don't do this for ones that are precompiled header related. - if f not in precomped: - excluded_configs.append((config_name, config)) - exclusions[f] = excluded_configs - # If any non-native rules use 'idl' as an extension exclude idl files. - # Exclude them now. - for f in excluded_idl: - excluded_configs = [] - for config_name, config in spec['configurations'].items(): - excluded_configs.append((config_name, config)) - exclusions[f] = excluded_configs - return exclusions + exclusions = {} + # Exclude excluded sources from being built. + for f in excluded_sources: + excluded_configs = [] + for config_name, config in spec["configurations"].items(): + precomped = [_FixPath(config.get(i, "")) for i in precomp_keys] + # Don't do this for ones that are precompiled header related. + if f not in precomped: + excluded_configs.append((config_name, config)) + exclusions[f] = excluded_configs + # If any non-native rules use 'idl' as an extension exclude idl files. + # Exclude them now. + for f in excluded_idl: + excluded_configs = [] + for config_name, config in spec["configurations"].items(): + excluded_configs.append((config_name, config)) + exclusions[f] = excluded_configs + return exclusions def _AddToolFilesToMSVS(p, spec): - # Add in tool files (rules). - tool_files = OrderedSet() - for _, config in spec['configurations'].items(): - for f in config.get('msvs_tool_files', []): - tool_files.add(f) - for f in tool_files: - p.AddToolFile(f) + # Add in tool files (rules). + tool_files = OrderedSet() + for _, config in spec["configurations"].items(): + for f in config.get("msvs_tool_files", []): + tool_files.add(f) + for f in tool_files: + p.AddToolFile(f) def _HandlePreCompiledHeaders(p, sources, spec): - # Pre-compiled header source stubs need a different compiler flag - # (generate precompiled header) and any source file not of the same - # kind (i.e. C vs. C++) as the precompiled header source stub needs - # to have use of precompiled headers disabled. - extensions_excluded_from_precompile = [] - for config_name, config in spec['configurations'].items(): - source = config.get('msvs_precompiled_source') - if source: - source = _FixPath(source) - # UsePrecompiledHeader=1 for if using precompiled headers. - tool = MSVSProject.Tool('VCCLCompilerTool', - {'UsePrecompiledHeader': '1'}) - p.AddFileConfig(source, _ConfigFullName(config_name, config), - {}, tools=[tool]) - basename, extension = os.path.splitext(source) - if extension == '.c': - extensions_excluded_from_precompile = ['.cc', '.cpp', '.cxx'] - else: - extensions_excluded_from_precompile = ['.c'] - def DisableForSourceTree(source_tree): - for source in source_tree: - if isinstance(source, MSVSProject.Filter): - DisableForSourceTree(source.contents) - else: - basename, extension = os.path.splitext(source) - if extension in extensions_excluded_from_precompile: - for config_name, config in spec['configurations'].items(): - tool = MSVSProject.Tool('VCCLCompilerTool', - {'UsePrecompiledHeader': '0', - 'ForcedIncludeFiles': '$(NOINHERIT)'}) - p.AddFileConfig(_FixPath(source), + # Pre-compiled header source stubs need a different compiler flag + # (generate precompiled header) and any source file not of the same + # kind (i.e. C vs. C++) as the precompiled header source stub needs + # to have use of precompiled headers disabled. + extensions_excluded_from_precompile = [] + for config_name, config in spec["configurations"].items(): + source = config.get("msvs_precompiled_source") + if source: + source = _FixPath(source) + # UsePrecompiledHeader=1 for if using precompiled headers. + tool = MSVSProject.Tool("VCCLCompilerTool", {"UsePrecompiledHeader": "1"}) + p.AddFileConfig( + source, _ConfigFullName(config_name, config), {}, tools=[tool] + ) + basename, extension = os.path.splitext(source) + if extension == ".c": + extensions_excluded_from_precompile = [".cc", ".cpp", ".cxx"] + else: + extensions_excluded_from_precompile = [".c"] + + def DisableForSourceTree(source_tree): + for source in source_tree: + if isinstance(source, MSVSProject.Filter): + DisableForSourceTree(source.contents) + else: + basename, extension = os.path.splitext(source) + if extension in extensions_excluded_from_precompile: + for config_name, config in spec["configurations"].items(): + tool = MSVSProject.Tool( + "VCCLCompilerTool", + { + "UsePrecompiledHeader": "0", + "ForcedIncludeFiles": "$(NOINHERIT)", + }, + ) + p.AddFileConfig( + _FixPath(source), _ConfigFullName(config_name, config), - {}, tools=[tool]) - # Do nothing if there was no precompiled source. - if extensions_excluded_from_precompile: - DisableForSourceTree(sources) + {}, + tools=[tool], + ) + + # Do nothing if there was no precompiled source. + if extensions_excluded_from_precompile: + DisableForSourceTree(sources) def _AddActions(actions_to_add, spec, relative_path_of_gyp_file): - # Add actions. - actions = spec.get('actions', []) - # Don't setup_env every time. When all the actions are run together in one - # batch file in VS, the PATH will grow too long. - # Membership in this set means that the cygwin environment has been set up, - # and does not need to be set up again. - have_setup_env = set() - for a in actions: - # Attach actions to the gyp file if nothing else is there. - inputs = a.get('inputs') or [relative_path_of_gyp_file] - attached_to = inputs[0] - need_setup_env = attached_to not in have_setup_env - cmd = _BuildCommandLineForRule(spec, a, has_input_path=False, - do_setup_env=need_setup_env) - have_setup_env.add(attached_to) - # Add the action. - _AddActionStep(actions_to_add, - inputs=inputs, - outputs=a.get('outputs', []), - description=a.get('message', a['action_name']), - command=cmd) + # Add actions. + actions = spec.get("actions", []) + # Don't setup_env every time. When all the actions are run together in one + # batch file in VS, the PATH will grow too long. + # Membership in this set means that the cygwin environment has been set up, + # and does not need to be set up again. + have_setup_env = set() + for a in actions: + # Attach actions to the gyp file if nothing else is there. + inputs = a.get("inputs") or [relative_path_of_gyp_file] + attached_to = inputs[0] + need_setup_env = attached_to not in have_setup_env + cmd = _BuildCommandLineForRule( + spec, a, has_input_path=False, do_setup_env=need_setup_env + ) + have_setup_env.add(attached_to) + # Add the action. + _AddActionStep( + actions_to_add, + inputs=inputs, + outputs=a.get("outputs", []), + description=a.get("message", a["action_name"]), + command=cmd, + ) def _WriteMSVSUserFile(project_path, version, spec): - # Add run_as and test targets. - if 'run_as' in spec: - run_as = spec['run_as'] - action = run_as.get('action', []) - environment = run_as.get('environment', []) - working_directory = run_as.get('working_directory', '.') - elif int(spec.get('test', 0)): - action = ['$(TargetPath)', '--gtest_print_time'] - environment = [] - working_directory = '.' - else: - return # Nothing to add - # Write out the user file. - user_file = _CreateMSVSUserFile(project_path, version, spec) - for config_name, c_data in spec['configurations'].items(): - user_file.AddDebugSettings(_ConfigFullName(config_name, c_data), - action, environment, working_directory) - user_file.WriteIfChanged() + # Add run_as and test targets. + if "run_as" in spec: + run_as = spec["run_as"] + action = run_as.get("action", []) + environment = run_as.get("environment", []) + working_directory = run_as.get("working_directory", ".") + elif int(spec.get("test", 0)): + action = ["$(TargetPath)", "--gtest_print_time"] + environment = [] + working_directory = "." + else: + return # Nothing to add + # Write out the user file. + user_file = _CreateMSVSUserFile(project_path, version, spec) + for config_name, c_data in spec["configurations"].items(): + user_file.AddDebugSettings( + _ConfigFullName(config_name, c_data), action, environment, working_directory + ) + user_file.WriteIfChanged() def _AddCopies(actions_to_add, spec): - copies = _GetCopies(spec) - for inputs, outputs, cmd, description in copies: - _AddActionStep(actions_to_add, inputs=inputs, outputs=outputs, - description=description, command=cmd) + copies = _GetCopies(spec) + for inputs, outputs, cmd, description in copies: + _AddActionStep( + actions_to_add, + inputs=inputs, + outputs=outputs, + description=description, + command=cmd, + ) def _GetCopies(spec): - copies = [] - # Add copies. - for cpy in spec.get('copies', []): - for src in cpy.get('files', []): - dst = os.path.join(cpy['destination'], os.path.basename(src)) - # _AddCustomBuildToolForMSVS() will call _FixPath() on the inputs and - # outputs, so do the same for our generated command line. - if src.endswith('/'): - src_bare = src[:-1] - base_dir = posixpath.split(src_bare)[0] - outer_dir = posixpath.split(src_bare)[1] - fixed_dst = _FixPath(dst) - full_dst = '"%s\\%s\\"' % (fixed_dst, outer_dir) - cmd = 'mkdir %s 2>nul & cd "%s" && xcopy /e /f /y "%s" %s' % ( - full_dst, _FixPath(base_dir), outer_dir, full_dst) - copies.append(([src], ['dummy_copies', dst], cmd, - 'Copying %s to %s' % (src, fixed_dst))) - else: - fix_dst = _FixPath(cpy['destination']) - cmd = 'mkdir "%s" 2>nul & set ERRORLEVEL=0 & copy /Y "%s" "%s"' % ( - fix_dst, _FixPath(src), _FixPath(dst)) - copies.append(([src], [dst], cmd, 'Copying %s to %s' % (src, fix_dst))) - return copies + copies = [] + # Add copies. + for cpy in spec.get("copies", []): + for src in cpy.get("files", []): + dst = os.path.join(cpy["destination"], os.path.basename(src)) + # _AddCustomBuildToolForMSVS() will call _FixPath() on the inputs and + # outputs, so do the same for our generated command line. + if src.endswith("/"): + src_bare = src[:-1] + base_dir = posixpath.split(src_bare)[0] + outer_dir = posixpath.split(src_bare)[1] + fixed_dst = _FixPath(dst) + full_dst = '"%s\\%s\\"' % (fixed_dst, outer_dir) + cmd = 'mkdir %s 2>nul & cd "%s" && xcopy /e /f /y "%s" %s' % ( + full_dst, + _FixPath(base_dir), + outer_dir, + full_dst, + ) + copies.append( + ( + [src], + ["dummy_copies", dst], + cmd, + "Copying %s to %s" % (src, fixed_dst), + ) + ) + else: + fix_dst = _FixPath(cpy["destination"]) + cmd = 'mkdir "%s" 2>nul & set ERRORLEVEL=0 & copy /Y "%s" "%s"' % ( + fix_dst, + _FixPath(src), + _FixPath(dst), + ) + copies.append(([src], [dst], cmd, "Copying %s to %s" % (src, fix_dst))) + return copies def _GetPathDict(root, path): - # |path| will eventually be empty (in the recursive calls) if it was initially - # relative; otherwise it will eventually end up as '\', 'D:\', etc. - if not path or path.endswith(os.sep): - return root - parent, folder = os.path.split(path) - parent_dict = _GetPathDict(root, parent) - if folder not in parent_dict: - parent_dict[folder] = dict() - return parent_dict[folder] + # |path| will eventually be empty (in the recursive calls) if it was initially + # relative; otherwise it will eventually end up as '\', 'D:\', etc. + if not path or path.endswith(os.sep): + return root + parent, folder = os.path.split(path) + parent_dict = _GetPathDict(root, parent) + if folder not in parent_dict: + parent_dict[folder] = dict() + return parent_dict[folder] def _DictsToFolders(base_path, bucket, flat): - # Convert to folders recursively. - children = [] - for folder, contents in bucket.items(): - if type(contents) == dict: - folder_children = _DictsToFolders(os.path.join(base_path, folder), - contents, flat) - if flat: - children += folder_children - else: - folder_children = MSVSNew.MSVSFolder(os.path.join(base_path, folder), - name='(' + folder + ')', - entries=folder_children) - children.append(folder_children) - else: - children.append(contents) - return children + # Convert to folders recursively. + children = [] + for folder, contents in bucket.items(): + if type(contents) == dict: + folder_children = _DictsToFolders( + os.path.join(base_path, folder), contents, flat + ) + if flat: + children += folder_children + else: + folder_children = MSVSNew.MSVSFolder( + os.path.join(base_path, folder), + name="(" + folder + ")", + entries=folder_children, + ) + children.append(folder_children) + else: + children.append(contents) + return children def _CollapseSingles(parent, node): - # Recursively explorer the tree of dicts looking for projects which are - # the sole item in a folder which has the same name as the project. Bring - # such projects up one level. - if (type(node) == dict and - len(node) == 1 and - list(node)[0] == parent + '.vcproj'): - return node[list(node)[0]] - if type(node) != dict: + # Recursively explorer the tree of dicts looking for projects which are + # the sole item in a folder which has the same name as the project. Bring + # such projects up one level. + if type(node) == dict and len(node) == 1 and next(iter(node)) == parent + ".vcproj": + return node[next(iter(node))] + if type(node) != dict: + return node + for child in node: + node[child] = _CollapseSingles(child, node[child]) return node - for child in node: - node[child] = _CollapseSingles(child, node[child]) - return node def _GatherSolutionFolders(sln_projects, project_objects, flat): - root = {} - # Convert into a tree of dicts on path. - for p in sln_projects: - gyp_file, target = gyp.common.ParseQualifiedTarget(p)[0:2] - gyp_dir = os.path.dirname(gyp_file) - path_dict = _GetPathDict(root, gyp_dir) - path_dict[target + '.vcproj'] = project_objects[p] - # Walk down from the top until we hit a folder that has more than one entry. - # In practice, this strips the top-level "src/" dir from the hierarchy in - # the solution. - while len(root) == 1 and type(root[list(root)[0]]) == dict: - root = root[list(root)[0]] - # Collapse singles. - root = _CollapseSingles('', root) - # Merge buckets until everything is a root entry. - return _DictsToFolders('', root, flat) + root = {} + # Convert into a tree of dicts on path. + for p in sln_projects: + gyp_file, target = gyp.common.ParseQualifiedTarget(p)[0:2] + gyp_dir = os.path.dirname(gyp_file) + path_dict = _GetPathDict(root, gyp_dir) + path_dict[target + ".vcproj"] = project_objects[p] + # Walk down from the top until we hit a folder that has more than one entry. + # In practice, this strips the top-level "src/" dir from the hierarchy in + # the solution. + while len(root) == 1 and type(root[next(iter(root))]) == dict: + root = root[next(iter(root))] + # Collapse singles. + root = _CollapseSingles("", root) + # Merge buckets until everything is a root entry. + return _DictsToFolders("", root, flat) def _GetPathOfProject(qualified_target, spec, options, msvs_version): - default_config = _GetDefaultConfiguration(spec) - proj_filename = default_config.get('msvs_existing_vcproj') - if not proj_filename: - proj_filename = (spec['target_name'] + options.suffix + - msvs_version.ProjectExtension()) - - build_file = gyp.common.BuildFile(qualified_target) - proj_path = os.path.join(os.path.dirname(build_file), proj_filename) - fix_prefix = None - if options.generator_output: - project_dir_path = os.path.dirname(os.path.abspath(proj_path)) - proj_path = os.path.join(options.generator_output, proj_path) - fix_prefix = gyp.common.RelativePath(project_dir_path, - os.path.dirname(proj_path)) - return proj_path, fix_prefix + default_config = _GetDefaultConfiguration(spec) + proj_filename = default_config.get("msvs_existing_vcproj") + if not proj_filename: + proj_filename = ( + spec["target_name"] + options.suffix + msvs_version.ProjectExtension() + ) + + build_file = gyp.common.BuildFile(qualified_target) + proj_path = os.path.join(os.path.dirname(build_file), proj_filename) + fix_prefix = None + if options.generator_output: + project_dir_path = os.path.dirname(os.path.abspath(proj_path)) + proj_path = os.path.join(options.generator_output, proj_path) + fix_prefix = gyp.common.RelativePath( + project_dir_path, os.path.dirname(proj_path) + ) + return proj_path, fix_prefix def _GetPlatformOverridesOfProject(spec): - # Prepare a dict indicating which project configurations are used for which - # solution configurations for this target. - config_platform_overrides = {} - for config_name, c in spec['configurations'].items(): - config_fullname = _ConfigFullName(config_name, c) - platform = c.get('msvs_target_platform', _ConfigPlatform(c)) - fixed_config_fullname = '%s|%s' % ( - _ConfigBaseName(config_name, _ConfigPlatform(c)), platform) - config_platform_overrides[config_fullname] = fixed_config_fullname - return config_platform_overrides + # Prepare a dict indicating which project configurations are used for which + # solution configurations for this target. + config_platform_overrides = {} + for config_name, c in spec["configurations"].items(): + config_fullname = _ConfigFullName(config_name, c) + platform = c.get("msvs_target_platform", _ConfigPlatform(c)) + fixed_config_fullname = "%s|%s" % ( + _ConfigBaseName(config_name, _ConfigPlatform(c)), + platform, + ) + config_platform_overrides[config_fullname] = fixed_config_fullname + return config_platform_overrides def _CreateProjectObjects(target_list, target_dicts, options, msvs_version): - """Create a MSVSProject object for the targets found in target list. + """Create a MSVSProject object for the targets found in target list. Arguments: target_list: the list of targets to generate project objects for. @@ -1857,46 +1963,50 @@ def _CreateProjectObjects(target_list, target_dicts, options, msvs_version): Returns: A set of created projects, keyed by target. """ - global fixpath_prefix - # Generate each project. - projects = {} - for qualified_target in target_list: - spec = target_dicts[qualified_target] - if spec['toolset'] != 'target': - raise GypError( - 'Multiple toolsets not supported in msvs build (target %s)' % - qualified_target) - proj_path, fixpath_prefix = _GetPathOfProject(qualified_target, spec, - options, msvs_version) - guid = _GetGuidOfProject(proj_path, spec) - overrides = _GetPlatformOverridesOfProject(spec) - build_file = gyp.common.BuildFile(qualified_target) - # Create object for this project. - obj = MSVSNew.MSVSProject( - proj_path, - name=spec['target_name'], - guid=guid, - spec=spec, - build_file=build_file, - config_platform_overrides=overrides, - fixpath_prefix=fixpath_prefix) - # Set project toolset if any (MS build only) - if msvs_version.UsesVcxproj(): - obj.set_msbuild_toolset( - _GetMsbuildToolsetOfProject(proj_path, spec, msvs_version)) - projects[qualified_target] = obj - # Set all the dependencies, but not if we are using an external builder like - # ninja - for project in projects.values(): - if not project.spec.get('msvs_external_builder'): - deps = project.spec.get('dependencies', []) - deps = [projects[d] for d in deps] - project.set_dependencies(deps) - return projects + global fixpath_prefix + # Generate each project. + projects = {} + for qualified_target in target_list: + spec = target_dicts[qualified_target] + if spec["toolset"] != "target": + raise GypError( + "Multiple toolsets not supported in msvs build (target %s)" + % qualified_target + ) + proj_path, fixpath_prefix = _GetPathOfProject( + qualified_target, spec, options, msvs_version + ) + guid = _GetGuidOfProject(proj_path, spec) + overrides = _GetPlatformOverridesOfProject(spec) + build_file = gyp.common.BuildFile(qualified_target) + # Create object for this project. + obj = MSVSNew.MSVSProject( + proj_path, + name=spec["target_name"], + guid=guid, + spec=spec, + build_file=build_file, + config_platform_overrides=overrides, + fixpath_prefix=fixpath_prefix, + ) + # Set project toolset if any (MS build only) + if msvs_version.UsesVcxproj(): + obj.set_msbuild_toolset( + _GetMsbuildToolsetOfProject(proj_path, spec, msvs_version) + ) + projects[qualified_target] = obj + # Set all the dependencies, but not if we are using an external builder like + # ninja + for project in projects.values(): + if not project.spec.get("msvs_external_builder"): + deps = project.spec.get("dependencies", []) + deps = [projects[d] for d in deps] + project.set_dependencies(deps) + return projects def _InitNinjaFlavor(params, target_list, target_dicts): - """Initialize targets for the ninja flavor. + """Initialize targets for the ninja flavor. This sets up the necessary variables in the targets to generate msvs projects that use ninja as an external builder. The variables in the spec are only set @@ -1907,106 +2017,115 @@ def _InitNinjaFlavor(params, target_list, target_dicts): target_list: List of target pairs: 'base/base.gyp:base'. target_dicts: Dict of target properties keyed on target pair. """ - for qualified_target in target_list: - spec = target_dicts[qualified_target] - if spec.get('msvs_external_builder'): - # The spec explicitly defined an external builder, so don't change it. - continue - - path_to_ninja = spec.get('msvs_path_to_ninja', 'ninja.exe') - - spec['msvs_external_builder'] = 'ninja' - if not spec.get('msvs_external_builder_out_dir'): - gyp_file, _, _ = gyp.common.ParseQualifiedTarget(qualified_target) - gyp_dir = os.path.dirname(gyp_file) - configuration = '$(Configuration)' - if params.get('target_arch') == 'x64': - configuration += '_x64' - if params.get('target_arch') == 'arm64': - configuration += '_arm64' - spec['msvs_external_builder_out_dir'] = os.path.join( - gyp.common.RelativePath(params['options'].toplevel_dir, gyp_dir), - ninja_generator.ComputeOutputDir(params), - configuration) - if not spec.get('msvs_external_builder_build_cmd'): - spec['msvs_external_builder_build_cmd'] = [ - path_to_ninja, - '-C', - '$(OutDir)', - '$(ProjectName)', - ] - if not spec.get('msvs_external_builder_clean_cmd'): - spec['msvs_external_builder_clean_cmd'] = [ - path_to_ninja, - '-C', - '$(OutDir)', - '-tclean', - '$(ProjectName)', - ] + for qualified_target in target_list: + spec = target_dicts[qualified_target] + if spec.get("msvs_external_builder"): + # The spec explicitly defined an external builder, so don't change it. + continue + + path_to_ninja = spec.get("msvs_path_to_ninja", "ninja.exe") + + spec["msvs_external_builder"] = "ninja" + if not spec.get("msvs_external_builder_out_dir"): + gyp_file, _, _ = gyp.common.ParseQualifiedTarget(qualified_target) + gyp_dir = os.path.dirname(gyp_file) + configuration = "$(Configuration)" + if params.get("target_arch") == "x64": + configuration += "_x64" + if params.get("target_arch") == "arm64": + configuration += "_arm64" + spec["msvs_external_builder_out_dir"] = os.path.join( + gyp.common.RelativePath(params["options"].toplevel_dir, gyp_dir), + ninja_generator.ComputeOutputDir(params), + configuration, + ) + if not spec.get("msvs_external_builder_build_cmd"): + spec["msvs_external_builder_build_cmd"] = [ + path_to_ninja, + "-C", + "$(OutDir)", + "$(ProjectName)", + ] + if not spec.get("msvs_external_builder_clean_cmd"): + spec["msvs_external_builder_clean_cmd"] = [ + path_to_ninja, + "-C", + "$(OutDir)", + "-tclean", + "$(ProjectName)", + ] def CalculateVariables(default_variables, params): - """Generated variables that require params to be known.""" - - generator_flags = params.get('generator_flags', {}) - - # Select project file format version (if unset, default to auto detecting). - msvs_version = MSVSVersion.SelectVisualStudioVersion( - generator_flags.get('msvs_version', 'auto')) - # Stash msvs_version for later (so we don't have to probe the system twice). - params['msvs_version'] = msvs_version - - # Set a variable so conditions can be based on msvs_version. - default_variables['MSVS_VERSION'] = msvs_version.ShortName() - - # To determine processor word size on Windows, in addition to checking - # PROCESSOR_ARCHITECTURE (which reflects the word size of the current - # process), it is also necessary to check PROCESSOR_ARCITEW6432 (which - # contains the actual word size of the system when running thru WOW64). - if (os.environ.get('PROCESSOR_ARCHITECTURE', '').find('64') >= 0 or - os.environ.get('PROCESSOR_ARCHITEW6432', '').find('64') >= 0): - default_variables['MSVS_OS_BITS'] = 64 - else: - default_variables['MSVS_OS_BITS'] = 32 + """Generated variables that require params to be known.""" + + generator_flags = params.get("generator_flags", {}) + + # Select project file format version (if unset, default to auto detecting). + msvs_version = MSVSVersion.SelectVisualStudioVersion( + generator_flags.get("msvs_version", "auto") + ) + # Stash msvs_version for later (so we don't have to probe the system twice). + params["msvs_version"] = msvs_version + + # Set a variable so conditions can be based on msvs_version. + default_variables["MSVS_VERSION"] = msvs_version.ShortName() + + # To determine processor word size on Windows, in addition to checking + # PROCESSOR_ARCHITECTURE (which reflects the word size of the current + # process), it is also necessary to check PROCESSOR_ARCITEW6432 (which + # contains the actual word size of the system when running thru WOW64). + if ( + os.environ.get("PROCESSOR_ARCHITECTURE", "").find("64") >= 0 + or os.environ.get("PROCESSOR_ARCHITEW6432", "").find("64") >= 0 + ): + default_variables["MSVS_OS_BITS"] = 64 + else: + default_variables["MSVS_OS_BITS"] = 32 - if gyp.common.GetFlavor(params) == 'ninja': - default_variables['SHARED_INTERMEDIATE_DIR'] = '$(OutDir)gen' + if gyp.common.GetFlavor(params) == "ninja": + default_variables["SHARED_INTERMEDIATE_DIR"] = "$(OutDir)gen" def PerformBuild(data, configurations, params): - options = params['options'] - msvs_version = params['msvs_version'] - devenv = os.path.join(msvs_version.path, 'Common7', 'IDE', 'devenv.com') - - for build_file, build_file_dict in data.items(): - (build_file_root, build_file_ext) = os.path.splitext(build_file) - if build_file_ext != '.gyp': - continue - sln_path = build_file_root + options.suffix + '.sln' - if options.generator_output: - sln_path = os.path.join(options.generator_output, sln_path) + options = params["options"] + msvs_version = params["msvs_version"] + devenv = os.path.join(msvs_version.path, "Common7", "IDE", "devenv.com") + + for build_file, build_file_dict in data.items(): + (build_file_root, build_file_ext) = os.path.splitext(build_file) + if build_file_ext != ".gyp": + continue + sln_path = build_file_root + options.suffix + ".sln" + if options.generator_output: + sln_path = os.path.join(options.generator_output, sln_path) - for config in configurations: - arguments = [devenv, sln_path, '/Build', config] - print('Building [%s]: %s' % (config, arguments)) - rtn = subprocess.check_call(arguments) + for config in configurations: + arguments = [devenv, sln_path, "/Build", config] + print("Building [%s]: %s" % (config, arguments)) + subprocess.check_call(arguments) def CalculateGeneratorInputInfo(params): - if params.get('flavor') == 'ninja': - toplevel = params['options'].toplevel_dir - qualified_out_dir = os.path.normpath(os.path.join( - toplevel, ninja_generator.ComputeOutputDir(params), - 'gypfiles-msvs-ninja')) - - global generator_filelist_paths - generator_filelist_paths = { - 'toplevel': toplevel, - 'qualified_out_dir': qualified_out_dir, - } + if params.get("flavor") == "ninja": + toplevel = params["options"].toplevel_dir + qualified_out_dir = os.path.normpath( + os.path.join( + toplevel, + ninja_generator.ComputeOutputDir(params), + "gypfiles-msvs-ninja", + ) + ) + + global generator_filelist_paths + generator_filelist_paths = { + "toplevel": toplevel, + "qualified_out_dir": qualified_out_dir, + } + def GenerateOutput(target_list, target_dicts, data, params): - """Generate .sln and .vcproj files. + """Generate .sln and .vcproj files. This is the entry point for this generator. Arguments: @@ -2014,82 +2133,87 @@ def GenerateOutput(target_list, target_dicts, data, params): target_dicts: Dict of target properties keyed on target pair. data: Dictionary containing per .gyp data. """ - global fixpath_prefix - - options = params['options'] - - # Get the project file format version back out of where we stashed it in - # GeneratorCalculatedVariables. - msvs_version = params['msvs_version'] - - generator_flags = params.get('generator_flags', {}) - - # Optionally shard targets marked with 'msvs_shard': SHARD_COUNT. - (target_list, target_dicts) = MSVSUtil.ShardTargets(target_list, target_dicts) - - # Optionally use the large PDB workaround for targets marked with - # 'msvs_large_pdb': 1. - (target_list, target_dicts) = MSVSUtil.InsertLargePdbShims( - target_list, target_dicts, generator_default_variables) - - # Optionally configure each spec to use ninja as the external builder. - if params.get('flavor') == 'ninja': - _InitNinjaFlavor(params, target_list, target_dicts) - - # Prepare the set of configurations. - configs = set() - for qualified_target in target_list: - spec = target_dicts[qualified_target] - for config_name, config in spec['configurations'].items(): - configs.add(_ConfigFullName(config_name, config)) - configs = list(configs) - - # Figure out all the projects that will be generated and their guids - project_objects = _CreateProjectObjects(target_list, target_dicts, options, - msvs_version) - - # Generate each project. - missing_sources = [] - for project in project_objects.values(): - fixpath_prefix = project.fixpath_prefix - missing_sources.extend(_GenerateProject(project, options, msvs_version, - generator_flags)) - fixpath_prefix = None - - for build_file in data: - # Validate build_file extension - if not build_file.endswith('.gyp'): - continue - sln_path = os.path.splitext(build_file)[0] + options.suffix + '.sln' - if options.generator_output: - sln_path = os.path.join(options.generator_output, sln_path) - # Get projects in the solution, and their dependents. - sln_projects = gyp.common.BuildFileTargets(target_list, build_file) - sln_projects += gyp.common.DeepDependencyTargets(target_dicts, sln_projects) - # Create folder hierarchy. - root_entries = _GatherSolutionFolders( - sln_projects, project_objects, flat=msvs_version.FlatSolution()) - # Create solution. - sln = MSVSNew.MSVSSolution(sln_path, - entries=root_entries, - variants=configs, - websiteProperties=False, - version=msvs_version) - sln.Write() - - if missing_sources: - error_message = "Missing input files:\n" + \ - '\n'.join(set(missing_sources)) - if generator_flags.get('msvs_error_on_missing_sources', False): - raise GypError(error_message) - else: - print("Warning: " + error_message, file=sys.stdout) + global fixpath_prefix + + options = params["options"] + + # Get the project file format version back out of where we stashed it in + # GeneratorCalculatedVariables. + msvs_version = params["msvs_version"] + + generator_flags = params.get("generator_flags", {}) + + # Optionally shard targets marked with 'msvs_shard': SHARD_COUNT. + (target_list, target_dicts) = MSVSUtil.ShardTargets(target_list, target_dicts) + + # Optionally use the large PDB workaround for targets marked with + # 'msvs_large_pdb': 1. + (target_list, target_dicts) = MSVSUtil.InsertLargePdbShims( + target_list, target_dicts, generator_default_variables + ) + + # Optionally configure each spec to use ninja as the external builder. + if params.get("flavor") == "ninja": + _InitNinjaFlavor(params, target_list, target_dicts) + + # Prepare the set of configurations. + configs = set() + for qualified_target in target_list: + spec = target_dicts[qualified_target] + for config_name, config in spec["configurations"].items(): + configs.add(_ConfigFullName(config_name, config)) + configs = list(configs) + + # Figure out all the projects that will be generated and their guids + project_objects = _CreateProjectObjects( + target_list, target_dicts, options, msvs_version + ) + + # Generate each project. + missing_sources = [] + for project in project_objects.values(): + fixpath_prefix = project.fixpath_prefix + missing_sources.extend( + _GenerateProject(project, options, msvs_version, generator_flags) + ) + fixpath_prefix = None + + for build_file in data: + # Validate build_file extension + if not build_file.endswith(".gyp"): + continue + sln_path = os.path.splitext(build_file)[0] + options.suffix + ".sln" + if options.generator_output: + sln_path = os.path.join(options.generator_output, sln_path) + # Get projects in the solution, and their dependents. + sln_projects = gyp.common.BuildFileTargets(target_list, build_file) + sln_projects += gyp.common.DeepDependencyTargets(target_dicts, sln_projects) + # Create folder hierarchy. + root_entries = _GatherSolutionFolders( + sln_projects, project_objects, flat=msvs_version.FlatSolution() + ) + # Create solution. + sln = MSVSNew.MSVSSolution( + sln_path, + entries=root_entries, + variants=configs, + websiteProperties=False, + version=msvs_version, + ) + sln.Write() + + if missing_sources: + error_message = "Missing input files:\n" + "\n".join(set(missing_sources)) + if generator_flags.get("msvs_error_on_missing_sources", False): + raise GypError(error_message) + else: + print("Warning: " + error_message, file=sys.stdout) -def _GenerateMSBuildFiltersFile(filters_path, source_files, - rule_dependencies, extension_to_rule_name, - platforms): - """Generate the filters file. +def _GenerateMSBuildFiltersFile( + filters_path, source_files, rule_dependencies, extension_to_rule_name, platforms +): + """Generate the filters file. This file is used by Visual Studio to organize the presentation of source files into folders. @@ -2099,29 +2223,43 @@ def _GenerateMSBuildFiltersFile(filters_path, source_files, source_files: The hierarchical structure of all the sources. extension_to_rule_name: A dictionary mapping file extensions to rules. """ - filter_group = [] - source_group = [] - _AppendFiltersForMSBuild('', source_files, rule_dependencies, - extension_to_rule_name, platforms, - filter_group, source_group) - if filter_group: - content = ['Project', - {'ToolsVersion': '4.0', - 'xmlns': 'http://schemas.microsoft.com/developer/msbuild/2003' - }, - ['ItemGroup'] + filter_group, - ['ItemGroup'] + source_group - ] - easy_xml.WriteXmlIfChanged(content, filters_path, pretty=True, win32=True) - elif os.path.exists(filters_path): - # We don't need this filter anymore. Delete the old filter file. - os.unlink(filters_path) - - -def _AppendFiltersForMSBuild(parent_filter_name, sources, rule_dependencies, - extension_to_rule_name, platforms, - filter_group, source_group): - """Creates the list of filters and sources to be added in the filter file. + filter_group = [] + source_group = [] + _AppendFiltersForMSBuild( + "", + source_files, + rule_dependencies, + extension_to_rule_name, + platforms, + filter_group, + source_group, + ) + if filter_group: + content = [ + "Project", + { + "ToolsVersion": "4.0", + "xmlns": "http://schemas.microsoft.com/developer/msbuild/2003", + }, + ["ItemGroup"] + filter_group, + ["ItemGroup"] + source_group, + ] + easy_xml.WriteXmlIfChanged(content, filters_path, pretty=True, win32=True) + elif os.path.exists(filters_path): + # We don't need this filter anymore. Delete the old filter file. + os.unlink(filters_path) + + +def _AppendFiltersForMSBuild( + parent_filter_name, + sources, + rule_dependencies, + extension_to_rule_name, + platforms, + filter_group, + source_group, +): + """Creates the list of filters and sources to be added in the filter file. Args: parent_filter_name: The name of the filter under which the sources are @@ -2131,36 +2269,47 @@ def _AppendFiltersForMSBuild(parent_filter_name, sources, rule_dependencies, filter_group: The list to which filter entries will be appended. source_group: The list to which source entries will be appeneded. """ - for source in sources: - if isinstance(source, MSVSProject.Filter): - # We have a sub-filter. Create the name of that sub-filter. - if not parent_filter_name: - filter_name = source.name - else: - filter_name = '%s\\%s' % (parent_filter_name, source.name) - # Add the filter to the group. - filter_group.append( - ['Filter', {'Include': filter_name}, - ['UniqueIdentifier', MSVSNew.MakeGuid(source.name)]]) - # Recurse and add its dependents. - _AppendFiltersForMSBuild(filter_name, source.contents, - rule_dependencies, extension_to_rule_name, - platforms, filter_group, source_group) - else: - # It's a source. Create a source entry. - _, element = _MapFileToMsBuildSourceType(source, rule_dependencies, - extension_to_rule_name, - platforms) - source_entry = [element, {'Include': source}] - # Specify the filter it is part of, if any. - if parent_filter_name: - source_entry.append(['Filter', parent_filter_name]) - source_group.append(source_entry) - - -def _MapFileToMsBuildSourceType(source, rule_dependencies, - extension_to_rule_name, platforms): - """Returns the group and element type of the source file. + for source in sources: + if isinstance(source, MSVSProject.Filter): + # We have a sub-filter. Create the name of that sub-filter. + if not parent_filter_name: + filter_name = source.name + else: + filter_name = "%s\\%s" % (parent_filter_name, source.name) + # Add the filter to the group. + filter_group.append( + [ + "Filter", + {"Include": filter_name}, + ["UniqueIdentifier", MSVSNew.MakeGuid(source.name)], + ] + ) + # Recurse and add its dependents. + _AppendFiltersForMSBuild( + filter_name, + source.contents, + rule_dependencies, + extension_to_rule_name, + platforms, + filter_group, + source_group, + ) + else: + # It's a source. Create a source entry. + _, element = _MapFileToMsBuildSourceType( + source, rule_dependencies, extension_to_rule_name, platforms + ) + source_entry = [element, {"Include": source}] + # Specify the filter it is part of, if any. + if parent_filter_name: + source_entry.append(["Filter", parent_filter_name]) + source_group.append(source_entry) + + +def _MapFileToMsBuildSourceType( + source, rule_dependencies, extension_to_rule_name, platforms +): + """Returns the group and element type of the source file. Arguments: source: The source file name. @@ -2169,85 +2318,93 @@ def _MapFileToMsBuildSourceType(source, rule_dependencies, Returns: A pair of (group this file should be part of, the label of element) """ - _, ext = os.path.splitext(source) - ext = ext.lower() - if ext in extension_to_rule_name: - group = 'rule' - element = extension_to_rule_name[ext] - elif ext in ['.cc', '.cpp', '.c', '.cxx', '.mm']: - group = 'compile' - element = 'ClCompile' - elif ext in ['.h', '.hxx']: - group = 'include' - element = 'ClInclude' - elif ext == '.rc': - group = 'resource' - element = 'ResourceCompile' - elif ext in ['.s', '.asm']: - group = 'masm' - element = 'MASM' - for platform in platforms: - if platform.lower() in ['arm', 'arm64']: - element = 'MARMASM' - elif ext == '.idl': - group = 'midl' - element = 'Midl' - elif source in rule_dependencies: - group = 'rule_dependency' - element = 'CustomBuild' - else: - group = 'none' - element = 'None' - return (group, element) - - -def _GenerateRulesForMSBuild(output_dir, options, spec, - sources, excluded_sources, - props_files_of_rules, targets_files_of_rules, - actions_to_add, rule_dependencies, - extension_to_rule_name): - # MSBuild rules are implemented using three files: an XML file, a .targets - # file and a .props file. - # See http://blogs.msdn.com/b/vcblog/archive/2010/04/21/quick-help-on-vs2010-custom-build-rule.aspx - # for more details. - rules = spec.get('rules', []) - rules_native = [r for r in rules if not int(r.get('msvs_external_rule', 0))] - rules_external = [r for r in rules if int(r.get('msvs_external_rule', 0))] - - msbuild_rules = [] - for rule in rules_native: - # Skip a rule with no action and no inputs. - if 'action' not in rule and not rule.get('rule_sources', []): - continue - msbuild_rule = MSBuildRule(rule, spec) - msbuild_rules.append(msbuild_rule) - rule_dependencies.update(msbuild_rule.additional_dependencies.split(';')) - extension_to_rule_name[msbuild_rule.extension] = msbuild_rule.rule_name - if msbuild_rules: - base = spec['target_name'] + options.suffix - props_name = base + '.props' - targets_name = base + '.targets' - xml_name = base + '.xml' - - props_files_of_rules.add(props_name) - targets_files_of_rules.add(targets_name) - - props_path = os.path.join(output_dir, props_name) - targets_path = os.path.join(output_dir, targets_name) - xml_path = os.path.join(output_dir, xml_name) - - _GenerateMSBuildRulePropsFile(props_path, msbuild_rules) - _GenerateMSBuildRuleTargetsFile(targets_path, msbuild_rules) - _GenerateMSBuildRuleXmlFile(xml_path, msbuild_rules) - - if rules_external: - _GenerateExternalRules(rules_external, output_dir, spec, - sources, options, actions_to_add) - _AdjustSourcesForRules(rules, sources, excluded_sources, True) + _, ext = os.path.splitext(source) + ext = ext.lower() + if ext in extension_to_rule_name: + group = "rule" + element = extension_to_rule_name[ext] + elif ext in [".cc", ".cpp", ".c", ".cxx", ".mm"]: + group = "compile" + element = "ClCompile" + elif ext in [".h", ".hxx"]: + group = "include" + element = "ClInclude" + elif ext == ".rc": + group = "resource" + element = "ResourceCompile" + elif ext in [".s", ".asm"]: + group = "masm" + element = "MASM" + for platform in platforms: + if platform.lower() in ["arm", "arm64"]: + element = "MARMASM" + elif ext == ".idl": + group = "midl" + element = "Midl" + elif source in rule_dependencies: + group = "rule_dependency" + element = "CustomBuild" + else: + group = "none" + element = "None" + return (group, element) + + +def _GenerateRulesForMSBuild( + output_dir, + options, + spec, + sources, + excluded_sources, + props_files_of_rules, + targets_files_of_rules, + actions_to_add, + rule_dependencies, + extension_to_rule_name, +): + # MSBuild rules are implemented using three files: an XML file, a .targets + # file and a .props file. + # See http://blogs.msdn.com/b/vcblog/archive/2010/04/21/quick-help-on-vs2010-custom-build-rule.aspx + # for more details. + rules = spec.get("rules", []) + rules_native = [r for r in rules if not int(r.get("msvs_external_rule", 0))] + rules_external = [r for r in rules if int(r.get("msvs_external_rule", 0))] + + msbuild_rules = [] + for rule in rules_native: + # Skip a rule with no action and no inputs. + if "action" not in rule and not rule.get("rule_sources", []): + continue + msbuild_rule = MSBuildRule(rule, spec) + msbuild_rules.append(msbuild_rule) + rule_dependencies.update(msbuild_rule.additional_dependencies.split(";")) + extension_to_rule_name[msbuild_rule.extension] = msbuild_rule.rule_name + if msbuild_rules: + base = spec["target_name"] + options.suffix + props_name = base + ".props" + targets_name = base + ".targets" + xml_name = base + ".xml" + + props_files_of_rules.add(props_name) + targets_files_of_rules.add(targets_name) + + props_path = os.path.join(output_dir, props_name) + targets_path = os.path.join(output_dir, targets_name) + xml_path = os.path.join(output_dir, xml_name) + + _GenerateMSBuildRulePropsFile(props_path, msbuild_rules) + _GenerateMSBuildRuleTargetsFile(targets_path, msbuild_rules) + _GenerateMSBuildRuleXmlFile(xml_path, msbuild_rules) + + if rules_external: + _GenerateExternalRules( + rules_external, output_dir, spec, sources, options, actions_to_add + ) + _AdjustSourcesForRules(rules, sources, excluded_sources, True) class MSBuildRule(object): - """Used to store information used to generate an MSBuild rule. + """Used to store information used to generate an MSBuild rule. Attributes: rule_name: The rule name, sanitized to use in XML. @@ -2266,718 +2423,842 @@ class MSBuildRule(object): command: The command used to run the rule. """ - def __init__(self, rule, spec): - self.display_name = rule['rule_name'] - # Assure that the rule name is only characters and numbers - self.rule_name = re.sub(r'\W', '_', self.display_name) - # Create the various element names, following the example set by the - # Visual Studio 2008 to 2010 conversion. I don't know if VS2010 - # is sensitive to the exact names. - self.target_name = '_' + self.rule_name - self.after_targets = self.rule_name + 'AfterTargets' - self.before_targets = self.rule_name + 'BeforeTargets' - self.depends_on = self.rule_name + 'DependsOn' - self.compute_output = 'Compute%sOutput' % self.rule_name - self.dirs_to_make = self.rule_name + 'DirsToMake' - self.inputs = self.rule_name + '_inputs' - self.tlog = self.rule_name + '_tlog' - self.extension = rule['extension'] - if not self.extension.startswith('.'): - self.extension = '.' + self.extension - - self.description = MSVSSettings.ConvertVCMacrosToMSBuild( - rule.get('message', self.rule_name)) - old_additional_dependencies = _FixPaths(rule.get('inputs', [])) - self.additional_dependencies = ( - ';'.join([MSVSSettings.ConvertVCMacrosToMSBuild(i) - for i in old_additional_dependencies])) - old_outputs = _FixPaths(rule.get('outputs', [])) - self.outputs = ';'.join([MSVSSettings.ConvertVCMacrosToMSBuild(i) - for i in old_outputs]) - old_command = _BuildCommandLineForRule(spec, rule, has_input_path=True, - do_setup_env=True) - self.command = MSVSSettings.ConvertVCMacrosToMSBuild(old_command) + def __init__(self, rule, spec): + self.display_name = rule["rule_name"] + # Assure that the rule name is only characters and numbers + self.rule_name = re.sub(r"\W", "_", self.display_name) + # Create the various element names, following the example set by the + # Visual Studio 2008 to 2010 conversion. I don't know if VS2010 + # is sensitive to the exact names. + self.target_name = "_" + self.rule_name + self.after_targets = self.rule_name + "AfterTargets" + self.before_targets = self.rule_name + "BeforeTargets" + self.depends_on = self.rule_name + "DependsOn" + self.compute_output = "Compute%sOutput" % self.rule_name + self.dirs_to_make = self.rule_name + "DirsToMake" + self.inputs = self.rule_name + "_inputs" + self.tlog = self.rule_name + "_tlog" + self.extension = rule["extension"] + if not self.extension.startswith("."): + self.extension = "." + self.extension + + self.description = MSVSSettings.ConvertVCMacrosToMSBuild( + rule.get("message", self.rule_name) + ) + old_additional_dependencies = _FixPaths(rule.get("inputs", [])) + self.additional_dependencies = ";".join( + [ + MSVSSettings.ConvertVCMacrosToMSBuild(i) + for i in old_additional_dependencies + ] + ) + old_outputs = _FixPaths(rule.get("outputs", [])) + self.outputs = ";".join( + [MSVSSettings.ConvertVCMacrosToMSBuild(i) for i in old_outputs] + ) + old_command = _BuildCommandLineForRule( + spec, rule, has_input_path=True, do_setup_env=True + ) + self.command = MSVSSettings.ConvertVCMacrosToMSBuild(old_command) def _GenerateMSBuildRulePropsFile(props_path, msbuild_rules): - """Generate the .props file.""" - content = ['Project', - {'xmlns': 'http://schemas.microsoft.com/developer/msbuild/2003'}] - for rule in msbuild_rules: - content.extend([ - ['PropertyGroup', - {'Condition': "'$(%s)' == '' and '$(%s)' == '' and " - "'$(ConfigurationType)' != 'Makefile'" % (rule.before_targets, - rule.after_targets) - }, - [rule.before_targets, 'Midl'], - [rule.after_targets, 'CustomBuild'], - ], - ['PropertyGroup', - [rule.depends_on, - {'Condition': "'$(ConfigurationType)' != 'Makefile'"}, - '_SelectedFiles;$(%s)' % rule.depends_on - ], - ], - ['ItemDefinitionGroup', - [rule.rule_name, - ['CommandLineTemplate', rule.command], - ['Outputs', rule.outputs], - ['ExecutionDescription', rule.description], - ['AdditionalDependencies', rule.additional_dependencies], - ], - ] - ]) - easy_xml.WriteXmlIfChanged(content, props_path, pretty=True, win32=True) + """Generate the .props file.""" + content = [ + "Project", + {"xmlns": "http://schemas.microsoft.com/developer/msbuild/2003"}, + ] + for rule in msbuild_rules: + content.extend( + [ + [ + "PropertyGroup", + { + "Condition": "'$(%s)' == '' and '$(%s)' == '' and " + "'$(ConfigurationType)' != 'Makefile'" + % (rule.before_targets, rule.after_targets) + }, + [rule.before_targets, "Midl"], + [rule.after_targets, "CustomBuild"], + ], + [ + "PropertyGroup", + [ + rule.depends_on, + {"Condition": "'$(ConfigurationType)' != 'Makefile'"}, + "_SelectedFiles;$(%s)" % rule.depends_on, + ], + ], + [ + "ItemDefinitionGroup", + [ + rule.rule_name, + ["CommandLineTemplate", rule.command], + ["Outputs", rule.outputs], + ["ExecutionDescription", rule.description], + ["AdditionalDependencies", rule.additional_dependencies], + ], + ], + ] + ) + easy_xml.WriteXmlIfChanged(content, props_path, pretty=True, win32=True) def _GenerateMSBuildRuleTargetsFile(targets_path, msbuild_rules): - """Generate the .targets file.""" - content = ['Project', - {'xmlns': 'http://schemas.microsoft.com/developer/msbuild/2003' - } - ] - item_group = [ - 'ItemGroup', - ['PropertyPageSchema', - {'Include': '$(MSBuildThisFileDirectory)$(MSBuildThisFileName).xml'} - ] - ] - for rule in msbuild_rules: - item_group.append( - ['AvailableItemName', - {'Include': rule.rule_name}, - ['Targets', rule.target_name], - ]) - content.append(item_group) - - for rule in msbuild_rules: - content.append( - ['UsingTask', - {'TaskName': rule.rule_name, - 'TaskFactory': 'XamlTaskFactory', - 'AssemblyName': 'Microsoft.Build.Tasks.v4.0' - }, - ['Task', '$(MSBuildThisFileDirectory)$(MSBuildThisFileName).xml'], - ]) - for rule in msbuild_rules: - rule_name = rule.rule_name - target_outputs = '%%(%s.Outputs)' % rule_name - target_inputs = ('%%(%s.Identity);%%(%s.AdditionalDependencies);' - '$(MSBuildProjectFile)') % (rule_name, rule_name) - rule_inputs = '%%(%s.Identity)' % rule_name - extension_condition = ("'%(Extension)'=='.obj' or " - "'%(Extension)'=='.res' or " - "'%(Extension)'=='.rsc' or " - "'%(Extension)'=='.lib'") - remove_section = [ - 'ItemGroup', - {'Condition': "'@(SelectedFiles)' != ''"}, - [rule_name, - {'Remove': '@(%s)' % rule_name, - 'Condition': "'%(Identity)' != '@(SelectedFiles)'" - } - ] - ] - inputs_section = [ - 'ItemGroup', - [rule.inputs, {'Include': '%%(%s.AdditionalDependencies)' % rule_name}] + """Generate the .targets file.""" + content = [ + "Project", + {"xmlns": "http://schemas.microsoft.com/developer/msbuild/2003"}, ] - logging_section = [ - 'ItemGroup', - [rule.tlog, - {'Include': '%%(%s.Outputs)' % rule_name, - 'Condition': ("'%%(%s.Outputs)' != '' and " - "'%%(%s.ExcludedFromBuild)' != 'true'" % - (rule_name, rule_name)) - }, - ['Source', "@(%s, '|')" % rule_name], - ['Inputs', "@(%s -> '%%(Fullpath)', ';')" % rule.inputs], + item_group = [ + "ItemGroup", + [ + "PropertyPageSchema", + {"Include": "$(MSBuildThisFileDirectory)$(MSBuildThisFileName).xml"}, ], ] - message_section = [ - 'Message', - {'Importance': 'High', - 'Text': '%%(%s.ExecutionDescription)' % rule_name - } - ] - write_tlog_section = [ - 'WriteLinesToFile', - {'Condition': "'@(%s)' != '' and '%%(%s.ExcludedFromBuild)' != " - "'true'" % (rule.tlog, rule.tlog), - 'File': '$(IntDir)$(ProjectName).write.1.tlog', - 'Lines': "^%%(%s.Source);@(%s->'%%(Fullpath)')" % (rule.tlog, - rule.tlog) - } - ] - read_tlog_section = [ - 'WriteLinesToFile', - {'Condition': "'@(%s)' != '' and '%%(%s.ExcludedFromBuild)' != " - "'true'" % (rule.tlog, rule.tlog), - 'File': '$(IntDir)$(ProjectName).read.1.tlog', - 'Lines': "^%%(%s.Source);%%(%s.Inputs)" % (rule.tlog, rule.tlog) - } - ] - command_and_input_section = [ - rule_name, - {'Condition': "'@(%s)' != '' and '%%(%s.ExcludedFromBuild)' != " - "'true'" % (rule_name, rule_name), - 'EchoOff': 'true', - 'StandardOutputImportance': 'High', - 'StandardErrorImportance': 'High', - 'CommandLineTemplate': '%%(%s.CommandLineTemplate)' % rule_name, - 'AdditionalOptions': '%%(%s.AdditionalOptions)' % rule_name, - 'Inputs': rule_inputs - } - ] - content.extend([ - ['Target', - {'Name': rule.target_name, - 'BeforeTargets': '$(%s)' % rule.before_targets, - 'AfterTargets': '$(%s)' % rule.after_targets, - 'Condition': "'@(%s)' != ''" % rule_name, - 'DependsOnTargets': '$(%s);%s' % (rule.depends_on, - rule.compute_output), - 'Outputs': target_outputs, - 'Inputs': target_inputs - }, - remove_section, - inputs_section, - logging_section, - message_section, - write_tlog_section, - read_tlog_section, - command_and_input_section, - ], - ['PropertyGroup', - ['ComputeLinkInputsTargets', - '$(ComputeLinkInputsTargets);', - '%s;' % rule.compute_output - ], - ['ComputeLibInputsTargets', - '$(ComputeLibInputsTargets);', - '%s;' % rule.compute_output - ], - ], - ['Target', - {'Name': rule.compute_output, - 'Condition': "'@(%s)' != ''" % rule_name - }, - ['ItemGroup', - [rule.dirs_to_make, - {'Condition': "'@(%s)' != '' and " - "'%%(%s.ExcludedFromBuild)' != 'true'" % (rule_name, rule_name), - 'Include': '%%(%s.Outputs)' % rule_name - } - ], - ['Link', - {'Include': '%%(%s.Identity)' % rule.dirs_to_make, - 'Condition': extension_condition - } - ], - ['Lib', - {'Include': '%%(%s.Identity)' % rule.dirs_to_make, - 'Condition': extension_condition - } - ], - ['ImpLib', - {'Include': '%%(%s.Identity)' % rule.dirs_to_make, - 'Condition': extension_condition - } - ], - ], - ['MakeDir', - {'Directories': ("@(%s->'%%(RootDir)%%(Directory)')" % - rule.dirs_to_make) - } - ] - ], - ]) - easy_xml.WriteXmlIfChanged(content, targets_path, pretty=True, win32=True) + for rule in msbuild_rules: + item_group.append( + [ + "AvailableItemName", + {"Include": rule.rule_name}, + ["Targets", rule.target_name], + ] + ) + content.append(item_group) + + for rule in msbuild_rules: + content.append( + [ + "UsingTask", + { + "TaskName": rule.rule_name, + "TaskFactory": "XamlTaskFactory", + "AssemblyName": "Microsoft.Build.Tasks.v4.0", + }, + ["Task", "$(MSBuildThisFileDirectory)$(MSBuildThisFileName).xml"], + ] + ) + for rule in msbuild_rules: + rule_name = rule.rule_name + target_outputs = "%%(%s.Outputs)" % rule_name + target_inputs = ( + "%%(%s.Identity);%%(%s.AdditionalDependencies);" "$(MSBuildProjectFile)" + ) % (rule_name, rule_name) + rule_inputs = "%%(%s.Identity)" % rule_name + extension_condition = ( + "'%(Extension)'=='.obj' or " + "'%(Extension)'=='.res' or " + "'%(Extension)'=='.rsc' or " + "'%(Extension)'=='.lib'" + ) + remove_section = [ + "ItemGroup", + {"Condition": "'@(SelectedFiles)' != ''"}, + [ + rule_name, + { + "Remove": "@(%s)" % rule_name, + "Condition": "'%(Identity)' != '@(SelectedFiles)'", + }, + ], + ] + inputs_section = [ + "ItemGroup", + [rule.inputs, {"Include": "%%(%s.AdditionalDependencies)" % rule_name}], + ] + logging_section = [ + "ItemGroup", + [ + rule.tlog, + { + "Include": "%%(%s.Outputs)" % rule_name, + "Condition": ( + "'%%(%s.Outputs)' != '' and " + "'%%(%s.ExcludedFromBuild)' != 'true'" % (rule_name, rule_name) + ), + }, + ["Source", "@(%s, '|')" % rule_name], + ["Inputs", "@(%s -> '%%(Fullpath)', ';')" % rule.inputs], + ], + ] + message_section = [ + "Message", + {"Importance": "High", "Text": "%%(%s.ExecutionDescription)" % rule_name}, + ] + write_tlog_section = [ + "WriteLinesToFile", + { + "Condition": "'@(%s)' != '' and '%%(%s.ExcludedFromBuild)' != " + "'true'" % (rule.tlog, rule.tlog), + "File": "$(IntDir)$(ProjectName).write.1.tlog", + "Lines": "^%%(%s.Source);@(%s->'%%(Fullpath)')" + % (rule.tlog, rule.tlog), + }, + ] + read_tlog_section = [ + "WriteLinesToFile", + { + "Condition": "'@(%s)' != '' and '%%(%s.ExcludedFromBuild)' != " + "'true'" % (rule.tlog, rule.tlog), + "File": "$(IntDir)$(ProjectName).read.1.tlog", + "Lines": "^%%(%s.Source);%%(%s.Inputs)" % (rule.tlog, rule.tlog), + }, + ] + command_and_input_section = [ + rule_name, + { + "Condition": "'@(%s)' != '' and '%%(%s.ExcludedFromBuild)' != " + "'true'" % (rule_name, rule_name), + "EchoOff": "true", + "StandardOutputImportance": "High", + "StandardErrorImportance": "High", + "CommandLineTemplate": "%%(%s.CommandLineTemplate)" % rule_name, + "AdditionalOptions": "%%(%s.AdditionalOptions)" % rule_name, + "Inputs": rule_inputs, + }, + ] + content.extend( + [ + [ + "Target", + { + "Name": rule.target_name, + "BeforeTargets": "$(%s)" % rule.before_targets, + "AfterTargets": "$(%s)" % rule.after_targets, + "Condition": "'@(%s)' != ''" % rule_name, + "DependsOnTargets": "$(%s);%s" + % (rule.depends_on, rule.compute_output), + "Outputs": target_outputs, + "Inputs": target_inputs, + }, + remove_section, + inputs_section, + logging_section, + message_section, + write_tlog_section, + read_tlog_section, + command_and_input_section, + ], + [ + "PropertyGroup", + [ + "ComputeLinkInputsTargets", + "$(ComputeLinkInputsTargets);", + "%s;" % rule.compute_output, + ], + [ + "ComputeLibInputsTargets", + "$(ComputeLibInputsTargets);", + "%s;" % rule.compute_output, + ], + ], + [ + "Target", + { + "Name": rule.compute_output, + "Condition": "'@(%s)' != ''" % rule_name, + }, + [ + "ItemGroup", + [ + rule.dirs_to_make, + { + "Condition": "'@(%s)' != '' and " + "'%%(%s.ExcludedFromBuild)' != 'true'" + % (rule_name, rule_name), + "Include": "%%(%s.Outputs)" % rule_name, + }, + ], + [ + "Link", + { + "Include": "%%(%s.Identity)" % rule.dirs_to_make, + "Condition": extension_condition, + }, + ], + [ + "Lib", + { + "Include": "%%(%s.Identity)" % rule.dirs_to_make, + "Condition": extension_condition, + }, + ], + [ + "ImpLib", + { + "Include": "%%(%s.Identity)" % rule.dirs_to_make, + "Condition": extension_condition, + }, + ], + ], + [ + "MakeDir", + { + "Directories": ( + "@(%s->'%%(RootDir)%%(Directory)')" % rule.dirs_to_make + ) + }, + ], + ], + ] + ) + easy_xml.WriteXmlIfChanged(content, targets_path, pretty=True, win32=True) def _GenerateMSBuildRuleXmlFile(xml_path, msbuild_rules): - # Generate the .xml file - content = [ - 'ProjectSchemaDefinitions', - {'xmlns': ('clr-namespace:Microsoft.Build.Framework.XamlTypes;' - 'assembly=Microsoft.Build.Framework'), - 'xmlns:x': 'http://schemas.microsoft.com/winfx/2006/xaml', - 'xmlns:sys': 'clr-namespace:System;assembly=mscorlib', - 'xmlns:transformCallback': - 'Microsoft.Cpp.Dev10.ConvertPropertyCallback' - } - ] - for rule in msbuild_rules: - content.extend([ - ['Rule', - {'Name': rule.rule_name, - 'PageTemplate': 'tool', - 'DisplayName': rule.display_name, - 'Order': '200' - }, - ['Rule.DataSource', - ['DataSource', - {'Persistence': 'ProjectFile', - 'ItemType': rule.rule_name - } - ] - ], - ['Rule.Categories', - ['Category', - {'Name': 'General'}, - ['Category.DisplayName', - ['sys:String', 'General'], - ], - ], - ['Category', - {'Name': 'Command Line', - 'Subtype': 'CommandLine' - }, - ['Category.DisplayName', - ['sys:String', 'Command Line'], - ], - ], - ], - ['StringListProperty', - {'Name': 'Inputs', - 'Category': 'Command Line', - 'IsRequired': 'true', - 'Switch': ' ' - }, - ['StringListProperty.DataSource', - ['DataSource', - {'Persistence': 'ProjectFile', - 'ItemType': rule.rule_name, - 'SourceType': 'Item' - } - ] - ], - ], - ['StringProperty', - {'Name': 'CommandLineTemplate', - 'DisplayName': 'Command Line', - 'Visible': 'False', - 'IncludeInCommandLine': 'False' - } - ], - ['DynamicEnumProperty', - {'Name': rule.before_targets, - 'Category': 'General', - 'EnumProvider': 'Targets', - 'IncludeInCommandLine': 'False' - }, - ['DynamicEnumProperty.DisplayName', - ['sys:String', 'Execute Before'], - ], - ['DynamicEnumProperty.Description', - ['sys:String', 'Specifies the targets for the build customization' - ' to run before.' - ], - ], - ['DynamicEnumProperty.ProviderSettings', - ['NameValuePair', - {'Name': 'Exclude', - 'Value': '^%s|^Compute' % rule.before_targets - } - ] - ], - ['DynamicEnumProperty.DataSource', - ['DataSource', - {'Persistence': 'ProjectFile', - 'HasConfigurationCondition': 'true' - } - ] - ], - ], - ['DynamicEnumProperty', - {'Name': rule.after_targets, - 'Category': 'General', - 'EnumProvider': 'Targets', - 'IncludeInCommandLine': 'False' - }, - ['DynamicEnumProperty.DisplayName', - ['sys:String', 'Execute After'], - ], - ['DynamicEnumProperty.Description', - ['sys:String', ('Specifies the targets for the build customization' - ' to run after.') - ], - ], - ['DynamicEnumProperty.ProviderSettings', - ['NameValuePair', - {'Name': 'Exclude', - 'Value': '^%s|^Compute' % rule.after_targets - } - ] - ], - ['DynamicEnumProperty.DataSource', - ['DataSource', - {'Persistence': 'ProjectFile', - 'ItemType': '', - 'HasConfigurationCondition': 'true' - } - ] - ], - ], - ['StringListProperty', - {'Name': 'Outputs', - 'DisplayName': 'Outputs', - 'Visible': 'False', - 'IncludeInCommandLine': 'False' - } - ], - ['StringProperty', - {'Name': 'ExecutionDescription', - 'DisplayName': 'Execution Description', - 'Visible': 'False', - 'IncludeInCommandLine': 'False' - } - ], - ['StringListProperty', - {'Name': 'AdditionalDependencies', - 'DisplayName': 'Additional Dependencies', - 'IncludeInCommandLine': 'False', - 'Visible': 'false' - } - ], - ['StringProperty', - {'Subtype': 'AdditionalOptions', - 'Name': 'AdditionalOptions', - 'Category': 'Command Line' - }, - ['StringProperty.DisplayName', - ['sys:String', 'Additional Options'], - ], - ['StringProperty.Description', - ['sys:String', 'Additional Options'], - ], - ], - ], - ['ItemType', - {'Name': rule.rule_name, - 'DisplayName': rule.display_name - } - ], - ['FileExtension', - {'Name': '*' + rule.extension, - 'ContentType': rule.rule_name - } - ], - ['ContentType', - {'Name': rule.rule_name, - 'DisplayName': '', - 'ItemType': rule.rule_name - } - ] - ]) - easy_xml.WriteXmlIfChanged(content, xml_path, pretty=True, win32=True) + # Generate the .xml file + content = [ + "ProjectSchemaDefinitions", + { + "xmlns": ( + "clr-namespace:Microsoft.Build.Framework.XamlTypes;" + "assembly=Microsoft.Build.Framework" + ), + "xmlns:x": "http://schemas.microsoft.com/winfx/2006/xaml", + "xmlns:sys": "clr-namespace:System;assembly=mscorlib", + "xmlns:transformCallback": "Microsoft.Cpp.Dev10.ConvertPropertyCallback", + }, + ] + for rule in msbuild_rules: + content.extend( + [ + [ + "Rule", + { + "Name": rule.rule_name, + "PageTemplate": "tool", + "DisplayName": rule.display_name, + "Order": "200", + }, + [ + "Rule.DataSource", + [ + "DataSource", + {"Persistence": "ProjectFile", "ItemType": rule.rule_name}, + ], + ], + [ + "Rule.Categories", + [ + "Category", + {"Name": "General"}, + ["Category.DisplayName", ["sys:String", "General"]], + ], + [ + "Category", + {"Name": "Command Line", "Subtype": "CommandLine"}, + ["Category.DisplayName", ["sys:String", "Command Line"]], + ], + ], + [ + "StringListProperty", + { + "Name": "Inputs", + "Category": "Command Line", + "IsRequired": "true", + "Switch": " ", + }, + [ + "StringListProperty.DataSource", + [ + "DataSource", + { + "Persistence": "ProjectFile", + "ItemType": rule.rule_name, + "SourceType": "Item", + }, + ], + ], + ], + [ + "StringProperty", + { + "Name": "CommandLineTemplate", + "DisplayName": "Command Line", + "Visible": "False", + "IncludeInCommandLine": "False", + }, + ], + [ + "DynamicEnumProperty", + { + "Name": rule.before_targets, + "Category": "General", + "EnumProvider": "Targets", + "IncludeInCommandLine": "False", + }, + [ + "DynamicEnumProperty.DisplayName", + ["sys:String", "Execute Before"], + ], + [ + "DynamicEnumProperty.Description", + [ + "sys:String", + "Specifies the targets for the build customization" + " to run before.", + ], + ], + [ + "DynamicEnumProperty.ProviderSettings", + [ + "NameValuePair", + { + "Name": "Exclude", + "Value": "^%s|^Compute" % rule.before_targets, + }, + ], + ], + [ + "DynamicEnumProperty.DataSource", + [ + "DataSource", + { + "Persistence": "ProjectFile", + "HasConfigurationCondition": "true", + }, + ], + ], + ], + [ + "DynamicEnumProperty", + { + "Name": rule.after_targets, + "Category": "General", + "EnumProvider": "Targets", + "IncludeInCommandLine": "False", + }, + [ + "DynamicEnumProperty.DisplayName", + ["sys:String", "Execute After"], + ], + [ + "DynamicEnumProperty.Description", + [ + "sys:String", + ( + "Specifies the targets for the build customization" + " to run after." + ), + ], + ], + [ + "DynamicEnumProperty.ProviderSettings", + [ + "NameValuePair", + { + "Name": "Exclude", + "Value": "^%s|^Compute" % rule.after_targets, + }, + ], + ], + [ + "DynamicEnumProperty.DataSource", + [ + "DataSource", + { + "Persistence": "ProjectFile", + "ItemType": "", + "HasConfigurationCondition": "true", + }, + ], + ], + ], + [ + "StringListProperty", + { + "Name": "Outputs", + "DisplayName": "Outputs", + "Visible": "False", + "IncludeInCommandLine": "False", + }, + ], + [ + "StringProperty", + { + "Name": "ExecutionDescription", + "DisplayName": "Execution Description", + "Visible": "False", + "IncludeInCommandLine": "False", + }, + ], + [ + "StringListProperty", + { + "Name": "AdditionalDependencies", + "DisplayName": "Additional Dependencies", + "IncludeInCommandLine": "False", + "Visible": "false", + }, + ], + [ + "StringProperty", + { + "Subtype": "AdditionalOptions", + "Name": "AdditionalOptions", + "Category": "Command Line", + }, + [ + "StringProperty.DisplayName", + ["sys:String", "Additional Options"], + ], + [ + "StringProperty.Description", + ["sys:String", "Additional Options"], + ], + ], + ], + [ + "ItemType", + {"Name": rule.rule_name, "DisplayName": rule.display_name}, + ], + [ + "FileExtension", + {"Name": "*" + rule.extension, "ContentType": rule.rule_name}, + ], + [ + "ContentType", + { + "Name": rule.rule_name, + "DisplayName": "", + "ItemType": rule.rule_name, + }, + ], + ] + ) + easy_xml.WriteXmlIfChanged(content, xml_path, pretty=True, win32=True) def _GetConfigurationAndPlatform(name, settings): - configuration = name.rsplit('_', 1)[0] - platform = settings.get('msvs_configuration_platform', 'Win32') - return (configuration, platform) + configuration = name.rsplit("_", 1)[0] + platform = settings.get("msvs_configuration_platform", "Win32") + return (configuration, platform) def _GetConfigurationCondition(name, settings): - return (r"'$(Configuration)|$(Platform)'=='%s|%s'" % - _GetConfigurationAndPlatform(name, settings)) + return r"'$(Configuration)|$(Platform)'=='%s|%s'" % _GetConfigurationAndPlatform( + name, settings + ) def _GetMSBuildProjectConfigurations(configurations): - group = ['ItemGroup', {'Label': 'ProjectConfigurations'}] - for (name, settings) in sorted(configurations.items()): - configuration, platform = _GetConfigurationAndPlatform(name, settings) - designation = '%s|%s' % (configuration, platform) - group.append( - ['ProjectConfiguration', {'Include': designation}, - ['Configuration', configuration], - ['Platform', platform]]) - return [group] + group = ["ItemGroup", {"Label": "ProjectConfigurations"}] + for (name, settings) in sorted(configurations.items()): + configuration, platform = _GetConfigurationAndPlatform(name, settings) + designation = "%s|%s" % (configuration, platform) + group.append( + [ + "ProjectConfiguration", + {"Include": designation}, + ["Configuration", configuration], + ["Platform", platform], + ] + ) + return [group] def _GetMSBuildGlobalProperties(spec, version, guid, gyp_file_name): - namespace = os.path.splitext(gyp_file_name)[0] - properties = [ - ['PropertyGroup', {'Label': 'Globals'}, - ['ProjectGuid', guid], - ['Keyword', 'Win32Proj'], - ['RootNamespace', namespace], - ['IgnoreWarnCompileDuplicatedFilename', 'true'], - ] + namespace = os.path.splitext(gyp_file_name)[0] + properties = [ + [ + "PropertyGroup", + {"Label": "Globals"}, + ["ProjectGuid", guid], + ["Keyword", "Win32Proj"], + ["RootNamespace", namespace], + ["IgnoreWarnCompileDuplicatedFilename", "true"], + ] ] - if os.environ.get('PROCESSOR_ARCHITECTURE') == 'AMD64' or \ - os.environ.get('PROCESSOR_ARCHITEW6432') == 'AMD64': - properties[0].append(['PreferredToolArchitecture', 'x64']) - - if spec.get('msvs_target_platform_version'): - target_platform_version = spec.get('msvs_target_platform_version') - properties[0].append(['WindowsTargetPlatformVersion', - target_platform_version]) - if spec.get('msvs_target_platform_minversion'): - target_platform_minversion = spec.get('msvs_target_platform_minversion') - properties[0].append(['WindowsTargetPlatformMinVersion', - target_platform_minversion]) - else: - properties[0].append(['WindowsTargetPlatformMinVersion', - target_platform_version]) - - if spec.get('msvs_enable_winrt'): - properties[0].append(['DefaultLanguage', 'en-US']) - properties[0].append(['AppContainerApplication', 'true']) - if spec.get('msvs_application_type_revision'): - app_type_revision = spec.get('msvs_application_type_revision') - properties[0].append(['ApplicationTypeRevision', app_type_revision]) - else: - properties[0].append(['ApplicationTypeRevision', '8.1']) - if spec.get('msvs_enable_winphone'): - properties[0].append(['ApplicationType', 'Windows Phone']) - else: - properties[0].append(['ApplicationType', 'Windows Store']) - - platform_name = None - msvs_windows_sdk_version = None - for configuration in spec['configurations'].values(): - platform_name = platform_name or _ConfigPlatform(configuration) - msvs_windows_sdk_version = (msvs_windows_sdk_version or - _ConfigWindowsTargetPlatformVersion(configuration, version)) - if platform_name and msvs_windows_sdk_version: - break - if msvs_windows_sdk_version: - properties[0].append(['WindowsTargetPlatformVersion', - str(msvs_windows_sdk_version)]) - elif version.compatible_sdks: - raise GypError('%s requires any SDK of %s version, but none were found' % - (version.description, version.compatible_sdks)) - - if platform_name == 'ARM': - properties[0].append(['WindowsSDKDesktopARMSupport', 'true']) - - return properties + if ( + os.environ.get("PROCESSOR_ARCHITECTURE") == "AMD64" + or os.environ.get("PROCESSOR_ARCHITEW6432") == "AMD64" + ): + properties[0].append(["PreferredToolArchitecture", "x64"]) + + if spec.get("msvs_target_platform_version"): + target_platform_version = spec.get("msvs_target_platform_version") + properties[0].append(["WindowsTargetPlatformVersion", target_platform_version]) + if spec.get("msvs_target_platform_minversion"): + target_platform_minversion = spec.get("msvs_target_platform_minversion") + properties[0].append( + ["WindowsTargetPlatformMinVersion", target_platform_minversion] + ) + else: + properties[0].append( + ["WindowsTargetPlatformMinVersion", target_platform_version] + ) + + if spec.get("msvs_enable_winrt"): + properties[0].append(["DefaultLanguage", "en-US"]) + properties[0].append(["AppContainerApplication", "true"]) + if spec.get("msvs_application_type_revision"): + app_type_revision = spec.get("msvs_application_type_revision") + properties[0].append(["ApplicationTypeRevision", app_type_revision]) + else: + properties[0].append(["ApplicationTypeRevision", "8.1"]) + if spec.get("msvs_enable_winphone"): + properties[0].append(["ApplicationType", "Windows Phone"]) + else: + properties[0].append(["ApplicationType", "Windows Store"]) + + platform_name = None + msvs_windows_sdk_version = None + for configuration in spec["configurations"].values(): + platform_name = platform_name or _ConfigPlatform(configuration) + msvs_windows_sdk_version = ( + msvs_windows_sdk_version + or _ConfigWindowsTargetPlatformVersion(configuration, version) + ) + if platform_name and msvs_windows_sdk_version: + break + if msvs_windows_sdk_version: + properties[0].append( + ["WindowsTargetPlatformVersion", str(msvs_windows_sdk_version)] + ) + elif version.compatible_sdks: + raise GypError( + "%s requires any SDK of %s version, but none were found" + % (version.description, version.compatible_sdks) + ) + + if platform_name == "ARM": + properties[0].append(["WindowsSDKDesktopARMSupport", "true"]) + + return properties def _GetMSBuildConfigurationDetails(spec, build_file): - properties = {} - for name, settings in spec['configurations'].items(): - msbuild_attributes = _GetMSBuildAttributes(spec, settings, build_file) - condition = _GetConfigurationCondition(name, settings) - character_set = msbuild_attributes.get('CharacterSet') - config_type = msbuild_attributes.get('ConfigurationType') - _AddConditionalProperty(properties, condition, 'ConfigurationType', - config_type) - if config_type == 'Driver': - _AddConditionalProperty(properties, condition, 'DriverType', 'WDM') - _AddConditionalProperty(properties, condition, 'TargetVersion', - _ConfigTargetVersion(settings)) - if character_set: - if 'msvs_enable_winrt' not in spec : - _AddConditionalProperty(properties, condition, 'CharacterSet', - character_set) - return _GetMSBuildPropertyGroup(spec, 'Configuration', properties) + properties = {} + for name, settings in spec["configurations"].items(): + msbuild_attributes = _GetMSBuildAttributes(spec, settings, build_file) + condition = _GetConfigurationCondition(name, settings) + character_set = msbuild_attributes.get("CharacterSet") + config_type = msbuild_attributes.get("ConfigurationType") + _AddConditionalProperty(properties, condition, "ConfigurationType", config_type) + if config_type == "Driver": + _AddConditionalProperty(properties, condition, "DriverType", "WDM") + _AddConditionalProperty( + properties, condition, "TargetVersion", _ConfigTargetVersion(settings) + ) + if character_set: + if "msvs_enable_winrt" not in spec: + _AddConditionalProperty( + properties, condition, "CharacterSet", character_set + ) + return _GetMSBuildPropertyGroup(spec, "Configuration", properties) def _GetMSBuildLocalProperties(msbuild_toolset): - # Currently the only local property we support is PlatformToolset - properties = {} - if msbuild_toolset: - properties = [ - ['PropertyGroup', {'Label': 'Locals'}, - ['PlatformToolset', msbuild_toolset], + # Currently the only local property we support is PlatformToolset + properties = {} + if msbuild_toolset: + properties = [ + [ + "PropertyGroup", + {"Label": "Locals"}, + ["PlatformToolset", msbuild_toolset], + ] ] - ] - return properties + return properties def _GetMSBuildPropertySheets(configurations): - user_props = r'$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props' - additional_props = {} - props_specified = False - for name, settings in sorted(configurations.items()): - configuration = _GetConfigurationCondition(name, settings) - if 'msbuild_props' in settings: - additional_props[configuration] = _FixPaths(settings['msbuild_props']) - props_specified = True - else: - additional_props[configuration] = '' - - if not props_specified: - return [ - ['ImportGroup', - {'Label': 'PropertySheets'}, - ['Import', - {'Project': user_props, - 'Condition': "exists('%s')" % user_props, - 'Label': 'LocalAppDataPlatform' - } - ] - ] - ] - else: - sheets = [] - for condition, props in additional_props.items(): - import_group = [ - 'ImportGroup', - {'Label': 'PropertySheets', - 'Condition': condition - }, - ['Import', - {'Project': user_props, - 'Condition': "exists('%s')" % user_props, - 'Label': 'LocalAppDataPlatform' - } + user_props = r"$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" + additional_props = {} + props_specified = False + for name, settings in sorted(configurations.items()): + configuration = _GetConfigurationCondition(name, settings) + if "msbuild_props" in settings: + additional_props[configuration] = _FixPaths(settings["msbuild_props"]) + props_specified = True + else: + additional_props[configuration] = "" + + if not props_specified: + return [ + [ + "ImportGroup", + {"Label": "PropertySheets"}, + [ + "Import", + { + "Project": user_props, + "Condition": "exists('%s')" % user_props, + "Label": "LocalAppDataPlatform", + }, + ], + ] ] - ] - for props_file in props: - import_group.append(['Import', {'Project':props_file}]) - sheets.append(import_group) - return sheets + else: + sheets = [] + for condition, props in additional_props.items(): + import_group = [ + "ImportGroup", + {"Label": "PropertySheets", "Condition": condition}, + [ + "Import", + { + "Project": user_props, + "Condition": "exists('%s')" % user_props, + "Label": "LocalAppDataPlatform", + }, + ], + ] + for props_file in props: + import_group.append(["Import", {"Project": props_file}]) + sheets.append(import_group) + return sheets + def _ConvertMSVSBuildAttributes(spec, config, build_file): - config_type = _GetMSVSConfigurationType(spec, build_file) - msvs_attributes = _GetMSVSAttributes(spec, config, config_type) - msbuild_attributes = {} - for a in msvs_attributes: - if a in ['IntermediateDirectory', 'OutputDirectory']: - directory = MSVSSettings.ConvertVCMacrosToMSBuild(msvs_attributes[a]) - if not directory.endswith('\\'): - directory += '\\' - msbuild_attributes[a] = directory - elif a == 'CharacterSet': - msbuild_attributes[a] = _ConvertMSVSCharacterSet(msvs_attributes[a]) - elif a == 'ConfigurationType': - msbuild_attributes[a] = _ConvertMSVSConfigurationType(msvs_attributes[a]) - else: - print('Warning: Do not know how to convert MSVS attribute ' + a) - return msbuild_attributes + config_type = _GetMSVSConfigurationType(spec, build_file) + msvs_attributes = _GetMSVSAttributes(spec, config, config_type) + msbuild_attributes = {} + for a in msvs_attributes: + if a in ["IntermediateDirectory", "OutputDirectory"]: + directory = MSVSSettings.ConvertVCMacrosToMSBuild(msvs_attributes[a]) + if not directory.endswith("\\"): + directory += "\\" + msbuild_attributes[a] = directory + elif a == "CharacterSet": + msbuild_attributes[a] = _ConvertMSVSCharacterSet(msvs_attributes[a]) + elif a == "ConfigurationType": + msbuild_attributes[a] = _ConvertMSVSConfigurationType(msvs_attributes[a]) + else: + print("Warning: Do not know how to convert MSVS attribute " + a) + return msbuild_attributes def _ConvertMSVSCharacterSet(char_set): - if char_set.isdigit(): - char_set = { - '0': 'MultiByte', - '1': 'Unicode', - '2': 'MultiByte', - }[char_set] - return char_set + if char_set.isdigit(): + char_set = {"0": "MultiByte", "1": "Unicode", "2": "MultiByte"}[char_set] + return char_set def _ConvertMSVSConfigurationType(config_type): - if config_type.isdigit(): - config_type = { - '1': 'Application', - '2': 'DynamicLibrary', - '4': 'StaticLibrary', - '5': 'Driver', - '10': 'Utility' - }[config_type] - return config_type + if config_type.isdigit(): + config_type = { + "1": "Application", + "2": "DynamicLibrary", + "4": "StaticLibrary", + "5": "Driver", + "10": "Utility", + }[config_type] + return config_type def _GetMSBuildAttributes(spec, config, build_file): - if 'msbuild_configuration_attributes' not in config: - msbuild_attributes = _ConvertMSVSBuildAttributes(spec, config, build_file) + if "msbuild_configuration_attributes" not in config: + msbuild_attributes = _ConvertMSVSBuildAttributes(spec, config, build_file) - else: - config_type = _GetMSVSConfigurationType(spec, build_file) - config_type = _ConvertMSVSConfigurationType(config_type) - msbuild_attributes = config.get('msbuild_configuration_attributes', {}) - msbuild_attributes.setdefault('ConfigurationType', config_type) - output_dir = msbuild_attributes.get('OutputDirectory', - '$(SolutionDir)$(Configuration)') - msbuild_attributes['OutputDirectory'] = _FixPath(output_dir) + '\\' - if 'IntermediateDirectory' not in msbuild_attributes: - intermediate = _FixPath('$(Configuration)') + '\\' - msbuild_attributes['IntermediateDirectory'] = intermediate - if 'CharacterSet' in msbuild_attributes: - msbuild_attributes['CharacterSet'] = _ConvertMSVSCharacterSet( - msbuild_attributes['CharacterSet']) - if 'TargetName' not in msbuild_attributes: - prefix = spec.get('product_prefix', '') - product_name = spec.get('product_name', '$(ProjectName)') - target_name = prefix + product_name - msbuild_attributes['TargetName'] = target_name - if 'TargetExt' not in msbuild_attributes and 'product_extension' in spec: - ext = spec.get('product_extension') - msbuild_attributes['TargetExt'] = '.' + ext - - if spec.get('msvs_external_builder'): - external_out_dir = spec.get('msvs_external_builder_out_dir', '.') - msbuild_attributes['OutputDirectory'] = _FixPath(external_out_dir) + '\\' - - # Make sure that 'TargetPath' matches 'Lib.OutputFile' or 'Link.OutputFile' - # (depending on the tool used) to avoid MSB8012 warning. - msbuild_tool_map = { - 'executable': 'Link', - 'shared_library': 'Link', - 'loadable_module': 'Link', - 'windows_driver': 'Link', - 'static_library': 'Lib', - } - msbuild_tool = msbuild_tool_map.get(spec['type']) - if msbuild_tool: - msbuild_settings = config['finalized_msbuild_settings'] - out_file = msbuild_settings[msbuild_tool].get('OutputFile') - if out_file: - msbuild_attributes['TargetPath'] = _FixPath(out_file) - target_ext = msbuild_settings[msbuild_tool].get('TargetExt') - if target_ext: - msbuild_attributes['TargetExt'] = target_ext + else: + config_type = _GetMSVSConfigurationType(spec, build_file) + config_type = _ConvertMSVSConfigurationType(config_type) + msbuild_attributes = config.get("msbuild_configuration_attributes", {}) + msbuild_attributes.setdefault("ConfigurationType", config_type) + output_dir = msbuild_attributes.get( + "OutputDirectory", "$(SolutionDir)$(Configuration)" + ) + msbuild_attributes["OutputDirectory"] = _FixPath(output_dir) + "\\" + if "IntermediateDirectory" not in msbuild_attributes: + intermediate = _FixPath("$(Configuration)") + "\\" + msbuild_attributes["IntermediateDirectory"] = intermediate + if "CharacterSet" in msbuild_attributes: + msbuild_attributes["CharacterSet"] = _ConvertMSVSCharacterSet( + msbuild_attributes["CharacterSet"] + ) + if "TargetName" not in msbuild_attributes: + prefix = spec.get("product_prefix", "") + product_name = spec.get("product_name", "$(ProjectName)") + target_name = prefix + product_name + msbuild_attributes["TargetName"] = target_name + if "TargetExt" not in msbuild_attributes and "product_extension" in spec: + ext = spec.get("product_extension") + msbuild_attributes["TargetExt"] = "." + ext + + if spec.get("msvs_external_builder"): + external_out_dir = spec.get("msvs_external_builder_out_dir", ".") + msbuild_attributes["OutputDirectory"] = _FixPath(external_out_dir) + "\\" + + # Make sure that 'TargetPath' matches 'Lib.OutputFile' or 'Link.OutputFile' + # (depending on the tool used) to avoid MSB8012 warning. + msbuild_tool_map = { + "executable": "Link", + "shared_library": "Link", + "loadable_module": "Link", + "windows_driver": "Link", + "static_library": "Lib", + } + msbuild_tool = msbuild_tool_map.get(spec["type"]) + if msbuild_tool: + msbuild_settings = config["finalized_msbuild_settings"] + out_file = msbuild_settings[msbuild_tool].get("OutputFile") + if out_file: + msbuild_attributes["TargetPath"] = _FixPath(out_file) + target_ext = msbuild_settings[msbuild_tool].get("TargetExt") + if target_ext: + msbuild_attributes["TargetExt"] = target_ext - return msbuild_attributes + return msbuild_attributes def _GetMSBuildConfigurationGlobalProperties(spec, configurations, build_file): - # TODO(jeanluc) We could optimize out the following and do it only if - # there are actions. - # TODO(jeanluc) Handle the equivalent of setting 'CYGWIN=nontsec'. - new_paths = [] - cygwin_dirs = spec.get('msvs_cygwin_dirs', ['.'])[0] - if cygwin_dirs: - cyg_path = '$(MSBuildProjectDirectory)\\%s\\bin\\' % _FixPath(cygwin_dirs) - new_paths.append(cyg_path) - # TODO(jeanluc) Change the convention to have both a cygwin_dir and a - # python_dir. - python_path = cyg_path.replace('cygwin\\bin', 'python_26') - new_paths.append(python_path) - if new_paths: - new_paths = '$(ExecutablePath);' + ';'.join(new_paths) - - properties = {} - for (name, configuration) in sorted(configurations.items()): - condition = _GetConfigurationCondition(name, configuration) - attributes = _GetMSBuildAttributes(spec, configuration, build_file) - msbuild_settings = configuration['finalized_msbuild_settings'] - _AddConditionalProperty(properties, condition, 'IntDir', - attributes['IntermediateDirectory']) - _AddConditionalProperty(properties, condition, 'OutDir', - attributes['OutputDirectory']) - _AddConditionalProperty(properties, condition, 'TargetName', - attributes['TargetName']) - if 'TargetExt' in attributes: - _AddConditionalProperty(properties, condition, 'TargetExt', - attributes['TargetExt']) - - if attributes.get('TargetPath'): - _AddConditionalProperty(properties, condition, 'TargetPath', - attributes['TargetPath']) - if attributes.get('TargetExt'): - _AddConditionalProperty(properties, condition, 'TargetExt', - attributes['TargetExt']) - - if new_paths: - _AddConditionalProperty(properties, condition, 'ExecutablePath', - new_paths) - tool_settings = msbuild_settings.get('', {}) - for name, value in sorted(tool_settings.items()): - formatted_value = _GetValueFormattedForMSBuild('', name, value) - _AddConditionalProperty(properties, condition, name, formatted_value) - return _GetMSBuildPropertyGroup(spec, None, properties) + # TODO(jeanluc) We could optimize out the following and do it only if + # there are actions. + # TODO(jeanluc) Handle the equivalent of setting 'CYGWIN=nontsec'. + new_paths = [] + cygwin_dirs = spec.get("msvs_cygwin_dirs", ["."])[0] + if cygwin_dirs: + cyg_path = "$(MSBuildProjectDirectory)\\%s\\bin\\" % _FixPath(cygwin_dirs) + new_paths.append(cyg_path) + # TODO(jeanluc) Change the convention to have both a cygwin_dir and a + # python_dir. + python_path = cyg_path.replace("cygwin\\bin", "python_26") + new_paths.append(python_path) + if new_paths: + new_paths = "$(ExecutablePath);" + ";".join(new_paths) + + properties = {} + for (name, configuration) in sorted(configurations.items()): + condition = _GetConfigurationCondition(name, configuration) + attributes = _GetMSBuildAttributes(spec, configuration, build_file) + msbuild_settings = configuration["finalized_msbuild_settings"] + _AddConditionalProperty( + properties, condition, "IntDir", attributes["IntermediateDirectory"] + ) + _AddConditionalProperty( + properties, condition, "OutDir", attributes["OutputDirectory"] + ) + _AddConditionalProperty( + properties, condition, "TargetName", attributes["TargetName"] + ) + if "TargetExt" in attributes: + _AddConditionalProperty( + properties, condition, "TargetExt", attributes["TargetExt"] + ) + + if attributes.get("TargetPath"): + _AddConditionalProperty( + properties, condition, "TargetPath", attributes["TargetPath"] + ) + if attributes.get("TargetExt"): + _AddConditionalProperty( + properties, condition, "TargetExt", attributes["TargetExt"] + ) + + if new_paths: + _AddConditionalProperty(properties, condition, "ExecutablePath", new_paths) + tool_settings = msbuild_settings.get("", {}) + for name, value in sorted(tool_settings.items()): + formatted_value = _GetValueFormattedForMSBuild("", name, value) + _AddConditionalProperty(properties, condition, name, formatted_value) + return _GetMSBuildPropertyGroup(spec, None, properties) def _AddConditionalProperty(properties, condition, name, value): - """Adds a property / conditional value pair to a dictionary. + """Adds a property / conditional value pair to a dictionary. Arguments: properties: The dictionary to be modified. The key is the name of the @@ -2987,21 +3268,21 @@ def _AddConditionalProperty(properties, condition, name, value): name: The name of the property. value: The value of the property. """ - if name not in properties: - properties[name] = {} - values = properties[name] - if value not in values: - values[value] = [] - conditions = values[value] - conditions.append(condition) + if name not in properties: + properties[name] = {} + values = properties[name] + if value not in values: + values[value] = [] + conditions = values[value] + conditions.append(condition) # Regex for msvs variable references ( i.e. $(FOO) ). -MSVS_VARIABLE_REFERENCE = re.compile(r'\$\(([a-zA-Z_][a-zA-Z0-9_]*)\)') +MSVS_VARIABLE_REFERENCE = re.compile(r"\$\(([a-zA-Z_][a-zA-Z0-9_]*)\)") def _GetMSBuildPropertyGroup(spec, label, properties): - """Returns a PropertyGroup definition for the specified properties. + """Returns a PropertyGroup definition for the specified properties. Arguments: spec: The target project dict. @@ -3010,191 +3291,215 @@ def _GetMSBuildPropertyGroup(spec, label, properties): property. The value is itself a dictionary; its key is the value and the value a list of condition for which this value is true. """ - group = ['PropertyGroup'] - if label: - group.append({'Label': label}) - num_configurations = len(spec['configurations']) - def GetEdges(node): - # Use a definition of edges such that user_of_variable -> used_varible. - # This happens to be easier in this case, since a variable's - # definition contains all variables it references in a single string. - edges = set() - for value in sorted(properties[node].keys()): - # Add to edges all $(...) references to variables. - # - # Variable references that refer to names not in properties are excluded - # These can exist for instance to refer built in definitions like - # $(SolutionDir). - # - # Self references are ignored. Self reference is used in a few places to - # append to the default value. I.e. PATH=$(PATH);other_path - edges.update(set([v for v in MSVS_VARIABLE_REFERENCE.findall(value) - if v in properties and v != node])) - return edges - properties_ordered = gyp.common.TopologicallySorted( - properties.keys(), GetEdges) - # Walk properties in the reverse of a topological sort on - # user_of_variable -> used_variable as this ensures variables are - # defined before they are used. - # NOTE: reverse(topsort(DAG)) = topsort(reverse_edges(DAG)) - for name in reversed(properties_ordered): - values = properties[name] - for value, conditions in sorted(values.items()): - if len(conditions) == num_configurations: - # If the value is the same all configurations, - # just add one unconditional entry. - group.append([name, value]) - else: - for condition in conditions: - group.append([name, {'Condition': condition}, value]) - return [group] + group = ["PropertyGroup"] + if label: + group.append({"Label": label}) + num_configurations = len(spec["configurations"]) + + def GetEdges(node): + # Use a definition of edges such that user_of_variable -> used_varible. + # This happens to be easier in this case, since a variable's + # definition contains all variables it references in a single string. + edges = set() + for value in sorted(properties[node].keys()): + # Add to edges all $(...) references to variables. + # + # Variable references that refer to names not in properties are excluded + # These can exist for instance to refer built in definitions like + # $(SolutionDir). + # + # Self references are ignored. Self reference is used in a few places to + # append to the default value. I.e. PATH=$(PATH);other_path + edges.update( + set( + [ + v + for v in MSVS_VARIABLE_REFERENCE.findall(value) + if v in properties and v != node + ] + ) + ) + return edges + + properties_ordered = gyp.common.TopologicallySorted(properties.keys(), GetEdges) + # Walk properties in the reverse of a topological sort on + # user_of_variable -> used_variable as this ensures variables are + # defined before they are used. + # NOTE: reverse(topsort(DAG)) = topsort(reverse_edges(DAG)) + for name in reversed(properties_ordered): + values = properties[name] + for value, conditions in sorted(values.items()): + if len(conditions) == num_configurations: + # If the value is the same all configurations, + # just add one unconditional entry. + group.append([name, value]) + else: + for condition in conditions: + group.append([name, {"Condition": condition}, value]) + return [group] def _GetMSBuildToolSettingsSections(spec, configurations): - groups = [] - for (name, configuration) in sorted(configurations.items()): - msbuild_settings = configuration['finalized_msbuild_settings'] - group = ['ItemDefinitionGroup', - {'Condition': _GetConfigurationCondition(name, configuration)} - ] - for tool_name, tool_settings in sorted(msbuild_settings.items()): - # Skip the tool named '' which is a holder of global settings handled - # by _GetMSBuildConfigurationGlobalProperties. - if tool_name: - if tool_settings: - tool = [tool_name] - for name, value in sorted(tool_settings.items()): - formatted_value = _GetValueFormattedForMSBuild(tool_name, name, - value) - tool.append([name, formatted_value]) - group.append(tool) - groups.append(group) - return groups + groups = [] + for (name, configuration) in sorted(configurations.items()): + msbuild_settings = configuration["finalized_msbuild_settings"] + group = [ + "ItemDefinitionGroup", + {"Condition": _GetConfigurationCondition(name, configuration)}, + ] + for tool_name, tool_settings in sorted(msbuild_settings.items()): + # Skip the tool named '' which is a holder of global settings handled + # by _GetMSBuildConfigurationGlobalProperties. + if tool_name: + if tool_settings: + tool = [tool_name] + for name, value in sorted(tool_settings.items()): + formatted_value = _GetValueFormattedForMSBuild( + tool_name, name, value + ) + tool.append([name, formatted_value]) + group.append(tool) + groups.append(group) + return groups def _FinalizeMSBuildSettings(spec, configuration): - if 'msbuild_settings' in configuration: - converted = False - msbuild_settings = configuration['msbuild_settings'] - MSVSSettings.ValidateMSBuildSettings(msbuild_settings) - else: - converted = True - msvs_settings = configuration.get('msvs_settings', {}) - msbuild_settings = MSVSSettings.ConvertToMSBuildSettings(msvs_settings) - include_dirs, midl_include_dirs, resource_include_dirs = \ - _GetIncludeDirs(configuration) - libraries = _GetLibraries(spec) - library_dirs = _GetLibraryDirs(configuration) - out_file, _, msbuild_tool = _GetOutputFilePathAndTool(spec, msbuild=True) - target_ext = _GetOutputTargetExt(spec) - defines = _GetDefines(configuration) - if converted: - # Visual Studio 2010 has TR1 - defines = [d for d in defines if d != '_HAS_TR1=0'] - # Warn of ignored settings - ignored_settings = ['msvs_tool_files'] - for ignored_setting in ignored_settings: - value = configuration.get(ignored_setting) - if value: - print('Warning: The automatic conversion to MSBuild does not handle ' - '%s. Ignoring setting of %s' % (ignored_setting, str(value))) - - defines = [_EscapeCppDefineForMSBuild(d) for d in defines] - disabled_warnings = _GetDisabledWarnings(configuration) - prebuild = configuration.get('msvs_prebuild') - postbuild = configuration.get('msvs_postbuild') - def_file = _GetModuleDefinition(spec) - precompiled_header = configuration.get('msvs_precompiled_header') - - # Add the information to the appropriate tool - # TODO(jeanluc) We could optimize and generate these settings only if - # the corresponding files are found, e.g. don't generate ResourceCompile - # if you don't have any resources. - _ToolAppend(msbuild_settings, 'ClCompile', - 'AdditionalIncludeDirectories', include_dirs) - _ToolAppend(msbuild_settings, 'Midl', - 'AdditionalIncludeDirectories', midl_include_dirs) - _ToolAppend(msbuild_settings, 'ResourceCompile', - 'AdditionalIncludeDirectories', resource_include_dirs) - # Add in libraries, note that even for empty libraries, we want this - # set, to prevent inheriting default libraries from the environment. - _ToolSetOrAppend(msbuild_settings, 'Link', 'AdditionalDependencies', - libraries) - _ToolAppend(msbuild_settings, 'Link', 'AdditionalLibraryDirectories', - library_dirs) - if out_file: - _ToolAppend(msbuild_settings, msbuild_tool, 'OutputFile', out_file, - only_if_unset=True) - if target_ext: - _ToolAppend(msbuild_settings, msbuild_tool, 'TargetExt', target_ext, - only_if_unset=True) - # Add defines. - _ToolAppend(msbuild_settings, 'ClCompile', - 'PreprocessorDefinitions', defines) - _ToolAppend(msbuild_settings, 'ResourceCompile', - 'PreprocessorDefinitions', defines) - # Add disabled warnings. - _ToolAppend(msbuild_settings, 'ClCompile', - 'DisableSpecificWarnings', disabled_warnings) - # Turn on precompiled headers if appropriate. - if precompiled_header: - precompiled_header = os.path.split(precompiled_header)[1] - _ToolAppend(msbuild_settings, 'ClCompile', 'PrecompiledHeader', 'Use') - _ToolAppend(msbuild_settings, 'ClCompile', - 'PrecompiledHeaderFile', precompiled_header) - _ToolAppend(msbuild_settings, 'ClCompile', - 'ForcedIncludeFiles', [precompiled_header]) - else: - _ToolAppend(msbuild_settings, 'ClCompile', 'PrecompiledHeader', 'NotUsing') - # Turn off WinRT compilation - _ToolAppend(msbuild_settings, 'ClCompile', 'CompileAsWinRT', 'false') - # Turn on import libraries if appropriate - if spec.get('msvs_requires_importlibrary'): - _ToolAppend(msbuild_settings, '', 'IgnoreImportLibrary', 'false') - # Loadable modules don't generate import libraries; - # tell dependent projects to not expect one. - if spec['type'] == 'loadable_module': - _ToolAppend(msbuild_settings, '', 'IgnoreImportLibrary', 'true') - # Set the module definition file if any. - if def_file: - _ToolAppend(msbuild_settings, 'Link', 'ModuleDefinitionFile', def_file) - configuration['finalized_msbuild_settings'] = msbuild_settings - if prebuild: - _ToolAppend(msbuild_settings, 'PreBuildEvent', 'Command', prebuild) - if postbuild: - _ToolAppend(msbuild_settings, 'PostBuildEvent', 'Command', postbuild) + if "msbuild_settings" in configuration: + converted = False + msbuild_settings = configuration["msbuild_settings"] + MSVSSettings.ValidateMSBuildSettings(msbuild_settings) + else: + converted = True + msvs_settings = configuration.get("msvs_settings", {}) + msbuild_settings = MSVSSettings.ConvertToMSBuildSettings(msvs_settings) + include_dirs, midl_include_dirs, resource_include_dirs = _GetIncludeDirs( + configuration + ) + libraries = _GetLibraries(spec) + library_dirs = _GetLibraryDirs(configuration) + out_file, _, msbuild_tool = _GetOutputFilePathAndTool(spec, msbuild=True) + target_ext = _GetOutputTargetExt(spec) + defines = _GetDefines(configuration) + if converted: + # Visual Studio 2010 has TR1 + defines = [d for d in defines if d != "_HAS_TR1=0"] + # Warn of ignored settings + ignored_settings = ["msvs_tool_files"] + for ignored_setting in ignored_settings: + value = configuration.get(ignored_setting) + if value: + print( + "Warning: The automatic conversion to MSBuild does not handle " + "%s. Ignoring setting of %s" % (ignored_setting, str(value)) + ) + + defines = [_EscapeCppDefineForMSBuild(d) for d in defines] + disabled_warnings = _GetDisabledWarnings(configuration) + prebuild = configuration.get("msvs_prebuild") + postbuild = configuration.get("msvs_postbuild") + def_file = _GetModuleDefinition(spec) + precompiled_header = configuration.get("msvs_precompiled_header") + + # Add the information to the appropriate tool + # TODO(jeanluc) We could optimize and generate these settings only if + # the corresponding files are found, e.g. don't generate ResourceCompile + # if you don't have any resources. + _ToolAppend( + msbuild_settings, "ClCompile", "AdditionalIncludeDirectories", include_dirs + ) + _ToolAppend( + msbuild_settings, "Midl", "AdditionalIncludeDirectories", midl_include_dirs + ) + _ToolAppend( + msbuild_settings, + "ResourceCompile", + "AdditionalIncludeDirectories", + resource_include_dirs, + ) + # Add in libraries, note that even for empty libraries, we want this + # set, to prevent inheriting default libraries from the environment. + _ToolSetOrAppend(msbuild_settings, "Link", "AdditionalDependencies", libraries) + _ToolAppend(msbuild_settings, "Link", "AdditionalLibraryDirectories", library_dirs) + if out_file: + _ToolAppend( + msbuild_settings, msbuild_tool, "OutputFile", out_file, only_if_unset=True + ) + if target_ext: + _ToolAppend( + msbuild_settings, msbuild_tool, "TargetExt", target_ext, only_if_unset=True + ) + # Add defines. + _ToolAppend(msbuild_settings, "ClCompile", "PreprocessorDefinitions", defines) + _ToolAppend(msbuild_settings, "ResourceCompile", "PreprocessorDefinitions", defines) + # Add disabled warnings. + _ToolAppend( + msbuild_settings, "ClCompile", "DisableSpecificWarnings", disabled_warnings + ) + # Turn on precompiled headers if appropriate. + if precompiled_header: + precompiled_header = os.path.split(precompiled_header)[1] + _ToolAppend(msbuild_settings, "ClCompile", "PrecompiledHeader", "Use") + _ToolAppend( + msbuild_settings, "ClCompile", "PrecompiledHeaderFile", precompiled_header + ) + _ToolAppend( + msbuild_settings, "ClCompile", "ForcedIncludeFiles", [precompiled_header] + ) + else: + _ToolAppend(msbuild_settings, "ClCompile", "PrecompiledHeader", "NotUsing") + # Turn off WinRT compilation + _ToolAppend(msbuild_settings, "ClCompile", "CompileAsWinRT", "false") + # Turn on import libraries if appropriate + if spec.get("msvs_requires_importlibrary"): + _ToolAppend(msbuild_settings, "", "IgnoreImportLibrary", "false") + # Loadable modules don't generate import libraries; + # tell dependent projects to not expect one. + if spec["type"] == "loadable_module": + _ToolAppend(msbuild_settings, "", "IgnoreImportLibrary", "true") + # Set the module definition file if any. + if def_file: + _ToolAppend(msbuild_settings, "Link", "ModuleDefinitionFile", def_file) + configuration["finalized_msbuild_settings"] = msbuild_settings + if prebuild: + _ToolAppend(msbuild_settings, "PreBuildEvent", "Command", prebuild) + if postbuild: + _ToolAppend(msbuild_settings, "PostBuildEvent", "Command", postbuild) def _GetValueFormattedForMSBuild(tool_name, name, value): - if type(value) == list: - # For some settings, VS2010 does not automatically extends the settings - # TODO(jeanluc) Is this what we want? - if name in ['AdditionalIncludeDirectories', - 'AdditionalLibraryDirectories', - 'AdditionalOptions', - 'DelayLoadDLLs', - 'DisableSpecificWarnings', - 'PreprocessorDefinitions']: - value.append('%%(%s)' % name) - # For most tools, entries in a list should be separated with ';' but some - # settings use a space. Check for those first. - exceptions = { - 'ClCompile': ['AdditionalOptions'], - 'Link': ['AdditionalOptions'], - 'Lib': ['AdditionalOptions']} - if tool_name in exceptions and name in exceptions[tool_name]: - char = ' ' + if type(value) == list: + # For some settings, VS2010 does not automatically extends the settings + # TODO(jeanluc) Is this what we want? + if name in [ + "AdditionalIncludeDirectories", + "AdditionalLibraryDirectories", + "AdditionalOptions", + "DelayLoadDLLs", + "DisableSpecificWarnings", + "PreprocessorDefinitions", + ]: + value.append("%%(%s)" % name) + # For most tools, entries in a list should be separated with ';' but some + # settings use a space. Check for those first. + exceptions = { + "ClCompile": ["AdditionalOptions"], + "Link": ["AdditionalOptions"], + "Lib": ["AdditionalOptions"], + } + if tool_name in exceptions and name in exceptions[tool_name]: + char = " " + else: + char = ";" + formatted_value = char.join( + [MSVSSettings.ConvertVCMacrosToMSBuild(i) for i in value] + ) else: - char = ';' - formatted_value = char.join( - [MSVSSettings.ConvertVCMacrosToMSBuild(i) for i in value]) - else: - formatted_value = MSVSSettings.ConvertVCMacrosToMSBuild(value) - return formatted_value + formatted_value = MSVSSettings.ConvertVCMacrosToMSBuild(value) + return formatted_value def _VerifySourcesExist(sources, root_dir): - """Verifies that all source files exist on disk. + """Verifies that all source files exist on disk. Checks that all regular source files, i.e. not created at run time, exist on disk. Missing files cause needless recompilation but no otherwise @@ -3206,253 +3511,320 @@ def _VerifySourcesExist(sources, root_dir): Returns: A list of source files that cannot be found on disk. """ - missing_sources = [] - for source in sources: - if isinstance(source, MSVSProject.Filter): - missing_sources.extend(_VerifySourcesExist(source.contents, root_dir)) - else: - if '$' not in source: - full_path = os.path.join(root_dir, source) - if not os.path.exists(full_path): - missing_sources.append(full_path) - return missing_sources - - -def _GetMSBuildSources(spec, sources, exclusions, rule_dependencies, - extension_to_rule_name, actions_spec, - sources_handled_by_action, list_excluded): - groups = ['none', 'masm', 'midl', 'include', 'compile', 'resource', 'rule', - 'rule_dependency'] - grouped_sources = {} - for g in groups: - grouped_sources[g] = [] - - _AddSources2(spec, sources, exclusions, grouped_sources, - rule_dependencies, extension_to_rule_name, - sources_handled_by_action, list_excluded) - sources = [] - for g in groups: - if grouped_sources[g]: - sources.append(['ItemGroup'] + grouped_sources[g]) - if actions_spec: - sources.append(['ItemGroup'] + actions_spec) - return sources - - -def _AddSources2(spec, sources, exclusions, grouped_sources, - rule_dependencies, extension_to_rule_name, - sources_handled_by_action, - list_excluded): - extensions_excluded_from_precompile = [] - for source in sources: - if isinstance(source, MSVSProject.Filter): - _AddSources2(spec, source.contents, exclusions, grouped_sources, - rule_dependencies, extension_to_rule_name, - sources_handled_by_action, - list_excluded) - else: - if not source in sources_handled_by_action: - detail = [] - excluded_configurations = exclusions.get(source, []) - if len(excluded_configurations) == len(spec['configurations']): - detail.append(['ExcludedFromBuild', 'true']) + missing_sources = [] + for source in sources: + if isinstance(source, MSVSProject.Filter): + missing_sources.extend(_VerifySourcesExist(source.contents, root_dir)) + else: + if "$" not in source: + full_path = os.path.join(root_dir, source) + if not os.path.exists(full_path): + missing_sources.append(full_path) + return missing_sources + + +def _GetMSBuildSources( + spec, + sources, + exclusions, + rule_dependencies, + extension_to_rule_name, + actions_spec, + sources_handled_by_action, + list_excluded, +): + groups = [ + "none", + "masm", + "midl", + "include", + "compile", + "resource", + "rule", + "rule_dependency", + ] + grouped_sources = {} + for g in groups: + grouped_sources[g] = [] + + _AddSources2( + spec, + sources, + exclusions, + grouped_sources, + rule_dependencies, + extension_to_rule_name, + sources_handled_by_action, + list_excluded, + ) + sources = [] + for g in groups: + if grouped_sources[g]: + sources.append(["ItemGroup"] + grouped_sources[g]) + if actions_spec: + sources.append(["ItemGroup"] + actions_spec) + return sources + + +def _AddSources2( + spec, + sources, + exclusions, + grouped_sources, + rule_dependencies, + extension_to_rule_name, + sources_handled_by_action, + list_excluded, +): + extensions_excluded_from_precompile = [] + for source in sources: + if isinstance(source, MSVSProject.Filter): + _AddSources2( + spec, + source.contents, + exclusions, + grouped_sources, + rule_dependencies, + extension_to_rule_name, + sources_handled_by_action, + list_excluded, + ) else: - for config_name, configuration in sorted(excluded_configurations): - condition = _GetConfigurationCondition(config_name, configuration) - detail.append(['ExcludedFromBuild', - {'Condition': condition}, - 'true']) - # Add precompile if needed - for config_name, configuration in spec['configurations'].items(): - precompiled_source = configuration.get('msvs_precompiled_source', '') - if precompiled_source != '': - precompiled_source = _FixPath(precompiled_source) - if not extensions_excluded_from_precompile: - # If the precompiled header is generated by a C source, we must - # not try to use it for C++ sources, and vice versa. - basename, extension = os.path.splitext(precompiled_source) - if extension == '.c': - extensions_excluded_from_precompile = ['.cc', '.cpp', '.cxx'] - else: - extensions_excluded_from_precompile = ['.c'] - - if precompiled_source == source: - condition = _GetConfigurationCondition(config_name, configuration) - detail.append(['PrecompiledHeader', - {'Condition': condition}, - 'Create' - ]) - else: - # Turn off precompiled header usage for source files of a - # different type than the file that generated the - # precompiled header. - for extension in extensions_excluded_from_precompile: - if source.endswith(extension): - detail.append(['PrecompiledHeader', '']) - detail.append(['ForcedIncludeFiles', '']) - - group, element = _MapFileToMsBuildSourceType(source, rule_dependencies, - extension_to_rule_name, - _GetUniquePlatforms(spec)) - grouped_sources[group].append([element, {'Include': source}] + detail) + if source not in sources_handled_by_action: + detail = [] + excluded_configurations = exclusions.get(source, []) + if len(excluded_configurations) == len(spec["configurations"]): + detail.append(["ExcludedFromBuild", "true"]) + else: + for config_name, configuration in sorted(excluded_configurations): + condition = _GetConfigurationCondition( + config_name, configuration + ) + detail.append( + ["ExcludedFromBuild", {"Condition": condition}, "true"] + ) + # Add precompile if needed + for config_name, configuration in spec["configurations"].items(): + precompiled_source = configuration.get( + "msvs_precompiled_source", "" + ) + if precompiled_source != "": + precompiled_source = _FixPath(precompiled_source) + if not extensions_excluded_from_precompile: + # If the precompiled header is generated by a C source, we must + # not try to use it for C++ sources, and vice versa. + basename, extension = os.path.splitext(precompiled_source) + if extension == ".c": + extensions_excluded_from_precompile = [ + ".cc", + ".cpp", + ".cxx", + ] + else: + extensions_excluded_from_precompile = [".c"] + + if precompiled_source == source: + condition = _GetConfigurationCondition( + config_name, configuration + ) + detail.append( + ["PrecompiledHeader", {"Condition": condition}, "Create"] + ) + else: + # Turn off precompiled header usage for source files of a + # different type than the file that generated the + # precompiled header. + for extension in extensions_excluded_from_precompile: + if source.endswith(extension): + detail.append(["PrecompiledHeader", ""]) + detail.append(["ForcedIncludeFiles", ""]) + + group, element = _MapFileToMsBuildSourceType( + source, + rule_dependencies, + extension_to_rule_name, + _GetUniquePlatforms(spec), + ) + grouped_sources[group].append([element, {"Include": source}] + detail) def _GetMSBuildProjectReferences(project): - references = [] - if project.dependencies: - group = ['ItemGroup'] - for dependency in project.dependencies: - guid = dependency.guid - project_dir = os.path.split(project.path)[0] - relative_path = gyp.common.RelativePath(dependency.path, project_dir) - project_ref = ['ProjectReference', - {'Include': relative_path}, - ['Project', guid], - ['ReferenceOutputAssembly', 'false'] - ] - for config in dependency.spec.get('configurations', {}).values(): - if config.get('msvs_use_library_dependency_inputs', 0): - project_ref.append(['UseLibraryDependencyInputs', 'true']) - break - # If it's disabled in any config, turn it off in the reference. - if config.get('msvs_2010_disable_uldi_when_referenced', 0): - project_ref.append(['UseLibraryDependencyInputs', 'false']) - break - group.append(project_ref) - references.append(group) - return references + references = [] + if project.dependencies: + group = ["ItemGroup"] + for dependency in project.dependencies: + guid = dependency.guid + project_dir = os.path.split(project.path)[0] + relative_path = gyp.common.RelativePath(dependency.path, project_dir) + project_ref = [ + "ProjectReference", + {"Include": relative_path}, + ["Project", guid], + ["ReferenceOutputAssembly", "false"], + ] + for config in dependency.spec.get("configurations", {}).values(): + if config.get("msvs_use_library_dependency_inputs", 0): + project_ref.append(["UseLibraryDependencyInputs", "true"]) + break + # If it's disabled in any config, turn it off in the reference. + if config.get("msvs_2010_disable_uldi_when_referenced", 0): + project_ref.append(["UseLibraryDependencyInputs", "false"]) + break + group.append(project_ref) + references.append(group) + return references def _GenerateMSBuildProject(project, options, version, generator_flags): - spec = project.spec - configurations = spec['configurations'] - project_dir, project_file_name = os.path.split(project.path) - gyp.common.EnsureDirExists(project.path) - # Prepare list of sources and excluded sources. - gyp_path = _NormalizedSource(project.build_file) - relative_path_of_gyp_file = gyp.common.RelativePath(gyp_path, project_dir) - - gyp_file = os.path.split(project.build_file)[1] - sources, excluded_sources = _PrepareListOfSources(spec, generator_flags, - gyp_file) - # Add rules. - actions_to_add = {} - props_files_of_rules = set() - targets_files_of_rules = set() - rule_dependencies = set() - extension_to_rule_name = {} - list_excluded = generator_flags.get('msvs_list_excluded_files', True) - - # Don't generate rules if we are using an external builder like ninja. - if not spec.get('msvs_external_builder'): - _GenerateRulesForMSBuild(project_dir, options, spec, - sources, excluded_sources, - props_files_of_rules, targets_files_of_rules, - actions_to_add, rule_dependencies, - extension_to_rule_name) - else: - rules = spec.get('rules', []) - _AdjustSourcesForRules(rules, sources, excluded_sources, True) - - sources, excluded_sources, excluded_idl = ( - _AdjustSourcesAndConvertToFilterHierarchy(spec, options, - project_dir, sources, - excluded_sources, - list_excluded, version)) - - # Don't add actions if we are using an external builder like ninja. - if not spec.get('msvs_external_builder'): - _AddActions(actions_to_add, spec, project.build_file) - _AddCopies(actions_to_add, spec) + spec = project.spec + configurations = spec["configurations"] + project_dir, project_file_name = os.path.split(project.path) + gyp.common.EnsureDirExists(project.path) + # Prepare list of sources and excluded sources. + + gyp_file = os.path.split(project.build_file)[1] + sources, excluded_sources = _PrepareListOfSources(spec, generator_flags, gyp_file) + # Add rules. + actions_to_add = {} + props_files_of_rules = set() + targets_files_of_rules = set() + rule_dependencies = set() + extension_to_rule_name = {} + list_excluded = generator_flags.get("msvs_list_excluded_files", True) + + # Don't generate rules if we are using an external builder like ninja. + if not spec.get("msvs_external_builder"): + _GenerateRulesForMSBuild( + project_dir, + options, + spec, + sources, + excluded_sources, + props_files_of_rules, + targets_files_of_rules, + actions_to_add, + rule_dependencies, + extension_to_rule_name, + ) + else: + rules = spec.get("rules", []) + _AdjustSourcesForRules(rules, sources, excluded_sources, True) + + sources, excluded_sources, excluded_idl = _AdjustSourcesAndConvertToFilterHierarchy( + spec, options, project_dir, sources, excluded_sources, list_excluded, version + ) + + # Don't add actions if we are using an external builder like ninja. + if not spec.get("msvs_external_builder"): + _AddActions(actions_to_add, spec, project.build_file) + _AddCopies(actions_to_add, spec) + + # NOTE: this stanza must appear after all actions have been decided. + # Don't excluded sources with actions attached, or they won't run. + excluded_sources = _FilterActionsFromExcluded(excluded_sources, actions_to_add) + + exclusions = _GetExcludedFilesFromBuild(spec, excluded_sources, excluded_idl) + actions_spec, sources_handled_by_action = _GenerateActionsForMSBuild( + spec, actions_to_add + ) + + _GenerateMSBuildFiltersFile( + project.path + ".filters", + sources, + rule_dependencies, + extension_to_rule_name, + _GetUniquePlatforms(spec), + ) + missing_sources = _VerifySourcesExist(sources, project_dir) + + for configuration in configurations.values(): + _FinalizeMSBuildSettings(spec, configuration) + + # Add attributes to root element + + import_default_section = [ + ["Import", {"Project": r"$(VCTargetsPath)\Microsoft.Cpp.Default.props"}] + ] + import_cpp_props_section = [ + ["Import", {"Project": r"$(VCTargetsPath)\Microsoft.Cpp.props"}] + ] + import_cpp_targets_section = [ + ["Import", {"Project": r"$(VCTargetsPath)\Microsoft.Cpp.targets"}] + ] + import_masm_props_section = [ + ["Import", {"Project": r"$(VCTargetsPath)\BuildCustomizations\masm.props"}] + ] + import_masm_targets_section = [ + ["Import", {"Project": r"$(VCTargetsPath)\BuildCustomizations\masm.targets"}] + ] + import_marmasm_props_section = [ + ["Import", {"Project": r"$(VCTargetsPath)\BuildCustomizations\marmasm.props"}] + ] + import_marmasm_targets_section = [ + ["Import", {"Project": r"$(VCTargetsPath)\BuildCustomizations\marmasm.targets"}] + ] + macro_section = [["PropertyGroup", {"Label": "UserMacros"}]] + + content = [ + "Project", + { + "xmlns": "http://schemas.microsoft.com/developer/msbuild/2003", + "ToolsVersion": version.ProjectVersion(), + "DefaultTargets": "Build", + }, + ] - # NOTE: this stanza must appear after all actions have been decided. - # Don't excluded sources with actions attached, or they won't run. - excluded_sources = _FilterActionsFromExcluded( - excluded_sources, actions_to_add) - - exclusions = _GetExcludedFilesFromBuild(spec, excluded_sources, excluded_idl) - actions_spec, sources_handled_by_action = _GenerateActionsForMSBuild( - spec, actions_to_add) - - _GenerateMSBuildFiltersFile(project.path + '.filters', sources, - rule_dependencies, - extension_to_rule_name, _GetUniquePlatforms(spec)) - missing_sources = _VerifySourcesExist(sources, project_dir) - - for configuration in configurations.values(): - _FinalizeMSBuildSettings(spec, configuration) - - # Add attributes to root element - - import_default_section = [ - ['Import', {'Project': r'$(VCTargetsPath)\Microsoft.Cpp.Default.props'}]] - import_cpp_props_section = [ - ['Import', {'Project': r'$(VCTargetsPath)\Microsoft.Cpp.props'}]] - import_cpp_targets_section = [ - ['Import', {'Project': r'$(VCTargetsPath)\Microsoft.Cpp.targets'}]] - import_masm_props_section = [ - ['Import', - {'Project': r'$(VCTargetsPath)\BuildCustomizations\masm.props'}]] - import_masm_targets_section = [ - ['Import', - {'Project': r'$(VCTargetsPath)\BuildCustomizations\masm.targets'}]] - import_marmasm_props_section = [ - ['Import', - {'Project': r'$(VCTargetsPath)\BuildCustomizations\marmasm.props'}]] - import_marmasm_targets_section = [ - ['Import', - {'Project': r'$(VCTargetsPath)\BuildCustomizations\marmasm.targets'}]] - macro_section = [['PropertyGroup', {'Label': 'UserMacros'}]] - - content = [ - 'Project', - {'xmlns': 'http://schemas.microsoft.com/developer/msbuild/2003', - 'ToolsVersion': version.ProjectVersion(), - 'DefaultTargets': 'Build' - }] - - content += _GetMSBuildProjectConfigurations(configurations) - content += _GetMSBuildGlobalProperties(spec, version, project.guid, - project_file_name) - content += import_default_section - content += _GetMSBuildConfigurationDetails(spec, project.build_file) - if spec.get('msvs_enable_winphone'): - content += _GetMSBuildLocalProperties('v120_wp81') - else: - content += _GetMSBuildLocalProperties(project.msbuild_toolset) - content += import_cpp_props_section - content += import_masm_props_section - if spec.get('msvs_enable_marmasm'): - content += import_marmasm_props_section - content += _GetMSBuildExtensions(props_files_of_rules) - content += _GetMSBuildPropertySheets(configurations) - content += macro_section - content += _GetMSBuildConfigurationGlobalProperties(spec, configurations, - project.build_file) - content += _GetMSBuildToolSettingsSections(spec, configurations) - content += _GetMSBuildSources( - spec, sources, exclusions, rule_dependencies, extension_to_rule_name, - actions_spec, sources_handled_by_action, list_excluded) - content += _GetMSBuildProjectReferences(project) - content += import_cpp_targets_section - content += import_masm_targets_section - if spec.get('msvs_enable_marmasm'): - content += import_marmasm_targets_section - content += _GetMSBuildExtensionTargets(targets_files_of_rules) - - if spec.get('msvs_external_builder'): - content += _GetMSBuildExternalBuilderTargets(spec) - - # TODO(jeanluc) File a bug to get rid of runas. We had in MSVS: - # has_run_as = _WriteMSVSUserFile(project.path, version, spec) - - easy_xml.WriteXmlIfChanged(content, project.path, pretty=True, win32=True) - - return missing_sources + content += _GetMSBuildProjectConfigurations(configurations) + content += _GetMSBuildGlobalProperties( + spec, version, project.guid, project_file_name + ) + content += import_default_section + content += _GetMSBuildConfigurationDetails(spec, project.build_file) + if spec.get("msvs_enable_winphone"): + content += _GetMSBuildLocalProperties("v120_wp81") + else: + content += _GetMSBuildLocalProperties(project.msbuild_toolset) + content += import_cpp_props_section + content += import_masm_props_section + if spec.get("msvs_enable_marmasm"): + content += import_marmasm_props_section + content += _GetMSBuildExtensions(props_files_of_rules) + content += _GetMSBuildPropertySheets(configurations) + content += macro_section + content += _GetMSBuildConfigurationGlobalProperties( + spec, configurations, project.build_file + ) + content += _GetMSBuildToolSettingsSections(spec, configurations) + content += _GetMSBuildSources( + spec, + sources, + exclusions, + rule_dependencies, + extension_to_rule_name, + actions_spec, + sources_handled_by_action, + list_excluded, + ) + content += _GetMSBuildProjectReferences(project) + content += import_cpp_targets_section + content += import_masm_targets_section + if spec.get("msvs_enable_marmasm"): + content += import_marmasm_targets_section + content += _GetMSBuildExtensionTargets(targets_files_of_rules) + + if spec.get("msvs_external_builder"): + content += _GetMSBuildExternalBuilderTargets(spec) + + # TODO(jeanluc) File a bug to get rid of runas. We had in MSVS: + # has_run_as = _WriteMSVSUserFile(project.path, version, spec) + + easy_xml.WriteXmlIfChanged(content, project.path, pretty=True, win32=True) + + return missing_sources def _GetMSBuildExternalBuilderTargets(spec): - """Return a list of MSBuild targets for external builders. + """Return a list of MSBuild targets for external builders. The "Build" and "Clean" targets are always generated. If the spec contains 'msvs_external_builder_clcompile_cmd', then the "ClCompile" target will also @@ -3463,47 +3835,52 @@ def _GetMSBuildExternalBuilderTargets(spec): Returns: List of MSBuild 'Target' specs. """ - build_cmd = _BuildCommandLineForRuleRaw( - spec, spec['msvs_external_builder_build_cmd'], - False, False, False, False) - build_target = ['Target', {'Name': 'Build'}] - build_target.append(['Exec', {'Command': build_cmd}]) - - clean_cmd = _BuildCommandLineForRuleRaw( - spec, spec['msvs_external_builder_clean_cmd'], - False, False, False, False) - clean_target = ['Target', {'Name': 'Clean'}] - clean_target.append(['Exec', {'Command': clean_cmd}]) - - targets = [build_target, clean_target] - - if spec.get('msvs_external_builder_clcompile_cmd'): - clcompile_cmd = _BuildCommandLineForRuleRaw( - spec, spec['msvs_external_builder_clcompile_cmd'], - False, False, False, False) - clcompile_target = ['Target', {'Name': 'ClCompile'}] - clcompile_target.append(['Exec', {'Command': clcompile_cmd}]) - targets.append(clcompile_target) - - return targets + build_cmd = _BuildCommandLineForRuleRaw( + spec, spec["msvs_external_builder_build_cmd"], False, False, False, False + ) + build_target = ["Target", {"Name": "Build"}] + build_target.append(["Exec", {"Command": build_cmd}]) + + clean_cmd = _BuildCommandLineForRuleRaw( + spec, spec["msvs_external_builder_clean_cmd"], False, False, False, False + ) + clean_target = ["Target", {"Name": "Clean"}] + clean_target.append(["Exec", {"Command": clean_cmd}]) + + targets = [build_target, clean_target] + + if spec.get("msvs_external_builder_clcompile_cmd"): + clcompile_cmd = _BuildCommandLineForRuleRaw( + spec, + spec["msvs_external_builder_clcompile_cmd"], + False, + False, + False, + False, + ) + clcompile_target = ["Target", {"Name": "ClCompile"}] + clcompile_target.append(["Exec", {"Command": clcompile_cmd}]) + targets.append(clcompile_target) + + return targets def _GetMSBuildExtensions(props_files_of_rules): - extensions = ['ImportGroup', {'Label': 'ExtensionSettings'}] - for props_file in props_files_of_rules: - extensions.append(['Import', {'Project': props_file}]) - return [extensions] + extensions = ["ImportGroup", {"Label": "ExtensionSettings"}] + for props_file in props_files_of_rules: + extensions.append(["Import", {"Project": props_file}]) + return [extensions] def _GetMSBuildExtensionTargets(targets_files_of_rules): - targets_node = ['ImportGroup', {'Label': 'ExtensionTargets'}] - for targets_file in sorted(targets_files_of_rules): - targets_node.append(['Import', {'Project': targets_file}]) - return [targets_node] + targets_node = ["ImportGroup", {"Label": "ExtensionTargets"}] + for targets_file in sorted(targets_files_of_rules): + targets_node.append(["Import", {"Project": targets_file}]) + return [targets_node] def _GenerateActionsForMSBuild(spec, actions_to_add): - """Add actions accumulated into an actions_to_add, merging as needed. + """Add actions accumulated into an actions_to_add, merging as needed. Arguments: spec: the target project dict @@ -3513,63 +3890,75 @@ def _GenerateActionsForMSBuild(spec, actions_to_add): Returns: A pair of (action specification, the sources handled by this action). """ - sources_handled_by_action = OrderedSet() - actions_spec = [] - for primary_input, actions in actions_to_add.items(): - inputs = OrderedSet() - outputs = OrderedSet() - descriptions = [] - commands = [] - for action in actions: - inputs.update(OrderedSet(action['inputs'])) - outputs.update(OrderedSet(action['outputs'])) - descriptions.append(action['description']) - cmd = action['command'] - # For most actions, add 'call' so that actions that invoke batch files - # return and continue executing. msbuild_use_call provides a way to - # disable this but I have not seen any adverse effect from doing that - # for everything. - if action.get('msbuild_use_call', True): - cmd = 'call ' + cmd - commands.append(cmd) - # Add the custom build action for one input file. - description = ', and also '.join(descriptions) - - # We can't join the commands simply with && because the command line will - # get too long. See also _AddActions: cygwin's setup_env mustn't be called - # for every invocation or the command that sets the PATH will grow too - # long. - command = '\r\n'.join([c + '\r\nif %errorlevel% neq 0 exit /b %errorlevel%' - for c in commands]) - _AddMSBuildAction(spec, - primary_input, - inputs, - outputs, - command, - description, - sources_handled_by_action, - actions_spec) - return actions_spec, sources_handled_by_action - - -def _AddMSBuildAction(spec, primary_input, inputs, outputs, cmd, description, - sources_handled_by_action, actions_spec): - command = MSVSSettings.ConvertVCMacrosToMSBuild(cmd) - primary_input = _FixPath(primary_input) - inputs_array = _FixPaths(inputs) - outputs_array = _FixPaths(outputs) - additional_inputs = ';'.join([i for i in inputs_array - if i != primary_input]) - outputs = ';'.join(outputs_array) - sources_handled_by_action.add(primary_input) - action_spec = ['CustomBuild', {'Include': primary_input}] - action_spec.extend( - # TODO(jeanluc) 'Document' for all or just if as_sources? - [['FileType', 'Document'], - ['Command', command], - ['Message', description], - ['Outputs', outputs] - ]) - if additional_inputs: - action_spec.append(['AdditionalInputs', additional_inputs]) - actions_spec.append(action_spec) + sources_handled_by_action = OrderedSet() + actions_spec = [] + for primary_input, actions in actions_to_add.items(): + inputs = OrderedSet() + outputs = OrderedSet() + descriptions = [] + commands = [] + for action in actions: + inputs.update(OrderedSet(action["inputs"])) + outputs.update(OrderedSet(action["outputs"])) + descriptions.append(action["description"]) + cmd = action["command"] + # For most actions, add 'call' so that actions that invoke batch files + # return and continue executing. msbuild_use_call provides a way to + # disable this but I have not seen any adverse effect from doing that + # for everything. + if action.get("msbuild_use_call", True): + cmd = "call " + cmd + commands.append(cmd) + # Add the custom build action for one input file. + description = ", and also ".join(descriptions) + + # We can't join the commands simply with && because the command line will + # get too long. See also _AddActions: cygwin's setup_env mustn't be called + # for every invocation or the command that sets the PATH will grow too + # long. + command = "\r\n".join( + [c + "\r\nif %errorlevel% neq 0 exit /b %errorlevel%" for c in commands] + ) + _AddMSBuildAction( + spec, + primary_input, + inputs, + outputs, + command, + description, + sources_handled_by_action, + actions_spec, + ) + return actions_spec, sources_handled_by_action + + +def _AddMSBuildAction( + spec, + primary_input, + inputs, + outputs, + cmd, + description, + sources_handled_by_action, + actions_spec, +): + command = MSVSSettings.ConvertVCMacrosToMSBuild(cmd) + primary_input = _FixPath(primary_input) + inputs_array = _FixPaths(inputs) + outputs_array = _FixPaths(outputs) + additional_inputs = ";".join([i for i in inputs_array if i != primary_input]) + outputs = ";".join(outputs_array) + sources_handled_by_action.add(primary_input) + action_spec = ["CustomBuild", {"Include": primary_input}] + action_spec.extend( + # TODO(jeanluc) 'Document' for all or just if as_sources? + [ + ["FileType", "Document"], + ["Command", command], + ["Message", description], + ["Outputs", outputs], + ] + ) + if additional_inputs: + action_spec.append(["AdditionalInputs", additional_inputs]) + actions_spec.append(action_spec) diff --git a/gyp/pylib/gyp/generator/msvs_test.py b/gyp/pylib/gyp/generator/msvs_test.py index 1b0cdd1720..e001f417d5 100755 --- a/gyp/pylib/gyp/generator/msvs_test.py +++ b/gyp/pylib/gyp/generator/msvs_test.py @@ -9,33 +9,39 @@ import unittest try: - from StringIO import StringIO # Python 2 + from StringIO import StringIO # Python 2 except ImportError: - from io import StringIO # Python 3 + from io import StringIO # Python 3 class TestSequenceFunctions(unittest.TestCase): - - def setUp(self): - self.stderr = StringIO() - - def test_GetLibraries(self): - self.assertEqual( - msvs._GetLibraries({}), - []) - self.assertEqual( - msvs._GetLibraries({'libraries': []}), - []) - self.assertEqual( - msvs._GetLibraries({'other':'foo', 'libraries': ['a.lib']}), - ['a.lib']) - self.assertEqual( - msvs._GetLibraries({'libraries': ['-la']}), - ['a.lib']) - self.assertEqual( - msvs._GetLibraries({'libraries': ['a.lib', 'b.lib', 'c.lib', '-lb.lib', - '-lb.lib', 'd.lib', 'a.lib']}), - ['c.lib', 'b.lib', 'd.lib', 'a.lib']) - -if __name__ == '__main__': - unittest.main() + def setUp(self): + self.stderr = StringIO() + + def test_GetLibraries(self): + self.assertEqual(msvs._GetLibraries({}), []) + self.assertEqual(msvs._GetLibraries({"libraries": []}), []) + self.assertEqual( + msvs._GetLibraries({"other": "foo", "libraries": ["a.lib"]}), ["a.lib"] + ) + self.assertEqual(msvs._GetLibraries({"libraries": ["-la"]}), ["a.lib"]) + self.assertEqual( + msvs._GetLibraries( + { + "libraries": [ + "a.lib", + "b.lib", + "c.lib", + "-lb.lib", + "-lb.lib", + "d.lib", + "a.lib", + ] + } + ), + ["c.lib", "b.lib", "d.lib", "a.lib"], + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py index d5006bf84a..19e00319a7 100644 --- a/gyp/pylib/gyp/generator/ninja.py +++ b/gyp/pylib/gyp/generator/ninja.py @@ -16,49 +16,47 @@ import sys import gyp import gyp.common -from gyp.common import OrderedSet import gyp.msvs_emulation import gyp.MSVSUtil as MSVSUtil import gyp.xcode_emulation + try: - from cStringIO import StringIO + from cStringIO import StringIO except ImportError: - from io import StringIO + from io import StringIO from gyp.common import GetEnvironFallback import gyp.ninja_syntax as ninja_syntax generator_default_variables = { - 'EXECUTABLE_PREFIX': '', - 'EXECUTABLE_SUFFIX': '', - 'STATIC_LIB_PREFIX': 'lib', - 'STATIC_LIB_SUFFIX': '.a', - 'SHARED_LIB_PREFIX': 'lib', - - # Gyp expects the following variables to be expandable by the build - # system to the appropriate locations. Ninja prefers paths to be - # known at gyp time. To resolve this, introduce special - # variables starting with $! and $| (which begin with a $ so gyp knows it - # should be treated specially, but is otherwise an invalid - # ninja/shell variable) that are passed to gyp here but expanded - # before writing out into the target .ninja files; see - # ExpandSpecial. - # $! is used for variables that represent a path and that can only appear at - # the start of a string, while $| is used for variables that can appear - # anywhere in a string. - 'INTERMEDIATE_DIR': '$!INTERMEDIATE_DIR', - 'SHARED_INTERMEDIATE_DIR': '$!PRODUCT_DIR/gen', - 'PRODUCT_DIR': '$!PRODUCT_DIR', - 'CONFIGURATION_NAME': '$|CONFIGURATION_NAME', - - # Special variables that may be used by gyp 'rule' targets. - # We generate definitions for these variables on the fly when processing a - # rule. - 'RULE_INPUT_ROOT': '${root}', - 'RULE_INPUT_DIRNAME': '${dirname}', - 'RULE_INPUT_PATH': '${source}', - 'RULE_INPUT_EXT': '${ext}', - 'RULE_INPUT_NAME': '${name}', + "EXECUTABLE_PREFIX": "", + "EXECUTABLE_SUFFIX": "", + "STATIC_LIB_PREFIX": "lib", + "STATIC_LIB_SUFFIX": ".a", + "SHARED_LIB_PREFIX": "lib", + # Gyp expects the following variables to be expandable by the build + # system to the appropriate locations. Ninja prefers paths to be + # known at gyp time. To resolve this, introduce special + # variables starting with $! and $| (which begin with a $ so gyp knows it + # should be treated specially, but is otherwise an invalid + # ninja/shell variable) that are passed to gyp here but expanded + # before writing out into the target .ninja files; see + # ExpandSpecial. + # $! is used for variables that represent a path and that can only appear at + # the start of a string, while $| is used for variables that can appear + # anywhere in a string. + "INTERMEDIATE_DIR": "$!INTERMEDIATE_DIR", + "SHARED_INTERMEDIATE_DIR": "$!PRODUCT_DIR/gen", + "PRODUCT_DIR": "$!PRODUCT_DIR", + "CONFIGURATION_NAME": "$|CONFIGURATION_NAME", + # Special variables that may be used by gyp 'rule' targets. + # We generate definitions for these variables on the fly when processing a + # rule. + "RULE_INPUT_ROOT": "${root}", + "RULE_INPUT_DIRNAME": "${dirname}", + "RULE_INPUT_PATH": "${source}", + "RULE_INPUT_EXT": "${ext}", + "RULE_INPUT_NAME": "${name}", } # Placates pylint. @@ -69,42 +67,43 @@ generator_supports_multiple_toolsets = gyp.common.CrossCompileRequested() + def StripPrefix(arg, prefix): - if arg.startswith(prefix): - return arg[len(prefix):] - return arg + if arg.startswith(prefix): + return arg[len(prefix) :] + return arg def QuoteShellArgument(arg, flavor): - """Quote a string such that it will be interpreted as a single argument + """Quote a string such that it will be interpreted as a single argument by the shell.""" - # Rather than attempting to enumerate the bad shell characters, just - # whitelist common OK ones and quote anything else. - if re.match(r'^[a-zA-Z0-9_=.\\/-]+$', arg): - return arg # No quoting necessary. - if flavor == 'win': - return gyp.msvs_emulation.QuoteForRspFile(arg) - return "'" + arg.replace("'", "'" + '"\'"' + "'") + "'" + # Rather than attempting to enumerate the bad shell characters, just + # whitelist common OK ones and quote anything else. + if re.match(r"^[a-zA-Z0-9_=.\\/-]+$", arg): + return arg # No quoting necessary. + if flavor == "win": + return gyp.msvs_emulation.QuoteForRspFile(arg) + return "'" + arg.replace("'", "'" + '"\'"' + "'") + "'" def Define(d, flavor): - """Takes a preprocessor define and returns a -D parameter that's ninja- and + """Takes a preprocessor define and returns a -D parameter that's ninja- and shell-escaped.""" - if flavor == 'win': - # cl.exe replaces literal # characters with = in preprocesor definitions for - # some reason. Octal-encode to work around that. - d = d.replace('#', '\\%03o' % ord('#')) - return QuoteShellArgument(ninja_syntax.escape('-D' + d), flavor) + if flavor == "win": + # cl.exe replaces literal # characters with = in preprocesor definitions for + # some reason. Octal-encode to work around that. + d = d.replace("#", "\\%03o" % ord("#")) + return QuoteShellArgument(ninja_syntax.escape("-D" + d), flavor) def AddArch(output, arch): - """Adds an arch string to an output path.""" - output, extension = os.path.splitext(output) - return '%s.%s%s' % (output, arch, extension) + """Adds an arch string to an output path.""" + output, extension = os.path.splitext(output) + return "%s.%s%s" % (output, arch, extension) class Target(object): - """Target represents the paths used within a single gyp target. + """Target represents the paths used within a single gyp target. Conceptually, building a single target A is a series of steps: @@ -126,67 +125,68 @@ class Target(object): variables only store concrete paths to single files, while methods compute derived values like "the last output of the target". """ - def __init__(self, type): - # Gyp type ("static_library", etc.) of this target. - self.type = type - # File representing whether any input dependencies necessary for - # dependent actions have completed. - self.preaction_stamp = None - # File representing whether any input dependencies necessary for - # dependent compiles have completed. - self.precompile_stamp = None - # File representing the completion of actions/rules/copies, if any. - self.actions_stamp = None - # Path to the output of the link step, if any. - self.binary = None - # Path to the file representing the completion of building the bundle, - # if any. - self.bundle = None - # On Windows, incremental linking requires linking against all the .objs - # that compose a .lib (rather than the .lib itself). That list is stored - # here. In this case, we also need to save the compile_deps for the target, - # so that the target that directly depends on the .objs can also depend - # on those. - self.component_objs = None - self.compile_deps = None - # Windows only. The import .lib is the output of a build step, but - # because dependents only link against the lib (not both the lib and the - # dll) we keep track of the import library here. - self.import_lib = None - # Track if this target contains any C++ files, to decide if gcc or g++ - # should be used for linking. - self.uses_cpp = False - - def Linkable(self): - """Return true if this is a target that can be linked against.""" - return self.type in ('static_library', 'shared_library') - - def UsesToc(self, flavor): - """Return true if the target should produce a restat rule based on a TOC + + def __init__(self, type): + # Gyp type ("static_library", etc.) of this target. + self.type = type + # File representing whether any input dependencies necessary for + # dependent actions have completed. + self.preaction_stamp = None + # File representing whether any input dependencies necessary for + # dependent compiles have completed. + self.precompile_stamp = None + # File representing the completion of actions/rules/copies, if any. + self.actions_stamp = None + # Path to the output of the link step, if any. + self.binary = None + # Path to the file representing the completion of building the bundle, + # if any. + self.bundle = None + # On Windows, incremental linking requires linking against all the .objs + # that compose a .lib (rather than the .lib itself). That list is stored + # here. In this case, we also need to save the compile_deps for the target, + # so that the target that directly depends on the .objs can also depend + # on those. + self.component_objs = None + self.compile_deps = None + # Windows only. The import .lib is the output of a build step, but + # because dependents only link against the lib (not both the lib and the + # dll) we keep track of the import library here. + self.import_lib = None + # Track if this target contains any C++ files, to decide if gcc or g++ + # should be used for linking. + self.uses_cpp = False + + def Linkable(self): + """Return true if this is a target that can be linked against.""" + return self.type in ("static_library", "shared_library") + + def UsesToc(self, flavor): + """Return true if the target should produce a restat rule based on a TOC file.""" - # For bundles, the .TOC should be produced for the binary, not for - # FinalOutput(). But the naive approach would put the TOC file into the - # bundle, so don't do this for bundles for now. - if flavor == 'win' or self.bundle: - return False - return self.type in ('shared_library', 'loadable_module') - - def PreActionInput(self, flavor): - """Return the path, if any, that should be used as a dependency of + # For bundles, the .TOC should be produced for the binary, not for + # FinalOutput(). But the naive approach would put the TOC file into the + # bundle, so don't do this for bundles for now. + if flavor == "win" or self.bundle: + return False + return self.type in ("shared_library", "loadable_module") + + def PreActionInput(self, flavor): + """Return the path, if any, that should be used as a dependency of any dependent action step.""" - if self.UsesToc(flavor): - return self.FinalOutput() + '.TOC' - return self.FinalOutput() or self.preaction_stamp + if self.UsesToc(flavor): + return self.FinalOutput() + ".TOC" + return self.FinalOutput() or self.preaction_stamp - def PreCompileInput(self): - """Return the path, if any, that should be used as a dependency of + def PreCompileInput(self): + """Return the path, if any, that should be used as a dependency of any dependent compile step.""" - return self.actions_stamp or self.precompile_stamp + return self.actions_stamp or self.precompile_stamp - def FinalOutput(self): - """Return the last output of the target, which depends on all prior + def FinalOutput(self): + """Return the last output of the target, which depends on all prior steps.""" - return self.bundle or self.binary or self.actions_stamp + return self.bundle or self.binary or self.actions_stamp # A small discourse on paths as used within the Ninja build: @@ -213,108 +213,116 @@ def FinalOutput(self): # an output file; the result can be namespaced such that it is unique # to the input file name as well as the output target name. + class NinjaWriter(object): - def __init__(self, hash_for_rules, target_outputs, base_dir, build_dir, - output_file, toplevel_build, output_file_name, flavor, - toplevel_dir=None): - """ + def __init__( + self, + hash_for_rules, + target_outputs, + base_dir, + build_dir, + output_file, + toplevel_build, + output_file_name, + flavor, + toplevel_dir=None, + ): + """ base_dir: path from source root to directory containing this gyp file, by gyp semantics, all input paths are relative to this build_dir: path from source root to build output toplevel_dir: path to the toplevel directory """ - self.hash_for_rules = hash_for_rules - self.target_outputs = target_outputs - self.base_dir = base_dir - self.build_dir = build_dir - self.ninja = ninja_syntax.Writer(output_file) - self.toplevel_build = toplevel_build - self.output_file_name = output_file_name - - self.flavor = flavor - self.abs_build_dir = None - if toplevel_dir is not None: - self.abs_build_dir = os.path.abspath(os.path.join(toplevel_dir, - build_dir)) - self.obj_ext = '.obj' if flavor == 'win' else '.o' - if flavor == 'win': - # See docstring of msvs_emulation.GenerateEnvironmentFiles(). - self.win_env = {} - for arch in ('x86', 'x64'): - self.win_env[arch] = 'environment.' + arch - - # Relative path from build output dir to base dir. - build_to_top = gyp.common.InvertRelativePath(build_dir, toplevel_dir) - self.build_to_base = os.path.join(build_to_top, base_dir) - # Relative path from base dir to build dir. - base_to_top = gyp.common.InvertRelativePath(base_dir, toplevel_dir) - self.base_to_build = os.path.join(base_to_top, build_dir) - - def ExpandSpecial(self, path, product_dir=None): - """Expand specials like $!PRODUCT_DIR in |path|. + self.hash_for_rules = hash_for_rules + self.target_outputs = target_outputs + self.base_dir = base_dir + self.build_dir = build_dir + self.ninja = ninja_syntax.Writer(output_file) + self.toplevel_build = toplevel_build + self.output_file_name = output_file_name + + self.flavor = flavor + self.abs_build_dir = None + if toplevel_dir is not None: + self.abs_build_dir = os.path.abspath(os.path.join(toplevel_dir, build_dir)) + self.obj_ext = ".obj" if flavor == "win" else ".o" + if flavor == "win": + # See docstring of msvs_emulation.GenerateEnvironmentFiles(). + self.win_env = {} + for arch in ("x86", "x64"): + self.win_env[arch] = "environment." + arch + + # Relative path from build output dir to base dir. + build_to_top = gyp.common.InvertRelativePath(build_dir, toplevel_dir) + self.build_to_base = os.path.join(build_to_top, base_dir) + # Relative path from base dir to build dir. + base_to_top = gyp.common.InvertRelativePath(base_dir, toplevel_dir) + self.base_to_build = os.path.join(base_to_top, build_dir) + + def ExpandSpecial(self, path, product_dir=None): + """Expand specials like $!PRODUCT_DIR in |path|. If |product_dir| is None, assumes the cwd is already the product dir. Otherwise, |product_dir| is the relative path to the product dir. """ - PRODUCT_DIR = '$!PRODUCT_DIR' - if PRODUCT_DIR in path: - if product_dir: - path = path.replace(PRODUCT_DIR, product_dir) - else: - path = path.replace(PRODUCT_DIR + '/', '') - path = path.replace(PRODUCT_DIR + '\\', '') - path = path.replace(PRODUCT_DIR, '.') - - INTERMEDIATE_DIR = '$!INTERMEDIATE_DIR' - if INTERMEDIATE_DIR in path: - int_dir = self.GypPathToUniqueOutput('gen') - # GypPathToUniqueOutput generates a path relative to the product dir, - # so insert product_dir in front if it is provided. - path = path.replace(INTERMEDIATE_DIR, - os.path.join(product_dir or '', int_dir)) - - CONFIGURATION_NAME = '$|CONFIGURATION_NAME' - path = path.replace(CONFIGURATION_NAME, self.config_name) - - return path - - def ExpandRuleVariables(self, path, root, dirname, source, ext, name): - if self.flavor == 'win': - path = self.msvs_settings.ConvertVSMacros( - path, config=self.config_name) - path = path.replace(generator_default_variables['RULE_INPUT_ROOT'], root) - path = path.replace(generator_default_variables['RULE_INPUT_DIRNAME'], - dirname) - path = path.replace(generator_default_variables['RULE_INPUT_PATH'], source) - path = path.replace(generator_default_variables['RULE_INPUT_EXT'], ext) - path = path.replace(generator_default_variables['RULE_INPUT_NAME'], name) - return path - - def GypPathToNinja(self, path, env=None): - """Translate a gyp path to a ninja path, optionally expanding environment + PRODUCT_DIR = "$!PRODUCT_DIR" + if PRODUCT_DIR in path: + if product_dir: + path = path.replace(PRODUCT_DIR, product_dir) + else: + path = path.replace(PRODUCT_DIR + "/", "") + path = path.replace(PRODUCT_DIR + "\\", "") + path = path.replace(PRODUCT_DIR, ".") + + INTERMEDIATE_DIR = "$!INTERMEDIATE_DIR" + if INTERMEDIATE_DIR in path: + int_dir = self.GypPathToUniqueOutput("gen") + # GypPathToUniqueOutput generates a path relative to the product dir, + # so insert product_dir in front if it is provided. + path = path.replace( + INTERMEDIATE_DIR, os.path.join(product_dir or "", int_dir) + ) + + CONFIGURATION_NAME = "$|CONFIGURATION_NAME" + path = path.replace(CONFIGURATION_NAME, self.config_name) + + return path + + def ExpandRuleVariables(self, path, root, dirname, source, ext, name): + if self.flavor == "win": + path = self.msvs_settings.ConvertVSMacros(path, config=self.config_name) + path = path.replace(generator_default_variables["RULE_INPUT_ROOT"], root) + path = path.replace(generator_default_variables["RULE_INPUT_DIRNAME"], dirname) + path = path.replace(generator_default_variables["RULE_INPUT_PATH"], source) + path = path.replace(generator_default_variables["RULE_INPUT_EXT"], ext) + path = path.replace(generator_default_variables["RULE_INPUT_NAME"], name) + return path + + def GypPathToNinja(self, path, env=None): + """Translate a gyp path to a ninja path, optionally expanding environment variable references in |path| with |env|. See the above discourse on path conversions.""" - if env: - if self.flavor == 'mac': - path = gyp.xcode_emulation.ExpandEnvVars(path, env) - elif self.flavor == 'win': - path = gyp.msvs_emulation.ExpandMacros(path, env) - if path.startswith('$!'): - expanded = self.ExpandSpecial(path) - if self.flavor == 'win': - expanded = os.path.normpath(expanded) - return expanded - if '$|' in path: - path = self.ExpandSpecial(path) - assert '$' not in path, path - return os.path.normpath(os.path.join(self.build_to_base, path)) - - def GypPathToUniqueOutput(self, path, qualified=True): - """Translate a gyp path to a ninja path for writing output. + if env: + if self.flavor == "mac": + path = gyp.xcode_emulation.ExpandEnvVars(path, env) + elif self.flavor == "win": + path = gyp.msvs_emulation.ExpandMacros(path, env) + if path.startswith("$!"): + expanded = self.ExpandSpecial(path) + if self.flavor == "win": + expanded = os.path.normpath(expanded) + return expanded + if "$|" in path: + path = self.ExpandSpecial(path) + assert "$" not in path, path + return os.path.normpath(os.path.join(self.build_to_base, path)) + + def GypPathToUniqueOutput(self, path, qualified=True): + """Translate a gyp path to a ninja path for writing output. If qualified is True, qualify the resulting filename with the name of the target. This is necessary when e.g. compiling the same @@ -322,2180 +330,2605 @@ def GypPathToUniqueOutput(self, path, qualified=True): See the above discourse on path conversions.""" - path = self.ExpandSpecial(path) - assert not path.startswith('$'), path - - # Translate the path following this scheme: - # Input: foo/bar.gyp, target targ, references baz/out.o - # Output: obj/foo/baz/targ.out.o (if qualified) - # obj/foo/baz/out.o (otherwise) - # (and obj.host instead of obj for cross-compiles) - # - # Why this scheme and not some other one? - # 1) for a given input, you can compute all derived outputs by matching - # its path, even if the input is brought via a gyp file with '..'. - # 2) simple files like libraries and stamps have a simple filename. - - obj = 'obj' - if self.toolset != 'target': - obj += '.' + self.toolset - - path_dir, path_basename = os.path.split(path) - assert not os.path.isabs(path_dir), ( - "'%s' can not be absolute path (see crbug.com/462153)." % path_dir) - - if qualified: - path_basename = self.name + '.' + path_basename - return os.path.normpath(os.path.join(obj, self.base_dir, path_dir, - path_basename)) - - def WriteCollapsedDependencies(self, name, targets, order_only=None): - """Given a list of targets, return a path for a single file + path = self.ExpandSpecial(path) + assert not path.startswith("$"), path + + # Translate the path following this scheme: + # Input: foo/bar.gyp, target targ, references baz/out.o + # Output: obj/foo/baz/targ.out.o (if qualified) + # obj/foo/baz/out.o (otherwise) + # (and obj.host instead of obj for cross-compiles) + # + # Why this scheme and not some other one? + # 1) for a given input, you can compute all derived outputs by matching + # its path, even if the input is brought via a gyp file with '..'. + # 2) simple files like libraries and stamps have a simple filename. + + obj = "obj" + if self.toolset != "target": + obj += "." + self.toolset + + path_dir, path_basename = os.path.split(path) + assert not os.path.isabs(path_dir), ( + "'%s' can not be absolute path (see crbug.com/462153)." % path_dir + ) + + if qualified: + path_basename = self.name + "." + path_basename + return os.path.normpath( + os.path.join(obj, self.base_dir, path_dir, path_basename) + ) + + def WriteCollapsedDependencies(self, name, targets, order_only=None): + """Given a list of targets, return a path for a single file representing the result of building all the targets or None. Uses a stamp file if necessary.""" - assert targets == [item for item in targets if item], targets - if len(targets) == 0: - assert not order_only - return None - if len(targets) > 1 or order_only: - stamp = self.GypPathToUniqueOutput(name + '.stamp') - targets = self.ninja.build(stamp, 'stamp', targets, order_only=order_only) - self.ninja.newline() - return targets[0] + assert targets == [item for item in targets if item], targets + if len(targets) == 0: + assert not order_only + return None + if len(targets) > 1 or order_only: + stamp = self.GypPathToUniqueOutput(name + ".stamp") + targets = self.ninja.build(stamp, "stamp", targets, order_only=order_only) + self.ninja.newline() + return targets[0] - def _SubninjaNameForArch(self, arch): - output_file_base = os.path.splitext(self.output_file_name)[0] - return '%s.%s.ninja' % (output_file_base, arch) + def _SubninjaNameForArch(self, arch): + output_file_base = os.path.splitext(self.output_file_name)[0] + return "%s.%s.ninja" % (output_file_base, arch) - def WriteSpec(self, spec, config_name, generator_flags): - """The main entry point for NinjaWriter: write the build rules for a spec. + def WriteSpec(self, spec, config_name, generator_flags): + """The main entry point for NinjaWriter: write the build rules for a spec. Returns a Target object, which represents the output paths for this spec. Returns None if there are no outputs (e.g. a settings-only 'none' type target).""" - self.config_name = config_name - self.name = spec['target_name'] - self.toolset = spec['toolset'] - config = spec['configurations'][config_name] - self.target = Target(spec['type']) - self.is_standalone_static_library = bool( - spec.get('standalone_static_library', 0)) - - self.target_rpath = generator_flags.get('target_rpath', r'\$$ORIGIN/lib/') - - self.is_mac_bundle = gyp.xcode_emulation.IsMacBundle(self.flavor, spec) - self.xcode_settings = self.msvs_settings = None - if self.flavor == 'mac': - self.xcode_settings = gyp.xcode_emulation.XcodeSettings(spec) - mac_toolchain_dir = generator_flags.get('mac_toolchain_dir', None) - if mac_toolchain_dir: - self.xcode_settings.mac_toolchain_dir = mac_toolchain_dir - - if self.flavor == 'win': - self.msvs_settings = gyp.msvs_emulation.MsvsSettings(spec, - generator_flags) - arch = self.msvs_settings.GetArch(config_name) - self.ninja.variable('arch', self.win_env[arch]) - self.ninja.variable('cc', '$cl_' + arch) - self.ninja.variable('cxx', '$cl_' + arch) - self.ninja.variable('cc_host', '$cl_' + arch) - self.ninja.variable('cxx_host', '$cl_' + arch) - self.ninja.variable('asm', '$ml_' + arch) - - if self.flavor == 'mac': - self.archs = self.xcode_settings.GetActiveArchs(config_name) - if len(self.archs) > 1: - self.arch_subninjas = dict( - (arch, ninja_syntax.Writer( - OpenOutput(os.path.join(self.toplevel_build, - self._SubninjaNameForArch(arch)), - 'w'))) - for arch in self.archs) - - # Compute predepends for all rules. - # actions_depends is the dependencies this target depends on before running - # any of its action/rule/copy steps. - # compile_depends is the dependencies this target depends on before running - # any of its compile steps. - actions_depends = [] - compile_depends = [] - # TODO(evan): it is rather confusing which things are lists and which - # are strings. Fix these. - if 'dependencies' in spec: - for dep in spec['dependencies']: - if dep in self.target_outputs: - target = self.target_outputs[dep] - actions_depends.append(target.PreActionInput(self.flavor)) - compile_depends.append(target.PreCompileInput()) - if target.uses_cpp: - self.target.uses_cpp = True - actions_depends = [item for item in actions_depends if item] - compile_depends = [item for item in compile_depends if item] - actions_depends = self.WriteCollapsedDependencies('actions_depends', - actions_depends) - compile_depends = self.WriteCollapsedDependencies('compile_depends', - compile_depends) - self.target.preaction_stamp = actions_depends - self.target.precompile_stamp = compile_depends - - # Write out actions, rules, and copies. These must happen before we - # compile any sources, so compute a list of predependencies for sources - # while we do it. - extra_sources = [] - mac_bundle_depends = [] - self.target.actions_stamp = self.WriteActionsRulesCopies( - spec, extra_sources, actions_depends, mac_bundle_depends) - - # If we have actions/rules/copies, we depend directly on those, but - # otherwise we depend on dependent target's actions/rules/copies etc. - # We never need to explicitly depend on previous target's link steps, - # because no compile ever depends on them. - compile_depends_stamp = (self.target.actions_stamp or compile_depends) - - # Write out the compilation steps, if any. - link_deps = [] - try: - sources = extra_sources + spec.get('sources', []) - except TypeError: - print('extra_sources: ', str(extra_sources)) - print('spec.get("sources"): ', str(spec.get('sources'))) - raise - if sources: - if self.flavor == 'mac' and len(self.archs) > 1: - # Write subninja file containing compile and link commands scoped to - # a single arch if a fat binary is being built. - for arch in self.archs: - self.ninja.subninja(self._SubninjaNameForArch(arch)) - - pch = None - if self.flavor == 'win': - gyp.msvs_emulation.VerifyMissingSources( - sources, self.abs_build_dir, generator_flags, self.GypPathToNinja) - pch = gyp.msvs_emulation.PrecompiledHeader( - self.msvs_settings, config_name, self.GypPathToNinja, - self.GypPathToUniqueOutput, self.obj_ext) - else: - pch = gyp.xcode_emulation.MacPrefixHeader( - self.xcode_settings, self.GypPathToNinja, - lambda path, lang: self.GypPathToUniqueOutput(path + '-' + lang)) - link_deps = self.WriteSources( - self.ninja, config_name, config, sources, compile_depends_stamp, pch, - spec) - # Some actions/rules output 'sources' that are already object files. - obj_outputs = [f for f in sources if f.endswith(self.obj_ext)] - if obj_outputs: - if self.flavor != 'mac' or len(self.archs) == 1: - link_deps += [self.GypPathToNinja(o) for o in obj_outputs] - else: - print("Warning: Actions/rules writing object files don't work with " - "multiarch targets, dropping. (target %s)" % spec['target_name']) - elif self.flavor == 'mac' and len(self.archs) > 1: - link_deps = collections.defaultdict(list) - - compile_deps = self.target.actions_stamp or actions_depends - if self.flavor == 'win' and self.target.type == 'static_library': - self.target.component_objs = link_deps - self.target.compile_deps = compile_deps - - # Write out a link step, if needed. - output = None - is_empty_bundle = not link_deps and not mac_bundle_depends - if link_deps or self.target.actions_stamp or actions_depends: - output = self.WriteTarget(spec, config_name, config, link_deps, - compile_deps) - if self.is_mac_bundle: - mac_bundle_depends.append(output) - - # Bundle all of the above together, if needed. - if self.is_mac_bundle: - output = self.WriteMacBundle(spec, mac_bundle_depends, is_empty_bundle) - - if not output: - return None - - assert self.target.FinalOutput(), output - return self.target - - def _WinIdlRule(self, source, prebuild, outputs): - """Handle the implicit VS .idl rule for one source file. Fills |outputs| - with files that are generated.""" - outdir, output, vars, flags = self.msvs_settings.GetIdlBuildData( - source, self.config_name) - outdir = self.GypPathToNinja(outdir) - def fix_path(path, rel=None): - path = os.path.join(outdir, path) - dirname, basename = os.path.split(source) - root, ext = os.path.splitext(basename) - path = self.ExpandRuleVariables( - path, root, dirname, source, ext, basename) - if rel: - path = os.path.relpath(path, rel) - return path - vars = [(name, fix_path(value, outdir)) for name, value in vars] - output = [fix_path(p) for p in output] - vars.append(('outdir', outdir)) - vars.append(('idlflags', flags)) - input = self.GypPathToNinja(source) - self.ninja.build(output, 'idl', input, - variables=vars, order_only=prebuild) - outputs.extend(output) - - def WriteWinIdlFiles(self, spec, prebuild): - """Writes rules to match MSVS's implicit idl handling.""" - assert self.flavor == 'win' - if self.msvs_settings.HasExplicitIdlRulesOrActions(spec): - return [] - outputs = [] - for source in filter(lambda x: x.endswith('.idl'), spec['sources']): - self._WinIdlRule(source, prebuild, outputs) - return outputs - - def WriteActionsRulesCopies(self, spec, extra_sources, prebuild, - mac_bundle_depends): - """Write out the Actions, Rules, and Copies steps. Return a path - representing the outputs of these steps.""" - outputs = [] - if self.is_mac_bundle: - mac_bundle_resources = spec.get('mac_bundle_resources', [])[:] - else: - mac_bundle_resources = [] - extra_mac_bundle_resources = [] - - if 'actions' in spec: - outputs += self.WriteActions(spec['actions'], extra_sources, prebuild, - extra_mac_bundle_resources) - if 'rules' in spec: - outputs += self.WriteRules(spec['rules'], extra_sources, prebuild, - mac_bundle_resources, - extra_mac_bundle_resources) - if 'copies' in spec: - outputs += self.WriteCopies(spec['copies'], prebuild, mac_bundle_depends) + self.config_name = config_name + self.name = spec["target_name"] + self.toolset = spec["toolset"] + config = spec["configurations"][config_name] + self.target = Target(spec["type"]) + self.is_standalone_static_library = bool( + spec.get("standalone_static_library", 0) + ) + + self.target_rpath = generator_flags.get("target_rpath", r"\$$ORIGIN/lib/") + + self.is_mac_bundle = gyp.xcode_emulation.IsMacBundle(self.flavor, spec) + self.xcode_settings = self.msvs_settings = None + if self.flavor == "mac": + self.xcode_settings = gyp.xcode_emulation.XcodeSettings(spec) + mac_toolchain_dir = generator_flags.get("mac_toolchain_dir", None) + if mac_toolchain_dir: + self.xcode_settings.mac_toolchain_dir = mac_toolchain_dir + + if self.flavor == "win": + self.msvs_settings = gyp.msvs_emulation.MsvsSettings(spec, generator_flags) + arch = self.msvs_settings.GetArch(config_name) + self.ninja.variable("arch", self.win_env[arch]) + self.ninja.variable("cc", "$cl_" + arch) + self.ninja.variable("cxx", "$cl_" + arch) + self.ninja.variable("cc_host", "$cl_" + arch) + self.ninja.variable("cxx_host", "$cl_" + arch) + self.ninja.variable("asm", "$ml_" + arch) + + if self.flavor == "mac": + self.archs = self.xcode_settings.GetActiveArchs(config_name) + if len(self.archs) > 1: + self.arch_subninjas = dict( + ( + arch, + ninja_syntax.Writer( + OpenOutput( + os.path.join( + self.toplevel_build, self._SubninjaNameForArch(arch) + ), + "w", + ) + ), + ) + for arch in self.archs + ) + + # Compute predepends for all rules. + # actions_depends is the dependencies this target depends on before running + # any of its action/rule/copy steps. + # compile_depends is the dependencies this target depends on before running + # any of its compile steps. + actions_depends = [] + compile_depends = [] + # TODO(evan): it is rather confusing which things are lists and which + # are strings. Fix these. + if "dependencies" in spec: + for dep in spec["dependencies"]: + if dep in self.target_outputs: + target = self.target_outputs[dep] + actions_depends.append(target.PreActionInput(self.flavor)) + compile_depends.append(target.PreCompileInput()) + if target.uses_cpp: + self.target.uses_cpp = True + actions_depends = [item for item in actions_depends if item] + compile_depends = [item for item in compile_depends if item] + actions_depends = self.WriteCollapsedDependencies( + "actions_depends", actions_depends + ) + compile_depends = self.WriteCollapsedDependencies( + "compile_depends", compile_depends + ) + self.target.preaction_stamp = actions_depends + self.target.precompile_stamp = compile_depends + + # Write out actions, rules, and copies. These must happen before we + # compile any sources, so compute a list of predependencies for sources + # while we do it. + extra_sources = [] + mac_bundle_depends = [] + self.target.actions_stamp = self.WriteActionsRulesCopies( + spec, extra_sources, actions_depends, mac_bundle_depends + ) + + # If we have actions/rules/copies, we depend directly on those, but + # otherwise we depend on dependent target's actions/rules/copies etc. + # We never need to explicitly depend on previous target's link steps, + # because no compile ever depends on them. + compile_depends_stamp = self.target.actions_stamp or compile_depends + + # Write out the compilation steps, if any. + link_deps = [] + try: + sources = extra_sources + spec.get("sources", []) + except TypeError: + print("extra_sources: ", str(extra_sources)) + print('spec.get("sources"): ', str(spec.get("sources"))) + raise + if sources: + if self.flavor == "mac" and len(self.archs) > 1: + # Write subninja file containing compile and link commands scoped to + # a single arch if a fat binary is being built. + for arch in self.archs: + self.ninja.subninja(self._SubninjaNameForArch(arch)) + + pch = None + if self.flavor == "win": + gyp.msvs_emulation.VerifyMissingSources( + sources, self.abs_build_dir, generator_flags, self.GypPathToNinja + ) + pch = gyp.msvs_emulation.PrecompiledHeader( + self.msvs_settings, + config_name, + self.GypPathToNinja, + self.GypPathToUniqueOutput, + self.obj_ext, + ) + else: + pch = gyp.xcode_emulation.MacPrefixHeader( + self.xcode_settings, + self.GypPathToNinja, + lambda path, lang: self.GypPathToUniqueOutput(path + "-" + lang), + ) + link_deps = self.WriteSources( + self.ninja, + config_name, + config, + sources, + compile_depends_stamp, + pch, + spec, + ) + # Some actions/rules output 'sources' that are already object files. + obj_outputs = [f for f in sources if f.endswith(self.obj_ext)] + if obj_outputs: + if self.flavor != "mac" or len(self.archs) == 1: + link_deps += [self.GypPathToNinja(o) for o in obj_outputs] + else: + print( + "Warning: Actions/rules writing object files don't work with " + "multiarch targets, dropping. (target %s)" % spec["target_name"] + ) + elif self.flavor == "mac" and len(self.archs) > 1: + link_deps = collections.defaultdict(list) + + compile_deps = self.target.actions_stamp or actions_depends + if self.flavor == "win" and self.target.type == "static_library": + self.target.component_objs = link_deps + self.target.compile_deps = compile_deps + + # Write out a link step, if needed. + output = None + is_empty_bundle = not link_deps and not mac_bundle_depends + if link_deps or self.target.actions_stamp or actions_depends: + output = self.WriteTarget( + spec, config_name, config, link_deps, compile_deps + ) + if self.is_mac_bundle: + mac_bundle_depends.append(output) + + # Bundle all of the above together, if needed. + if self.is_mac_bundle: + output = self.WriteMacBundle(spec, mac_bundle_depends, is_empty_bundle) - if 'sources' in spec and self.flavor == 'win': - outputs += self.WriteWinIdlFiles(spec, prebuild) + if not output: + return None - if self.xcode_settings and self.xcode_settings.IsIosFramework(): - self.WriteiOSFrameworkHeaders(spec, outputs, prebuild) + assert self.target.FinalOutput(), output + return self.target - stamp = self.WriteCollapsedDependencies('actions_rules_copies', outputs) + def _WinIdlRule(self, source, prebuild, outputs): + """Handle the implicit VS .idl rule for one source file. Fills |outputs| + with files that are generated.""" + outdir, output, vars, flags = self.msvs_settings.GetIdlBuildData( + source, self.config_name + ) + outdir = self.GypPathToNinja(outdir) + + def fix_path(path, rel=None): + path = os.path.join(outdir, path) + dirname, basename = os.path.split(source) + root, ext = os.path.splitext(basename) + path = self.ExpandRuleVariables(path, root, dirname, source, ext, basename) + if rel: + path = os.path.relpath(path, rel) + return path + + vars = [(name, fix_path(value, outdir)) for name, value in vars] + output = [fix_path(p) for p in output] + vars.append(("outdir", outdir)) + vars.append(("idlflags", flags)) + input = self.GypPathToNinja(source) + self.ninja.build(output, "idl", input, variables=vars, order_only=prebuild) + outputs.extend(output) + + def WriteWinIdlFiles(self, spec, prebuild): + """Writes rules to match MSVS's implicit idl handling.""" + assert self.flavor == "win" + if self.msvs_settings.HasExplicitIdlRulesOrActions(spec): + return [] + outputs = [] + for source in filter(lambda x: x.endswith(".idl"), spec["sources"]): + self._WinIdlRule(source, prebuild, outputs) + return outputs + + def WriteActionsRulesCopies( + self, spec, extra_sources, prebuild, mac_bundle_depends + ): + """Write out the Actions, Rules, and Copies steps. Return a path + representing the outputs of these steps.""" + outputs = [] + if self.is_mac_bundle: + mac_bundle_resources = spec.get("mac_bundle_resources", [])[:] + else: + mac_bundle_resources = [] + extra_mac_bundle_resources = [] + + if "actions" in spec: + outputs += self.WriteActions( + spec["actions"], extra_sources, prebuild, extra_mac_bundle_resources + ) + if "rules" in spec: + outputs += self.WriteRules( + spec["rules"], + extra_sources, + prebuild, + mac_bundle_resources, + extra_mac_bundle_resources, + ) + if "copies" in spec: + outputs += self.WriteCopies(spec["copies"], prebuild, mac_bundle_depends) + + if "sources" in spec and self.flavor == "win": + outputs += self.WriteWinIdlFiles(spec, prebuild) + + if self.xcode_settings and self.xcode_settings.IsIosFramework(): + self.WriteiOSFrameworkHeaders(spec, outputs, prebuild) + + stamp = self.WriteCollapsedDependencies("actions_rules_copies", outputs) - if self.is_mac_bundle: - xcassets = self.WriteMacBundleResources( - extra_mac_bundle_resources + mac_bundle_resources, mac_bundle_depends) - partial_info_plist = self.WriteMacXCassets(xcassets, mac_bundle_depends) - self.WriteMacInfoPlist(partial_info_plist, mac_bundle_depends) + if self.is_mac_bundle: + xcassets = self.WriteMacBundleResources( + extra_mac_bundle_resources + mac_bundle_resources, mac_bundle_depends + ) + partial_info_plist = self.WriteMacXCassets(xcassets, mac_bundle_depends) + self.WriteMacInfoPlist(partial_info_plist, mac_bundle_depends) - return stamp + return stamp - def GenerateDescription(self, verb, message, fallback): - """Generate and return a description of a build step. + def GenerateDescription(self, verb, message, fallback): + """Generate and return a description of a build step. |verb| is the short summary, e.g. ACTION or RULE. |message| is a hand-written description, or None if not available. |fallback| is the gyp-level name of the step, usable as a fallback. """ - if self.toolset != 'target': - verb += '(%s)' % self.toolset - if message: - return '%s %s' % (verb, self.ExpandSpecial(message)) - else: - return '%s %s: %s' % (verb, self.name, fallback) - - def WriteActions(self, actions, extra_sources, prebuild, - extra_mac_bundle_resources): - # Actions cd into the base directory. - env = self.GetToolchainEnv() - all_outputs = [] - for action in actions: - # First write out a rule for the action. - name = '%s_%s' % (action['action_name'], self.hash_for_rules) - description = self.GenerateDescription('ACTION', - action.get('message', None), - name) - is_cygwin = (self.msvs_settings.IsRuleRunUnderCygwin(action) - if self.flavor == 'win' else False) - args = action['action'] - depfile = action.get('depfile', None) - if depfile: - depfile = self.ExpandSpecial(depfile, self.base_to_build) - pool = 'console' if int(action.get('ninja_use_console', 0)) else None - rule_name, _ = self.WriteNewNinjaRule(name, args, description, - is_cygwin, env, pool, - depfile=depfile) - - inputs = [self.GypPathToNinja(i, env) for i in action['inputs']] - if int(action.get('process_outputs_as_sources', False)): - extra_sources += action['outputs'] - if int(action.get('process_outputs_as_mac_bundle_resources', False)): - extra_mac_bundle_resources += action['outputs'] - outputs = [self.GypPathToNinja(o, env) for o in action['outputs']] - - # Then write out an edge using the rule. - self.ninja.build(outputs, rule_name, inputs, - order_only=prebuild) - all_outputs += outputs - - self.ninja.newline() - - return all_outputs - - def WriteRules(self, rules, extra_sources, prebuild, - mac_bundle_resources, extra_mac_bundle_resources): - env = self.GetToolchainEnv() - all_outputs = [] - for rule in rules: - # Skip a rule with no action and no inputs. - if 'action' not in rule and not rule.get('rule_sources', []): - continue - - # First write out a rule for the rule action. - name = '%s_%s' % (rule['rule_name'], self.hash_for_rules) - - args = rule['action'] - description = self.GenerateDescription( - 'RULE', - rule.get('message', None), - ('%s ' + generator_default_variables['RULE_INPUT_PATH']) % name) - is_cygwin = (self.msvs_settings.IsRuleRunUnderCygwin(rule) - if self.flavor == 'win' else False) - pool = 'console' if int(rule.get('ninja_use_console', 0)) else None - rule_name, args = self.WriteNewNinjaRule( - name, args, description, is_cygwin, env, pool) - - # TODO: if the command references the outputs directly, we should - # simplify it to just use $out. - - # Rules can potentially make use of some special variables which - # must vary per source file. - # Compute the list of variables we'll need to provide. - special_locals = ('source', 'root', 'dirname', 'ext', 'name') - needed_variables = set(['source']) - for argument in args: - for var in special_locals: - if '${%s}' % var in argument: - needed_variables.add(var) - needed_variables = sorted(needed_variables) - - def cygwin_munge(path): - # pylint: disable=cell-var-from-loop - if is_cygwin: - return path.replace('\\', '/') - return path - - inputs = [self.GypPathToNinja(i, env) for i in rule.get('inputs', [])] - - # If there are n source files matching the rule, and m additional rule - # inputs, then adding 'inputs' to each build edge written below will - # write m * n inputs. Collapsing reduces this to m + n. - sources = rule.get('rule_sources', []) - num_inputs = len(inputs) - if prebuild: - num_inputs += 1 - if num_inputs > 2 and len(sources) > 2: - inputs = [self.WriteCollapsedDependencies( - rule['rule_name'], inputs, order_only=prebuild)] - prebuild = [] - - # For each source file, write an edge that generates all the outputs. - for source in sources: - source = os.path.normpath(source) - dirname, basename = os.path.split(source) - root, ext = os.path.splitext(basename) - - # Gather the list of inputs and outputs, expanding $vars if possible. - outputs = [self.ExpandRuleVariables(o, root, dirname, - source, ext, basename) - for o in rule['outputs']] - - if int(rule.get('process_outputs_as_sources', False)): - extra_sources += outputs - - was_mac_bundle_resource = source in mac_bundle_resources - if was_mac_bundle_resource or \ - int(rule.get('process_outputs_as_mac_bundle_resources', False)): - extra_mac_bundle_resources += outputs - # Note: This is n_resources * n_outputs_in_rule. Put to-be-removed - # items in a set and remove them all in a single pass if this becomes - # a performance issue. - if was_mac_bundle_resource: - mac_bundle_resources.remove(source) - - extra_bindings = [] - for var in needed_variables: - if var == 'root': - extra_bindings.append(('root', cygwin_munge(root))) - elif var == 'dirname': - # '$dirname' is a parameter to the rule action, which means - # it shouldn't be converted to a Ninja path. But we don't - # want $!PRODUCT_DIR in there either. - dirname_expanded = self.ExpandSpecial(dirname, self.base_to_build) - extra_bindings.append(('dirname', cygwin_munge(dirname_expanded))) - elif var == 'source': - # '$source' is a parameter to the rule action, which means - # it shouldn't be converted to a Ninja path. But we don't - # want $!PRODUCT_DIR in there either. - source_expanded = self.ExpandSpecial(source, self.base_to_build) - extra_bindings.append(('source', cygwin_munge(source_expanded))) - elif var == 'ext': - extra_bindings.append(('ext', ext)) - elif var == 'name': - extra_bindings.append(('name', cygwin_munge(basename))) - else: - assert var is None, repr(var) - - outputs = [self.GypPathToNinja(o, env) for o in outputs] - if self.flavor == 'win': - # WriteNewNinjaRule uses unique_name for creating an rsp file on win. - extra_bindings.append(('unique_name', - hashlib.md5(outputs[0]).hexdigest())) - - self.ninja.build(outputs, rule_name, self.GypPathToNinja(source), - implicit=inputs, - order_only=prebuild, - variables=extra_bindings) - - all_outputs.extend(outputs) - - return all_outputs - - def WriteCopies(self, copies, prebuild, mac_bundle_depends): - outputs = [] - if self.xcode_settings: - extra_env = self.xcode_settings.GetPerTargetSettings() - env = self.GetToolchainEnv(additional_settings=extra_env) - else: - env = self.GetToolchainEnv() - for copy in copies: - for path in copy['files']: - # Normalize the path so trailing slashes don't confuse us. - path = os.path.normpath(path) - basename = os.path.split(path)[1] - src = self.GypPathToNinja(path, env) - dst = self.GypPathToNinja(os.path.join(copy['destination'], basename), - env) - outputs += self.ninja.build(dst, 'copy', src, order_only=prebuild) - if self.is_mac_bundle: - # gyp has mac_bundle_resources to copy things into a bundle's - # Resources folder, but there's no built-in way to copy files to other - # places in the bundle. Hence, some targets use copies for this. Check - # if this file is copied into the current bundle, and if so add it to - # the bundle depends so that dependent targets get rebuilt if the copy - # input changes. - if dst.startswith(self.xcode_settings.GetBundleContentsFolderPath()): - mac_bundle_depends.append(dst) - - return outputs - - def WriteiOSFrameworkHeaders(self, spec, outputs, prebuild): - """Prebuild steps to generate hmap files and copy headers to destination.""" - framework = self.ComputeMacBundleOutput() - all_sources = spec['sources'] - copy_headers = spec['mac_framework_headers'] - output = self.GypPathToUniqueOutput('headers.hmap') - self.xcode_settings.header_map_path = output - all_headers = map(self.GypPathToNinja, - filter(lambda x:x.endswith(('.h')), all_sources)) - variables = [('framework', framework), - ('copy_headers', map(self.GypPathToNinja, copy_headers))] - outputs.extend(self.ninja.build( - output, 'compile_ios_framework_headers', all_headers, - variables=variables, order_only=prebuild)) - - def WriteMacBundleResources(self, resources, bundle_depends): - """Writes ninja edges for 'mac_bundle_resources'.""" - xcassets = [] - - extra_env = self.xcode_settings.GetPerTargetSettings() - env = self.GetSortedXcodeEnv(additional_settings=extra_env) - env = self.ComputeExportEnvString(env) - isBinary = self.xcode_settings.IsBinaryOutputFormat(self.config_name) - - for output, res in gyp.xcode_emulation.GetMacBundleResources( - generator_default_variables['PRODUCT_DIR'], - self.xcode_settings, map(self.GypPathToNinja, resources)): - output = self.ExpandSpecial(output) - if os.path.splitext(output)[-1] != '.xcassets': - self.ninja.build(output, 'mac_tool', res, - variables=[('mactool_cmd', 'copy-bundle-resource'), \ - ('env', env), ('binary', isBinary)]) - bundle_depends.append(output) - else: - xcassets.append(res) - return xcassets - - def WriteMacXCassets(self, xcassets, bundle_depends): - """Writes ninja edges for 'mac_bundle_resources' .xcassets files. + if self.toolset != "target": + verb += "(%s)" % self.toolset + if message: + return "%s %s" % (verb, self.ExpandSpecial(message)) + else: + return "%s %s: %s" % (verb, self.name, fallback) + + def WriteActions( + self, actions, extra_sources, prebuild, extra_mac_bundle_resources + ): + # Actions cd into the base directory. + env = self.GetToolchainEnv() + all_outputs = [] + for action in actions: + # First write out a rule for the action. + name = "%s_%s" % (action["action_name"], self.hash_for_rules) + description = self.GenerateDescription( + "ACTION", action.get("message", None), name + ) + is_cygwin = ( + self.msvs_settings.IsRuleRunUnderCygwin(action) + if self.flavor == "win" + else False + ) + args = action["action"] + depfile = action.get("depfile", None) + if depfile: + depfile = self.ExpandSpecial(depfile, self.base_to_build) + pool = "console" if int(action.get("ninja_use_console", 0)) else None + rule_name, _ = self.WriteNewNinjaRule( + name, args, description, is_cygwin, env, pool, depfile=depfile + ) + + inputs = [self.GypPathToNinja(i, env) for i in action["inputs"]] + if int(action.get("process_outputs_as_sources", False)): + extra_sources += action["outputs"] + if int(action.get("process_outputs_as_mac_bundle_resources", False)): + extra_mac_bundle_resources += action["outputs"] + outputs = [self.GypPathToNinja(o, env) for o in action["outputs"]] + + # Then write out an edge using the rule. + self.ninja.build(outputs, rule_name, inputs, order_only=prebuild) + all_outputs += outputs + + self.ninja.newline() + + return all_outputs + + def WriteRules( + self, + rules, + extra_sources, + prebuild, + mac_bundle_resources, + extra_mac_bundle_resources, + ): + env = self.GetToolchainEnv() + all_outputs = [] + for rule in rules: + # Skip a rule with no action and no inputs. + if "action" not in rule and not rule.get("rule_sources", []): + continue + + # First write out a rule for the rule action. + name = "%s_%s" % (rule["rule_name"], self.hash_for_rules) + + args = rule["action"] + description = self.GenerateDescription( + "RULE", + rule.get("message", None), + ("%s " + generator_default_variables["RULE_INPUT_PATH"]) % name, + ) + is_cygwin = ( + self.msvs_settings.IsRuleRunUnderCygwin(rule) + if self.flavor == "win" + else False + ) + pool = "console" if int(rule.get("ninja_use_console", 0)) else None + rule_name, args = self.WriteNewNinjaRule( + name, args, description, is_cygwin, env, pool + ) + + # TODO: if the command references the outputs directly, we should + # simplify it to just use $out. + + # Rules can potentially make use of some special variables which + # must vary per source file. + # Compute the list of variables we'll need to provide. + special_locals = ("source", "root", "dirname", "ext", "name") + needed_variables = set(["source"]) + for argument in args: + for var in special_locals: + if "${%s}" % var in argument: + needed_variables.add(var) + needed_variables = sorted(needed_variables) + + def cygwin_munge(path): + # pylint: disable=cell-var-from-loop + if is_cygwin: + return path.replace("\\", "/") + return path + + inputs = [self.GypPathToNinja(i, env) for i in rule.get("inputs", [])] + + # If there are n source files matching the rule, and m additional rule + # inputs, then adding 'inputs' to each build edge written below will + # write m * n inputs. Collapsing reduces this to m + n. + sources = rule.get("rule_sources", []) + num_inputs = len(inputs) + if prebuild: + num_inputs += 1 + if num_inputs > 2 and len(sources) > 2: + inputs = [ + self.WriteCollapsedDependencies( + rule["rule_name"], inputs, order_only=prebuild + ) + ] + prebuild = [] + + # For each source file, write an edge that generates all the outputs. + for source in sources: + source = os.path.normpath(source) + dirname, basename = os.path.split(source) + root, ext = os.path.splitext(basename) + + # Gather the list of inputs and outputs, expanding $vars if possible. + outputs = [ + self.ExpandRuleVariables(o, root, dirname, source, ext, basename) + for o in rule["outputs"] + ] + + if int(rule.get("process_outputs_as_sources", False)): + extra_sources += outputs + + was_mac_bundle_resource = source in mac_bundle_resources + if was_mac_bundle_resource or int( + rule.get("process_outputs_as_mac_bundle_resources", False) + ): + extra_mac_bundle_resources += outputs + # Note: This is n_resources * n_outputs_in_rule. Put to-be-removed + # items in a set and remove them all in a single pass if this becomes + # a performance issue. + if was_mac_bundle_resource: + mac_bundle_resources.remove(source) + + extra_bindings = [] + for var in needed_variables: + if var == "root": + extra_bindings.append(("root", cygwin_munge(root))) + elif var == "dirname": + # '$dirname' is a parameter to the rule action, which means + # it shouldn't be converted to a Ninja path. But we don't + # want $!PRODUCT_DIR in there either. + dirname_expanded = self.ExpandSpecial( + dirname, self.base_to_build + ) + extra_bindings.append( + ("dirname", cygwin_munge(dirname_expanded)) + ) + elif var == "source": + # '$source' is a parameter to the rule action, which means + # it shouldn't be converted to a Ninja path. But we don't + # want $!PRODUCT_DIR in there either. + source_expanded = self.ExpandSpecial(source, self.base_to_build) + extra_bindings.append(("source", cygwin_munge(source_expanded))) + elif var == "ext": + extra_bindings.append(("ext", ext)) + elif var == "name": + extra_bindings.append(("name", cygwin_munge(basename))) + else: + assert var is None, repr(var) + + outputs = [self.GypPathToNinja(o, env) for o in outputs] + if self.flavor == "win": + # WriteNewNinjaRule uses unique_name for creating an rsp file on win. + extra_bindings.append( + ("unique_name", hashlib.md5(outputs[0]).hexdigest()) + ) + + self.ninja.build( + outputs, + rule_name, + self.GypPathToNinja(source), + implicit=inputs, + order_only=prebuild, + variables=extra_bindings, + ) + + all_outputs.extend(outputs) + + return all_outputs + + def WriteCopies(self, copies, prebuild, mac_bundle_depends): + outputs = [] + if self.xcode_settings: + extra_env = self.xcode_settings.GetPerTargetSettings() + env = self.GetToolchainEnv(additional_settings=extra_env) + else: + env = self.GetToolchainEnv() + for to_copy in copies: + for path in to_copy["files"]: + # Normalize the path so trailing slashes don't confuse us. + path = os.path.normpath(path) + basename = os.path.split(path)[1] + src = self.GypPathToNinja(path, env) + dst = self.GypPathToNinja( + os.path.join(to_copy["destination"], basename), env + ) + outputs += self.ninja.build(dst, "copy", src, order_only=prebuild) + if self.is_mac_bundle: + # gyp has mac_bundle_resources to copy things into a bundle's + # Resources folder, but there's no built-in way to copy files to other + # places in the bundle. Hence, some targets use copies for this. Check + # if this file is copied into the current bundle, and if so add it to + # the bundle depends so that dependent targets get rebuilt if the copy + # input changes. + if dst.startswith( + self.xcode_settings.GetBundleContentsFolderPath() + ): + mac_bundle_depends.append(dst) + + return outputs + + def WriteiOSFrameworkHeaders(self, spec, outputs, prebuild): + """Prebuild steps to generate hmap files and copy headers to destination.""" + framework = self.ComputeMacBundleOutput() + all_sources = spec["sources"] + copy_headers = spec["mac_framework_headers"] + output = self.GypPathToUniqueOutput("headers.hmap") + self.xcode_settings.header_map_path = output + all_headers = map( + self.GypPathToNinja, filter(lambda x: x.endswith((".h")), all_sources) + ) + variables = [ + ("framework", framework), + ("copy_headers", map(self.GypPathToNinja, copy_headers)), + ] + outputs.extend( + self.ninja.build( + output, + "compile_ios_framework_headers", + all_headers, + variables=variables, + order_only=prebuild, + ) + ) + + def WriteMacBundleResources(self, resources, bundle_depends): + """Writes ninja edges for 'mac_bundle_resources'.""" + xcassets = [] + + extra_env = self.xcode_settings.GetPerTargetSettings() + env = self.GetSortedXcodeEnv(additional_settings=extra_env) + env = self.ComputeExportEnvString(env) + isBinary = self.xcode_settings.IsBinaryOutputFormat(self.config_name) + + for output, res in gyp.xcode_emulation.GetMacBundleResources( + generator_default_variables["PRODUCT_DIR"], + self.xcode_settings, + map(self.GypPathToNinja, resources), + ): + output = self.ExpandSpecial(output) + if os.path.splitext(output)[-1] != ".xcassets": + self.ninja.build( + output, + "mac_tool", + res, + variables=[ + ("mactool_cmd", "copy-bundle-resource"), + ("env", env), + ("binary", isBinary), + ], + ) + bundle_depends.append(output) + else: + xcassets.append(res) + return xcassets + + def WriteMacXCassets(self, xcassets, bundle_depends): + """Writes ninja edges for 'mac_bundle_resources' .xcassets files. This add an invocation of 'actool' via the 'mac_tool.py' helper script. It assumes that the assets catalogs define at least one imageset and thus an Assets.car file will be generated in the application resources directory. If this is not the case, then the build will probably be done at each invocation of ninja.""" - if not xcassets: - return - - extra_arguments = {} - settings_to_arg = { - 'XCASSETS_APP_ICON': 'app-icon', - 'XCASSETS_LAUNCH_IMAGE': 'launch-image', - } - settings = self.xcode_settings.xcode_settings[self.config_name] - for settings_key, arg_name in settings_to_arg.items(): - value = settings.get(settings_key) - if value: - extra_arguments[arg_name] = value - - partial_info_plist = None - if extra_arguments: - partial_info_plist = self.GypPathToUniqueOutput( - 'assetcatalog_generated_info.plist') - extra_arguments['output-partial-info-plist'] = partial_info_plist - - outputs = [] - outputs.append( - os.path.join( - self.xcode_settings.GetBundleResourceFolder(), - 'Assets.car')) - if partial_info_plist: - outputs.append(partial_info_plist) - - keys = QuoteShellArgument(json.dumps(extra_arguments), self.flavor) - extra_env = self.xcode_settings.GetPerTargetSettings() - env = self.GetSortedXcodeEnv(additional_settings=extra_env) - env = self.ComputeExportEnvString(env) - - bundle_depends.extend(self.ninja.build( - outputs, 'compile_xcassets', xcassets, - variables=[('env', env), ('keys', keys)])) - return partial_info_plist - - def WriteMacInfoPlist(self, partial_info_plist, bundle_depends): - """Write build rules for bundle Info.plist files.""" - info_plist, out, defines, extra_env = gyp.xcode_emulation.GetMacInfoPlist( - generator_default_variables['PRODUCT_DIR'], - self.xcode_settings, self.GypPathToNinja) - if not info_plist: - return - out = self.ExpandSpecial(out) - if defines: - # Create an intermediate file to store preprocessed results. - intermediate_plist = self.GypPathToUniqueOutput( - os.path.basename(info_plist)) - defines = ' '.join([Define(d, self.flavor) for d in defines]) - info_plist = self.ninja.build( - intermediate_plist, 'preprocess_infoplist', info_plist, - variables=[('defines',defines)]) - - env = self.GetSortedXcodeEnv(additional_settings=extra_env) - env = self.ComputeExportEnvString(env) - - if partial_info_plist: - intermediate_plist = self.GypPathToUniqueOutput('merged_info.plist') - info_plist = self.ninja.build( - intermediate_plist, 'merge_infoplist', - [partial_info_plist, info_plist]) - - keys = self.xcode_settings.GetExtraPlistItems(self.config_name) - keys = QuoteShellArgument(json.dumps(keys), self.flavor) - isBinary = self.xcode_settings.IsBinaryOutputFormat(self.config_name) - self.ninja.build(out, 'copy_infoplist', info_plist, - variables=[('env', env), ('keys', keys), - ('binary', isBinary)]) - bundle_depends.append(out) - - def WriteSources(self, ninja_file, config_name, config, sources, predepends, - precompiled_header, spec): - """Write build rules to compile all of |sources|.""" - if self.toolset == 'host': - self.ninja.variable('ar', '$ar_host') - self.ninja.variable('cc', '$cc_host') - self.ninja.variable('cxx', '$cxx_host') - self.ninja.variable('ld', '$ld_host') - self.ninja.variable('ldxx', '$ldxx_host') - self.ninja.variable('nm', '$nm_host') - self.ninja.variable('readelf', '$readelf_host') - - if self.flavor != 'mac' or len(self.archs) == 1: - return self.WriteSourcesForArch( - self.ninja, config_name, config, sources, predepends, - precompiled_header, spec) - else: - return dict((arch, self.WriteSourcesForArch( - self.arch_subninjas[arch], config_name, config, sources, predepends, - precompiled_header, spec, arch=arch)) - for arch in self.archs) - - def WriteSourcesForArch(self, ninja_file, config_name, config, sources, - predepends, precompiled_header, spec, arch=None): - """Write build rules to compile all of |sources|.""" - - extra_defines = [] - if self.flavor == 'mac': - cflags = self.xcode_settings.GetCflags(config_name, arch=arch) - cflags_c = self.xcode_settings.GetCflagsC(config_name) - cflags_cc = self.xcode_settings.GetCflagsCC(config_name) - cflags_objc = ['$cflags_c'] + \ - self.xcode_settings.GetCflagsObjC(config_name) - cflags_objcc = ['$cflags_cc'] + \ - self.xcode_settings.GetCflagsObjCC(config_name) - elif self.flavor == 'win': - asmflags = self.msvs_settings.GetAsmflags(config_name) - cflags = self.msvs_settings.GetCflags(config_name) - cflags_c = self.msvs_settings.GetCflagsC(config_name) - cflags_cc = self.msvs_settings.GetCflagsCC(config_name) - extra_defines = self.msvs_settings.GetComputedDefines(config_name) - # See comment at cc_command for why there's two .pdb files. - pdbpath_c = pdbpath_cc = self.msvs_settings.GetCompilerPdbName( - config_name, self.ExpandSpecial) - if not pdbpath_c: - obj = 'obj' - if self.toolset != 'target': - obj += '.' + self.toolset - pdbpath = os.path.normpath(os.path.join(obj, self.base_dir, self.name)) - pdbpath_c = pdbpath + '.c.pdb' - pdbpath_cc = pdbpath + '.cc.pdb' - self.WriteVariableList(ninja_file, 'pdbname_c', [pdbpath_c]) - self.WriteVariableList(ninja_file, 'pdbname_cc', [pdbpath_cc]) - self.WriteVariableList(ninja_file, 'pchprefix', [self.name]) - else: - cflags = config.get('cflags', []) - cflags_c = config.get('cflags_c', []) - cflags_cc = config.get('cflags_cc', []) - - # Respect environment variables related to build, but target-specific - # flags can still override them. - if self.toolset == 'target': - cflags_c = (os.environ.get('CPPFLAGS', '').split() + - os.environ.get('CFLAGS', '').split() + cflags_c) - cflags_cc = (os.environ.get('CPPFLAGS', '').split() + - os.environ.get('CXXFLAGS', '').split() + cflags_cc) - elif self.toolset == 'host': - cflags_c = (os.environ.get('CPPFLAGS_host', '').split() + - os.environ.get('CFLAGS_host', '').split() + cflags_c) - cflags_cc = (os.environ.get('CPPFLAGS_host', '').split() + - os.environ.get('CXXFLAGS_host', '').split() + cflags_cc) - - defines = config.get('defines', []) + extra_defines - self.WriteVariableList(ninja_file, 'defines', - [Define(d, self.flavor) for d in defines]) - if self.flavor == 'win': - self.WriteVariableList(ninja_file, 'asmflags', - map(self.ExpandSpecial, asmflags)) - self.WriteVariableList(ninja_file, 'rcflags', - [QuoteShellArgument(self.ExpandSpecial(f), self.flavor) - for f in self.msvs_settings.GetRcflags(config_name, - self.GypPathToNinja)]) - - include_dirs = config.get('include_dirs', []) - - env = self.GetToolchainEnv() - if self.flavor == 'win': - include_dirs = self.msvs_settings.AdjustIncludeDirs(include_dirs, - config_name) - self.WriteVariableList(ninja_file, 'includes', - [QuoteShellArgument('-I' + self.GypPathToNinja(i, env), self.flavor) - for i in include_dirs]) - - if self.flavor == 'win': - midl_include_dirs = config.get('midl_include_dirs', []) - midl_include_dirs = self.msvs_settings.AdjustMidlIncludeDirs( - midl_include_dirs, config_name) - self.WriteVariableList(ninja_file, 'midl_includes', - [QuoteShellArgument('-I' + self.GypPathToNinja(i, env), self.flavor) - for i in midl_include_dirs]) - - pch_commands = precompiled_header.GetPchBuildCommands(arch) - if self.flavor == 'mac': - # Most targets use no precompiled headers, so only write these if needed. - for ext, var in [('c', 'cflags_pch_c'), ('cc', 'cflags_pch_cc'), - ('m', 'cflags_pch_objc'), ('mm', 'cflags_pch_objcc')]: - include = precompiled_header.GetInclude(ext, arch) - if include: ninja_file.variable(var, include) - - arflags = config.get('arflags', []) - - self.WriteVariableList(ninja_file, 'cflags', - map(self.ExpandSpecial, cflags)) - self.WriteVariableList(ninja_file, 'cflags_c', - map(self.ExpandSpecial, cflags_c)) - self.WriteVariableList(ninja_file, 'cflags_cc', - map(self.ExpandSpecial, cflags_cc)) - if self.flavor == 'mac': - self.WriteVariableList(ninja_file, 'cflags_objc', - map(self.ExpandSpecial, cflags_objc)) - self.WriteVariableList(ninja_file, 'cflags_objcc', - map(self.ExpandSpecial, cflags_objcc)) - self.WriteVariableList(ninja_file, 'arflags', - map(self.ExpandSpecial, arflags)) - ninja_file.newline() - outputs = [] - has_rc_source = False - for source in sources: - filename, ext = os.path.splitext(source) - ext = ext[1:] - obj_ext = self.obj_ext - if ext in ('cc', 'cpp', 'cxx'): - command = 'cxx' - self.target.uses_cpp = True - elif ext == 'c' or (ext == 'S' and self.flavor != 'win'): - command = 'cc' - elif ext == 's' and self.flavor != 'win': # Doesn't generate .o.d files. - command = 'cc_s' - elif (self.flavor == 'win' and ext == 'asm' and - not self.msvs_settings.HasExplicitAsmRules(spec)): - command = 'asm' - # Add the _asm suffix as msvs is capable of handling .cc and - # .asm files of the same name without collision. - obj_ext = '_asm.obj' - elif self.flavor == 'mac' and ext == 'm': - command = 'objc' - elif self.flavor == 'mac' and ext == 'mm': - command = 'objcxx' - self.target.uses_cpp = True - elif self.flavor == 'win' and ext == 'rc': - command = 'rc' - obj_ext = '.res' - has_rc_source = True - else: - # Ignore unhandled extensions. - continue - input = self.GypPathToNinja(source) - output = self.GypPathToUniqueOutput(filename + obj_ext) - if arch is not None: - output = AddArch(output, arch) - implicit = precompiled_header.GetObjDependencies([input], [output], arch) - variables = [] - if self.flavor == 'win': - variables, output, implicit = precompiled_header.GetFlagsModifications( - input, output, implicit, command, cflags_c, cflags_cc, - self.ExpandSpecial) - ninja_file.build(output, command, input, - implicit=[gch for _, _, gch in implicit], - order_only=predepends, variables=variables) - outputs.append(output) - - if has_rc_source: - resource_include_dirs = config.get('resource_include_dirs', include_dirs) - self.WriteVariableList(ninja_file, 'resource_includes', - [QuoteShellArgument('-I' + self.GypPathToNinja(i, env), self.flavor) - for i in resource_include_dirs]) - - self.WritePchTargets(ninja_file, pch_commands) - - ninja_file.newline() - return outputs - - def WritePchTargets(self, ninja_file, pch_commands): - """Writes ninja rules to compile prefix headers.""" - if not pch_commands: - return - - for gch, lang_flag, lang, input in pch_commands: - var_name = { - 'c': 'cflags_pch_c', - 'cc': 'cflags_pch_cc', - 'm': 'cflags_pch_objc', - 'mm': 'cflags_pch_objcc', - }[lang] - - map = { 'c': 'cc', 'cc': 'cxx', 'm': 'objc', 'mm': 'objcxx', } - cmd = map.get(lang) - ninja_file.build(gch, cmd, input, variables=[(var_name, lang_flag)]) - - def WriteLink(self, spec, config_name, config, link_deps, compile_deps): - """Write out a link step. Fills out target.binary. """ - if self.flavor != 'mac' or len(self.archs) == 1: - return self.WriteLinkForArch( - self.ninja, spec, config_name, config, link_deps, compile_deps) - else: - output = self.ComputeOutput(spec) - inputs = [self.WriteLinkForArch(self.arch_subninjas[arch], spec, - config_name, config, link_deps[arch], - compile_deps, arch=arch) - for arch in self.archs] - extra_bindings = [] - build_output = output - if not self.is_mac_bundle: - self.AppendPostbuildVariable(extra_bindings, spec, output, output) - - # TODO(yyanagisawa): more work needed to fix: - # https://code.google.com/p/gyp/issues/detail?id=411 - if (spec['type'] in ('shared_library', 'loadable_module') and - not self.is_mac_bundle): - extra_bindings.append(('lib', output)) - self.ninja.build([output, output + '.TOC'], 'solipo', inputs, - variables=extra_bindings) - else: - self.ninja.build(build_output, 'lipo', inputs, variables=extra_bindings) - return output - - def WriteLinkForArch(self, ninja_file, spec, config_name, config, - link_deps, compile_deps, arch=None): - """Write out a link step. Fills out target.binary. """ - command = { - 'executable': 'link', - 'loadable_module': 'solink_module', - 'shared_library': 'solink', - }[spec['type']] - command_suffix = '' - - implicit_deps = set() - solibs = set() - order_deps = set() - - if compile_deps: - # Normally, the compiles of the target already depend on compile_deps, - # but a shared_library target might have no sources and only link together - # a few static_library deps, so the link step also needs to depend - # on compile_deps to make sure actions in the shared_library target - # get run before the link. - order_deps.add(compile_deps) - - if 'dependencies' in spec: - # Two kinds of dependencies: - # - Linkable dependencies (like a .a or a .so): add them to the link line. - # - Non-linkable dependencies (like a rule that generates a file - # and writes a stamp file): add them to implicit_deps - extra_link_deps = set() - for dep in spec['dependencies']: - target = self.target_outputs.get(dep) - if not target: - continue - linkable = target.Linkable() - if linkable: - new_deps = [] - if (self.flavor == 'win' and - target.component_objs and - self.msvs_settings.IsUseLibraryDependencyInputs(config_name)): - new_deps = target.component_objs - if target.compile_deps: - order_deps.add(target.compile_deps) - elif self.flavor == 'win' and target.import_lib: - new_deps = [target.import_lib] - elif target.UsesToc(self.flavor): - solibs.add(target.binary) - implicit_deps.add(target.binary + '.TOC') - else: - new_deps = [target.binary] - for new_dep in new_deps: - if new_dep not in extra_link_deps: - extra_link_deps.add(new_dep) - link_deps.append(new_dep) - - final_output = target.FinalOutput() - if not linkable or final_output != target.binary: - implicit_deps.add(final_output) - - extra_bindings = [] - if self.target.uses_cpp and self.flavor != 'win': - extra_bindings.append(('ld', '$ldxx')) - - output = self.ComputeOutput(spec, arch) - if arch is None and not self.is_mac_bundle: - self.AppendPostbuildVariable(extra_bindings, spec, output, output) - - is_executable = spec['type'] == 'executable' - # The ldflags config key is not used on mac or win. On those platforms - # linker flags are set via xcode_settings and msvs_settings, respectively. - env_ldflags = os.environ.get('LDFLAGS', '').split() - if self.flavor == 'mac': - ldflags = self.xcode_settings.GetLdflags(config_name, - self.ExpandSpecial(generator_default_variables['PRODUCT_DIR']), - self.GypPathToNinja, arch) - ldflags = env_ldflags + ldflags - elif self.flavor == 'win': - manifest_base_name = self.GypPathToUniqueOutput( - self.ComputeOutputFileName(spec)) - ldflags, intermediate_manifest, manifest_files = \ - self.msvs_settings.GetLdflags(config_name, self.GypPathToNinja, - self.ExpandSpecial, manifest_base_name, - output, is_executable, - self.toplevel_build) - ldflags = env_ldflags + ldflags - self.WriteVariableList(ninja_file, 'manifests', manifest_files) - implicit_deps = implicit_deps.union(manifest_files) - if intermediate_manifest: + if not xcassets: + return + + extra_arguments = {} + settings_to_arg = { + "XCASSETS_APP_ICON": "app-icon", + "XCASSETS_LAUNCH_IMAGE": "launch-image", + } + settings = self.xcode_settings.xcode_settings[self.config_name] + for settings_key, arg_name in settings_to_arg.items(): + value = settings.get(settings_key) + if value: + extra_arguments[arg_name] = value + + partial_info_plist = None + if extra_arguments: + partial_info_plist = self.GypPathToUniqueOutput( + "assetcatalog_generated_info.plist" + ) + extra_arguments["output-partial-info-plist"] = partial_info_plist + + outputs = [] + outputs.append( + os.path.join(self.xcode_settings.GetBundleResourceFolder(), "Assets.car") + ) + if partial_info_plist: + outputs.append(partial_info_plist) + + keys = QuoteShellArgument(json.dumps(extra_arguments), self.flavor) + extra_env = self.xcode_settings.GetPerTargetSettings() + env = self.GetSortedXcodeEnv(additional_settings=extra_env) + env = self.ComputeExportEnvString(env) + + bundle_depends.extend( + self.ninja.build( + outputs, + "compile_xcassets", + xcassets, + variables=[("env", env), ("keys", keys)], + ) + ) + return partial_info_plist + + def WriteMacInfoPlist(self, partial_info_plist, bundle_depends): + """Write build rules for bundle Info.plist files.""" + info_plist, out, defines, extra_env = gyp.xcode_emulation.GetMacInfoPlist( + generator_default_variables["PRODUCT_DIR"], + self.xcode_settings, + self.GypPathToNinja, + ) + if not info_plist: + return + out = self.ExpandSpecial(out) + if defines: + # Create an intermediate file to store preprocessed results. + intermediate_plist = self.GypPathToUniqueOutput( + os.path.basename(info_plist) + ) + defines = " ".join([Define(d, self.flavor) for d in defines]) + info_plist = self.ninja.build( + intermediate_plist, + "preprocess_infoplist", + info_plist, + variables=[("defines", defines)], + ) + + env = self.GetSortedXcodeEnv(additional_settings=extra_env) + env = self.ComputeExportEnvString(env) + + if partial_info_plist: + intermediate_plist = self.GypPathToUniqueOutput("merged_info.plist") + info_plist = self.ninja.build( + intermediate_plist, "merge_infoplist", [partial_info_plist, info_plist] + ) + + keys = self.xcode_settings.GetExtraPlistItems(self.config_name) + keys = QuoteShellArgument(json.dumps(keys), self.flavor) + isBinary = self.xcode_settings.IsBinaryOutputFormat(self.config_name) + self.ninja.build( + out, + "copy_infoplist", + info_plist, + variables=[("env", env), ("keys", keys), ("binary", isBinary)], + ) + bundle_depends.append(out) + + def WriteSources( + self, + ninja_file, + config_name, + config, + sources, + predepends, + precompiled_header, + spec, + ): + """Write build rules to compile all of |sources|.""" + if self.toolset == "host": + self.ninja.variable("ar", "$ar_host") + self.ninja.variable("cc", "$cc_host") + self.ninja.variable("cxx", "$cxx_host") + self.ninja.variable("ld", "$ld_host") + self.ninja.variable("ldxx", "$ldxx_host") + self.ninja.variable("nm", "$nm_host") + self.ninja.variable("readelf", "$readelf_host") + + if self.flavor != "mac" or len(self.archs) == 1: + return self.WriteSourcesForArch( + self.ninja, + config_name, + config, + sources, + predepends, + precompiled_header, + spec, + ) + else: + return dict( + ( + arch, + self.WriteSourcesForArch( + self.arch_subninjas[arch], + config_name, + config, + sources, + predepends, + precompiled_header, + spec, + arch=arch, + ), + ) + for arch in self.archs + ) + + def WriteSourcesForArch( + self, + ninja_file, + config_name, + config, + sources, + predepends, + precompiled_header, + spec, + arch=None, + ): + """Write build rules to compile all of |sources|.""" + + extra_defines = [] + if self.flavor == "mac": + cflags = self.xcode_settings.GetCflags(config_name, arch=arch) + cflags_c = self.xcode_settings.GetCflagsC(config_name) + cflags_cc = self.xcode_settings.GetCflagsCC(config_name) + cflags_objc = ["$cflags_c"] + self.xcode_settings.GetCflagsObjC(config_name) + cflags_objcc = ["$cflags_cc"] + self.xcode_settings.GetCflagsObjCC( + config_name + ) + elif self.flavor == "win": + asmflags = self.msvs_settings.GetAsmflags(config_name) + cflags = self.msvs_settings.GetCflags(config_name) + cflags_c = self.msvs_settings.GetCflagsC(config_name) + cflags_cc = self.msvs_settings.GetCflagsCC(config_name) + extra_defines = self.msvs_settings.GetComputedDefines(config_name) + # See comment at cc_command for why there's two .pdb files. + pdbpath_c = pdbpath_cc = self.msvs_settings.GetCompilerPdbName( + config_name, self.ExpandSpecial + ) + if not pdbpath_c: + obj = "obj" + if self.toolset != "target": + obj += "." + self.toolset + pdbpath = os.path.normpath(os.path.join(obj, self.base_dir, self.name)) + pdbpath_c = pdbpath + ".c.pdb" + pdbpath_cc = pdbpath + ".cc.pdb" + self.WriteVariableList(ninja_file, "pdbname_c", [pdbpath_c]) + self.WriteVariableList(ninja_file, "pdbname_cc", [pdbpath_cc]) + self.WriteVariableList(ninja_file, "pchprefix", [self.name]) + else: + cflags = config.get("cflags", []) + cflags_c = config.get("cflags_c", []) + cflags_cc = config.get("cflags_cc", []) + + # Respect environment variables related to build, but target-specific + # flags can still override them. + if self.toolset == "target": + cflags_c = ( + os.environ.get("CPPFLAGS", "").split() + + os.environ.get("CFLAGS", "").split() + + cflags_c + ) + cflags_cc = ( + os.environ.get("CPPFLAGS", "").split() + + os.environ.get("CXXFLAGS", "").split() + + cflags_cc + ) + elif self.toolset == "host": + cflags_c = ( + os.environ.get("CPPFLAGS_host", "").split() + + os.environ.get("CFLAGS_host", "").split() + + cflags_c + ) + cflags_cc = ( + os.environ.get("CPPFLAGS_host", "").split() + + os.environ.get("CXXFLAGS_host", "").split() + + cflags_cc + ) + + defines = config.get("defines", []) + extra_defines self.WriteVariableList( - ninja_file, 'intermediatemanifest', [intermediate_manifest]) - command_suffix = _GetWinLinkRuleNameSuffix( - self.msvs_settings.IsEmbedManifest(config_name)) - def_file = self.msvs_settings.GetDefFile(self.GypPathToNinja) - if def_file: - implicit_deps.add(def_file) - else: - # Respect environment variables related to build, but target-specific - # flags can still override them. - ldflags = env_ldflags + config.get('ldflags', []) - if is_executable and len(solibs): - rpath = 'lib/' - if self.toolset != 'target': - rpath += self.toolset - ldflags.append(r'-Wl,-rpath=\$$ORIGIN/%s' % rpath) + ninja_file, "defines", [Define(d, self.flavor) for d in defines] + ) + if self.flavor == "win": + self.WriteVariableList( + ninja_file, "asmflags", map(self.ExpandSpecial, asmflags) + ) + self.WriteVariableList( + ninja_file, + "rcflags", + [ + QuoteShellArgument(self.ExpandSpecial(f), self.flavor) + for f in self.msvs_settings.GetRcflags( + config_name, self.GypPathToNinja + ) + ], + ) + + include_dirs = config.get("include_dirs", []) + + env = self.GetToolchainEnv() + if self.flavor == "win": + include_dirs = self.msvs_settings.AdjustIncludeDirs( + include_dirs, config_name + ) + self.WriteVariableList( + ninja_file, + "includes", + [ + QuoteShellArgument("-I" + self.GypPathToNinja(i, env), self.flavor) + for i in include_dirs + ], + ) + + if self.flavor == "win": + midl_include_dirs = config.get("midl_include_dirs", []) + midl_include_dirs = self.msvs_settings.AdjustMidlIncludeDirs( + midl_include_dirs, config_name + ) + self.WriteVariableList( + ninja_file, + "midl_includes", + [ + QuoteShellArgument("-I" + self.GypPathToNinja(i, env), self.flavor) + for i in midl_include_dirs + ], + ) + + pch_commands = precompiled_header.GetPchBuildCommands(arch) + if self.flavor == "mac": + # Most targets use no precompiled headers, so only write these if needed. + for ext, var in [ + ("c", "cflags_pch_c"), + ("cc", "cflags_pch_cc"), + ("m", "cflags_pch_objc"), + ("mm", "cflags_pch_objcc"), + ]: + include = precompiled_header.GetInclude(ext, arch) + if include: + ninja_file.variable(var, include) + + arflags = config.get("arflags", []) + + self.WriteVariableList(ninja_file, "cflags", map(self.ExpandSpecial, cflags)) + self.WriteVariableList( + ninja_file, "cflags_c", map(self.ExpandSpecial, cflags_c) + ) + self.WriteVariableList( + ninja_file, "cflags_cc", map(self.ExpandSpecial, cflags_cc) + ) + if self.flavor == "mac": + self.WriteVariableList( + ninja_file, "cflags_objc", map(self.ExpandSpecial, cflags_objc) + ) + self.WriteVariableList( + ninja_file, "cflags_objcc", map(self.ExpandSpecial, cflags_objcc) + ) + self.WriteVariableList(ninja_file, "arflags", map(self.ExpandSpecial, arflags)) + ninja_file.newline() + outputs = [] + has_rc_source = False + for source in sources: + filename, ext = os.path.splitext(source) + ext = ext[1:] + obj_ext = self.obj_ext + if ext in ("cc", "cpp", "cxx"): + command = "cxx" + self.target.uses_cpp = True + elif ext == "c" or (ext == "S" and self.flavor != "win"): + command = "cc" + elif ext == "s" and self.flavor != "win": # Doesn't generate .o.d files. + command = "cc_s" + elif ( + self.flavor == "win" + and ext == "asm" + and not self.msvs_settings.HasExplicitAsmRules(spec) + ): + command = "asm" + # Add the _asm suffix as msvs is capable of handling .cc and + # .asm files of the same name without collision. + obj_ext = "_asm.obj" + elif self.flavor == "mac" and ext == "m": + command = "objc" + elif self.flavor == "mac" and ext == "mm": + command = "objcxx" + self.target.uses_cpp = True + elif self.flavor == "win" and ext == "rc": + command = "rc" + obj_ext = ".res" + has_rc_source = True + else: + # Ignore unhandled extensions. + continue + input = self.GypPathToNinja(source) + output = self.GypPathToUniqueOutput(filename + obj_ext) + if arch is not None: + output = AddArch(output, arch) + implicit = precompiled_header.GetObjDependencies([input], [output], arch) + variables = [] + if self.flavor == "win": + variables, output, implicit = precompiled_header.GetFlagsModifications( + input, + output, + implicit, + command, + cflags_c, + cflags_cc, + self.ExpandSpecial, + ) + ninja_file.build( + output, + command, + input, + implicit=[gch for _, _, gch in implicit], + order_only=predepends, + variables=variables, + ) + outputs.append(output) + + if has_rc_source: + resource_include_dirs = config.get("resource_include_dirs", include_dirs) + self.WriteVariableList( + ninja_file, + "resource_includes", + [ + QuoteShellArgument("-I" + self.GypPathToNinja(i, env), self.flavor) + for i in resource_include_dirs + ], + ) + + self.WritePchTargets(ninja_file, pch_commands) + + ninja_file.newline() + return outputs + + def WritePchTargets(self, ninja_file, pch_commands): + """Writes ninja rules to compile prefix headers.""" + if not pch_commands: + return + + for gch, lang_flag, lang, input in pch_commands: + var_name = { + "c": "cflags_pch_c", + "cc": "cflags_pch_cc", + "m": "cflags_pch_objc", + "mm": "cflags_pch_objcc", + }[lang] + + map = { + "c": "cc", + "cc": "cxx", + "m": "objc", + "mm": "objcxx", + } + cmd = map.get(lang) + ninja_file.build(gch, cmd, input, variables=[(var_name, lang_flag)]) + + def WriteLink(self, spec, config_name, config, link_deps, compile_deps): + """Write out a link step. Fills out target.binary. """ + if self.flavor != "mac" or len(self.archs) == 1: + return self.WriteLinkForArch( + self.ninja, spec, config_name, config, link_deps, compile_deps + ) else: - ldflags.append('-Wl,-rpath=%s' % self.target_rpath) - ldflags.append('-Wl,-rpath-link=%s' % rpath) - self.WriteVariableList(ninja_file, 'ldflags', - map(self.ExpandSpecial, ldflags)) - - library_dirs = config.get('library_dirs', []) - if self.flavor == 'win': - library_dirs = [self.msvs_settings.ConvertVSMacros(l, config_name) - for l in library_dirs] - library_dirs = ['/LIBPATH:' + QuoteShellArgument(self.GypPathToNinja(l), - self.flavor) - for l in library_dirs] - else: - library_dirs = [QuoteShellArgument('-L' + self.GypPathToNinja(l), - self.flavor) - for l in library_dirs] - - libraries = gyp.common.uniquer(map(self.ExpandSpecial, - spec.get('libraries', []))) - if self.flavor == 'mac': - libraries = self.xcode_settings.AdjustLibraries(libraries, config_name) - elif self.flavor == 'win': - libraries = self.msvs_settings.AdjustLibraries(libraries) - - self.WriteVariableList(ninja_file, 'libs', library_dirs + libraries) - - linked_binary = output - - if command in ('solink', 'solink_module'): - extra_bindings.append(('soname', os.path.split(output)[1])) - extra_bindings.append(('lib', - gyp.common.EncodePOSIXShellArgument(output))) - if self.flavor != 'win': - link_file_list = output - if self.is_mac_bundle: - # 'Dependency Framework.framework/Versions/A/Dependency Framework' -> - # 'Dependency Framework.framework.rsp' - link_file_list = self.xcode_settings.GetWrapperName() - if arch: - link_file_list += '.' + arch - link_file_list += '.rsp' - # If an rspfile contains spaces, ninja surrounds the filename with - # quotes around it and then passes it to open(), creating a file with - # quotes in its name (and when looking for the rsp file, the name - # makes it through bash which strips the quotes) :-/ - link_file_list = link_file_list.replace(' ', '_') - extra_bindings.append( - ('link_file_list', - gyp.common.EncodePOSIXShellArgument(link_file_list))) - if self.flavor == 'win': - extra_bindings.append(('binary', output)) - if ('/NOENTRY' not in ldflags and - not self.msvs_settings.GetNoImportLibrary(config_name)): - self.target.import_lib = output + '.lib' - extra_bindings.append(('implibflag', - '/IMPLIB:%s' % self.target.import_lib)) - pdbname = self.msvs_settings.GetPDBName( - config_name, self.ExpandSpecial, output + '.pdb') - output = [output, self.target.import_lib] - if pdbname: - output.append(pdbname) - elif not self.is_mac_bundle: - output = [output, output + '.TOC'] - else: - command = command + '_notoc' - elif self.flavor == 'win': - extra_bindings.append(('binary', output)) - pdbname = self.msvs_settings.GetPDBName( - config_name, self.ExpandSpecial, output + '.pdb') - if pdbname: - output = [output, pdbname] - - - if len(solibs): - extra_bindings.append(('solibs', - gyp.common.EncodePOSIXShellList(sorted(solibs)))) - - ninja_file.build(output, command + command_suffix, link_deps, - implicit=sorted(implicit_deps), - order_only=list(order_deps), - variables=extra_bindings) - return linked_binary - - def WriteTarget(self, spec, config_name, config, link_deps, compile_deps): - extra_link_deps = any(self.target_outputs.get(dep).Linkable() - for dep in spec.get('dependencies', []) - if dep in self.target_outputs) - if spec['type'] == 'none' or (not link_deps and not extra_link_deps): - # TODO(evan): don't call this function for 'none' target types, as - # it doesn't do anything, and we fake out a 'binary' with a stamp file. - self.target.binary = compile_deps - self.target.type = 'none' - elif spec['type'] == 'static_library': - self.target.binary = self.ComputeOutput(spec) - if (self.flavor not in ('mac', 'openbsd', 'netbsd', 'win') and not - self.is_standalone_static_library): - self.ninja.build(self.target.binary, 'alink_thin', link_deps, - order_only=compile_deps) - else: + output = self.ComputeOutput(spec) + inputs = [ + self.WriteLinkForArch( + self.arch_subninjas[arch], + spec, + config_name, + config, + link_deps[arch], + compile_deps, + arch=arch, + ) + for arch in self.archs + ] + extra_bindings = [] + build_output = output + if not self.is_mac_bundle: + self.AppendPostbuildVariable(extra_bindings, spec, output, output) + + # TODO(yyanagisawa): more work needed to fix: + # https://code.google.com/p/gyp/issues/detail?id=411 + if ( + spec["type"] in ("shared_library", "loadable_module") + and not self.is_mac_bundle + ): + extra_bindings.append(("lib", output)) + self.ninja.build( + [output, output + ".TOC"], + "solipo", + inputs, + variables=extra_bindings, + ) + else: + self.ninja.build(build_output, "lipo", inputs, variables=extra_bindings) + return output + + def WriteLinkForArch( + self, ninja_file, spec, config_name, config, link_deps, compile_deps, arch=None + ): + """Write out a link step. Fills out target.binary. """ + command = { + "executable": "link", + "loadable_module": "solink_module", + "shared_library": "solink", + }[spec["type"]] + command_suffix = "" + + implicit_deps = set() + solibs = set() + order_deps = set() + + if compile_deps: + # Normally, the compiles of the target already depend on compile_deps, + # but a shared_library target might have no sources and only link together + # a few static_library deps, so the link step also needs to depend + # on compile_deps to make sure actions in the shared_library target + # get run before the link. + order_deps.add(compile_deps) + + if "dependencies" in spec: + # Two kinds of dependencies: + # - Linkable dependencies (like a .a or a .so): add them to the link line. + # - Non-linkable dependencies (like a rule that generates a file + # and writes a stamp file): add them to implicit_deps + extra_link_deps = set() + for dep in spec["dependencies"]: + target = self.target_outputs.get(dep) + if not target: + continue + linkable = target.Linkable() + if linkable: + new_deps = [] + if ( + self.flavor == "win" + and target.component_objs + and self.msvs_settings.IsUseLibraryDependencyInputs(config_name) + ): + new_deps = target.component_objs + if target.compile_deps: + order_deps.add(target.compile_deps) + elif self.flavor == "win" and target.import_lib: + new_deps = [target.import_lib] + elif target.UsesToc(self.flavor): + solibs.add(target.binary) + implicit_deps.add(target.binary + ".TOC") + else: + new_deps = [target.binary] + for new_dep in new_deps: + if new_dep not in extra_link_deps: + extra_link_deps.add(new_dep) + link_deps.append(new_dep) + + final_output = target.FinalOutput() + if not linkable or final_output != target.binary: + implicit_deps.add(final_output) + + extra_bindings = [] + if self.target.uses_cpp and self.flavor != "win": + extra_bindings.append(("ld", "$ldxx")) + + output = self.ComputeOutput(spec, arch) + if arch is None and not self.is_mac_bundle: + self.AppendPostbuildVariable(extra_bindings, spec, output, output) + + is_executable = spec["type"] == "executable" + # The ldflags config key is not used on mac or win. On those platforms + # linker flags are set via xcode_settings and msvs_settings, respectively. + env_ldflags = os.environ.get("LDFLAGS", "").split() + if self.flavor == "mac": + ldflags = self.xcode_settings.GetLdflags( + config_name, + self.ExpandSpecial(generator_default_variables["PRODUCT_DIR"]), + self.GypPathToNinja, + arch, + ) + ldflags = env_ldflags + ldflags + elif self.flavor == "win": + manifest_base_name = self.GypPathToUniqueOutput( + self.ComputeOutputFileName(spec) + ) + ( + ldflags, + intermediate_manifest, + manifest_files, + ) = self.msvs_settings.GetLdflags( + config_name, + self.GypPathToNinja, + self.ExpandSpecial, + manifest_base_name, + output, + is_executable, + self.toplevel_build, + ) + ldflags = env_ldflags + ldflags + self.WriteVariableList(ninja_file, "manifests", manifest_files) + implicit_deps = implicit_deps.union(manifest_files) + if intermediate_manifest: + self.WriteVariableList( + ninja_file, "intermediatemanifest", [intermediate_manifest] + ) + command_suffix = _GetWinLinkRuleNameSuffix( + self.msvs_settings.IsEmbedManifest(config_name) + ) + def_file = self.msvs_settings.GetDefFile(self.GypPathToNinja) + if def_file: + implicit_deps.add(def_file) + else: + # Respect environment variables related to build, but target-specific + # flags can still override them. + ldflags = env_ldflags + config.get("ldflags", []) + if is_executable and len(solibs): + rpath = "lib/" + if self.toolset != "target": + rpath += self.toolset + ldflags.append(r"-Wl,-rpath=\$$ORIGIN/%s" % rpath) + else: + ldflags.append("-Wl,-rpath=%s" % self.target_rpath) + ldflags.append("-Wl,-rpath-link=%s" % rpath) + self.WriteVariableList(ninja_file, "ldflags", map(self.ExpandSpecial, ldflags)) + + library_dirs = config.get("library_dirs", []) + if self.flavor == "win": + library_dirs = [ + self.msvs_settings.ConvertVSMacros(l, config_name) for l in library_dirs + ] + library_dirs = [ + "/LIBPATH:" + QuoteShellArgument(self.GypPathToNinja(l), self.flavor) + for l in library_dirs + ] + else: + library_dirs = [ + QuoteShellArgument("-L" + self.GypPathToNinja(l), self.flavor) + for l in library_dirs + ] + + libraries = gyp.common.uniquer( + map(self.ExpandSpecial, spec.get("libraries", [])) + ) + if self.flavor == "mac": + libraries = self.xcode_settings.AdjustLibraries(libraries, config_name) + elif self.flavor == "win": + libraries = self.msvs_settings.AdjustLibraries(libraries) + + self.WriteVariableList(ninja_file, "libs", library_dirs + libraries) + + linked_binary = output + + if command in ("solink", "solink_module"): + extra_bindings.append(("soname", os.path.split(output)[1])) + extra_bindings.append(("lib", gyp.common.EncodePOSIXShellArgument(output))) + if self.flavor != "win": + link_file_list = output + if self.is_mac_bundle: + # 'Dependency Framework.framework/Versions/A/Dependency Framework' -> + # 'Dependency Framework.framework.rsp' + link_file_list = self.xcode_settings.GetWrapperName() + if arch: + link_file_list += "." + arch + link_file_list += ".rsp" + # If an rspfile contains spaces, ninja surrounds the filename with + # quotes around it and then passes it to open(), creating a file with + # quotes in its name (and when looking for the rsp file, the name + # makes it through bash which strips the quotes) :-/ + link_file_list = link_file_list.replace(" ", "_") + extra_bindings.append( + ( + "link_file_list", + gyp.common.EncodePOSIXShellArgument(link_file_list), + ) + ) + if self.flavor == "win": + extra_bindings.append(("binary", output)) + if ( + "/NOENTRY" not in ldflags + and not self.msvs_settings.GetNoImportLibrary(config_name) + ): + self.target.import_lib = output + ".lib" + extra_bindings.append( + ("implibflag", "/IMPLIB:%s" % self.target.import_lib) + ) + pdbname = self.msvs_settings.GetPDBName( + config_name, self.ExpandSpecial, output + ".pdb" + ) + output = [output, self.target.import_lib] + if pdbname: + output.append(pdbname) + elif not self.is_mac_bundle: + output = [output, output + ".TOC"] + else: + command = command + "_notoc" + elif self.flavor == "win": + extra_bindings.append(("binary", output)) + pdbname = self.msvs_settings.GetPDBName( + config_name, self.ExpandSpecial, output + ".pdb" + ) + if pdbname: + output = [output, pdbname] + + if len(solibs): + extra_bindings.append( + ("solibs", gyp.common.EncodePOSIXShellList(sorted(solibs))) + ) + + ninja_file.build( + output, + command + command_suffix, + link_deps, + implicit=sorted(implicit_deps), + order_only=list(order_deps), + variables=extra_bindings, + ) + return linked_binary + + def WriteTarget(self, spec, config_name, config, link_deps, compile_deps): + extra_link_deps = any( + self.target_outputs.get(dep).Linkable() + for dep in spec.get("dependencies", []) + if dep in self.target_outputs + ) + if spec["type"] == "none" or (not link_deps and not extra_link_deps): + # TODO(evan): don't call this function for 'none' target types, as + # it doesn't do anything, and we fake out a 'binary' with a stamp file. + self.target.binary = compile_deps + self.target.type = "none" + elif spec["type"] == "static_library": + self.target.binary = self.ComputeOutput(spec) + if ( + self.flavor not in ("mac", "openbsd", "netbsd", "win") + and not self.is_standalone_static_library + ): + self.ninja.build( + self.target.binary, "alink_thin", link_deps, order_only=compile_deps + ) + else: + variables = [] + if self.xcode_settings: + libtool_flags = self.xcode_settings.GetLibtoolflags(config_name) + if libtool_flags: + variables.append(("libtool_flags", libtool_flags)) + if self.msvs_settings: + libflags = self.msvs_settings.GetLibFlags( + config_name, self.GypPathToNinja + ) + variables.append(("libflags", libflags)) + + if self.flavor != "mac" or len(self.archs) == 1: + self.AppendPostbuildVariable( + variables, spec, self.target.binary, self.target.binary + ) + self.ninja.build( + self.target.binary, + "alink", + link_deps, + order_only=compile_deps, + variables=variables, + ) + else: + inputs = [] + for arch in self.archs: + output = self.ComputeOutput(spec, arch) + self.arch_subninjas[arch].build( + output, + "alink", + link_deps[arch], + order_only=compile_deps, + variables=variables, + ) + inputs.append(output) + # TODO: It's not clear if libtool_flags should be passed to the alink + # call that combines single-arch .a files into a fat .a file. + self.AppendPostbuildVariable( + variables, spec, self.target.binary, self.target.binary + ) + self.ninja.build( + self.target.binary, + "alink", + inputs, + # FIXME: test proving order_only=compile_deps isn't + # needed. + variables=variables, + ) + else: + self.target.binary = self.WriteLink( + spec, config_name, config, link_deps, compile_deps + ) + return self.target.binary + + def WriteMacBundle(self, spec, mac_bundle_depends, is_empty): + assert self.is_mac_bundle + package_framework = spec["type"] in ("shared_library", "loadable_module") + output = self.ComputeMacBundleOutput() + if is_empty: + output += ".stamp" variables = [] - if self.xcode_settings: - libtool_flags = self.xcode_settings.GetLibtoolflags(config_name) - if libtool_flags: - variables.append(('libtool_flags', libtool_flags)) - if self.msvs_settings: - libflags = self.msvs_settings.GetLibFlags(config_name, - self.GypPathToNinja) - variables.append(('libflags', libflags)) - - if self.flavor != 'mac' or len(self.archs) == 1: - self.AppendPostbuildVariable(variables, spec, - self.target.binary, self.target.binary) - self.ninja.build(self.target.binary, 'alink', link_deps, - order_only=compile_deps, variables=variables) + self.AppendPostbuildVariable( + variables, + spec, + output, + self.target.binary, + is_command_start=not package_framework, + ) + if package_framework and not is_empty: + if spec["type"] == "shared_library" and self.xcode_settings.isIOS: + self.ninja.build( + output, + "package_ios_framework", + mac_bundle_depends, + variables=variables, + ) + else: + variables.append(("version", self.xcode_settings.GetFrameworkVersion())) + self.ninja.build( + output, "package_framework", mac_bundle_depends, variables=variables + ) else: - inputs = [] - for arch in self.archs: - output = self.ComputeOutput(spec, arch) - self.arch_subninjas[arch].build(output, 'alink', link_deps[arch], - order_only=compile_deps, - variables=variables) - inputs.append(output) - # TODO: It's not clear if libtool_flags should be passed to the alink - # call that combines single-arch .a files into a fat .a file. - self.AppendPostbuildVariable(variables, spec, - self.target.binary, self.target.binary) - self.ninja.build(self.target.binary, 'alink', inputs, - # FIXME: test proving order_only=compile_deps isn't - # needed. - variables=variables) - else: - self.target.binary = self.WriteLink(spec, config_name, config, link_deps, - compile_deps) - return self.target.binary - - def WriteMacBundle(self, spec, mac_bundle_depends, is_empty): - assert self.is_mac_bundle - package_framework = spec['type'] in ('shared_library', 'loadable_module') - output = self.ComputeMacBundleOutput() - if is_empty: - output += '.stamp' - variables = [] - self.AppendPostbuildVariable(variables, spec, output, self.target.binary, - is_command_start=not package_framework) - if package_framework and not is_empty: - if spec['type'] == 'shared_library' and self.xcode_settings.isIOS: - self.ninja.build(output, 'package_ios_framework', mac_bundle_depends, - variables=variables) - else: - variables.append(('version', self.xcode_settings.GetFrameworkVersion())) - self.ninja.build(output, 'package_framework', mac_bundle_depends, - variables=variables) - else: - self.ninja.build(output, 'stamp', mac_bundle_depends, - variables=variables) - self.target.bundle = output - return output - - def GetToolchainEnv(self, additional_settings=None): - """Returns the variables toolchain would set for build steps.""" - env = self.GetSortedXcodeEnv(additional_settings=additional_settings) - if self.flavor == 'win': - env = self.GetMsvsToolchainEnv( - additional_settings=additional_settings) - return env - - def GetMsvsToolchainEnv(self, additional_settings=None): - """Returns the variables Visual Studio would set for build steps.""" - return self.msvs_settings.GetVSMacroEnv('$!PRODUCT_DIR', - config=self.config_name) - - def GetSortedXcodeEnv(self, additional_settings=None): - """Returns the variables Xcode would set for build steps.""" - assert self.abs_build_dir - abs_build_dir = self.abs_build_dir - return gyp.xcode_emulation.GetSortedXcodeEnv( - self.xcode_settings, abs_build_dir, - os.path.join(abs_build_dir, self.build_to_base), self.config_name, - additional_settings) - - def GetSortedXcodePostbuildEnv(self): - """Returns the variables Xcode would set for postbuild steps.""" - postbuild_settings = {} - # CHROMIUM_STRIP_SAVE_FILE is a chromium-specific hack. - # TODO(thakis): It would be nice to have some general mechanism instead. - strip_save_file = self.xcode_settings.GetPerTargetSetting( - 'CHROMIUM_STRIP_SAVE_FILE') - if strip_save_file: - postbuild_settings['CHROMIUM_STRIP_SAVE_FILE'] = strip_save_file - return self.GetSortedXcodeEnv(additional_settings=postbuild_settings) - - def AppendPostbuildVariable(self, variables, spec, output, binary, - is_command_start=False): - """Adds a 'postbuild' variable if there is a postbuild for |output|.""" - postbuild = self.GetPostbuildCommand(spec, output, binary, is_command_start) - if postbuild: - variables.append(('postbuilds', postbuild)) - - def GetPostbuildCommand(self, spec, output, output_binary, is_command_start): - """Returns a shell command that runs all the postbuilds, and removes + self.ninja.build(output, "stamp", mac_bundle_depends, variables=variables) + self.target.bundle = output + return output + + def GetToolchainEnv(self, additional_settings=None): + """Returns the variables toolchain would set for build steps.""" + env = self.GetSortedXcodeEnv(additional_settings=additional_settings) + if self.flavor == "win": + env = self.GetMsvsToolchainEnv(additional_settings=additional_settings) + return env + + def GetMsvsToolchainEnv(self, additional_settings=None): + """Returns the variables Visual Studio would set for build steps.""" + return self.msvs_settings.GetVSMacroEnv( + "$!PRODUCT_DIR", config=self.config_name + ) + + def GetSortedXcodeEnv(self, additional_settings=None): + """Returns the variables Xcode would set for build steps.""" + assert self.abs_build_dir + abs_build_dir = self.abs_build_dir + return gyp.xcode_emulation.GetSortedXcodeEnv( + self.xcode_settings, + abs_build_dir, + os.path.join(abs_build_dir, self.build_to_base), + self.config_name, + additional_settings, + ) + + def GetSortedXcodePostbuildEnv(self): + """Returns the variables Xcode would set for postbuild steps.""" + postbuild_settings = {} + # CHROMIUM_STRIP_SAVE_FILE is a chromium-specific hack. + # TODO(thakis): It would be nice to have some general mechanism instead. + strip_save_file = self.xcode_settings.GetPerTargetSetting( + "CHROMIUM_STRIP_SAVE_FILE" + ) + if strip_save_file: + postbuild_settings["CHROMIUM_STRIP_SAVE_FILE"] = strip_save_file + return self.GetSortedXcodeEnv(additional_settings=postbuild_settings) + + def AppendPostbuildVariable( + self, variables, spec, output, binary, is_command_start=False + ): + """Adds a 'postbuild' variable if there is a postbuild for |output|.""" + postbuild = self.GetPostbuildCommand(spec, output, binary, is_command_start) + if postbuild: + variables.append(("postbuilds", postbuild)) + + def GetPostbuildCommand(self, spec, output, output_binary, is_command_start): + """Returns a shell command that runs all the postbuilds, and removes |output| if any of them fails. If |is_command_start| is False, then the returned string will start with ' && '.""" - if not self.xcode_settings or spec['type'] == 'none' or not output: - return '' - output = QuoteShellArgument(output, self.flavor) - postbuilds = gyp.xcode_emulation.GetSpecPostbuildCommands(spec, quiet=True) - if output_binary is not None: - postbuilds = self.xcode_settings.AddImplicitPostbuilds( - self.config_name, - os.path.normpath(os.path.join(self.base_to_build, output)), - QuoteShellArgument( - os.path.normpath(os.path.join(self.base_to_build, output_binary)), - self.flavor), - postbuilds, quiet=True) - - if not postbuilds: - return '' - # Postbuilds expect to be run in the gyp file's directory, so insert an - # implicit postbuild to cd to there. - postbuilds.insert(0, gyp.common.EncodePOSIXShellList( - ['cd', self.build_to_base])) - env = self.ComputeExportEnvString(self.GetSortedXcodePostbuildEnv()) - # G will be non-null if any postbuild fails. Run all postbuilds in a - # subshell. - commands = env + ' (' + \ - ' && '.join([ninja_syntax.escape(command) for command in postbuilds]) - command_string = (commands + '); G=$$?; ' - # Remove the final output if any postbuild failed. - '((exit $$G) || rm -rf %s) ' % output + '&& exit $$G)') - if is_command_start: - return '(' + command_string + ' && ' - else: - return '$ && (' + command_string + if not self.xcode_settings or spec["type"] == "none" or not output: + return "" + output = QuoteShellArgument(output, self.flavor) + postbuilds = gyp.xcode_emulation.GetSpecPostbuildCommands(spec, quiet=True) + if output_binary is not None: + postbuilds = self.xcode_settings.AddImplicitPostbuilds( + self.config_name, + os.path.normpath(os.path.join(self.base_to_build, output)), + QuoteShellArgument( + os.path.normpath(os.path.join(self.base_to_build, output_binary)), + self.flavor, + ), + postbuilds, + quiet=True, + ) + + if not postbuilds: + return "" + # Postbuilds expect to be run in the gyp file's directory, so insert an + # implicit postbuild to cd to there. + postbuilds.insert( + 0, gyp.common.EncodePOSIXShellList(["cd", self.build_to_base]) + ) + env = self.ComputeExportEnvString(self.GetSortedXcodePostbuildEnv()) + # G will be non-null if any postbuild fails. Run all postbuilds in a + # subshell. + commands = ( + env + + " (" + + " && ".join([ninja_syntax.escape(command) for command in postbuilds]) + ) + command_string = ( + commands + + "); G=$$?; " + # Remove the final output if any postbuild failed. + "((exit $$G) || rm -rf %s) " % output + + "&& exit $$G)" + ) + if is_command_start: + return "(" + command_string + " && " + else: + return "$ && (" + command_string - def ComputeExportEnvString(self, env): - """Given an environment, returns a string looking like + def ComputeExportEnvString(self, env): + """Given an environment, returns a string looking like 'export FOO=foo; export BAR="${FOO} bar;' that exports |env| to the shell.""" - export_str = [] - for k, v in env: - export_str.append('export %s=%s;' % - (k, ninja_syntax.escape(gyp.common.EncodePOSIXShellArgument(v)))) - return ' '.join(export_str) - - def ComputeMacBundleOutput(self): - """Return the 'output' (full output path) to a bundle output directory.""" - assert self.is_mac_bundle - path = generator_default_variables['PRODUCT_DIR'] - return self.ExpandSpecial( - os.path.join(path, self.xcode_settings.GetWrapperName())) - - def ComputeOutputFileName(self, spec, type=None): - """Compute the filename of the final output for the current target.""" - if not type: - type = spec['type'] - - default_variables = copy.copy(generator_default_variables) - CalculateVariables(default_variables, {'flavor': self.flavor}) - - # Compute filename prefix: the product prefix, or a default for - # the product type. - DEFAULT_PREFIX = { - 'loadable_module': default_variables['SHARED_LIB_PREFIX'], - 'shared_library': default_variables['SHARED_LIB_PREFIX'], - 'static_library': default_variables['STATIC_LIB_PREFIX'], - 'executable': default_variables['EXECUTABLE_PREFIX'], - } - prefix = spec.get('product_prefix', DEFAULT_PREFIX.get(type, '')) - - # Compute filename extension: the product extension, or a default - # for the product type. - DEFAULT_EXTENSION = { - 'loadable_module': default_variables['SHARED_LIB_SUFFIX'], - 'shared_library': default_variables['SHARED_LIB_SUFFIX'], - 'static_library': default_variables['STATIC_LIB_SUFFIX'], - 'executable': default_variables['EXECUTABLE_SUFFIX'], - } - extension = spec.get('product_extension') - if extension: - extension = '.' + extension - else: - extension = DEFAULT_EXTENSION.get(type, '') - - if 'product_name' in spec: - # If we were given an explicit name, use that. - target = spec['product_name'] - else: - # Otherwise, derive a name from the target name. - target = spec['target_name'] - if prefix == 'lib': - # Snip out an extra 'lib' from libs if appropriate. - target = StripPrefix(target, 'lib') - - if type in ('static_library', 'loadable_module', 'shared_library', - 'executable'): - return '%s%s%s' % (prefix, target, extension) - elif type == 'none': - return '%s.stamp' % target - else: - raise Exception('Unhandled output type %s' % type) - - def ComputeOutput(self, spec, arch=None): - """Compute the path for the final output of the spec.""" - type = spec['type'] - - if self.flavor == 'win': - override = self.msvs_settings.GetOutputName(self.config_name, - self.ExpandSpecial) - if override: - return override + export_str = [] + for k, v in env: + export_str.append( + "export %s=%s;" + % (k, ninja_syntax.escape(gyp.common.EncodePOSIXShellArgument(v))) + ) + return " ".join(export_str) + + def ComputeMacBundleOutput(self): + """Return the 'output' (full output path) to a bundle output directory.""" + assert self.is_mac_bundle + path = generator_default_variables["PRODUCT_DIR"] + return self.ExpandSpecial( + os.path.join(path, self.xcode_settings.GetWrapperName()) + ) + + def ComputeOutputFileName(self, spec, type=None): + """Compute the filename of the final output for the current target.""" + if not type: + type = spec["type"] + + default_variables = copy.copy(generator_default_variables) + CalculateVariables(default_variables, {"flavor": self.flavor}) + + # Compute filename prefix: the product prefix, or a default for + # the product type. + DEFAULT_PREFIX = { + "loadable_module": default_variables["SHARED_LIB_PREFIX"], + "shared_library": default_variables["SHARED_LIB_PREFIX"], + "static_library": default_variables["STATIC_LIB_PREFIX"], + "executable": default_variables["EXECUTABLE_PREFIX"], + } + prefix = spec.get("product_prefix", DEFAULT_PREFIX.get(type, "")) + + # Compute filename extension: the product extension, or a default + # for the product type. + DEFAULT_EXTENSION = { + "loadable_module": default_variables["SHARED_LIB_SUFFIX"], + "shared_library": default_variables["SHARED_LIB_SUFFIX"], + "static_library": default_variables["STATIC_LIB_SUFFIX"], + "executable": default_variables["EXECUTABLE_SUFFIX"], + } + extension = spec.get("product_extension") + if extension: + extension = "." + extension + else: + extension = DEFAULT_EXTENSION.get(type, "") - if arch is None and self.flavor == 'mac' and type in ( - 'static_library', 'executable', 'shared_library', 'loadable_module'): - filename = self.xcode_settings.GetExecutablePath() - else: - filename = self.ComputeOutputFileName(spec, type) - - if arch is None and 'product_dir' in spec: - path = os.path.join(spec['product_dir'], filename) - return self.ExpandSpecial(path) - - # Some products go into the output root, libraries go into shared library - # dir, and everything else goes into the normal place. - type_in_output_root = ['executable', 'loadable_module'] - if self.flavor == 'mac' and self.toolset == 'target': - type_in_output_root += ['shared_library', 'static_library'] - elif self.flavor == 'win' and self.toolset == 'target': - type_in_output_root += ['shared_library'] - - if arch is not None: - # Make sure partial executables don't end up in a bundle or the regular - # output directory. - archdir = 'arch' - if self.toolset != 'target': - archdir = os.path.join('arch', '%s' % self.toolset) - return os.path.join(archdir, AddArch(filename, arch)) - elif type in type_in_output_root or self.is_standalone_static_library: - return filename - elif type == 'shared_library': - libdir = 'lib' - if self.toolset != 'target': - libdir = os.path.join('lib', '%s' % self.toolset) - return os.path.join(libdir, filename) - else: - return self.GypPathToUniqueOutput(filename, qualified=False) + if "product_name" in spec: + # If we were given an explicit name, use that. + target = spec["product_name"] + else: + # Otherwise, derive a name from the target name. + target = spec["target_name"] + if prefix == "lib": + # Snip out an extra 'lib' from libs if appropriate. + target = StripPrefix(target, "lib") + + if type in ( + "static_library", + "loadable_module", + "shared_library", + "executable", + ): + return "%s%s%s" % (prefix, target, extension) + elif type == "none": + return "%s.stamp" % target + else: + raise Exception("Unhandled output type %s" % type) + + def ComputeOutput(self, spec, arch=None): + """Compute the path for the final output of the spec.""" + type = spec["type"] + + if self.flavor == "win": + override = self.msvs_settings.GetOutputName( + self.config_name, self.ExpandSpecial + ) + if override: + return override + + if ( + arch is None + and self.flavor == "mac" + and type + in ("static_library", "executable", "shared_library", "loadable_module") + ): + filename = self.xcode_settings.GetExecutablePath() + else: + filename = self.ComputeOutputFileName(spec, type) + + if arch is None and "product_dir" in spec: + path = os.path.join(spec["product_dir"], filename) + return self.ExpandSpecial(path) + + # Some products go into the output root, libraries go into shared library + # dir, and everything else goes into the normal place. + type_in_output_root = ["executable", "loadable_module"] + if self.flavor == "mac" and self.toolset == "target": + type_in_output_root += ["shared_library", "static_library"] + elif self.flavor == "win" and self.toolset == "target": + type_in_output_root += ["shared_library"] + + if arch is not None: + # Make sure partial executables don't end up in a bundle or the regular + # output directory. + archdir = "arch" + if self.toolset != "target": + archdir = os.path.join("arch", "%s" % self.toolset) + return os.path.join(archdir, AddArch(filename, arch)) + elif type in type_in_output_root or self.is_standalone_static_library: + return filename + elif type == "shared_library": + libdir = "lib" + if self.toolset != "target": + libdir = os.path.join("lib", "%s" % self.toolset) + return os.path.join(libdir, filename) + else: + return self.GypPathToUniqueOutput(filename, qualified=False) - def WriteVariableList(self, ninja_file, var, values): - assert not isinstance(values, str) - if values is None: - values = [] - ninja_file.variable(var, ' '.join(values)) + def WriteVariableList(self, ninja_file, var, values): + assert not isinstance(values, str) + if values is None: + values = [] + ninja_file.variable(var, " ".join(values)) - def WriteNewNinjaRule(self, name, args, description, is_cygwin, env, pool, - depfile=None): - """Write out a new ninja "rule" statement for a given command. + def WriteNewNinjaRule( + self, name, args, description, is_cygwin, env, pool, depfile=None + ): + """Write out a new ninja "rule" statement for a given command. Returns the name of the new rule, and a copy of |args| with variables expanded.""" - if self.flavor == 'win': - args = [self.msvs_settings.ConvertVSMacros( - arg, self.base_to_build, config=self.config_name) - for arg in args] - description = self.msvs_settings.ConvertVSMacros( - description, config=self.config_name) - elif self.flavor == 'mac': - # |env| is an empty list on non-mac. - args = [gyp.xcode_emulation.ExpandEnvVars(arg, env) for arg in args] - description = gyp.xcode_emulation.ExpandEnvVars(description, env) - - # TODO: we shouldn't need to qualify names; we do it because - # currently the ninja rule namespace is global, but it really - # should be scoped to the subninja. - rule_name = self.name - if self.toolset == 'target': - rule_name += '.' + self.toolset - rule_name += '.' + name - rule_name = re.sub('[^a-zA-Z0-9_]', '_', rule_name) - - # Remove variable references, but not if they refer to the magic rule - # variables. This is not quite right, as it also protects these for - # actions, not just for rules where they are valid. Good enough. - protect = [ '${root}', '${dirname}', '${source}', '${ext}', '${name}' ] - protect = '(?!' + '|'.join(map(re.escape, protect)) + ')' - description = re.sub(protect + r'\$', '_', description) - - # gyp dictates that commands are run from the base directory. - # cd into the directory before running, and adjust paths in - # the arguments to point to the proper locations. - rspfile = None - rspfile_content = None - args = [self.ExpandSpecial(arg, self.base_to_build) for arg in args] - if self.flavor == 'win': - rspfile = rule_name + '.$unique_name.rsp' - # The cygwin case handles this inside the bash sub-shell. - run_in = '' if is_cygwin else ' ' + self.build_to_base - if is_cygwin: - rspfile_content = self.msvs_settings.BuildCygwinBashCommandLine( - args, self.build_to_base) - else: - rspfile_content = gyp.msvs_emulation.EncodeRspFileList(args) - command = ('%s gyp-win-tool action-wrapper $arch ' % sys.executable + - rspfile + run_in) - else: - env = self.ComputeExportEnvString(env) - command = gyp.common.EncodePOSIXShellList(args) - command = 'cd %s; ' % self.build_to_base + env + command - - # GYP rules/actions express being no-ops by not touching their outputs. - # Avoid executing downstream dependencies in this case by specifying - # restat=1 to ninja. - self.ninja.rule(rule_name, command, description, depfile=depfile, - restat=True, pool=pool, - rspfile=rspfile, rspfile_content=rspfile_content) - self.ninja.newline() - - return rule_name, args + if self.flavor == "win": + args = [ + self.msvs_settings.ConvertVSMacros( + arg, self.base_to_build, config=self.config_name + ) + for arg in args + ] + description = self.msvs_settings.ConvertVSMacros( + description, config=self.config_name + ) + elif self.flavor == "mac": + # |env| is an empty list on non-mac. + args = [gyp.xcode_emulation.ExpandEnvVars(arg, env) for arg in args] + description = gyp.xcode_emulation.ExpandEnvVars(description, env) + + # TODO: we shouldn't need to qualify names; we do it because + # currently the ninja rule namespace is global, but it really + # should be scoped to the subninja. + rule_name = self.name + if self.toolset == "target": + rule_name += "." + self.toolset + rule_name += "." + name + rule_name = re.sub("[^a-zA-Z0-9_]", "_", rule_name) + + # Remove variable references, but not if they refer to the magic rule + # variables. This is not quite right, as it also protects these for + # actions, not just for rules where they are valid. Good enough. + protect = ["${root}", "${dirname}", "${source}", "${ext}", "${name}"] + protect = "(?!" + "|".join(map(re.escape, protect)) + ")" + description = re.sub(protect + r"\$", "_", description) + + # gyp dictates that commands are run from the base directory. + # cd into the directory before running, and adjust paths in + # the arguments to point to the proper locations. + rspfile = None + rspfile_content = None + args = [self.ExpandSpecial(arg, self.base_to_build) for arg in args] + if self.flavor == "win": + rspfile = rule_name + ".$unique_name.rsp" + # The cygwin case handles this inside the bash sub-shell. + run_in = "" if is_cygwin else " " + self.build_to_base + if is_cygwin: + rspfile_content = self.msvs_settings.BuildCygwinBashCommandLine( + args, self.build_to_base + ) + else: + rspfile_content = gyp.msvs_emulation.EncodeRspFileList(args) + command = ( + "%s gyp-win-tool action-wrapper $arch " % sys.executable + + rspfile + + run_in + ) + else: + env = self.ComputeExportEnvString(env) + command = gyp.common.EncodePOSIXShellList(args) + command = "cd %s; " % self.build_to_base + env + command + + # GYP rules/actions express being no-ops by not touching their outputs. + # Avoid executing downstream dependencies in this case by specifying + # restat=1 to ninja. + self.ninja.rule( + rule_name, + command, + description, + depfile=depfile, + restat=True, + pool=pool, + rspfile=rspfile, + rspfile_content=rspfile_content, + ) + self.ninja.newline() + + return rule_name, args def CalculateVariables(default_variables, params): - """Calculate additional variables for use in the build (called by gyp).""" - global generator_additional_non_configuration_keys - global generator_additional_path_sections - flavor = gyp.common.GetFlavor(params) - if flavor == 'mac': - default_variables.setdefault('OS', 'mac') - default_variables.setdefault('SHARED_LIB_SUFFIX', '.dylib') - default_variables.setdefault('SHARED_LIB_DIR', - generator_default_variables['PRODUCT_DIR']) - default_variables.setdefault('LIB_DIR', - generator_default_variables['PRODUCT_DIR']) - - # Copy additional generator configuration data from Xcode, which is shared - # by the Mac Ninja generator. - import gyp.generator.xcode as xcode_generator - generator_additional_non_configuration_keys = getattr(xcode_generator, - 'generator_additional_non_configuration_keys', []) - generator_additional_path_sections = getattr(xcode_generator, - 'generator_additional_path_sections', []) - global generator_extra_sources_for_rules - generator_extra_sources_for_rules = getattr(xcode_generator, - 'generator_extra_sources_for_rules', []) - elif flavor == 'win': - exts = gyp.MSVSUtil.TARGET_TYPE_EXT - default_variables.setdefault('OS', 'win') - default_variables['EXECUTABLE_SUFFIX'] = '.' + exts['executable'] - default_variables['STATIC_LIB_PREFIX'] = '' - default_variables['STATIC_LIB_SUFFIX'] = '.' + exts['static_library'] - default_variables['SHARED_LIB_PREFIX'] = '' - default_variables['SHARED_LIB_SUFFIX'] = '.' + exts['shared_library'] - - # Copy additional generator configuration data from VS, which is shared - # by the Windows Ninja generator. - import gyp.generator.msvs as msvs_generator - generator_additional_non_configuration_keys = getattr(msvs_generator, - 'generator_additional_non_configuration_keys', []) - generator_additional_path_sections = getattr(msvs_generator, - 'generator_additional_path_sections', []) - - gyp.msvs_emulation.CalculateCommonVariables(default_variables, params) - else: - operating_system = flavor - if flavor == 'android': - operating_system = 'linux' # Keep this legacy behavior for now. - default_variables.setdefault('OS', operating_system) - default_variables.setdefault('SHARED_LIB_SUFFIX', '.so') - default_variables.setdefault('SHARED_LIB_DIR', - os.path.join('$!PRODUCT_DIR', 'lib')) - default_variables.setdefault('LIB_DIR', - os.path.join('$!PRODUCT_DIR', 'obj')) + """Calculate additional variables for use in the build (called by gyp).""" + global generator_additional_non_configuration_keys + global generator_additional_path_sections + flavor = gyp.common.GetFlavor(params) + if flavor == "mac": + default_variables.setdefault("OS", "mac") + default_variables.setdefault("SHARED_LIB_SUFFIX", ".dylib") + default_variables.setdefault( + "SHARED_LIB_DIR", generator_default_variables["PRODUCT_DIR"] + ) + default_variables.setdefault( + "LIB_DIR", generator_default_variables["PRODUCT_DIR"] + ) + + # Copy additional generator configuration data from Xcode, which is shared + # by the Mac Ninja generator. + import gyp.generator.xcode as xcode_generator + + generator_additional_non_configuration_keys = getattr( + xcode_generator, "generator_additional_non_configuration_keys", [] + ) + generator_additional_path_sections = getattr( + xcode_generator, "generator_additional_path_sections", [] + ) + global generator_extra_sources_for_rules + generator_extra_sources_for_rules = getattr( + xcode_generator, "generator_extra_sources_for_rules", [] + ) + elif flavor == "win": + exts = gyp.MSVSUtil.TARGET_TYPE_EXT + default_variables.setdefault("OS", "win") + default_variables["EXECUTABLE_SUFFIX"] = "." + exts["executable"] + default_variables["STATIC_LIB_PREFIX"] = "" + default_variables["STATIC_LIB_SUFFIX"] = "." + exts["static_library"] + default_variables["SHARED_LIB_PREFIX"] = "" + default_variables["SHARED_LIB_SUFFIX"] = "." + exts["shared_library"] + + # Copy additional generator configuration data from VS, which is shared + # by the Windows Ninja generator. + import gyp.generator.msvs as msvs_generator + + generator_additional_non_configuration_keys = getattr( + msvs_generator, "generator_additional_non_configuration_keys", [] + ) + generator_additional_path_sections = getattr( + msvs_generator, "generator_additional_path_sections", [] + ) + + gyp.msvs_emulation.CalculateCommonVariables(default_variables, params) + else: + operating_system = flavor + if flavor == "android": + operating_system = "linux" # Keep this legacy behavior for now. + default_variables.setdefault("OS", operating_system) + default_variables.setdefault("SHARED_LIB_SUFFIX", ".so") + default_variables.setdefault( + "SHARED_LIB_DIR", os.path.join("$!PRODUCT_DIR", "lib") + ) + default_variables.setdefault("LIB_DIR", os.path.join("$!PRODUCT_DIR", "obj")) + def ComputeOutputDir(params): - """Returns the path from the toplevel_dir to the build output directory.""" - # generator_dir: relative path from pwd to where make puts build files. - # Makes migrating from make to ninja easier, ninja doesn't put anything here. - generator_dir = os.path.relpath(params['options'].generator_output or '.') + """Returns the path from the toplevel_dir to the build output directory.""" + # generator_dir: relative path from pwd to where make puts build files. + # Makes migrating from make to ninja easier, ninja doesn't put anything here. + generator_dir = os.path.relpath(params["options"].generator_output or ".") - # output_dir: relative path from generator_dir to the build directory. - output_dir = params.get('generator_flags', {}).get('output_dir', 'out') + # output_dir: relative path from generator_dir to the build directory. + output_dir = params.get("generator_flags", {}).get("output_dir", "out") - # Relative path from source root to our output files. e.g. "out" - return os.path.normpath(os.path.join(generator_dir, output_dir)) + # Relative path from source root to our output files. e.g. "out" + return os.path.normpath(os.path.join(generator_dir, output_dir)) def CalculateGeneratorInputInfo(params): - """Called by __init__ to initialize generator values based on params.""" - # E.g. "out/gypfiles" - toplevel = params['options'].toplevel_dir - qualified_out_dir = os.path.normpath(os.path.join( - toplevel, ComputeOutputDir(params), 'gypfiles')) - - global generator_filelist_paths - generator_filelist_paths = { - 'toplevel': toplevel, - 'qualified_out_dir': qualified_out_dir, - } + """Called by __init__ to initialize generator values based on params.""" + # E.g. "out/gypfiles" + toplevel = params["options"].toplevel_dir + qualified_out_dir = os.path.normpath( + os.path.join(toplevel, ComputeOutputDir(params), "gypfiles") + ) + + global generator_filelist_paths + generator_filelist_paths = { + "toplevel": toplevel, + "qualified_out_dir": qualified_out_dir, + } -def OpenOutput(path, mode='w'): - """Open |path| for writing, creating directories if necessary.""" - gyp.common.EnsureDirExists(path) - return open(path, mode) +def OpenOutput(path, mode="w"): + """Open |path| for writing, creating directories if necessary.""" + gyp.common.EnsureDirExists(path) + return open(path, mode) def CommandWithWrapper(cmd, wrappers, prog): - wrapper = wrappers.get(cmd, '') - if wrapper: - return wrapper + ' ' + prog - return prog + wrapper = wrappers.get(cmd, "") + if wrapper: + return wrapper + " " + prog + return prog def GetDefaultConcurrentLinks(): - """Returns a best-guess for a number of concurrent links.""" - pool_size = int(os.environ.get('GYP_LINK_CONCURRENCY', 0)) - if pool_size: - return pool_size - - if sys.platform in ('win32', 'cygwin'): - import ctypes - - class MEMORYSTATUSEX(ctypes.Structure): - _fields_ = [ - ("dwLength", ctypes.c_ulong), - ("dwMemoryLoad", ctypes.c_ulong), - ("ullTotalPhys", ctypes.c_ulonglong), - ("ullAvailPhys", ctypes.c_ulonglong), - ("ullTotalPageFile", ctypes.c_ulonglong), - ("ullAvailPageFile", ctypes.c_ulonglong), - ("ullTotalVirtual", ctypes.c_ulonglong), - ("ullAvailVirtual", ctypes.c_ulonglong), - ("sullAvailExtendedVirtual", ctypes.c_ulonglong), - ] - - stat = MEMORYSTATUSEX() - stat.dwLength = ctypes.sizeof(stat) - ctypes.windll.kernel32.GlobalMemoryStatusEx(ctypes.byref(stat)) - - # VS 2015 uses 20% more working set than VS 2013 and can consume all RAM - # on a 64 GB machine. - mem_limit = max(1, stat.ullTotalPhys / (5 * (2 ** 30))) # total / 5GB - hard_cap = max(1, int(os.environ.get('GYP_LINK_CONCURRENCY_MAX', 2**32))) - return min(mem_limit, hard_cap) - elif sys.platform.startswith('linux'): - if os.path.exists("/proc/meminfo"): - with open("/proc/meminfo") as meminfo: - memtotal_re = re.compile(r'^MemTotal:\s*(\d*)\s*kB') - for line in meminfo: - match = memtotal_re.match(line) - if not match: - continue - # Allow 8Gb per link on Linux because Gold is quite memory hungry - return max(1, int(match.group(1)) / (8 * (2 ** 20))) - return 1 - elif sys.platform == 'darwin': - try: - avail_bytes = int(subprocess.check_output(['sysctl', '-n', 'hw.memsize'])) - # A static library debug build of Chromium's unit_tests takes ~2.7GB, so - # 4GB per ld process allows for some more bloat. - return max(1, avail_bytes / (4 * (2 ** 30))) # total / 4GB - except: - return 1 - else: - # TODO(scottmg): Implement this for other platforms. - return 1 + """Returns a best-guess for a number of concurrent links.""" + pool_size = int(os.environ.get("GYP_LINK_CONCURRENCY", 0)) + if pool_size: + return pool_size + + if sys.platform in ("win32", "cygwin"): + import ctypes + + class MEMORYSTATUSEX(ctypes.Structure): + _fields_ = [ + ("dwLength", ctypes.c_ulong), + ("dwMemoryLoad", ctypes.c_ulong), + ("ullTotalPhys", ctypes.c_ulonglong), + ("ullAvailPhys", ctypes.c_ulonglong), + ("ullTotalPageFile", ctypes.c_ulonglong), + ("ullAvailPageFile", ctypes.c_ulonglong), + ("ullTotalVirtual", ctypes.c_ulonglong), + ("ullAvailVirtual", ctypes.c_ulonglong), + ("sullAvailExtendedVirtual", ctypes.c_ulonglong), + ] + + stat = MEMORYSTATUSEX() + stat.dwLength = ctypes.sizeof(stat) + ctypes.windll.kernel32.GlobalMemoryStatusEx(ctypes.byref(stat)) + + # VS 2015 uses 20% more working set than VS 2013 and can consume all RAM + # on a 64 GB machine. + mem_limit = max(1, stat.ullTotalPhys // (5 * (2 ** 30))) # total / 5GB + hard_cap = max(1, int(os.environ.get("GYP_LINK_CONCURRENCY_MAX", 2 ** 32))) + return min(mem_limit, hard_cap) + elif sys.platform.startswith("linux"): + if os.path.exists("/proc/meminfo"): + with open("/proc/meminfo") as meminfo: + memtotal_re = re.compile(r"^MemTotal:\s*(\d*)\s*kB") + for line in meminfo: + match = memtotal_re.match(line) + if not match: + continue + # Allow 8Gb per link on Linux because Gold is quite memory hungry + return max(1, int(match.group(1)) // (8 * (2 ** 20))) + return 1 + elif sys.platform == "darwin": + try: + avail_bytes = int(subprocess.check_output(["sysctl", "-n", "hw.memsize"])) + # A static library debug build of Chromium's unit_tests takes ~2.7GB, so + # 4GB per ld process allows for some more bloat. + return max(1, avail_bytes // (4 * (2 ** 30))) # total / 4GB + except subprocess.CalledProcessError: + return 1 + else: + # TODO(scottmg): Implement this for other platforms. + return 1 def _GetWinLinkRuleNameSuffix(embed_manifest): - """Returns the suffix used to select an appropriate linking rule depending on + """Returns the suffix used to select an appropriate linking rule depending on whether the manifest embedding is enabled.""" - return '_embed' if embed_manifest else '' + return "_embed" if embed_manifest else "" def _AddWinLinkRules(master_ninja, embed_manifest): - """Adds link rules for Windows platform to |master_ninja|.""" - def FullLinkCommand(ldcmd, out, binary_type): - resource_name = { - 'exe': '1', - 'dll': '2', - }[binary_type] - return '%(python)s gyp-win-tool link-with-manifests $arch %(embed)s ' \ - '%(out)s "%(ldcmd)s" %(resname)s $mt $rc "$intermediatemanifest" ' \ - '$manifests' % { - 'python': sys.executable, - 'out': out, - 'ldcmd': ldcmd, - 'resname': resource_name, - 'embed': embed_manifest } - rule_name_suffix = _GetWinLinkRuleNameSuffix(embed_manifest) - use_separate_mspdbsrv = ( - int(os.environ.get('GYP_USE_SEPARATE_MSPDBSRV', '0')) != 0) - dlldesc = 'LINK%s(DLL) $binary' % rule_name_suffix.upper() - dllcmd = ('%s gyp-win-tool link-wrapper $arch %s ' - '$ld /nologo $implibflag /DLL /OUT:$binary ' - '@$binary.rsp' % (sys.executable, use_separate_mspdbsrv)) - dllcmd = FullLinkCommand(dllcmd, '$binary', 'dll') - master_ninja.rule('solink' + rule_name_suffix, - description=dlldesc, command=dllcmd, - rspfile='$binary.rsp', - rspfile_content='$libs $in_newline $ldflags', - restat=True, - pool='link_pool') - master_ninja.rule('solink_module' + rule_name_suffix, - description=dlldesc, command=dllcmd, - rspfile='$binary.rsp', - rspfile_content='$libs $in_newline $ldflags', - restat=True, - pool='link_pool') - # Note that ldflags goes at the end so that it has the option of - # overriding default settings earlier in the command line. - exe_cmd = ('%s gyp-win-tool link-wrapper $arch %s ' - '$ld /nologo /OUT:$binary @$binary.rsp' % - (sys.executable, use_separate_mspdbsrv)) - exe_cmd = FullLinkCommand(exe_cmd, '$binary', 'exe') - master_ninja.rule('link' + rule_name_suffix, - description='LINK%s $binary' % rule_name_suffix.upper(), - command=exe_cmd, - rspfile='$binary.rsp', - rspfile_content='$in_newline $libs $ldflags', - pool='link_pool') - - -def GenerateOutputForConfig(target_list, target_dicts, data, params, - config_name): - options = params['options'] - flavor = gyp.common.GetFlavor(params) - generator_flags = params.get('generator_flags', {}) - - # build_dir: relative path from source root to our output files. - # e.g. "out/Debug" - build_dir = os.path.normpath( - os.path.join(ComputeOutputDir(params), config_name)) - - toplevel_build = os.path.join(options.toplevel_dir, build_dir) - - master_ninja_file = OpenOutput(os.path.join(toplevel_build, 'build.ninja')) - master_ninja = ninja_syntax.Writer(master_ninja_file, width=120) - - # Put build-time support tools in out/{config_name}. - gyp.common.CopyTool(flavor, toplevel_build, generator_flags) - - # Grab make settings for CC/CXX. - # The rules are - # - The priority from low to high is gcc/g++, the 'make_global_settings' in - # gyp, the environment variable. - # - If there is no 'make_global_settings' for CC.host/CXX.host or - # 'CC_host'/'CXX_host' environment variable, cc_host/cxx_host should be set - # to cc/cxx. - if flavor == 'win': - ar = 'lib.exe' - # cc and cxx must be set to the correct architecture by overriding with one - # of cl_x86 or cl_x64 below. - cc = 'UNSET' - cxx = 'UNSET' - ld = 'link.exe' - ld_host = '$ld' - else: - ar = 'ar' - cc = 'cc' - cxx = 'c++' - ld = '$cc' - ldxx = '$cxx' - ld_host = '$cc_host' - ldxx_host = '$cxx_host' - - ar_host = ar - cc_host = None - cxx_host = None - cc_host_global_setting = None - cxx_host_global_setting = None - clang_cl = None - nm = 'nm' - nm_host = 'nm' - readelf = 'readelf' - readelf_host = 'readelf' - - build_file, _, _ = gyp.common.ParseQualifiedTarget(target_list[0]) - make_global_settings = data[build_file].get('make_global_settings', []) - build_to_root = gyp.common.InvertRelativePath(build_dir, - options.toplevel_dir) - wrappers = {} - for key, value in make_global_settings: - if key == 'AR': - ar = os.path.join(build_to_root, value) - if key == 'AR.host': - ar_host = os.path.join(build_to_root, value) - if key == 'CC': - cc = os.path.join(build_to_root, value) - if cc.endswith('clang-cl'): - clang_cl = cc - if key == 'CXX': - cxx = os.path.join(build_to_root, value) - if key == 'CC.host': - cc_host = os.path.join(build_to_root, value) - cc_host_global_setting = value - if key == 'CXX.host': - cxx_host = os.path.join(build_to_root, value) - cxx_host_global_setting = value - if key == 'LD': - ld = os.path.join(build_to_root, value) - if key == 'LD.host': - ld_host = os.path.join(build_to_root, value) - if key == 'LDXX': - ldxx = os.path.join(build_to_root, value) - if key == 'LDXX.host': - ldxx_host = os.path.join(build_to_root, value) - if key == 'NM': - nm = os.path.join(build_to_root, value) - if key == 'NM.host': - nm_host = os.path.join(build_to_root, value) - if key == 'READELF': - readelf = os.path.join(build_to_root, value) - if key == 'READELF.host': - readelf_host = os.path.join(build_to_root, value) - if key.endswith('_wrapper'): - wrappers[key[:-len('_wrapper')]] = os.path.join(build_to_root, value) - - # Support wrappers from environment variables too. - for key, value in os.environ.items(): - if key.lower().endswith('_wrapper'): - key_prefix = key[:-len('_wrapper')] - key_prefix = re.sub(r'\.HOST$', '.host', key_prefix) - wrappers[key_prefix] = os.path.join(build_to_root, value) - - mac_toolchain_dir = generator_flags.get('mac_toolchain_dir', None) - if mac_toolchain_dir: - wrappers['LINK'] = "export DEVELOPER_DIR='%s' &&" % mac_toolchain_dir - - if flavor == 'win': - configs = [target_dicts[qualified_target]['configurations'][config_name] - for qualified_target in target_list] - shared_system_includes = None - if not generator_flags.get('ninja_use_custom_environment_files', 0): - shared_system_includes = \ - gyp.msvs_emulation.ExtractSharedMSVSSystemIncludes( - configs, generator_flags) - cl_paths = gyp.msvs_emulation.GenerateEnvironmentFiles( - toplevel_build, generator_flags, shared_system_includes, OpenOutput) - for arch, path in sorted(cl_paths.items()): - if clang_cl: - # If we have selected clang-cl, use that instead. - path = clang_cl - command = CommandWithWrapper('CC', wrappers, - QuoteShellArgument(path, 'win')) - if clang_cl: - # Use clang-cl to cross-compile for x86 or x86_64. - command += (' -m32' if arch == 'x86' else ' -m64') - master_ninja.variable('cl_' + arch, command) - - cc = GetEnvironFallback(['CC_target', 'CC'], cc) - master_ninja.variable('cc', CommandWithWrapper('CC', wrappers, cc)) - cxx = GetEnvironFallback(['CXX_target', 'CXX'], cxx) - master_ninja.variable('cxx', CommandWithWrapper('CXX', wrappers, cxx)) - - if flavor == 'win': - master_ninja.variable('ld', ld) - master_ninja.variable('idl', 'midl.exe') - master_ninja.variable('ar', ar) - master_ninja.variable('rc', 'rc.exe') - master_ninja.variable('ml_x86', 'ml.exe') - master_ninja.variable('ml_x64', 'ml64.exe') - master_ninja.variable('mt', 'mt.exe') - else: - master_ninja.variable('ld', CommandWithWrapper('LINK', wrappers, ld)) - master_ninja.variable('ldxx', CommandWithWrapper('LINK', wrappers, ldxx)) - master_ninja.variable('ar', GetEnvironFallback(['AR_target', 'AR'], ar)) - if flavor != 'mac': - # Mac does not use readelf/nm for .TOC generation, so avoiding polluting - # the master ninja with extra unused variables. - master_ninja.variable( - 'nm', GetEnvironFallback(['NM_target', 'NM'], nm)) - master_ninja.variable( - 'readelf', GetEnvironFallback(['READELF_target', 'READELF'], readelf)) - - if generator_supports_multiple_toolsets: - if not cc_host: - cc_host = cc - if not cxx_host: - cxx_host = cxx - - master_ninja.variable('ar_host', GetEnvironFallback(['AR_host'], ar_host)) - master_ninja.variable('nm_host', GetEnvironFallback(['NM_host'], nm_host)) - master_ninja.variable('readelf_host', - GetEnvironFallback(['READELF_host'], readelf_host)) - cc_host = GetEnvironFallback(['CC_host'], cc_host) - cxx_host = GetEnvironFallback(['CXX_host'], cxx_host) - - # The environment variable could be used in 'make_global_settings', like - # ['CC.host', '$(CC)'] or ['CXX.host', '$(CXX)'], transform them here. - if '$(CC)' in cc_host and cc_host_global_setting: - cc_host = cc_host_global_setting.replace('$(CC)', cc) - if '$(CXX)' in cxx_host and cxx_host_global_setting: - cxx_host = cxx_host_global_setting.replace('$(CXX)', cxx) - master_ninja.variable('cc_host', - CommandWithWrapper('CC.host', wrappers, cc_host)) - master_ninja.variable('cxx_host', - CommandWithWrapper('CXX.host', wrappers, cxx_host)) - if flavor == 'win': - master_ninja.variable('ld_host', ld_host) - master_ninja.variable('ldxx_host', ldxx_host) - else: - master_ninja.variable('ld_host', CommandWithWrapper( - 'LINK', wrappers, ld_host)) - master_ninja.variable('ldxx_host', CommandWithWrapper( - 'LINK', wrappers, ldxx_host)) - - master_ninja.newline() - - master_ninja.pool('link_pool', depth=GetDefaultConcurrentLinks()) - master_ninja.newline() - - deps = 'msvc' if flavor == 'win' else 'gcc' - - if flavor != 'win': - master_ninja.rule( - 'cc', - description='CC $out', - command=('$cc -MMD -MF $out.d $defines $includes $cflags $cflags_c ' - '$cflags_pch_c -c $in -o $out'), - depfile='$out.d', - deps=deps) - master_ninja.rule( - 'cc_s', - description='CC $out', - command=('$cc $defines $includes $cflags $cflags_c ' - '$cflags_pch_c -c $in -o $out')) - master_ninja.rule( - 'cxx', - description='CXX $out', - command=('$cxx -MMD -MF $out.d $defines $includes $cflags $cflags_cc ' - '$cflags_pch_cc -c $in -o $out'), - depfile='$out.d', - deps=deps) - else: - # TODO(scottmg) Separate pdb names is a test to see if it works around - # http://crbug.com/142362. It seems there's a race between the creation of - # the .pdb by the precompiled header step for .cc and the compilation of - # .c files. This should be handled by mspdbsrv, but rarely errors out with - # c1xx : fatal error C1033: cannot open program database - # By making the rules target separate pdb files this might be avoided. - cc_command = ('ninja -t msvc -e $arch ' + - '-- ' - '$cc /nologo /showIncludes /FC ' - '@$out.rsp /c $in /Fo$out /Fd$pdbname_c ') - cxx_command = ('ninja -t msvc -e $arch ' + - '-- ' - '$cxx /nologo /showIncludes /FC ' - '@$out.rsp /c $in /Fo$out /Fd$pdbname_cc ') - master_ninja.rule( - 'cc', - description='CC $out', - command=cc_command, - rspfile='$out.rsp', - rspfile_content='$defines $includes $cflags $cflags_c', - deps=deps) - master_ninja.rule( - 'cxx', - description='CXX $out', - command=cxx_command, - rspfile='$out.rsp', - rspfile_content='$defines $includes $cflags $cflags_cc', - deps=deps) - master_ninja.rule( - 'idl', - description='IDL $in', - command=('%s gyp-win-tool midl-wrapper $arch $outdir ' - '$tlb $h $dlldata $iid $proxy $in ' - '$midl_includes $idlflags' % sys.executable)) - master_ninja.rule( - 'rc', - description='RC $in', - # Note: $in must be last otherwise rc.exe complains. - command=('%s gyp-win-tool rc-wrapper ' - '$arch $rc $defines $resource_includes $rcflags /fo$out $in' % - sys.executable)) - master_ninja.rule( - 'asm', - description='ASM $out', - command=('%s gyp-win-tool asm-wrapper ' - '$arch $asm $defines $includes $asmflags /c /Fo $out $in' % - sys.executable)) - - if flavor != 'mac' and flavor != 'win': - master_ninja.rule( - 'alink', - description='AR $out', - command='rm -f $out && $ar rcs $arflags $out $in') - master_ninja.rule( - 'alink_thin', - description='AR $out', - command='rm -f $out && $ar rcsT $arflags $out $in') - - # This allows targets that only need to depend on $lib's API to declare an - # order-only dependency on $lib.TOC and avoid relinking such downstream - # dependencies when $lib changes only in non-public ways. - # The resulting string leaves an uninterpolated %{suffix} which - # is used in the final substitution below. - mtime_preserving_solink_base = ( - 'if [ ! -e $lib -o ! -e $lib.TOC ]; then ' - '%(solink)s && %(extract_toc)s > $lib.TOC; else ' - '%(solink)s && %(extract_toc)s > $lib.tmp && ' - 'if ! cmp -s $lib.tmp $lib.TOC; then mv $lib.tmp $lib.TOC ; ' - 'fi; fi' - % { 'solink': - '$ld -shared $ldflags -o $lib -Wl,-soname=$soname %(suffix)s', - 'extract_toc': - ('{ $readelf -d $lib | grep SONAME ; ' - '$nm -gD -f p $lib | cut -f1-2 -d\' \'; }')}) - - master_ninja.rule( - 'solink', - description='SOLINK $lib', - restat=True, - command=mtime_preserving_solink_base % {'suffix': '@$link_file_list'}, - rspfile='$link_file_list', - rspfile_content= - '-Wl,--whole-archive $in $solibs -Wl,--no-whole-archive $libs', - pool='link_pool') - master_ninja.rule( - 'solink_module', - description='SOLINK(module) $lib', - restat=True, - command=mtime_preserving_solink_base % {'suffix': '@$link_file_list'}, - rspfile='$link_file_list', - rspfile_content='-Wl,--start-group $in $solibs $libs -Wl,--end-group', - pool='link_pool') - master_ninja.rule( - 'link', - description='LINK $out', - command=('$ld $ldflags -o $out ' - '-Wl,--start-group $in $solibs $libs -Wl,--end-group'), - pool='link_pool') - elif flavor == 'win': + """Adds link rules for Windows platform to |master_ninja|.""" + + def FullLinkCommand(ldcmd, out, binary_type): + resource_name = {"exe": "1", "dll": "2"}[binary_type] + return ( + "%(python)s gyp-win-tool link-with-manifests $arch %(embed)s " + '%(out)s "%(ldcmd)s" %(resname)s $mt $rc "$intermediatemanifest" ' + "$manifests" + % { + "python": sys.executable, + "out": out, + "ldcmd": ldcmd, + "resname": resource_name, + "embed": embed_manifest, + } + ) + + rule_name_suffix = _GetWinLinkRuleNameSuffix(embed_manifest) + use_separate_mspdbsrv = int(os.environ.get("GYP_USE_SEPARATE_MSPDBSRV", "0")) != 0 + dlldesc = "LINK%s(DLL) $binary" % rule_name_suffix.upper() + dllcmd = ( + "%s gyp-win-tool link-wrapper $arch %s " + "$ld /nologo $implibflag /DLL /OUT:$binary " + "@$binary.rsp" % (sys.executable, use_separate_mspdbsrv) + ) + dllcmd = FullLinkCommand(dllcmd, "$binary", "dll") master_ninja.rule( - 'alink', - description='LIB $out', - command=('%s gyp-win-tool link-wrapper $arch False ' - '$ar /nologo /ignore:4221 /OUT:$out @$out.rsp' % - sys.executable), - rspfile='$out.rsp', - rspfile_content='$in_newline $libflags') - _AddWinLinkRules(master_ninja, embed_manifest=True) - _AddWinLinkRules(master_ninja, embed_manifest=False) - else: + "solink" + rule_name_suffix, + description=dlldesc, + command=dllcmd, + rspfile="$binary.rsp", + rspfile_content="$libs $in_newline $ldflags", + restat=True, + pool="link_pool", + ) master_ninja.rule( - 'objc', - description='OBJC $out', - command=('$cc -MMD -MF $out.d $defines $includes $cflags $cflags_objc ' - '$cflags_pch_objc -c $in -o $out'), - depfile='$out.d', - deps=deps) + "solink_module" + rule_name_suffix, + description=dlldesc, + command=dllcmd, + rspfile="$binary.rsp", + rspfile_content="$libs $in_newline $ldflags", + restat=True, + pool="link_pool", + ) + # Note that ldflags goes at the end so that it has the option of + # overriding default settings earlier in the command line. + exe_cmd = ( + "%s gyp-win-tool link-wrapper $arch %s " + "$ld /nologo /OUT:$binary @$binary.rsp" + % (sys.executable, use_separate_mspdbsrv) + ) + exe_cmd = FullLinkCommand(exe_cmd, "$binary", "exe") master_ninja.rule( - 'objcxx', - description='OBJCXX $out', - command=('$cxx -MMD -MF $out.d $defines $includes $cflags $cflags_objcc ' - '$cflags_pch_objcc -c $in -o $out'), - depfile='$out.d', - deps=deps) - master_ninja.rule( - 'alink', - description='LIBTOOL-STATIC $out, POSTBUILDS', - command='rm -f $out && ' - './gyp-mac-tool filter-libtool libtool $libtool_flags ' - '-static -o $out $in' - '$postbuilds') - master_ninja.rule( - 'lipo', - description='LIPO $out, POSTBUILDS', - command='rm -f $out && lipo -create $in -output $out$postbuilds') - master_ninja.rule( - 'solipo', - description='SOLIPO $out, POSTBUILDS', - command=( - 'rm -f $lib $lib.TOC && lipo -create $in -output $lib$postbuilds &&' - '%(extract_toc)s > $lib.TOC' - % { 'extract_toc': - '{ otool -l $lib | grep LC_ID_DYLIB -A 5; ' - 'nm -gP $lib | cut -f1-2 -d\' \' | grep -v U$$; true; }'})) - - - # Record the public interface of $lib in $lib.TOC. See the corresponding - # comment in the posix section above for details. - solink_base = '$ld %(type)s $ldflags -o $lib %(suffix)s' - mtime_preserving_solink_base = ( - 'if [ ! -e $lib -o ! -e $lib.TOC ] || ' - # Always force dependent targets to relink if this library - # reexports something. Handling this correctly would require - # recursive TOC dumping but this is rare in practice, so punt. - 'otool -l $lib | grep -q LC_REEXPORT_DYLIB ; then ' - '%(solink)s && %(extract_toc)s > $lib.TOC; ' - 'else ' - '%(solink)s && %(extract_toc)s > $lib.tmp && ' - 'if ! cmp -s $lib.tmp $lib.TOC; then ' - 'mv $lib.tmp $lib.TOC ; ' - 'fi; ' - 'fi' - % { 'solink': solink_base, - 'extract_toc': - '{ otool -l $lib | grep LC_ID_DYLIB -A 5; ' - 'nm -gP $lib | cut -f1-2 -d\' \' | grep -v U$$; true; }'}) - - - solink_suffix = '@$link_file_list$postbuilds' - master_ninja.rule( - 'solink', - description='SOLINK $lib, POSTBUILDS', - restat=True, - command=mtime_preserving_solink_base % {'suffix': solink_suffix, - 'type': '-shared'}, - rspfile='$link_file_list', - rspfile_content='$in $solibs $libs', - pool='link_pool') - master_ninja.rule( - 'solink_notoc', - description='SOLINK $lib, POSTBUILDS', - restat=True, - command=solink_base % {'suffix':solink_suffix, 'type': '-shared'}, - rspfile='$link_file_list', - rspfile_content='$in $solibs $libs', - pool='link_pool') - - master_ninja.rule( - 'solink_module', - description='SOLINK(module) $lib, POSTBUILDS', - restat=True, - command=mtime_preserving_solink_base % {'suffix': solink_suffix, - 'type': '-bundle'}, - rspfile='$link_file_list', - rspfile_content='$in $solibs $libs', - pool='link_pool') - master_ninja.rule( - 'solink_module_notoc', - description='SOLINK(module) $lib, POSTBUILDS', - restat=True, - command=solink_base % {'suffix': solink_suffix, 'type': '-bundle'}, - rspfile='$link_file_list', - rspfile_content='$in $solibs $libs', - pool='link_pool') - - master_ninja.rule( - 'link', - description='LINK $out, POSTBUILDS', - command=('$ld $ldflags -o $out ' - '$in $solibs $libs$postbuilds'), - pool='link_pool') - master_ninja.rule( - 'preprocess_infoplist', - description='PREPROCESS INFOPLIST $out', - command=('$cc -E -P -Wno-trigraphs -x c $defines $in -o $out && ' - 'plutil -convert xml1 $out $out')) - master_ninja.rule( - 'copy_infoplist', - description='COPY INFOPLIST $in', - command='$env ./gyp-mac-tool copy-info-plist $in $out $binary $keys') - master_ninja.rule( - 'merge_infoplist', - description='MERGE INFOPLISTS $in', - command='$env ./gyp-mac-tool merge-info-plist $out $in') - master_ninja.rule( - 'compile_xcassets', - description='COMPILE XCASSETS $in', - command='$env ./gyp-mac-tool compile-xcassets $keys $in') - master_ninja.rule( - 'compile_ios_framework_headers', - description='COMPILE HEADER MAPS AND COPY FRAMEWORK HEADERS $in', - command='$env ./gyp-mac-tool compile-ios-framework-header-map $out ' - '$framework $in && $env ./gyp-mac-tool ' - 'copy-ios-framework-headers $framework $copy_headers') - master_ninja.rule( - 'mac_tool', - description='MACTOOL $mactool_cmd $in', - command='$env ./gyp-mac-tool $mactool_cmd $in $out $binary') - master_ninja.rule( - 'package_framework', - description='PACKAGE FRAMEWORK $out, POSTBUILDS', - command='./gyp-mac-tool package-framework $out $version$postbuilds ' - '&& touch $out') - master_ninja.rule( - 'package_ios_framework', - description='PACKAGE IOS FRAMEWORK $out, POSTBUILDS', - command='./gyp-mac-tool package-ios-framework $out $postbuilds ' - '&& touch $out') - if flavor == 'win': - master_ninja.rule( - 'stamp', - description='STAMP $out', - command='%s gyp-win-tool stamp $out' % sys.executable) - else: - master_ninja.rule( - 'stamp', - description='STAMP $out', - command='${postbuilds}touch $out') - if flavor == 'win': - master_ninja.rule( - 'copy', - description='COPY $in $out', - command='%s gyp-win-tool recursive-mirror $in $out' % sys.executable) - elif flavor == 'zos': - master_ninja.rule( - 'copy', - description='COPY $in $out', - command='rm -rf $out && cp -fRP $in $out') - else: - master_ninja.rule( - 'copy', - description='COPY $in $out', - command='ln -f $in $out 2>/dev/null || (rm -rf $out && cp -af $in $out)') - master_ninja.newline() - - all_targets = set() - for build_file in params['build_files']: - for target in gyp.common.AllTargets(target_list, - target_dicts, - os.path.normpath(build_file)): - all_targets.add(target) - all_outputs = set() - - # target_outputs is a map from qualified target name to a Target object. - target_outputs = {} - # target_short_names is a map from target short name to a list of Target - # objects. - target_short_names = {} - - # short name of targets that were skipped because they didn't contain anything - # interesting. - # NOTE: there may be overlap between this an non_empty_target_names. - empty_target_names = set() - - # Set of non-empty short target names. - # NOTE: there may be overlap between this an empty_target_names. - non_empty_target_names = set() - - for qualified_target in target_list: - # qualified_target is like: third_party/icu/icu.gyp:icui18n#target - build_file, name, toolset = \ - gyp.common.ParseQualifiedTarget(qualified_target) - - this_make_global_settings = data[build_file].get('make_global_settings', []) - assert make_global_settings == this_make_global_settings, ( - "make_global_settings needs to be the same for all targets. %s vs. %s" % - (this_make_global_settings, make_global_settings)) - - spec = target_dicts[qualified_target] - if flavor == 'mac': - gyp.xcode_emulation.MergeGlobalXcodeSettingsToSpec(data[build_file], spec) - - # If build_file is a symlink, we must not follow it because there's a chance - # it could point to a path above toplevel_dir, and we cannot correctly deal - # with that case at the moment. - build_file = gyp.common.RelativePath(build_file, options.toplevel_dir, - False) - - qualified_target_for_hash = gyp.common.QualifiedTarget(build_file, name, - toolset) - qualified_target_for_hash = qualified_target_for_hash.encode('utf-8') - hash_for_rules = hashlib.md5(qualified_target_for_hash).hexdigest() - - base_path = os.path.dirname(build_file) - obj = 'obj' - if toolset != 'target': - obj += '.' + toolset - output_file = os.path.join(obj, base_path, name + '.ninja') - - ninja_output = StringIO() - writer = NinjaWriter(hash_for_rules, target_outputs, base_path, build_dir, - ninja_output, - toplevel_build, output_file, - flavor, toplevel_dir=options.toplevel_dir) - - target = writer.WriteSpec(spec, config_name, generator_flags) - - if ninja_output.tell() > 0: - # Only create files for ninja files that actually have contents. - with OpenOutput(os.path.join(toplevel_build, output_file)) as ninja_file: - ninja_file.write(ninja_output.getvalue()) - ninja_output.close() - master_ninja.subninja(output_file) - - if target: - if name != target.FinalOutput() and spec['toolset'] == 'target': - target_short_names.setdefault(name, []).append(target) - target_outputs[qualified_target] = target - if qualified_target in all_targets: - all_outputs.add(target.FinalOutput()) - non_empty_target_names.add(name) + "link" + rule_name_suffix, + description="LINK%s $binary" % rule_name_suffix.upper(), + command=exe_cmd, + rspfile="$binary.rsp", + rspfile_content="$in_newline $libs $ldflags", + pool="link_pool", + ) + + +def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name): + options = params["options"] + flavor = gyp.common.GetFlavor(params) + generator_flags = params.get("generator_flags", {}) + + # build_dir: relative path from source root to our output files. + # e.g. "out/Debug" + build_dir = os.path.normpath(os.path.join(ComputeOutputDir(params), config_name)) + + toplevel_build = os.path.join(options.toplevel_dir, build_dir) + + master_ninja_file = OpenOutput(os.path.join(toplevel_build, "build.ninja")) + master_ninja = ninja_syntax.Writer(master_ninja_file, width=120) + + # Put build-time support tools in out/{config_name}. + gyp.common.CopyTool(flavor, toplevel_build, generator_flags) + + # Grab make settings for CC/CXX. + # The rules are + # - The priority from low to high is gcc/g++, the 'make_global_settings' in + # gyp, the environment variable. + # - If there is no 'make_global_settings' for CC.host/CXX.host or + # 'CC_host'/'CXX_host' environment variable, cc_host/cxx_host should be set + # to cc/cxx. + if flavor == "win": + ar = "lib.exe" + # cc and cxx must be set to the correct architecture by overriding with one + # of cl_x86 or cl_x64 below. + cc = "UNSET" + cxx = "UNSET" + ld = "link.exe" + ld_host = "$ld" + else: + ar = "ar" + cc = "cc" + cxx = "c++" + ld = "$cc" + ldxx = "$cxx" + ld_host = "$cc_host" + ldxx_host = "$cxx_host" + + ar_host = ar + cc_host = None + cxx_host = None + cc_host_global_setting = None + cxx_host_global_setting = None + clang_cl = None + nm = "nm" + nm_host = "nm" + readelf = "readelf" + readelf_host = "readelf" + + build_file, _, _ = gyp.common.ParseQualifiedTarget(target_list[0]) + make_global_settings = data[build_file].get("make_global_settings", []) + build_to_root = gyp.common.InvertRelativePath(build_dir, options.toplevel_dir) + wrappers = {} + for key, value in make_global_settings: + if key == "AR": + ar = os.path.join(build_to_root, value) + if key == "AR.host": + ar_host = os.path.join(build_to_root, value) + if key == "CC": + cc = os.path.join(build_to_root, value) + if cc.endswith("clang-cl"): + clang_cl = cc + if key == "CXX": + cxx = os.path.join(build_to_root, value) + if key == "CC.host": + cc_host = os.path.join(build_to_root, value) + cc_host_global_setting = value + if key == "CXX.host": + cxx_host = os.path.join(build_to_root, value) + cxx_host_global_setting = value + if key == "LD": + ld = os.path.join(build_to_root, value) + if key == "LD.host": + ld_host = os.path.join(build_to_root, value) + if key == "LDXX": + ldxx = os.path.join(build_to_root, value) + if key == "LDXX.host": + ldxx_host = os.path.join(build_to_root, value) + if key == "NM": + nm = os.path.join(build_to_root, value) + if key == "NM.host": + nm_host = os.path.join(build_to_root, value) + if key == "READELF": + readelf = os.path.join(build_to_root, value) + if key == "READELF.host": + readelf_host = os.path.join(build_to_root, value) + if key.endswith("_wrapper"): + wrappers[key[: -len("_wrapper")]] = os.path.join(build_to_root, value) + + # Support wrappers from environment variables too. + for key, value in os.environ.items(): + if key.lower().endswith("_wrapper"): + key_prefix = key[: -len("_wrapper")] + key_prefix = re.sub(r"\.HOST$", ".host", key_prefix) + wrappers[key_prefix] = os.path.join(build_to_root, value) + + mac_toolchain_dir = generator_flags.get("mac_toolchain_dir", None) + if mac_toolchain_dir: + wrappers["LINK"] = "export DEVELOPER_DIR='%s' &&" % mac_toolchain_dir + + if flavor == "win": + configs = [ + target_dicts[qualified_target]["configurations"][config_name] + for qualified_target in target_list + ] + shared_system_includes = None + if not generator_flags.get("ninja_use_custom_environment_files", 0): + shared_system_includes = gyp.msvs_emulation.ExtractSharedMSVSSystemIncludes( + configs, generator_flags + ) + cl_paths = gyp.msvs_emulation.GenerateEnvironmentFiles( + toplevel_build, generator_flags, shared_system_includes, OpenOutput + ) + for arch, path in sorted(cl_paths.items()): + if clang_cl: + # If we have selected clang-cl, use that instead. + path = clang_cl + command = CommandWithWrapper( + "CC", wrappers, QuoteShellArgument(path, "win") + ) + if clang_cl: + # Use clang-cl to cross-compile for x86 or x86_64. + command += " -m32" if arch == "x86" else " -m64" + master_ninja.variable("cl_" + arch, command) + + cc = GetEnvironFallback(["CC_target", "CC"], cc) + master_ninja.variable("cc", CommandWithWrapper("CC", wrappers, cc)) + cxx = GetEnvironFallback(["CXX_target", "CXX"], cxx) + master_ninja.variable("cxx", CommandWithWrapper("CXX", wrappers, cxx)) + + if flavor == "win": + master_ninja.variable("ld", ld) + master_ninja.variable("idl", "midl.exe") + master_ninja.variable("ar", ar) + master_ninja.variable("rc", "rc.exe") + master_ninja.variable("ml_x86", "ml.exe") + master_ninja.variable("ml_x64", "ml64.exe") + master_ninja.variable("mt", "mt.exe") else: - empty_target_names.add(name) + master_ninja.variable("ld", CommandWithWrapper("LINK", wrappers, ld)) + master_ninja.variable("ldxx", CommandWithWrapper("LINK", wrappers, ldxx)) + master_ninja.variable("ar", GetEnvironFallback(["AR_target", "AR"], ar)) + if flavor != "mac": + # Mac does not use readelf/nm for .TOC generation, so avoiding polluting + # the master ninja with extra unused variables. + master_ninja.variable("nm", GetEnvironFallback(["NM_target", "NM"], nm)) + master_ninja.variable( + "readelf", GetEnvironFallback(["READELF_target", "READELF"], readelf) + ) + + if generator_supports_multiple_toolsets: + if not cc_host: + cc_host = cc + if not cxx_host: + cxx_host = cxx + + master_ninja.variable("ar_host", GetEnvironFallback(["AR_host"], ar_host)) + master_ninja.variable("nm_host", GetEnvironFallback(["NM_host"], nm_host)) + master_ninja.variable( + "readelf_host", GetEnvironFallback(["READELF_host"], readelf_host) + ) + cc_host = GetEnvironFallback(["CC_host"], cc_host) + cxx_host = GetEnvironFallback(["CXX_host"], cxx_host) + + # The environment variable could be used in 'make_global_settings', like + # ['CC.host', '$(CC)'] or ['CXX.host', '$(CXX)'], transform them here. + if "$(CC)" in cc_host and cc_host_global_setting: + cc_host = cc_host_global_setting.replace("$(CC)", cc) + if "$(CXX)" in cxx_host and cxx_host_global_setting: + cxx_host = cxx_host_global_setting.replace("$(CXX)", cxx) + master_ninja.variable( + "cc_host", CommandWithWrapper("CC.host", wrappers, cc_host) + ) + master_ninja.variable( + "cxx_host", CommandWithWrapper("CXX.host", wrappers, cxx_host) + ) + if flavor == "win": + master_ninja.variable("ld_host", ld_host) + master_ninja.variable("ldxx_host", ldxx_host) + else: + master_ninja.variable( + "ld_host", CommandWithWrapper("LINK", wrappers, ld_host) + ) + master_ninja.variable( + "ldxx_host", CommandWithWrapper("LINK", wrappers, ldxx_host) + ) - if target_short_names: - # Write a short name to build this target. This benefits both the - # "build chrome" case as well as the gyp tests, which expect to be - # able to run actions and build libraries by their short name. master_ninja.newline() - master_ninja.comment('Short names for targets.') - for short_name in sorted(target_short_names): - master_ninja.build(short_name, 'phony', [x.FinalOutput() for x in - target_short_names[short_name]]) - - # Write phony targets for any empty targets that weren't written yet. As - # short names are not necessarily unique only do this for short names that - # haven't already been output for another target. - empty_target_names = empty_target_names - non_empty_target_names - if empty_target_names: + + master_ninja.pool("link_pool", depth=GetDefaultConcurrentLinks()) master_ninja.newline() - master_ninja.comment('Empty targets (output for completeness).') - for name in sorted(empty_target_names): - master_ninja.build(name, 'phony') - if all_outputs: + deps = "msvc" if flavor == "win" else "gcc" + + if flavor != "win": + master_ninja.rule( + "cc", + description="CC $out", + command=( + "$cc -MMD -MF $out.d $defines $includes $cflags $cflags_c " + "$cflags_pch_c -c $in -o $out" + ), + depfile="$out.d", + deps=deps, + ) + master_ninja.rule( + "cc_s", + description="CC $out", + command=( + "$cc $defines $includes $cflags $cflags_c " + "$cflags_pch_c -c $in -o $out" + ), + ) + master_ninja.rule( + "cxx", + description="CXX $out", + command=( + "$cxx -MMD -MF $out.d $defines $includes $cflags $cflags_cc " + "$cflags_pch_cc -c $in -o $out" + ), + depfile="$out.d", + deps=deps, + ) + else: + # TODO(scottmg) Separate pdb names is a test to see if it works around + # http://crbug.com/142362. It seems there's a race between the creation of + # the .pdb by the precompiled header step for .cc and the compilation of + # .c files. This should be handled by mspdbsrv, but rarely errors out with + # c1xx : fatal error C1033: cannot open program database + # By making the rules target separate pdb files this might be avoided. + cc_command = ( + "ninja -t msvc -e $arch " + "-- " + "$cc /nologo /showIncludes /FC " + "@$out.rsp /c $in /Fo$out /Fd$pdbname_c " + ) + cxx_command = ( + "ninja -t msvc -e $arch " + "-- " + "$cxx /nologo /showIncludes /FC " + "@$out.rsp /c $in /Fo$out /Fd$pdbname_cc " + ) + master_ninja.rule( + "cc", + description="CC $out", + command=cc_command, + rspfile="$out.rsp", + rspfile_content="$defines $includes $cflags $cflags_c", + deps=deps, + ) + master_ninja.rule( + "cxx", + description="CXX $out", + command=cxx_command, + rspfile="$out.rsp", + rspfile_content="$defines $includes $cflags $cflags_cc", + deps=deps, + ) + master_ninja.rule( + "idl", + description="IDL $in", + command=( + "%s gyp-win-tool midl-wrapper $arch $outdir " + "$tlb $h $dlldata $iid $proxy $in " + "$midl_includes $idlflags" % sys.executable + ), + ) + master_ninja.rule( + "rc", + description="RC $in", + # Note: $in must be last otherwise rc.exe complains. + command=( + "%s gyp-win-tool rc-wrapper " + "$arch $rc $defines $resource_includes $rcflags /fo$out $in" + % sys.executable + ), + ) + master_ninja.rule( + "asm", + description="ASM $out", + command=( + "%s gyp-win-tool asm-wrapper " + "$arch $asm $defines $includes $asmflags /c /Fo $out $in" + % sys.executable + ), + ) + + if flavor != "mac" and flavor != "win": + master_ninja.rule( + "alink", + description="AR $out", + command="rm -f $out && $ar rcs $arflags $out $in", + ) + master_ninja.rule( + "alink_thin", + description="AR $out", + command="rm -f $out && $ar rcsT $arflags $out $in", + ) + + # This allows targets that only need to depend on $lib's API to declare an + # order-only dependency on $lib.TOC and avoid relinking such downstream + # dependencies when $lib changes only in non-public ways. + # The resulting string leaves an uninterpolated %{suffix} which + # is used in the final substitution below. + mtime_preserving_solink_base = ( + "if [ ! -e $lib -o ! -e $lib.TOC ]; then " + "%(solink)s && %(extract_toc)s > $lib.TOC; else " + "%(solink)s && %(extract_toc)s > $lib.tmp && " + "if ! cmp -s $lib.tmp $lib.TOC; then mv $lib.tmp $lib.TOC ; " + "fi; fi" + % { + "solink": "$ld -shared $ldflags -o $lib -Wl,-soname=$soname %(suffix)s", + "extract_toc": ( + "{ $readelf -d $lib | grep SONAME ; " + "$nm -gD -f p $lib | cut -f1-2 -d' '; }" + ), + } + ) + + master_ninja.rule( + "solink", + description="SOLINK $lib", + restat=True, + command=mtime_preserving_solink_base % {"suffix": "@$link_file_list"}, + rspfile="$link_file_list", + rspfile_content="-Wl,--whole-archive $in $solibs -Wl,--no-whole-archive $libs", + pool="link_pool", + ) + master_ninja.rule( + "solink_module", + description="SOLINK(module) $lib", + restat=True, + command=mtime_preserving_solink_base % {"suffix": "@$link_file_list"}, + rspfile="$link_file_list", + rspfile_content="-Wl,--start-group $in $solibs $libs -Wl,--end-group", + pool="link_pool", + ) + master_ninja.rule( + "link", + description="LINK $out", + command=( + "$ld $ldflags -o $out " + "-Wl,--start-group $in $solibs $libs -Wl,--end-group" + ), + pool="link_pool", + ) + elif flavor == "win": + master_ninja.rule( + "alink", + description="LIB $out", + command=( + "%s gyp-win-tool link-wrapper $arch False " + "$ar /nologo /ignore:4221 /OUT:$out @$out.rsp" % sys.executable + ), + rspfile="$out.rsp", + rspfile_content="$in_newline $libflags", + ) + _AddWinLinkRules(master_ninja, embed_manifest=True) + _AddWinLinkRules(master_ninja, embed_manifest=False) + else: + master_ninja.rule( + "objc", + description="OBJC $out", + command=( + "$cc -MMD -MF $out.d $defines $includes $cflags $cflags_objc " + "$cflags_pch_objc -c $in -o $out" + ), + depfile="$out.d", + deps=deps, + ) + master_ninja.rule( + "objcxx", + description="OBJCXX $out", + command=( + "$cxx -MMD -MF $out.d $defines $includes $cflags $cflags_objcc " + "$cflags_pch_objcc -c $in -o $out" + ), + depfile="$out.d", + deps=deps, + ) + master_ninja.rule( + "alink", + description="LIBTOOL-STATIC $out, POSTBUILDS", + command="rm -f $out && " + "./gyp-mac-tool filter-libtool libtool $libtool_flags " + "-static -o $out $in" + "$postbuilds", + ) + master_ninja.rule( + "lipo", + description="LIPO $out, POSTBUILDS", + command="rm -f $out && lipo -create $in -output $out$postbuilds", + ) + master_ninja.rule( + "solipo", + description="SOLIPO $out, POSTBUILDS", + command=( + "rm -f $lib $lib.TOC && lipo -create $in -output $lib$postbuilds &&" + "%(extract_toc)s > $lib.TOC" + % { + "extract_toc": "{ otool -l $lib | grep LC_ID_DYLIB -A 5; " + "nm -gP $lib | cut -f1-2 -d' ' | grep -v U$$; true; }" + } + ), + ) + + # Record the public interface of $lib in $lib.TOC. See the corresponding + # comment in the posix section above for details. + solink_base = "$ld %(type)s $ldflags -o $lib %(suffix)s" + mtime_preserving_solink_base = ( + "if [ ! -e $lib -o ! -e $lib.TOC ] || " + # Always force dependent targets to relink if this library + # reexports something. Handling this correctly would require + # recursive TOC dumping but this is rare in practice, so punt. + "otool -l $lib | grep -q LC_REEXPORT_DYLIB ; then " + "%(solink)s && %(extract_toc)s > $lib.TOC; " + "else " + "%(solink)s && %(extract_toc)s > $lib.tmp && " + "if ! cmp -s $lib.tmp $lib.TOC; then " + "mv $lib.tmp $lib.TOC ; " + "fi; " + "fi" + % { + "solink": solink_base, + "extract_toc": "{ otool -l $lib | grep LC_ID_DYLIB -A 5; " + "nm -gP $lib | cut -f1-2 -d' ' | grep -v U$$; true; }", + } + ) + + solink_suffix = "@$link_file_list$postbuilds" + master_ninja.rule( + "solink", + description="SOLINK $lib, POSTBUILDS", + restat=True, + command=mtime_preserving_solink_base + % {"suffix": solink_suffix, "type": "-shared"}, + rspfile="$link_file_list", + rspfile_content="$in $solibs $libs", + pool="link_pool", + ) + master_ninja.rule( + "solink_notoc", + description="SOLINK $lib, POSTBUILDS", + restat=True, + command=solink_base % {"suffix": solink_suffix, "type": "-shared"}, + rspfile="$link_file_list", + rspfile_content="$in $solibs $libs", + pool="link_pool", + ) + + master_ninja.rule( + "solink_module", + description="SOLINK(module) $lib, POSTBUILDS", + restat=True, + command=mtime_preserving_solink_base + % {"suffix": solink_suffix, "type": "-bundle"}, + rspfile="$link_file_list", + rspfile_content="$in $solibs $libs", + pool="link_pool", + ) + master_ninja.rule( + "solink_module_notoc", + description="SOLINK(module) $lib, POSTBUILDS", + restat=True, + command=solink_base % {"suffix": solink_suffix, "type": "-bundle"}, + rspfile="$link_file_list", + rspfile_content="$in $solibs $libs", + pool="link_pool", + ) + + master_ninja.rule( + "link", + description="LINK $out, POSTBUILDS", + command=("$ld $ldflags -o $out " "$in $solibs $libs$postbuilds"), + pool="link_pool", + ) + master_ninja.rule( + "preprocess_infoplist", + description="PREPROCESS INFOPLIST $out", + command=( + "$cc -E -P -Wno-trigraphs -x c $defines $in -o $out && " + "plutil -convert xml1 $out $out" + ), + ) + master_ninja.rule( + "copy_infoplist", + description="COPY INFOPLIST $in", + command="$env ./gyp-mac-tool copy-info-plist $in $out $binary $keys", + ) + master_ninja.rule( + "merge_infoplist", + description="MERGE INFOPLISTS $in", + command="$env ./gyp-mac-tool merge-info-plist $out $in", + ) + master_ninja.rule( + "compile_xcassets", + description="COMPILE XCASSETS $in", + command="$env ./gyp-mac-tool compile-xcassets $keys $in", + ) + master_ninja.rule( + "compile_ios_framework_headers", + description="COMPILE HEADER MAPS AND COPY FRAMEWORK HEADERS $in", + command="$env ./gyp-mac-tool compile-ios-framework-header-map $out " + "$framework $in && $env ./gyp-mac-tool " + "copy-ios-framework-headers $framework $copy_headers", + ) + master_ninja.rule( + "mac_tool", + description="MACTOOL $mactool_cmd $in", + command="$env ./gyp-mac-tool $mactool_cmd $in $out $binary", + ) + master_ninja.rule( + "package_framework", + description="PACKAGE FRAMEWORK $out, POSTBUILDS", + command="./gyp-mac-tool package-framework $out $version$postbuilds " + "&& touch $out", + ) + master_ninja.rule( + "package_ios_framework", + description="PACKAGE IOS FRAMEWORK $out, POSTBUILDS", + command="./gyp-mac-tool package-ios-framework $out $postbuilds " + "&& touch $out", + ) + if flavor == "win": + master_ninja.rule( + "stamp", + description="STAMP $out", + command="%s gyp-win-tool stamp $out" % sys.executable, + ) + else: + master_ninja.rule( + "stamp", description="STAMP $out", command="${postbuilds}touch $out" + ) + if flavor == "win": + master_ninja.rule( + "copy", + description="COPY $in $out", + command="%s gyp-win-tool recursive-mirror $in $out" % sys.executable, + ) + elif flavor == "zos": + master_ninja.rule( + "copy", + description="COPY $in $out", + command="rm -rf $out && cp -fRP $in $out", + ) + else: + master_ninja.rule( + "copy", + description="COPY $in $out", + command="ln -f $in $out 2>/dev/null || (rm -rf $out && cp -af $in $out)", + ) master_ninja.newline() - master_ninja.build('all', 'phony', sorted(all_outputs)) - master_ninja.default(generator_flags.get('default_target', 'all')) - master_ninja_file.close() + all_targets = set() + for build_file in params["build_files"]: + for target in gyp.common.AllTargets( + target_list, target_dicts, os.path.normpath(build_file) + ): + all_targets.add(target) + all_outputs = set() + + # target_outputs is a map from qualified target name to a Target object. + target_outputs = {} + # target_short_names is a map from target short name to a list of Target + # objects. + target_short_names = {} + + # short name of targets that were skipped because they didn't contain anything + # interesting. + # NOTE: there may be overlap between this an non_empty_target_names. + empty_target_names = set() + + # Set of non-empty short target names. + # NOTE: there may be overlap between this an empty_target_names. + non_empty_target_names = set() + + for qualified_target in target_list: + # qualified_target is like: third_party/icu/icu.gyp:icui18n#target + build_file, name, toolset = gyp.common.ParseQualifiedTarget(qualified_target) + + this_make_global_settings = data[build_file].get("make_global_settings", []) + assert make_global_settings == this_make_global_settings, ( + "make_global_settings needs to be the same for all targets. %s vs. %s" + % (this_make_global_settings, make_global_settings) + ) + + spec = target_dicts[qualified_target] + if flavor == "mac": + gyp.xcode_emulation.MergeGlobalXcodeSettingsToSpec(data[build_file], spec) + + # If build_file is a symlink, we must not follow it because there's a chance + # it could point to a path above toplevel_dir, and we cannot correctly deal + # with that case at the moment. + build_file = gyp.common.RelativePath(build_file, options.toplevel_dir, False) + + qualified_target_for_hash = gyp.common.QualifiedTarget( + build_file, name, toolset + ) + qualified_target_for_hash = qualified_target_for_hash.encode("utf-8") + hash_for_rules = hashlib.md5(qualified_target_for_hash).hexdigest() + + base_path = os.path.dirname(build_file) + obj = "obj" + if toolset != "target": + obj += "." + toolset + output_file = os.path.join(obj, base_path, name + ".ninja") + + ninja_output = StringIO() + writer = NinjaWriter( + hash_for_rules, + target_outputs, + base_path, + build_dir, + ninja_output, + toplevel_build, + output_file, + flavor, + toplevel_dir=options.toplevel_dir, + ) + + target = writer.WriteSpec(spec, config_name, generator_flags) + + if ninja_output.tell() > 0: + # Only create files for ninja files that actually have contents. + with OpenOutput(os.path.join(toplevel_build, output_file)) as ninja_file: + ninja_file.write(ninja_output.getvalue()) + ninja_output.close() + master_ninja.subninja(output_file) + + if target: + if name != target.FinalOutput() and spec["toolset"] == "target": + target_short_names.setdefault(name, []).append(target) + target_outputs[qualified_target] = target + if qualified_target in all_targets: + all_outputs.add(target.FinalOutput()) + non_empty_target_names.add(name) + else: + empty_target_names.add(name) + + if target_short_names: + # Write a short name to build this target. This benefits both the + # "build chrome" case as well as the gyp tests, which expect to be + # able to run actions and build libraries by their short name. + master_ninja.newline() + master_ninja.comment("Short names for targets.") + for short_name in sorted(target_short_names): + master_ninja.build( + short_name, + "phony", + [x.FinalOutput() for x in target_short_names[short_name]], + ) + + # Write phony targets for any empty targets that weren't written yet. As + # short names are not necessarily unique only do this for short names that + # haven't already been output for another target. + empty_target_names = empty_target_names - non_empty_target_names + if empty_target_names: + master_ninja.newline() + master_ninja.comment("Empty targets (output for completeness).") + for name in sorted(empty_target_names): + master_ninja.build(name, "phony") + + if all_outputs: + master_ninja.newline() + master_ninja.build("all", "phony", sorted(all_outputs)) + master_ninja.default(generator_flags.get("default_target", "all")) + + master_ninja_file.close() def PerformBuild(data, configurations, params): - options = params['options'] - for config in configurations: - builddir = os.path.join(options.toplevel_dir, 'out', config) - arguments = ['ninja', '-C', builddir] - print('Building [%s]: %s' % (config, arguments)) - subprocess.check_call(arguments) + options = params["options"] + for config in configurations: + builddir = os.path.join(options.toplevel_dir, "out", config) + arguments = ["ninja", "-C", builddir] + print("Building [%s]: %s" % (config, arguments)) + subprocess.check_call(arguments) def CallGenerateOutputForConfig(arglist): - # Ignore the interrupt signal so that the parent process catches it and - # kills all multiprocessing children. - signal.signal(signal.SIGINT, signal.SIG_IGN) + # Ignore the interrupt signal so that the parent process catches it and + # kills all multiprocessing children. + signal.signal(signal.SIGINT, signal.SIG_IGN) - (target_list, target_dicts, data, params, config_name) = arglist - GenerateOutputForConfig(target_list, target_dicts, data, params, config_name) + (target_list, target_dicts, data, params, config_name) = arglist + GenerateOutputForConfig(target_list, target_dicts, data, params, config_name) def GenerateOutput(target_list, target_dicts, data, params): - # Update target_dicts for iOS device builds. - target_dicts = gyp.xcode_emulation.CloneConfigurationForDeviceAndEmulator( - target_dicts) - - user_config = params.get('generator_flags', {}).get('config', None) - if gyp.common.GetFlavor(params) == 'win': - target_list, target_dicts = MSVSUtil.ShardTargets(target_list, target_dicts) - target_list, target_dicts = MSVSUtil.InsertLargePdbShims( - target_list, target_dicts, generator_default_variables) - - if user_config: - GenerateOutputForConfig(target_list, target_dicts, data, params, - user_config) - else: - config_names = target_dicts[target_list[0]]['configurations'].keys() - if params['parallel']: - try: - pool = multiprocessing.Pool(len(config_names)) - arglists = [] - for config_name in config_names: - arglists.append( - (target_list, target_dicts, data, params, config_name)) - pool.map(CallGenerateOutputForConfig, arglists) - except KeyboardInterrupt as e: - pool.terminate() - raise e + # Update target_dicts for iOS device builds. + target_dicts = gyp.xcode_emulation.CloneConfigurationForDeviceAndEmulator( + target_dicts + ) + + user_config = params.get("generator_flags", {}).get("config", None) + if gyp.common.GetFlavor(params) == "win": + target_list, target_dicts = MSVSUtil.ShardTargets(target_list, target_dicts) + target_list, target_dicts = MSVSUtil.InsertLargePdbShims( + target_list, target_dicts, generator_default_variables + ) + + if user_config: + GenerateOutputForConfig(target_list, target_dicts, data, params, user_config) else: - for config_name in config_names: - GenerateOutputForConfig(target_list, target_dicts, data, params, - config_name) + config_names = target_dicts[target_list[0]]["configurations"] + if params["parallel"]: + try: + pool = multiprocessing.Pool(len(config_names)) + arglists = [] + for config_name in config_names: + arglists.append( + (target_list, target_dicts, data, params, config_name) + ) + pool.map(CallGenerateOutputForConfig, arglists) + except KeyboardInterrupt as e: + pool.terminate() + raise e + else: + for config_name in config_names: + GenerateOutputForConfig( + target_list, target_dicts, data, params, config_name + ) diff --git a/gyp/pylib/gyp/generator/ninja_test.py b/gyp/pylib/gyp/generator/ninja_test.py index c8adc251c9..abadcd9828 100644 --- a/gyp/pylib/gyp/generator/ninja_test.py +++ b/gyp/pylib/gyp/generator/ninja_test.py @@ -13,34 +13,43 @@ class TestPrefixesAndSuffixes(unittest.TestCase): - def test_BinaryNamesWindows(self): - # These cannot run on non-Windows as they require a VS installation to - # correctly handle variable expansion. - if sys.platform.startswith('win'): - writer = ninja.NinjaWriter('foo', 'wee', '.', '.', 'build.ninja', '.', - 'build.ninja', 'win') - spec = { 'target_name': 'wee' } - self.assertTrue(writer.ComputeOutputFileName(spec, 'executable'). - endswith('.exe')) - self.assertTrue(writer.ComputeOutputFileName(spec, 'shared_library'). - endswith('.dll')) - self.assertTrue(writer.ComputeOutputFileName(spec, 'static_library'). - endswith('.lib')) - - def test_BinaryNamesLinux(self): - writer = ninja.NinjaWriter('foo', 'wee', '.', '.', 'build.ninja', '.', - 'build.ninja', 'linux') - spec = { 'target_name': 'wee' } - self.assertTrue('.' not in writer.ComputeOutputFileName(spec, - 'executable')) - self.assertTrue(writer.ComputeOutputFileName(spec, 'shared_library'). - startswith('lib')) - self.assertTrue(writer.ComputeOutputFileName(spec, 'static_library'). - startswith('lib')) - self.assertTrue(writer.ComputeOutputFileName(spec, 'shared_library'). - endswith('.so')) - self.assertTrue(writer.ComputeOutputFileName(spec, 'static_library'). - endswith('.a')) - -if __name__ == '__main__': - unittest.main() + def test_BinaryNamesWindows(self): + # These cannot run on non-Windows as they require a VS installation to + # correctly handle variable expansion. + if sys.platform.startswith("win"): + writer = ninja.NinjaWriter( + "foo", "wee", ".", ".", "build.ninja", ".", "build.ninja", "win" + ) + spec = {"target_name": "wee"} + self.assertTrue( + writer.ComputeOutputFileName(spec, "executable").endswith(".exe") + ) + self.assertTrue( + writer.ComputeOutputFileName(spec, "shared_library").endswith(".dll") + ) + self.assertTrue( + writer.ComputeOutputFileName(spec, "static_library").endswith(".lib") + ) + + def test_BinaryNamesLinux(self): + writer = ninja.NinjaWriter( + "foo", "wee", ".", ".", "build.ninja", ".", "build.ninja", "linux" + ) + spec = {"target_name": "wee"} + self.assertTrue("." not in writer.ComputeOutputFileName(spec, "executable")) + self.assertTrue( + writer.ComputeOutputFileName(spec, "shared_library").startswith("lib") + ) + self.assertTrue( + writer.ComputeOutputFileName(spec, "static_library").startswith("lib") + ) + self.assertTrue( + writer.ComputeOutputFileName(spec, "shared_library").endswith(".so") + ) + self.assertTrue( + writer.ComputeOutputFileName(spec, "static_library").endswith(".a") + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/gyp/pylib/gyp/generator/xcode.py b/gyp/pylib/gyp/generator/xcode.py index 4917ba77b9..535c5400ce 100644 --- a/gyp/pylib/gyp/generator/xcode.py +++ b/gyp/pylib/gyp/generator/xcode.py @@ -27,511 +27,543 @@ # PROJECT_DERIVED_FILE_DIR, is unsuitable because while it is project-specific, # it is not configuration-specific. INTERMEDIATE_DIR is defined as # $(PROJECT_DERIVED_FILE_DIR)/$(CONFIGURATION). -_intermediate_var = 'INTERMEDIATE_DIR' +_intermediate_var = "INTERMEDIATE_DIR" # SHARED_INTERMEDIATE_DIR is the same, except that it is shared among all # targets that share the same BUILT_PRODUCTS_DIR. -_shared_intermediate_var = 'SHARED_INTERMEDIATE_DIR' +_shared_intermediate_var = "SHARED_INTERMEDIATE_DIR" -_library_search_paths_var = 'LIBRARY_SEARCH_PATHS' +_library_search_paths_var = "LIBRARY_SEARCH_PATHS" generator_default_variables = { - 'EXECUTABLE_PREFIX': '', - 'EXECUTABLE_SUFFIX': '', - 'STATIC_LIB_PREFIX': 'lib', - 'SHARED_LIB_PREFIX': 'lib', - 'STATIC_LIB_SUFFIX': '.a', - 'SHARED_LIB_SUFFIX': '.dylib', - # INTERMEDIATE_DIR is a place for targets to build up intermediate products. - # It is specific to each build environment. It is only guaranteed to exist - # and be constant within the context of a project, corresponding to a single - # input file. Some build environments may allow their intermediate directory - # to be shared on a wider scale, but this is not guaranteed. - 'INTERMEDIATE_DIR': '$(%s)' % _intermediate_var, - 'OS': 'mac', - 'PRODUCT_DIR': '$(BUILT_PRODUCTS_DIR)', - 'LIB_DIR': '$(BUILT_PRODUCTS_DIR)', - 'RULE_INPUT_ROOT': '$(INPUT_FILE_BASE)', - 'RULE_INPUT_EXT': '$(INPUT_FILE_SUFFIX)', - 'RULE_INPUT_NAME': '$(INPUT_FILE_NAME)', - 'RULE_INPUT_PATH': '$(INPUT_FILE_PATH)', - 'RULE_INPUT_DIRNAME': '$(INPUT_FILE_DIRNAME)', - 'SHARED_INTERMEDIATE_DIR': '$(%s)' % _shared_intermediate_var, - 'CONFIGURATION_NAME': '$(CONFIGURATION)', + "EXECUTABLE_PREFIX": "", + "EXECUTABLE_SUFFIX": "", + "STATIC_LIB_PREFIX": "lib", + "SHARED_LIB_PREFIX": "lib", + "STATIC_LIB_SUFFIX": ".a", + "SHARED_LIB_SUFFIX": ".dylib", + # INTERMEDIATE_DIR is a place for targets to build up intermediate products. + # It is specific to each build environment. It is only guaranteed to exist + # and be constant within the context of a project, corresponding to a single + # input file. Some build environments may allow their intermediate directory + # to be shared on a wider scale, but this is not guaranteed. + "INTERMEDIATE_DIR": "$(%s)" % _intermediate_var, + "OS": "mac", + "PRODUCT_DIR": "$(BUILT_PRODUCTS_DIR)", + "LIB_DIR": "$(BUILT_PRODUCTS_DIR)", + "RULE_INPUT_ROOT": "$(INPUT_FILE_BASE)", + "RULE_INPUT_EXT": "$(INPUT_FILE_SUFFIX)", + "RULE_INPUT_NAME": "$(INPUT_FILE_NAME)", + "RULE_INPUT_PATH": "$(INPUT_FILE_PATH)", + "RULE_INPUT_DIRNAME": "$(INPUT_FILE_DIRNAME)", + "SHARED_INTERMEDIATE_DIR": "$(%s)" % _shared_intermediate_var, + "CONFIGURATION_NAME": "$(CONFIGURATION)", } # The Xcode-specific sections that hold paths. generator_additional_path_sections = [ - 'mac_bundle_resources', - 'mac_framework_headers', - 'mac_framework_private_headers', - # 'mac_framework_dirs', input already handles _dirs endings. + "mac_bundle_resources", + "mac_framework_headers", + "mac_framework_private_headers", + # 'mac_framework_dirs', input already handles _dirs endings. ] # The Xcode-specific keys that exist on targets and aren't moved down to # configurations. generator_additional_non_configuration_keys = [ - 'ios_app_extension', - 'ios_watch_app', - 'ios_watchkit_extension', - 'mac_bundle', - 'mac_bundle_resources', - 'mac_framework_headers', - 'mac_framework_private_headers', - 'mac_xctest_bundle', - 'mac_xcuitest_bundle', - 'xcode_create_dependents_test_runner', + "ios_app_extension", + "ios_watch_app", + "ios_watchkit_extension", + "mac_bundle", + "mac_bundle_resources", + "mac_framework_headers", + "mac_framework_private_headers", + "mac_xctest_bundle", + "mac_xcuitest_bundle", + "xcode_create_dependents_test_runner", ] # We want to let any rules apply to files that are resources also. generator_extra_sources_for_rules = [ - 'mac_bundle_resources', - 'mac_framework_headers', - 'mac_framework_private_headers', + "mac_bundle_resources", + "mac_framework_headers", + "mac_framework_private_headers", ] generator_filelist_paths = None # Xcode's standard set of library directories, which don't need to be duplicated # in LIBRARY_SEARCH_PATHS. This list is not exhaustive, but that's okay. -xcode_standard_library_dirs = frozenset([ - '$(SDKROOT)/usr/lib', - '$(SDKROOT)/usr/local/lib', -]) +xcode_standard_library_dirs = frozenset( + ["$(SDKROOT)/usr/lib", "$(SDKROOT)/usr/local/lib"] +) + def CreateXCConfigurationList(configuration_names): - xccl = gyp.xcodeproj_file.XCConfigurationList({'buildConfigurations': []}) - if len(configuration_names) == 0: - configuration_names = ['Default'] - for configuration_name in configuration_names: - xcbc = gyp.xcodeproj_file.XCBuildConfiguration({ - 'name': configuration_name}) - xccl.AppendProperty('buildConfigurations', xcbc) - xccl.SetProperty('defaultConfigurationName', configuration_names[0]) - return xccl + xccl = gyp.xcodeproj_file.XCConfigurationList({"buildConfigurations": []}) + if len(configuration_names) == 0: + configuration_names = ["Default"] + for configuration_name in configuration_names: + xcbc = gyp.xcodeproj_file.XCBuildConfiguration({"name": configuration_name}) + xccl.AppendProperty("buildConfigurations", xcbc) + xccl.SetProperty("defaultConfigurationName", configuration_names[0]) + return xccl class XcodeProject(object): - def __init__(self, gyp_path, path, build_file_dict): - self.gyp_path = gyp_path - self.path = path - self.project = gyp.xcodeproj_file.PBXProject(path=path) - projectDirPath = gyp.common.RelativePath( - os.path.dirname(os.path.abspath(self.gyp_path)), - os.path.dirname(path) or '.') - self.project.SetProperty('projectDirPath', projectDirPath) - self.project_file = \ - gyp.xcodeproj_file.XCProjectFile({'rootObject': self.project}) - self.build_file_dict = build_file_dict - - # TODO(mark): add destructor that cleans up self.path if created_dir is - # True and things didn't complete successfully. Or do something even - # better with "try"? - self.created_dir = False - try: - os.makedirs(self.path) - self.created_dir = True - except OSError as e: - if e.errno != errno.EEXIST: - raise - - def Finalize1(self, xcode_targets, serialize_all_tests): - # Collect a list of all of the build configuration names used by the - # various targets in the file. It is very heavily advised to keep each - # target in an entire project (even across multiple project files) using - # the same set of configuration names. - configurations = [] - for xct in self.project.GetProperty('targets'): - xccl = xct.GetProperty('buildConfigurationList') - xcbcs = xccl.GetProperty('buildConfigurations') - for xcbc in xcbcs: - name = xcbc.GetProperty('name') - if name not in configurations: - configurations.append(name) - - # Replace the XCConfigurationList attached to the PBXProject object with - # a new one specifying all of the configuration names used by the various - # targets. - try: - xccl = CreateXCConfigurationList(configurations) - self.project.SetProperty('buildConfigurationList', xccl) - except: - sys.stderr.write("Problem with gyp file %s\n" % self.gyp_path) - raise - - # The need for this setting is explained above where _intermediate_var is - # defined. The comments below about wanting to avoid project-wide build - # settings apply here too, but this needs to be set on a project-wide basis - # so that files relative to the _intermediate_var setting can be displayed - # properly in the Xcode UI. - # - # Note that for configuration-relative files such as anything relative to - # _intermediate_var, for the purposes of UI tree view display, Xcode will - # only resolve the configuration name once, when the project file is - # opened. If the active build configuration is changed, the project file - # must be closed and reopened if it is desired for the tree view to update. - # This is filed as Apple radar 6588391. - xccl.SetBuildSetting(_intermediate_var, - '$(PROJECT_DERIVED_FILE_DIR)/$(CONFIGURATION)') - xccl.SetBuildSetting(_shared_intermediate_var, - '$(SYMROOT)/DerivedSources/$(CONFIGURATION)') - - # Set user-specified project-wide build settings and config files. This - # is intended to be used very sparingly. Really, almost everything should - # go into target-specific build settings sections. The project-wide - # settings are only intended to be used in cases where Xcode attempts to - # resolve variable references in a project context as opposed to a target - # context, such as when resolving sourceTree references while building up - # the tree tree view for UI display. - # Any values set globally are applied to all configurations, then any - # per-configuration values are applied. - for xck, xcv in self.build_file_dict.get('xcode_settings', {}).items(): - xccl.SetBuildSetting(xck, xcv) - if 'xcode_config_file' in self.build_file_dict: - config_ref = self.project.AddOrGetFileInRootGroup( - self.build_file_dict['xcode_config_file']) - xccl.SetBaseConfiguration(config_ref) - build_file_configurations = self.build_file_dict.get('configurations', {}) - if build_file_configurations: - for config_name in configurations: - build_file_configuration_named = \ - build_file_configurations.get(config_name, {}) - if build_file_configuration_named: - xcc = xccl.ConfigurationNamed(config_name) - for xck, xcv in build_file_configuration_named.get('xcode_settings', - {}).items(): - xcc.SetBuildSetting(xck, xcv) - if 'xcode_config_file' in build_file_configuration_named: + def __init__(self, gyp_path, path, build_file_dict): + self.gyp_path = gyp_path + self.path = path + self.project = gyp.xcodeproj_file.PBXProject(path=path) + projectDirPath = gyp.common.RelativePath( + os.path.dirname(os.path.abspath(self.gyp_path)), + os.path.dirname(path) or ".", + ) + self.project.SetProperty("projectDirPath", projectDirPath) + self.project_file = gyp.xcodeproj_file.XCProjectFile( + {"rootObject": self.project} + ) + self.build_file_dict = build_file_dict + + # TODO(mark): add destructor that cleans up self.path if created_dir is + # True and things didn't complete successfully. Or do something even + # better with "try"? + self.created_dir = False + try: + os.makedirs(self.path) + self.created_dir = True + except OSError as e: + if e.errno != errno.EEXIST: + raise + + def Finalize1(self, xcode_targets, serialize_all_tests): + # Collect a list of all of the build configuration names used by the + # various targets in the file. It is very heavily advised to keep each + # target in an entire project (even across multiple project files) using + # the same set of configuration names. + configurations = [] + for xct in self.project.GetProperty("targets"): + xccl = xct.GetProperty("buildConfigurationList") + xcbcs = xccl.GetProperty("buildConfigurations") + for xcbc in xcbcs: + name = xcbc.GetProperty("name") + if name not in configurations: + configurations.append(name) + + # Replace the XCConfigurationList attached to the PBXProject object with + # a new one specifying all of the configuration names used by the various + # targets. + try: + xccl = CreateXCConfigurationList(configurations) + self.project.SetProperty("buildConfigurationList", xccl) + except Exception: + sys.stderr.write("Problem with gyp file %s\n" % self.gyp_path) + raise + + # The need for this setting is explained above where _intermediate_var is + # defined. The comments below about wanting to avoid project-wide build + # settings apply here too, but this needs to be set on a project-wide basis + # so that files relative to the _intermediate_var setting can be displayed + # properly in the Xcode UI. + # + # Note that for configuration-relative files such as anything relative to + # _intermediate_var, for the purposes of UI tree view display, Xcode will + # only resolve the configuration name once, when the project file is + # opened. If the active build configuration is changed, the project file + # must be closed and reopened if it is desired for the tree view to update. + # This is filed as Apple radar 6588391. + xccl.SetBuildSetting( + _intermediate_var, "$(PROJECT_DERIVED_FILE_DIR)/$(CONFIGURATION)" + ) + xccl.SetBuildSetting( + _shared_intermediate_var, "$(SYMROOT)/DerivedSources/$(CONFIGURATION)" + ) + + # Set user-specified project-wide build settings and config files. This + # is intended to be used very sparingly. Really, almost everything should + # go into target-specific build settings sections. The project-wide + # settings are only intended to be used in cases where Xcode attempts to + # resolve variable references in a project context as opposed to a target + # context, such as when resolving sourceTree references while building up + # the tree tree view for UI display. + # Any values set globally are applied to all configurations, then any + # per-configuration values are applied. + for xck, xcv in self.build_file_dict.get("xcode_settings", {}).items(): + xccl.SetBuildSetting(xck, xcv) + if "xcode_config_file" in self.build_file_dict: config_ref = self.project.AddOrGetFileInRootGroup( - build_file_configurations[config_name]['xcode_config_file']) - xcc.SetBaseConfiguration(config_ref) - - # Sort the targets based on how they appeared in the input. - # TODO(mark): Like a lot of other things here, this assumes internal - # knowledge of PBXProject - in this case, of its "targets" property. - - # ordinary_targets are ordinary targets that are already in the project - # file. run_test_targets are the targets that run unittests and should be - # used for the Run All Tests target. support_targets are the action/rule - # targets used by GYP file targets, just kept for the assert check. - ordinary_targets = [] - run_test_targets = [] - support_targets = [] - - # targets is full list of targets in the project. - targets = [] - - # does the it define it's own "all"? - has_custom_all = False - - # targets_for_all is the list of ordinary_targets that should be listed - # in this project's "All" target. It includes each non_runtest_target - # that does not have suppress_wildcard set. - targets_for_all = [] - - for target in self.build_file_dict['targets']: - target_name = target['target_name'] - toolset = target['toolset'] - qualified_target = gyp.common.QualifiedTarget(self.gyp_path, target_name, - toolset) - xcode_target = xcode_targets[qualified_target] - # Make sure that the target being added to the sorted list is already in - # the unsorted list. - assert xcode_target in self.project._properties['targets'] - targets.append(xcode_target) - ordinary_targets.append(xcode_target) - if xcode_target.support_target: - support_targets.append(xcode_target.support_target) - targets.append(xcode_target.support_target) - - if not int(target.get('suppress_wildcard', False)): - targets_for_all.append(xcode_target) - - if target_name.lower() == 'all': - has_custom_all = True - - # If this target has a 'run_as' attribute, add its target to the - # targets, and add it to the test targets. - if target.get('run_as'): - # Make a target to run something. It should have one - # dependency, the parent xcode target. - xccl = CreateXCConfigurationList(configurations) - run_target = gyp.xcodeproj_file.PBXAggregateTarget({ - 'name': 'Run ' + target_name, - 'productName': xcode_target.GetProperty('productName'), - 'buildConfigurationList': xccl, - }, - parent=self.project) - run_target.AddDependency(xcode_target) - - command = target['run_as'] - script = '' - if command.get('working_directory'): - script = script + 'cd "%s"\n' % \ - gyp.xcodeproj_file.ConvertVariablesToShellSyntax( - command.get('working_directory')) - - if command.get('environment'): - script = script + "\n".join( - ['export %s="%s"' % - (key, gyp.xcodeproj_file.ConvertVariablesToShellSyntax(val)) - for (key, val) in command.get('environment').items()]) + "\n" - - # Some test end up using sockets, files on disk, etc. and can get - # confused if more then one test runs at a time. The generator - # flag 'xcode_serialize_all_test_runs' controls the forcing of all - # tests serially. It defaults to True. To get serial runs this - # little bit of python does the same as the linux flock utility to - # make sure only one runs at a time. - command_prefix = '' - if serialize_all_tests: - command_prefix = \ -"""python -c "import fcntl, subprocess, sys + self.build_file_dict["xcode_config_file"] + ) + xccl.SetBaseConfiguration(config_ref) + build_file_configurations = self.build_file_dict.get("configurations", {}) + if build_file_configurations: + for config_name in configurations: + build_file_configuration_named = build_file_configurations.get( + config_name, {} + ) + if build_file_configuration_named: + xcc = xccl.ConfigurationNamed(config_name) + for xck, xcv in build_file_configuration_named.get( + "xcode_settings", {} + ).items(): + xcc.SetBuildSetting(xck, xcv) + if "xcode_config_file" in build_file_configuration_named: + config_ref = self.project.AddOrGetFileInRootGroup( + build_file_configurations[config_name]["xcode_config_file"] + ) + xcc.SetBaseConfiguration(config_ref) + + # Sort the targets based on how they appeared in the input. + # TODO(mark): Like a lot of other things here, this assumes internal + # knowledge of PBXProject - in this case, of its "targets" property. + + # ordinary_targets are ordinary targets that are already in the project + # file. run_test_targets are the targets that run unittests and should be + # used for the Run All Tests target. support_targets are the action/rule + # targets used by GYP file targets, just kept for the assert check. + ordinary_targets = [] + run_test_targets = [] + support_targets = [] + + # targets is full list of targets in the project. + targets = [] + + # does the it define it's own "all"? + has_custom_all = False + + # targets_for_all is the list of ordinary_targets that should be listed + # in this project's "All" target. It includes each non_runtest_target + # that does not have suppress_wildcard set. + targets_for_all = [] + + for target in self.build_file_dict["targets"]: + target_name = target["target_name"] + toolset = target["toolset"] + qualified_target = gyp.common.QualifiedTarget( + self.gyp_path, target_name, toolset + ) + xcode_target = xcode_targets[qualified_target] + # Make sure that the target being added to the sorted list is already in + # the unsorted list. + assert xcode_target in self.project._properties["targets"] + targets.append(xcode_target) + ordinary_targets.append(xcode_target) + if xcode_target.support_target: + support_targets.append(xcode_target.support_target) + targets.append(xcode_target.support_target) + + if not int(target.get("suppress_wildcard", False)): + targets_for_all.append(xcode_target) + + if target_name.lower() == "all": + has_custom_all = True + + # If this target has a 'run_as' attribute, add its target to the + # targets, and add it to the test targets. + if target.get("run_as"): + # Make a target to run something. It should have one + # dependency, the parent xcode target. + xccl = CreateXCConfigurationList(configurations) + run_target = gyp.xcodeproj_file.PBXAggregateTarget( + { + "name": "Run " + target_name, + "productName": xcode_target.GetProperty("productName"), + "buildConfigurationList": xccl, + }, + parent=self.project, + ) + run_target.AddDependency(xcode_target) + + command = target["run_as"] + script = "" + if command.get("working_directory"): + script = ( + script + + 'cd "%s"\n' + % gyp.xcodeproj_file.ConvertVariablesToShellSyntax( + command.get("working_directory") + ) + ) + + if command.get("environment"): + script = ( + script + + "\n".join( + [ + 'export %s="%s"' + % ( + key, + gyp.xcodeproj_file.ConvertVariablesToShellSyntax( + val + ), + ) + for (key, val) in command.get("environment").items() + ] + ) + + "\n" + ) + + # Some test end up using sockets, files on disk, etc. and can get + # confused if more then one test runs at a time. The generator + # flag 'xcode_serialize_all_test_runs' controls the forcing of all + # tests serially. It defaults to True. To get serial runs this + # little bit of python does the same as the linux flock utility to + # make sure only one runs at a time. + command_prefix = "" + if serialize_all_tests: + command_prefix = """python -c "import fcntl, subprocess, sys file = open('$TMPDIR/GYP_serialize_test_runs', 'a') fcntl.flock(file.fileno(), fcntl.LOCK_EX) sys.exit(subprocess.call(sys.argv[1:]))" """ - # If we were unable to exec for some reason, we want to exit - # with an error, and fixup variable references to be shell - # syntax instead of xcode syntax. - script = script + 'exec ' + command_prefix + '%s\nexit 1\n' % \ - gyp.xcodeproj_file.ConvertVariablesToShellSyntax( - gyp.common.EncodePOSIXShellList(command.get('action'))) - - ssbp = gyp.xcodeproj_file.PBXShellScriptBuildPhase({ - 'shellScript': script, - 'showEnvVarsInLog': 0, - }) - run_target.AppendProperty('buildPhases', ssbp) - - # Add the run target to the project file. - targets.append(run_target) - run_test_targets.append(run_target) - xcode_target.test_runner = run_target - - - # Make sure that the list of targets being replaced is the same length as - # the one replacing it, but allow for the added test runner targets. - assert len(self.project._properties['targets']) == \ - len(ordinary_targets) + len(support_targets) - - self.project._properties['targets'] = targets - - # Get rid of unnecessary levels of depth in groups like the Source group. - self.project.RootGroupsTakeOverOnlyChildren(True) - - # Sort the groups nicely. Do this after sorting the targets, because the - # Products group is sorted based on the order of the targets. - self.project.SortGroups() - - # Create an "All" target if there's more than one target in this project - # file and the project didn't define its own "All" target. Put a generated - # "All" target first so that people opening up the project for the first - # time will build everything by default. - if len(targets_for_all) > 1 and not has_custom_all: - xccl = CreateXCConfigurationList(configurations) - all_target = gyp.xcodeproj_file.PBXAggregateTarget( - { - 'buildConfigurationList': xccl, - 'name': 'All', - }, - parent=self.project) - - for target in targets_for_all: - all_target.AddDependency(target) - - # TODO(mark): This is evil because it relies on internal knowledge of - # PBXProject._properties. It's important to get the "All" target first, - # though. - self.project._properties['targets'].insert(0, all_target) - - # The same, but for run_test_targets. - if len(run_test_targets) > 1: - xccl = CreateXCConfigurationList(configurations) - run_all_tests_target = gyp.xcodeproj_file.PBXAggregateTarget( - { - 'buildConfigurationList': xccl, - 'name': 'Run All Tests', - }, - parent=self.project) - for run_test_target in run_test_targets: - run_all_tests_target.AddDependency(run_test_target) - - # Insert after the "All" target, which must exist if there is more than - # one run_test_target. - self.project._properties['targets'].insert(1, run_all_tests_target) - - def Finalize2(self, xcode_targets, xcode_target_to_target_dict): - # Finalize2 needs to happen in a separate step because the process of - # updating references to other projects depends on the ordering of targets - # within remote project files. Finalize1 is responsible for sorting duty, - # and once all project files are sorted, Finalize2 can come in and update - # these references. - - # To support making a "test runner" target that will run all the tests - # that are direct dependents of any given target, we look for - # xcode_create_dependents_test_runner being set on an Aggregate target, - # and generate a second target that will run the tests runners found under - # the marked target. - for bf_tgt in self.build_file_dict['targets']: - if int(bf_tgt.get('xcode_create_dependents_test_runner', 0)): - tgt_name = bf_tgt['target_name'] - toolset = bf_tgt['toolset'] - qualified_target = gyp.common.QualifiedTarget(self.gyp_path, - tgt_name, toolset) - xcode_target = xcode_targets[qualified_target] - if isinstance(xcode_target, gyp.xcodeproj_file.PBXAggregateTarget): - # Collect all the run test targets. - all_run_tests = [] - pbxtds = xcode_target.GetProperty('dependencies') - for pbxtd in pbxtds: - pbxcip = pbxtd.GetProperty('targetProxy') - dependency_xct = pbxcip.GetProperty('remoteGlobalIDString') - if hasattr(dependency_xct, 'test_runner'): - all_run_tests.append(dependency_xct.test_runner) - - # Directly depend on all the runners as they depend on the target - # that builds them. - if len(all_run_tests) > 0: - run_all_target = gyp.xcodeproj_file.PBXAggregateTarget({ - 'name': 'Run %s Tests' % tgt_name, - 'productName': tgt_name, - }, - parent=self.project) - for run_test_target in all_run_tests: - run_all_target.AddDependency(run_test_target) - - # Insert the test runner after the related target. - idx = self.project._properties['targets'].index(xcode_target) - self.project._properties['targets'].insert(idx + 1, run_all_target) - - # Update all references to other projects, to make sure that the lists of - # remote products are complete. Otherwise, Xcode will fill them in when - # it opens the project file, which will result in unnecessary diffs. - # TODO(mark): This is evil because it relies on internal knowledge of - # PBXProject._other_pbxprojects. - for other_pbxproject in self.project._other_pbxprojects.keys(): - self.project.AddOrGetProjectReference(other_pbxproject) - - self.project.SortRemoteProductReferences() - - # Give everything an ID. - self.project_file.ComputeIDs() - - # Make sure that no two objects in the project file have the same ID. If - # multiple objects wind up with the same ID, upon loading the file, Xcode - # will only recognize one object (the last one in the file?) and the - # results are unpredictable. - self.project_file.EnsureNoIDCollisions() - - def Write(self): - # Write the project file to a temporary location first. Xcode watches for - # changes to the project file and presents a UI sheet offering to reload - # the project when it does change. However, in some cases, especially when - # multiple projects are open or when Xcode is busy, things don't work so - # seamlessly. Sometimes, Xcode is able to detect that a project file has - # changed but can't unload it because something else is referencing it. - # To mitigate this problem, and to avoid even having Xcode present the UI - # sheet when an open project is rewritten for inconsequential changes, the - # project file is written to a temporary file in the xcodeproj directory - # first. The new temporary file is then compared to the existing project - # file, if any. If they differ, the new file replaces the old; otherwise, - # the new project file is simply deleted. Xcode properly detects a file - # being renamed over an open project file as a change and so it remains - # able to present the "project file changed" sheet under this system. - # Writing to a temporary file first also avoids the possible problem of - # Xcode rereading an incomplete project file. - (output_fd, new_pbxproj_path) = \ - tempfile.mkstemp(suffix='.tmp', prefix='project.pbxproj.gyp.', - dir=self.path) - - try: - output_file = os.fdopen(output_fd, 'wb') - - self.project_file.Print(output_file) - output_file.close() - - pbxproj_path = os.path.join(self.path, 'project.pbxproj') - - same = False - try: - same = filecmp.cmp(pbxproj_path, new_pbxproj_path, False) - except OSError as e: - if e.errno != errno.ENOENT: - raise - - if same: - # The new file is identical to the old one, just get rid of the new - # one. - os.unlink(new_pbxproj_path) - else: - # The new file is different from the old one, or there is no old one. - # Rename the new file to the permanent name. - # - # tempfile.mkstemp uses an overly restrictive mode, resulting in a - # file that can only be read by the owner, regardless of the umask. - # There's no reason to not respect the umask here, which means that - # an extra hoop is required to fetch it and reset the new file's mode. - # - # No way to get the umask without setting a new one? Set a safe one - # and then set it back to the old value. - umask = os.umask(0o77) - os.umask(umask) - - os.chmod(new_pbxproj_path, 0o666 & ~umask) - os.rename(new_pbxproj_path, pbxproj_path) - - except Exception: - # Don't leave turds behind. In fact, if this code was responsible for - # creating the xcodeproj directory, get rid of that too. - os.unlink(new_pbxproj_path) - if self.created_dir: - shutil.rmtree(self.path, True) - raise + # If we were unable to exec for some reason, we want to exit + # with an error, and fixup variable references to be shell + # syntax instead of xcode syntax. + script = ( + script + + "exec " + + command_prefix + + "%s\nexit 1\n" + % gyp.xcodeproj_file.ConvertVariablesToShellSyntax( + gyp.common.EncodePOSIXShellList(command.get("action")) + ) + ) + + ssbp = gyp.xcodeproj_file.PBXShellScriptBuildPhase( + {"shellScript": script, "showEnvVarsInLog": 0} + ) + run_target.AppendProperty("buildPhases", ssbp) + + # Add the run target to the project file. + targets.append(run_target) + run_test_targets.append(run_target) + xcode_target.test_runner = run_target + + # Make sure that the list of targets being replaced is the same length as + # the one replacing it, but allow for the added test runner targets. + assert len(self.project._properties["targets"]) == len(ordinary_targets) + len( + support_targets + ) + + self.project._properties["targets"] = targets + + # Get rid of unnecessary levels of depth in groups like the Source group. + self.project.RootGroupsTakeOverOnlyChildren(True) + + # Sort the groups nicely. Do this after sorting the targets, because the + # Products group is sorted based on the order of the targets. + self.project.SortGroups() + + # Create an "All" target if there's more than one target in this project + # file and the project didn't define its own "All" target. Put a generated + # "All" target first so that people opening up the project for the first + # time will build everything by default. + if len(targets_for_all) > 1 and not has_custom_all: + xccl = CreateXCConfigurationList(configurations) + all_target = gyp.xcodeproj_file.PBXAggregateTarget( + {"buildConfigurationList": xccl, "name": "All"}, parent=self.project + ) + + for target in targets_for_all: + all_target.AddDependency(target) + + # TODO(mark): This is evil because it relies on internal knowledge of + # PBXProject._properties. It's important to get the "All" target first, + # though. + self.project._properties["targets"].insert(0, all_target) + + # The same, but for run_test_targets. + if len(run_test_targets) > 1: + xccl = CreateXCConfigurationList(configurations) + run_all_tests_target = gyp.xcodeproj_file.PBXAggregateTarget( + {"buildConfigurationList": xccl, "name": "Run All Tests"}, + parent=self.project, + ) + for run_test_target in run_test_targets: + run_all_tests_target.AddDependency(run_test_target) + + # Insert after the "All" target, which must exist if there is more than + # one run_test_target. + self.project._properties["targets"].insert(1, run_all_tests_target) + + def Finalize2(self, xcode_targets, xcode_target_to_target_dict): + # Finalize2 needs to happen in a separate step because the process of + # updating references to other projects depends on the ordering of targets + # within remote project files. Finalize1 is responsible for sorting duty, + # and once all project files are sorted, Finalize2 can come in and update + # these references. + + # To support making a "test runner" target that will run all the tests + # that are direct dependents of any given target, we look for + # xcode_create_dependents_test_runner being set on an Aggregate target, + # and generate a second target that will run the tests runners found under + # the marked target. + for bf_tgt in self.build_file_dict["targets"]: + if int(bf_tgt.get("xcode_create_dependents_test_runner", 0)): + tgt_name = bf_tgt["target_name"] + toolset = bf_tgt["toolset"] + qualified_target = gyp.common.QualifiedTarget( + self.gyp_path, tgt_name, toolset + ) + xcode_target = xcode_targets[qualified_target] + if isinstance(xcode_target, gyp.xcodeproj_file.PBXAggregateTarget): + # Collect all the run test targets. + all_run_tests = [] + pbxtds = xcode_target.GetProperty("dependencies") + for pbxtd in pbxtds: + pbxcip = pbxtd.GetProperty("targetProxy") + dependency_xct = pbxcip.GetProperty("remoteGlobalIDString") + if hasattr(dependency_xct, "test_runner"): + all_run_tests.append(dependency_xct.test_runner) + + # Directly depend on all the runners as they depend on the target + # that builds them. + if len(all_run_tests) > 0: + run_all_target = gyp.xcodeproj_file.PBXAggregateTarget( + { + "name": "Run %s Tests" % tgt_name, + "productName": tgt_name, + }, + parent=self.project, + ) + for run_test_target in all_run_tests: + run_all_target.AddDependency(run_test_target) + + # Insert the test runner after the related target. + idx = self.project._properties["targets"].index(xcode_target) + self.project._properties["targets"].insert( + idx + 1, run_all_target + ) + + # Update all references to other projects, to make sure that the lists of + # remote products are complete. Otherwise, Xcode will fill them in when + # it opens the project file, which will result in unnecessary diffs. + # TODO(mark): This is evil because it relies on internal knowledge of + # PBXProject._other_pbxprojects. + for other_pbxproject in self.project._other_pbxprojects.keys(): + self.project.AddOrGetProjectReference(other_pbxproject) + + self.project.SortRemoteProductReferences() + + # Give everything an ID. + self.project_file.ComputeIDs() + + # Make sure that no two objects in the project file have the same ID. If + # multiple objects wind up with the same ID, upon loading the file, Xcode + # will only recognize one object (the last one in the file?) and the + # results are unpredictable. + self.project_file.EnsureNoIDCollisions() + + def Write(self): + # Write the project file to a temporary location first. Xcode watches for + # changes to the project file and presents a UI sheet offering to reload + # the project when it does change. However, in some cases, especially when + # multiple projects are open or when Xcode is busy, things don't work so + # seamlessly. Sometimes, Xcode is able to detect that a project file has + # changed but can't unload it because something else is referencing it. + # To mitigate this problem, and to avoid even having Xcode present the UI + # sheet when an open project is rewritten for inconsequential changes, the + # project file is written to a temporary file in the xcodeproj directory + # first. The new temporary file is then compared to the existing project + # file, if any. If they differ, the new file replaces the old; otherwise, + # the new project file is simply deleted. Xcode properly detects a file + # being renamed over an open project file as a change and so it remains + # able to present the "project file changed" sheet under this system. + # Writing to a temporary file first also avoids the possible problem of + # Xcode rereading an incomplete project file. + (output_fd, new_pbxproj_path) = tempfile.mkstemp( + suffix=".tmp", prefix="project.pbxproj.gyp.", dir=self.path + ) + + try: + output_file = os.fdopen(output_fd, "w") + + self.project_file.Print(output_file) + output_file.close() + + pbxproj_path = os.path.join(self.path, "project.pbxproj") + + same = False + try: + same = filecmp.cmp(pbxproj_path, new_pbxproj_path, False) + except OSError as e: + if e.errno != errno.ENOENT: + raise + + if same: + # The new file is identical to the old one, just get rid of the new + # one. + os.unlink(new_pbxproj_path) + else: + # The new file is different from the old one, or there is no old one. + # Rename the new file to the permanent name. + # + # tempfile.mkstemp uses an overly restrictive mode, resulting in a + # file that can only be read by the owner, regardless of the umask. + # There's no reason to not respect the umask here, which means that + # an extra hoop is required to fetch it and reset the new file's mode. + # + # No way to get the umask without setting a new one? Set a safe one + # and then set it back to the old value. + umask = os.umask(0o77) + os.umask(umask) + + os.chmod(new_pbxproj_path, 0o666 & ~umask) + os.rename(new_pbxproj_path, pbxproj_path) + + except Exception: + # Don't leave turds behind. In fact, if this code was responsible for + # creating the xcodeproj directory, get rid of that too. + os.unlink(new_pbxproj_path) + if self.created_dir: + shutil.rmtree(self.path, True) + raise def AddSourceToTarget(source, type, pbxp, xct): - # TODO(mark): Perhaps source_extensions and library_extensions can be made a - # little bit fancier. - source_extensions = ['c', 'cc', 'cpp', 'cxx', 'm', 'mm', 's', 'swift'] - - # .o is conceptually more of a "source" than a "library," but Xcode thinks - # of "sources" as things to compile and "libraries" (or "frameworks") as - # things to link with. Adding an object file to an Xcode target's frameworks - # phase works properly. - library_extensions = ['a', 'dylib', 'framework', 'o'] - - basename = posixpath.basename(source) - (root, ext) = posixpath.splitext(basename) - if ext: - ext = ext[1:].lower() - - if ext in source_extensions and type != 'none': - xct.SourcesPhase().AddFile(source) - elif ext in library_extensions and type != 'none': - xct.FrameworksPhase().AddFile(source) - else: - # Files that aren't added to a sources or frameworks build phase can still - # go into the project file, just not as part of a build phase. - pbxp.AddOrGetFileInRootGroup(source) + # TODO(mark): Perhaps source_extensions and library_extensions can be made a + # little bit fancier. + source_extensions = ["c", "cc", "cpp", "cxx", "m", "mm", "s", "swift"] + + # .o is conceptually more of a "source" than a "library," but Xcode thinks + # of "sources" as things to compile and "libraries" (or "frameworks") as + # things to link with. Adding an object file to an Xcode target's frameworks + # phase works properly. + library_extensions = ["a", "dylib", "framework", "o"] + + basename = posixpath.basename(source) + (root, ext) = posixpath.splitext(basename) + if ext: + ext = ext[1:].lower() + + if ext in source_extensions and type != "none": + xct.SourcesPhase().AddFile(source) + elif ext in library_extensions and type != "none": + xct.FrameworksPhase().AddFile(source) + else: + # Files that aren't added to a sources or frameworks build phase can still + # go into the project file, just not as part of a build phase. + pbxp.AddOrGetFileInRootGroup(source) def AddResourceToTarget(resource, pbxp, xct): - # TODO(mark): Combine with AddSourceToTarget above? Or just inline this call - # where it's used. - xct.ResourcesPhase().AddFile(resource) + # TODO(mark): Combine with AddSourceToTarget above? Or just inline this call + # where it's used. + xct.ResourcesPhase().AddFile(resource) def AddHeaderToTarget(header, pbxp, xct, is_public): - # TODO(mark): Combine with AddSourceToTarget above? Or just inline this call - # where it's used. - settings = '{ATTRIBUTES = (%s, ); }' % ('Private', 'Public')[is_public] - xct.HeadersPhase().AddFile(header, settings) + # TODO(mark): Combine with AddSourceToTarget above? Or just inline this call + # where it's used. + settings = "{ATTRIBUTES = (%s, ); }" % ("Private", "Public")[is_public] + xct.HeadersPhase().AddFile(header, settings) + + +_xcode_variable_re = re.compile(r"(\$\((.*?)\))") -_xcode_variable_re = re.compile(r'(\$\((.*?)\))') def ExpandXcodeVariables(string, expansions): - """Expands Xcode-style $(VARIABLES) in string per the expansions dict. + """Expands Xcode-style $(VARIABLES) in string per the expansions dict. In some rare cases, it is appropriate to expand Xcode variables when a project file is generated. For any substring $(VAR) in string, if VAR is a @@ -540,774 +572,823 @@ def ExpandXcodeVariables(string, expansions): dict will remain in the returned string. """ - matches = _xcode_variable_re.findall(string) - if matches is None: - return string + matches = _xcode_variable_re.findall(string) + if matches is None: + return string - matches.reverse() - for match in matches: - (to_replace, variable) = match - if not variable in expansions: - continue + matches.reverse() + for match in matches: + (to_replace, variable) = match + if variable not in expansions: + continue - replacement = expansions[variable] - string = re.sub(re.escape(to_replace), replacement, string) + replacement = expansions[variable] + string = re.sub(re.escape(to_replace), replacement, string) + + return string - return string + +_xcode_define_re = re.compile(r"([\\\"\' ])") -_xcode_define_re = re.compile(r'([\\\"\' ])') def EscapeXcodeDefine(s): - """We must escape the defines that we give to XCode so that it knows not to + """We must escape the defines that we give to XCode so that it knows not to split on spaces and to respect backslash and quote literals. However, we must not quote the define, or Xcode will incorrectly intepret variables especially $(inherited).""" - return re.sub(_xcode_define_re, r'\\\1', s) + return re.sub(_xcode_define_re, r"\\\1", s) def PerformBuild(data, configurations, params): - options = params['options'] + options = params["options"] - for build_file, build_file_dict in data.items(): - (build_file_root, build_file_ext) = os.path.splitext(build_file) - if build_file_ext != '.gyp': - continue - xcodeproj_path = build_file_root + options.suffix + '.xcodeproj' - if options.generator_output: - xcodeproj_path = os.path.join(options.generator_output, xcodeproj_path) + for build_file, build_file_dict in data.items(): + (build_file_root, build_file_ext) = os.path.splitext(build_file) + if build_file_ext != ".gyp": + continue + xcodeproj_path = build_file_root + options.suffix + ".xcodeproj" + if options.generator_output: + xcodeproj_path = os.path.join(options.generator_output, xcodeproj_path) - for config in configurations: - arguments = ['xcodebuild', '-project', xcodeproj_path] - arguments += ['-configuration', config] - print("Building [%s]: %s" % (config, arguments)) - subprocess.check_call(arguments) + for config in configurations: + arguments = ["xcodebuild", "-project", xcodeproj_path] + arguments += ["-configuration", config] + print("Building [%s]: %s" % (config, arguments)) + subprocess.check_call(arguments) def CalculateGeneratorInputInfo(params): - toplevel = params['options'].toplevel_dir - if params.get('flavor') == 'ninja': - generator_dir = os.path.relpath(params['options'].generator_output or '.') - output_dir = params.get('generator_flags', {}).get('output_dir', 'out') - output_dir = os.path.normpath(os.path.join(generator_dir, output_dir)) - qualified_out_dir = os.path.normpath(os.path.join( - toplevel, output_dir, 'gypfiles-xcode-ninja')) - else: - output_dir = os.path.normpath(os.path.join(toplevel, 'xcodebuild')) - qualified_out_dir = os.path.normpath(os.path.join( - toplevel, output_dir, 'gypfiles')) - - global generator_filelist_paths - generator_filelist_paths = { - 'toplevel': toplevel, - 'qualified_out_dir': qualified_out_dir, - } + toplevel = params["options"].toplevel_dir + if params.get("flavor") == "ninja": + generator_dir = os.path.relpath(params["options"].generator_output or ".") + output_dir = params.get("generator_flags", {}).get("output_dir", "out") + output_dir = os.path.normpath(os.path.join(generator_dir, output_dir)) + qualified_out_dir = os.path.normpath( + os.path.join(toplevel, output_dir, "gypfiles-xcode-ninja") + ) + else: + output_dir = os.path.normpath(os.path.join(toplevel, "xcodebuild")) + qualified_out_dir = os.path.normpath( + os.path.join(toplevel, output_dir, "gypfiles") + ) + + global generator_filelist_paths + generator_filelist_paths = { + "toplevel": toplevel, + "qualified_out_dir": qualified_out_dir, + } def GenerateOutput(target_list, target_dicts, data, params): - # Optionally configure each spec to use ninja as the external builder. - ninja_wrapper = params.get('flavor') == 'ninja' - if ninja_wrapper: - (target_list, target_dicts, data) = \ - gyp.xcode_ninja.CreateWrapper(target_list, target_dicts, data, params) - - options = params['options'] - generator_flags = params.get('generator_flags', {}) - parallel_builds = generator_flags.get('xcode_parallel_builds', True) - serialize_all_tests = \ - generator_flags.get('xcode_serialize_all_test_runs', True) - upgrade_check_project_version = \ - generator_flags.get('xcode_upgrade_check_project_version', None) - - # Format upgrade_check_project_version with leading zeros as needed. - if upgrade_check_project_version: - upgrade_check_project_version = str(upgrade_check_project_version) - while len(upgrade_check_project_version) < 4: - upgrade_check_project_version = '0' + upgrade_check_project_version - - skip_excluded_files = \ - not generator_flags.get('xcode_list_excluded_files', True) - xcode_projects = {} - for build_file, build_file_dict in data.items(): - (build_file_root, build_file_ext) = os.path.splitext(build_file) - if build_file_ext != '.gyp': - continue - xcodeproj_path = build_file_root + options.suffix + '.xcodeproj' - if options.generator_output: - xcodeproj_path = os.path.join(options.generator_output, xcodeproj_path) - xcp = XcodeProject(build_file, xcodeproj_path, build_file_dict) - xcode_projects[build_file] = xcp - pbxp = xcp.project - - # Set project-level attributes from multiple options - project_attributes = {} - if parallel_builds: - project_attributes['BuildIndependentTargetsInParallel'] = 'YES' + # Optionally configure each spec to use ninja as the external builder. + ninja_wrapper = params.get("flavor") == "ninja" + if ninja_wrapper: + (target_list, target_dicts, data) = gyp.xcode_ninja.CreateWrapper( + target_list, target_dicts, data, params + ) + + options = params["options"] + generator_flags = params.get("generator_flags", {}) + parallel_builds = generator_flags.get("xcode_parallel_builds", True) + serialize_all_tests = generator_flags.get("xcode_serialize_all_test_runs", True) + upgrade_check_project_version = generator_flags.get( + "xcode_upgrade_check_project_version", None + ) + + # Format upgrade_check_project_version with leading zeros as needed. if upgrade_check_project_version: - project_attributes['LastUpgradeCheck'] = upgrade_check_project_version - project_attributes['LastTestingUpgradeCheck'] = \ - upgrade_check_project_version - project_attributes['LastSwiftUpdateCheck'] = \ - upgrade_check_project_version - pbxp.SetProperty('attributes', project_attributes) - - # Add gyp/gypi files to project - if not generator_flags.get('standalone'): - main_group = pbxp.GetProperty('mainGroup') - build_group = gyp.xcodeproj_file.PBXGroup({'name': 'Build'}) - main_group.AppendChild(build_group) - for included_file in build_file_dict['included_files']: - build_group.AddOrGetFileByPath(included_file, False) - - xcode_targets = {} - xcode_target_to_target_dict = {} - for qualified_target in target_list: - [build_file, target_name, toolset] = \ - gyp.common.ParseQualifiedTarget(qualified_target) - - spec = target_dicts[qualified_target] - if spec['toolset'] != 'target': - raise Exception( - 'Multiple toolsets not supported in xcode build (target %s)' % - qualified_target) - configuration_names = [spec['default_configuration']] - for configuration_name in sorted(spec['configurations'].keys()): - if configuration_name not in configuration_names: - configuration_names.append(configuration_name) - xcp = xcode_projects[build_file] - pbxp = xcp.project - - # Set up the configurations for the target according to the list of names - # supplied. - xccl = CreateXCConfigurationList(configuration_names) - - # Create an XCTarget subclass object for the target. The type with - # "+bundle" appended will be used if the target has "mac_bundle" set. - # loadable_modules not in a mac_bundle are mapped to - # com.googlecode.gyp.xcode.bundle, a pseudo-type that xcode.py interprets - # to create a single-file mh_bundle. - _types = { - 'executable': 'com.apple.product-type.tool', - 'loadable_module': 'com.googlecode.gyp.xcode.bundle', - 'shared_library': 'com.apple.product-type.library.dynamic', - 'static_library': 'com.apple.product-type.library.static', - 'mac_kernel_extension': 'com.apple.product-type.kernel-extension', - 'executable+bundle': 'com.apple.product-type.application', - 'loadable_module+bundle': 'com.apple.product-type.bundle', - 'loadable_module+xctest': 'com.apple.product-type.bundle.unit-test', - 'loadable_module+xcuitest': 'com.apple.product-type.bundle.ui-testing', - 'shared_library+bundle': 'com.apple.product-type.framework', - 'executable+extension+bundle': 'com.apple.product-type.app-extension', - 'executable+watch+extension+bundle': - 'com.apple.product-type.watchkit-extension', - 'executable+watch+bundle': - 'com.apple.product-type.application.watchapp', - 'mac_kernel_extension+bundle': 'com.apple.product-type.kernel-extension', - } - - target_properties = { - 'buildConfigurationList': xccl, - 'name': target_name, - } + upgrade_check_project_version = str(upgrade_check_project_version) + while len(upgrade_check_project_version) < 4: + upgrade_check_project_version = "0" + upgrade_check_project_version + + skip_excluded_files = not generator_flags.get("xcode_list_excluded_files", True) + xcode_projects = {} + for build_file, build_file_dict in data.items(): + (build_file_root, build_file_ext) = os.path.splitext(build_file) + if build_file_ext != ".gyp": + continue + xcodeproj_path = build_file_root + options.suffix + ".xcodeproj" + if options.generator_output: + xcodeproj_path = os.path.join(options.generator_output, xcodeproj_path) + xcp = XcodeProject(build_file, xcodeproj_path, build_file_dict) + xcode_projects[build_file] = xcp + pbxp = xcp.project + + # Set project-level attributes from multiple options + project_attributes = {} + if parallel_builds: + project_attributes["BuildIndependentTargetsInParallel"] = "YES" + if upgrade_check_project_version: + project_attributes["LastUpgradeCheck"] = upgrade_check_project_version + project_attributes[ + "LastTestingUpgradeCheck" + ] = upgrade_check_project_version + project_attributes["LastSwiftUpdateCheck"] = upgrade_check_project_version + pbxp.SetProperty("attributes", project_attributes) + + # Add gyp/gypi files to project + if not generator_flags.get("standalone"): + main_group = pbxp.GetProperty("mainGroup") + build_group = gyp.xcodeproj_file.PBXGroup({"name": "Build"}) + main_group.AppendChild(build_group) + for included_file in build_file_dict["included_files"]: + build_group.AddOrGetFileByPath(included_file, False) + + xcode_targets = {} + xcode_target_to_target_dict = {} + for qualified_target in target_list: + [build_file, target_name, toolset] = gyp.common.ParseQualifiedTarget( + qualified_target + ) + + spec = target_dicts[qualified_target] + if spec["toolset"] != "target": + raise Exception( + "Multiple toolsets not supported in xcode build (target %s)" + % qualified_target + ) + configuration_names = [spec["default_configuration"]] + for configuration_name in sorted(spec["configurations"].keys()): + if configuration_name not in configuration_names: + configuration_names.append(configuration_name) + xcp = xcode_projects[build_file] + pbxp = xcp.project + + # Set up the configurations for the target according to the list of names + # supplied. + xccl = CreateXCConfigurationList(configuration_names) + + # Create an XCTarget subclass object for the target. The type with + # "+bundle" appended will be used if the target has "mac_bundle" set. + # loadable_modules not in a mac_bundle are mapped to + # com.googlecode.gyp.xcode.bundle, a pseudo-type that xcode.py interprets + # to create a single-file mh_bundle. + _types = { + "executable": "com.apple.product-type.tool", + "loadable_module": "com.googlecode.gyp.xcode.bundle", + "shared_library": "com.apple.product-type.library.dynamic", + "static_library": "com.apple.product-type.library.static", + "mac_kernel_extension": "com.apple.product-type.kernel-extension", + "executable+bundle": "com.apple.product-type.application", + "loadable_module+bundle": "com.apple.product-type.bundle", + "loadable_module+xctest": "com.apple.product-type.bundle.unit-test", + "loadable_module+xcuitest": "com.apple.product-type.bundle.ui-testing", + "shared_library+bundle": "com.apple.product-type.framework", + "executable+extension+bundle": "com.apple.product-type.app-extension", + "executable+watch+extension+bundle": "com.apple.product-type.watchkit-extension", + "executable+watch+bundle": "com.apple.product-type.application.watchapp", + "mac_kernel_extension+bundle": "com.apple.product-type.kernel-extension", + } - type = spec['type'] - is_xctest = int(spec.get('mac_xctest_bundle', 0)) - is_xcuitest = int(spec.get('mac_xcuitest_bundle', 0)) - is_bundle = int(spec.get('mac_bundle', 0)) or is_xctest - is_app_extension = int(spec.get('ios_app_extension', 0)) - is_watchkit_extension = int(spec.get('ios_watchkit_extension', 0)) - is_watch_app = int(spec.get('ios_watch_app', 0)) - if type != 'none': - type_bundle_key = type - if is_xcuitest: - type_bundle_key += '+xcuitest' - assert type == 'loadable_module', ( - 'mac_xcuitest_bundle targets must have type loadable_module ' - '(target %s)' % target_name) - elif is_xctest: - type_bundle_key += '+xctest' - assert type == 'loadable_module', ( - 'mac_xctest_bundle targets must have type loadable_module ' - '(target %s)' % target_name) - elif is_app_extension: - assert is_bundle, ('ios_app_extension flag requires mac_bundle ' - '(target %s)' % target_name) - type_bundle_key += '+extension+bundle' - elif is_watchkit_extension: - assert is_bundle, ('ios_watchkit_extension flag requires mac_bundle ' - '(target %s)' % target_name) - type_bundle_key += '+watch+extension+bundle' - elif is_watch_app: - assert is_bundle, ('ios_watch_app flag requires mac_bundle ' - '(target %s)' % target_name) - type_bundle_key += '+watch+bundle' - elif is_bundle: - type_bundle_key += '+bundle' - - xctarget_type = gyp.xcodeproj_file.PBXNativeTarget - try: - target_properties['productType'] = _types[type_bundle_key] - except KeyError as e: - gyp.common.ExceptionAppend(e, "-- unknown product type while " - "writing target %s" % target_name) - raise - else: - xctarget_type = gyp.xcodeproj_file.PBXAggregateTarget - assert not is_bundle, ( - 'mac_bundle targets cannot have type none (target "%s")' % - target_name) - assert not is_xcuitest, ( - 'mac_xcuitest_bundle targets cannot have type none (target "%s")' % - target_name) - assert not is_xctest, ( - 'mac_xctest_bundle targets cannot have type none (target "%s")' % - target_name) - - target_product_name = spec.get('product_name') - if target_product_name is not None: - target_properties['productName'] = target_product_name - - xct = xctarget_type(target_properties, parent=pbxp, - force_outdir=spec.get('product_dir'), - force_prefix=spec.get('product_prefix'), - force_extension=spec.get('product_extension')) - pbxp.AppendProperty('targets', xct) - xcode_targets[qualified_target] = xct - xcode_target_to_target_dict[xct] = spec - - spec_actions = spec.get('actions', []) - spec_rules = spec.get('rules', []) - - # Xcode has some "issues" with checking dependencies for the "Compile - # sources" step with any source files/headers generated by actions/rules. - # To work around this, if a target is building anything directly (not - # type "none"), then a second target is used to run the GYP actions/rules - # and is made a dependency of this target. This way the work is done - # before the dependency checks for what should be recompiled. - support_xct = None - # The Xcode "issues" don't affect xcode-ninja builds, since the dependency - # logic all happens in ninja. Don't bother creating the extra targets in - # that case. - if type != 'none' and (spec_actions or spec_rules) and not ninja_wrapper: - support_xccl = CreateXCConfigurationList(configuration_names) - support_target_suffix = generator_flags.get( - 'support_target_suffix', ' Support') - support_target_properties = { - 'buildConfigurationList': support_xccl, - 'name': target_name + support_target_suffix, - } - if target_product_name: - support_target_properties['productName'] = \ - target_product_name + ' Support' - support_xct = \ - gyp.xcodeproj_file.PBXAggregateTarget(support_target_properties, - parent=pbxp) - pbxp.AppendProperty('targets', support_xct) - xct.AddDependency(support_xct) - # Hang the support target off the main target so it can be tested/found - # by the generator during Finalize. - xct.support_target = support_xct - - prebuild_index = 0 - - # Add custom shell script phases for "actions" sections. - for action in spec_actions: - # There's no need to write anything into the script to ensure that the - # output directories already exist, because Xcode will look at the - # declared outputs and automatically ensure that they exist for us. - - # Do we have a message to print when this action runs? - message = action.get('message') - if message: - message = 'echo note: ' + gyp.common.EncodePOSIXShellArgument(message) - else: - message = '' - - # Turn the list into a string that can be passed to a shell. - action_string = gyp.common.EncodePOSIXShellList(action['action']) - - # Convert Xcode-type variable references to sh-compatible environment - # variable references. - message_sh = gyp.xcodeproj_file.ConvertVariablesToShellSyntax(message) - action_string_sh = gyp.xcodeproj_file.ConvertVariablesToShellSyntax( - action_string) - - script = '' - # Include the optional message - if message_sh: - script += message_sh + '\n' - # Be sure the script runs in exec, and that if exec fails, the script - # exits signalling an error. - script += 'exec ' + action_string_sh + '\nexit 1\n' - ssbp = gyp.xcodeproj_file.PBXShellScriptBuildPhase({ - 'inputPaths': action['inputs'], - 'name': 'Action "' + action['action_name'] + '"', - 'outputPaths': action['outputs'], - 'shellScript': script, - 'showEnvVarsInLog': 0, - }) - - if support_xct: - support_xct.AppendProperty('buildPhases', ssbp) - else: - # TODO(mark): this assumes too much knowledge of the internals of - # xcodeproj_file; some of these smarts should move into xcodeproj_file - # itself. - xct._properties['buildPhases'].insert(prebuild_index, ssbp) - prebuild_index = prebuild_index + 1 - - # TODO(mark): Should verify that at most one of these is specified. - if int(action.get('process_outputs_as_sources', False)): - for output in action['outputs']: - AddSourceToTarget(output, type, pbxp, xct) - - if int(action.get('process_outputs_as_mac_bundle_resources', False)): - for output in action['outputs']: - AddResourceToTarget(output, pbxp, xct) - - # tgt_mac_bundle_resources holds the list of bundle resources so - # the rule processing can check against it. - if is_bundle: - tgt_mac_bundle_resources = spec.get('mac_bundle_resources', []) - else: - tgt_mac_bundle_resources = [] - - # Add custom shell script phases driving "make" for "rules" sections. - # - # Xcode's built-in rule support is almost powerful enough to use directly, - # but there are a few significant deficiencies that render them unusable. - # There are workarounds for some of its inadequacies, but in aggregate, - # the workarounds added complexity to the generator, and some workarounds - # actually require input files to be crafted more carefully than I'd like. - # Consequently, until Xcode rules are made more capable, "rules" input - # sections will be handled in Xcode output by shell script build phases - # performed prior to the compilation phase. - # - # The following problems with Xcode rules were found. The numbers are - # Apple radar IDs. I hope that these shortcomings are addressed, I really - # liked having the rules handled directly in Xcode during the period that - # I was prototyping this. - # - # 6588600 Xcode compiles custom script rule outputs too soon, compilation - # fails. This occurs when rule outputs from distinct inputs are - # interdependent. The only workaround is to put rules and their - # inputs in a separate target from the one that compiles the rule - # outputs. This requires input file cooperation and it means that - # process_outputs_as_sources is unusable. - # 6584932 Need to declare that custom rule outputs should be excluded from - # compilation. A possible workaround is to lie to Xcode about a - # rule's output, giving it a dummy file it doesn't know how to - # compile. The rule action script would need to touch the dummy. - # 6584839 I need a way to declare additional inputs to a custom rule. - # A possible workaround is a shell script phase prior to - # compilation that touches a rule's primary input files if any - # would-be additional inputs are newer than the output. Modifying - # the source tree - even just modification times - feels dirty. - # 6564240 Xcode "custom script" build rules always dump all environment - # variables. This is a low-prioroty problem and is not a - # show-stopper. - rules_by_ext = {} - for rule in spec_rules: - rules_by_ext[rule['extension']] = rule - - # First, some definitions: - # - # A "rule source" is a file that was listed in a target's "sources" - # list and will have a rule applied to it on the basis of matching the - # rule's "extensions" attribute. Rule sources are direct inputs to - # rules. - # - # Rule definitions may specify additional inputs in their "inputs" - # attribute. These additional inputs are used for dependency tracking - # purposes. - # - # A "concrete output" is a rule output with input-dependent variables - # resolved. For example, given a rule with: - # 'extension': 'ext', 'outputs': ['$(INPUT_FILE_BASE).cc'], - # if the target's "sources" list contained "one.ext" and "two.ext", - # the "concrete output" for rule input "two.ext" would be "two.cc". If - # a rule specifies multiple outputs, each input file that the rule is - # applied to will have the same number of concrete outputs. - # - # If any concrete outputs are outdated or missing relative to their - # corresponding rule_source or to any specified additional input, the - # rule action must be performed to generate the concrete outputs. - - # concrete_outputs_by_rule_source will have an item at the same index - # as the rule['rule_sources'] that it corresponds to. Each item is a - # list of all of the concrete outputs for the rule_source. - concrete_outputs_by_rule_source = [] - - # concrete_outputs_all is a flat list of all concrete outputs that this - # rule is able to produce, given the known set of input files - # (rule_sources) that apply to it. - concrete_outputs_all = [] - - # messages & actions are keyed by the same indices as rule['rule_sources'] - # and concrete_outputs_by_rule_source. They contain the message and - # action to perform after resolving input-dependent variables. The - # message is optional, in which case None is stored for each rule source. - messages = [] - actions = [] - - for rule_source in rule.get('rule_sources', []): - rule_source_dirname, rule_source_basename = \ - posixpath.split(rule_source) - (rule_source_root, rule_source_ext) = \ - posixpath.splitext(rule_source_basename) - - # These are the same variable names that Xcode uses for its own native - # rule support. Because Xcode's rule engine is not being used, they - # need to be expanded as they are written to the makefile. - rule_input_dict = { - 'INPUT_FILE_BASE': rule_source_root, - 'INPUT_FILE_SUFFIX': rule_source_ext, - 'INPUT_FILE_NAME': rule_source_basename, - 'INPUT_FILE_PATH': rule_source, - 'INPUT_FILE_DIRNAME': rule_source_dirname, + target_properties = { + "buildConfigurationList": xccl, + "name": target_name, } - concrete_outputs_for_this_rule_source = [] - for output in rule.get('outputs', []): - # Fortunately, Xcode and make both use $(VAR) format for their - # variables, so the expansion is the only transformation necessary. - # Any remaning $(VAR)-type variables in the string can be given - # directly to make, which will pick up the correct settings from - # what Xcode puts into the environment. - concrete_output = ExpandXcodeVariables(output, rule_input_dict) - concrete_outputs_for_this_rule_source.append(concrete_output) - - # Add all concrete outputs to the project. - pbxp.AddOrGetFileInRootGroup(concrete_output) - - concrete_outputs_by_rule_source.append( \ - concrete_outputs_for_this_rule_source) - concrete_outputs_all.extend(concrete_outputs_for_this_rule_source) - - # TODO(mark): Should verify that at most one of these is specified. - if int(rule.get('process_outputs_as_sources', False)): - for output in concrete_outputs_for_this_rule_source: - AddSourceToTarget(output, type, pbxp, xct) - - # If the file came from the mac_bundle_resources list or if the rule - # is marked to process outputs as bundle resource, do so. - was_mac_bundle_resource = rule_source in tgt_mac_bundle_resources - if was_mac_bundle_resource or \ - int(rule.get('process_outputs_as_mac_bundle_resources', False)): - for output in concrete_outputs_for_this_rule_source: - AddResourceToTarget(output, pbxp, xct) - - # Do we have a message to print when this rule runs? - message = rule.get('message') - if message: - message = gyp.common.EncodePOSIXShellArgument(message) - message = ExpandXcodeVariables(message, rule_input_dict) - messages.append(message) - - # Turn the list into a string that can be passed to a shell. - action_string = gyp.common.EncodePOSIXShellList(rule['action']) - - action = ExpandXcodeVariables(action_string, rule_input_dict) - actions.append(action) - - if len(concrete_outputs_all) > 0: - # TODO(mark): There's a possibility for collision here. Consider - # target "t" rule "A_r" and target "t_A" rule "r". - makefile_name = '%s.make' % re.sub( - '[^a-zA-Z0-9_]', '_' , '%s_%s' % (target_name, rule['rule_name'])) - makefile_path = os.path.join(xcode_projects[build_file].path, - makefile_name) - # TODO(mark): try/close? Write to a temporary file and swap it only - # if it's got changes? - makefile = open(makefile_path, 'wb') - - # make will build the first target in the makefile by default. By - # convention, it's called "all". List all (or at least one) - # concrete output for each rule source as a prerequisite of the "all" - # target. - makefile.write('all: \\\n') - for concrete_output_index in \ - range(0, len(concrete_outputs_by_rule_source)): - # Only list the first (index [0]) concrete output of each input - # in the "all" target. Otherwise, a parallel make (-j > 1) would - # attempt to process each input multiple times simultaneously. - # Otherwise, "all" could just contain the entire list of - # concrete_outputs_all. - concrete_output = \ - concrete_outputs_by_rule_source[concrete_output_index][0] - if concrete_output_index == len(concrete_outputs_by_rule_source) - 1: - eol = '' - else: - eol = ' \\' - makefile.write(' %s%s\n' % (concrete_output, eol)) - - for (rule_source, concrete_outputs, message, action) in \ - zip(rule['rule_sources'], concrete_outputs_by_rule_source, - messages, actions): - makefile.write('\n') - - # Add a rule that declares it can build each concrete output of a - # rule source. Collect the names of the directories that are - # required. - concrete_output_dirs = [] - for concrete_output_index in range(0, len(concrete_outputs)): - concrete_output = concrete_outputs[concrete_output_index] - if concrete_output_index == 0: - bol = '' + type = spec["type"] + is_xctest = int(spec.get("mac_xctest_bundle", 0)) + is_xcuitest = int(spec.get("mac_xcuitest_bundle", 0)) + is_bundle = int(spec.get("mac_bundle", 0)) or is_xctest + is_app_extension = int(spec.get("ios_app_extension", 0)) + is_watchkit_extension = int(spec.get("ios_watchkit_extension", 0)) + is_watch_app = int(spec.get("ios_watch_app", 0)) + if type != "none": + type_bundle_key = type + if is_xcuitest: + type_bundle_key += "+xcuitest" + assert type == "loadable_module", ( + "mac_xcuitest_bundle targets must have type loadable_module " + "(target %s)" % target_name + ) + elif is_xctest: + type_bundle_key += "+xctest" + assert type == "loadable_module", ( + "mac_xctest_bundle targets must have type loadable_module " + "(target %s)" % target_name + ) + elif is_app_extension: + assert is_bundle, ( + "ios_app_extension flag requires mac_bundle " + "(target %s)" % target_name + ) + type_bundle_key += "+extension+bundle" + elif is_watchkit_extension: + assert is_bundle, ( + "ios_watchkit_extension flag requires mac_bundle " + "(target %s)" % target_name + ) + type_bundle_key += "+watch+extension+bundle" + elif is_watch_app: + assert is_bundle, ( + "ios_watch_app flag requires mac_bundle " + "(target %s)" % target_name + ) + type_bundle_key += "+watch+bundle" + elif is_bundle: + type_bundle_key += "+bundle" + + xctarget_type = gyp.xcodeproj_file.PBXNativeTarget + try: + target_properties["productType"] = _types[type_bundle_key] + except KeyError as e: + gyp.common.ExceptionAppend( + e, + "-- unknown product type while " "writing target %s" % target_name, + ) + raise + else: + xctarget_type = gyp.xcodeproj_file.PBXAggregateTarget + assert not is_bundle, ( + 'mac_bundle targets cannot have type none (target "%s")' % target_name + ) + assert not is_xcuitest, ( + 'mac_xcuitest_bundle targets cannot have type none (target "%s")' + % target_name + ) + assert not is_xctest, ( + 'mac_xctest_bundle targets cannot have type none (target "%s")' + % target_name + ) + + target_product_name = spec.get("product_name") + if target_product_name is not None: + target_properties["productName"] = target_product_name + + xct = xctarget_type( + target_properties, + parent=pbxp, + force_outdir=spec.get("product_dir"), + force_prefix=spec.get("product_prefix"), + force_extension=spec.get("product_extension"), + ) + pbxp.AppendProperty("targets", xct) + xcode_targets[qualified_target] = xct + xcode_target_to_target_dict[xct] = spec + + spec_actions = spec.get("actions", []) + spec_rules = spec.get("rules", []) + + # Xcode has some "issues" with checking dependencies for the "Compile + # sources" step with any source files/headers generated by actions/rules. + # To work around this, if a target is building anything directly (not + # type "none"), then a second target is used to run the GYP actions/rules + # and is made a dependency of this target. This way the work is done + # before the dependency checks for what should be recompiled. + support_xct = None + # The Xcode "issues" don't affect xcode-ninja builds, since the dependency + # logic all happens in ninja. Don't bother creating the extra targets in + # that case. + if type != "none" and (spec_actions or spec_rules) and not ninja_wrapper: + support_xccl = CreateXCConfigurationList(configuration_names) + support_target_suffix = generator_flags.get( + "support_target_suffix", " Support" + ) + support_target_properties = { + "buildConfigurationList": support_xccl, + "name": target_name + support_target_suffix, + } + if target_product_name: + support_target_properties["productName"] = ( + target_product_name + " Support" + ) + support_xct = gyp.xcodeproj_file.PBXAggregateTarget( + support_target_properties, parent=pbxp + ) + pbxp.AppendProperty("targets", support_xct) + xct.AddDependency(support_xct) + # Hang the support target off the main target so it can be tested/found + # by the generator during Finalize. + xct.support_target = support_xct + + prebuild_index = 0 + + # Add custom shell script phases for "actions" sections. + for action in spec_actions: + # There's no need to write anything into the script to ensure that the + # output directories already exist, because Xcode will look at the + # declared outputs and automatically ensure that they exist for us. + + # Do we have a message to print when this action runs? + message = action.get("message") + if message: + message = "echo note: " + gyp.common.EncodePOSIXShellArgument(message) else: - bol = ' ' - makefile.write('%s%s \\\n' % (bol, concrete_output)) - - concrete_output_dir = posixpath.dirname(concrete_output) - if (concrete_output_dir and - concrete_output_dir not in concrete_output_dirs): - concrete_output_dirs.append(concrete_output_dir) - - makefile.write(' : \\\n') - - # The prerequisites for this rule are the rule source itself and - # the set of additional rule inputs, if any. - prerequisites = [rule_source] - prerequisites.extend(rule.get('inputs', [])) - for prerequisite_index in range(0, len(prerequisites)): - prerequisite = prerequisites[prerequisite_index] - if prerequisite_index == len(prerequisites) - 1: - eol = '' + message = "" + + # Turn the list into a string that can be passed to a shell. + action_string = gyp.common.EncodePOSIXShellList(action["action"]) + + # Convert Xcode-type variable references to sh-compatible environment + # variable references. + message_sh = gyp.xcodeproj_file.ConvertVariablesToShellSyntax(message) + action_string_sh = gyp.xcodeproj_file.ConvertVariablesToShellSyntax( + action_string + ) + + script = "" + # Include the optional message + if message_sh: + script += message_sh + "\n" + # Be sure the script runs in exec, and that if exec fails, the script + # exits signalling an error. + script += "exec " + action_string_sh + "\nexit 1\n" + ssbp = gyp.xcodeproj_file.PBXShellScriptBuildPhase( + { + "inputPaths": action["inputs"], + "name": 'Action "' + action["action_name"] + '"', + "outputPaths": action["outputs"], + "shellScript": script, + "showEnvVarsInLog": 0, + } + ) + + if support_xct: + support_xct.AppendProperty("buildPhases", ssbp) else: - eol = ' \\' - makefile.write(' %s%s\n' % (prerequisite, eol)) - - # Make sure that output directories exist before executing the rule - # action. - if len(concrete_output_dirs) > 0: - makefile.write('\t@mkdir -p "%s"\n' % - '" "'.join(concrete_output_dirs)) - - # The rule message and action have already had the necessary variable - # substitutions performed. - if message: - # Mark it with note: so Xcode picks it up in build output. - makefile.write('\t@echo note: %s\n' % message) - makefile.write('\t%s\n' % action) - - makefile.close() - - # It might be nice to ensure that needed output directories exist - # here rather than in each target in the Makefile, but that wouldn't - # work if there ever was a concrete output that had an input-dependent - # variable anywhere other than in the leaf position. - - # Don't declare any inputPaths or outputPaths. If they're present, - # Xcode will provide a slight optimization by only running the script - # phase if any output is missing or outdated relative to any input. - # Unfortunately, it will also assume that all outputs are touched by - # the script, and if the outputs serve as files in a compilation - # phase, they will be unconditionally rebuilt. Since make might not - # rebuild everything that could be declared here as an output, this - # extra compilation activity is unnecessary. With inputPaths and - # outputPaths not supplied, make will always be called, but it knows - # enough to not do anything when everything is up-to-date. - - # To help speed things up, pass -j COUNT to make so it does some work - # in parallel. Don't use ncpus because Xcode will build ncpus targets - # in parallel and if each target happens to have a rules step, there - # would be ncpus^2 things going. With a machine that has 2 quad-core - # Xeons, a build can quickly run out of processes based on - # scheduling/other tasks, and randomly failing builds are no good. - script = \ -"""JOB_COUNT="$(/usr/sbin/sysctl -n hw.ncpu)" + # TODO(mark): this assumes too much knowledge of the internals of + # xcodeproj_file; some of these smarts should move into xcodeproj_file + # itself. + xct._properties["buildPhases"].insert(prebuild_index, ssbp) + prebuild_index = prebuild_index + 1 + + # TODO(mark): Should verify that at most one of these is specified. + if int(action.get("process_outputs_as_sources", False)): + for output in action["outputs"]: + AddSourceToTarget(output, type, pbxp, xct) + + if int(action.get("process_outputs_as_mac_bundle_resources", False)): + for output in action["outputs"]: + AddResourceToTarget(output, pbxp, xct) + + # tgt_mac_bundle_resources holds the list of bundle resources so + # the rule processing can check against it. + if is_bundle: + tgt_mac_bundle_resources = spec.get("mac_bundle_resources", []) + else: + tgt_mac_bundle_resources = [] + + # Add custom shell script phases driving "make" for "rules" sections. + # + # Xcode's built-in rule support is almost powerful enough to use directly, + # but there are a few significant deficiencies that render them unusable. + # There are workarounds for some of its inadequacies, but in aggregate, + # the workarounds added complexity to the generator, and some workarounds + # actually require input files to be crafted more carefully than I'd like. + # Consequently, until Xcode rules are made more capable, "rules" input + # sections will be handled in Xcode output by shell script build phases + # performed prior to the compilation phase. + # + # The following problems with Xcode rules were found. The numbers are + # Apple radar IDs. I hope that these shortcomings are addressed, I really + # liked having the rules handled directly in Xcode during the period that + # I was prototyping this. + # + # 6588600 Xcode compiles custom script rule outputs too soon, compilation + # fails. This occurs when rule outputs from distinct inputs are + # interdependent. The only workaround is to put rules and their + # inputs in a separate target from the one that compiles the rule + # outputs. This requires input file cooperation and it means that + # process_outputs_as_sources is unusable. + # 6584932 Need to declare that custom rule outputs should be excluded from + # compilation. A possible workaround is to lie to Xcode about a + # rule's output, giving it a dummy file it doesn't know how to + # compile. The rule action script would need to touch the dummy. + # 6584839 I need a way to declare additional inputs to a custom rule. + # A possible workaround is a shell script phase prior to + # compilation that touches a rule's primary input files if any + # would-be additional inputs are newer than the output. Modifying + # the source tree - even just modification times - feels dirty. + # 6564240 Xcode "custom script" build rules always dump all environment + # variables. This is a low-prioroty problem and is not a + # show-stopper. + rules_by_ext = {} + for rule in spec_rules: + rules_by_ext[rule["extension"]] = rule + + # First, some definitions: + # + # A "rule source" is a file that was listed in a target's "sources" + # list and will have a rule applied to it on the basis of matching the + # rule's "extensions" attribute. Rule sources are direct inputs to + # rules. + # + # Rule definitions may specify additional inputs in their "inputs" + # attribute. These additional inputs are used for dependency tracking + # purposes. + # + # A "concrete output" is a rule output with input-dependent variables + # resolved. For example, given a rule with: + # 'extension': 'ext', 'outputs': ['$(INPUT_FILE_BASE).cc'], + # if the target's "sources" list contained "one.ext" and "two.ext", + # the "concrete output" for rule input "two.ext" would be "two.cc". If + # a rule specifies multiple outputs, each input file that the rule is + # applied to will have the same number of concrete outputs. + # + # If any concrete outputs are outdated or missing relative to their + # corresponding rule_source or to any specified additional input, the + # rule action must be performed to generate the concrete outputs. + + # concrete_outputs_by_rule_source will have an item at the same index + # as the rule['rule_sources'] that it corresponds to. Each item is a + # list of all of the concrete outputs for the rule_source. + concrete_outputs_by_rule_source = [] + + # concrete_outputs_all is a flat list of all concrete outputs that this + # rule is able to produce, given the known set of input files + # (rule_sources) that apply to it. + concrete_outputs_all = [] + + # messages & actions are keyed by the same indices as rule['rule_sources'] + # and concrete_outputs_by_rule_source. They contain the message and + # action to perform after resolving input-dependent variables. The + # message is optional, in which case None is stored for each rule source. + messages = [] + actions = [] + + for rule_source in rule.get("rule_sources", []): + rule_source_dirname, rule_source_basename = posixpath.split(rule_source) + (rule_source_root, rule_source_ext) = posixpath.splitext( + rule_source_basename + ) + + # These are the same variable names that Xcode uses for its own native + # rule support. Because Xcode's rule engine is not being used, they + # need to be expanded as they are written to the makefile. + rule_input_dict = { + "INPUT_FILE_BASE": rule_source_root, + "INPUT_FILE_SUFFIX": rule_source_ext, + "INPUT_FILE_NAME": rule_source_basename, + "INPUT_FILE_PATH": rule_source, + "INPUT_FILE_DIRNAME": rule_source_dirname, + } + + concrete_outputs_for_this_rule_source = [] + for output in rule.get("outputs", []): + # Fortunately, Xcode and make both use $(VAR) format for their + # variables, so the expansion is the only transformation necessary. + # Any remaning $(VAR)-type variables in the string can be given + # directly to make, which will pick up the correct settings from + # what Xcode puts into the environment. + concrete_output = ExpandXcodeVariables(output, rule_input_dict) + concrete_outputs_for_this_rule_source.append(concrete_output) + + # Add all concrete outputs to the project. + pbxp.AddOrGetFileInRootGroup(concrete_output) + + concrete_outputs_by_rule_source.append( + concrete_outputs_for_this_rule_source + ) + concrete_outputs_all.extend(concrete_outputs_for_this_rule_source) + + # TODO(mark): Should verify that at most one of these is specified. + if int(rule.get("process_outputs_as_sources", False)): + for output in concrete_outputs_for_this_rule_source: + AddSourceToTarget(output, type, pbxp, xct) + + # If the file came from the mac_bundle_resources list or if the rule + # is marked to process outputs as bundle resource, do so. + was_mac_bundle_resource = rule_source in tgt_mac_bundle_resources + if was_mac_bundle_resource or int( + rule.get("process_outputs_as_mac_bundle_resources", False) + ): + for output in concrete_outputs_for_this_rule_source: + AddResourceToTarget(output, pbxp, xct) + + # Do we have a message to print when this rule runs? + message = rule.get("message") + if message: + message = gyp.common.EncodePOSIXShellArgument(message) + message = ExpandXcodeVariables(message, rule_input_dict) + messages.append(message) + + # Turn the list into a string that can be passed to a shell. + action_string = gyp.common.EncodePOSIXShellList(rule["action"]) + + action = ExpandXcodeVariables(action_string, rule_input_dict) + actions.append(action) + + if len(concrete_outputs_all) > 0: + # TODO(mark): There's a possibility for collision here. Consider + # target "t" rule "A_r" and target "t_A" rule "r". + makefile_name = "%s.make" % re.sub( + "[^a-zA-Z0-9_]", "_", "%s_%s" % (target_name, rule["rule_name"]) + ) + makefile_path = os.path.join( + xcode_projects[build_file].path, makefile_name + ) + # TODO(mark): try/close? Write to a temporary file and swap it only + # if it's got changes? + makefile = open(makefile_path, "w") + + # make will build the first target in the makefile by default. By + # convention, it's called "all". List all (or at least one) + # concrete output for each rule source as a prerequisite of the "all" + # target. + makefile.write("all: \\\n") + for concrete_output_index, concrete_output_by_rule_source in enumerate( + concrete_outputs_by_rule_source + ): + # Only list the first (index [0]) concrete output of each input + # in the "all" target. Otherwise, a parallel make (-j > 1) would + # attempt to process each input multiple times simultaneously. + # Otherwise, "all" could just contain the entire list of + # concrete_outputs_all. + concrete_output = concrete_output_by_rule_source[0] + if ( + concrete_output_index + == len(concrete_outputs_by_rule_source) - 1 + ): + eol = "" + else: + eol = " \\" + makefile.write(" %s%s\n" % (concrete_output, eol)) + + for (rule_source, concrete_outputs, message, action) in zip( + rule["rule_sources"], + concrete_outputs_by_rule_source, + messages, + actions, + ): + makefile.write("\n") + + # Add a rule that declares it can build each concrete output of a + # rule source. Collect the names of the directories that are + # required. + concrete_output_dirs = [] + for concrete_output_index, concrete_output in enumerate( + concrete_outputs + ): + if concrete_output_index == 0: + bol = "" + else: + bol = " " + makefile.write("%s%s \\\n" % (bol, concrete_output)) + + concrete_output_dir = posixpath.dirname(concrete_output) + if ( + concrete_output_dir + and concrete_output_dir not in concrete_output_dirs + ): + concrete_output_dirs.append(concrete_output_dir) + + makefile.write(" : \\\n") + + # The prerequisites for this rule are the rule source itself and + # the set of additional rule inputs, if any. + prerequisites = [rule_source] + prerequisites.extend(rule.get("inputs", [])) + for prerequisite_index, prerequisite in enumerate(prerequisites): + if prerequisite_index == len(prerequisites) - 1: + eol = "" + else: + eol = " \\" + makefile.write(" %s%s\n" % (prerequisite, eol)) + + # Make sure that output directories exist before executing the rule + # action. + if len(concrete_output_dirs) > 0: + makefile.write( + '\t@mkdir -p "%s"\n' % '" "'.join(concrete_output_dirs) + ) + + # The rule message and action have already had the necessary variable + # substitutions performed. + if message: + # Mark it with note: so Xcode picks it up in build output. + makefile.write("\t@echo note: %s\n" % message) + makefile.write("\t%s\n" % action) + + makefile.close() + + # It might be nice to ensure that needed output directories exist + # here rather than in each target in the Makefile, but that wouldn't + # work if there ever was a concrete output that had an input-dependent + # variable anywhere other than in the leaf position. + + # Don't declare any inputPaths or outputPaths. If they're present, + # Xcode will provide a slight optimization by only running the script + # phase if any output is missing or outdated relative to any input. + # Unfortunately, it will also assume that all outputs are touched by + # the script, and if the outputs serve as files in a compilation + # phase, they will be unconditionally rebuilt. Since make might not + # rebuild everything that could be declared here as an output, this + # extra compilation activity is unnecessary. With inputPaths and + # outputPaths not supplied, make will always be called, but it knows + # enough to not do anything when everything is up-to-date. + + # To help speed things up, pass -j COUNT to make so it does some work + # in parallel. Don't use ncpus because Xcode will build ncpus targets + # in parallel and if each target happens to have a rules step, there + # would be ncpus^2 things going. With a machine that has 2 quad-core + # Xeons, a build can quickly run out of processes based on + # scheduling/other tasks, and randomly failing builds are no good. + script = ( + """JOB_COUNT="$(/usr/sbin/sysctl -n hw.ncpu)" if [ "${JOB_COUNT}" -gt 4 ]; then JOB_COUNT=4 fi exec xcrun make -f "${PROJECT_FILE_PATH}/%s" -j "${JOB_COUNT}" exit 1 -""" % makefile_name - ssbp = gyp.xcodeproj_file.PBXShellScriptBuildPhase({ - 'name': 'Rule "' + rule['rule_name'] + '"', - 'shellScript': script, - 'showEnvVarsInLog': 0, - }) - - if support_xct: - support_xct.AppendProperty('buildPhases', ssbp) - else: - # TODO(mark): this assumes too much knowledge of the internals of - # xcodeproj_file; some of these smarts should move into xcodeproj_file - # itself. - xct._properties['buildPhases'].insert(prebuild_index, ssbp) - prebuild_index = prebuild_index + 1 - - # Extra rule inputs also go into the project file. Concrete outputs were - # already added when they were computed. - groups = ['inputs', 'inputs_excluded'] - if skip_excluded_files: - groups = [x for x in groups if not x.endswith('_excluded')] - for group in groups: - for item in rule.get(group, []): - pbxp.AddOrGetFileInRootGroup(item) - - # Add "sources". - for source in spec.get('sources', []): - (source_root, source_extension) = posixpath.splitext(source) - if source_extension[1:] not in rules_by_ext: - # AddSourceToTarget will add the file to a root group if it's not - # already there. - AddSourceToTarget(source, type, pbxp, xct) - else: - pbxp.AddOrGetFileInRootGroup(source) - - # Add "mac_bundle_resources" and "mac_framework_private_headers" if - # it's a bundle of any type. - if is_bundle: - for resource in tgt_mac_bundle_resources: - (resource_root, resource_extension) = posixpath.splitext(resource) - if resource_extension[1:] not in rules_by_ext: - AddResourceToTarget(resource, pbxp, xct) - else: - pbxp.AddOrGetFileInRootGroup(resource) - - for header in spec.get('mac_framework_private_headers', []): - AddHeaderToTarget(header, pbxp, xct, False) - - # Add "mac_framework_headers". These can be valid for both frameworks - # and static libraries. - if is_bundle or type == 'static_library': - for header in spec.get('mac_framework_headers', []): - AddHeaderToTarget(header, pbxp, xct, True) - - # Add "copies". - pbxcp_dict = {} - for copy_group in spec.get('copies', []): - dest = copy_group['destination'] - if dest[0] not in ('/', '$'): - # Relative paths are relative to $(SRCROOT). - dest = '$(SRCROOT)/' + dest - - code_sign = int(copy_group.get('xcode_code_sign', 0)) - settings = (None, '{ATTRIBUTES = (CodeSignOnCopy, ); }')[code_sign] - - # Coalesce multiple "copies" sections in the same target with the same - # "destination" property into the same PBXCopyFilesBuildPhase, otherwise - # they'll wind up with ID collisions. - pbxcp = pbxcp_dict.get(dest, None) - if pbxcp is None: - pbxcp = gyp.xcodeproj_file.PBXCopyFilesBuildPhase({ - 'name': 'Copy to ' + copy_group['destination'] - }, - parent=xct) - pbxcp.SetDestination(dest) - - # TODO(mark): The usual comment about this knowing too much about - # gyp.xcodeproj_file internals applies. - xct._properties['buildPhases'].insert(prebuild_index, pbxcp) - - pbxcp_dict[dest] = pbxcp - - for file in copy_group['files']: - pbxcp.AddFile(file, settings) - - # Excluded files can also go into the project file. - if not skip_excluded_files: - for key in ['sources', 'mac_bundle_resources', 'mac_framework_headers', - 'mac_framework_private_headers']: - excluded_key = key + '_excluded' - for item in spec.get(excluded_key, []): - pbxp.AddOrGetFileInRootGroup(item) - - # So can "inputs" and "outputs" sections of "actions" groups. - groups = ['inputs', 'inputs_excluded', 'outputs', 'outputs_excluded'] - if skip_excluded_files: - groups = [x for x in groups if not x.endswith('_excluded')] - for action in spec.get('actions', []): - for group in groups: - for item in action.get(group, []): - # Exclude anything in BUILT_PRODUCTS_DIR. They're products, not - # sources. - if not item.startswith('$(BUILT_PRODUCTS_DIR)/'): - pbxp.AddOrGetFileInRootGroup(item) - - for postbuild in spec.get('postbuilds', []): - action_string_sh = gyp.common.EncodePOSIXShellList(postbuild['action']) - script = 'exec ' + action_string_sh + '\nexit 1\n' - - # Make the postbuild step depend on the output of ld or ar from this - # target. Apparently putting the script step after the link step isn't - # sufficient to ensure proper ordering in all cases. With an input - # declared but no outputs, the script step should run every time, as - # desired. - ssbp = gyp.xcodeproj_file.PBXShellScriptBuildPhase({ - 'inputPaths': ['$(BUILT_PRODUCTS_DIR)/$(EXECUTABLE_PATH)'], - 'name': 'Postbuild "' + postbuild['postbuild_name'] + '"', - 'shellScript': script, - 'showEnvVarsInLog': 0, - }) - xct.AppendProperty('buildPhases', ssbp) - - # Add dependencies before libraries, because adding a dependency may imply - # adding a library. It's preferable to keep dependencies listed first - # during a link phase so that they can override symbols that would - # otherwise be provided by libraries, which will usually include system - # libraries. On some systems, ld is finicky and even requires the - # libraries to be ordered in such a way that unresolved symbols in - # earlier-listed libraries may only be resolved by later-listed libraries. - # The Mac linker doesn't work that way, but other platforms do, and so - # their linker invocations need to be constructed in this way. There's - # no compelling reason for Xcode's linker invocations to differ. - - if 'dependencies' in spec: - for dependency in spec['dependencies']: - xct.AddDependency(xcode_targets[dependency]) - # The support project also gets the dependencies (in case they are - # needed for the actions/rules to work). - if support_xct: - support_xct.AddDependency(xcode_targets[dependency]) - - if 'libraries' in spec: - for library in spec['libraries']: - xct.FrameworksPhase().AddFile(library) - # Add the library's directory to LIBRARY_SEARCH_PATHS if necessary. - # I wish Xcode handled this automatically. - library_dir = posixpath.dirname(library) - if library_dir not in xcode_standard_library_dirs and ( - not xct.HasBuildSetting(_library_search_paths_var) or - library_dir not in xct.GetBuildSetting(_library_search_paths_var)): - xct.AppendBuildSetting(_library_search_paths_var, library_dir) - - for configuration_name in configuration_names: - configuration = spec['configurations'][configuration_name] - xcbc = xct.ConfigurationNamed(configuration_name) - for include_dir in configuration.get('mac_framework_dirs', []): - xcbc.AppendBuildSetting('FRAMEWORK_SEARCH_PATHS', include_dir) - for include_dir in configuration.get('include_dirs', []): - xcbc.AppendBuildSetting('HEADER_SEARCH_PATHS', include_dir) - for library_dir in configuration.get('library_dirs', []): - if library_dir not in xcode_standard_library_dirs and ( - not xcbc.HasBuildSetting(_library_search_paths_var) or - library_dir not in xcbc.GetBuildSetting(_library_search_paths_var)): - xcbc.AppendBuildSetting(_library_search_paths_var, library_dir) - - if 'defines' in configuration: - for define in configuration['defines']: - set_define = EscapeXcodeDefine(define) - xcbc.AppendBuildSetting('GCC_PREPROCESSOR_DEFINITIONS', set_define) - if 'xcode_settings' in configuration: - for xck, xcv in configuration['xcode_settings'].items(): - xcbc.SetBuildSetting(xck, xcv) - if 'xcode_config_file' in configuration: - config_ref = pbxp.AddOrGetFileInRootGroup( - configuration['xcode_config_file']) - xcbc.SetBaseConfiguration(config_ref) - - build_files = [] - for build_file, build_file_dict in data.items(): - if build_file.endswith('.gyp'): - build_files.append(build_file) - - for build_file in build_files: - xcode_projects[build_file].Finalize1(xcode_targets, serialize_all_tests) - - for build_file in build_files: - xcode_projects[build_file].Finalize2(xcode_targets, - xcode_target_to_target_dict) - - for build_file in build_files: - xcode_projects[build_file].Write() +""" + % makefile_name + ) + ssbp = gyp.xcodeproj_file.PBXShellScriptBuildPhase( + { + "name": 'Rule "' + rule["rule_name"] + '"', + "shellScript": script, + "showEnvVarsInLog": 0, + } + ) + + if support_xct: + support_xct.AppendProperty("buildPhases", ssbp) + else: + # TODO(mark): this assumes too much knowledge of the internals of + # xcodeproj_file; some of these smarts should move into xcodeproj_file + # itself. + xct._properties["buildPhases"].insert(prebuild_index, ssbp) + prebuild_index = prebuild_index + 1 + + # Extra rule inputs also go into the project file. Concrete outputs were + # already added when they were computed. + groups = ["inputs", "inputs_excluded"] + if skip_excluded_files: + groups = [x for x in groups if not x.endswith("_excluded")] + for group in groups: + for item in rule.get(group, []): + pbxp.AddOrGetFileInRootGroup(item) + + # Add "sources". + for source in spec.get("sources", []): + (source_root, source_extension) = posixpath.splitext(source) + if source_extension[1:] not in rules_by_ext: + # AddSourceToTarget will add the file to a root group if it's not + # already there. + AddSourceToTarget(source, type, pbxp, xct) + else: + pbxp.AddOrGetFileInRootGroup(source) + + # Add "mac_bundle_resources" and "mac_framework_private_headers" if + # it's a bundle of any type. + if is_bundle: + for resource in tgt_mac_bundle_resources: + (resource_root, resource_extension) = posixpath.splitext(resource) + if resource_extension[1:] not in rules_by_ext: + AddResourceToTarget(resource, pbxp, xct) + else: + pbxp.AddOrGetFileInRootGroup(resource) + + for header in spec.get("mac_framework_private_headers", []): + AddHeaderToTarget(header, pbxp, xct, False) + + # Add "mac_framework_headers". These can be valid for both frameworks + # and static libraries. + if is_bundle or type == "static_library": + for header in spec.get("mac_framework_headers", []): + AddHeaderToTarget(header, pbxp, xct, True) + + # Add "copies". + pbxcp_dict = {} + for copy_group in spec.get("copies", []): + dest = copy_group["destination"] + if dest[0] not in ("/", "$"): + # Relative paths are relative to $(SRCROOT). + dest = "$(SRCROOT)/" + dest + + code_sign = int(copy_group.get("xcode_code_sign", 0)) + settings = (None, "{ATTRIBUTES = (CodeSignOnCopy, ); }")[code_sign] + + # Coalesce multiple "copies" sections in the same target with the same + # "destination" property into the same PBXCopyFilesBuildPhase, otherwise + # they'll wind up with ID collisions. + pbxcp = pbxcp_dict.get(dest, None) + if pbxcp is None: + pbxcp = gyp.xcodeproj_file.PBXCopyFilesBuildPhase( + {"name": "Copy to " + copy_group["destination"]}, parent=xct + ) + pbxcp.SetDestination(dest) + + # TODO(mark): The usual comment about this knowing too much about + # gyp.xcodeproj_file internals applies. + xct._properties["buildPhases"].insert(prebuild_index, pbxcp) + + pbxcp_dict[dest] = pbxcp + + for file in copy_group["files"]: + pbxcp.AddFile(file, settings) + + # Excluded files can also go into the project file. + if not skip_excluded_files: + for key in [ + "sources", + "mac_bundle_resources", + "mac_framework_headers", + "mac_framework_private_headers", + ]: + excluded_key = key + "_excluded" + for item in spec.get(excluded_key, []): + pbxp.AddOrGetFileInRootGroup(item) + + # So can "inputs" and "outputs" sections of "actions" groups. + groups = ["inputs", "inputs_excluded", "outputs", "outputs_excluded"] + if skip_excluded_files: + groups = [x for x in groups if not x.endswith("_excluded")] + for action in spec.get("actions", []): + for group in groups: + for item in action.get(group, []): + # Exclude anything in BUILT_PRODUCTS_DIR. They're products, not + # sources. + if not item.startswith("$(BUILT_PRODUCTS_DIR)/"): + pbxp.AddOrGetFileInRootGroup(item) + + for postbuild in spec.get("postbuilds", []): + action_string_sh = gyp.common.EncodePOSIXShellList(postbuild["action"]) + script = "exec " + action_string_sh + "\nexit 1\n" + + # Make the postbuild step depend on the output of ld or ar from this + # target. Apparently putting the script step after the link step isn't + # sufficient to ensure proper ordering in all cases. With an input + # declared but no outputs, the script step should run every time, as + # desired. + ssbp = gyp.xcodeproj_file.PBXShellScriptBuildPhase( + { + "inputPaths": ["$(BUILT_PRODUCTS_DIR)/$(EXECUTABLE_PATH)"], + "name": 'Postbuild "' + postbuild["postbuild_name"] + '"', + "shellScript": script, + "showEnvVarsInLog": 0, + } + ) + xct.AppendProperty("buildPhases", ssbp) + + # Add dependencies before libraries, because adding a dependency may imply + # adding a library. It's preferable to keep dependencies listed first + # during a link phase so that they can override symbols that would + # otherwise be provided by libraries, which will usually include system + # libraries. On some systems, ld is finicky and even requires the + # libraries to be ordered in such a way that unresolved symbols in + # earlier-listed libraries may only be resolved by later-listed libraries. + # The Mac linker doesn't work that way, but other platforms do, and so + # their linker invocations need to be constructed in this way. There's + # no compelling reason for Xcode's linker invocations to differ. + + if "dependencies" in spec: + for dependency in spec["dependencies"]: + xct.AddDependency(xcode_targets[dependency]) + # The support project also gets the dependencies (in case they are + # needed for the actions/rules to work). + if support_xct: + support_xct.AddDependency(xcode_targets[dependency]) + + if "libraries" in spec: + for library in spec["libraries"]: + xct.FrameworksPhase().AddFile(library) + # Add the library's directory to LIBRARY_SEARCH_PATHS if necessary. + # I wish Xcode handled this automatically. + library_dir = posixpath.dirname(library) + if library_dir not in xcode_standard_library_dirs and ( + not xct.HasBuildSetting(_library_search_paths_var) + or library_dir not in xct.GetBuildSetting(_library_search_paths_var) + ): + xct.AppendBuildSetting(_library_search_paths_var, library_dir) + + for configuration_name in configuration_names: + configuration = spec["configurations"][configuration_name] + xcbc = xct.ConfigurationNamed(configuration_name) + for include_dir in configuration.get("mac_framework_dirs", []): + xcbc.AppendBuildSetting("FRAMEWORK_SEARCH_PATHS", include_dir) + for include_dir in configuration.get("include_dirs", []): + xcbc.AppendBuildSetting("HEADER_SEARCH_PATHS", include_dir) + for library_dir in configuration.get("library_dirs", []): + if library_dir not in xcode_standard_library_dirs and ( + not xcbc.HasBuildSetting(_library_search_paths_var) + or library_dir + not in xcbc.GetBuildSetting(_library_search_paths_var) + ): + xcbc.AppendBuildSetting(_library_search_paths_var, library_dir) + + if "defines" in configuration: + for define in configuration["defines"]: + set_define = EscapeXcodeDefine(define) + xcbc.AppendBuildSetting("GCC_PREPROCESSOR_DEFINITIONS", set_define) + if "xcode_settings" in configuration: + for xck, xcv in configuration["xcode_settings"].items(): + xcbc.SetBuildSetting(xck, xcv) + if "xcode_config_file" in configuration: + config_ref = pbxp.AddOrGetFileInRootGroup( + configuration["xcode_config_file"] + ) + xcbc.SetBaseConfiguration(config_ref) + + build_files = [] + for build_file, build_file_dict in data.items(): + if build_file.endswith(".gyp"): + build_files.append(build_file) + + for build_file in build_files: + xcode_projects[build_file].Finalize1(xcode_targets, serialize_all_tests) + + for build_file in build_files: + xcode_projects[build_file].Finalize2(xcode_targets, xcode_target_to_target_dict) + + for build_file in build_files: + xcode_projects[build_file].Write() diff --git a/gyp/pylib/gyp/generator/xcode_test.py b/gyp/pylib/gyp/generator/xcode_test.py index 260324a43f..51fbca6a2a 100644 --- a/gyp/pylib/gyp/generator/xcode_test.py +++ b/gyp/pylib/gyp/generator/xcode_test.py @@ -12,12 +12,14 @@ class TestEscapeXcodeDefine(unittest.TestCase): - if sys.platform == 'darwin': - def test_InheritedRemainsUnescaped(self): - self.assertEqual(xcode.EscapeXcodeDefine('$(inherited)'), '$(inherited)') + if sys.platform == "darwin": - def test_Escaping(self): - self.assertEqual(xcode.EscapeXcodeDefine('a b"c\\'), 'a\\ b\\"c\\\\') + def test_InheritedRemainsUnescaped(self): + self.assertEqual(xcode.EscapeXcodeDefine("$(inherited)"), "$(inherited)") -if __name__ == '__main__': - unittest.main() + def test_Escaping(self): + self.assertEqual(xcode.EscapeXcodeDefine('a b"c\\'), 'a\\ b\\"c\\\\') + + +if __name__ == "__main__": + unittest.main() diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index 1f40abb069..139df75405 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -9,7 +9,6 @@ import gyp.common import gyp.simple_copy import multiprocessing -import optparse import os.path import re import shlex @@ -17,7 +16,6 @@ import subprocess import sys import threading -import time import traceback from distutils.version import StrictVersion from gyp.common import GypError @@ -27,28 +25,28 @@ # A list of types that are treated as linkable. linkable_types = [ - 'executable', - 'shared_library', - 'loadable_module', - 'mac_kernel_extension', - 'windows_driver', + "executable", + "shared_library", + "loadable_module", + "mac_kernel_extension", + "windows_driver", ] # A list of sections that contain links to other targets. -dependency_sections = ['dependencies', 'export_dependent_settings'] +dependency_sections = ["dependencies", "export_dependent_settings"] # base_path_sections is a list of sections defined by GYP that contain # pathnames. The generators can provide more keys, the two lists are merged # into path_sections, but you should call IsPathSection instead of using either # list directly. base_path_sections = [ - 'destination', - 'files', - 'include_dirs', - 'inputs', - 'libraries', - 'outputs', - 'sources', + "destination", + "files", + "include_dirs", + "inputs", + "libraries", + "outputs", + "sources", ] path_sections = set() @@ -57,77 +55,78 @@ per_process_data = {} per_process_aux_data = {} + def IsPathSection(section): - # If section ends in one of the '=+?!' characters, it's applied to a section - # without the trailing characters. '/' is notably absent from this list, - # because there's no way for a regular expression to be treated as a path. - while section and section[-1:] in '=+?!': - section = section[:-1] - - if section in path_sections: - return True - - # Sections mathing the regexp '_(dir|file|path)s?$' are also - # considered PathSections. Using manual string matching since that - # is much faster than the regexp and this can be called hundreds of - # thousands of times so micro performance matters. - if "_" in section: - tail = section[-6:] - if tail[-1] == 's': - tail = tail[:-1] - if tail[-5:] in ('_file', '_path'): - return True - return tail[-4:] == '_dir' - - return False + # If section ends in one of the '=+?!' characters, it's applied to a section + # without the trailing characters. '/' is notably absent from this list, + # because there's no way for a regular expression to be treated as a path. + while section and section[-1:] in "=+?!": + section = section[:-1] + + if section in path_sections: + return True + + # Sections mathing the regexp '_(dir|file|path)s?$' are also + # considered PathSections. Using manual string matching since that + # is much faster than the regexp and this can be called hundreds of + # thousands of times so micro performance matters. + if "_" in section: + tail = section[-6:] + if tail[-1] == "s": + tail = tail[:-1] + if tail[-5:] in ("_file", "_path"): + return True + return tail[-4:] == "_dir" + + return False + # base_non_configuration_keys is a list of key names that belong in the target # itself and should not be propagated into its configurations. It is merged # with a list that can come from the generator to # create non_configuration_keys. base_non_configuration_keys = [ - # Sections that must exist inside targets and not configurations. - 'actions', - 'configurations', - 'copies', - 'default_configuration', - 'dependencies', - 'dependencies_original', - 'libraries', - 'postbuilds', - 'product_dir', - 'product_extension', - 'product_name', - 'product_prefix', - 'rules', - 'run_as', - 'sources', - 'standalone_static_library', - 'suppress_wildcard', - 'target_name', - 'toolset', - 'toolsets', - 'type', - - # Sections that can be found inside targets or configurations, but that - # should not be propagated from targets into their configurations. - 'variables', + # Sections that must exist inside targets and not configurations. + "actions", + "configurations", + "copies", + "default_configuration", + "dependencies", + "dependencies_original", + "libraries", + "postbuilds", + "product_dir", + "product_extension", + "product_name", + "product_prefix", + "rules", + "run_as", + "sources", + "standalone_static_library", + "suppress_wildcard", + "target_name", + "toolset", + "toolsets", + "type", + # Sections that can be found inside targets or configurations, but that + # should not be propagated from targets into their configurations. + "variables", ] non_configuration_keys = [] # Keys that do not belong inside a configuration dictionary. invalid_configuration_keys = [ - 'actions', - 'all_dependent_settings', - 'configurations', - 'dependencies', - 'direct_dependent_settings', - 'libraries', - 'link_settings', - 'sources', - 'standalone_static_library', - 'target_name', - 'type', + "actions", + "all_dependent_settings", + "configurations", + "dependencies", + "direct_dependent_settings", + "libraries", + "link_settings", + "sources", + "standalone_static_library", + "target_name", + "type", ] # Controls whether or not the generator supports multiple toolsets. @@ -139,8 +138,9 @@ def IsPathSection(section): # } generator_filelist_paths = None + def GetIncludedBuildFiles(build_file_path, aux_data, included=None): - """Return a list of all build files included into build_file_path. + """Return a list of all build files included into build_file_path. The returned list will contain build_file_path as well as all other files that it included, either directly or indirectly. Note that the list may @@ -158,535 +158,595 @@ def GetIncludedBuildFiles(build_file_path, aux_data, included=None): in the list will be relative to the current directory. """ - if included is None: - included = [] + if included is None: + included = [] - if build_file_path in included: - return included + if build_file_path in included: + return included - included.append(build_file_path) + included.append(build_file_path) - for included_build_file in aux_data[build_file_path].get('included', []): - GetIncludedBuildFiles(included_build_file, aux_data, included) + for included_build_file in aux_data[build_file_path].get("included", []): + GetIncludedBuildFiles(included_build_file, aux_data, included) - return included + return included def CheckedEval(file_contents): - """Return the eval of a gyp file. + """Return the eval of a gyp file. The gyp file is restricted to dictionaries and lists only, and repeated keys are not allowed. Note that this is slower than eval() is. """ - syntax_tree = ast.parse(file_contents) - assert isinstance(syntax_tree, ast.Module) - c1 = syntax_tree.body - assert len(c1) == 1 - c2 = c1[0] - assert isinstance(c2, ast.Expr) - return CheckNode(c2.value, []) + syntax_tree = ast.parse(file_contents) + assert isinstance(syntax_tree, ast.Module) + c1 = syntax_tree.body + assert len(c1) == 1 + c2 = c1[0] + assert isinstance(c2, ast.Expr) + return CheckNode(c2.value, []) def CheckNode(node, keypath): - if isinstance(node, ast.Dict): - c = node.getChildren() - dict = {} - for key, value in zip(node.keys, node.values): - assert isinstance(key, ast.Str) - key = key.s - if key in dict: - raise GypError("Key '" + key + "' repeated at level " + - repr(len(keypath) + 1) + " with key path '" + - '.'.join(keypath) + "'") - kp = list(keypath) # Make a copy of the list for descending this node. - kp.append(key) - dict[key] = CheckNode(value, kp) - return dict - elif isinstance(node, ast.List): - children = [] - for index, child in enumerate(node.elts): - kp = list(keypath) # Copy list. - kp.append(repr(index)) - children.append(CheckNode(child, kp)) - return children - elif isinstance(node, ast.Str): - return node.s - else: - raise TypeError("Unknown AST node at key path '" + '.'.join(keypath) + - "': " + repr(node)) - - -def LoadOneBuildFile(build_file_path, data, aux_data, includes, - is_target, check): - if build_file_path in data: - return data[build_file_path] - - if os.path.exists(build_file_path): - # Open the build file for read ('r') with universal-newlines mode ('U') - # to make sure platform specific newlines ('\r\n' or '\r') are converted to '\n' - # which otherwise will fail eval() - if sys.platform == 'zos': - # On z/OS, universal-newlines mode treats the file as an ascii file. But since - # node-gyp produces ebcdic files, do not use that mode. - build_file_contents = open(build_file_path, 'r').read() + if isinstance(node, ast.Dict): + dict = {} + for key, value in zip(node.keys, node.values): + assert isinstance(key, ast.Str) + key = key.s + if key in dict: + raise GypError( + "Key '" + + key + + "' repeated at level " + + repr(len(keypath) + 1) + + " with key path '" + + ".".join(keypath) + + "'" + ) + kp = list(keypath) # Make a copy of the list for descending this node. + kp.append(key) + dict[key] = CheckNode(value, kp) + return dict + elif isinstance(node, ast.List): + children = [] + for index, child in enumerate(node.elts): + kp = list(keypath) # Copy list. + kp.append(repr(index)) + children.append(CheckNode(child, kp)) + return children + elif isinstance(node, ast.Str): + return node.s else: - build_file_contents = open(build_file_path, 'rU').read() - else: - raise GypError("%s not found (cwd: %s)" % (build_file_path, os.getcwd())) - - build_file_data = None - try: - if check: - build_file_data = CheckedEval(build_file_contents) + raise TypeError( + "Unknown AST node at key path '" + ".".join(keypath) + "': " + repr(node) + ) + + +def LoadOneBuildFile(build_file_path, data, aux_data, includes, is_target, check): + if build_file_path in data: + return data[build_file_path] + + if os.path.exists(build_file_path): + # Open the build file for read ('r') with universal-newlines mode ('U') + # to make sure platform specific newlines ('\r\n' or '\r') are converted to '\n' + # which otherwise will fail eval() + if sys.platform == "zos": + # On z/OS, universal-newlines mode treats the file as an ascii file. But since + # node-gyp produces ebcdic files, do not use that mode. + build_file_contents = open(build_file_path, "r").read() + else: + build_file_contents = open(build_file_path, "rU").read() else: - build_file_data = eval(build_file_contents, {'__builtins__': {}}, - None) - except SyntaxError as e: - e.filename = build_file_path - raise - except Exception as e: - gyp.common.ExceptionAppend(e, 'while reading ' + build_file_path) - raise - - if type(build_file_data) is not dict: - raise GypError("%s does not evaluate to a dictionary." % build_file_path) - - data[build_file_path] = build_file_data - aux_data[build_file_path] = {} - - # Scan for includes and merge them in. - if ('skip_includes' not in build_file_data or - not build_file_data['skip_includes']): + raise GypError("%s not found (cwd: %s)" % (build_file_path, os.getcwd())) + + build_file_data = None try: - if is_target: - LoadBuildFileIncludesIntoDict(build_file_data, build_file_path, data, - aux_data, includes, check) - else: - LoadBuildFileIncludesIntoDict(build_file_data, build_file_path, data, - aux_data, None, check) + if check: + build_file_data = CheckedEval(build_file_contents) + else: + build_file_data = eval(build_file_contents, {"__builtins__": {}}, None) + except SyntaxError as e: + e.filename = build_file_path + raise except Exception as e: - gyp.common.ExceptionAppend(e, - 'while reading includes of ' + build_file_path) - raise - - return build_file_data - - -def LoadBuildFileIncludesIntoDict(subdict, subdict_path, data, aux_data, - includes, check): - includes_list = [] - if includes != None: - includes_list.extend(includes) - if 'includes' in subdict: - for include in subdict['includes']: - # "include" is specified relative to subdict_path, so compute the real - # path to include by appending the provided "include" to the directory - # in which subdict_path resides. - relative_include = \ - os.path.normpath(os.path.join(os.path.dirname(subdict_path), include)) - includes_list.append(relative_include) - # Unhook the includes list, it's no longer needed. - del subdict['includes'] - - # Merge in the included files. - for include in includes_list: - if not 'included' in aux_data[subdict_path]: - aux_data[subdict_path]['included'] = [] - aux_data[subdict_path]['included'].append(include) - - gyp.DebugOutput(gyp.DEBUG_INCLUDES, "Loading Included File: '%s'", include) - - MergeDicts(subdict, - LoadOneBuildFile(include, data, aux_data, None, False, check), - subdict_path, include) - - # Recurse into subdictionaries. - for k, v in subdict.items(): - if type(v) is dict: - LoadBuildFileIncludesIntoDict(v, subdict_path, data, aux_data, - None, check) - elif type(v) is list: - LoadBuildFileIncludesIntoList(v, subdict_path, data, aux_data, - check) + gyp.common.ExceptionAppend(e, "while reading " + build_file_path) + raise + + if type(build_file_data) is not dict: + raise GypError("%s does not evaluate to a dictionary." % build_file_path) + + data[build_file_path] = build_file_data + aux_data[build_file_path] = {} + + # Scan for includes and merge them in. + if "skip_includes" not in build_file_data or not build_file_data["skip_includes"]: + try: + if is_target: + LoadBuildFileIncludesIntoDict( + build_file_data, build_file_path, data, aux_data, includes, check + ) + else: + LoadBuildFileIncludesIntoDict( + build_file_data, build_file_path, data, aux_data, None, check + ) + except Exception as e: + gyp.common.ExceptionAppend( + e, "while reading includes of " + build_file_path + ) + raise + + return build_file_data + + +def LoadBuildFileIncludesIntoDict( + subdict, subdict_path, data, aux_data, includes, check +): + includes_list = [] + if includes is not None: + includes_list.extend(includes) + if "includes" in subdict: + for include in subdict["includes"]: + # "include" is specified relative to subdict_path, so compute the real + # path to include by appending the provided "include" to the directory + # in which subdict_path resides. + relative_include = os.path.normpath( + os.path.join(os.path.dirname(subdict_path), include) + ) + includes_list.append(relative_include) + # Unhook the includes list, it's no longer needed. + del subdict["includes"] + + # Merge in the included files. + for include in includes_list: + if "included" not in aux_data[subdict_path]: + aux_data[subdict_path]["included"] = [] + aux_data[subdict_path]["included"].append(include) + + gyp.DebugOutput(gyp.DEBUG_INCLUDES, "Loading Included File: '%s'", include) + + MergeDicts( + subdict, + LoadOneBuildFile(include, data, aux_data, None, False, check), + subdict_path, + include, + ) + + # Recurse into subdictionaries. + for k, v in subdict.items(): + if type(v) is dict: + LoadBuildFileIncludesIntoDict(v, subdict_path, data, aux_data, None, check) + elif type(v) is list: + LoadBuildFileIncludesIntoList(v, subdict_path, data, aux_data, check) # This recurses into lists so that it can look for dicts. def LoadBuildFileIncludesIntoList(sublist, sublist_path, data, aux_data, check): - for item in sublist: - if type(item) is dict: - LoadBuildFileIncludesIntoDict(item, sublist_path, data, aux_data, - None, check) - elif type(item) is list: - LoadBuildFileIncludesIntoList(item, sublist_path, data, aux_data, check) + for item in sublist: + if type(item) is dict: + LoadBuildFileIncludesIntoDict( + item, sublist_path, data, aux_data, None, check + ) + elif type(item) is list: + LoadBuildFileIncludesIntoList(item, sublist_path, data, aux_data, check) + # Processes toolsets in all the targets. This recurses into condition entries # since they can contain toolsets as well. def ProcessToolsetsInDict(data): - if 'targets' in data: - target_list = data['targets'] - new_target_list = [] - for target in target_list: - # If this target already has an explicit 'toolset', and no 'toolsets' - # list, don't modify it further. - if 'toolset' in target and 'toolsets' not in target: - new_target_list.append(target) - continue - if multiple_toolsets: - toolsets = target.get('toolsets', ['target']) - else: - toolsets = ['target'] - # Make sure this 'toolsets' definition is only processed once. - if 'toolsets' in target: - del target['toolsets'] - if len(toolsets) > 0: - # Optimization: only do copies if more than one toolset is specified. - for build in toolsets[1:]: - new_target = gyp.simple_copy.deepcopy(target) - new_target['toolset'] = build - new_target_list.append(new_target) - target['toolset'] = toolsets[0] - new_target_list.append(target) - data['targets'] = new_target_list - if 'conditions' in data: - for condition in data['conditions']: - if type(condition) is list: - for condition_dict in condition[1:]: - if type(condition_dict) is dict: - ProcessToolsetsInDict(condition_dict) + if "targets" in data: + target_list = data["targets"] + new_target_list = [] + for target in target_list: + # If this target already has an explicit 'toolset', and no 'toolsets' + # list, don't modify it further. + if "toolset" in target and "toolsets" not in target: + new_target_list.append(target) + continue + if multiple_toolsets: + toolsets = target.get("toolsets", ["target"]) + else: + toolsets = ["target"] + # Make sure this 'toolsets' definition is only processed once. + if "toolsets" in target: + del target["toolsets"] + if len(toolsets) > 0: + # Optimization: only do copies if more than one toolset is specified. + for build in toolsets[1:]: + new_target = gyp.simple_copy.deepcopy(target) + new_target["toolset"] = build + new_target_list.append(new_target) + target["toolset"] = toolsets[0] + new_target_list.append(target) + data["targets"] = new_target_list + if "conditions" in data: + for condition in data["conditions"]: + if type(condition) is list: + for condition_dict in condition[1:]: + if type(condition_dict) is dict: + ProcessToolsetsInDict(condition_dict) # TODO(mark): I don't love this name. It just means that it's going to load # a build file that contains targets and is expected to provide a targets dict # that contains the targets... -def LoadTargetBuildFile(build_file_path, data, aux_data, variables, includes, - depth, check, load_dependencies): - # If depth is set, predefine the DEPTH variable to be a relative path from - # this build file's directory to the directory identified by depth. - if depth: - # TODO(dglazkov) The backslash/forward-slash replacement at the end is a - # temporary measure. This should really be addressed by keeping all paths - # in POSIX until actual project generation. - d = gyp.common.RelativePath(depth, os.path.dirname(build_file_path)) - if d == '': - variables['DEPTH'] = '.' +def LoadTargetBuildFile( + build_file_path, + data, + aux_data, + variables, + includes, + depth, + check, + load_dependencies, +): + # If depth is set, predefine the DEPTH variable to be a relative path from + # this build file's directory to the directory identified by depth. + if depth: + # TODO(dglazkov) The backslash/forward-slash replacement at the end is a + # temporary measure. This should really be addressed by keeping all paths + # in POSIX until actual project generation. + d = gyp.common.RelativePath(depth, os.path.dirname(build_file_path)) + if d == "": + variables["DEPTH"] = "." + else: + variables["DEPTH"] = d.replace("\\", "/") + + # The 'target_build_files' key is only set when loading target build files in + # the non-parallel code path, where LoadTargetBuildFile is called + # recursively. In the parallel code path, we don't need to check whether the + # |build_file_path| has already been loaded, because the 'scheduled' set in + # ParallelState guarantees that we never load the same |build_file_path| + # twice. + if "target_build_files" in data: + if build_file_path in data["target_build_files"]: + # Already loaded. + return False + data["target_build_files"].add(build_file_path) + + gyp.DebugOutput( + gyp.DEBUG_INCLUDES, "Loading Target Build File '%s'", build_file_path + ) + + build_file_data = LoadOneBuildFile( + build_file_path, data, aux_data, includes, True, check + ) + + # Store DEPTH for later use in generators. + build_file_data["_DEPTH"] = depth + + # Set up the included_files key indicating which .gyp files contributed to + # this target dict. + if "included_files" in build_file_data: + raise GypError(build_file_path + " must not contain included_files key") + + included = GetIncludedBuildFiles(build_file_path, aux_data) + build_file_data["included_files"] = [] + for included_file in included: + # included_file is relative to the current directory, but it needs to + # be made relative to build_file_path's directory. + included_relative = gyp.common.RelativePath( + included_file, os.path.dirname(build_file_path) + ) + build_file_data["included_files"].append(included_relative) + + # Do a first round of toolsets expansion so that conditions can be defined + # per toolset. + ProcessToolsetsInDict(build_file_data) + + # Apply "pre"/"early" variable expansions and condition evaluations. + ProcessVariablesAndConditionsInDict( + build_file_data, PHASE_EARLY, variables, build_file_path + ) + + # Since some toolsets might have been defined conditionally, perform + # a second round of toolsets expansion now. + ProcessToolsetsInDict(build_file_data) + + # Look at each project's target_defaults dict, and merge settings into + # targets. + if "target_defaults" in build_file_data: + if "targets" not in build_file_data: + raise GypError("Unable to find targets in build file %s" % build_file_path) + + index = 0 + while index < len(build_file_data["targets"]): + # This procedure needs to give the impression that target_defaults is + # used as defaults, and the individual targets inherit from that. + # The individual targets need to be merged into the defaults. Make + # a deep copy of the defaults for each target, merge the target dict + # as found in the input file into that copy, and then hook up the + # copy with the target-specific data merged into it as the replacement + # target dict. + old_target_dict = build_file_data["targets"][index] + new_target_dict = gyp.simple_copy.deepcopy( + build_file_data["target_defaults"] + ) + MergeDicts( + new_target_dict, old_target_dict, build_file_path, build_file_path + ) + build_file_data["targets"][index] = new_target_dict + index += 1 + + # No longer needed. + del build_file_data["target_defaults"] + + # Look for dependencies. This means that dependency resolution occurs + # after "pre" conditionals and variable expansion, but before "post" - + # in other words, you can't put a "dependencies" section inside a "post" + # conditional within a target. + + dependencies = [] + if "targets" in build_file_data: + for target_dict in build_file_data["targets"]: + if "dependencies" not in target_dict: + continue + for dependency in target_dict["dependencies"]: + dependencies.append( + gyp.common.ResolveTarget(build_file_path, dependency, None)[0] + ) + + if load_dependencies: + for dependency in dependencies: + try: + LoadTargetBuildFile( + dependency, + data, + aux_data, + variables, + includes, + depth, + check, + load_dependencies, + ) + except Exception as e: + gyp.common.ExceptionAppend( + e, "while loading dependencies of %s" % build_file_path + ) + raise else: - variables['DEPTH'] = d.replace('\\', '/') - - # The 'target_build_files' key is only set when loading target build files in - # the non-parallel code path, where LoadTargetBuildFile is called - # recursively. In the parallel code path, we don't need to check whether the - # |build_file_path| has already been loaded, because the 'scheduled' set in - # ParallelState guarantees that we never load the same |build_file_path| - # twice. - if 'target_build_files' in data: - if build_file_path in data['target_build_files']: - # Already loaded. - return False - data['target_build_files'].add(build_file_path) - - gyp.DebugOutput(gyp.DEBUG_INCLUDES, - "Loading Target Build File '%s'", build_file_path) - - build_file_data = LoadOneBuildFile(build_file_path, data, aux_data, - includes, True, check) - - # Store DEPTH for later use in generators. - build_file_data['_DEPTH'] = depth - - # Set up the included_files key indicating which .gyp files contributed to - # this target dict. - if 'included_files' in build_file_data: - raise GypError(build_file_path + ' must not contain included_files key') - - included = GetIncludedBuildFiles(build_file_path, aux_data) - build_file_data['included_files'] = [] - for included_file in included: - # included_file is relative to the current directory, but it needs to - # be made relative to build_file_path's directory. - included_relative = \ - gyp.common.RelativePath(included_file, - os.path.dirname(build_file_path)) - build_file_data['included_files'].append(included_relative) - - # Do a first round of toolsets expansion so that conditions can be defined - # per toolset. - ProcessToolsetsInDict(build_file_data) - - # Apply "pre"/"early" variable expansions and condition evaluations. - ProcessVariablesAndConditionsInDict( - build_file_data, PHASE_EARLY, variables, build_file_path) - - # Since some toolsets might have been defined conditionally, perform - # a second round of toolsets expansion now. - ProcessToolsetsInDict(build_file_data) - - # Look at each project's target_defaults dict, and merge settings into - # targets. - if 'target_defaults' in build_file_data: - if 'targets' not in build_file_data: - raise GypError("Unable to find targets in build file %s" % - build_file_path) + return (build_file_path, dependencies) - index = 0 - while index < len(build_file_data['targets']): - # This procedure needs to give the impression that target_defaults is - # used as defaults, and the individual targets inherit from that. - # The individual targets need to be merged into the defaults. Make - # a deep copy of the defaults for each target, merge the target dict - # as found in the input file into that copy, and then hook up the - # copy with the target-specific data merged into it as the replacement - # target dict. - old_target_dict = build_file_data['targets'][index] - new_target_dict = gyp.simple_copy.deepcopy( - build_file_data['target_defaults']) - MergeDicts(new_target_dict, old_target_dict, - build_file_path, build_file_path) - build_file_data['targets'][index] = new_target_dict - index += 1 - - # No longer needed. - del build_file_data['target_defaults'] - - # Look for dependencies. This means that dependency resolution occurs - # after "pre" conditionals and variable expansion, but before "post" - - # in other words, you can't put a "dependencies" section inside a "post" - # conditional within a target. - - dependencies = [] - if 'targets' in build_file_data: - for target_dict in build_file_data['targets']: - if 'dependencies' not in target_dict: - continue - for dependency in target_dict['dependencies']: - dependencies.append( - gyp.common.ResolveTarget(build_file_path, dependency, None)[0]) - - if load_dependencies: - for dependency in dependencies: - try: - LoadTargetBuildFile(dependency, data, aux_data, variables, - includes, depth, check, load_dependencies) - except Exception as e: - gyp.common.ExceptionAppend( - e, 'while loading dependencies of %s' % build_file_path) - raise - else: - return (build_file_path, dependencies) -def CallLoadTargetBuildFile(global_flags, - build_file_path, variables, - includes, depth, check, - generator_input_info): - """Wrapper around LoadTargetBuildFile for parallel processing. +def CallLoadTargetBuildFile( + global_flags, + build_file_path, + variables, + includes, + depth, + check, + generator_input_info, +): + """Wrapper around LoadTargetBuildFile for parallel processing. This wrapper is used when LoadTargetBuildFile is executed in a worker process. """ - try: - signal.signal(signal.SIGINT, signal.SIG_IGN) - - # Apply globals so that the worker process behaves the same. - for key, value in global_flags.items(): - globals()[key] = value - - SetGeneratorGlobals(generator_input_info) - result = LoadTargetBuildFile(build_file_path, per_process_data, - per_process_aux_data, variables, - includes, depth, check, False) - if not result: - return result - - (build_file_path, dependencies) = result - - # We can safely pop the build_file_data from per_process_data because it - # will never be referenced by this process again, so we don't need to keep - # it in the cache. - build_file_data = per_process_data.pop(build_file_path) - - # This gets serialized and sent back to the main process via a pipe. - # It's handled in LoadTargetBuildFileCallback. - return (build_file_path, - build_file_data, - dependencies) - except GypError as e: - sys.stderr.write("gyp: %s\n" % e) - return None - except Exception as e: - print('Exception:', e, file=sys.stderr) - print(traceback.format_exc(), file=sys.stderr) - return None + try: + signal.signal(signal.SIGINT, signal.SIG_IGN) + + # Apply globals so that the worker process behaves the same. + for key, value in global_flags.items(): + globals()[key] = value + + SetGeneratorGlobals(generator_input_info) + result = LoadTargetBuildFile( + build_file_path, + per_process_data, + per_process_aux_data, + variables, + includes, + depth, + check, + False, + ) + if not result: + return result + + (build_file_path, dependencies) = result + + # We can safely pop the build_file_data from per_process_data because it + # will never be referenced by this process again, so we don't need to keep + # it in the cache. + build_file_data = per_process_data.pop(build_file_path) + + # This gets serialized and sent back to the main process via a pipe. + # It's handled in LoadTargetBuildFileCallback. + return (build_file_path, build_file_data, dependencies) + except GypError as e: + sys.stderr.write("gyp: %s\n" % e) + return None + except Exception as e: + print("Exception:", e, file=sys.stderr) + print(traceback.format_exc(), file=sys.stderr) + return None class ParallelProcessingError(Exception): - pass + pass class ParallelState(object): - """Class to keep track of state when processing input files in parallel. + """Class to keep track of state when processing input files in parallel. If build files are loaded in parallel, use this to keep track of state during farming out and processing parallel jobs. It's stored in a global so that the callback function can have access to it. """ - def __init__(self): - # The multiprocessing pool. - self.pool = None - # The condition variable used to protect this object and notify - # the main loop when there might be more data to process. - self.condition = None - # The "data" dict that was passed to LoadTargetBuildFileParallel - self.data = None - # The number of parallel calls outstanding; decremented when a response - # was received. - self.pending = 0 - # The set of all build files that have been scheduled, so we don't - # schedule the same one twice. - self.scheduled = set() - # A list of dependency build file paths that haven't been scheduled yet. - self.dependencies = [] - # Flag to indicate if there was an error in a child process. - self.error = False - - def LoadTargetBuildFileCallback(self, result): - """Handle the results of running LoadTargetBuildFile in another process. + def __init__(self): + # The multiprocessing pool. + self.pool = None + # The condition variable used to protect this object and notify + # the main loop when there might be more data to process. + self.condition = None + # The "data" dict that was passed to LoadTargetBuildFileParallel + self.data = None + # The number of parallel calls outstanding; decremented when a response + # was received. + self.pending = 0 + # The set of all build files that have been scheduled, so we don't + # schedule the same one twice. + self.scheduled = set() + # A list of dependency build file paths that haven't been scheduled yet. + self.dependencies = [] + # Flag to indicate if there was an error in a child process. + self.error = False + + def LoadTargetBuildFileCallback(self, result): + """Handle the results of running LoadTargetBuildFile in another process. """ - self.condition.acquire() - if not result: - self.error = True - self.condition.notify() - self.condition.release() - return - (build_file_path0, build_file_data0, dependencies0) = result - self.data[build_file_path0] = build_file_data0 - self.data['target_build_files'].add(build_file_path0) - for new_dependency in dependencies0: - if new_dependency not in self.scheduled: - self.scheduled.add(new_dependency) - self.dependencies.append(new_dependency) - self.pending -= 1 - self.condition.notify() - self.condition.release() - - -def LoadTargetBuildFilesParallel(build_files, data, variables, includes, depth, - check, generator_input_info): - parallel_state = ParallelState() - parallel_state.condition = threading.Condition() - # Make copies of the build_files argument that we can modify while working. - parallel_state.dependencies = list(build_files) - parallel_state.scheduled = set(build_files) - parallel_state.pending = 0 - parallel_state.data = data - - try: - parallel_state.condition.acquire() - while parallel_state.dependencies or parallel_state.pending: - if parallel_state.error: - break - if not parallel_state.dependencies: - parallel_state.condition.wait() - continue - - dependency = parallel_state.dependencies.pop() - - parallel_state.pending += 1 - global_flags = { - 'path_sections': globals()['path_sections'], - 'non_configuration_keys': globals()['non_configuration_keys'], - 'multiple_toolsets': globals()['multiple_toolsets']} - - if not parallel_state.pool: - parallel_state.pool = multiprocessing.Pool(multiprocessing.cpu_count()) - parallel_state.pool.apply_async( - CallLoadTargetBuildFile, - args = (global_flags, dependency, - variables, includes, depth, check, generator_input_info), - callback = parallel_state.LoadTargetBuildFileCallback) - except KeyboardInterrupt as e: - parallel_state.pool.terminate() - raise e - - parallel_state.condition.release() - - parallel_state.pool.close() - parallel_state.pool.join() - parallel_state.pool = None - - if parallel_state.error: - sys.exit(1) + self.condition.acquire() + if not result: + self.error = True + self.condition.notify() + self.condition.release() + return + (build_file_path0, build_file_data0, dependencies0) = result + self.data[build_file_path0] = build_file_data0 + self.data["target_build_files"].add(build_file_path0) + for new_dependency in dependencies0: + if new_dependency not in self.scheduled: + self.scheduled.add(new_dependency) + self.dependencies.append(new_dependency) + self.pending -= 1 + self.condition.notify() + self.condition.release() + + +def LoadTargetBuildFilesParallel( + build_files, data, variables, includes, depth, check, generator_input_info +): + parallel_state = ParallelState() + parallel_state.condition = threading.Condition() + # Make copies of the build_files argument that we can modify while working. + parallel_state.dependencies = list(build_files) + parallel_state.scheduled = set(build_files) + parallel_state.pending = 0 + parallel_state.data = data + + try: + parallel_state.condition.acquire() + while parallel_state.dependencies or parallel_state.pending: + if parallel_state.error: + break + if not parallel_state.dependencies: + parallel_state.condition.wait() + continue + + dependency = parallel_state.dependencies.pop() + + parallel_state.pending += 1 + global_flags = { + "path_sections": globals()["path_sections"], + "non_configuration_keys": globals()["non_configuration_keys"], + "multiple_toolsets": globals()["multiple_toolsets"], + } + + if not parallel_state.pool: + parallel_state.pool = multiprocessing.Pool(multiprocessing.cpu_count()) + parallel_state.pool.apply_async( + CallLoadTargetBuildFile, + args=( + global_flags, + dependency, + variables, + includes, + depth, + check, + generator_input_info, + ), + callback=parallel_state.LoadTargetBuildFileCallback, + ) + except KeyboardInterrupt as e: + parallel_state.pool.terminate() + raise e + + parallel_state.condition.release() + + parallel_state.pool.close() + parallel_state.pool.join() + parallel_state.pool = None + + if parallel_state.error: + sys.exit(1) + # Look for the bracket that matches the first bracket seen in a # string, and return the start and end as a tuple. For example, if # the input is something like "<(foo <(bar)) blah", then it would # return (1, 13), indicating the entire string except for the leading # "<" and trailing " blah". -LBRACKETS= set('{[(') -BRACKETS = {'}': '{', ']': '[', ')': '('} +LBRACKETS = set("{[(") +BRACKETS = {"}": "{", "]": "[", ")": "("} + + def FindEnclosingBracketGroup(input_str): - stack = [] - start = -1 - for index, char in enumerate(input_str): - if char in LBRACKETS: - stack.append(char) - if start == -1: - start = index - elif char in BRACKETS: - if not stack: - return (-1, -1) - if stack.pop() != BRACKETS[char]: - return (-1, -1) - if not stack: - return (start, index + 1) - return (-1, -1) + stack = [] + start = -1 + for index, char in enumerate(input_str): + if char in LBRACKETS: + stack.append(char) + if start == -1: + start = index + elif char in BRACKETS: + if not stack: + return (-1, -1) + if stack.pop() != BRACKETS[char]: + return (-1, -1) + if not stack: + return (start, index + 1) + return (-1, -1) def IsStrCanonicalInt(string): - """Returns True if |string| is in its canonical integer form. + """Returns True if |string| is in its canonical integer form. The canonical form is such that str(int(string)) == string. """ - if type(string) is str: - # This function is called a lot so for maximum performance, avoid - # involving regexps which would otherwise make the code much - # shorter. Regexps would need twice the time of this function. - if string: - if string == "0": - return True - if string[0] == "-": - string = string[1:] - if not string: - return False - if '1' <= string[0] <= '9': - return string.isdigit() - - return False + if type(string) is str: + # This function is called a lot so for maximum performance, avoid + # involving regexps which would otherwise make the code much + # shorter. Regexps would need twice the time of this function. + if string: + if string == "0": + return True + if string[0] == "-": + string = string[1:] + if not string: + return False + if "1" <= string[0] <= "9": + return string.isdigit() + + return False # This matches things like "<(asdf)", "(?P<(?:(?:!?@?)|\|)?)' - r'(?P[-a-zA-Z0-9_.]+)?' - r'\((?P\s*\[?)' - r'(?P.*?)(\]?)\))') + r"(?P(?P<(?:(?:!?@?)|\|)?)" + r"(?P[-a-zA-Z0-9_.]+)?" + r"\((?P\s*\[?)" + r"(?P.*?)(\]?)\))" +) # This matches the same as early_variable_re, but with '>' instead of '<'. late_variable_re = re.compile( - r'(?P(?P>(?:(?:!?@?)|\|)?)' - r'(?P[-a-zA-Z0-9_.]+)?' - r'\((?P\s*\[?)' - r'(?P.*?)(\]?)\))') + r"(?P(?P>(?:(?:!?@?)|\|)?)" + r"(?P[-a-zA-Z0-9_.]+)?" + r"\((?P\s*\[?)" + r"(?P.*?)(\]?)\))" +) # This matches the same as early_variable_re, but with '^' instead of '<'. latelate_variable_re = re.compile( - r'(?P(?P[\^](?:(?:!?@?)|\|)?)' - r'(?P[-a-zA-Z0-9_.]+)?' - r'\((?P\s*\[?)' - r'(?P.*?)(\]?)\))') + r"(?P(?P[\^](?:(?:!?@?)|\|)?)" + r"(?P[-a-zA-Z0-9_.]+)?" + r"\((?P\s*\[?)" + r"(?P.*?)(\]?)\))" +) # Global cache of results from running commands so they don't have to be run # more then once. @@ -694,12 +754,12 @@ def IsStrCanonicalInt(string): def FixupPlatformCommand(cmd): - if sys.platform == 'win32': - if type(cmd) is list: - cmd = [re.sub('^cat ', 'type ', cmd[0])] + cmd[1:] - else: - cmd = re.sub('^cat ', 'type ', cmd) - return cmd + if sys.platform == "win32": + if type(cmd) is list: + cmd = [re.sub("^cat ", "type ", cmd[0])] + cmd[1:] + else: + cmd = re.sub("^cat ", "type ", cmd) + return cmd PHASE_EARLY = 0 @@ -708,640 +768,702 @@ def FixupPlatformCommand(cmd): def ExpandVariables(input, phase, variables, build_file): - # Look for the pattern that gets expanded into variables - if phase == PHASE_EARLY: - variable_re = early_variable_re - expansion_symbol = '<' - elif phase == PHASE_LATE: - variable_re = late_variable_re - expansion_symbol = '>' - elif phase == PHASE_LATELATE: - variable_re = latelate_variable_re - expansion_symbol = '^' - else: - assert False - - input_str = str(input) - if IsStrCanonicalInt(input_str): - return int(input_str) - - # Do a quick scan to determine if an expensive regex search is warranted. - if expansion_symbol not in input_str: - return input_str - - # Get the entire list of matches as a list of MatchObject instances. - # (using findall here would return strings instead of MatchObjects). - matches = list(variable_re.finditer(input_str)) - if not matches: - return input_str - - output = input_str - # Reverse the list of matches so that replacements are done right-to-left. - # That ensures that earlier replacements won't mess up the string in a - # way that causes later calls to find the earlier substituted text instead - # of what's intended for replacement. - matches.reverse() - for match_group in matches: - match = match_group.groupdict() - gyp.DebugOutput(gyp.DEBUG_VARIABLES, "Matches: %r", match) - # match['replace'] is the substring to look for, match['type'] - # is the character code for the replacement type (< > ! <| >| <@ - # >@ !@), match['is_array'] contains a '[' for command - # arrays, and match['content'] is the name of the variable (< >) - # or command to run (!). match['command_string'] is an optional - # command string. Currently, only 'pymod_do_main' is supported. - - # run_command is true if a ! variant is used. - run_command = '!' in match['type'] - command_string = match['command_string'] - - # file_list is true if a | variant is used. - file_list = '|' in match['type'] - - # Capture these now so we can adjust them later. - replace_start = match_group.start('replace') - replace_end = match_group.end('replace') - - # Find the ending paren, and re-evaluate the contained string. - (c_start, c_end) = FindEnclosingBracketGroup(input_str[replace_start:]) - - # Adjust the replacement range to match the entire command - # found by FindEnclosingBracketGroup (since the variable_re - # probably doesn't match the entire command if it contained - # nested variables). - replace_end = replace_start + c_end - - # Find the "real" replacement, matching the appropriate closing - # paren, and adjust the replacement start and end. - replacement = input_str[replace_start:replace_end] - - # Figure out what the contents of the variable parens are. - contents_start = replace_start + c_start + 1 - contents_end = replace_end - 1 - contents = input_str[contents_start:contents_end] - - # Do filter substitution now for <|(). - # Admittedly, this is different than the evaluation order in other - # contexts. However, since filtration has no chance to run on <|(), - # this seems like the only obvious way to give them access to filters. - if file_list: - processed_variables = gyp.simple_copy.deepcopy(variables) - ProcessListFiltersInDict(contents, processed_variables) - # Recurse to expand variables in the contents - contents = ExpandVariables(contents, phase, - processed_variables, build_file) + # Look for the pattern that gets expanded into variables + if phase == PHASE_EARLY: + variable_re = early_variable_re + expansion_symbol = "<" + elif phase == PHASE_LATE: + variable_re = late_variable_re + expansion_symbol = ">" + elif phase == PHASE_LATELATE: + variable_re = latelate_variable_re + expansion_symbol = "^" else: - # Recurse to expand variables in the contents - contents = ExpandVariables(contents, phase, variables, build_file) - - # Strip off leading/trailing whitespace so that variable matches are - # simpler below (and because they are rarely needed). - contents = contents.strip() - - # expand_to_list is true if an @ variant is used. In that case, - # the expansion should result in a list. Note that the caller - # is to be expecting a list in return, and not all callers do - # because not all are working in list context. Also, for list - # expansions, there can be no other text besides the variable - # expansion in the input string. - expand_to_list = '@' in match['type'] and input_str == replacement - - if run_command or file_list: - # Find the build file's directory, so commands can be run or file lists - # generated relative to it. - build_file_dir = os.path.dirname(build_file) - if build_file_dir == '' and not file_list: - # If build_file is just a leaf filename indicating a file in the - # current directory, build_file_dir might be an empty string. Set - # it to None to signal to subprocess.Popen that it should run the - # command in the current directory. - build_file_dir = None - - # Support <|(listfile.txt ...) which generates a file - # containing items from a gyp list, generated at gyp time. - # This works around actions/rules which have more inputs than will - # fit on the command line. - if file_list: - if type(contents) is list: - contents_list = contents - else: - contents_list = contents.split(' ') - replacement = contents_list[0] - if os.path.isabs(replacement): - raise GypError('| cannot handle absolute paths, got "%s"' % replacement) - - if not generator_filelist_paths: - path = os.path.join(build_file_dir, replacement) - else: - if os.path.isabs(build_file_dir): - toplevel = generator_filelist_paths['toplevel'] - rel_build_file_dir = gyp.common.RelativePath(build_file_dir, toplevel) + assert False + + input_str = str(input) + if IsStrCanonicalInt(input_str): + return int(input_str) + + # Do a quick scan to determine if an expensive regex search is warranted. + if expansion_symbol not in input_str: + return input_str + + # Get the entire list of matches as a list of MatchObject instances. + # (using findall here would return strings instead of MatchObjects). + matches = list(variable_re.finditer(input_str)) + if not matches: + return input_str + + output = input_str + # Reverse the list of matches so that replacements are done right-to-left. + # That ensures that earlier replacements won't mess up the string in a + # way that causes later calls to find the earlier substituted text instead + # of what's intended for replacement. + matches.reverse() + for match_group in matches: + match = match_group.groupdict() + gyp.DebugOutput(gyp.DEBUG_VARIABLES, "Matches: %r", match) + # match['replace'] is the substring to look for, match['type'] + # is the character code for the replacement type (< > ! <| >| <@ + # >@ !@), match['is_array'] contains a '[' for command + # arrays, and match['content'] is the name of the variable (< >) + # or command to run (!). match['command_string'] is an optional + # command string. Currently, only 'pymod_do_main' is supported. + + # run_command is true if a ! variant is used. + run_command = "!" in match["type"] + command_string = match["command_string"] + + # file_list is true if a | variant is used. + file_list = "|" in match["type"] + + # Capture these now so we can adjust them later. + replace_start = match_group.start("replace") + replace_end = match_group.end("replace") + + # Find the ending paren, and re-evaluate the contained string. + (c_start, c_end) = FindEnclosingBracketGroup(input_str[replace_start:]) + + # Adjust the replacement range to match the entire command + # found by FindEnclosingBracketGroup (since the variable_re + # probably doesn't match the entire command if it contained + # nested variables). + replace_end = replace_start + c_end + + # Find the "real" replacement, matching the appropriate closing + # paren, and adjust the replacement start and end. + replacement = input_str[replace_start:replace_end] + + # Figure out what the contents of the variable parens are. + contents_start = replace_start + c_start + 1 + contents_end = replace_end - 1 + contents = input_str[contents_start:contents_end] + + # Do filter substitution now for <|(). + # Admittedly, this is different than the evaluation order in other + # contexts. However, since filtration has no chance to run on <|(), + # this seems like the only obvious way to give them access to filters. + if file_list: + processed_variables = gyp.simple_copy.deepcopy(variables) + ProcessListFiltersInDict(contents, processed_variables) + # Recurse to expand variables in the contents + contents = ExpandVariables(contents, phase, processed_variables, build_file) else: - rel_build_file_dir = build_file_dir - qualified_out_dir = generator_filelist_paths['qualified_out_dir'] - path = os.path.join(qualified_out_dir, rel_build_file_dir, replacement) - gyp.common.EnsureDirExists(path) - - replacement = gyp.common.RelativePath(path, build_file_dir) - f = gyp.common.WriteOnDiff(path) - for i in contents_list[1:]: - f.write('%s\n' % i) - f.close() - - elif run_command: - use_shell = True - if match['is_array']: - contents = eval(contents) - use_shell = False - - # Check for a cached value to avoid executing commands, or generating - # file lists more than once. The cache key contains the command to be - # run as well as the directory to run it from, to account for commands - # that depend on their current directory. - # TODO(http://code.google.com/p/gyp/issues/detail?id=111): In theory, - # someone could author a set of GYP files where each time the command - # is invoked it produces different output by design. When the need - # arises, the syntax should be extended to support no caching off a - # command's output so it is run every time. - cache_key = (str(contents), build_file_dir) - cached_value = cached_command_results.get(cache_key, None) - if cached_value is None: - gyp.DebugOutput(gyp.DEBUG_VARIABLES, - "Executing command '%s' in directory '%s'", - contents, build_file_dir) - - replacement = '' - - if command_string == 'pymod_do_main': - # (sources/) etc. to resolve to - # and empty list if undefined. This allows actions to: - # 'action!': [ - # '>@(_sources!)', - # ], - # 'action/': [ - # '>@(_sources/)', - # ], - replacement = [] else: - raise GypError('Undefined variable ' + contents + - ' in ' + build_file) - else: - replacement = variables[contents] - - if isinstance(replacement, bytes) and not isinstance(replacement, str): - replacement = replacement.decode("utf-8") # done on Python 3 only - if type(replacement) is list: - for item in replacement: - if isinstance(item, bytes) and not isinstance(item, str): - item = item.decode("utf-8") # done on Python 3 only - if not contents[-1] == '/' and type(item) not in (str, int): - raise GypError('Variable ' + contents + - ' must expand to a string or list of strings; ' + - 'list contains a ' + - item.__class__.__name__) - # Run through the list and handle variable expansions in it. Since - # the list is guaranteed not to contain dicts, this won't do anything - # with conditions sections. - ProcessVariablesAndConditionsInList(replacement, phase, variables, - build_file) - elif type(replacement) not in (str, int): - raise GypError('Variable ' + contents + - ' must expand to a string or list of strings; ' + - 'found a ' + replacement.__class__.__name__) - - if expand_to_list: - # Expanding in list context. It's guaranteed that there's only one - # replacement to do in |input_str| and that it's this replacement. See - # above. - if type(replacement) is list: - # If it's already a list, make a copy. - output = replacement[:] - else: - # Split it the same way sh would split arguments. - output = shlex.split(str(replacement)) + if contents not in variables: + if contents[-1] in ["!", "/"]: + # In order to allow cross-compiles (nacl) to happen more naturally, + # we will allow references to >(sources/) etc. to resolve to + # and empty list if undefined. This allows actions to: + # 'action!': [ + # '>@(_sources!)', + # ], + # 'action/': [ + # '>@(_sources/)', + # ], + replacement = [] + else: + raise GypError( + "Undefined variable " + contents + " in " + build_file + ) + else: + replacement = variables[contents] + + if isinstance(replacement, bytes) and not isinstance(replacement, str): + replacement = replacement.decode("utf-8") # done on Python 3 only + if type(replacement) is list: + for item in replacement: + if isinstance(item, bytes) and not isinstance(item, str): + item = item.decode("utf-8") # done on Python 3 only + if not contents[-1] == "/" and type(item) not in (str, int): + raise GypError( + "Variable " + + contents + + " must expand to a string or list of strings; " + + "list contains a " + + item.__class__.__name__ + ) + # Run through the list and handle variable expansions in it. Since + # the list is guaranteed not to contain dicts, this won't do anything + # with conditions sections. + ProcessVariablesAndConditionsInList( + replacement, phase, variables, build_file + ) + elif type(replacement) not in (str, int): + raise GypError( + "Variable " + + contents + + " must expand to a string or list of strings; " + + "found a " + + replacement.__class__.__name__ + ) + + if expand_to_list: + # Expanding in list context. It's guaranteed that there's only one + # replacement to do in |input_str| and that it's this replacement. See + # above. + if type(replacement) is list: + # If it's already a list, make a copy. + output = replacement[:] + else: + # Split it the same way sh would split arguments. + output = shlex.split(str(replacement)) + else: + # Expanding in string context. + encoded_replacement = "" + if type(replacement) is list: + # When expanding a list into string context, turn the list items + # into a string in a way that will work with a subprocess call. + # + # TODO(mark): This isn't completely correct. This should + # call a generator-provided function that observes the + # proper list-to-argument quoting rules on a specific + # platform instead of just calling the POSIX encoding + # routine. + encoded_replacement = gyp.common.EncodePOSIXShellList(replacement) + else: + encoded_replacement = replacement + + output = ( + output[:replace_start] + str(encoded_replacement) + output[replace_end:] + ) + # Prepare for the next match iteration. + input_str = output + + if output == input: + gyp.DebugOutput( + gyp.DEBUG_VARIABLES, + "Found only identity matches on %r, avoiding infinite " "recursion.", + output, + ) else: - # Expanding in string context. - encoded_replacement = '' - if type(replacement) is list: - # When expanding a list into string context, turn the list items - # into a string in a way that will work with a subprocess call. - # - # TODO(mark): This isn't completely correct. This should - # call a generator-provided function that observes the - # proper list-to-argument quoting rules on a specific - # platform instead of just calling the POSIX encoding - # routine. - encoded_replacement = gyp.common.EncodePOSIXShellList(replacement) - else: - encoded_replacement = replacement - - output = output[:replace_start] + str(encoded_replacement) + \ - output[replace_end:] - # Prepare for the next match iteration. - input_str = output - - if output == input: - gyp.DebugOutput(gyp.DEBUG_VARIABLES, - "Found only identity matches on %r, avoiding infinite " - "recursion.", - output) - else: - # Look for more matches now that we've replaced some, to deal with - # expanding local variables (variables defined in the same - # variables block as this one). - gyp.DebugOutput(gyp.DEBUG_VARIABLES, "Found output %r, recursing.", output) + # Look for more matches now that we've replaced some, to deal with + # expanding local variables (variables defined in the same + # variables block as this one). + gyp.DebugOutput(gyp.DEBUG_VARIABLES, "Found output %r, recursing.", output) + if type(output) is list: + if output and type(output[0]) is list: + # Leave output alone if it's a list of lists. + # We don't want such lists to be stringified. + pass + else: + new_output = [] + for item in output: + new_output.append( + ExpandVariables(item, phase, variables, build_file) + ) + output = new_output + else: + output = ExpandVariables(output, phase, variables, build_file) + + # Convert all strings that are canonically-represented integers into integers. if type(output) is list: - if output and type(output[0]) is list: - # Leave output alone if it's a list of lists. - # We don't want such lists to be stringified. - pass - else: - new_output = [] - for item in output: - new_output.append( - ExpandVariables(item, phase, variables, build_file)) - output = new_output - else: - output = ExpandVariables(output, phase, variables, build_file) + for index, outstr in enumerate(output): + if IsStrCanonicalInt(outstr): + output[index] = int(outstr) + elif IsStrCanonicalInt(output): + output = int(output) - # Convert all strings that are canonically-represented integers into integers. - if type(output) is list: - for index in range(0, len(output)): - if IsStrCanonicalInt(output[index]): - output[index] = int(output[index]) - elif IsStrCanonicalInt(output): - output = int(output) + return output - return output # The same condition is often evaluated over and over again so it # makes sense to cache as much as possible between evaluations. cached_conditions_asts = {} + def EvalCondition(condition, conditions_key, phase, variables, build_file): - """Returns the dict that should be used or None if the result was + """Returns the dict that should be used or None if the result was that nothing should be used.""" - if type(condition) is not list: - raise GypError(conditions_key + ' must be a list') - if len(condition) < 2: - # It's possible that condition[0] won't work in which case this - # attempt will raise its own IndexError. That's probably fine. - raise GypError(conditions_key + ' ' + condition[0] + - ' must be at least length 2, not ' + str(len(condition))) - - i = 0 - result = None - while i < len(condition): - cond_expr = condition[i] - true_dict = condition[i + 1] - if type(true_dict) is not dict: - raise GypError('{} {} must be followed by a dictionary, not {}'.format( - conditions_key, cond_expr, type(true_dict))) - if len(condition) > i + 2 and type(condition[i + 2]) is dict: - false_dict = condition[i + 2] - i = i + 3 - if i != len(condition): - raise GypError('{} {} has {} unexpected trailing items'.format( - conditions_key, cond_expr, len(condition) - i)) - else: - false_dict = None - i = i + 2 - if result is None: - result = EvalSingleCondition( - cond_expr, true_dict, false_dict, phase, variables, build_file) + if type(condition) is not list: + raise GypError(conditions_key + " must be a list") + if len(condition) < 2: + # It's possible that condition[0] won't work in which case this + # attempt will raise its own IndexError. That's probably fine. + raise GypError( + conditions_key + + " " + + condition[0] + + " must be at least length 2, not " + + str(len(condition)) + ) + + i = 0 + result = None + while i < len(condition): + cond_expr = condition[i] + true_dict = condition[i + 1] + if type(true_dict) is not dict: + raise GypError( + "{} {} must be followed by a dictionary, not {}".format( + conditions_key, cond_expr, type(true_dict) + ) + ) + if len(condition) > i + 2 and type(condition[i + 2]) is dict: + false_dict = condition[i + 2] + i = i + 3 + if i != len(condition): + raise GypError( + "{} {} has {} unexpected trailing items".format( + conditions_key, cond_expr, len(condition) - i + ) + ) + else: + false_dict = None + i = i + 2 + if result is None: + result = EvalSingleCondition( + cond_expr, true_dict, false_dict, phase, variables, build_file + ) - return result + return result -def EvalSingleCondition( - cond_expr, true_dict, false_dict, phase, variables, build_file): - """Returns true_dict if cond_expr evaluates to true, and false_dict +def EvalSingleCondition(cond_expr, true_dict, false_dict, phase, variables, build_file): + """Returns true_dict if cond_expr evaluates to true, and false_dict otherwise.""" - # Do expansions on the condition itself. Since the condition can naturally - # contain variable references without needing to resort to GYP expansion - # syntax, this is of dubious value for variables, but someone might want to - # use a command expansion directly inside a condition. - cond_expr_expanded = ExpandVariables(cond_expr, phase, variables, - build_file) - if type(cond_expr_expanded) not in (str, int): - raise ValueError( - 'Variable expansion in this context permits str and int ' + \ - 'only, found ' + cond_expr_expanded.__class__.__name__) - - try: - if cond_expr_expanded in cached_conditions_asts: - ast_code = cached_conditions_asts[cond_expr_expanded] - else: - ast_code = compile(cond_expr_expanded, '', 'eval') - cached_conditions_asts[cond_expr_expanded] = ast_code - env = {'__builtins__': {}, 'v': StrictVersion} - if eval(ast_code, env, variables): - return true_dict - return false_dict - except SyntaxError as e: - syntax_error = SyntaxError('%s while evaluating condition \'%s\' in %s ' - 'at character %d.' % - (str(e.args[0]), e.text, build_file, e.offset), - e.filename, e.lineno, e.offset, e.text) - raise syntax_error - except NameError as e: - gyp.common.ExceptionAppend(e, 'while evaluating condition \'%s\' in %s' % - (cond_expr_expanded, build_file)) - raise GypError(e) + # Do expansions on the condition itself. Since the condition can naturally + # contain variable references without needing to resort to GYP expansion + # syntax, this is of dubious value for variables, but someone might want to + # use a command expansion directly inside a condition. + cond_expr_expanded = ExpandVariables(cond_expr, phase, variables, build_file) + if type(cond_expr_expanded) not in (str, int): + raise ValueError( + "Variable expansion in this context permits str and int " + + "only, found " + + cond_expr_expanded.__class__.__name__ + ) + + try: + if cond_expr_expanded in cached_conditions_asts: + ast_code = cached_conditions_asts[cond_expr_expanded] + else: + ast_code = compile(cond_expr_expanded, "", "eval") + cached_conditions_asts[cond_expr_expanded] = ast_code + env = {"__builtins__": {}, "v": StrictVersion} + if eval(ast_code, env, variables): + return true_dict + return false_dict + except SyntaxError as e: + syntax_error = SyntaxError( + "%s while evaluating condition '%s' in %s " + "at character %d." % (str(e.args[0]), e.text, build_file, e.offset), + e.filename, + e.lineno, + e.offset, + e.text, + ) + raise syntax_error + except NameError as e: + gyp.common.ExceptionAppend( + e, + "while evaluating condition '%s' in %s" % (cond_expr_expanded, build_file), + ) + raise GypError(e) def ProcessConditionsInDict(the_dict, phase, variables, build_file): - # Process a 'conditions' or 'target_conditions' section in the_dict, - # depending on phase. - # early -> conditions - # late -> target_conditions - # latelate -> no conditions - # - # Each item in a conditions list consists of cond_expr, a string expression - # evaluated as the condition, and true_dict, a dict that will be merged into - # the_dict if cond_expr evaluates to true. Optionally, a third item, - # false_dict, may be present. false_dict is merged into the_dict if - # cond_expr evaluates to false. - # - # Any dict merged into the_dict will be recursively processed for nested - # conditionals and other expansions, also according to phase, immediately - # prior to being merged. - - if phase == PHASE_EARLY: - conditions_key = 'conditions' - elif phase == PHASE_LATE: - conditions_key = 'target_conditions' - elif phase == PHASE_LATELATE: - return - else: - assert False - - if not conditions_key in the_dict: - return - - conditions_list = the_dict[conditions_key] - # Unhook the conditions list, it's no longer needed. - del the_dict[conditions_key] - - for condition in conditions_list: - merge_dict = EvalCondition(condition, conditions_key, phase, variables, - build_file) - - if merge_dict != None: - # Expand variables and nested conditinals in the merge_dict before - # merging it. - ProcessVariablesAndConditionsInDict(merge_dict, phase, - variables, build_file) - - MergeDicts(the_dict, merge_dict, build_file, build_file) + # Process a 'conditions' or 'target_conditions' section in the_dict, + # depending on phase. + # early -> conditions + # late -> target_conditions + # latelate -> no conditions + # + # Each item in a conditions list consists of cond_expr, a string expression + # evaluated as the condition, and true_dict, a dict that will be merged into + # the_dict if cond_expr evaluates to true. Optionally, a third item, + # false_dict, may be present. false_dict is merged into the_dict if + # cond_expr evaluates to false. + # + # Any dict merged into the_dict will be recursively processed for nested + # conditionals and other expansions, also according to phase, immediately + # prior to being merged. + + if phase == PHASE_EARLY: + conditions_key = "conditions" + elif phase == PHASE_LATE: + conditions_key = "target_conditions" + elif phase == PHASE_LATELATE: + return + else: + assert False + + if conditions_key not in the_dict: + return + + conditions_list = the_dict[conditions_key] + # Unhook the conditions list, it's no longer needed. + del the_dict[conditions_key] + + for condition in conditions_list: + merge_dict = EvalCondition( + condition, conditions_key, phase, variables, build_file + ) + + if merge_dict is not None: + # Expand variables and nested conditinals in the merge_dict before + # merging it. + ProcessVariablesAndConditionsInDict( + merge_dict, phase, variables, build_file + ) + + MergeDicts(the_dict, merge_dict, build_file, build_file) def LoadAutomaticVariablesFromDict(variables, the_dict): - # Any keys with plain string values in the_dict become automatic variables. - # The variable name is the key name with a "_" character prepended. - for key, value in the_dict.items(): - if type(value) in (str, int, list): - variables['_' + key] = value + # Any keys with plain string values in the_dict become automatic variables. + # The variable name is the key name with a "_" character prepended. + for key, value in the_dict.items(): + if type(value) in (str, int, list): + variables["_" + key] = value def LoadVariablesFromVariablesDict(variables, the_dict, the_dict_key): - # Any keys in the_dict's "variables" dict, if it has one, becomes a - # variable. The variable name is the key name in the "variables" dict. - # Variables that end with the % character are set only if they are unset in - # the variables dict. the_dict_key is the name of the key that accesses - # the_dict in the_dict's parent dict. If the_dict's parent is not a dict - # (it could be a list or it could be parentless because it is a root dict), - # the_dict_key will be None. - for key, value in the_dict.get('variables', {}).items(): - if type(value) not in (str, int, list): - continue - - if key.endswith('%'): - variable_name = key[:-1] - if variable_name in variables: - # If the variable is already set, don't set it. - continue - if the_dict_key == 'variables' and variable_name in the_dict: - # If the variable is set without a % in the_dict, and the_dict is a - # variables dict (making |variables| a variables sub-dict of a - # variables dict), use the_dict's definition. - value = the_dict[variable_name] - else: - variable_name = key + # Any keys in the_dict's "variables" dict, if it has one, becomes a + # variable. The variable name is the key name in the "variables" dict. + # Variables that end with the % character are set only if they are unset in + # the variables dict. the_dict_key is the name of the key that accesses + # the_dict in the_dict's parent dict. If the_dict's parent is not a dict + # (it could be a list or it could be parentless because it is a root dict), + # the_dict_key will be None. + for key, value in the_dict.get("variables", {}).items(): + if type(value) not in (str, int, list): + continue + + if key.endswith("%"): + variable_name = key[:-1] + if variable_name in variables: + # If the variable is already set, don't set it. + continue + if the_dict_key == "variables" and variable_name in the_dict: + # If the variable is set without a % in the_dict, and the_dict is a + # variables dict (making |variables| a variables sub-dict of a + # variables dict), use the_dict's definition. + value = the_dict[variable_name] + else: + variable_name = key - variables[variable_name] = value + variables[variable_name] = value -def ProcessVariablesAndConditionsInDict(the_dict, phase, variables_in, - build_file, the_dict_key=None): - """Handle all variable and command expansion and conditional evaluation. +def ProcessVariablesAndConditionsInDict( + the_dict, phase, variables_in, build_file, the_dict_key=None +): + """Handle all variable and command expansion and conditional evaluation. This function is the public entry point for all variable expansions and conditional evaluations. The variables_in dictionary will not be modified by this function. """ - # Make a copy of the variables_in dict that can be modified during the - # loading of automatics and the loading of the variables dict. - variables = variables_in.copy() - LoadAutomaticVariablesFromDict(variables, the_dict) - - if 'variables' in the_dict: - # Make sure all the local variables are added to the variables - # list before we process them so that you can reference one - # variable from another. They will be fully expanded by recursion - # in ExpandVariables. - for key, value in the_dict['variables'].items(): - variables[key] = value - - # Handle the associated variables dict first, so that any variable - # references within can be resolved prior to using them as variables. - # Pass a copy of the variables dict to avoid having it be tainted. - # Otherwise, it would have extra automatics added for everything that - # should just be an ordinary variable in this scope. - ProcessVariablesAndConditionsInDict(the_dict['variables'], phase, - variables, build_file, 'variables') - - LoadVariablesFromVariablesDict(variables, the_dict, the_dict_key) - - for key, value in the_dict.items(): - # Skip "variables", which was already processed if present. - if key != 'variables' and type(value) is str: - expanded = ExpandVariables(value, phase, variables, build_file) - if type(expanded) not in (str, int): - raise ValueError( - 'Variable expansion in this context permits str and int ' + \ - 'only, found ' + expanded.__class__.__name__ + ' for ' + key) - the_dict[key] = expanded - - # Variable expansion may have resulted in changes to automatics. Reload. - # TODO(mark): Optimization: only reload if no changes were made. - variables = variables_in.copy() - LoadAutomaticVariablesFromDict(variables, the_dict) - LoadVariablesFromVariablesDict(variables, the_dict, the_dict_key) - - # Process conditions in this dict. This is done after variable expansion - # so that conditions may take advantage of expanded variables. For example, - # if the_dict contains: - # {'type': '<(library_type)', - # 'conditions': [['_type=="static_library"', { ... }]]}, - # _type, as used in the condition, will only be set to the value of - # library_type if variable expansion is performed before condition - # processing. However, condition processing should occur prior to recursion - # so that variables (both automatic and "variables" dict type) may be - # adjusted by conditions sections, merged into the_dict, and have the - # intended impact on contained dicts. - # - # This arrangement means that a "conditions" section containing a "variables" - # section will only have those variables effective in subdicts, not in - # the_dict. The workaround is to put a "conditions" section within a - # "variables" section. For example: - # {'conditions': [['os=="mac"', {'variables': {'define': 'IS_MAC'}}]], - # 'defines': ['<(define)'], - # 'my_subdict': {'defines': ['<(define)']}}, - # will not result in "IS_MAC" being appended to the "defines" list in the - # current scope but would result in it being appended to the "defines" list - # within "my_subdict". By comparison: - # {'variables': {'conditions': [['os=="mac"', {'define': 'IS_MAC'}]]}, - # 'defines': ['<(define)'], - # 'my_subdict': {'defines': ['<(define)']}}, - # will append "IS_MAC" to both "defines" lists. - - # Evaluate conditions sections, allowing variable expansions within them - # as well as nested conditionals. This will process a 'conditions' or - # 'target_conditions' section, perform appropriate merging and recursive - # conditional and variable processing, and then remove the conditions section - # from the_dict if it is present. - ProcessConditionsInDict(the_dict, phase, variables, build_file) - - # Conditional processing may have resulted in changes to automatics or the - # variables dict. Reload. - variables = variables_in.copy() - LoadAutomaticVariablesFromDict(variables, the_dict) - LoadVariablesFromVariablesDict(variables, the_dict, the_dict_key) - - # Recurse into child dicts, or process child lists which may result in - # further recursion into descendant dicts. - for key, value in the_dict.items(): - # Skip "variables" and string values, which were already processed if - # present. - if key == 'variables' or type(value) is str: - continue - if type(value) is dict: - # Pass a copy of the variables dict so that subdicts can't influence - # parents. - ProcessVariablesAndConditionsInDict(value, phase, variables, - build_file, key) - elif type(value) is list: - # The list itself can't influence the variables dict, and - # ProcessVariablesAndConditionsInList will make copies of the variables - # dict if it needs to pass it to something that can influence it. No - # copy is necessary here. - ProcessVariablesAndConditionsInList(value, phase, variables, - build_file) - elif type(value) is not int: - raise TypeError('Unknown type ' + value.__class__.__name__ + \ - ' for ' + key) - - -def ProcessVariablesAndConditionsInList(the_list, phase, variables, - build_file): - # Iterate using an index so that new values can be assigned into the_list. - index = 0 - while index < len(the_list): - item = the_list[index] - if type(item) is dict: - # Make a copy of the variables dict so that it won't influence anything - # outside of its own scope. - ProcessVariablesAndConditionsInDict(item, phase, variables, build_file) - elif type(item) is list: - ProcessVariablesAndConditionsInList(item, phase, variables, build_file) - elif type(item) is str: - expanded = ExpandVariables(item, phase, variables, build_file) - if type(expanded) in (str, int): - the_list[index] = expanded - elif type(expanded) is list: - the_list[index:index+1] = expanded - index += len(expanded) - - # index now identifies the next item to examine. Continue right now - # without falling into the index increment below. - continue - else: - raise ValueError( - 'Variable expansion in this context permits strings and ' + \ - 'lists only, found ' + expanded.__class__.__name__ + ' at ' + \ - index) - elif type(item) is not int: - raise TypeError('Unknown type ' + item.__class__.__name__ + \ - ' at index ' + index) - index = index + 1 + # Make a copy of the variables_in dict that can be modified during the + # loading of automatics and the loading of the variables dict. + variables = variables_in.copy() + LoadAutomaticVariablesFromDict(variables, the_dict) + + if "variables" in the_dict: + # Make sure all the local variables are added to the variables + # list before we process them so that you can reference one + # variable from another. They will be fully expanded by recursion + # in ExpandVariables. + for key, value in the_dict["variables"].items(): + variables[key] = value + + # Handle the associated variables dict first, so that any variable + # references within can be resolved prior to using them as variables. + # Pass a copy of the variables dict to avoid having it be tainted. + # Otherwise, it would have extra automatics added for everything that + # should just be an ordinary variable in this scope. + ProcessVariablesAndConditionsInDict( + the_dict["variables"], phase, variables, build_file, "variables" + ) + + LoadVariablesFromVariablesDict(variables, the_dict, the_dict_key) + + for key, value in the_dict.items(): + # Skip "variables", which was already processed if present. + if key != "variables" and type(value) is str: + expanded = ExpandVariables(value, phase, variables, build_file) + if type(expanded) not in (str, int): + raise ValueError( + "Variable expansion in this context permits str and int " + + "only, found " + + expanded.__class__.__name__ + + " for " + + key + ) + the_dict[key] = expanded + + # Variable expansion may have resulted in changes to automatics. Reload. + # TODO(mark): Optimization: only reload if no changes were made. + variables = variables_in.copy() + LoadAutomaticVariablesFromDict(variables, the_dict) + LoadVariablesFromVariablesDict(variables, the_dict, the_dict_key) + + # Process conditions in this dict. This is done after variable expansion + # so that conditions may take advantage of expanded variables. For example, + # if the_dict contains: + # {'type': '<(library_type)', + # 'conditions': [['_type=="static_library"', { ... }]]}, + # _type, as used in the condition, will only be set to the value of + # library_type if variable expansion is performed before condition + # processing. However, condition processing should occur prior to recursion + # so that variables (both automatic and "variables" dict type) may be + # adjusted by conditions sections, merged into the_dict, and have the + # intended impact on contained dicts. + # + # This arrangement means that a "conditions" section containing a "variables" + # section will only have those variables effective in subdicts, not in + # the_dict. The workaround is to put a "conditions" section within a + # "variables" section. For example: + # {'conditions': [['os=="mac"', {'variables': {'define': 'IS_MAC'}}]], + # 'defines': ['<(define)'], + # 'my_subdict': {'defines': ['<(define)']}}, + # will not result in "IS_MAC" being appended to the "defines" list in the + # current scope but would result in it being appended to the "defines" list + # within "my_subdict". By comparison: + # {'variables': {'conditions': [['os=="mac"', {'define': 'IS_MAC'}]]}, + # 'defines': ['<(define)'], + # 'my_subdict': {'defines': ['<(define)']}}, + # will append "IS_MAC" to both "defines" lists. + + # Evaluate conditions sections, allowing variable expansions within them + # as well as nested conditionals. This will process a 'conditions' or + # 'target_conditions' section, perform appropriate merging and recursive + # conditional and variable processing, and then remove the conditions section + # from the_dict if it is present. + ProcessConditionsInDict(the_dict, phase, variables, build_file) + + # Conditional processing may have resulted in changes to automatics or the + # variables dict. Reload. + variables = variables_in.copy() + LoadAutomaticVariablesFromDict(variables, the_dict) + LoadVariablesFromVariablesDict(variables, the_dict, the_dict_key) + + # Recurse into child dicts, or process child lists which may result in + # further recursion into descendant dicts. + for key, value in the_dict.items(): + # Skip "variables" and string values, which were already processed if + # present. + if key == "variables" or type(value) is str: + continue + if type(value) is dict: + # Pass a copy of the variables dict so that subdicts can't influence + # parents. + ProcessVariablesAndConditionsInDict( + value, phase, variables, build_file, key + ) + elif type(value) is list: + # The list itself can't influence the variables dict, and + # ProcessVariablesAndConditionsInList will make copies of the variables + # dict if it needs to pass it to something that can influence it. No + # copy is necessary here. + ProcessVariablesAndConditionsInList(value, phase, variables, build_file) + elif type(value) is not int: + raise TypeError("Unknown type " + value.__class__.__name__ + " for " + key) + + +def ProcessVariablesAndConditionsInList(the_list, phase, variables, build_file): + # Iterate using an index so that new values can be assigned into the_list. + index = 0 + while index < len(the_list): + item = the_list[index] + if type(item) is dict: + # Make a copy of the variables dict so that it won't influence anything + # outside of its own scope. + ProcessVariablesAndConditionsInDict(item, phase, variables, build_file) + elif type(item) is list: + ProcessVariablesAndConditionsInList(item, phase, variables, build_file) + elif type(item) is str: + expanded = ExpandVariables(item, phase, variables, build_file) + if type(expanded) in (str, int): + the_list[index] = expanded + elif type(expanded) is list: + the_list[index : index + 1] = expanded + index += len(expanded) + + # index now identifies the next item to examine. Continue right now + # without falling into the index increment below. + continue + else: + raise ValueError( + "Variable expansion in this context permits strings and " + + "lists only, found " + + expanded.__class__.__name__ + + " at " + + index + ) + elif type(item) is not int: + raise TypeError( + "Unknown type " + item.__class__.__name__ + " at index " + index + ) + index = index + 1 def BuildTargetsDict(data): - """Builds a dict mapping fully-qualified target names to their target dicts. + """Builds a dict mapping fully-qualified target names to their target dicts. |data| is a dict mapping loaded build files by pathname relative to the current directory. Values in |data| are build file contents. For each @@ -1353,21 +1475,21 @@ def BuildTargetsDict(data): the dicts in the "targets" lists. """ - targets = {} - for build_file in data['target_build_files']: - for target in data[build_file].get('targets', []): - target_name = gyp.common.QualifiedTarget(build_file, - target['target_name'], - target['toolset']) - if target_name in targets: - raise GypError('Duplicate target definitions for ' + target_name) - targets[target_name] = target + targets = {} + for build_file in data["target_build_files"]: + for target in data[build_file].get("targets", []): + target_name = gyp.common.QualifiedTarget( + build_file, target["target_name"], target["toolset"] + ) + if target_name in targets: + raise GypError("Duplicate target definitions for " + target_name) + targets[target_name] = target - return targets + return targets def QualifyDependencies(targets): - """Make dependency links fully-qualified relative to the current directory. + """Make dependency links fully-qualified relative to the current directory. |targets| is a dict mapping fully-qualified target names to their target dicts. For each target in this dict, keys known to contain dependency @@ -1377,36 +1499,46 @@ def QualifyDependencies(targets): similar dict. """ - all_dependency_sections = [dep + op - for dep in dependency_sections - for op in ('', '!', '/')] - - for target, target_dict in targets.items(): - target_build_file = gyp.common.BuildFile(target) - toolset = target_dict['toolset'] - for dependency_key in all_dependency_sections: - dependencies = target_dict.get(dependency_key, []) - for index in range(0, len(dependencies)): - dep_file, dep_target, dep_toolset = gyp.common.ResolveTarget( - target_build_file, dependencies[index], toolset) - if not multiple_toolsets: - # Ignore toolset specification in the dependency if it is specified. - dep_toolset = toolset - dependency = gyp.common.QualifiedTarget(dep_file, - dep_target, - dep_toolset) - dependencies[index] = dependency - - # Make sure anything appearing in a list other than "dependencies" also - # appears in the "dependencies" list. - if dependency_key != 'dependencies' and \ - dependency not in target_dict['dependencies']: - raise GypError('Found ' + dependency + ' in ' + dependency_key + - ' of ' + target + ', but not in dependencies') + all_dependency_sections = [ + dep + op for dep in dependency_sections for op in ("", "!", "/") + ] + + for target, target_dict in targets.items(): + target_build_file = gyp.common.BuildFile(target) + toolset = target_dict["toolset"] + for dependency_key in all_dependency_sections: + dependencies = target_dict.get(dependency_key, []) + for index, dep in enumerate(dependencies): + dep_file, dep_target, dep_toolset = gyp.common.ResolveTarget( + target_build_file, dep, toolset + ) + if not multiple_toolsets: + # Ignore toolset specification in the dependency if it is specified. + dep_toolset = toolset + dependency = gyp.common.QualifiedTarget( + dep_file, dep_target, dep_toolset + ) + dependencies[index] = dependency + + # Make sure anything appearing in a list other than "dependencies" also + # appears in the "dependencies" list. + if ( + dependency_key != "dependencies" + and dependency not in target_dict["dependencies"] + ): + raise GypError( + "Found " + + dependency + + " in " + + dependency_key + + " of " + + target + + ", but not in dependencies" + ) def ExpandWildcardDependencies(targets, data): - """Expands dependencies specified as build_file:*. + """Expands dependencies specified as build_file:*. For each target in |targets|, examines sections containing links to other targets. If any such section contains a link of the form build_file:*, it @@ -1421,110 +1553,130 @@ def ExpandWildcardDependencies(targets, data): dependency list, must be qualified when this function is called. """ - for target, target_dict in targets.items(): - toolset = target_dict['toolset'] - target_build_file = gyp.common.BuildFile(target) - for dependency_key in dependency_sections: - dependencies = target_dict.get(dependency_key, []) - - # Loop this way instead of "for dependency in" or "for index in range" - # because the dependencies list will be modified within the loop body. - index = 0 - while index < len(dependencies): - (dependency_build_file, dependency_target, dependency_toolset) = \ - gyp.common.ParseQualifiedTarget(dependencies[index]) - if dependency_target != '*' and dependency_toolset != '*': - # Not a wildcard. Keep it moving. - index = index + 1 - continue - - if dependency_build_file == target_build_file: - # It's an error for a target to depend on all other targets in - # the same file, because a target cannot depend on itself. - raise GypError('Found wildcard in ' + dependency_key + ' of ' + - target + ' referring to same build file') - - # Take the wildcard out and adjust the index so that the next - # dependency in the list will be processed the next time through the - # loop. - del dependencies[index] - index = index - 1 - - # Loop through the targets in the other build file, adding them to - # this target's list of dependencies in place of the removed - # wildcard. - dependency_target_dicts = data[dependency_build_file]['targets'] - for dependency_target_dict in dependency_target_dicts: - if int(dependency_target_dict.get('suppress_wildcard', False)): - continue - dependency_target_name = dependency_target_dict['target_name'] - if (dependency_target != '*' and - dependency_target != dependency_target_name): - continue - dependency_target_toolset = dependency_target_dict['toolset'] - if (dependency_toolset != '*' and - dependency_toolset != dependency_target_toolset): - continue - dependency = gyp.common.QualifiedTarget(dependency_build_file, - dependency_target_name, - dependency_target_toolset) - index = index + 1 - dependencies.insert(index, dependency) - - index = index + 1 + for target, target_dict in targets.items(): + target_build_file = gyp.common.BuildFile(target) + for dependency_key in dependency_sections: + dependencies = target_dict.get(dependency_key, []) + + # Loop this way instead of "for dependency in" or "for index in range" + # because the dependencies list will be modified within the loop body. + index = 0 + while index < len(dependencies): + ( + dependency_build_file, + dependency_target, + dependency_toolset, + ) = gyp.common.ParseQualifiedTarget(dependencies[index]) + if dependency_target != "*" and dependency_toolset != "*": + # Not a wildcard. Keep it moving. + index = index + 1 + continue + + if dependency_build_file == target_build_file: + # It's an error for a target to depend on all other targets in + # the same file, because a target cannot depend on itself. + raise GypError( + "Found wildcard in " + + dependency_key + + " of " + + target + + " referring to same build file" + ) + + # Take the wildcard out and adjust the index so that the next + # dependency in the list will be processed the next time through the + # loop. + del dependencies[index] + index = index - 1 + + # Loop through the targets in the other build file, adding them to + # this target's list of dependencies in place of the removed + # wildcard. + dependency_target_dicts = data[dependency_build_file]["targets"] + for dependency_target_dict in dependency_target_dicts: + if int(dependency_target_dict.get("suppress_wildcard", False)): + continue + dependency_target_name = dependency_target_dict["target_name"] + if ( + dependency_target != "*" + and dependency_target != dependency_target_name + ): + continue + dependency_target_toolset = dependency_target_dict["toolset"] + if ( + dependency_toolset != "*" + and dependency_toolset != dependency_target_toolset + ): + continue + dependency = gyp.common.QualifiedTarget( + dependency_build_file, + dependency_target_name, + dependency_target_toolset, + ) + index = index + 1 + dependencies.insert(index, dependency) + + index = index + 1 def Unify(l): - """Removes duplicate elements from l, keeping the first element.""" - seen = {} - return [seen.setdefault(e, e) for e in l if e not in seen] + """Removes duplicate elements from l, keeping the first element.""" + seen = {} + return [seen.setdefault(e, e) for e in l if e not in seen] def RemoveDuplicateDependencies(targets): - """Makes sure every dependency appears only once in all targets's dependency + """Makes sure every dependency appears only once in all targets's dependency lists.""" - for target_name, target_dict in targets.items(): - for dependency_key in dependency_sections: - dependencies = target_dict.get(dependency_key, []) - if dependencies: - target_dict[dependency_key] = Unify(dependencies) + for target_name, target_dict in targets.items(): + for dependency_key in dependency_sections: + dependencies = target_dict.get(dependency_key, []) + if dependencies: + target_dict[dependency_key] = Unify(dependencies) def Filter(l, item): - """Removes item from l.""" - res = {} - return [res.setdefault(e, e) for e in l if e != item] + """Removes item from l.""" + res = {} + return [res.setdefault(e, e) for e in l if e != item] def RemoveSelfDependencies(targets): - """Remove self dependencies from targets that have the prune_self_dependency + """Remove self dependencies from targets that have the prune_self_dependency variable set.""" - for target_name, target_dict in targets.items(): - for dependency_key in dependency_sections: - dependencies = target_dict.get(dependency_key, []) - if dependencies: - for t in dependencies: - if t == target_name: - if targets[t].get('variables', {}).get('prune_self_dependency', 0): - target_dict[dependency_key] = Filter(dependencies, target_name) + for target_name, target_dict in targets.items(): + for dependency_key in dependency_sections: + dependencies = target_dict.get(dependency_key, []) + if dependencies: + for t in dependencies: + if t == target_name: + if ( + targets[t] + .get("variables", {}) + .get("prune_self_dependency", 0) + ): + target_dict[dependency_key] = Filter( + dependencies, target_name + ) def RemoveLinkDependenciesFromNoneTargets(targets): - """Remove dependencies having the 'link_dependency' attribute from the 'none' + """Remove dependencies having the 'link_dependency' attribute from the 'none' targets.""" - for target_name, target_dict in targets.items(): - for dependency_key in dependency_sections: - dependencies = target_dict.get(dependency_key, []) - if dependencies: - for t in dependencies: - if target_dict.get('type', None) == 'none': - if targets[t].get('variables', {}).get('link_dependency', 0): - target_dict[dependency_key] = \ - Filter(target_dict[dependency_key], t) + for target_name, target_dict in targets.items(): + for dependency_key in dependency_sections: + dependencies = target_dict.get(dependency_key, []) + if dependencies: + for t in dependencies: + if target_dict.get("type", None) == "none": + if targets[t].get("variables", {}).get("link_dependency", 0): + target_dict[dependency_key] = Filter( + target_dict[dependency_key], t + ) class DependencyGraphNode(object): - """ + """ Attributes: ref: A reference to an object that this DependencyGraphNode represents. @@ -1532,101 +1684,102 @@ class DependencyGraphNode(object): dependents: List of DependencyGraphNodes that depend on this one. """ - class CircularException(GypError): - pass + class CircularException(GypError): + pass - def __init__(self, ref): - self.ref = ref - self.dependencies = [] - self.dependents = [] - - def __repr__(self): - return '' % self.ref - - def FlattenToList(self): - # flat_list is the sorted list of dependencies - actually, the list items - # are the "ref" attributes of DependencyGraphNodes. Every target will - # appear in flat_list after all of its dependencies, and before all of its - # dependents. - flat_list = OrderedSet() - - def ExtractNodeRef(node): - """Extracts the object that the node represents from the given node.""" - return node.ref - - # in_degree_zeros is the list of DependencyGraphNodes that have no - # dependencies not in flat_list. Initially, it is a copy of the children - # of this node, because when the graph was built, nodes with no - # dependencies were made implicit dependents of the root node. - in_degree_zeros = sorted(self.dependents[:], key=ExtractNodeRef) - - while in_degree_zeros: - # Nodes in in_degree_zeros have no dependencies not in flat_list, so they - # can be appended to flat_list. Take these nodes out of in_degree_zeros - # as work progresses, so that the next node to process from the list can - # always be accessed at a consistent position. - node = in_degree_zeros.pop() - flat_list.add(node.ref) - - # Look at dependents of the node just added to flat_list. Some of them - # may now belong in in_degree_zeros. - for node_dependent in sorted(node.dependents, key=ExtractNodeRef): - is_in_degree_zero = True - # TODO: We want to check through the - # node_dependent.dependencies list but if it's long and we - # always start at the beginning, then we get O(n^2) behaviour. - for node_dependent_dependency in (sorted(node_dependent.dependencies, - key=ExtractNodeRef)): - if not node_dependent_dependency.ref in flat_list: - # The dependent one or more dependencies not in flat_list. There - # will be more chances to add it to flat_list when examining - # it again as a dependent of those other dependencies, provided - # that there are no cycles. - is_in_degree_zero = False - break - - if is_in_degree_zero: - # All of the dependent's dependencies are already in flat_list. Add - # it to in_degree_zeros where it will be processed in a future - # iteration of the outer loop. - in_degree_zeros += [node_dependent] - - return list(flat_list) - - def FindCycles(self): - """ + def __init__(self, ref): + self.ref = ref + self.dependencies = [] + self.dependents = [] + + def __repr__(self): + return "" % self.ref + + def FlattenToList(self): + # flat_list is the sorted list of dependencies - actually, the list items + # are the "ref" attributes of DependencyGraphNodes. Every target will + # appear in flat_list after all of its dependencies, and before all of its + # dependents. + flat_list = OrderedSet() + + def ExtractNodeRef(node): + """Extracts the object that the node represents from the given node.""" + return node.ref + + # in_degree_zeros is the list of DependencyGraphNodes that have no + # dependencies not in flat_list. Initially, it is a copy of the children + # of this node, because when the graph was built, nodes with no + # dependencies were made implicit dependents of the root node. + in_degree_zeros = sorted(self.dependents[:], key=ExtractNodeRef) + + while in_degree_zeros: + # Nodes in in_degree_zeros have no dependencies not in flat_list, so they + # can be appended to flat_list. Take these nodes out of in_degree_zeros + # as work progresses, so that the next node to process from the list can + # always be accessed at a consistent position. + node = in_degree_zeros.pop() + flat_list.add(node.ref) + + # Look at dependents of the node just added to flat_list. Some of them + # may now belong in in_degree_zeros. + for node_dependent in sorted(node.dependents, key=ExtractNodeRef): + is_in_degree_zero = True + # TODO: We want to check through the + # node_dependent.dependencies list but if it's long and we + # always start at the beginning, then we get O(n^2) behaviour. + for node_dependent_dependency in sorted( + node_dependent.dependencies, key=ExtractNodeRef + ): + if node_dependent_dependency.ref not in flat_list: + # The dependent one or more dependencies not in flat_list. There + # will be more chances to add it to flat_list when examining + # it again as a dependent of those other dependencies, provided + # that there are no cycles. + is_in_degree_zero = False + break + + if is_in_degree_zero: + # All of the dependent's dependencies are already in flat_list. Add + # it to in_degree_zeros where it will be processed in a future + # iteration of the outer loop. + in_degree_zeros += [node_dependent] + + return list(flat_list) + + def FindCycles(self): + """ Returns a list of cycles in the graph, where each cycle is its own list. """ - results = [] - visited = set() + results = [] + visited = set() - def Visit(node, path): - for child in node.dependents: - if child in path: - results.append([child] + path[:path.index(child) + 1]) - elif not child in visited: - visited.add(child) - Visit(child, [child] + path) + def Visit(node, path): + for child in node.dependents: + if child in path: + results.append([child] + path[: path.index(child) + 1]) + elif child not in visited: + visited.add(child) + Visit(child, [child] + path) - visited.add(self) - Visit(self, [self]) + visited.add(self) + Visit(self, [self]) - return results + return results - def DirectDependencies(self, dependencies=None): - """Returns a list of just direct dependencies.""" - if dependencies is None: - dependencies = [] + def DirectDependencies(self, dependencies=None): + """Returns a list of just direct dependencies.""" + if dependencies is None: + dependencies = [] - for dependency in self.dependencies: - # Check for None, corresponding to the root node. - if dependency.ref != None and dependency.ref not in dependencies: - dependencies.append(dependency.ref) + for dependency in self.dependencies: + # Check for None, corresponding to the root node. + if dependency.ref and dependency.ref not in dependencies: + dependencies.append(dependency.ref) - return dependencies + return dependencies - def _AddImportedDependencies(self, targets, dependencies=None): - """Given a list of direct dependencies, adds indirect dependencies that + def _AddImportedDependencies(self, targets, dependencies=None): + """Given a list of direct dependencies, adds indirect dependencies that other dependencies have declared to export their settings. This method does not operate on self. Rather, it operates on the list @@ -1643,58 +1796,60 @@ def _AddImportedDependencies(self, targets, dependencies=None): public entry point. """ - if dependencies is None: - dependencies = [] - - index = 0 - while index < len(dependencies): - dependency = dependencies[index] - dependency_dict = targets[dependency] - # Add any dependencies whose settings should be imported to the list - # if not already present. Newly-added items will be checked for - # their own imports when the list iteration reaches them. - # Rather than simply appending new items, insert them after the - # dependency that exported them. This is done to more closely match - # the depth-first method used by DeepDependencies. - add_index = 1 - for imported_dependency in \ - dependency_dict.get('export_dependent_settings', []): - if imported_dependency not in dependencies: - dependencies.insert(index + add_index, imported_dependency) - add_index = add_index + 1 - index = index + 1 - - return dependencies - - def DirectAndImportedDependencies(self, targets, dependencies=None): - """Returns a list of a target's direct dependencies and all indirect + if dependencies is None: + dependencies = [] + + index = 0 + while index < len(dependencies): + dependency = dependencies[index] + dependency_dict = targets[dependency] + # Add any dependencies whose settings should be imported to the list + # if not already present. Newly-added items will be checked for + # their own imports when the list iteration reaches them. + # Rather than simply appending new items, insert them after the + # dependency that exported them. This is done to more closely match + # the depth-first method used by DeepDependencies. + add_index = 1 + for imported_dependency in dependency_dict.get( + "export_dependent_settings", [] + ): + if imported_dependency not in dependencies: + dependencies.insert(index + add_index, imported_dependency) + add_index = add_index + 1 + index = index + 1 + + return dependencies + + def DirectAndImportedDependencies(self, targets, dependencies=None): + """Returns a list of a target's direct dependencies and all indirect dependencies that a dependency has advertised settings should be exported through the dependency for. """ - dependencies = self.DirectDependencies(dependencies) - return self._AddImportedDependencies(targets, dependencies) - - def DeepDependencies(self, dependencies=None): - """Returns an OrderedSet of all of a target's dependencies, recursively.""" - if dependencies is None: - # Using a list to get ordered output and a set to do fast "is it - # already added" checks. - dependencies = OrderedSet() - - for dependency in self.dependencies: - # Check for None, corresponding to the root node. - if dependency.ref is None: - continue - if dependency.ref not in dependencies: - dependency.DeepDependencies(dependencies) - dependencies.add(dependency.ref) - - return dependencies - - def _LinkDependenciesInternal(self, targets, include_shared_libraries, - dependencies=None, initial=True): - """Returns an OrderedSet of dependency targets that are linked + dependencies = self.DirectDependencies(dependencies) + return self._AddImportedDependencies(targets, dependencies) + + def DeepDependencies(self, dependencies=None): + """Returns an OrderedSet of all of a target's dependencies, recursively.""" + if dependencies is None: + # Using a list to get ordered output and a set to do fast "is it + # already added" checks. + dependencies = OrderedSet() + + for dependency in self.dependencies: + # Check for None, corresponding to the root node. + if dependency.ref is None: + continue + if dependency.ref not in dependencies: + dependency.DeepDependencies(dependencies) + dependencies.add(dependency.ref) + + return dependencies + + def _LinkDependenciesInternal( + self, targets, include_shared_libraries, dependencies=None, initial=True + ): + """Returns an OrderedSet of dependency targets that are linked into this target. This function has a split personality, depending on the setting of @@ -1708,625 +1863,683 @@ def _LinkDependenciesInternal(self, targets, include_shared_libraries, If |include_shared_libraries| is False, the resulting dependencies will not include shared_library targets that are linked into this target. """ - if dependencies is None: - # Using a list to get ordered output and a set to do fast "is it - # already added" checks. - dependencies = OrderedSet() - - # Check for None, corresponding to the root node. - if self.ref is None: - return dependencies - - # It's kind of sucky that |targets| has to be passed into this function, - # but that's presently the easiest way to access the target dicts so that - # this function can find target types. - - if 'target_name' not in targets[self.ref]: - raise GypError("Missing 'target_name' field in target.") - - if 'type' not in targets[self.ref]: - raise GypError("Missing 'type' field in target %s" % - targets[self.ref]['target_name']) - - target_type = targets[self.ref]['type'] - - is_linkable = target_type in linkable_types - - if initial and not is_linkable: - # If this is the first target being examined and it's not linkable, - # return an empty list of link dependencies, because the link - # dependencies are intended to apply to the target itself (initial is - # True) and this target won't be linked. - return dependencies - - # Don't traverse 'none' targets if explicitly excluded. - if (target_type == 'none' and - not targets[self.ref].get('dependencies_traverse', True)): - dependencies.add(self.ref) - return dependencies - - # Executables, mac kernel extensions, windows drivers and loadable modules - # are already fully and finally linked. Nothing else can be a link - # dependency of them, there can only be dependencies in the sense that a - # dependent target might run an executable or load the loadable_module. - if not initial and target_type in ('executable', 'loadable_module', - 'mac_kernel_extension', - 'windows_driver'): - return dependencies - - # Shared libraries are already fully linked. They should only be included - # in |dependencies| when adjusting static library dependencies (in order to - # link against the shared_library's import lib), but should not be included - # in |dependencies| when propagating link_settings. - # The |include_shared_libraries| flag controls which of these two cases we - # are handling. - if (not initial and target_type == 'shared_library' and - not include_shared_libraries): - return dependencies - - # The target is linkable, add it to the list of link dependencies. - if self.ref not in dependencies: - dependencies.add(self.ref) - if initial or not is_linkable: - # If this is a subsequent target and it's linkable, don't look any - # further for linkable dependencies, as they'll already be linked into - # this target linkable. Always look at dependencies of the initial - # target, and always look at dependencies of non-linkables. - for dependency in self.dependencies: - dependency._LinkDependenciesInternal(targets, - include_shared_libraries, - dependencies, False) - - return dependencies - - def DependenciesForLinkSettings(self, targets): - """ + if dependencies is None: + # Using a list to get ordered output and a set to do fast "is it + # already added" checks. + dependencies = OrderedSet() + + # Check for None, corresponding to the root node. + if self.ref is None: + return dependencies + + # It's kind of sucky that |targets| has to be passed into this function, + # but that's presently the easiest way to access the target dicts so that + # this function can find target types. + + if "target_name" not in targets[self.ref]: + raise GypError("Missing 'target_name' field in target.") + + if "type" not in targets[self.ref]: + raise GypError( + "Missing 'type' field in target %s" % targets[self.ref]["target_name"] + ) + + target_type = targets[self.ref]["type"] + + is_linkable = target_type in linkable_types + + if initial and not is_linkable: + # If this is the first target being examined and it's not linkable, + # return an empty list of link dependencies, because the link + # dependencies are intended to apply to the target itself (initial is + # True) and this target won't be linked. + return dependencies + + # Don't traverse 'none' targets if explicitly excluded. + if target_type == "none" and not targets[self.ref].get( + "dependencies_traverse", True + ): + dependencies.add(self.ref) + return dependencies + + # Executables, mac kernel extensions, windows drivers and loadable modules + # are already fully and finally linked. Nothing else can be a link + # dependency of them, there can only be dependencies in the sense that a + # dependent target might run an executable or load the loadable_module. + if not initial and target_type in ( + "executable", + "loadable_module", + "mac_kernel_extension", + "windows_driver", + ): + return dependencies + + # Shared libraries are already fully linked. They should only be included + # in |dependencies| when adjusting static library dependencies (in order to + # link against the shared_library's import lib), but should not be included + # in |dependencies| when propagating link_settings. + # The |include_shared_libraries| flag controls which of these two cases we + # are handling. + if ( + not initial + and target_type == "shared_library" + and not include_shared_libraries + ): + return dependencies + + # The target is linkable, add it to the list of link dependencies. + if self.ref not in dependencies: + dependencies.add(self.ref) + if initial or not is_linkable: + # If this is a subsequent target and it's linkable, don't look any + # further for linkable dependencies, as they'll already be linked into + # this target linkable. Always look at dependencies of the initial + # target, and always look at dependencies of non-linkables. + for dependency in self.dependencies: + dependency._LinkDependenciesInternal( + targets, include_shared_libraries, dependencies, False + ) + + return dependencies + + def DependenciesForLinkSettings(self, targets): + """ Returns a list of dependency targets whose link_settings should be merged into this target. """ - # TODO(sbaig) Currently, chrome depends on the bug that shared libraries' - # link_settings are propagated. So for now, we will allow it, unless the - # 'allow_sharedlib_linksettings_propagation' flag is explicitly set to - # False. Once chrome is fixed, we can remove this flag. - include_shared_libraries = \ - targets[self.ref].get('allow_sharedlib_linksettings_propagation', True) - return self._LinkDependenciesInternal(targets, include_shared_libraries) - - def DependenciesToLinkAgainst(self, targets): - """ + # TODO(sbaig) Currently, chrome depends on the bug that shared libraries' + # link_settings are propagated. So for now, we will allow it, unless the + # 'allow_sharedlib_linksettings_propagation' flag is explicitly set to + # False. Once chrome is fixed, we can remove this flag. + include_shared_libraries = targets[self.ref].get( + "allow_sharedlib_linksettings_propagation", True + ) + return self._LinkDependenciesInternal(targets, include_shared_libraries) + + def DependenciesToLinkAgainst(self, targets): + """ Returns a list of dependency targets that are linked into this target. """ - return self._LinkDependenciesInternal(targets, True) + return self._LinkDependenciesInternal(targets, True) def BuildDependencyList(targets): - # Create a DependencyGraphNode for each target. Put it into a dict for easy - # access. - dependency_nodes = {} - for target, spec in targets.items(): - if target not in dependency_nodes: - dependency_nodes[target] = DependencyGraphNode(target) - - # Set up the dependency links. Targets that have no dependencies are treated - # as dependent on root_node. - root_node = DependencyGraphNode(None) - for target, spec in targets.items(): - target_node = dependency_nodes[target] - target_build_file = gyp.common.BuildFile(target) - dependencies = spec.get('dependencies') - if not dependencies: - target_node.dependencies = [root_node] - root_node.dependents.append(target_node) - else: - for dependency in dependencies: - dependency_node = dependency_nodes.get(dependency) - if not dependency_node: - raise GypError("Dependency '%s' not found while " - "trying to load target %s" % (dependency, target)) - target_node.dependencies.append(dependency_node) - dependency_node.dependents.append(target_node) - - flat_list = root_node.FlattenToList() - - # If there's anything left unvisited, there must be a circular dependency - # (cycle). - if len(flat_list) != len(targets): - if not root_node.dependents: - # If all targets have dependencies, add the first target as a dependent - # of root_node so that the cycle can be discovered from root_node. - target = targets.keys()[0] - target_node = dependency_nodes[target] - target_node.dependencies.append(root_node) - root_node.dependents.append(target_node) - - cycles = [] - for cycle in root_node.FindCycles(): - paths = [node.ref for node in cycle] - cycles.append('Cycle: %s' % ' -> '.join(paths)) - raise DependencyGraphNode.CircularException( - 'Cycles in dependency graph detected:\n' + '\n'.join(cycles)) - - return [dependency_nodes, flat_list] + # Create a DependencyGraphNode for each target. Put it into a dict for easy + # access. + dependency_nodes = {} + for target, spec in targets.items(): + if target not in dependency_nodes: + dependency_nodes[target] = DependencyGraphNode(target) + + # Set up the dependency links. Targets that have no dependencies are treated + # as dependent on root_node. + root_node = DependencyGraphNode(None) + for target, spec in targets.items(): + target_node = dependency_nodes[target] + dependencies = spec.get("dependencies") + if not dependencies: + target_node.dependencies = [root_node] + root_node.dependents.append(target_node) + else: + for dependency in dependencies: + dependency_node = dependency_nodes.get(dependency) + if not dependency_node: + raise GypError( + "Dependency '%s' not found while " + "trying to load target %s" % (dependency, target) + ) + target_node.dependencies.append(dependency_node) + dependency_node.dependents.append(target_node) + + flat_list = root_node.FlattenToList() + + # If there's anything left unvisited, there must be a circular dependency + # (cycle). + if len(flat_list) != len(targets): + if not root_node.dependents: + # If all targets have dependencies, add the first target as a dependent + # of root_node so that the cycle can be discovered from root_node. + target = next(iter(targets)) + target_node = dependency_nodes[target] + target_node.dependencies.append(root_node) + root_node.dependents.append(target_node) + + cycles = [] + for cycle in root_node.FindCycles(): + paths = [node.ref for node in cycle] + cycles.append("Cycle: %s" % " -> ".join(paths)) + raise DependencyGraphNode.CircularException( + "Cycles in dependency graph detected:\n" + "\n".join(cycles) + ) + + return [dependency_nodes, flat_list] def VerifyNoGYPFileCircularDependencies(targets): - # Create a DependencyGraphNode for each gyp file containing a target. Put - # it into a dict for easy access. - dependency_nodes = {} - for target in targets: - build_file = gyp.common.BuildFile(target) - if not build_file in dependency_nodes: - dependency_nodes[build_file] = DependencyGraphNode(build_file) - - # Set up the dependency links. - for target, spec in targets.items(): - build_file = gyp.common.BuildFile(target) - build_file_node = dependency_nodes[build_file] - target_dependencies = spec.get('dependencies', []) - for dependency in target_dependencies: - try: - dependency_build_file = gyp.common.BuildFile(dependency) - except GypError as e: - gyp.common.ExceptionAppend( - e, 'while computing dependencies of .gyp file %s' % build_file) - raise - - if dependency_build_file == build_file: - # A .gyp file is allowed to refer back to itself. - continue - dependency_node = dependency_nodes.get(dependency_build_file) - if not dependency_node: - raise GypError("Dependency '%s' not found" % dependency_build_file) - if dependency_node not in build_file_node.dependencies: - build_file_node.dependencies.append(dependency_node) - dependency_node.dependents.append(build_file_node) - - - # Files that have no dependencies are treated as dependent on root_node. - root_node = DependencyGraphNode(None) - for build_file_node in dependency_nodes.values(): - if len(build_file_node.dependencies) == 0: - build_file_node.dependencies.append(root_node) - root_node.dependents.append(build_file_node) - - flat_list = root_node.FlattenToList() - - # If there's anything left unvisited, there must be a circular dependency - # (cycle). - if len(flat_list) != len(dependency_nodes): - if not root_node.dependents: - # If all files have dependencies, add the first file as a dependent - # of root_node so that the cycle can be discovered from root_node. - file_node = dependency_nodes.values()[0] - file_node.dependencies.append(root_node) - root_node.dependents.append(file_node) - cycles = [] - for cycle in root_node.FindCycles(): - paths = [node.ref for node in cycle] - cycles.append('Cycle: %s' % ' -> '.join(paths)) - raise DependencyGraphNode.CircularException( - 'Cycles in .gyp file dependency graph detected:\n' + '\n'.join(cycles)) + # Create a DependencyGraphNode for each gyp file containing a target. Put + # it into a dict for easy access. + dependency_nodes = {} + for target in targets: + build_file = gyp.common.BuildFile(target) + if build_file not in dependency_nodes: + dependency_nodes[build_file] = DependencyGraphNode(build_file) + + # Set up the dependency links. + for target, spec in targets.items(): + build_file = gyp.common.BuildFile(target) + build_file_node = dependency_nodes[build_file] + target_dependencies = spec.get("dependencies", []) + for dependency in target_dependencies: + try: + dependency_build_file = gyp.common.BuildFile(dependency) + except GypError as e: + gyp.common.ExceptionAppend( + e, "while computing dependencies of .gyp file %s" % build_file + ) + raise + + if dependency_build_file == build_file: + # A .gyp file is allowed to refer back to itself. + continue + dependency_node = dependency_nodes.get(dependency_build_file) + if not dependency_node: + raise GypError("Dependency '%s' not found" % dependency_build_file) + if dependency_node not in build_file_node.dependencies: + build_file_node.dependencies.append(dependency_node) + dependency_node.dependents.append(build_file_node) + + # Files that have no dependencies are treated as dependent on root_node. + root_node = DependencyGraphNode(None) + for build_file_node in dependency_nodes.values(): + if len(build_file_node.dependencies) == 0: + build_file_node.dependencies.append(root_node) + root_node.dependents.append(build_file_node) + + flat_list = root_node.FlattenToList() + + # If there's anything left unvisited, there must be a circular dependency + # (cycle). + if len(flat_list) != len(dependency_nodes): + if not root_node.dependents: + # If all files have dependencies, add the first file as a dependent + # of root_node so that the cycle can be discovered from root_node. + file_node = next(iter(dependency_nodes.values())) + file_node.dependencies.append(root_node) + root_node.dependents.append(file_node) + cycles = [] + for cycle in root_node.FindCycles(): + paths = [node.ref for node in cycle] + cycles.append("Cycle: %s" % " -> ".join(paths)) + raise DependencyGraphNode.CircularException( + "Cycles in .gyp file dependency graph detected:\n" + "\n".join(cycles) + ) def DoDependentSettings(key, flat_list, targets, dependency_nodes): - # key should be one of all_dependent_settings, direct_dependent_settings, - # or link_settings. - - for target in flat_list: - target_dict = targets[target] - build_file = gyp.common.BuildFile(target) + # key should be one of all_dependent_settings, direct_dependent_settings, + # or link_settings. - if key == 'all_dependent_settings': - dependencies = dependency_nodes[target].DeepDependencies() - elif key == 'direct_dependent_settings': - dependencies = \ - dependency_nodes[target].DirectAndImportedDependencies(targets) - elif key == 'link_settings': - dependencies = \ - dependency_nodes[target].DependenciesForLinkSettings(targets) - else: - raise GypError("DoDependentSettings doesn't know how to determine " - 'dependencies for ' + key) - - for dependency in dependencies: - dependency_dict = targets[dependency] - if not key in dependency_dict: - continue - dependency_build_file = gyp.common.BuildFile(dependency) - MergeDicts(target_dict, dependency_dict[key], - build_file, dependency_build_file) - - -def AdjustStaticLibraryDependencies(flat_list, targets, dependency_nodes, - sort_dependencies): - # Recompute target "dependencies" properties. For each static library - # target, remove "dependencies" entries referring to other static libraries, - # unless the dependency has the "hard_dependency" attribute set. For each - # linkable target, add a "dependencies" entry referring to all of the - # target's computed list of link dependencies (including static libraries - # if no such entry is already present. - for target in flat_list: - target_dict = targets[target] - target_type = target_dict['type'] - - if target_type == 'static_library': - if not 'dependencies' in target_dict: - continue - - target_dict['dependencies_original'] = target_dict.get( - 'dependencies', [])[:] - - # A static library should not depend on another static library unless - # the dependency relationship is "hard," which should only be done when - # a dependent relies on some side effect other than just the build - # product, like a rule or action output. Further, if a target has a - # non-hard dependency, but that dependency exports a hard dependency, - # the non-hard dependency can safely be removed, but the exported hard - # dependency must be added to the target to keep the same dependency - # ordering. - dependencies = \ - dependency_nodes[target].DirectAndImportedDependencies(targets) - index = 0 - while index < len(dependencies): - dependency = dependencies[index] - dependency_dict = targets[dependency] - - # Remove every non-hard static library dependency and remove every - # non-static library dependency that isn't a direct dependency. - if (dependency_dict['type'] == 'static_library' and \ - not dependency_dict.get('hard_dependency', False)) or \ - (dependency_dict['type'] != 'static_library' and \ - not dependency in target_dict['dependencies']): - # Take the dependency out of the list, and don't increment index - # because the next dependency to analyze will shift into the index - # formerly occupied by the one being removed. - del dependencies[index] + for target in flat_list: + target_dict = targets[target] + build_file = gyp.common.BuildFile(target) + + if key == "all_dependent_settings": + dependencies = dependency_nodes[target].DeepDependencies() + elif key == "direct_dependent_settings": + dependencies = dependency_nodes[target].DirectAndImportedDependencies( + targets + ) + elif key == "link_settings": + dependencies = dependency_nodes[target].DependenciesForLinkSettings(targets) else: - index = index + 1 - - # Update the dependencies. If the dependencies list is empty, it's not - # needed, so unhook it. - if len(dependencies) > 0: - target_dict['dependencies'] = dependencies - else: - del target_dict['dependencies'] - - elif target_type in linkable_types: - # Get a list of dependency targets that should be linked into this - # target. Add them to the dependencies list if they're not already - # present. - - link_dependencies = \ - dependency_nodes[target].DependenciesToLinkAgainst(targets) - for dependency in link_dependencies: - if dependency == target: - continue - if not 'dependencies' in target_dict: - target_dict['dependencies'] = [] - if not dependency in target_dict['dependencies']: - target_dict['dependencies'].append(dependency) - # Sort the dependencies list in the order from dependents to dependencies. - # e.g. If A and B depend on C and C depends on D, sort them in A, B, C, D. - # Note: flat_list is already sorted in the order from dependencies to - # dependents. - if sort_dependencies and 'dependencies' in target_dict: - target_dict['dependencies'] = [dep for dep in reversed(flat_list) - if dep in target_dict['dependencies']] + raise GypError( + "DoDependentSettings doesn't know how to determine " + "dependencies for " + key + ) + + for dependency in dependencies: + dependency_dict = targets[dependency] + if key not in dependency_dict: + continue + dependency_build_file = gyp.common.BuildFile(dependency) + MergeDicts( + target_dict, dependency_dict[key], build_file, dependency_build_file + ) + + +def AdjustStaticLibraryDependencies( + flat_list, targets, dependency_nodes, sort_dependencies +): + # Recompute target "dependencies" properties. For each static library + # target, remove "dependencies" entries referring to other static libraries, + # unless the dependency has the "hard_dependency" attribute set. For each + # linkable target, add a "dependencies" entry referring to all of the + # target's computed list of link dependencies (including static libraries + # if no such entry is already present. + for target in flat_list: + target_dict = targets[target] + target_type = target_dict["type"] + + if target_type == "static_library": + if "dependencies" not in target_dict: + continue + + target_dict["dependencies_original"] = target_dict.get("dependencies", [])[ + : + ] + + # A static library should not depend on another static library unless + # the dependency relationship is "hard," which should only be done when + # a dependent relies on some side effect other than just the build + # product, like a rule or action output. Further, if a target has a + # non-hard dependency, but that dependency exports a hard dependency, + # the non-hard dependency can safely be removed, but the exported hard + # dependency must be added to the target to keep the same dependency + # ordering. + dependencies = dependency_nodes[target].DirectAndImportedDependencies( + targets + ) + index = 0 + while index < len(dependencies): + dependency = dependencies[index] + dependency_dict = targets[dependency] + + # Remove every non-hard static library dependency and remove every + # non-static library dependency that isn't a direct dependency. + if ( + dependency_dict["type"] == "static_library" + and not dependency_dict.get("hard_dependency", False) + ) or ( + dependency_dict["type"] != "static_library" + and dependency not in target_dict["dependencies"] + ): + # Take the dependency out of the list, and don't increment index + # because the next dependency to analyze will shift into the index + # formerly occupied by the one being removed. + del dependencies[index] + else: + index = index + 1 + + # Update the dependencies. If the dependencies list is empty, it's not + # needed, so unhook it. + if len(dependencies) > 0: + target_dict["dependencies"] = dependencies + else: + del target_dict["dependencies"] + + elif target_type in linkable_types: + # Get a list of dependency targets that should be linked into this + # target. Add them to the dependencies list if they're not already + # present. + + link_dependencies = dependency_nodes[target].DependenciesToLinkAgainst( + targets + ) + for dependency in link_dependencies: + if dependency == target: + continue + if "dependencies" not in target_dict: + target_dict["dependencies"] = [] + if dependency not in target_dict["dependencies"]: + target_dict["dependencies"].append(dependency) + # Sort the dependencies list in the order from dependents to dependencies. + # e.g. If A and B depend on C and C depends on D, sort them in A, B, C, D. + # Note: flat_list is already sorted in the order from dependencies to + # dependents. + if sort_dependencies and "dependencies" in target_dict: + target_dict["dependencies"] = [ + dep + for dep in reversed(flat_list) + if dep in target_dict["dependencies"] + ] # Initialize this here to speed up MakePathRelative. -exception_re = re.compile(r'''["']?[-/$<>^]''') +exception_re = re.compile(r"""["']?[-/$<>^]""") def MakePathRelative(to_file, fro_file, item): - # If item is a relative path, it's relative to the build file dict that it's - # coming from. Fix it up to make it relative to the build file dict that - # it's going into. - # Exception: any |item| that begins with these special characters is - # returned without modification. - # / Used when a path is already absolute (shortcut optimization; - # such paths would be returned as absolute anyway) - # $ Used for build environment variables - # - Used for some build environment flags (such as -lapr-1 in a - # "libraries" section) - # < Used for our own variable and command expansions (see ExpandVariables) - # > Used for our own variable and command expansions (see ExpandVariables) - # ^ Used for our own variable and command expansions (see ExpandVariables) - # - # "/' Used when a value is quoted. If these are present, then we - # check the second character instead. - # - if to_file == fro_file or exception_re.match(item): - return item - else: - # TODO(dglazkov) The backslash/forward-slash replacement at the end is a - # temporary measure. This should really be addressed by keeping all paths - # in POSIX until actual project generation. - ret = os.path.normpath(os.path.join( - gyp.common.RelativePath(os.path.dirname(fro_file), - os.path.dirname(to_file)), - item)).replace('\\', '/') - if item.endswith('/'): - ret += '/' - return ret - -def MergeLists(to, fro, to_file, fro_file, is_paths=False, append=True): - # Python documentation recommends objects which do not support hash - # set this value to None. Python library objects follow this rule. - is_hashable = lambda val: val.__hash__ - - # If x is hashable, returns whether x is in s. Else returns whether x is in l. - def is_in_set_or_list(x, s, l): - if is_hashable(x): - return x in s - return x in l - - prepend_index = 0 - - # Make membership testing of hashables in |to| (in particular, strings) - # faster. - hashable_to_set = set(x for x in to if is_hashable(x)) - for item in fro: - singleton = False - if type(item) in (str, int): - # The cheap and easy case. - if is_paths: - to_item = MakePathRelative(to_file, fro_file, item) - else: - to_item = item - - if not (type(item) is str and item.startswith('-')): - # Any string that doesn't begin with a "-" is a singleton - it can - # only appear once in a list, to be enforced by the list merge append - # or prepend. - singleton = True - elif type(item) is dict: - # Make a copy of the dictionary, continuing to look for paths to fix. - # The other intelligent aspects of merge processing won't apply because - # item is being merged into an empty dict. - to_item = {} - MergeDicts(to_item, item, to_file, fro_file) - elif type(item) is list: - # Recurse, making a copy of the list. If the list contains any - # descendant dicts, path fixing will occur. Note that here, custom - # values for is_paths and append are dropped; those are only to be - # applied to |to| and |fro|, not sublists of |fro|. append shouldn't - # matter anyway because the new |to_item| list is empty. - to_item = [] - MergeLists(to_item, item, to_file, fro_file) - else: - raise TypeError( - 'Attempt to merge list item of unsupported type ' + \ - item.__class__.__name__) - - if append: - # If appending a singleton that's already in the list, don't append. - # This ensures that the earliest occurrence of the item will stay put. - if not singleton or not is_in_set_or_list(to_item, hashable_to_set, to): - to.append(to_item) - if is_hashable(to_item): - hashable_to_set.add(to_item) - else: - # If prepending a singleton that's already in the list, remove the - # existing instance and proceed with the prepend. This ensures that the - # item appears at the earliest possible position in the list. - while singleton and to_item in to: - to.remove(to_item) - - # Don't just insert everything at index 0. That would prepend the new - # items to the list in reverse order, which would be an unwelcome - # surprise. - to.insert(prepend_index, to_item) - if is_hashable(to_item): - hashable_to_set.add(to_item) - prepend_index = prepend_index + 1 - - -def MergeDicts(to, fro, to_file, fro_file): - # I wanted to name the parameter "from" but it's a Python keyword... - for k, v in fro.items(): - # It would be nice to do "if not k in to: to[k] = v" but that wouldn't give - # copy semantics. Something else may want to merge from the |fro| dict - # later, and having the same dict ref pointed to twice in the tree isn't - # what anyone wants considering that the dicts may subsequently be - # modified. - if k in to: - bad_merge = False - if type(v) in (str, int): - if type(to[k]) not in (str, int): - bad_merge = True - elif type(v) is not type(to[k]): - bad_merge = True - - if bad_merge: - raise TypeError( - 'Attempt to merge dict value of type ' + v.__class__.__name__ + \ - ' into incompatible type ' + to[k].__class__.__name__ + \ - ' for key ' + k) - if type(v) in (str, int): - # Overwrite the existing value, if any. Cheap and easy. - is_path = IsPathSection(k) - if is_path: - to[k] = MakePathRelative(to_file, fro_file, v) - else: - to[k] = v - elif type(v) is dict: - # Recurse, guaranteeing copies will be made of objects that require it. - if not k in to: - to[k] = {} - MergeDicts(to[k], v, to_file, fro_file) - elif type(v) is list: - # Lists in dicts can be merged with different policies, depending on - # how the key in the "from" dict (k, the from-key) is written. - # - # If the from-key has ...the to-list will have this action - # this character appended:... applied when receiving the from-list: - # = replace - # + prepend - # ? set, only if to-list does not yet exist - # (none) append - # - # This logic is list-specific, but since it relies on the associated - # dict key, it's checked in this dict-oriented function. - ext = k[-1] - append = True - if ext == '=': - list_base = k[:-1] - lists_incompatible = [list_base, list_base + '?'] - to[list_base] = [] - elif ext == '+': - list_base = k[:-1] - lists_incompatible = [list_base + '=', list_base + '?'] - append = False - elif ext == '?': - list_base = k[:-1] - lists_incompatible = [list_base, list_base + '=', list_base + '+'] - else: - list_base = k - lists_incompatible = [list_base + '=', list_base + '?'] - - # Some combinations of merge policies appearing together are meaningless. - # It's stupid to replace and append simultaneously, for example. Append - # and prepend are the only policies that can coexist. - for list_incompatible in lists_incompatible: - if list_incompatible in fro: - raise GypError('Incompatible list policies ' + k + ' and ' + - list_incompatible) - - if list_base in to: - if ext == '?': - # If the key ends in "?", the list will only be merged if it doesn't - # already exist. - continue - elif type(to[list_base]) is not list: - # This may not have been checked above if merging in a list with an - # extension character. - raise TypeError( - 'Attempt to merge dict value of type ' + v.__class__.__name__ + \ - ' into incompatible type ' + to[list_base].__class__.__name__ + \ - ' for key ' + list_base + '(' + k + ')') - else: - to[list_base] = [] - - # Call MergeLists, which will make copies of objects that require it. - # MergeLists can recurse back into MergeDicts, although this will be - # to make copies of dicts (with paths fixed), there will be no - # subsequent dict "merging" once entering a list because lists are - # always replaced, appended to, or prepended to. - is_paths = IsPathSection(list_base) - MergeLists(to[list_base], v, to_file, fro_file, is_paths, append) + # If item is a relative path, it's relative to the build file dict that it's + # coming from. Fix it up to make it relative to the build file dict that + # it's going into. + # Exception: any |item| that begins with these special characters is + # returned without modification. + # / Used when a path is already absolute (shortcut optimization; + # such paths would be returned as absolute anyway) + # $ Used for build environment variables + # - Used for some build environment flags (such as -lapr-1 in a + # "libraries" section) + # < Used for our own variable and command expansions (see ExpandVariables) + # > Used for our own variable and command expansions (see ExpandVariables) + # ^ Used for our own variable and command expansions (see ExpandVariables) + # + # "/' Used when a value is quoted. If these are present, then we + # check the second character instead. + # + if to_file == fro_file or exception_re.match(item): + return item else: - raise TypeError( - 'Attempt to merge dict value of unsupported type ' + \ - v.__class__.__name__ + ' for key ' + k) + # TODO(dglazkov) The backslash/forward-slash replacement at the end is a + # temporary measure. This should really be addressed by keeping all paths + # in POSIX until actual project generation. + ret = os.path.normpath( + os.path.join( + gyp.common.RelativePath( + os.path.dirname(fro_file), os.path.dirname(to_file) + ), + item, + ) + ).replace("\\", "/") + if item.endswith("/"): + ret += "/" + return ret -def MergeConfigWithInheritance(new_configuration_dict, build_file, - target_dict, configuration, visited): - # Skip if previously visted. - if configuration in visited: - return - - # Look at this configuration. - configuration_dict = target_dict['configurations'][configuration] +def MergeLists(to, fro, to_file, fro_file, is_paths=False, append=True): + # Python documentation recommends objects which do not support hash + # set this value to None. Python library objects follow this rule. + def is_hashable(val): + return val.__hash__ + + # If x is hashable, returns whether x is in s. Else returns whether x is in l. + def is_in_set_or_list(x, s, l): + if is_hashable(x): + return x in s + return x in l + + prepend_index = 0 + + # Make membership testing of hashables in |to| (in particular, strings) + # faster. + hashable_to_set = set(x for x in to if is_hashable(x)) + for item in fro: + singleton = False + if type(item) in (str, int): + # The cheap and easy case. + if is_paths: + to_item = MakePathRelative(to_file, fro_file, item) + else: + to_item = item + + if not (type(item) is str and item.startswith("-")): + # Any string that doesn't begin with a "-" is a singleton - it can + # only appear once in a list, to be enforced by the list merge append + # or prepend. + singleton = True + elif type(item) is dict: + # Make a copy of the dictionary, continuing to look for paths to fix. + # The other intelligent aspects of merge processing won't apply because + # item is being merged into an empty dict. + to_item = {} + MergeDicts(to_item, item, to_file, fro_file) + elif type(item) is list: + # Recurse, making a copy of the list. If the list contains any + # descendant dicts, path fixing will occur. Note that here, custom + # values for is_paths and append are dropped; those are only to be + # applied to |to| and |fro|, not sublists of |fro|. append shouldn't + # matter anyway because the new |to_item| list is empty. + to_item = [] + MergeLists(to_item, item, to_file, fro_file) + else: + raise TypeError( + "Attempt to merge list item of unsupported type " + + item.__class__.__name__ + ) + + if append: + # If appending a singleton that's already in the list, don't append. + # This ensures that the earliest occurrence of the item will stay put. + if not singleton or not is_in_set_or_list(to_item, hashable_to_set, to): + to.append(to_item) + if is_hashable(to_item): + hashable_to_set.add(to_item) + else: + # If prepending a singleton that's already in the list, remove the + # existing instance and proceed with the prepend. This ensures that the + # item appears at the earliest possible position in the list. + while singleton and to_item in to: + to.remove(to_item) - # Merge in parents. - for parent in configuration_dict.get('inherit_from', []): - MergeConfigWithInheritance(new_configuration_dict, build_file, - target_dict, parent, visited + [configuration]) + # Don't just insert everything at index 0. That would prepend the new + # items to the list in reverse order, which would be an unwelcome + # surprise. + to.insert(prepend_index, to_item) + if is_hashable(to_item): + hashable_to_set.add(to_item) + prepend_index = prepend_index + 1 - # Merge it into the new config. - MergeDicts(new_configuration_dict, configuration_dict, - build_file, build_file) - # Drop abstract. - if 'abstract' in new_configuration_dict: - del new_configuration_dict['abstract'] +def MergeDicts(to, fro, to_file, fro_file): + # I wanted to name the parameter "from" but it's a Python keyword... + for k, v in fro.items(): + # It would be nice to do "if not k in to: to[k] = v" but that wouldn't give + # copy semantics. Something else may want to merge from the |fro| dict + # later, and having the same dict ref pointed to twice in the tree isn't + # what anyone wants considering that the dicts may subsequently be + # modified. + if k in to: + bad_merge = False + if type(v) in (str, int): + if type(to[k]) not in (str, int): + bad_merge = True + elif not isinstance(v, type(to[k])): + bad_merge = True + + if bad_merge: + raise TypeError( + "Attempt to merge dict value of type " + + v.__class__.__name__ + + " into incompatible type " + + to[k].__class__.__name__ + + " for key " + + k + ) + if type(v) in (str, int): + # Overwrite the existing value, if any. Cheap and easy. + is_path = IsPathSection(k) + if is_path: + to[k] = MakePathRelative(to_file, fro_file, v) + else: + to[k] = v + elif type(v) is dict: + # Recurse, guaranteeing copies will be made of objects that require it. + if k not in to: + to[k] = {} + MergeDicts(to[k], v, to_file, fro_file) + elif type(v) is list: + # Lists in dicts can be merged with different policies, depending on + # how the key in the "from" dict (k, the from-key) is written. + # + # If the from-key has ...the to-list will have this action + # this character appended:... applied when receiving the from-list: + # = replace + # + prepend + # ? set, only if to-list does not yet exist + # (none) append + # + # This logic is list-specific, but since it relies on the associated + # dict key, it's checked in this dict-oriented function. + ext = k[-1] + append = True + if ext == "=": + list_base = k[:-1] + lists_incompatible = [list_base, list_base + "?"] + to[list_base] = [] + elif ext == "+": + list_base = k[:-1] + lists_incompatible = [list_base + "=", list_base + "?"] + append = False + elif ext == "?": + list_base = k[:-1] + lists_incompatible = [list_base, list_base + "=", list_base + "+"] + else: + list_base = k + lists_incompatible = [list_base + "=", list_base + "?"] + + # Some combinations of merge policies appearing together are meaningless. + # It's stupid to replace and append simultaneously, for example. Append + # and prepend are the only policies that can coexist. + for list_incompatible in lists_incompatible: + if list_incompatible in fro: + raise GypError( + "Incompatible list policies " + k + " and " + list_incompatible + ) + + if list_base in to: + if ext == "?": + # If the key ends in "?", the list will only be merged if it doesn't + # already exist. + continue + elif type(to[list_base]) is not list: + # This may not have been checked above if merging in a list with an + # extension character. + raise TypeError( + "Attempt to merge dict value of type " + + v.__class__.__name__ + + " into incompatible type " + + to[list_base].__class__.__name__ + + " for key " + + list_base + + "(" + + k + + ")" + ) + else: + to[list_base] = [] + + # Call MergeLists, which will make copies of objects that require it. + # MergeLists can recurse back into MergeDicts, although this will be + # to make copies of dicts (with paths fixed), there will be no + # subsequent dict "merging" once entering a list because lists are + # always replaced, appended to, or prepended to. + is_paths = IsPathSection(list_base) + MergeLists(to[list_base], v, to_file, fro_file, is_paths, append) + else: + raise TypeError( + "Attempt to merge dict value of unsupported type " + + v.__class__.__name__ + + " for key " + + k + ) + + +def MergeConfigWithInheritance( + new_configuration_dict, build_file, target_dict, configuration, visited +): + # Skip if previously visted. + if configuration in visited: + return + + # Look at this configuration. + configuration_dict = target_dict["configurations"][configuration] + + # Merge in parents. + for parent in configuration_dict.get("inherit_from", []): + MergeConfigWithInheritance( + new_configuration_dict, + build_file, + target_dict, + parent, + visited + [configuration], + ) + + # Merge it into the new config. + MergeDicts(new_configuration_dict, configuration_dict, build_file, build_file) + + # Drop abstract. + if "abstract" in new_configuration_dict: + del new_configuration_dict["abstract"] def SetUpConfigurations(target, target_dict): - # key_suffixes is a list of key suffixes that might appear on key names. - # These suffixes are handled in conditional evaluations (for =, +, and ?) - # and rules/exclude processing (for ! and /). Keys with these suffixes - # should be treated the same as keys without. - key_suffixes = ['=', '+', '?', '!', '/'] - - build_file = gyp.common.BuildFile(target) - - # Provide a single configuration by default if none exists. - # TODO(mark): Signal an error if default_configurations exists but - # configurations does not. - if not 'configurations' in target_dict: - target_dict['configurations'] = {'Default': {}} - if not 'default_configuration' in target_dict: - concrete = [i for (i, config) in target_dict['configurations'].items() - if not config.get('abstract')] - target_dict['default_configuration'] = sorted(concrete)[0] - - merged_configurations = {} - configs = target_dict['configurations'] - for (configuration, old_configuration_dict) in configs.items(): - # Skip abstract configurations (saves work only). - if old_configuration_dict.get('abstract'): - continue - # Configurations inherit (most) settings from the enclosing target scope. - # Get the inheritance relationship right by making a copy of the target - # dict. - new_configuration_dict = {} - for (key, target_val) in target_dict.items(): - key_ext = key[-1:] - if key_ext in key_suffixes: - key_base = key[:-1] - else: - key_base = key - if not key_base in non_configuration_keys: - new_configuration_dict[key] = gyp.simple_copy.deepcopy(target_val) - - # Merge in configuration (with all its parents first). - MergeConfigWithInheritance(new_configuration_dict, build_file, - target_dict, configuration, []) - - merged_configurations[configuration] = new_configuration_dict - - # Put the new configurations back into the target dict as a configuration. - for configuration in merged_configurations.keys(): - target_dict['configurations'][configuration] = ( - merged_configurations[configuration]) - - # Now drop all the abstract ones. - for configuration in list(target_dict['configurations']): - old_configuration_dict = target_dict['configurations'][configuration] - if old_configuration_dict.get('abstract'): - del target_dict['configurations'][configuration] - - # Now that all of the target's configurations have been built, go through - # the target dict's keys and remove everything that's been moved into a - # "configurations" section. - delete_keys = [] - for key in target_dict: - key_ext = key[-1:] - if key_ext in key_suffixes: - key_base = key[:-1] - else: - key_base = key - if not key_base in non_configuration_keys: - delete_keys.append(key) - for key in delete_keys: - del target_dict[key] + # key_suffixes is a list of key suffixes that might appear on key names. + # These suffixes are handled in conditional evaluations (for =, +, and ?) + # and rules/exclude processing (for ! and /). Keys with these suffixes + # should be treated the same as keys without. + key_suffixes = ["=", "+", "?", "!", "/"] - # Check the configurations to see if they contain invalid keys. - for configuration in target_dict['configurations'].keys(): - configuration_dict = target_dict['configurations'][configuration] - for key in configuration_dict.keys(): - if key in invalid_configuration_keys: - raise GypError('%s not allowed in the %s configuration, found in ' - 'target %s' % (key, configuration, target)) + build_file = gyp.common.BuildFile(target) + # Provide a single configuration by default if none exists. + # TODO(mark): Signal an error if default_configurations exists but + # configurations does not. + if "configurations" not in target_dict: + target_dict["configurations"] = {"Default": {}} + if "default_configuration" not in target_dict: + concrete = [ + i + for (i, config) in target_dict["configurations"].items() + if not config.get("abstract") + ] + target_dict["default_configuration"] = sorted(concrete)[0] + + merged_configurations = {} + configs = target_dict["configurations"] + for (configuration, old_configuration_dict) in configs.items(): + # Skip abstract configurations (saves work only). + if old_configuration_dict.get("abstract"): + continue + # Configurations inherit (most) settings from the enclosing target scope. + # Get the inheritance relationship right by making a copy of the target + # dict. + new_configuration_dict = {} + for (key, target_val) in target_dict.items(): + key_ext = key[-1:] + if key_ext in key_suffixes: + key_base = key[:-1] + else: + key_base = key + if key_base not in non_configuration_keys: + new_configuration_dict[key] = gyp.simple_copy.deepcopy(target_val) + + # Merge in configuration (with all its parents first). + MergeConfigWithInheritance( + new_configuration_dict, build_file, target_dict, configuration, [] + ) + + merged_configurations[configuration] = new_configuration_dict + + # Put the new configurations back into the target dict as a configuration. + for configuration in merged_configurations.keys(): + target_dict["configurations"][configuration] = merged_configurations[ + configuration + ] + + # Now drop all the abstract ones. + configs = target_dict["configurations"] + target_dict["configurations"] = { + k: v for k, v in configs.items() if not v.get("abstract") + } + + # Now that all of the target's configurations have been built, go through + # the target dict's keys and remove everything that's been moved into a + # "configurations" section. + delete_keys = [] + for key in target_dict: + key_ext = key[-1:] + if key_ext in key_suffixes: + key_base = key[:-1] + else: + key_base = key + if key_base not in non_configuration_keys: + delete_keys.append(key) + for key in delete_keys: + del target_dict[key] + + # Check the configurations to see if they contain invalid keys. + for configuration in target_dict["configurations"].keys(): + configuration_dict = target_dict["configurations"][configuration] + for key in configuration_dict.keys(): + if key in invalid_configuration_keys: + raise GypError( + "%s not allowed in the %s configuration, found in " + "target %s" % (key, configuration, target) + ) def ProcessListFiltersInDict(name, the_dict): - """Process regular expression and exclusion-based filters on lists. + """Process regular expression and exclusion-based filters on lists. An exclusion list is in a dict key named with a trailing "!", like "sources!". Every item in such a list is removed from the associated @@ -2348,151 +2561,163 @@ def ProcessListFiltersInDict(name, the_dict): patterns can still cause items to be excluded after matching an "include". """ - # Look through the dictionary for any lists whose keys end in "!" or "/". - # These are lists that will be treated as exclude lists and regular - # expression-based exclude/include lists. Collect the lists that are - # needed first, looking for the lists that they operate on, and assemble - # then into |lists|. This is done in a separate loop up front, because - # the _included and _excluded keys need to be added to the_dict, and that - # can't be done while iterating through it. - - lists = [] - del_lists = [] - for key, value in the_dict.items(): - operation = key[-1] - if operation != '!' and operation != '/': - continue - - if type(value) is not list: - raise ValueError(name + ' key ' + key + ' must be list, not ' + \ - value.__class__.__name__) - - list_key = key[:-1] - if list_key not in the_dict: - # This happens when there's a list like "sources!" but no corresponding - # "sources" list. Since there's nothing for it to operate on, queue up - # the "sources!" list for deletion now. - del_lists.append(key) - continue - - if type(the_dict[list_key]) is not list: - value = the_dict[list_key] - raise ValueError(name + ' key ' + list_key + \ - ' must be list, not ' + \ - value.__class__.__name__ + ' when applying ' + \ - {'!': 'exclusion', '/': 'regex'}[operation]) - - if not list_key in lists: - lists.append(list_key) - - # Delete the lists that are known to be unneeded at this point. - for del_list in del_lists: - del the_dict[del_list] - - for list_key in lists: - the_list = the_dict[list_key] - - # Initialize the list_actions list, which is parallel to the_list. Each - # item in list_actions identifies whether the corresponding item in - # the_list should be excluded, unconditionally preserved (included), or - # whether no exclusion or inclusion has been applied. Items for which - # no exclusion or inclusion has been applied (yet) have value -1, items - # excluded have value 0, and items included have value 1. Includes and - # excludes override previous actions. All items in list_actions are - # initialized to -1 because no excludes or includes have been processed - # yet. - list_actions = list((-1,) * len(the_list)) - - exclude_key = list_key + '!' - if exclude_key in the_dict: - for exclude_item in the_dict[exclude_key]: - for index in range(0, len(the_list)): - if exclude_item == the_list[index]: - # This item matches the exclude_item, so set its action to 0 - # (exclude). - list_actions[index] = 0 - - # The "whatever!" list is no longer needed, dump it. - del the_dict[exclude_key] - - regex_key = list_key + '/' - if regex_key in the_dict: - for regex_item in the_dict[regex_key]: - [action, pattern] = regex_item - pattern_re = re.compile(pattern) - - if action == 'exclude': - # This item matches an exclude regex, so set its value to 0 (exclude). - action_value = 0 - elif action == 'include': - # This item matches an include regex, so set its value to 1 (include). - action_value = 1 - else: - # This is an action that doesn't make any sense. - raise ValueError('Unrecognized action ' + action + ' in ' + name + \ - ' key ' + regex_key) - - for index in range(0, len(the_list)): - list_item = the_list[index] - if list_actions[index] == action_value: - # Even if the regex matches, nothing will change so continue (regex - # searches are expensive). + # Look through the dictionary for any lists whose keys end in "!" or "/". + # These are lists that will be treated as exclude lists and regular + # expression-based exclude/include lists. Collect the lists that are + # needed first, looking for the lists that they operate on, and assemble + # then into |lists|. This is done in a separate loop up front, because + # the _included and _excluded keys need to be added to the_dict, and that + # can't be done while iterating through it. + + lists = [] + del_lists = [] + for key, value in the_dict.items(): + operation = key[-1] + if operation != "!" and operation != "/": continue - if pattern_re.search(list_item): - # Regular expression match. - list_actions[index] = action_value - # The "whatever/" list is no longer needed, dump it. - del the_dict[regex_key] + if type(value) is not list: + raise ValueError( + name + " key " + key + " must be list, not " + value.__class__.__name__ + ) + + list_key = key[:-1] + if list_key not in the_dict: + # This happens when there's a list like "sources!" but no corresponding + # "sources" list. Since there's nothing for it to operate on, queue up + # the "sources!" list for deletion now. + del_lists.append(key) + continue - # Add excluded items to the excluded list. - # - # Note that exclude_key ("sources!") is different from excluded_key - # ("sources_excluded"). The exclude_key list is input and it was already - # processed and deleted; the excluded_key list is output and it's about - # to be created. - excluded_key = list_key + '_excluded' - if excluded_key in the_dict: - raise GypError(name + ' key ' + excluded_key + - ' must not be present prior ' - ' to applying exclusion/regex filters for ' + list_key) - - excluded_list = [] - - # Go backwards through the list_actions list so that as items are deleted, - # the indices of items that haven't been seen yet don't shift. That means - # that things need to be prepended to excluded_list to maintain them in the - # same order that they existed in the_list. - for index in range(len(list_actions) - 1, -1, -1): - if list_actions[index] == 0: - # Dump anything with action 0 (exclude). Keep anything with action 1 - # (include) or -1 (no include or exclude seen for the item). - excluded_list.insert(0, the_list[index]) - del the_list[index] - - # If anything was excluded, put the excluded list into the_dict at - # excluded_key. - if len(excluded_list) > 0: - the_dict[excluded_key] = excluded_list - - # Now recurse into subdicts and lists that may contain dicts. - for key, value in the_dict.items(): - if type(value) is dict: - ProcessListFiltersInDict(key, value) - elif type(value) is list: - ProcessListFiltersInList(key, value) + if type(the_dict[list_key]) is not list: + value = the_dict[list_key] + raise ValueError( + name + + " key " + + list_key + + " must be list, not " + + value.__class__.__name__ + + " when applying " + + {"!": "exclusion", "/": "regex"}[operation] + ) + + if list_key not in lists: + lists.append(list_key) + + # Delete the lists that are known to be unneeded at this point. + for del_list in del_lists: + del the_dict[del_list] + + for list_key in lists: + the_list = the_dict[list_key] + + # Initialize the list_actions list, which is parallel to the_list. Each + # item in list_actions identifies whether the corresponding item in + # the_list should be excluded, unconditionally preserved (included), or + # whether no exclusion or inclusion has been applied. Items for which + # no exclusion or inclusion has been applied (yet) have value -1, items + # excluded have value 0, and items included have value 1. Includes and + # excludes override previous actions. All items in list_actions are + # initialized to -1 because no excludes or includes have been processed + # yet. + list_actions = list((-1,) * len(the_list)) + + exclude_key = list_key + "!" + if exclude_key in the_dict: + for exclude_item in the_dict[exclude_key]: + for index, list_item in enumerate(the_list): + if exclude_item == list_item: + # This item matches the exclude_item, so set its action to 0 + # (exclude). + list_actions[index] = 0 + + # The "whatever!" list is no longer needed, dump it. + del the_dict[exclude_key] + + regex_key = list_key + "/" + if regex_key in the_dict: + for regex_item in the_dict[regex_key]: + [action, pattern] = regex_item + pattern_re = re.compile(pattern) + + if action == "exclude": + # This item matches an exclude regex, so set its value to 0 (exclude). + action_value = 0 + elif action == "include": + # This item matches an include regex, so set its value to 1 (include). + action_value = 1 + else: + # This is an action that doesn't make any sense. + raise ValueError( + "Unrecognized action " + + action + + " in " + + name + + " key " + + regex_key + ) + + for index, list_item in enumerate(the_list): + if list_actions[index] == action_value: + # Even if the regex matches, nothing will change so continue (regex + # searches are expensive). + continue + if pattern_re.search(list_item): + # Regular expression match. + list_actions[index] = action_value + + # The "whatever/" list is no longer needed, dump it. + del the_dict[regex_key] + + # Add excluded items to the excluded list. + # + # Note that exclude_key ("sources!") is different from excluded_key + # ("sources_excluded"). The exclude_key list is input and it was already + # processed and deleted; the excluded_key list is output and it's about + # to be created. + excluded_key = list_key + "_excluded" + if excluded_key in the_dict: + raise GypError( + name + " key " + excluded_key + " must not be present prior " + " to applying exclusion/regex filters for " + list_key + ) + + excluded_list = [] + + # Go backwards through the list_actions list so that as items are deleted, + # the indices of items that haven't been seen yet don't shift. That means + # that things need to be prepended to excluded_list to maintain them in the + # same order that they existed in the_list. + for index in range(len(list_actions) - 1, -1, -1): + if list_actions[index] == 0: + # Dump anything with action 0 (exclude). Keep anything with action 1 + # (include) or -1 (no include or exclude seen for the item). + excluded_list.insert(0, the_list[index]) + del the_list[index] + + # If anything was excluded, put the excluded list into the_dict at + # excluded_key. + if len(excluded_list) > 0: + the_dict[excluded_key] = excluded_list + + # Now recurse into subdicts and lists that may contain dicts. + for key, value in the_dict.items(): + if type(value) is dict: + ProcessListFiltersInDict(key, value) + elif type(value) is list: + ProcessListFiltersInList(key, value) def ProcessListFiltersInList(name, the_list): - for item in the_list: - if type(item) is dict: - ProcessListFiltersInDict(name, item) - elif type(item) is list: - ProcessListFiltersInList(name, item) + for item in the_list: + if type(item) is dict: + ProcessListFiltersInDict(name, item) + elif type(item) is list: + ProcessListFiltersInList(name, item) def ValidateTargetType(target, target_dict): - """Ensures the 'type' field on the target is one of the known types. + """Ensures the 'type' field on the target is one of the known types. Arguments: target: string, name of target. @@ -2500,52 +2725,63 @@ def ValidateTargetType(target, target_dict): Raises an exception on error. """ - VALID_TARGET_TYPES = ('executable', 'loadable_module', - 'static_library', 'shared_library', - 'mac_kernel_extension', 'none', 'windows_driver') - target_type = target_dict.get('type', None) - if target_type not in VALID_TARGET_TYPES: - raise GypError("Target %s has an invalid target type '%s'. " - "Must be one of %s." % - (target, target_type, '/'.join(VALID_TARGET_TYPES))) - if (target_dict.get('standalone_static_library', 0) and - not target_type == 'static_library'): - raise GypError('Target %s has type %s but standalone_static_library flag is' - ' only valid for static_library type.' % (target, - target_type)) - - -def ValidateSourcesInTarget(target, target_dict, build_file, - duplicate_basename_check): - if not duplicate_basename_check: - return - if target_dict.get('type', None) != 'static_library': - return - sources = target_dict.get('sources', []) - basenames = {} - for source in sources: - name, ext = os.path.splitext(source) - is_compiled_file = ext in [ - '.c', '.cc', '.cpp', '.cxx', '.m', '.mm', '.s', '.S'] - if not is_compiled_file: - continue - basename = os.path.basename(name) # Don't include extension. - basenames.setdefault(basename, []).append(source) - - error = '' - for basename, files in basenames.items(): - if len(files) > 1: - error += ' %s: %s\n' % (basename, ' '.join(files)) - - if error: - print('static library %s has several files with the same basename:\n' % target - + error + 'libtool on Mac cannot handle that. Use ' - '--no-duplicate-basename-check to disable this validation.') - raise GypError('Duplicate basenames in sources section, see list above') + VALID_TARGET_TYPES = ( + "executable", + "loadable_module", + "static_library", + "shared_library", + "mac_kernel_extension", + "none", + "windows_driver", + ) + target_type = target_dict.get("type", None) + if target_type not in VALID_TARGET_TYPES: + raise GypError( + "Target %s has an invalid target type '%s'. " + "Must be one of %s." % (target, target_type, "/".join(VALID_TARGET_TYPES)) + ) + if ( + target_dict.get("standalone_static_library", 0) + and not target_type == "static_library" + ): + raise GypError( + "Target %s has type %s but standalone_static_library flag is" + " only valid for static_library type." % (target, target_type) + ) + + +def ValidateSourcesInTarget(target, target_dict, build_file, duplicate_basename_check): + if not duplicate_basename_check: + return + if target_dict.get("type", None) != "static_library": + return + sources = target_dict.get("sources", []) + basenames = {} + for source in sources: + name, ext = os.path.splitext(source) + is_compiled_file = ext in [".c", ".cc", ".cpp", ".cxx", ".m", ".mm", ".s", ".S"] + if not is_compiled_file: + continue + basename = os.path.basename(name) # Don't include extension. + basenames.setdefault(basename, []).append(source) + + error = "" + for basename, files in basenames.items(): + if len(files) > 1: + error += " %s: %s\n" % (basename, " ".join(files)) + + if error: + print( + "static library %s has several files with the same basename:\n" % target + + error + + "libtool on Mac cannot handle that. Use " + "--no-duplicate-basename-check to disable this validation." + ) + raise GypError("Duplicate basenames in sources section, see list above") def ValidateRulesInTarget(target, target_dict, extra_sources_for_rules): - """Ensures that the rules sections in target_dict are valid and consistent, + """Ensures that the rules sections in target_dict are valid and consistent, and determines which sources they apply to. Arguments: @@ -2555,357 +2791,393 @@ def ValidateRulesInTarget(target, target_dict, extra_sources_for_rules): addition to 'sources'. """ - # Dicts to map between values found in rules' 'rule_name' and 'extension' - # keys and the rule dicts themselves. - rule_names = {} - rule_extensions = {} - - rules = target_dict.get('rules', []) - for rule in rules: - # Make sure that there's no conflict among rule names and extensions. - rule_name = rule['rule_name'] - if rule_name in rule_names: - raise GypError('rule %s exists in duplicate, target %s' % - (rule_name, target)) - rule_names[rule_name] = rule - - rule_extension = rule['extension'] - if rule_extension.startswith('.'): - rule_extension = rule_extension[1:] - if rule_extension in rule_extensions: - raise GypError(('extension %s associated with multiple rules, ' + - 'target %s rules %s and %s') % - (rule_extension, target, - rule_extensions[rule_extension]['rule_name'], - rule_name)) - rule_extensions[rule_extension] = rule - - # Make sure rule_sources isn't already there. It's going to be - # created below if needed. - if 'rule_sources' in rule: - raise GypError( - 'rule_sources must not exist in input, target %s rule %s' % - (target, rule_name)) - - rule_sources = [] - source_keys = ['sources'] - source_keys.extend(extra_sources_for_rules) - for source_key in source_keys: - for source in target_dict.get(source_key, []): - (source_root, source_extension) = os.path.splitext(source) - if source_extension.startswith('.'): - source_extension = source_extension[1:] - if source_extension == rule_extension: - rule_sources.append(source) - - if len(rule_sources) > 0: - rule['rule_sources'] = rule_sources + # Dicts to map between values found in rules' 'rule_name' and 'extension' + # keys and the rule dicts themselves. + rule_names = {} + rule_extensions = {} + + rules = target_dict.get("rules", []) + for rule in rules: + # Make sure that there's no conflict among rule names and extensions. + rule_name = rule["rule_name"] + if rule_name in rule_names: + raise GypError( + "rule %s exists in duplicate, target %s" % (rule_name, target) + ) + rule_names[rule_name] = rule + + rule_extension = rule["extension"] + if rule_extension.startswith("."): + rule_extension = rule_extension[1:] + if rule_extension in rule_extensions: + raise GypError( + ( + "extension %s associated with multiple rules, " + + "target %s rules %s and %s" + ) + % ( + rule_extension, + target, + rule_extensions[rule_extension]["rule_name"], + rule_name, + ) + ) + rule_extensions[rule_extension] = rule + + # Make sure rule_sources isn't already there. It's going to be + # created below if needed. + if "rule_sources" in rule: + raise GypError( + "rule_sources must not exist in input, target %s rule %s" + % (target, rule_name) + ) + + rule_sources = [] + source_keys = ["sources"] + source_keys.extend(extra_sources_for_rules) + for source_key in source_keys: + for source in target_dict.get(source_key, []): + (source_root, source_extension) = os.path.splitext(source) + if source_extension.startswith("."): + source_extension = source_extension[1:] + if source_extension == rule_extension: + rule_sources.append(source) + + if len(rule_sources) > 0: + rule["rule_sources"] = rule_sources def ValidateRunAsInTarget(target, target_dict, build_file): - target_name = target_dict.get('target_name') - run_as = target_dict.get('run_as') - if not run_as: - return - if type(run_as) is not dict: - raise GypError("The 'run_as' in target %s from file %s should be a " - "dictionary." % - (target_name, build_file)) - action = run_as.get('action') - if not action: - raise GypError("The 'run_as' in target %s from file %s must have an " - "'action' section." % - (target_name, build_file)) - if type(action) is not list: - raise GypError("The 'action' for 'run_as' in target %s from file %s " - "must be a list." % - (target_name, build_file)) - working_directory = run_as.get('working_directory') - if working_directory and type(working_directory) is not str: - raise GypError("The 'working_directory' for 'run_as' in target %s " - "in file %s should be a string." % - (target_name, build_file)) - environment = run_as.get('environment') - if environment and type(environment) is not dict: - raise GypError("The 'environment' for 'run_as' in target %s " - "in file %s should be a dictionary." % - (target_name, build_file)) + target_name = target_dict.get("target_name") + run_as = target_dict.get("run_as") + if not run_as: + return + if type(run_as) is not dict: + raise GypError( + "The 'run_as' in target %s from file %s should be a " + "dictionary." % (target_name, build_file) + ) + action = run_as.get("action") + if not action: + raise GypError( + "The 'run_as' in target %s from file %s must have an " + "'action' section." % (target_name, build_file) + ) + if type(action) is not list: + raise GypError( + "The 'action' for 'run_as' in target %s from file %s " + "must be a list." % (target_name, build_file) + ) + working_directory = run_as.get("working_directory") + if working_directory and type(working_directory) is not str: + raise GypError( + "The 'working_directory' for 'run_as' in target %s " + "in file %s should be a string." % (target_name, build_file) + ) + environment = run_as.get("environment") + if environment and type(environment) is not dict: + raise GypError( + "The 'environment' for 'run_as' in target %s " + "in file %s should be a dictionary." % (target_name, build_file) + ) def ValidateActionsInTarget(target, target_dict, build_file): - '''Validates the inputs to the actions in a target.''' - target_name = target_dict.get('target_name') - actions = target_dict.get('actions', []) - for action in actions: - action_name = action.get('action_name') - if not action_name: - raise GypError("Anonymous action in target %s. " - "An action must have an 'action_name' field." % - target_name) - inputs = action.get('inputs', None) - if inputs is None: - raise GypError('Action in target %s has no inputs.' % target_name) - action_command = action.get('action') - if action_command and not action_command[0]: - raise GypError("Empty action as command in target %s." % target_name) + """Validates the inputs to the actions in a target.""" + target_name = target_dict.get("target_name") + actions = target_dict.get("actions", []) + for action in actions: + action_name = action.get("action_name") + if not action_name: + raise GypError( + "Anonymous action in target %s. " + "An action must have an 'action_name' field." % target_name + ) + inputs = action.get("inputs", None) + if inputs is None: + raise GypError("Action in target %s has no inputs." % target_name) + action_command = action.get("action") + if action_command and not action_command[0]: + raise GypError("Empty action as command in target %s." % target_name) def TurnIntIntoStrInDict(the_dict): - """Given dict the_dict, recursively converts all integers into strings. + """Given dict the_dict, recursively converts all integers into strings. """ - # Use items instead of iteritems because there's no need to try to look at - # reinserted keys and their associated values. - for k, v in the_dict.items(): - if type(v) is int: - v = str(v) - the_dict[k] = v - elif type(v) is dict: - TurnIntIntoStrInDict(v) - elif type(v) is list: - TurnIntIntoStrInList(v) - - if type(k) is int: - del the_dict[k] - the_dict[str(k)] = v + # Use items instead of iteritems because there's no need to try to look at + # reinserted keys and their associated values. + for k, v in the_dict.items(): + if type(v) is int: + v = str(v) + the_dict[k] = v + elif type(v) is dict: + TurnIntIntoStrInDict(v) + elif type(v) is list: + TurnIntIntoStrInList(v) + + if type(k) is int: + del the_dict[k] + the_dict[str(k)] = v def TurnIntIntoStrInList(the_list): - """Given list the_list, recursively converts all integers into strings. + """Given list the_list, recursively converts all integers into strings. """ - for index in range(0, len(the_list)): - item = the_list[index] - if type(item) is int: - the_list[index] = str(item) - elif type(item) is dict: - TurnIntIntoStrInDict(item) - elif type(item) is list: - TurnIntIntoStrInList(item) - - -def PruneUnwantedTargets(targets, flat_list, dependency_nodes, root_targets, - data): - """Return only the targets that are deep dependencies of |root_targets|.""" - qualified_root_targets = [] - for target in root_targets: - target = target.strip() - qualified_targets = gyp.common.FindQualifiedTargets(target, flat_list) - if not qualified_targets: - raise GypError("Could not find target %s" % target) - qualified_root_targets.extend(qualified_targets) - - wanted_targets = {} - for target in qualified_root_targets: - wanted_targets[target] = targets[target] - for dependency in dependency_nodes[target].DeepDependencies(): - wanted_targets[dependency] = targets[dependency] - - wanted_flat_list = [t for t in flat_list if t in wanted_targets] - - # Prune unwanted targets from each build_file's data dict. - for build_file in data['target_build_files']: - if not 'targets' in data[build_file]: - continue - new_targets = [] - for target in data[build_file]['targets']: - qualified_name = gyp.common.QualifiedTarget(build_file, - target['target_name'], - target['toolset']) - if qualified_name in wanted_targets: - new_targets.append(target) - data[build_file]['targets'] = new_targets - - return wanted_targets, wanted_flat_list + for index, item in enumerate(the_list): + if type(item) is int: + the_list[index] = str(item) + elif type(item) is dict: + TurnIntIntoStrInDict(item) + elif type(item) is list: + TurnIntIntoStrInList(item) + + +def PruneUnwantedTargets(targets, flat_list, dependency_nodes, root_targets, data): + """Return only the targets that are deep dependencies of |root_targets|.""" + qualified_root_targets = [] + for target in root_targets: + target = target.strip() + qualified_targets = gyp.common.FindQualifiedTargets(target, flat_list) + if not qualified_targets: + raise GypError("Could not find target %s" % target) + qualified_root_targets.extend(qualified_targets) + + wanted_targets = {} + for target in qualified_root_targets: + wanted_targets[target] = targets[target] + for dependency in dependency_nodes[target].DeepDependencies(): + wanted_targets[dependency] = targets[dependency] + + wanted_flat_list = [t for t in flat_list if t in wanted_targets] + + # Prune unwanted targets from each build_file's data dict. + for build_file in data["target_build_files"]: + if "targets" not in data[build_file]: + continue + new_targets = [] + for target in data[build_file]["targets"]: + qualified_name = gyp.common.QualifiedTarget( + build_file, target["target_name"], target["toolset"] + ) + if qualified_name in wanted_targets: + new_targets.append(target) + data[build_file]["targets"] = new_targets + + return wanted_targets, wanted_flat_list def VerifyNoCollidingTargets(targets): - """Verify that no two targets in the same directory share the same name. + """Verify that no two targets in the same directory share the same name. Arguments: targets: A list of targets in the form 'path/to/file.gyp:target_name'. """ - # Keep a dict going from 'subdirectory:target_name' to 'foo.gyp'. - used = {} - for target in targets: - # Separate out 'path/to/file.gyp, 'target_name' from - # 'path/to/file.gyp:target_name'. - path, name = target.rsplit(':', 1) - # Separate out 'path/to', 'file.gyp' from 'path/to/file.gyp'. - subdir, gyp = os.path.split(path) - # Use '.' for the current directory '', so that the error messages make - # more sense. - if not subdir: - subdir = '.' - # Prepare a key like 'path/to:target_name'. - key = subdir + ':' + name - if key in used: - # Complain if this target is already used. - raise GypError('Duplicate target name "%s" in directory "%s" used both ' - 'in "%s" and "%s".' % (name, subdir, gyp, used[key])) - used[key] = gyp + # Keep a dict going from 'subdirectory:target_name' to 'foo.gyp'. + used = {} + for target in targets: + # Separate out 'path/to/file.gyp, 'target_name' from + # 'path/to/file.gyp:target_name'. + path, name = target.rsplit(":", 1) + # Separate out 'path/to', 'file.gyp' from 'path/to/file.gyp'. + subdir, gyp = os.path.split(path) + # Use '.' for the current directory '', so that the error messages make + # more sense. + if not subdir: + subdir = "." + # Prepare a key like 'path/to:target_name'. + key = subdir + ":" + name + if key in used: + # Complain if this target is already used. + raise GypError( + 'Duplicate target name "%s" in directory "%s" used both ' + 'in "%s" and "%s".' % (name, subdir, gyp, used[key]) + ) + used[key] = gyp def SetGeneratorGlobals(generator_input_info): - # Set up path_sections and non_configuration_keys with the default data plus - # the generator-specific data. - global path_sections - path_sections = set(base_path_sections) - path_sections.update(generator_input_info['path_sections']) - - global non_configuration_keys - non_configuration_keys = base_non_configuration_keys[:] - non_configuration_keys.extend(generator_input_info['non_configuration_keys']) - - global multiple_toolsets - multiple_toolsets = generator_input_info[ - 'generator_supports_multiple_toolsets'] - - global generator_filelist_paths - generator_filelist_paths = generator_input_info['generator_filelist_paths'] - - -def Load(build_files, variables, includes, depth, generator_input_info, check, - circular_check, duplicate_basename_check, parallel, root_targets): - SetGeneratorGlobals(generator_input_info) - # A generator can have other lists (in addition to sources) be processed - # for rules. - extra_sources_for_rules = generator_input_info['extra_sources_for_rules'] - - # Load build files. This loads every target-containing build file into - # the |data| dictionary such that the keys to |data| are build file names, - # and the values are the entire build file contents after "early" or "pre" - # processing has been done and includes have been resolved. - # NOTE: data contains both "target" files (.gyp) and "includes" (.gypi), as - # well as meta-data (e.g. 'included_files' key). 'target_build_files' keeps - # track of the keys corresponding to "target" files. - data = {'target_build_files': set()} - # Normalize paths everywhere. This is important because paths will be - # used as keys to the data dict and for references between input files. - build_files = set(map(os.path.normpath, build_files)) - if parallel: - LoadTargetBuildFilesParallel(build_files, data, variables, includes, depth, - check, generator_input_info) - else: - aux_data = {} - for build_file in build_files: - try: - LoadTargetBuildFile(build_file, data, aux_data, - variables, includes, depth, check, True) - except Exception as e: - gyp.common.ExceptionAppend(e, 'while trying to load %s' % build_file) - raise - - # Build a dict to access each target's subdict by qualified name. - targets = BuildTargetsDict(data) - - # Fully qualify all dependency links. - QualifyDependencies(targets) - - # Remove self-dependencies from targets that have 'prune_self_dependencies' - # set to 1. - RemoveSelfDependencies(targets) - - # Expand dependencies specified as build_file:*. - ExpandWildcardDependencies(targets, data) - - # Remove all dependencies marked as 'link_dependency' from the targets of - # type 'none'. - RemoveLinkDependenciesFromNoneTargets(targets) - - # Apply exclude (!) and regex (/) list filters only for dependency_sections. - for target_name, target_dict in targets.items(): - tmp_dict = {} - for key_base in dependency_sections: - for op in ('', '!', '/'): - key = key_base + op - if key in target_dict: - tmp_dict[key] = target_dict[key] - del target_dict[key] - ProcessListFiltersInDict(target_name, tmp_dict) - # Write the results back to |target_dict|. - for key in tmp_dict: - target_dict[key] = tmp_dict[key] - - # Make sure every dependency appears at most once. - RemoveDuplicateDependencies(targets) - - if circular_check: - # Make sure that any targets in a.gyp don't contain dependencies in other - # .gyp files that further depend on a.gyp. - VerifyNoGYPFileCircularDependencies(targets) - - [dependency_nodes, flat_list] = BuildDependencyList(targets) - - if root_targets: - # Remove, from |targets| and |flat_list|, the targets that are not deep - # dependencies of the targets specified in |root_targets|. - targets, flat_list = PruneUnwantedTargets( - targets, flat_list, dependency_nodes, root_targets, data) - - # Check that no two targets in the same directory have the same name. - VerifyNoCollidingTargets(flat_list) - - # Handle dependent settings of various types. - for settings_type in ['all_dependent_settings', - 'direct_dependent_settings', - 'link_settings']: - DoDependentSettings(settings_type, flat_list, targets, dependency_nodes) - - # Take out the dependent settings now that they've been published to all - # of the targets that require them. + # Set up path_sections and non_configuration_keys with the default data plus + # the generator-specific data. + global path_sections + path_sections = set(base_path_sections) + path_sections.update(generator_input_info["path_sections"]) + + global non_configuration_keys + non_configuration_keys = base_non_configuration_keys[:] + non_configuration_keys.extend(generator_input_info["non_configuration_keys"]) + + global multiple_toolsets + multiple_toolsets = generator_input_info["generator_supports_multiple_toolsets"] + + global generator_filelist_paths + generator_filelist_paths = generator_input_info["generator_filelist_paths"] + + +def Load( + build_files, + variables, + includes, + depth, + generator_input_info, + check, + circular_check, + duplicate_basename_check, + parallel, + root_targets, +): + SetGeneratorGlobals(generator_input_info) + # A generator can have other lists (in addition to sources) be processed + # for rules. + extra_sources_for_rules = generator_input_info["extra_sources_for_rules"] + + # Load build files. This loads every target-containing build file into + # the |data| dictionary such that the keys to |data| are build file names, + # and the values are the entire build file contents after "early" or "pre" + # processing has been done and includes have been resolved. + # NOTE: data contains both "target" files (.gyp) and "includes" (.gypi), as + # well as meta-data (e.g. 'included_files' key). 'target_build_files' keeps + # track of the keys corresponding to "target" files. + data = {"target_build_files": set()} + # Normalize paths everywhere. This is important because paths will be + # used as keys to the data dict and for references between input files. + build_files = set(map(os.path.normpath, build_files)) + if parallel: + LoadTargetBuildFilesParallel( + build_files, data, variables, includes, depth, check, generator_input_info + ) + else: + aux_data = {} + for build_file in build_files: + try: + LoadTargetBuildFile( + build_file, data, aux_data, variables, includes, depth, check, True + ) + except Exception as e: + gyp.common.ExceptionAppend(e, "while trying to load %s" % build_file) + raise + + # Build a dict to access each target's subdict by qualified name. + targets = BuildTargetsDict(data) + + # Fully qualify all dependency links. + QualifyDependencies(targets) + + # Remove self-dependencies from targets that have 'prune_self_dependencies' + # set to 1. + RemoveSelfDependencies(targets) + + # Expand dependencies specified as build_file:*. + ExpandWildcardDependencies(targets, data) + + # Remove all dependencies marked as 'link_dependency' from the targets of + # type 'none'. + RemoveLinkDependenciesFromNoneTargets(targets) + + # Apply exclude (!) and regex (/) list filters only for dependency_sections. + for target_name, target_dict in targets.items(): + tmp_dict = {} + for key_base in dependency_sections: + for op in ("", "!", "/"): + key = key_base + op + if key in target_dict: + tmp_dict[key] = target_dict[key] + del target_dict[key] + ProcessListFiltersInDict(target_name, tmp_dict) + # Write the results back to |target_dict|. + for key in tmp_dict: + target_dict[key] = tmp_dict[key] + + # Make sure every dependency appears at most once. + RemoveDuplicateDependencies(targets) + + if circular_check: + # Make sure that any targets in a.gyp don't contain dependencies in other + # .gyp files that further depend on a.gyp. + VerifyNoGYPFileCircularDependencies(targets) + + [dependency_nodes, flat_list] = BuildDependencyList(targets) + + if root_targets: + # Remove, from |targets| and |flat_list|, the targets that are not deep + # dependencies of the targets specified in |root_targets|. + targets, flat_list = PruneUnwantedTargets( + targets, flat_list, dependency_nodes, root_targets, data + ) + + # Check that no two targets in the same directory have the same name. + VerifyNoCollidingTargets(flat_list) + + # Handle dependent settings of various types. + for settings_type in [ + "all_dependent_settings", + "direct_dependent_settings", + "link_settings", + ]: + DoDependentSettings(settings_type, flat_list, targets, dependency_nodes) + + # Take out the dependent settings now that they've been published to all + # of the targets that require them. + for target in flat_list: + if settings_type in targets[target]: + del targets[target][settings_type] + + # Make sure static libraries don't declare dependencies on other static + # libraries, but that linkables depend on all unlinked static libraries + # that they need so that their link steps will be correct. + gii = generator_input_info + if gii["generator_wants_static_library_dependencies_adjusted"]: + AdjustStaticLibraryDependencies( + flat_list, + targets, + dependency_nodes, + gii["generator_wants_sorted_dependencies"], + ) + + # Apply "post"/"late"/"target" variable expansions and condition evaluations. for target in flat_list: - if settings_type in targets[target]: - del targets[target][settings_type] - - # Make sure static libraries don't declare dependencies on other static - # libraries, but that linkables depend on all unlinked static libraries - # that they need so that their link steps will be correct. - gii = generator_input_info - if gii['generator_wants_static_library_dependencies_adjusted']: - AdjustStaticLibraryDependencies(flat_list, targets, dependency_nodes, - gii['generator_wants_sorted_dependencies']) - - # Apply "post"/"late"/"target" variable expansions and condition evaluations. - for target in flat_list: - target_dict = targets[target] - build_file = gyp.common.BuildFile(target) - ProcessVariablesAndConditionsInDict( - target_dict, PHASE_LATE, variables, build_file) + target_dict = targets[target] + build_file = gyp.common.BuildFile(target) + ProcessVariablesAndConditionsInDict( + target_dict, PHASE_LATE, variables, build_file + ) - # Move everything that can go into a "configurations" section into one. - for target in flat_list: - target_dict = targets[target] - SetUpConfigurations(target, target_dict) + # Move everything that can go into a "configurations" section into one. + for target in flat_list: + target_dict = targets[target] + SetUpConfigurations(target, target_dict) - # Apply exclude (!) and regex (/) list filters. - for target in flat_list: - target_dict = targets[target] - ProcessListFiltersInDict(target, target_dict) + # Apply exclude (!) and regex (/) list filters. + for target in flat_list: + target_dict = targets[target] + ProcessListFiltersInDict(target, target_dict) - # Apply "latelate" variable expansions and condition evaluations. - for target in flat_list: - target_dict = targets[target] - build_file = gyp.common.BuildFile(target) - ProcessVariablesAndConditionsInDict( - target_dict, PHASE_LATELATE, variables, build_file) - - # Make sure that the rules make sense, and build up rule_sources lists as - # needed. Not all generators will need to use the rule_sources lists, but - # some may, and it seems best to build the list in a common spot. - # Also validate actions and run_as elements in targets. - for target in flat_list: - target_dict = targets[target] - build_file = gyp.common.BuildFile(target) - ValidateTargetType(target, target_dict) - ValidateSourcesInTarget(target, target_dict, build_file, - duplicate_basename_check) - ValidateRulesInTarget(target, target_dict, extra_sources_for_rules) - ValidateRunAsInTarget(target, target_dict, build_file) - ValidateActionsInTarget(target, target_dict, build_file) - - # Generators might not expect ints. Turn them into strs. - TurnIntIntoStrInDict(data) - - # TODO(mark): Return |data| for now because the generator needs a list of - # build files that came in. In the future, maybe it should just accept - # a list, and not the whole data dict. - return [flat_list, targets, data] + # Apply "latelate" variable expansions and condition evaluations. + for target in flat_list: + target_dict = targets[target] + build_file = gyp.common.BuildFile(target) + ProcessVariablesAndConditionsInDict( + target_dict, PHASE_LATELATE, variables, build_file + ) + + # Make sure that the rules make sense, and build up rule_sources lists as + # needed. Not all generators will need to use the rule_sources lists, but + # some may, and it seems best to build the list in a common spot. + # Also validate actions and run_as elements in targets. + for target in flat_list: + target_dict = targets[target] + build_file = gyp.common.BuildFile(target) + ValidateTargetType(target, target_dict) + ValidateSourcesInTarget( + target, target_dict, build_file, duplicate_basename_check + ) + ValidateRulesInTarget(target, target_dict, extra_sources_for_rules) + ValidateRunAsInTarget(target, target_dict, build_file) + ValidateActionsInTarget(target, target_dict, build_file) + + # Generators might not expect ints. Turn them into strs. + TurnIntIntoStrInDict(data) + + # TODO(mark): Return |data| for now because the generator needs a list of + # build files that came in. In the future, maybe it should just accept + # a list, and not the whole data dict. + return [flat_list, targets, data] diff --git a/gyp/pylib/gyp/input_test.py b/gyp/pylib/gyp/input_test.py index 1bc5e3d308..6672ddc014 100755 --- a/gyp/pylib/gyp/input_test.py +++ b/gyp/pylib/gyp/input_test.py @@ -8,83 +8,91 @@ import gyp.input import unittest -import sys class TestFindCycles(unittest.TestCase): - def setUp(self): - self.nodes = {} - for x in ('a', 'b', 'c', 'd', 'e'): - self.nodes[x] = gyp.input.DependencyGraphNode(x) - - def _create_dependency(self, dependent, dependency): - dependent.dependencies.append(dependency) - dependency.dependents.append(dependent) - - def test_no_cycle_empty_graph(self): - for label, node in self.nodes.items(): - self.assertEqual([], node.FindCycles()) - - def test_no_cycle_line(self): - self._create_dependency(self.nodes['a'], self.nodes['b']) - self._create_dependency(self.nodes['b'], self.nodes['c']) - self._create_dependency(self.nodes['c'], self.nodes['d']) - - for label, node in self.nodes.items(): - self.assertEqual([], node.FindCycles()) - - def test_no_cycle_dag(self): - self._create_dependency(self.nodes['a'], self.nodes['b']) - self._create_dependency(self.nodes['a'], self.nodes['c']) - self._create_dependency(self.nodes['b'], self.nodes['c']) - - for label, node in self.nodes.items(): - self.assertEqual([], node.FindCycles()) - - def test_cycle_self_reference(self): - self._create_dependency(self.nodes['a'], self.nodes['a']) - - self.assertEqual([[self.nodes['a'], self.nodes['a']]], - self.nodes['a'].FindCycles()) - - def test_cycle_two_nodes(self): - self._create_dependency(self.nodes['a'], self.nodes['b']) - self._create_dependency(self.nodes['b'], self.nodes['a']) - - self.assertEqual([[self.nodes['a'], self.nodes['b'], self.nodes['a']]], - self.nodes['a'].FindCycles()) - self.assertEqual([[self.nodes['b'], self.nodes['a'], self.nodes['b']]], - self.nodes['b'].FindCycles()) - - def test_two_cycles(self): - self._create_dependency(self.nodes['a'], self.nodes['b']) - self._create_dependency(self.nodes['b'], self.nodes['a']) - - self._create_dependency(self.nodes['b'], self.nodes['c']) - self._create_dependency(self.nodes['c'], self.nodes['b']) - - cycles = self.nodes['a'].FindCycles() - self.assertTrue( - [self.nodes['a'], self.nodes['b'], self.nodes['a']] in cycles) - self.assertTrue( - [self.nodes['b'], self.nodes['c'], self.nodes['b']] in cycles) - self.assertEqual(2, len(cycles)) - - def test_big_cycle(self): - self._create_dependency(self.nodes['a'], self.nodes['b']) - self._create_dependency(self.nodes['b'], self.nodes['c']) - self._create_dependency(self.nodes['c'], self.nodes['d']) - self._create_dependency(self.nodes['d'], self.nodes['e']) - self._create_dependency(self.nodes['e'], self.nodes['a']) - - self.assertEqual([[self.nodes['a'], - self.nodes['b'], - self.nodes['c'], - self.nodes['d'], - self.nodes['e'], - self.nodes['a']]], - self.nodes['a'].FindCycles()) - - -if __name__ == '__main__': - unittest.main() + def setUp(self): + self.nodes = {} + for x in ("a", "b", "c", "d", "e"): + self.nodes[x] = gyp.input.DependencyGraphNode(x) + + def _create_dependency(self, dependent, dependency): + dependent.dependencies.append(dependency) + dependency.dependents.append(dependent) + + def test_no_cycle_empty_graph(self): + for label, node in self.nodes.items(): + self.assertEqual([], node.FindCycles()) + + def test_no_cycle_line(self): + self._create_dependency(self.nodes["a"], self.nodes["b"]) + self._create_dependency(self.nodes["b"], self.nodes["c"]) + self._create_dependency(self.nodes["c"], self.nodes["d"]) + + for label, node in self.nodes.items(): + self.assertEqual([], node.FindCycles()) + + def test_no_cycle_dag(self): + self._create_dependency(self.nodes["a"], self.nodes["b"]) + self._create_dependency(self.nodes["a"], self.nodes["c"]) + self._create_dependency(self.nodes["b"], self.nodes["c"]) + + for label, node in self.nodes.items(): + self.assertEqual([], node.FindCycles()) + + def test_cycle_self_reference(self): + self._create_dependency(self.nodes["a"], self.nodes["a"]) + + self.assertEqual( + [[self.nodes["a"], self.nodes["a"]]], self.nodes["a"].FindCycles() + ) + + def test_cycle_two_nodes(self): + self._create_dependency(self.nodes["a"], self.nodes["b"]) + self._create_dependency(self.nodes["b"], self.nodes["a"]) + + self.assertEqual( + [[self.nodes["a"], self.nodes["b"], self.nodes["a"]]], + self.nodes["a"].FindCycles(), + ) + self.assertEqual( + [[self.nodes["b"], self.nodes["a"], self.nodes["b"]]], + self.nodes["b"].FindCycles(), + ) + + def test_two_cycles(self): + self._create_dependency(self.nodes["a"], self.nodes["b"]) + self._create_dependency(self.nodes["b"], self.nodes["a"]) + + self._create_dependency(self.nodes["b"], self.nodes["c"]) + self._create_dependency(self.nodes["c"], self.nodes["b"]) + + cycles = self.nodes["a"].FindCycles() + self.assertTrue([self.nodes["a"], self.nodes["b"], self.nodes["a"]] in cycles) + self.assertTrue([self.nodes["b"], self.nodes["c"], self.nodes["b"]] in cycles) + self.assertEqual(2, len(cycles)) + + def test_big_cycle(self): + self._create_dependency(self.nodes["a"], self.nodes["b"]) + self._create_dependency(self.nodes["b"], self.nodes["c"]) + self._create_dependency(self.nodes["c"], self.nodes["d"]) + self._create_dependency(self.nodes["d"], self.nodes["e"]) + self._create_dependency(self.nodes["e"], self.nodes["a"]) + + self.assertEqual( + [ + [ + self.nodes["a"], + self.nodes["b"], + self.nodes["c"], + self.nodes["d"], + self.nodes["e"], + self.nodes["a"], + ] + ], + self.nodes["a"].FindCycles(), + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/gyp/pylib/gyp/mac_tool.py b/gyp/pylib/gyp/mac_tool.py index 781a8633bc..07412578d1 100755 --- a/gyp/pylib/gyp/mac_tool.py +++ b/gyp/pylib/gyp/mac_tool.py @@ -18,7 +18,6 @@ import plistlib import re import shutil -import string import struct import subprocess import sys @@ -28,328 +27,345 @@ def main(args): - executor = MacTool() - exit_code = executor.Dispatch(args) - if exit_code is not None: - sys.exit(exit_code) + executor = MacTool() + exit_code = executor.Dispatch(args) + if exit_code is not None: + sys.exit(exit_code) class MacTool(object): - """This class performs all the Mac tooling steps. The methods can either be + """This class performs all the Mac tooling steps. The methods can either be executed directly, or dispatched from an argument list.""" - def Dispatch(self, args): - """Dispatches a string command to a method.""" - if len(args) < 1: - raise Exception("Not enough arguments") + def Dispatch(self, args): + """Dispatches a string command to a method.""" + if len(args) < 1: + raise Exception("Not enough arguments") - method = "Exec%s" % self._CommandifyName(args[0]) - return getattr(self, method)(*args[1:]) + method = "Exec%s" % self._CommandifyName(args[0]) + return getattr(self, method)(*args[1:]) - def _CommandifyName(self, name_string): - """Transforms a tool name like copy-info-plist to CopyInfoPlist""" - return name_string.title().replace('-', '') + def _CommandifyName(self, name_string): + """Transforms a tool name like copy-info-plist to CopyInfoPlist""" + return name_string.title().replace("-", "") - def ExecCopyBundleResource(self, source, dest, convert_to_binary): - """Copies a resource file to the bundle/Resources directory, performing any + def ExecCopyBundleResource(self, source, dest, convert_to_binary): + """Copies a resource file to the bundle/Resources directory, performing any necessary compilation on each resource.""" - convert_to_binary = convert_to_binary == 'True' - extension = os.path.splitext(source)[1].lower() - if os.path.isdir(source): - # Copy tree. - # TODO(thakis): This copies file attributes like mtime, while the - # single-file branch below doesn't. This should probably be changed to - # be consistent with the single-file branch. - if os.path.exists(dest): - shutil.rmtree(dest) - shutil.copytree(source, dest) - elif extension == '.xib': - return self._CopyXIBFile(source, dest) - elif extension == '.storyboard': - return self._CopyXIBFile(source, dest) - elif extension == '.strings' and not convert_to_binary: - self._CopyStringsFile(source, dest) - else: - if os.path.exists(dest): - os.unlink(dest) - shutil.copy(source, dest) - - if convert_to_binary and extension in ('.plist', '.strings'): - self._ConvertToBinary(dest) - - def _CopyXIBFile(self, source, dest): - """Compiles a XIB file with ibtool into a binary plist in the bundle.""" - - # ibtool sometimes crashes with relative paths. See crbug.com/314728. - base = os.path.dirname(os.path.realpath(__file__)) - if os.path.relpath(source): - source = os.path.join(base, source) - if os.path.relpath(dest): - dest = os.path.join(base, dest) - - args = ['xcrun', 'ibtool', '--errors', '--warnings', '--notices'] - - if os.environ['XCODE_VERSION_ACTUAL'] > '0700': - args.extend(['--auto-activate-custom-fonts']) - if 'IPHONEOS_DEPLOYMENT_TARGET' in os.environ: - args.extend([ - '--target-device', 'iphone', '--target-device', 'ipad', - '--minimum-deployment-target', - os.environ['IPHONEOS_DEPLOYMENT_TARGET'], - ]) - else: - args.extend([ - '--target-device', 'mac', - '--minimum-deployment-target', - os.environ['MACOSX_DEPLOYMENT_TARGET'], - ]) - - args.extend(['--output-format', 'human-readable-text', '--compile', dest, - source]) - - ibtool_section_re = re.compile(r'/\*.*\*/') - ibtool_re = re.compile(r'.*note:.*is clipping its content') - try: - stdout = subprocess.check_output(args) - except subprocess.CalledProcessError as e: - print(e.output) - raise - current_section_header = None - for line in stdout.splitlines(): - if ibtool_section_re.match(line): - current_section_header = line - elif not ibtool_re.match(line): - if current_section_header: - print(current_section_header) - current_section_header = None - print(line) - return 0 - - def _ConvertToBinary(self, dest): - subprocess.check_call([ - 'xcrun', 'plutil', '-convert', 'binary1', '-o', dest, dest]) - - def _CopyStringsFile(self, source, dest): - """Copies a .strings file using iconv to reconvert the input into UTF-16.""" - input_code = self._DetectInputEncoding(source) or "UTF-8" - - # Xcode's CpyCopyStringsFile / builtin-copyStrings seems to call - # CFPropertyListCreateFromXMLData() behind the scenes; at least it prints - # CFPropertyListCreateFromXMLData(): Old-style plist parser: missing - # semicolon in dictionary. - # on invalid files. Do the same kind of validation. - import CoreFoundation - with open(source, 'rb') as in_file: - s = in_file.read() - d = CoreFoundation.CFDataCreate(None, s, len(s)) - _, error = CoreFoundation.CFPropertyListCreateFromXMLData(None, d, 0, None) - if error: - return - - with open(dest, 'wb') as fp: - fp.write(s.decode(input_code).encode('UTF-16')) - - def _DetectInputEncoding(self, file_name): - """Reads the first few bytes from file_name and tries to guess the text + convert_to_binary = convert_to_binary == "True" + extension = os.path.splitext(source)[1].lower() + if os.path.isdir(source): + # Copy tree. + # TODO(thakis): This copies file attributes like mtime, while the + # single-file branch below doesn't. This should probably be changed to + # be consistent with the single-file branch. + if os.path.exists(dest): + shutil.rmtree(dest) + shutil.copytree(source, dest) + elif extension == ".xib": + return self._CopyXIBFile(source, dest) + elif extension == ".storyboard": + return self._CopyXIBFile(source, dest) + elif extension == ".strings" and not convert_to_binary: + self._CopyStringsFile(source, dest) + else: + if os.path.exists(dest): + os.unlink(dest) + shutil.copy(source, dest) + + if convert_to_binary and extension in (".plist", ".strings"): + self._ConvertToBinary(dest) + + def _CopyXIBFile(self, source, dest): + """Compiles a XIB file with ibtool into a binary plist in the bundle.""" + + # ibtool sometimes crashes with relative paths. See crbug.com/314728. + base = os.path.dirname(os.path.realpath(__file__)) + if os.path.relpath(source): + source = os.path.join(base, source) + if os.path.relpath(dest): + dest = os.path.join(base, dest) + + args = ["xcrun", "ibtool", "--errors", "--warnings", "--notices"] + + if os.environ["XCODE_VERSION_ACTUAL"] > "0700": + args.extend(["--auto-activate-custom-fonts"]) + if "IPHONEOS_DEPLOYMENT_TARGET" in os.environ: + args.extend( + [ + "--target-device", + "iphone", + "--target-device", + "ipad", + "--minimum-deployment-target", + os.environ["IPHONEOS_DEPLOYMENT_TARGET"], + ] + ) + else: + args.extend( + [ + "--target-device", + "mac", + "--minimum-deployment-target", + os.environ["MACOSX_DEPLOYMENT_TARGET"], + ] + ) + + args.extend( + ["--output-format", "human-readable-text", "--compile", dest, source] + ) + + ibtool_section_re = re.compile(r"/\*.*\*/") + ibtool_re = re.compile(r".*note:.*is clipping its content") + try: + stdout = subprocess.check_output(args) + except subprocess.CalledProcessError as e: + print(e.output) + raise + current_section_header = None + for line in stdout.splitlines(): + if ibtool_section_re.match(line): + current_section_header = line + elif not ibtool_re.match(line): + if current_section_header: + print(current_section_header) + current_section_header = None + print(line) + return 0 + + def _ConvertToBinary(self, dest): + subprocess.check_call( + ["xcrun", "plutil", "-convert", "binary1", "-o", dest, dest] + ) + + def _CopyStringsFile(self, source, dest): + """Copies a .strings file using iconv to reconvert the input into UTF-16.""" + input_code = self._DetectInputEncoding(source) or "UTF-8" + + # Xcode's CpyCopyStringsFile / builtin-copyStrings seems to call + # CFPropertyListCreateFromXMLData() behind the scenes; at least it prints + # CFPropertyListCreateFromXMLData(): Old-style plist parser: missing + # semicolon in dictionary. + # on invalid files. Do the same kind of validation. + import CoreFoundation + + with open(source, "rb") as in_file: + s = in_file.read() + d = CoreFoundation.CFDataCreate(None, s, len(s)) + _, error = CoreFoundation.CFPropertyListCreateFromXMLData(None, d, 0, None) + if error: + return + + with open(dest, "wb") as fp: + fp.write(s.decode(input_code).encode("UTF-16")) + + def _DetectInputEncoding(self, file_name): + """Reads the first few bytes from file_name and tries to guess the text encoding. Returns None as a guess if it can't detect it.""" - with open(file_name, 'rb') as fp: - try: - header = fp.read(3) - except Exception: - return None - if header.startswith(("\xFE\xFF", "\xFF\xFE")): - return "UTF-16" - elif header.startswith("\xEF\xBB\xBF"): - return "UTF-8" - else: - return None - - def ExecCopyInfoPlist(self, source, dest, convert_to_binary, *keys): - """Copies the |source| Info.plist to the destination directory |dest|.""" - # Read the source Info.plist into memory. - with open(source, 'r') as fd: - lines = fd.read() - - # Insert synthesized key/value pairs (e.g. BuildMachineOSBuild). - plist = plistlib.readPlistFromString(lines) - if keys: - plist = dict(plist.items() + json.loads(keys[0]).items()) - lines = plistlib.writePlistToString(plist) - - # Go through all the environment variables and replace them as variables in - # the file. - IDENT_RE = re.compile(r'[_/\s]') - for key in os.environ: - if key.startswith('_'): - continue - evar = '${%s}' % key - evalue = os.environ[key] - lines = string.replace(lines, evar, evalue) - - # Xcode supports various suffices on environment variables, which are - # all undocumented. :rfc1034identifier is used in the standard project - # template these days, and :identifier was used earlier. They are used to - # convert non-url characters into things that look like valid urls -- - # except that the replacement character for :identifier, '_' isn't valid - # in a URL either -- oops, hence :rfc1034identifier was born. - evar = '${%s:identifier}' % key - evalue = IDENT_RE.sub('_', os.environ[key]) - lines = string.replace(lines, evar, evalue) - - evar = '${%s:rfc1034identifier}' % key - evalue = IDENT_RE.sub('-', os.environ[key]) - lines = string.replace(lines, evar, evalue) - - # Remove any keys with values that haven't been replaced. - lines = lines.splitlines() - for i in range(len(lines)): - if lines[i].strip().startswith("${"): - lines[i] = None - lines[i - 1] = None - lines = '\n'.join(line for line in lines if line is not None) - - # Write out the file with variables replaced. - with open(dest, 'w') as fd: - fd.write(lines) - - # Now write out PkgInfo file now that the Info.plist file has been - # "compiled". - self._WritePkgInfo(dest) - - if convert_to_binary == 'True': - self._ConvertToBinary(dest) - - def _WritePkgInfo(self, info_plist): - """This writes the PkgInfo file from the data stored in Info.plist.""" - plist = plistlib.readPlist(info_plist) - if not plist: - return - - # Only create PkgInfo for executable types. - package_type = plist['CFBundlePackageType'] - if package_type != 'APPL': - return - - # The format of PkgInfo is eight characters, representing the bundle type - # and bundle signature, each four characters. If that is missing, four - # '?' characters are used instead. - signature_code = plist.get('CFBundleSignature', '????') - if len(signature_code) != 4: # Wrong length resets everything, too. - signature_code = '?' * 4 - - dest = os.path.join(os.path.dirname(info_plist), 'PkgInfo') - with open(dest, 'w') as fp: - fp.write('%s%s' % (package_type, signature_code)) - - def ExecFlock(self, lockfile, *cmd_list): - """Emulates the most basic behavior of Linux's flock(1).""" - # Rely on exception handling to report errors. - fd = os.open(lockfile, os.O_RDONLY|os.O_NOCTTY|os.O_CREAT, 0o666) - fcntl.flock(fd, fcntl.LOCK_EX) - return subprocess.call(cmd_list) - - def ExecFilterLibtool(self, *cmd_list): - """Calls libtool and filters out '/path/to/libtool: file: foo.o has no + with open(file_name, "rb") as fp: + try: + header = fp.read(3) + except Exception: + return None + if header.startswith(b"\xFE\xFF"): + return "UTF-16" + elif header.startswith(b"\xFF\xFE"): + return "UTF-16" + elif header.startswith(b"\xEF\xBB\xBF"): + return "UTF-8" + else: + return None + + def ExecCopyInfoPlist(self, source, dest, convert_to_binary, *keys): + """Copies the |source| Info.plist to the destination directory |dest|.""" + # Read the source Info.plist into memory. + with open(source, "r") as fd: + lines = fd.read() + + # Insert synthesized key/value pairs (e.g. BuildMachineOSBuild). + plist = plistlib.readPlistFromString(lines) + if keys: + plist.update(json.loads(keys[0])) + lines = plistlib.writePlistToString(plist) + + # Go through all the environment variables and replace them as variables in + # the file. + IDENT_RE = re.compile(r"[_/\s]") + for key in os.environ: + if key.startswith("_"): + continue + evar = "${%s}" % key + evalue = os.environ[key] + lines = lines.replace(lines, evar, evalue) + + # Xcode supports various suffices on environment variables, which are + # all undocumented. :rfc1034identifier is used in the standard project + # template these days, and :identifier was used earlier. They are used to + # convert non-url characters into things that look like valid urls -- + # except that the replacement character for :identifier, '_' isn't valid + # in a URL either -- oops, hence :rfc1034identifier was born. + evar = "${%s:identifier}" % key + evalue = IDENT_RE.sub("_", os.environ[key]) + lines = lines.replace(lines, evar, evalue) + + evar = "${%s:rfc1034identifier}" % key + evalue = IDENT_RE.sub("-", os.environ[key]) + lines = lines.replace(lines, evar, evalue) + + # Remove any keys with values that haven't been replaced. + lines = lines.splitlines() + for i in range(len(lines)): + if lines[i].strip().startswith("${"): + lines[i] = None + lines[i - 1] = None + lines = "\n".join(line for line in lines if line is not None) + + # Write out the file with variables replaced. + with open(dest, "w") as fd: + fd.write(lines) + + # Now write out PkgInfo file now that the Info.plist file has been + # "compiled". + self._WritePkgInfo(dest) + + if convert_to_binary == "True": + self._ConvertToBinary(dest) + + def _WritePkgInfo(self, info_plist): + """This writes the PkgInfo file from the data stored in Info.plist.""" + plist = plistlib.readPlist(info_plist) + if not plist: + return + + # Only create PkgInfo for executable types. + package_type = plist["CFBundlePackageType"] + if package_type != "APPL": + return + + # The format of PkgInfo is eight characters, representing the bundle type + # and bundle signature, each four characters. If that is missing, four + # '?' characters are used instead. + signature_code = plist.get("CFBundleSignature", "????") + if len(signature_code) != 4: # Wrong length resets everything, too. + signature_code = "?" * 4 + + dest = os.path.join(os.path.dirname(info_plist), "PkgInfo") + with open(dest, "w") as fp: + fp.write("%s%s" % (package_type, signature_code)) + + def ExecFlock(self, lockfile, *cmd_list): + """Emulates the most basic behavior of Linux's flock(1).""" + # Rely on exception handling to report errors. + fd = os.open(lockfile, os.O_RDONLY | os.O_NOCTTY | os.O_CREAT, 0o666) + fcntl.flock(fd, fcntl.LOCK_EX) + return subprocess.call(cmd_list) + + def ExecFilterLibtool(self, *cmd_list): + """Calls libtool and filters out '/path/to/libtool: file: foo.o has no symbols'.""" - libtool_re = re.compile(r'^.*libtool: (?:for architecture: \S* )?' - r'file: .* has no symbols$') - libtool_re5 = re.compile( - r'^.*libtool: warning for library: ' + - r'.* the table of contents is empty ' + - r'\(no object file members in the library define global symbols\)$') - env = os.environ.copy() - # Ref: - # http://www.opensource.apple.com/source/cctools/cctools-809/misc/libtool.c - # The problem with this flag is that it resets the file mtime on the file to - # epoch=0, e.g. 1970-1-1 or 1969-12-31 depending on timezone. - env['ZERO_AR_DATE'] = '1' - libtoolout = subprocess.Popen(cmd_list, stderr=subprocess.PIPE, env=env) - _, err = libtoolout.communicate() - if PY3: - err = err.decode('utf-8') - for line in err.splitlines(): - if not libtool_re.match(line) and not libtool_re5.match(line): - print(line, file=sys.stderr) - # Unconditionally touch the output .a file on the command line if present - # and the command succeeded. A bit hacky. - if not libtoolout.returncode: - for i in range(len(cmd_list) - 1): - if cmd_list[i] == "-o" and cmd_list[i+1].endswith('.a'): - os.utime(cmd_list[i+1], None) - break - return libtoolout.returncode - - def ExecPackageIosFramework(self, framework): - # Find the name of the binary based on the part before the ".framework". - binary = os.path.basename(framework).split('.')[0] - module_path = os.path.join(framework, 'Modules') - if not os.path.exists(module_path): - os.mkdir(module_path) - module_template = 'framework module %s {\n' \ - ' umbrella header "%s.h"\n' \ - '\n' \ - ' export *\n' \ - ' module * { export * }\n' \ - '}\n' % (binary, binary) - - with open(os.path.join(module_path, 'module.modulemap'), "w") as module_file: - module_file.write(module_template) - - def ExecPackageFramework(self, framework, version): - """Takes a path to Something.framework and the Current version of that and + libtool_re = re.compile( + r"^.*libtool: (?:for architecture: \S* )?" r"file: .* has no symbols$" + ) + libtool_re5 = re.compile( + r"^.*libtool: warning for library: " + + r".* the table of contents is empty " + + r"\(no object file members in the library define global symbols\)$" + ) + env = os.environ.copy() + # Ref: + # http://www.opensource.apple.com/source/cctools/cctools-809/misc/libtool.c + # The problem with this flag is that it resets the file mtime on the file to + # epoch=0, e.g. 1970-1-1 or 1969-12-31 depending on timezone. + env["ZERO_AR_DATE"] = "1" + libtoolout = subprocess.Popen(cmd_list, stderr=subprocess.PIPE, env=env) + _, err = libtoolout.communicate() + if PY3: + err = err.decode("utf-8") + for line in err.splitlines(): + if not libtool_re.match(line) and not libtool_re5.match(line): + print(line, file=sys.stderr) + # Unconditionally touch the output .a file on the command line if present + # and the command succeeded. A bit hacky. + if not libtoolout.returncode: + for i in range(len(cmd_list) - 1): + if cmd_list[i] == "-o" and cmd_list[i + 1].endswith(".a"): + os.utime(cmd_list[i + 1], None) + break + return libtoolout.returncode + + def ExecPackageIosFramework(self, framework): + # Find the name of the binary based on the part before the ".framework". + binary = os.path.basename(framework).split(".")[0] + module_path = os.path.join(framework, "Modules") + if not os.path.exists(module_path): + os.mkdir(module_path) + module_template = ( + "framework module %s {\n" + ' umbrella header "%s.h"\n' + "\n" + " export *\n" + " module * { export * }\n" + "}\n" % (binary, binary) + ) + + with open(os.path.join(module_path, "module.modulemap"), "w") as module_file: + module_file.write(module_template) + + def ExecPackageFramework(self, framework, version): + """Takes a path to Something.framework and the Current version of that and sets up all the symlinks.""" - # Find the name of the binary based on the part before the ".framework". - binary = os.path.basename(framework).split('.')[0] + # Find the name of the binary based on the part before the ".framework". + binary = os.path.basename(framework).split(".")[0] - CURRENT = 'Current' - RESOURCES = 'Resources' - VERSIONS = 'Versions' + CURRENT = "Current" + RESOURCES = "Resources" + VERSIONS = "Versions" - if not os.path.exists(os.path.join(framework, VERSIONS, version, binary)): - # Binary-less frameworks don't seem to contain symlinks (see e.g. - # chromium's out/Debug/org.chromium.Chromium.manifest/ bundle). - return + if not os.path.exists(os.path.join(framework, VERSIONS, version, binary)): + # Binary-less frameworks don't seem to contain symlinks (see e.g. + # chromium's out/Debug/org.chromium.Chromium.manifest/ bundle). + return - # Move into the framework directory to set the symlinks correctly. - pwd = os.getcwd() - os.chdir(framework) + # Move into the framework directory to set the symlinks correctly. + pwd = os.getcwd() + os.chdir(framework) - # Set up the Current version. - self._Relink(version, os.path.join(VERSIONS, CURRENT)) + # Set up the Current version. + self._Relink(version, os.path.join(VERSIONS, CURRENT)) - # Set up the root symlinks. - self._Relink(os.path.join(VERSIONS, CURRENT, binary), binary) - self._Relink(os.path.join(VERSIONS, CURRENT, RESOURCES), RESOURCES) + # Set up the root symlinks. + self._Relink(os.path.join(VERSIONS, CURRENT, binary), binary) + self._Relink(os.path.join(VERSIONS, CURRENT, RESOURCES), RESOURCES) - # Back to where we were before! - os.chdir(pwd) + # Back to where we were before! + os.chdir(pwd) - def _Relink(self, dest, link): - """Creates a symlink to |dest| named |link|. If |link| already exists, + def _Relink(self, dest, link): + """Creates a symlink to |dest| named |link|. If |link| already exists, it is overwritten.""" - if os.path.lexists(link): - os.remove(link) - os.symlink(dest, link) - - def ExecCompileIosFrameworkHeaderMap(self, out, framework, *all_headers): - framework_name = os.path.basename(framework).split('.')[0] - all_headers = [os.path.abspath(header) for header in all_headers] - filelist = {} - for header in all_headers: - filename = os.path.basename(header) - filelist[filename] = header - filelist[os.path.join(framework_name, filename)] = header - WriteHmap(out, filelist) - - def ExecCopyIosFrameworkHeaders(self, framework, *copy_headers): - header_path = os.path.join(framework, 'Headers') - if not os.path.exists(header_path): - os.makedirs(header_path) - for header in copy_headers: - shutil.copy(header, os.path.join(header_path, os.path.basename(header))) - - def ExecCompileXcassets(self, keys, *inputs): - """Compiles multiple .xcassets files into a single .car file. + if os.path.lexists(link): + os.remove(link) + os.symlink(dest, link) + + def ExecCompileIosFrameworkHeaderMap(self, out, framework, *all_headers): + framework_name = os.path.basename(framework).split(".")[0] + all_headers = [os.path.abspath(header) for header in all_headers] + filelist = {} + for header in all_headers: + filename = os.path.basename(header) + filelist[filename] = header + filelist[os.path.join(framework_name, filename)] = header + WriteHmap(out, filelist) + + def ExecCopyIosFrameworkHeaders(self, framework, *copy_headers): + header_path = os.path.join(framework, "Headers") + if not os.path.exists(header_path): + os.makedirs(header_path) + for header in copy_headers: + shutil.copy(header, os.path.join(header_path, os.path.basename(header))) + + def ExecCompileXcassets(self, keys, *inputs): + """Compiles multiple .xcassets files into a single .car file. This invokes 'actool' to compile all the inputs .xcassets files. The |keys| arguments is a json-encoded dictionary of extra arguments to @@ -359,57 +375,77 @@ def ExecCompileXcassets(self, keys, *inputs): Note that 'actool' does not create the Assets.car file if the asset catalogs does not contains imageset. """ - command_line = [ - 'xcrun', 'actool', '--output-format', 'human-readable-text', - '--compress-pngs', '--notices', '--warnings', '--errors', - ] - is_iphone_target = 'IPHONEOS_DEPLOYMENT_TARGET' in os.environ - if is_iphone_target: - platform = os.environ['CONFIGURATION'].split('-')[-1] - if platform not in ('iphoneos', 'iphonesimulator'): - platform = 'iphonesimulator' - command_line.extend([ - '--platform', platform, '--target-device', 'iphone', - '--target-device', 'ipad', '--minimum-deployment-target', - os.environ['IPHONEOS_DEPLOYMENT_TARGET'], '--compile', - os.path.abspath(os.environ['CONTENTS_FOLDER_PATH']), - ]) - else: - command_line.extend([ - '--platform', 'macosx', '--target-device', 'mac', - '--minimum-deployment-target', os.environ['MACOSX_DEPLOYMENT_TARGET'], - '--compile', - os.path.abspath(os.environ['UNLOCALIZED_RESOURCES_FOLDER_PATH']), - ]) - if keys: - keys = json.loads(keys) - for key, value in keys.items(): - arg_name = '--' + key - if isinstance(value, bool): - if value: - command_line.append(arg_name) - elif isinstance(value, list): - for v in value: - command_line.append(arg_name) - command_line.append(str(v)) + command_line = [ + "xcrun", + "actool", + "--output-format", + "human-readable-text", + "--compress-pngs", + "--notices", + "--warnings", + "--errors", + ] + is_iphone_target = "IPHONEOS_DEPLOYMENT_TARGET" in os.environ + if is_iphone_target: + platform = os.environ["CONFIGURATION"].split("-")[-1] + if platform not in ("iphoneos", "iphonesimulator"): + platform = "iphonesimulator" + command_line.extend( + [ + "--platform", + platform, + "--target-device", + "iphone", + "--target-device", + "ipad", + "--minimum-deployment-target", + os.environ["IPHONEOS_DEPLOYMENT_TARGET"], + "--compile", + os.path.abspath(os.environ["CONTENTS_FOLDER_PATH"]), + ] + ) else: - command_line.append(arg_name) - command_line.append(str(value)) - # Note: actool crashes if inputs path are relative, so use os.path.abspath - # to get absolute path name for inputs. - command_line.extend(map(os.path.abspath, inputs)) - subprocess.check_call(command_line) - - def ExecMergeInfoPlist(self, output, *inputs): - """Merge multiple .plist files into a single .plist file.""" - merged_plist = {} - for path in inputs: - plist = self._LoadPlistMaybeBinary(path) - self._MergePlist(merged_plist, plist) - plistlib.writePlist(merged_plist, output) - - def ExecCodeSignBundle(self, key, entitlements, provisioning, path, preserve): - """Code sign a bundle. + command_line.extend( + [ + "--platform", + "macosx", + "--target-device", + "mac", + "--minimum-deployment-target", + os.environ["MACOSX_DEPLOYMENT_TARGET"], + "--compile", + os.path.abspath(os.environ["UNLOCALIZED_RESOURCES_FOLDER_PATH"]), + ] + ) + if keys: + keys = json.loads(keys) + for key, value in keys.items(): + arg_name = "--" + key + if isinstance(value, bool): + if value: + command_line.append(arg_name) + elif isinstance(value, list): + for v in value: + command_line.append(arg_name) + command_line.append(str(v)) + else: + command_line.append(arg_name) + command_line.append(str(value)) + # Note: actool crashes if inputs path are relative, so use os.path.abspath + # to get absolute path name for inputs. + command_line.extend(map(os.path.abspath, inputs)) + subprocess.check_call(command_line) + + def ExecMergeInfoPlist(self, output, *inputs): + """Merge multiple .plist files into a single .plist file.""" + merged_plist = {} + for path in inputs: + plist = self._LoadPlistMaybeBinary(path) + self._MergePlist(merged_plist, plist) + plistlib.writePlist(merged_plist, output) + + def ExecCodeSignBundle(self, key, entitlements, provisioning, path, preserve): + """Code sign a bundle. This function tries to code sign an iOS bundle, following the same algorithm as Xcode: @@ -418,21 +454,23 @@ def ExecCodeSignBundle(self, key, entitlements, provisioning, path, preserve): 2. copy Entitlements.plist from user or SDK next to the bundle, 3. code sign the bundle. """ - substitutions, overrides = self._InstallProvisioningProfile( - provisioning, self._GetCFBundleIdentifier()) - entitlements_path = self._InstallEntitlements( - entitlements, substitutions, overrides) - - args = ['codesign', '--force', '--sign', key] - if preserve == 'True': - args.extend(['--deep', '--preserve-metadata=identifier,entitlements']) - else: - args.extend(['--entitlements', entitlements_path]) - args.extend(['--timestamp=none', path]) - subprocess.check_call(args) - - def _InstallProvisioningProfile(self, profile, bundle_identifier): - """Installs embedded.mobileprovision into the bundle. + substitutions, overrides = self._InstallProvisioningProfile( + provisioning, self._GetCFBundleIdentifier() + ) + entitlements_path = self._InstallEntitlements( + entitlements, substitutions, overrides + ) + + args = ["codesign", "--force", "--sign", key] + if preserve == "True": + args.extend(["--deep", "--preserve-metadata=identifier,entitlements"]) + else: + args.extend(["--entitlements", entitlements_path]) + args.extend(["--timestamp=none", path]) + subprocess.check_call(args) + + def _InstallProvisioningProfile(self, profile, bundle_identifier): + """Installs embedded.mobileprovision into the bundle. Args: profile: string, optional, short name of the .mobileprovision file @@ -444,18 +482,20 @@ def _InstallProvisioningProfile(self, profile, bundle_identifier): A tuple containing two dictionary: variables substitutions and values to overrides when generating the entitlements file. """ - source_path, provisioning_data, team_id = self._FindProvisioningProfile( - profile, bundle_identifier) - target_path = os.path.join( - os.environ['BUILT_PRODUCTS_DIR'], - os.environ['CONTENTS_FOLDER_PATH'], - 'embedded.mobileprovision') - shutil.copy2(source_path, target_path) - substitutions = self._GetSubstitutions(bundle_identifier, team_id + '.') - return substitutions, provisioning_data['Entitlements'] - - def _FindProvisioningProfile(self, profile, bundle_identifier): - """Finds the .mobileprovision file to use for signing the bundle. + source_path, provisioning_data, team_id = self._FindProvisioningProfile( + profile, bundle_identifier + ) + target_path = os.path.join( + os.environ["BUILT_PRODUCTS_DIR"], + os.environ["CONTENTS_FOLDER_PATH"], + "embedded.mobileprovision", + ) + shutil.copy2(source_path, target_path) + substitutions = self._GetSubstitutions(bundle_identifier, team_id + ".") + return substitutions, provisioning_data["Entitlements"] + + def _FindProvisioningProfile(self, profile, bundle_identifier): + """Finds the .mobileprovision file to use for signing the bundle. Checks all the installed provisioning profiles (or if the user specified the PROVISIONING_PROFILE variable, only consult it) and select the most @@ -475,40 +515,52 @@ def _FindProvisioningProfile(self, profile, bundle_identifier): Raises: SystemExit: if no .mobileprovision can be used to sign the bundle. """ - profiles_dir = os.path.join( - os.environ['HOME'], 'Library', 'MobileDevice', 'Provisioning Profiles') - if not os.path.isdir(profiles_dir): - print('cannot find mobile provisioning for %s' % (bundle_identifier), file=sys.stderr) - sys.exit(1) - provisioning_profiles = None - if profile: - profile_path = os.path.join(profiles_dir, profile + '.mobileprovision') - if os.path.exists(profile_path): - provisioning_profiles = [profile_path] - if not provisioning_profiles: - provisioning_profiles = glob.glob( - os.path.join(profiles_dir, '*.mobileprovision')) - valid_provisioning_profiles = {} - for profile_path in provisioning_profiles: - profile_data = self._LoadProvisioningProfile(profile_path) - app_id_pattern = profile_data.get( - 'Entitlements', {}).get('application-identifier', '') - for team_identifier in profile_data.get('TeamIdentifier', []): - app_id = '%s.%s' % (team_identifier, bundle_identifier) - if fnmatch.fnmatch(app_id, app_id_pattern): - valid_provisioning_profiles[app_id_pattern] = ( - profile_path, profile_data, team_identifier) - if not valid_provisioning_profiles: - print('cannot find mobile provisioning for %s' % (bundle_identifier), file=sys.stderr) - sys.exit(1) - # If the user has multiple provisioning profiles installed that can be - # used for ${bundle_identifier}, pick the most specific one (ie. the - # provisioning profile whose pattern is the longest). - selected_key = max(valid_provisioning_profiles, key=lambda v: len(v)) - return valid_provisioning_profiles[selected_key] - - def _LoadProvisioningProfile(self, profile_path): - """Extracts the plist embedded in a provisioning profile. + profiles_dir = os.path.join( + os.environ["HOME"], "Library", "MobileDevice", "Provisioning Profiles" + ) + if not os.path.isdir(profiles_dir): + print( + "cannot find mobile provisioning for %s" % (bundle_identifier), + file=sys.stderr, + ) + sys.exit(1) + provisioning_profiles = None + if profile: + profile_path = os.path.join(profiles_dir, profile + ".mobileprovision") + if os.path.exists(profile_path): + provisioning_profiles = [profile_path] + if not provisioning_profiles: + provisioning_profiles = glob.glob( + os.path.join(profiles_dir, "*.mobileprovision") + ) + valid_provisioning_profiles = {} + for profile_path in provisioning_profiles: + profile_data = self._LoadProvisioningProfile(profile_path) + app_id_pattern = profile_data.get("Entitlements", {}).get( + "application-identifier", "" + ) + for team_identifier in profile_data.get("TeamIdentifier", []): + app_id = "%s.%s" % (team_identifier, bundle_identifier) + if fnmatch.fnmatch(app_id, app_id_pattern): + valid_provisioning_profiles[app_id_pattern] = ( + profile_path, + profile_data, + team_identifier, + ) + if not valid_provisioning_profiles: + print( + "cannot find mobile provisioning for %s" % (bundle_identifier), + file=sys.stderr, + ) + sys.exit(1) + # If the user has multiple provisioning profiles installed that can be + # used for ${bundle_identifier}, pick the most specific one (ie. the + # provisioning profile whose pattern is the longest). + selected_key = max(valid_provisioning_profiles, key=lambda v: len(v)) + return valid_provisioning_profiles[selected_key] + + def _LoadProvisioningProfile(self, profile_path): + """Extracts the plist embedded in a provisioning profile. Args: profile_path: string, path to the .mobileprovision file @@ -516,26 +568,27 @@ def _LoadProvisioningProfile(self, profile_path): Returns: Content of the plist embedded in the provisioning profile as a dictionary. """ - with tempfile.NamedTemporaryFile() as temp: - subprocess.check_call([ - 'security', 'cms', '-D', '-i', profile_path, '-o', temp.name]) - return self._LoadPlistMaybeBinary(temp.name) - - def _MergePlist(self, merged_plist, plist): - """Merge |plist| into |merged_plist|.""" - for key, value in plist.items(): - if isinstance(value, dict): - merged_value = merged_plist.get(key, {}) - if isinstance(merged_value, dict): - self._MergePlist(merged_value, value) - merged_plist[key] = merged_value - else: - merged_plist[key] = value - else: - merged_plist[key] = value - - def _LoadPlistMaybeBinary(self, plist_path): - """Loads into a memory a plist possibly encoded in binary format. + with tempfile.NamedTemporaryFile() as temp: + subprocess.check_call( + ["security", "cms", "-D", "-i", profile_path, "-o", temp.name] + ) + return self._LoadPlistMaybeBinary(temp.name) + + def _MergePlist(self, merged_plist, plist): + """Merge |plist| into |merged_plist|.""" + for key, value in plist.items(): + if isinstance(value, dict): + merged_value = merged_plist.get(key, {}) + if isinstance(merged_value, dict): + self._MergePlist(merged_value, value) + merged_plist[key] = merged_value + else: + merged_plist[key] = value + else: + merged_plist[key] = value + + def _LoadPlistMaybeBinary(self, plist_path): + """Loads into a memory a plist possibly encoded in binary format. This is a wrapper around plistlib.readPlist that tries to convert the plist to the XML format if it can't be parsed (assuming that it is in @@ -547,20 +600,20 @@ def _LoadPlistMaybeBinary(self, plist_path): Returns: Content of the plist as a dictionary. """ - try: - # First, try to read the file using plistlib that only supports XML, - # and if an exception is raised, convert a temporary copy to XML and - # load that copy. - return plistlib.readPlist(plist_path) - except: - pass - with tempfile.NamedTemporaryFile() as temp: - shutil.copy2(plist_path, temp.name) - subprocess.check_call(['plutil', '-convert', 'xml1', temp.name]) - return plistlib.readPlist(temp.name) - - def _GetSubstitutions(self, bundle_identifier, app_identifier_prefix): - """Constructs a dictionary of variable substitutions for Entitlements.plist. + try: + # First, try to read the file using plistlib that only supports XML, + # and if an exception is raised, convert a temporary copy to XML and + # load that copy. + return plistlib.readPlist(plist_path) + except Exception: + pass + with tempfile.NamedTemporaryFile() as temp: + shutil.copy2(plist_path, temp.name) + subprocess.check_call(["plutil", "-convert", "xml1", temp.name]) + return plistlib.readPlist(temp.name) + + def _GetSubstitutions(self, bundle_identifier, app_identifier_prefix): + """Constructs a dictionary of variable substitutions for Entitlements.plist. Args: bundle_identifier: string, value of CFBundleIdentifier from Info.plist @@ -569,25 +622,25 @@ def _GetSubstitutions(self, bundle_identifier, app_identifier_prefix): Returns: Dictionary of substitutions to apply when generating Entitlements.plist. """ - return { - 'CFBundleIdentifier': bundle_identifier, - 'AppIdentifierPrefix': app_identifier_prefix, - } + return { + "CFBundleIdentifier": bundle_identifier, + "AppIdentifierPrefix": app_identifier_prefix, + } - def _GetCFBundleIdentifier(self): - """Extracts CFBundleIdentifier value from Info.plist in the bundle. + def _GetCFBundleIdentifier(self): + """Extracts CFBundleIdentifier value from Info.plist in the bundle. Returns: Value of CFBundleIdentifier in the Info.plist located in the bundle. """ - info_plist_path = os.path.join( - os.environ['TARGET_BUILD_DIR'], - os.environ['INFOPLIST_PATH']) - info_plist_data = self._LoadPlistMaybeBinary(info_plist_path) - return info_plist_data['CFBundleIdentifier'] + info_plist_path = os.path.join( + os.environ["TARGET_BUILD_DIR"], os.environ["INFOPLIST_PATH"] + ) + info_plist_data = self._LoadPlistMaybeBinary(info_plist_path) + return info_plist_data["CFBundleIdentifier"] - def _InstallEntitlements(self, entitlements, substitutions, overrides): - """Generates and install the ${BundleName}.xcent entitlements file. + def _InstallEntitlements(self, entitlements, substitutions, overrides): + """Generates and install the ${BundleName}.xcent entitlements file. Expands variables "$(variable)" pattern in the source entitlements file, add extra entitlements defined in the .mobileprovision file and the copy @@ -602,26 +655,24 @@ def _InstallEntitlements(self, entitlements, substitutions, overrides): Returns: Path to the generated entitlements file. """ - source_path = entitlements - target_path = os.path.join( - os.environ['BUILT_PRODUCTS_DIR'], - os.environ['PRODUCT_NAME'] + '.xcent') - if not source_path: - source_path = os.path.join( - os.environ['SDKROOT'], - 'Entitlements.plist') - shutil.copy2(source_path, target_path) - data = self._LoadPlistMaybeBinary(target_path) - data = self._ExpandVariables(data, substitutions) - if overrides: - for key in overrides: - if key not in data: - data[key] = overrides[key] - plistlib.writePlist(data, target_path) - return target_path - - def _ExpandVariables(self, data, substitutions): - """Expands variables "$(variable)" in data. + source_path = entitlements + target_path = os.path.join( + os.environ["BUILT_PRODUCTS_DIR"], os.environ["PRODUCT_NAME"] + ".xcent" + ) + if not source_path: + source_path = os.path.join(os.environ["SDKROOT"], "Entitlements.plist") + shutil.copy2(source_path, target_path) + data = self._LoadPlistMaybeBinary(target_path) + data = self._ExpandVariables(data, substitutions) + if overrides: + for key in overrides: + if key not in data: + data[key] = overrides[key] + plistlib.writePlist(data, target_path) + return target_path + + def _ExpandVariables(self, data, substitutions): + """Expands variables "$(variable)" in data. Args: data: object, can be either string, list or dictionary @@ -632,21 +683,23 @@ def _ExpandVariables(self, data, substitutions): by the corresponding value found in substitutions, or left intact if the key was not found. """ - if isinstance(data, str): - for key, value in substitutions.items(): - data = data.replace('$(%s)' % key, value) - return data - if isinstance(data, list): - return [self._ExpandVariables(v, substitutions) for v in data] - if isinstance(data, dict): - return {k: self._ExpandVariables(data[k], substitutions) for k in data} - return data + if isinstance(data, str): + for key, value in substitutions.items(): + data = data.replace("$(%s)" % key, value) + return data + if isinstance(data, list): + return [self._ExpandVariables(v, substitutions) for v in data] + if isinstance(data, dict): + return {k: self._ExpandVariables(data[k], substitutions) for k in data} + return data + def NextGreaterPowerOf2(x): - return 2**(x).bit_length() + return 2 ** (x).bit_length() + def WriteHmap(output_name, filelist): - """Generates a header map based on |filelist|. + """Generates a header map based on |filelist|. Per Mark Mentovai: A header map is structured essentially as a hash table, keyed by names used @@ -657,56 +710,67 @@ def WriteHmap(output_name, filelist): while also looking at the implementation in clang in: https://llvm.org/svn/llvm-project/cfe/trunk/lib/Lex/HeaderMap.cpp """ - magic = 1751998832 - version = 1 - _reserved = 0 - count = len(filelist) - capacity = NextGreaterPowerOf2(count) - strings_offset = 24 + (12 * capacity) - max_value_length = max(len(value) for value in filelist.values()) - - out = open(output_name, "wb") - out.write(struct.pack(' 0 or arg.count('/') > 1: - arg = os.path.normpath(arg) + # Use a heuristic to try to find args that are paths, and normalize them + if arg.find("/") > 0 or arg.count("/") > 1: + arg = os.path.normpath(arg) - # For a literal quote, CommandLineToArgvW requires 2n+1 backslashes - # preceding it, and results in n backslashes + the quote. So we substitute - # in 2* what we match, +1 more, plus the quote. - arg = windows_quoter_regex.sub(lambda mo: 2 * mo.group(1) + '\\"', arg) + # For a literal quote, CommandLineToArgvW requires 2n+1 backslashes + # preceding it, and results in n backslashes + the quote. So we substitute + # in 2* what we match, +1 more, plus the quote. + arg = windows_quoter_regex.sub(lambda mo: 2 * mo.group(1) + '\\"', arg) - # %'s also need to be doubled otherwise they're interpreted as batch - # positional arguments. Also make sure to escape the % so that they're - # passed literally through escaping so they can be singled to just the - # original %. Otherwise, trying to pass the literal representation that - # looks like an environment variable to the shell (e.g. %PATH%) would fail. - arg = arg.replace('%', '%%') + # %'s also need to be doubled otherwise they're interpreted as batch + # positional arguments. Also make sure to escape the % so that they're + # passed literally through escaping so they can be singled to just the + # original %. Otherwise, trying to pass the literal representation that + # looks like an environment variable to the shell (e.g. %PATH%) would fail. + arg = arg.replace("%", "%%") - # These commands are used in rsp files, so no escaping for the shell (via ^) - # is necessary. + # These commands are used in rsp files, so no escaping for the shell (via ^) + # is necessary. - # Finally, wrap the whole thing in quotes so that the above quote rule - # applies and whitespace isn't a word break. - return '"' + arg + '"' + # Finally, wrap the whole thing in quotes so that the above quote rule + # applies and whitespace isn't a word break. + return '"' + arg + '"' def EncodeRspFileList(args): - """Process a list of arguments using QuoteCmdExeArgument.""" - # Note that the first argument is assumed to be the command. Don't add - # quotes around it because then built-ins like 'echo', etc. won't work. - # Take care to normpath only the path in the case of 'call ../x.bat' because - # otherwise the whole thing is incorrectly interpreted as a path and not - # normalized correctly. - if not args: return '' - if args[0].startswith('call '): - call, program = args[0].split(' ', 1) - program = call + ' ' + os.path.normpath(program) - else: - program = os.path.normpath(args[0]) - return program + ' ' + ' '.join(QuoteForRspFile(arg) for arg in args[1:]) + """Process a list of arguments using QuoteCmdExeArgument.""" + # Note that the first argument is assumed to be the command. Don't add + # quotes around it because then built-ins like 'echo', etc. won't work. + # Take care to normpath only the path in the case of 'call ../x.bat' because + # otherwise the whole thing is incorrectly interpreted as a path and not + # normalized correctly. + if not args: + return "" + if args[0].startswith("call "): + call, program = args[0].split(" ", 1) + program = call + " " + os.path.normpath(program) + else: + program = os.path.normpath(args[0]) + return program + " " + " ".join(QuoteForRspFile(arg) for arg in args[1:]) def _GenericRetrieve(root, default, path): - """Given a list of dictionary keys |path| and a tree of dicts |root|, find + """Given a list of dictionary keys |path| and a tree of dicts |root|, find value at path, or return |default| if any of the path doesn't exist.""" - if not root: - return default - if not path: - return root - return _GenericRetrieve(root.get(path[0]), default, path[1:]) + if not root: + return default + if not path: + return root + return _GenericRetrieve(root.get(path[0]), default, path[1:]) def _AddPrefix(element, prefix): - """Add |prefix| to |element| or each subelement if element is iterable.""" - if element is None: - return element - # Note, not Iterable because we don't want to handle strings like that. - if isinstance(element, list) or isinstance(element, tuple): - return [prefix + e for e in element] - else: - return prefix + element + """Add |prefix| to |element| or each subelement if element is iterable.""" + if element is None: + return element + # Note, not Iterable because we don't want to handle strings like that. + if isinstance(element, list) or isinstance(element, tuple): + return [prefix + e for e in element] + else: + return prefix + element def _DoRemapping(element, map): - """If |element| then remap it through |map|. If |element| is iterable then + """If |element| then remap it through |map|. If |element| is iterable then each item will be remapped. Any elements not found will be removed.""" - if map is not None and element is not None: - if not callable(map): - map = map.get # Assume it's a dict, otherwise a callable to do the remap. - if isinstance(element, list) or isinstance(element, tuple): - element = filter(None, [map(elem) for elem in element]) - else: - element = map(element) - return element + if map is not None and element is not None: + if not callable(map): + map = map.get # Assume it's a dict, otherwise a callable to do the remap. + if isinstance(element, list) or isinstance(element, tuple): + element = filter(None, [map(elem) for elem in element]) + else: + element = map(element) + return element def _AppendOrReturn(append, element): - """If |append| is None, simply return |element|. If |append| is not None, + """If |append| is None, simply return |element|. If |append| is not None, then add |element| to it, adding each item in |element| if it's a list or tuple.""" - if append is not None and element is not None: - if isinstance(element, list) or isinstance(element, tuple): - append.extend(element) + if append is not None and element is not None: + if isinstance(element, list) or isinstance(element, tuple): + append.extend(element) + else: + append.append(element) else: - append.append(element) - else: - return element + return element def _FindDirectXInstallation(): - """Try to find an installation location for the DirectX SDK. Check for the + """Try to find an installation location for the DirectX SDK. Check for the standard environment variable, and if that doesn't exist, try to find via the registry. May return None if not found in either location.""" - # Return previously calculated value, if there is one - if hasattr(_FindDirectXInstallation, 'dxsdk_dir'): - return _FindDirectXInstallation.dxsdk_dir - - dxsdk_dir = os.environ.get('DXSDK_DIR') - if not dxsdk_dir: - # Setup params to pass to and attempt to launch reg.exe. - cmd = ['reg.exe', 'query', r'HKLM\Software\Microsoft\DirectX', '/s'] - p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) - stdout = p.communicate()[0] - if PY3: - stdout = stdout.decode('utf-8') - for line in stdout.splitlines(): - if 'InstallPath' in line: - dxsdk_dir = line.split(' ')[3] + "\\" - - # Cache return value - _FindDirectXInstallation.dxsdk_dir = dxsdk_dir - return dxsdk_dir + # Return previously calculated value, if there is one + if hasattr(_FindDirectXInstallation, "dxsdk_dir"): + return _FindDirectXInstallation.dxsdk_dir + + dxsdk_dir = os.environ.get("DXSDK_DIR") + if not dxsdk_dir: + # Setup params to pass to and attempt to launch reg.exe. + cmd = ["reg.exe", "query", r"HKLM\Software\Microsoft\DirectX", "/s"] + p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + stdout = p.communicate()[0] + if PY3: + stdout = stdout.decode("utf-8") + for line in stdout.splitlines(): + if "InstallPath" in line: + dxsdk_dir = line.split(" ")[3] + "\\" + + # Cache return value + _FindDirectXInstallation.dxsdk_dir = dxsdk_dir + return dxsdk_dir def GetGlobalVSMacroEnv(vs_version): - """Get a dict of variables mapping internal VS macro names to their gyp + """Get a dict of variables mapping internal VS macro names to their gyp equivalents. Returns all variables that are independent of the target.""" - env = {} - # '$(VSInstallDir)' and '$(VCInstallDir)' are available when and only when - # Visual Studio is actually installed. - if vs_version.Path(): - env['$(VSInstallDir)'] = vs_version.Path() - env['$(VCInstallDir)'] = os.path.join(vs_version.Path(), 'VC') + '\\' - # Chromium uses DXSDK_DIR in include/lib paths, but it may or may not be - # set. This happens when the SDK is sync'd via src-internal, rather than - # by typical end-user installation of the SDK. If it's not set, we don't - # want to leave the unexpanded variable in the path, so simply strip it. - dxsdk_dir = _FindDirectXInstallation() - env['$(DXSDK_DIR)'] = dxsdk_dir if dxsdk_dir else '' - # Try to find an installation location for the Windows DDK by checking - # the WDK_DIR environment variable, may be None. - env['$(WDK_DIR)'] = os.environ.get('WDK_DIR', '') - return env + env = {} + # '$(VSInstallDir)' and '$(VCInstallDir)' are available when and only when + # Visual Studio is actually installed. + if vs_version.Path(): + env["$(VSInstallDir)"] = vs_version.Path() + env["$(VCInstallDir)"] = os.path.join(vs_version.Path(), "VC") + "\\" + # Chromium uses DXSDK_DIR in include/lib paths, but it may or may not be + # set. This happens when the SDK is sync'd via src-internal, rather than + # by typical end-user installation of the SDK. If it's not set, we don't + # want to leave the unexpanded variable in the path, so simply strip it. + dxsdk_dir = _FindDirectXInstallation() + env["$(DXSDK_DIR)"] = dxsdk_dir if dxsdk_dir else "" + # Try to find an installation location for the Windows DDK by checking + # the WDK_DIR environment variable, may be None. + env["$(WDK_DIR)"] = os.environ.get("WDK_DIR", "") + return env + def ExtractSharedMSVSSystemIncludes(configs, generator_flags): - """Finds msvs_system_include_dirs that are common to all targets, removes + """Finds msvs_system_include_dirs that are common to all targets, removes them from all targets, and returns an OrderedSet containing them.""" - all_system_includes = OrderedSet( - configs[0].get('msvs_system_include_dirs', [])) - for config in configs[1:]: - system_includes = config.get('msvs_system_include_dirs', []) - all_system_includes = all_system_includes & OrderedSet(system_includes) - if not all_system_includes: - return None - # Expand macros in all_system_includes. - env = GetGlobalVSMacroEnv(GetVSVersion(generator_flags)) - expanded_system_includes = OrderedSet([ExpandMacros(include, env) - for include in all_system_includes]) - if any(['$' in include for include in expanded_system_includes]): - # Some path relies on target-specific variables, bail. - return None - - # Remove system includes shared by all targets from the targets. - for config in configs: - includes = config.get('msvs_system_include_dirs', []) - if includes: # Don't insert a msvs_system_include_dirs key if not needed. - # This must check the unexpanded includes list: - new_includes = [i for i in includes if i not in all_system_includes] - config['msvs_system_include_dirs'] = new_includes - return expanded_system_includes + all_system_includes = OrderedSet(configs[0].get("msvs_system_include_dirs", [])) + for config in configs[1:]: + system_includes = config.get("msvs_system_include_dirs", []) + all_system_includes = all_system_includes & OrderedSet(system_includes) + if not all_system_includes: + return None + # Expand macros in all_system_includes. + env = GetGlobalVSMacroEnv(GetVSVersion(generator_flags)) + expanded_system_includes = OrderedSet( + [ExpandMacros(include, env) for include in all_system_includes] + ) + if any(["$" in include for include in expanded_system_includes]): + # Some path relies on target-specific variables, bail. + return None + + # Remove system includes shared by all targets from the targets. + for config in configs: + includes = config.get("msvs_system_include_dirs", []) + if includes: # Don't insert a msvs_system_include_dirs key if not needed. + # This must check the unexpanded includes list: + new_includes = [i for i in includes if i not in all_system_includes] + config["msvs_system_include_dirs"] = new_includes + return expanded_system_includes class MsvsSettings(object): - """A class that understands the gyp 'msvs_...' values (especially the + """A class that understands the gyp 'msvs_...' values (especially the msvs_settings field). They largely correpond to the VS2008 IDE DOM. This class helps map those settings to command line options.""" - def __init__(self, spec, generator_flags): - self.spec = spec - self.vs_version = GetVSVersion(generator_flags) - - supported_fields = [ - ('msvs_configuration_attributes', dict), - ('msvs_settings', dict), - ('msvs_system_include_dirs', list), - ('msvs_disabled_warnings', list), - ('msvs_precompiled_header', str), - ('msvs_precompiled_source', str), - ('msvs_configuration_platform', str), - ('msvs_target_platform', str), + def __init__(self, spec, generator_flags): + self.spec = spec + self.vs_version = GetVSVersion(generator_flags) + + supported_fields = [ + ("msvs_configuration_attributes", dict), + ("msvs_settings", dict), + ("msvs_system_include_dirs", list), + ("msvs_disabled_warnings", list), + ("msvs_precompiled_header", str), + ("msvs_precompiled_source", str), + ("msvs_configuration_platform", str), + ("msvs_target_platform", str), + ] + configs = spec["configurations"] + for field, default in supported_fields: + setattr(self, field, {}) + for configname, config in configs.items(): + getattr(self, field)[configname] = config.get(field, default()) + + self.msvs_cygwin_dirs = spec.get("msvs_cygwin_dirs", ["."]) + + unsupported_fields = [ + "msvs_prebuild", + "msvs_postbuild", ] - configs = spec['configurations'] - for field, default in supported_fields: - setattr(self, field, {}) - for configname, config in configs.items(): - getattr(self, field)[configname] = config.get(field, default()) - - self.msvs_cygwin_dirs = spec.get('msvs_cygwin_dirs', ['.']) - - unsupported_fields = [ - 'msvs_prebuild', - 'msvs_postbuild', - ] - unsupported = [] - for field in unsupported_fields: - for config in configs.values(): - if field in config: - unsupported += ["%s not supported (target %s)." % - (field, spec['target_name'])] - if unsupported: - raise Exception('\n'.join(unsupported)) - - def GetExtension(self): - """Returns the extension for the target, with no leading dot. + unsupported = [] + for field in unsupported_fields: + for config in configs.values(): + if field in config: + unsupported += [ + "%s not supported (target %s)." % (field, spec["target_name"]) + ] + if unsupported: + raise Exception("\n".join(unsupported)) + + def GetExtension(self): + """Returns the extension for the target, with no leading dot. Uses 'product_extension' if specified, otherwise uses MSVS defaults based on the target type. """ - ext = self.spec.get('product_extension', None) - if ext: - return ext - return gyp.MSVSUtil.TARGET_TYPE_EXT.get(self.spec['type'], '') + ext = self.spec.get("product_extension", None) + if ext: + return ext + return gyp.MSVSUtil.TARGET_TYPE_EXT.get(self.spec["type"], "") - def GetVSMacroEnv(self, base_to_build=None, config=None): - """Get a dict of variables mapping internal VS macro names to their gyp + def GetVSMacroEnv(self, base_to_build=None, config=None): + """Get a dict of variables mapping internal VS macro names to their gyp equivalents.""" - target_arch = self.GetArch(config) - if target_arch == 'x86': - target_platform = 'Win32' - else: - target_platform = target_arch - target_name = self.spec.get('product_prefix', '') + \ - self.spec.get('product_name', self.spec['target_name']) - target_dir = base_to_build + '\\' if base_to_build else '' - target_ext = '.' + self.GetExtension() - target_file_name = target_name + target_ext - - replacements = { - '$(InputName)': '${root}', - '$(InputPath)': '${source}', - '$(IntDir)': '$!INTERMEDIATE_DIR', - '$(OutDir)\\': target_dir, - '$(PlatformName)': target_platform, - '$(ProjectDir)\\': '', - '$(ProjectName)': self.spec['target_name'], - '$(TargetDir)\\': target_dir, - '$(TargetExt)': target_ext, - '$(TargetFileName)': target_file_name, - '$(TargetName)': target_name, - '$(TargetPath)': os.path.join(target_dir, target_file_name), - } - replacements.update(GetGlobalVSMacroEnv(self.vs_version)) - return replacements - - def ConvertVSMacros(self, s, base_to_build=None, config=None): - """Convert from VS macro names to something equivalent.""" - env = self.GetVSMacroEnv(base_to_build, config=config) - return ExpandMacros(s, env) - - def AdjustLibraries(self, libraries): - """Strip -l from library if it's specified with that.""" - libs = [lib[2:] if lib.startswith('-l') else lib for lib in libraries] - return [lib + '.lib' if not lib.lower().endswith('.lib') \ - and not lib.lower().endswith('.obj') else lib for lib in libs] - - def _GetAndMunge(self, field, path, default, prefix, append, map): - """Retrieve a value from |field| at |path| or return |default|. If + target_arch = self.GetArch(config) + if target_arch == "x86": + target_platform = "Win32" + else: + target_platform = target_arch + target_name = self.spec.get("product_prefix", "") + self.spec.get( + "product_name", self.spec["target_name"] + ) + target_dir = base_to_build + "\\" if base_to_build else "" + target_ext = "." + self.GetExtension() + target_file_name = target_name + target_ext + + replacements = { + "$(InputName)": "${root}", + "$(InputPath)": "${source}", + "$(IntDir)": "$!INTERMEDIATE_DIR", + "$(OutDir)\\": target_dir, + "$(PlatformName)": target_platform, + "$(ProjectDir)\\": "", + "$(ProjectName)": self.spec["target_name"], + "$(TargetDir)\\": target_dir, + "$(TargetExt)": target_ext, + "$(TargetFileName)": target_file_name, + "$(TargetName)": target_name, + "$(TargetPath)": os.path.join(target_dir, target_file_name), + } + replacements.update(GetGlobalVSMacroEnv(self.vs_version)) + return replacements + + def ConvertVSMacros(self, s, base_to_build=None, config=None): + """Convert from VS macro names to something equivalent.""" + env = self.GetVSMacroEnv(base_to_build, config=config) + return ExpandMacros(s, env) + + def AdjustLibraries(self, libraries): + """Strip -l from library if it's specified with that.""" + libs = [lib[2:] if lib.startswith("-l") else lib for lib in libraries] + return [ + lib + ".lib" + if not lib.lower().endswith(".lib") and not lib.lower().endswith(".obj") + else lib + for lib in libs + ] + + def _GetAndMunge(self, field, path, default, prefix, append, map): + """Retrieve a value from |field| at |path| or return |default|. If |append| is specified, and the item is found, it will be appended to that object instead of returned. If |map| is specified, results will be remapped through |map| before being returned or appended.""" - result = _GenericRetrieve(field, default, path) - result = _DoRemapping(result, map) - result = _AddPrefix(result, prefix) - return _AppendOrReturn(append, result) - - class _GetWrapper(object): - def __init__(self, parent, field, base_path, append=None): - self.parent = parent - self.field = field - self.base_path = [base_path] - self.append = append - def __call__(self, name, map=None, prefix='', default=None): - return self.parent._GetAndMunge(self.field, self.base_path + [name], - default=default, prefix=prefix, append=self.append, map=map) - - def GetArch(self, config): - """Get architecture based on msvs_configuration_platform and + result = _GenericRetrieve(field, default, path) + result = _DoRemapping(result, map) + result = _AddPrefix(result, prefix) + return _AppendOrReturn(append, result) + + class _GetWrapper(object): + def __init__(self, parent, field, base_path, append=None): + self.parent = parent + self.field = field + self.base_path = [base_path] + self.append = append + + def __call__(self, name, map=None, prefix="", default=None): + return self.parent._GetAndMunge( + self.field, + self.base_path + [name], + default=default, + prefix=prefix, + append=self.append, + map=map, + ) + + def GetArch(self, config): + """Get architecture based on msvs_configuration_platform and msvs_target_platform. Returns either 'x86' or 'x64'.""" - configuration_platform = self.msvs_configuration_platform.get(config, '') - platform = self.msvs_target_platform.get(config, '') - if not platform: # If no specific override, use the configuration's. - platform = configuration_platform - # Map from platform to architecture. - return {'Win32': 'x86', 'x64': 'x64', 'ARM64': 'arm64'}.get(platform, 'x86') - - def _TargetConfig(self, config): - """Returns the target-specific configuration.""" - # There's two levels of architecture/platform specification in VS. The - # first level is globally for the configuration (this is what we consider - # "the" config at the gyp level, which will be something like 'Debug' or - # 'Release'), VS2015 and later only use this level - if self.vs_version.short_name >= 2015: - return config - # and a second target-specific configuration, which is an - # override for the global one. |config| is remapped here to take into - # account the local target-specific overrides to the global configuration. - arch = self.GetArch(config) - if arch == 'x64' and not config.endswith('_x64'): - config += '_x64' - if arch == 'x86' and config.endswith('_x64'): - config = config.rsplit('_', 1)[0] - return config - - def _Setting(self, path, config, - default=None, prefix='', append=None, map=None): - """_GetAndMunge for msvs_settings.""" - return self._GetAndMunge( - self.msvs_settings[config], path, default, prefix, append, map) - - def _ConfigAttrib(self, path, config, - default=None, prefix='', append=None, map=None): - """_GetAndMunge for msvs_configuration_attributes.""" - return self._GetAndMunge( - self.msvs_configuration_attributes[config], - path, default, prefix, append, map) - - def AdjustIncludeDirs(self, include_dirs, config): - """Updates include_dirs to expand VS specific paths, and adds the system + configuration_platform = self.msvs_configuration_platform.get(config, "") + platform = self.msvs_target_platform.get(config, "") + if not platform: # If no specific override, use the configuration's. + platform = configuration_platform + # Map from platform to architecture. + return {"Win32": "x86", "x64": "x64", "ARM64": "arm64"}.get(platform, "x86") + + def _TargetConfig(self, config): + """Returns the target-specific configuration.""" + # There's two levels of architecture/platform specification in VS. The + # first level is globally for the configuration (this is what we consider + # "the" config at the gyp level, which will be something like 'Debug' or + # 'Release'), VS2015 and later only use this level + if self.vs_version.short_name >= 2015: + return config + # and a second target-specific configuration, which is an + # override for the global one. |config| is remapped here to take into + # account the local target-specific overrides to the global configuration. + arch = self.GetArch(config) + if arch == "x64" and not config.endswith("_x64"): + config += "_x64" + if arch == "x86" and config.endswith("_x64"): + config = config.rsplit("_", 1)[0] + return config + + def _Setting(self, path, config, default=None, prefix="", append=None, map=None): + """_GetAndMunge for msvs_settings.""" + return self._GetAndMunge( + self.msvs_settings[config], path, default, prefix, append, map + ) + + def _ConfigAttrib( + self, path, config, default=None, prefix="", append=None, map=None + ): + """_GetAndMunge for msvs_configuration_attributes.""" + return self._GetAndMunge( + self.msvs_configuration_attributes[config], + path, + default, + prefix, + append, + map, + ) + + def AdjustIncludeDirs(self, include_dirs, config): + """Updates include_dirs to expand VS specific paths, and adds the system include dirs used for platform SDK and similar.""" - config = self._TargetConfig(config) - includes = include_dirs + self.msvs_system_include_dirs[config] - includes.extend(self._Setting( - ('VCCLCompilerTool', 'AdditionalIncludeDirectories'), config, default=[])) - return [self.ConvertVSMacros(p, config=config) for p in includes] - - def AdjustMidlIncludeDirs(self, midl_include_dirs, config): - """Updates midl_include_dirs to expand VS specific paths, and adds the + config = self._TargetConfig(config) + includes = include_dirs + self.msvs_system_include_dirs[config] + includes.extend( + self._Setting( + ("VCCLCompilerTool", "AdditionalIncludeDirectories"), config, default=[] + ) + ) + return [self.ConvertVSMacros(p, config=config) for p in includes] + + def AdjustMidlIncludeDirs(self, midl_include_dirs, config): + """Updates midl_include_dirs to expand VS specific paths, and adds the system include dirs used for platform SDK and similar.""" - config = self._TargetConfig(config) - includes = midl_include_dirs + self.msvs_system_include_dirs[config] - includes.extend(self._Setting( - ('VCMIDLTool', 'AdditionalIncludeDirectories'), config, default=[])) - return [self.ConvertVSMacros(p, config=config) for p in includes] - - def GetComputedDefines(self, config): - """Returns the set of defines that are injected to the defines list based + config = self._TargetConfig(config) + includes = midl_include_dirs + self.msvs_system_include_dirs[config] + includes.extend( + self._Setting( + ("VCMIDLTool", "AdditionalIncludeDirectories"), config, default=[] + ) + ) + return [self.ConvertVSMacros(p, config=config) for p in includes] + + def GetComputedDefines(self, config): + """Returns the set of defines that are injected to the defines list based on other VS settings.""" - config = self._TargetConfig(config) - defines = [] - if self._ConfigAttrib(['CharacterSet'], config) == '1': - defines.extend(('_UNICODE', 'UNICODE')) - if self._ConfigAttrib(['CharacterSet'], config) == '2': - defines.append('_MBCS') - defines.extend(self._Setting( - ('VCCLCompilerTool', 'PreprocessorDefinitions'), config, default=[])) - return defines - - def GetCompilerPdbName(self, config, expand_special): - """Get the pdb file name that should be used for compiler invocations, or + config = self._TargetConfig(config) + defines = [] + if self._ConfigAttrib(["CharacterSet"], config) == "1": + defines.extend(("_UNICODE", "UNICODE")) + if self._ConfigAttrib(["CharacterSet"], config) == "2": + defines.append("_MBCS") + defines.extend( + self._Setting( + ("VCCLCompilerTool", "PreprocessorDefinitions"), config, default=[] + ) + ) + return defines + + def GetCompilerPdbName(self, config, expand_special): + """Get the pdb file name that should be used for compiler invocations, or None if there's no explicit name specified.""" - config = self._TargetConfig(config) - pdbname = self._Setting( - ('VCCLCompilerTool', 'ProgramDataBaseFileName'), config) - if pdbname: - pdbname = expand_special(self.ConvertVSMacros(pdbname)) - return pdbname - - def GetMapFileName(self, config, expand_special): - """Gets the explicitly overridden map file name for a target or returns None + config = self._TargetConfig(config) + pdbname = self._Setting(("VCCLCompilerTool", "ProgramDataBaseFileName"), config) + if pdbname: + pdbname = expand_special(self.ConvertVSMacros(pdbname)) + return pdbname + + def GetMapFileName(self, config, expand_special): + """Gets the explicitly overridden map file name for a target or returns None if it's not set.""" - config = self._TargetConfig(config) - map_file = self._Setting(('VCLinkerTool', 'MapFileName'), config) - if map_file: - map_file = expand_special(self.ConvertVSMacros(map_file, config=config)) - return map_file - - def GetOutputName(self, config, expand_special): - """Gets the explicitly overridden output name for a target or returns None + config = self._TargetConfig(config) + map_file = self._Setting(("VCLinkerTool", "MapFileName"), config) + if map_file: + map_file = expand_special(self.ConvertVSMacros(map_file, config=config)) + return map_file + + def GetOutputName(self, config, expand_special): + """Gets the explicitly overridden output name for a target or returns None if it's not overridden.""" - config = self._TargetConfig(config) - type = self.spec['type'] - root = 'VCLibrarianTool' if type == 'static_library' else 'VCLinkerTool' - # TODO(scottmg): Handle OutputDirectory without OutputFile. - output_file = self._Setting((root, 'OutputFile'), config) - if output_file: - output_file = expand_special(self.ConvertVSMacros( - output_file, config=config)) - return output_file - - def GetPDBName(self, config, expand_special, default): - """Gets the explicitly overridden pdb name for a target or returns + config = self._TargetConfig(config) + type = self.spec["type"] + root = "VCLibrarianTool" if type == "static_library" else "VCLinkerTool" + # TODO(scottmg): Handle OutputDirectory without OutputFile. + output_file = self._Setting((root, "OutputFile"), config) + if output_file: + output_file = expand_special( + self.ConvertVSMacros(output_file, config=config) + ) + return output_file + + def GetPDBName(self, config, expand_special, default): + """Gets the explicitly overridden pdb name for a target or returns default if it's not overridden, or if no pdb will be generated.""" - config = self._TargetConfig(config) - output_file = self._Setting(('VCLinkerTool', 'ProgramDatabaseFile'), config) - generate_debug_info = self._Setting( - ('VCLinkerTool', 'GenerateDebugInformation'), config) - if generate_debug_info == 'true': - if output_file: - return expand_special(self.ConvertVSMacros(output_file, config=config)) - else: - return default - else: - return None - - def GetNoImportLibrary(self, config): - """If NoImportLibrary: true, ninja will not expect the output to include + config = self._TargetConfig(config) + output_file = self._Setting(("VCLinkerTool", "ProgramDatabaseFile"), config) + generate_debug_info = self._Setting( + ("VCLinkerTool", "GenerateDebugInformation"), config + ) + if generate_debug_info == "true": + if output_file: + return expand_special(self.ConvertVSMacros(output_file, config=config)) + else: + return default + else: + return None + + def GetNoImportLibrary(self, config): + """If NoImportLibrary: true, ninja will not expect the output to include an import library.""" - config = self._TargetConfig(config) - noimplib = self._Setting(('NoImportLibrary',), config) - return noimplib == 'true' - - def GetAsmflags(self, config): - """Returns the flags that need to be added to ml invocations.""" - config = self._TargetConfig(config) - asmflags = [] - safeseh = self._Setting(('MASM', 'UseSafeExceptionHandlers'), config) - if safeseh == 'true': - asmflags.append('/safeseh') - return asmflags - - def GetCflags(self, config): - """Returns the flags that need to be added to .c and .cc compilations.""" - config = self._TargetConfig(config) - cflags = [] - cflags.extend(['/wd' + w for w in self.msvs_disabled_warnings[config]]) - cl = self._GetWrapper(self, self.msvs_settings[config], - 'VCCLCompilerTool', append=cflags) - cl('Optimization', - map={'0': 'd', '1': '1', '2': '2', '3': 'x'}, prefix='/O', default='2') - cl('InlineFunctionExpansion', prefix='/Ob') - cl('DisableSpecificWarnings', prefix='/wd') - cl('StringPooling', map={'true': '/GF'}) - cl('EnableFiberSafeOptimizations', map={'true': '/GT'}) - cl('OmitFramePointers', map={'false': '-', 'true': ''}, prefix='/Oy') - cl('EnableIntrinsicFunctions', map={'false': '-', 'true': ''}, prefix='/Oi') - cl('FavorSizeOrSpeed', map={'1': 't', '2': 's'}, prefix='/O') - cl('FloatingPointModel', - map={'0': 'precise', '1': 'strict', '2': 'fast'}, prefix='/fp:', - default='0') - cl('CompileAsManaged', map={'false': '', 'true': '/clr'}) - cl('WholeProgramOptimization', map={'true': '/GL'}) - cl('WarningLevel', prefix='/W') - cl('WarnAsError', map={'true': '/WX'}) - cl('CallingConvention', - map={'0': 'd', '1': 'r', '2': 'z', '3': 'v'}, prefix='/G') - cl('DebugInformationFormat', - map={'1': '7', '3': 'i', '4': 'I'}, prefix='/Z') - cl('RuntimeTypeInfo', map={'true': '/GR', 'false': '/GR-'}) - cl('EnableFunctionLevelLinking', map={'true': '/Gy', 'false': '/Gy-'}) - cl('MinimalRebuild', map={'true': '/Gm'}) - cl('BufferSecurityCheck', map={'true': '/GS', 'false': '/GS-'}) - cl('BasicRuntimeChecks', map={'1': 's', '2': 'u', '3': '1'}, prefix='/RTC') - cl('RuntimeLibrary', - map={'0': 'T', '1': 'Td', '2': 'D', '3': 'Dd'}, prefix='/M') - cl('ExceptionHandling', map={'1': 'sc','2': 'a'}, prefix='/EH') - cl('DefaultCharIsUnsigned', map={'true': '/J'}) - cl('TreatWChar_tAsBuiltInType', - map={'false': '-', 'true': ''}, prefix='/Zc:wchar_t') - cl('EnablePREfast', map={'true': '/analyze'}) - cl('AdditionalOptions', prefix='') - cl('EnableEnhancedInstructionSet', - map={'1': 'SSE', '2': 'SSE2', '3': 'AVX', '4': 'IA32', '5': 'AVX2'}, - prefix='/arch:') - cflags.extend(['/FI' + f for f in self._Setting( - ('VCCLCompilerTool', 'ForcedIncludeFiles'), config, default=[])]) - if self.vs_version.project_version >= 12.0: - # New flag introduced in VS2013 (project version 12.0) Forces writes to - # the program database (PDB) to be serialized through MSPDBSRV.EXE. - # https://msdn.microsoft.com/en-us/library/dn502518.aspx - cflags.append('/FS') - # ninja handles parallelism by itself, don't have the compiler do it too. - cflags = filter(lambda x: not x.startswith('/MP'), cflags) - return cflags - - def _GetPchFlags(self, config, extension): - """Get the flags to be added to the cflags for precompiled header support. + config = self._TargetConfig(config) + noimplib = self._Setting(("NoImportLibrary",), config) + return noimplib == "true" + + def GetAsmflags(self, config): + """Returns the flags that need to be added to ml invocations.""" + config = self._TargetConfig(config) + asmflags = [] + safeseh = self._Setting(("MASM", "UseSafeExceptionHandlers"), config) + if safeseh == "true": + asmflags.append("/safeseh") + return asmflags + + def GetCflags(self, config): + """Returns the flags that need to be added to .c and .cc compilations.""" + config = self._TargetConfig(config) + cflags = [] + cflags.extend(["/wd" + w for w in self.msvs_disabled_warnings[config]]) + cl = self._GetWrapper( + self, self.msvs_settings[config], "VCCLCompilerTool", append=cflags + ) + cl( + "Optimization", + map={"0": "d", "1": "1", "2": "2", "3": "x"}, + prefix="/O", + default="2", + ) + cl("InlineFunctionExpansion", prefix="/Ob") + cl("DisableSpecificWarnings", prefix="/wd") + cl("StringPooling", map={"true": "/GF"}) + cl("EnableFiberSafeOptimizations", map={"true": "/GT"}) + cl("OmitFramePointers", map={"false": "-", "true": ""}, prefix="/Oy") + cl("EnableIntrinsicFunctions", map={"false": "-", "true": ""}, prefix="/Oi") + cl("FavorSizeOrSpeed", map={"1": "t", "2": "s"}, prefix="/O") + cl( + "FloatingPointModel", + map={"0": "precise", "1": "strict", "2": "fast"}, + prefix="/fp:", + default="0", + ) + cl("CompileAsManaged", map={"false": "", "true": "/clr"}) + cl("WholeProgramOptimization", map={"true": "/GL"}) + cl("WarningLevel", prefix="/W") + cl("WarnAsError", map={"true": "/WX"}) + cl( + "CallingConvention", + map={"0": "d", "1": "r", "2": "z", "3": "v"}, + prefix="/G", + ) + cl("DebugInformationFormat", map={"1": "7", "3": "i", "4": "I"}, prefix="/Z") + cl("RuntimeTypeInfo", map={"true": "/GR", "false": "/GR-"}) + cl("EnableFunctionLevelLinking", map={"true": "/Gy", "false": "/Gy-"}) + cl("MinimalRebuild", map={"true": "/Gm"}) + cl("BufferSecurityCheck", map={"true": "/GS", "false": "/GS-"}) + cl("BasicRuntimeChecks", map={"1": "s", "2": "u", "3": "1"}, prefix="/RTC") + cl( + "RuntimeLibrary", + map={"0": "T", "1": "Td", "2": "D", "3": "Dd"}, + prefix="/M", + ) + cl("ExceptionHandling", map={"1": "sc", "2": "a"}, prefix="/EH") + cl("DefaultCharIsUnsigned", map={"true": "/J"}) + cl( + "TreatWChar_tAsBuiltInType", + map={"false": "-", "true": ""}, + prefix="/Zc:wchar_t", + ) + cl("EnablePREfast", map={"true": "/analyze"}) + cl("AdditionalOptions", prefix="") + cl( + "EnableEnhancedInstructionSet", + map={"1": "SSE", "2": "SSE2", "3": "AVX", "4": "IA32", "5": "AVX2"}, + prefix="/arch:", + ) + cflags.extend( + [ + "/FI" + f + for f in self._Setting( + ("VCCLCompilerTool", "ForcedIncludeFiles"), config, default=[] + ) + ] + ) + if self.vs_version.project_version >= 12.0: + # New flag introduced in VS2013 (project version 12.0) Forces writes to + # the program database (PDB) to be serialized through MSPDBSRV.EXE. + # https://msdn.microsoft.com/en-us/library/dn502518.aspx + cflags.append("/FS") + # ninja handles parallelism by itself, don't have the compiler do it too. + cflags = [x for x in cflags if not x.startswith("/MP")] + return cflags + + def _GetPchFlags(self, config, extension): + """Get the flags to be added to the cflags for precompiled header support. """ - config = self._TargetConfig(config) - # The PCH is only built once by a particular source file. Usage of PCH must - # only be for the same language (i.e. C vs. C++), so only include the pch - # flags when the language matches. - if self.msvs_precompiled_header[config]: - source_ext = os.path.splitext(self.msvs_precompiled_source[config])[1] - if _LanguageMatchesForPch(source_ext, extension): - pch = self.msvs_precompiled_header[config] - pchbase = os.path.split(pch)[1] - return ['/Yu' + pch, '/FI' + pch, '/Fp${pchprefix}.' + pchbase + '.pch'] - return [] - - def GetCflagsC(self, config): - """Returns the flags that need to be added to .c compilations.""" - config = self._TargetConfig(config) - return self._GetPchFlags(config, '.c') - - def GetCflagsCC(self, config): - """Returns the flags that need to be added to .cc compilations.""" - config = self._TargetConfig(config) - return ['/TP'] + self._GetPchFlags(config, '.cc') - - def _GetAdditionalLibraryDirectories(self, root, config, gyp_to_build_path): - """Get and normalize the list of paths in AdditionalLibraryDirectories + config = self._TargetConfig(config) + # The PCH is only built once by a particular source file. Usage of PCH must + # only be for the same language (i.e. C vs. C++), so only include the pch + # flags when the language matches. + if self.msvs_precompiled_header[config]: + source_ext = os.path.splitext(self.msvs_precompiled_source[config])[1] + if _LanguageMatchesForPch(source_ext, extension): + pch = self.msvs_precompiled_header[config] + pchbase = os.path.split(pch)[1] + return ["/Yu" + pch, "/FI" + pch, "/Fp${pchprefix}." + pchbase + ".pch"] + return [] + + def GetCflagsC(self, config): + """Returns the flags that need to be added to .c compilations.""" + config = self._TargetConfig(config) + return self._GetPchFlags(config, ".c") + + def GetCflagsCC(self, config): + """Returns the flags that need to be added to .cc compilations.""" + config = self._TargetConfig(config) + return ["/TP"] + self._GetPchFlags(config, ".cc") + + def _GetAdditionalLibraryDirectories(self, root, config, gyp_to_build_path): + """Get and normalize the list of paths in AdditionalLibraryDirectories setting.""" - config = self._TargetConfig(config) - libpaths = self._Setting((root, 'AdditionalLibraryDirectories'), - config, default=[]) - libpaths = [os.path.normpath( - gyp_to_build_path(self.ConvertVSMacros(p, config=config))) - for p in libpaths] - return ['/LIBPATH:"' + p + '"' for p in libpaths] - - def GetLibFlags(self, config, gyp_to_build_path): - """Returns the flags that need to be added to lib commands.""" - config = self._TargetConfig(config) - libflags = [] - lib = self._GetWrapper(self, self.msvs_settings[config], - 'VCLibrarianTool', append=libflags) - libflags.extend(self._GetAdditionalLibraryDirectories( - 'VCLibrarianTool', config, gyp_to_build_path)) - lib('LinkTimeCodeGeneration', map={'true': '/LTCG'}) - lib('TargetMachine', map={'1': 'X86', '17': 'X64', '3': 'ARM'}, - prefix='/MACHINE:') - lib('AdditionalOptions') - return libflags - - def GetDefFile(self, gyp_to_build_path): - """Returns the .def file from sources, if any. Otherwise returns None.""" - spec = self.spec - if spec['type'] in ('shared_library', 'loadable_module', 'executable'): - def_files = [s for s in spec.get('sources', []) - if s.lower().endswith('.def')] - if len(def_files) == 1: - return gyp_to_build_path(def_files[0]) - elif len(def_files) > 1: - raise Exception("Multiple .def files") - return None - - def _GetDefFileAsLdflags(self, ldflags, gyp_to_build_path): - """.def files get implicitly converted to a ModuleDefinitionFile for the + config = self._TargetConfig(config) + libpaths = self._Setting( + (root, "AdditionalLibraryDirectories"), config, default=[] + ) + libpaths = [ + os.path.normpath(gyp_to_build_path(self.ConvertVSMacros(p, config=config))) + for p in libpaths + ] + return ['/LIBPATH:"' + p + '"' for p in libpaths] + + def GetLibFlags(self, config, gyp_to_build_path): + """Returns the flags that need to be added to lib commands.""" + config = self._TargetConfig(config) + libflags = [] + lib = self._GetWrapper( + self, self.msvs_settings[config], "VCLibrarianTool", append=libflags + ) + libflags.extend( + self._GetAdditionalLibraryDirectories( + "VCLibrarianTool", config, gyp_to_build_path + ) + ) + lib("LinkTimeCodeGeneration", map={"true": "/LTCG"}) + lib( + "TargetMachine", + map={"1": "X86", "17": "X64", "3": "ARM"}, + prefix="/MACHINE:", + ) + lib("AdditionalOptions") + return libflags + + def GetDefFile(self, gyp_to_build_path): + """Returns the .def file from sources, if any. Otherwise returns None.""" + spec = self.spec + if spec["type"] in ("shared_library", "loadable_module", "executable"): + def_files = [ + s for s in spec.get("sources", []) if s.lower().endswith(".def") + ] + if len(def_files) == 1: + return gyp_to_build_path(def_files[0]) + elif len(def_files) > 1: + raise Exception("Multiple .def files") + return None + + def _GetDefFileAsLdflags(self, ldflags, gyp_to_build_path): + """.def files get implicitly converted to a ModuleDefinitionFile for the linker in the VS generator. Emulate that behaviour here.""" - def_file = self.GetDefFile(gyp_to_build_path) - if def_file: - ldflags.append('/DEF:"%s"' % def_file) + def_file = self.GetDefFile(gyp_to_build_path) + if def_file: + ldflags.append('/DEF:"%s"' % def_file) - def GetPGDName(self, config, expand_special): - """Gets the explicitly overridden pgd name for a target or returns None + def GetPGDName(self, config, expand_special): + """Gets the explicitly overridden pgd name for a target or returns None if it's not overridden.""" - config = self._TargetConfig(config) - output_file = self._Setting( - ('VCLinkerTool', 'ProfileGuidedDatabase'), config) - if output_file: - output_file = expand_special(self.ConvertVSMacros( - output_file, config=config)) - return output_file - - def GetLdflags(self, config, gyp_to_build_path, expand_special, - manifest_base_name, output_name, is_executable, build_dir): - """Returns the flags that need to be added to link commands, and the + config = self._TargetConfig(config) + output_file = self._Setting(("VCLinkerTool", "ProfileGuidedDatabase"), config) + if output_file: + output_file = expand_special( + self.ConvertVSMacros(output_file, config=config) + ) + return output_file + + def GetLdflags( + self, + config, + gyp_to_build_path, + expand_special, + manifest_base_name, + output_name, + is_executable, + build_dir, + ): + """Returns the flags that need to be added to link commands, and the manifest files.""" - config = self._TargetConfig(config) - ldflags = [] - ld = self._GetWrapper(self, self.msvs_settings[config], - 'VCLinkerTool', append=ldflags) - self._GetDefFileAsLdflags(ldflags, gyp_to_build_path) - ld('GenerateDebugInformation', map={'true': '/DEBUG'}) - # TODO: These 'map' values come from machineTypeOption enum, - # and does not have an official value for ARM64 in VS2017 (yet). - # It needs to verify the ARM64 value when machineTypeOption is updated. - ld('TargetMachine', map={'1': 'X86', '17': 'X64', '3': 'ARM', '18': 'ARM64'}, - prefix='/MACHINE:') - ldflags.extend(self._GetAdditionalLibraryDirectories( - 'VCLinkerTool', config, gyp_to_build_path)) - ld('DelayLoadDLLs', prefix='/DELAYLOAD:') - ld('TreatLinkerWarningAsErrors', prefix='/WX', - map={'true': '', 'false': ':NO'}) - out = self.GetOutputName(config, expand_special) - if out: - ldflags.append('/OUT:' + out) - pdb = self.GetPDBName(config, expand_special, output_name + '.pdb') - if pdb: - ldflags.append('/PDB:' + pdb) - pgd = self.GetPGDName(config, expand_special) - if pgd: - ldflags.append('/PGD:' + pgd) - map_file = self.GetMapFileName(config, expand_special) - ld('GenerateMapFile', map={'true': '/MAP:' + map_file if map_file - else '/MAP'}) - ld('MapExports', map={'true': '/MAPINFO:EXPORTS'}) - ld('AdditionalOptions', prefix='') - - minimum_required_version = self._Setting( - ('VCLinkerTool', 'MinimumRequiredVersion'), config, default='') - if minimum_required_version: - minimum_required_version = ',' + minimum_required_version - ld('SubSystem', - map={'1': 'CONSOLE%s' % minimum_required_version, - '2': 'WINDOWS%s' % minimum_required_version}, - prefix='/SUBSYSTEM:') - - stack_reserve_size = self._Setting( - ('VCLinkerTool', 'StackReserveSize'), config, default='') - if stack_reserve_size: - stack_commit_size = self._Setting( - ('VCLinkerTool', 'StackCommitSize'), config, default='') - if stack_commit_size: - stack_commit_size = ',' + stack_commit_size - ldflags.append('/STACK:%s%s' % (stack_reserve_size, stack_commit_size)) - - ld('TerminalServerAware', map={'1': ':NO', '2': ''}, prefix='/TSAWARE') - ld('LinkIncremental', map={'1': ':NO', '2': ''}, prefix='/INCREMENTAL') - ld('BaseAddress', prefix='/BASE:') - ld('FixedBaseAddress', map={'1': ':NO', '2': ''}, prefix='/FIXED') - ld('RandomizedBaseAddress', - map={'1': ':NO', '2': ''}, prefix='/DYNAMICBASE') - ld('DataExecutionPrevention', - map={'1': ':NO', '2': ''}, prefix='/NXCOMPAT') - ld('OptimizeReferences', map={'1': 'NOREF', '2': 'REF'}, prefix='/OPT:') - ld('ForceSymbolReferences', prefix='/INCLUDE:') - ld('EnableCOMDATFolding', map={'1': 'NOICF', '2': 'ICF'}, prefix='/OPT:') - ld('LinkTimeCodeGeneration', - map={'1': '', '2': ':PGINSTRUMENT', '3': ':PGOPTIMIZE', - '4': ':PGUPDATE'}, - prefix='/LTCG') - ld('IgnoreDefaultLibraryNames', prefix='/NODEFAULTLIB:') - ld('ResourceOnlyDLL', map={'true': '/NOENTRY'}) - ld('EntryPointSymbol', prefix='/ENTRY:') - ld('Profile', map={'true': '/PROFILE'}) - ld('LargeAddressAware', - map={'1': ':NO', '2': ''}, prefix='/LARGEADDRESSAWARE') - # TODO(scottmg): This should sort of be somewhere else (not really a flag). - ld('AdditionalDependencies', prefix='') - - if self.GetArch(config) == 'x86': - safeseh_default = 'true' - else: - safeseh_default = None - ld('ImageHasSafeExceptionHandlers', - map={'false': ':NO', 'true': ''}, prefix='/SAFESEH', - default=safeseh_default) - - # If the base address is not specifically controlled, DYNAMICBASE should - # be on by default. - base_flags = filter(lambda x: 'DYNAMICBASE' in x or x == '/FIXED', - ldflags) - if not base_flags: - ldflags.append('/DYNAMICBASE') - - # If the NXCOMPAT flag has not been specified, default to on. Despite the - # documentation that says this only defaults to on when the subsystem is - # Vista or greater (which applies to the linker), the IDE defaults it on - # unless it's explicitly off. - if not filter(lambda x: 'NXCOMPAT' in x, ldflags): - ldflags.append('/NXCOMPAT') - - have_def_file = filter(lambda x: x.startswith('/DEF:'), ldflags) - manifest_flags, intermediate_manifest, manifest_files = \ - self._GetLdManifestFlags(config, manifest_base_name, gyp_to_build_path, - is_executable and not have_def_file, build_dir) - ldflags.extend(manifest_flags) - return ldflags, intermediate_manifest, manifest_files - - def _GetLdManifestFlags(self, config, name, gyp_to_build_path, - allow_isolation, build_dir): - """Returns a 3-tuple: + config = self._TargetConfig(config) + ldflags = [] + ld = self._GetWrapper( + self, self.msvs_settings[config], "VCLinkerTool", append=ldflags + ) + self._GetDefFileAsLdflags(ldflags, gyp_to_build_path) + ld("GenerateDebugInformation", map={"true": "/DEBUG"}) + # TODO: These 'map' values come from machineTypeOption enum, + # and does not have an official value for ARM64 in VS2017 (yet). + # It needs to verify the ARM64 value when machineTypeOption is updated. + ld( + "TargetMachine", + map={"1": "X86", "17": "X64", "3": "ARM", "18": "ARM64"}, + prefix="/MACHINE:", + ) + ldflags.extend( + self._GetAdditionalLibraryDirectories( + "VCLinkerTool", config, gyp_to_build_path + ) + ) + ld("DelayLoadDLLs", prefix="/DELAYLOAD:") + ld("TreatLinkerWarningAsErrors", prefix="/WX", map={"true": "", "false": ":NO"}) + out = self.GetOutputName(config, expand_special) + if out: + ldflags.append("/OUT:" + out) + pdb = self.GetPDBName(config, expand_special, output_name + ".pdb") + if pdb: + ldflags.append("/PDB:" + pdb) + pgd = self.GetPGDName(config, expand_special) + if pgd: + ldflags.append("/PGD:" + pgd) + map_file = self.GetMapFileName(config, expand_special) + ld("GenerateMapFile", map={"true": "/MAP:" + map_file if map_file else "/MAP"}) + ld("MapExports", map={"true": "/MAPINFO:EXPORTS"}) + ld("AdditionalOptions", prefix="") + + minimum_required_version = self._Setting( + ("VCLinkerTool", "MinimumRequiredVersion"), config, default="" + ) + if minimum_required_version: + minimum_required_version = "," + minimum_required_version + ld( + "SubSystem", + map={ + "1": "CONSOLE%s" % minimum_required_version, + "2": "WINDOWS%s" % minimum_required_version, + }, + prefix="/SUBSYSTEM:", + ) + + stack_reserve_size = self._Setting( + ("VCLinkerTool", "StackReserveSize"), config, default="" + ) + if stack_reserve_size: + stack_commit_size = self._Setting( + ("VCLinkerTool", "StackCommitSize"), config, default="" + ) + if stack_commit_size: + stack_commit_size = "," + stack_commit_size + ldflags.append("/STACK:%s%s" % (stack_reserve_size, stack_commit_size)) + + ld("TerminalServerAware", map={"1": ":NO", "2": ""}, prefix="/TSAWARE") + ld("LinkIncremental", map={"1": ":NO", "2": ""}, prefix="/INCREMENTAL") + ld("BaseAddress", prefix="/BASE:") + ld("FixedBaseAddress", map={"1": ":NO", "2": ""}, prefix="/FIXED") + ld("RandomizedBaseAddress", map={"1": ":NO", "2": ""}, prefix="/DYNAMICBASE") + ld("DataExecutionPrevention", map={"1": ":NO", "2": ""}, prefix="/NXCOMPAT") + ld("OptimizeReferences", map={"1": "NOREF", "2": "REF"}, prefix="/OPT:") + ld("ForceSymbolReferences", prefix="/INCLUDE:") + ld("EnableCOMDATFolding", map={"1": "NOICF", "2": "ICF"}, prefix="/OPT:") + ld( + "LinkTimeCodeGeneration", + map={"1": "", "2": ":PGINSTRUMENT", "3": ":PGOPTIMIZE", "4": ":PGUPDATE"}, + prefix="/LTCG", + ) + ld("IgnoreDefaultLibraryNames", prefix="/NODEFAULTLIB:") + ld("ResourceOnlyDLL", map={"true": "/NOENTRY"}) + ld("EntryPointSymbol", prefix="/ENTRY:") + ld("Profile", map={"true": "/PROFILE"}) + ld("LargeAddressAware", map={"1": ":NO", "2": ""}, prefix="/LARGEADDRESSAWARE") + # TODO(scottmg): This should sort of be somewhere else (not really a flag). + ld("AdditionalDependencies", prefix="") + + if self.GetArch(config) == "x86": + safeseh_default = "true" + else: + safeseh_default = None + ld( + "ImageHasSafeExceptionHandlers", + map={"false": ":NO", "true": ""}, + prefix="/SAFESEH", + default=safeseh_default, + ) + + # If the base address is not specifically controlled, DYNAMICBASE should + # be on by default. + if not any("DYNAMICBASE" in flag or flag == "/FIXED" for flag in ldflags): + ldflags.append("/DYNAMICBASE") + + # If the NXCOMPAT flag has not been specified, default to on. Despite the + # documentation that says this only defaults to on when the subsystem is + # Vista or greater (which applies to the linker), the IDE defaults it on + # unless it's explicitly off. + if not any("NXCOMPAT" in flag for flag in ldflags): + ldflags.append("/NXCOMPAT") + + have_def_file = any(flag.startswith("/DEF:") for flag in ldflags) + ( + manifest_flags, + intermediate_manifest, + manifest_files, + ) = self._GetLdManifestFlags( + config, + manifest_base_name, + gyp_to_build_path, + is_executable and not have_def_file, + build_dir, + ) + ldflags.extend(manifest_flags) + return ldflags, intermediate_manifest, manifest_files + + def _GetLdManifestFlags( + self, config, name, gyp_to_build_path, allow_isolation, build_dir + ): + """Returns a 3-tuple: - the set of flags that need to be added to the link to generate a default manifest - the intermediate manifest that the linker will generate that should be used to assert it doesn't add anything to the merged one. - the list of all the manifest files to be merged by the manifest tool and included into the link.""" - generate_manifest = self._Setting(('VCLinkerTool', 'GenerateManifest'), - config, - default='true') - if generate_manifest != 'true': - # This means not only that the linker should not generate the intermediate - # manifest but also that the manifest tool should do nothing even when - # additional manifests are specified. - return ['/MANIFEST:NO'], [], [] - - output_name = name + '.intermediate.manifest' - flags = [ - '/MANIFEST', - '/ManifestFile:' + output_name, - ] - - # Instead of using the MANIFESTUAC flags, we generate a .manifest to - # include into the list of manifests. This allows us to avoid the need to - # do two passes during linking. The /MANIFEST flag and /ManifestFile are - # still used, and the intermediate manifest is used to assert that the - # final manifest we get from merging all the additional manifest files - # (plus the one we generate here) isn't modified by merging the - # intermediate into it. - - # Always NO, because we generate a manifest file that has what we want. - flags.append('/MANIFESTUAC:NO') - - config = self._TargetConfig(config) - enable_uac = self._Setting(('VCLinkerTool', 'EnableUAC'), config, - default='true') - manifest_files = [] - generated_manifest_outer = \ -"" \ -"%s" \ -"" - if enable_uac == 'true': - execution_level = self._Setting(('VCLinkerTool', 'UACExecutionLevel'), - config, default='0') - execution_level_map = { - '0': 'asInvoker', - '1': 'highestAvailable', - '2': 'requireAdministrator' - } - - ui_access = self._Setting(('VCLinkerTool', 'UACUIAccess'), config, - default='false') - - inner = ''' + generate_manifest = self._Setting( + ("VCLinkerTool", "GenerateManifest"), config, default="true" + ) + if generate_manifest != "true": + # This means not only that the linker should not generate the intermediate + # manifest but also that the manifest tool should do nothing even when + # additional manifests are specified. + return ["/MANIFEST:NO"], [], [] + + output_name = name + ".intermediate.manifest" + flags = [ + "/MANIFEST", + "/ManifestFile:" + output_name, + ] + + # Instead of using the MANIFESTUAC flags, we generate a .manifest to + # include into the list of manifests. This allows us to avoid the need to + # do two passes during linking. The /MANIFEST flag and /ManifestFile are + # still used, and the intermediate manifest is used to assert that the + # final manifest we get from merging all the additional manifest files + # (plus the one we generate here) isn't modified by merging the + # intermediate into it. + + # Always NO, because we generate a manifest file that has what we want. + flags.append("/MANIFESTUAC:NO") + + config = self._TargetConfig(config) + enable_uac = self._Setting( + ("VCLinkerTool", "EnableUAC"), config, default="true" + ) + manifest_files = [] + generated_manifest_outer = ( + "" + "%s" + "" + ) + if enable_uac == "true": + execution_level = self._Setting( + ("VCLinkerTool", "UACExecutionLevel"), config, default="0" + ) + execution_level_map = { + "0": "asInvoker", + "1": "highestAvailable", + "2": "requireAdministrator", + } + + ui_access = self._Setting( + ("VCLinkerTool", "UACUIAccess"), config, default="false" + ) + + inner = """ -''' % (execution_level_map[execution_level], ui_access) - else: - inner = '' - - generated_manifest_contents = generated_manifest_outer % inner - generated_name = name + '.generated.manifest' - # Need to join with the build_dir here as we're writing it during - # generation time, but we return the un-joined version because the build - # will occur in that directory. We only write the file if the contents - # have changed so that simply regenerating the project files doesn't - # cause a relink. - build_dir_generated_name = os.path.join(build_dir, generated_name) - gyp.common.EnsureDirExists(build_dir_generated_name) - f = gyp.common.WriteOnDiff(build_dir_generated_name) - f.write(generated_manifest_contents) - f.close() - manifest_files = [generated_name] - - if allow_isolation: - flags.append('/ALLOWISOLATION') - - manifest_files += self._GetAdditionalManifestFiles(config, - gyp_to_build_path) - return flags, output_name, manifest_files - - def _GetAdditionalManifestFiles(self, config, gyp_to_build_path): - """Gets additional manifest files that are added to the default one +""" % ( + execution_level_map[execution_level], + ui_access, + ) + else: + inner = "" + + generated_manifest_contents = generated_manifest_outer % inner + generated_name = name + ".generated.manifest" + # Need to join with the build_dir here as we're writing it during + # generation time, but we return the un-joined version because the build + # will occur in that directory. We only write the file if the contents + # have changed so that simply regenerating the project files doesn't + # cause a relink. + build_dir_generated_name = os.path.join(build_dir, generated_name) + gyp.common.EnsureDirExists(build_dir_generated_name) + f = gyp.common.WriteOnDiff(build_dir_generated_name) + f.write(generated_manifest_contents) + f.close() + manifest_files = [generated_name] + + if allow_isolation: + flags.append("/ALLOWISOLATION") + + manifest_files += self._GetAdditionalManifestFiles(config, gyp_to_build_path) + return flags, output_name, manifest_files + + def _GetAdditionalManifestFiles(self, config, gyp_to_build_path): + """Gets additional manifest files that are added to the default one generated by the linker.""" - files = self._Setting(('VCManifestTool', 'AdditionalManifestFiles'), config, - default=[]) - if isinstance(files, str): - files = files.split(';') - return [os.path.normpath( - gyp_to_build_path(self.ConvertVSMacros(f, config=config))) - for f in files] - - def IsUseLibraryDependencyInputs(self, config): - """Returns whether the target should be linked via Use Library Dependency + files = self._Setting( + ("VCManifestTool", "AdditionalManifestFiles"), config, default=[] + ) + if isinstance(files, str): + files = files.split(";") + return [ + os.path.normpath(gyp_to_build_path(self.ConvertVSMacros(f, config=config))) + for f in files + ] + + def IsUseLibraryDependencyInputs(self, config): + """Returns whether the target should be linked via Use Library Dependency Inputs (using component .objs of a given .lib).""" - config = self._TargetConfig(config) - uldi = self._Setting(('VCLinkerTool', 'UseLibraryDependencyInputs'), config) - return uldi == 'true' - - def IsEmbedManifest(self, config): - """Returns whether manifest should be linked into binary.""" - config = self._TargetConfig(config) - embed = self._Setting(('VCManifestTool', 'EmbedManifest'), config, - default='true') - return embed == 'true' - - def IsLinkIncremental(self, config): - """Returns whether the target should be linked incrementally.""" - config = self._TargetConfig(config) - link_inc = self._Setting(('VCLinkerTool', 'LinkIncremental'), config) - return link_inc != '1' - - def GetRcflags(self, config, gyp_to_ninja_path): - """Returns the flags that need to be added to invocations of the resource + config = self._TargetConfig(config) + uldi = self._Setting(("VCLinkerTool", "UseLibraryDependencyInputs"), config) + return uldi == "true" + + def IsEmbedManifest(self, config): + """Returns whether manifest should be linked into binary.""" + config = self._TargetConfig(config) + embed = self._Setting( + ("VCManifestTool", "EmbedManifest"), config, default="true" + ) + return embed == "true" + + def IsLinkIncremental(self, config): + """Returns whether the target should be linked incrementally.""" + config = self._TargetConfig(config) + link_inc = self._Setting(("VCLinkerTool", "LinkIncremental"), config) + return link_inc != "1" + + def GetRcflags(self, config, gyp_to_ninja_path): + """Returns the flags that need to be added to invocations of the resource compiler.""" - config = self._TargetConfig(config) - rcflags = [] - rc = self._GetWrapper(self, self.msvs_settings[config], - 'VCResourceCompilerTool', append=rcflags) - rc('AdditionalIncludeDirectories', map=gyp_to_ninja_path, prefix='/I') - rcflags.append('/I' + gyp_to_ninja_path('.')) - rc('PreprocessorDefinitions', prefix='/d') - # /l arg must be in hex without leading '0x' - rc('Culture', prefix='/l', map=lambda x: hex(int(x))[2:]) - return rcflags - - def BuildCygwinBashCommandLine(self, args, path_to_base): - """Build a command line that runs args via cygwin bash. We assume that all + config = self._TargetConfig(config) + rcflags = [] + rc = self._GetWrapper( + self, self.msvs_settings[config], "VCResourceCompilerTool", append=rcflags + ) + rc("AdditionalIncludeDirectories", map=gyp_to_ninja_path, prefix="/I") + rcflags.append("/I" + gyp_to_ninja_path(".")) + rc("PreprocessorDefinitions", prefix="/d") + # /l arg must be in hex without leading '0x' + rc("Culture", prefix="/l", map=lambda x: hex(int(x))[2:]) + return rcflags + + def BuildCygwinBashCommandLine(self, args, path_to_base): + """Build a command line that runs args via cygwin bash. We assume that all incoming paths are in Windows normpath'd form, so they need to be converted to posix style for the part of the command line that's passed to bash. We also have to do some Visual Studio macro emulation here because @@ -820,216 +923,246 @@ def BuildCygwinBashCommandLine(self, args, path_to_base): contain ninja variables cannot be fixed here (for example ${source}), so the outer generator needs to make sure that the paths that are written out are in posix style, if the command line will be used here.""" - cygwin_dir = os.path.normpath( - os.path.join(path_to_base, self.msvs_cygwin_dirs[0])) - cd = ('cd %s' % path_to_base).replace('\\', '/') - args = [a.replace('\\', '/').replace('"', '\\"') for a in args] - args = ["'%s'" % a.replace("'", "'\\''") for a in args] - bash_cmd = ' '.join(args) - cmd = ( - 'call "%s\\setup_env.bat" && set CYGWIN=nontsec && ' % cygwin_dir + - 'bash -c "%s ; %s"' % (cd, bash_cmd)) - return cmd - - def IsRuleRunUnderCygwin(self, rule): - """Determine if an action should be run under cygwin. If the variable is + cygwin_dir = os.path.normpath( + os.path.join(path_to_base, self.msvs_cygwin_dirs[0]) + ) + cd = ("cd %s" % path_to_base).replace("\\", "/") + args = [a.replace("\\", "/").replace('"', '\\"') for a in args] + args = ["'%s'" % a.replace("'", "'\\''") for a in args] + bash_cmd = " ".join(args) + cmd = ( + 'call "%s\\setup_env.bat" && set CYGWIN=nontsec && ' % cygwin_dir + + 'bash -c "%s ; %s"' % (cd, bash_cmd) + ) + return cmd + + def IsRuleRunUnderCygwin(self, rule): + """Determine if an action should be run under cygwin. If the variable is unset, or set to 1 we use cygwin.""" - return int(rule.get('msvs_cygwin_shell', - self.spec.get('msvs_cygwin_shell', 1))) != 0 - - def _HasExplicitRuleForExtension(self, spec, extension): - """Determine if there's an explicit rule for a particular extension.""" - for rule in spec.get('rules', []): - if rule['extension'] == extension: - return True - return False - - def _HasExplicitIdlActions(self, spec): - """Determine if an action should not run midl for .idl files.""" - return any([action.get('explicit_idl_action', 0) - for action in spec.get('actions', [])]) - - def HasExplicitIdlRulesOrActions(self, spec): - """Determine if there's an explicit rule or action for idl files. When + return ( + int(rule.get("msvs_cygwin_shell", self.spec.get("msvs_cygwin_shell", 1))) + != 0 + ) + + def _HasExplicitRuleForExtension(self, spec, extension): + """Determine if there's an explicit rule for a particular extension.""" + for rule in spec.get("rules", []): + if rule["extension"] == extension: + return True + return False + + def _HasExplicitIdlActions(self, spec): + """Determine if an action should not run midl for .idl files.""" + return any( + [action.get("explicit_idl_action", 0) for action in spec.get("actions", [])] + ) + + def HasExplicitIdlRulesOrActions(self, spec): + """Determine if there's an explicit rule or action for idl files. When there isn't we need to generate implicit rules to build MIDL .idl files.""" - return (self._HasExplicitRuleForExtension(spec, 'idl') or - self._HasExplicitIdlActions(spec)) + return self._HasExplicitRuleForExtension( + spec, "idl" + ) or self._HasExplicitIdlActions(spec) - def HasExplicitAsmRules(self, spec): - """Determine if there's an explicit rule for asm files. When there isn't we + def HasExplicitAsmRules(self, spec): + """Determine if there's an explicit rule for asm files. When there isn't we need to generate implicit rules to assemble .asm files.""" - return self._HasExplicitRuleForExtension(spec, 'asm') + return self._HasExplicitRuleForExtension(spec, "asm") - def GetIdlBuildData(self, source, config): - """Determine the implicit outputs for an idl file. Returns output + def GetIdlBuildData(self, source, config): + """Determine the implicit outputs for an idl file. Returns output directory, outputs, and variables and flags that are required.""" - config = self._TargetConfig(config) - midl_get = self._GetWrapper(self, self.msvs_settings[config], 'VCMIDLTool') - def midl(name, default=None): - return self.ConvertVSMacros(midl_get(name, default=default), - config=config) - tlb = midl('TypeLibraryName', default='${root}.tlb') - header = midl('HeaderFileName', default='${root}.h') - dlldata = midl('DLLDataFileName', default='dlldata.c') - iid = midl('InterfaceIdentifierFileName', default='${root}_i.c') - proxy = midl('ProxyFileName', default='${root}_p.c') - # Note that .tlb is not included in the outputs as it is not always - # generated depending on the content of the input idl file. - outdir = midl('OutputDirectory', default='') - output = [header, dlldata, iid, proxy] - variables = [('tlb', tlb), - ('h', header), - ('dlldata', dlldata), - ('iid', iid), - ('proxy', proxy)] - # TODO(scottmg): Are there configuration settings to set these flags? - target_platform = self.GetArch(config) - if target_platform == 'x86': - target_platform = 'win32' - flags = ['/char', 'signed', '/env', target_platform, '/Oicf'] - return outdir, output, variables, flags + config = self._TargetConfig(config) + midl_get = self._GetWrapper(self, self.msvs_settings[config], "VCMIDLTool") + + def midl(name, default=None): + return self.ConvertVSMacros(midl_get(name, default=default), config=config) + + tlb = midl("TypeLibraryName", default="${root}.tlb") + header = midl("HeaderFileName", default="${root}.h") + dlldata = midl("DLLDataFileName", default="dlldata.c") + iid = midl("InterfaceIdentifierFileName", default="${root}_i.c") + proxy = midl("ProxyFileName", default="${root}_p.c") + # Note that .tlb is not included in the outputs as it is not always + # generated depending on the content of the input idl file. + outdir = midl("OutputDirectory", default="") + output = [header, dlldata, iid, proxy] + variables = [ + ("tlb", tlb), + ("h", header), + ("dlldata", dlldata), + ("iid", iid), + ("proxy", proxy), + ] + # TODO(scottmg): Are there configuration settings to set these flags? + target_platform = self.GetArch(config) + if target_platform == "x86": + target_platform = "win32" + flags = ["/char", "signed", "/env", target_platform, "/Oicf"] + return outdir, output, variables, flags def _LanguageMatchesForPch(source_ext, pch_source_ext): - c_exts = ('.c',) - cc_exts = ('.cc', '.cxx', '.cpp') - return ((source_ext in c_exts and pch_source_ext in c_exts) or - (source_ext in cc_exts and pch_source_ext in cc_exts)) + c_exts = (".c",) + cc_exts = (".cc", ".cxx", ".cpp") + return (source_ext in c_exts and pch_source_ext in c_exts) or ( + source_ext in cc_exts and pch_source_ext in cc_exts + ) class PrecompiledHeader(object): - """Helper to generate dependencies and build rules to handle generation of + """Helper to generate dependencies and build rules to handle generation of precompiled headers. Interface matches the GCH handler in xcode_emulation.py. """ - def __init__( - self, settings, config, gyp_to_build_path, gyp_to_unique_output, obj_ext): - self.settings = settings - self.config = config - pch_source = self.settings.msvs_precompiled_source[self.config] - self.pch_source = gyp_to_build_path(pch_source) - filename, _ = os.path.splitext(pch_source) - self.output_obj = gyp_to_unique_output(filename + obj_ext).lower() - - def _PchHeader(self): - """Get the header that will appear in an #include line for all source + + def __init__( + self, settings, config, gyp_to_build_path, gyp_to_unique_output, obj_ext + ): + self.settings = settings + self.config = config + pch_source = self.settings.msvs_precompiled_source[self.config] + self.pch_source = gyp_to_build_path(pch_source) + filename, _ = os.path.splitext(pch_source) + self.output_obj = gyp_to_unique_output(filename + obj_ext).lower() + + def _PchHeader(self): + """Get the header that will appear in an #include line for all source files.""" - return self.settings.msvs_precompiled_header[self.config] + return self.settings.msvs_precompiled_header[self.config] - def GetObjDependencies(self, sources, objs, arch): - """Given a list of sources files and the corresponding object files, + def GetObjDependencies(self, sources, objs, arch): + """Given a list of sources files and the corresponding object files, returns a list of the pch files that should be depended upon. The additional wrapping in the return value is for interface compatibility with make.py on Mac, and xcode_emulation.py.""" - assert arch is None - if not self._PchHeader(): - return [] - pch_ext = os.path.splitext(self.pch_source)[1] - for source in sources: - if _LanguageMatchesForPch(os.path.splitext(source)[1], pch_ext): - return [(None, None, self.output_obj)] - return [] - - def GetPchBuildCommands(self, arch): - """Not used on Windows as there are no additional build steps required + assert arch is None + if not self._PchHeader(): + return [] + pch_ext = os.path.splitext(self.pch_source)[1] + for source in sources: + if _LanguageMatchesForPch(os.path.splitext(source)[1], pch_ext): + return [(None, None, self.output_obj)] + return [] + + def GetPchBuildCommands(self, arch): + """Not used on Windows as there are no additional build steps required (instead, existing steps are modified in GetFlagsModifications below).""" - return [] + return [] - def GetFlagsModifications(self, input, output, implicit, command, - cflags_c, cflags_cc, expand_special): - """Get the modified cflags and implicit dependencies that should be used + def GetFlagsModifications( + self, input, output, implicit, command, cflags_c, cflags_cc, expand_special + ): + """Get the modified cflags and implicit dependencies that should be used for the pch compilation step.""" - if input == self.pch_source: - pch_output = ['/Yc' + self._PchHeader()] - if command == 'cxx': - return ([('cflags_cc', map(expand_special, cflags_cc + pch_output))], - self.output_obj, []) - elif command == 'cc': - return ([('cflags_c', map(expand_special, cflags_c + pch_output))], - self.output_obj, []) - return [], output, implicit + if input == self.pch_source: + pch_output = ["/Yc" + self._PchHeader()] + if command == "cxx": + return ( + [("cflags_cc", map(expand_special, cflags_cc + pch_output))], + self.output_obj, + [], + ) + elif command == "cc": + return ( + [("cflags_c", map(expand_special, cflags_c + pch_output))], + self.output_obj, + [], + ) + return [], output, implicit vs_version = None + + def GetVSVersion(generator_flags): - global vs_version - if not vs_version: - vs_version = gyp.MSVSVersion.SelectVisualStudioVersion( - generator_flags.get('msvs_version', 'auto'), - allow_fallback=False) - return vs_version + global vs_version + if not vs_version: + vs_version = gyp.MSVSVersion.SelectVisualStudioVersion( + generator_flags.get("msvs_version", "auto"), allow_fallback=False + ) + return vs_version + def _GetVsvarsSetupArgs(generator_flags, arch): - vs = GetVSVersion(generator_flags) - return vs.SetupScript() + vs = GetVSVersion(generator_flags) + return vs.SetupScript() + def ExpandMacros(string, expansions): - """Expand $(Variable) per expansions dict. See MsvsSettings.GetVSMacroEnv + """Expand $(Variable) per expansions dict. See MsvsSettings.GetVSMacroEnv for the canonical way to retrieve a suitable dict.""" - if '$' in string: - for old, new in expansions.items(): - assert '$(' not in new, new - string = string.replace(old, new) - return string + if "$" in string: + for old, new in expansions.items(): + assert "$(" not in new, new + string = string.replace(old, new) + return string + def _ExtractImportantEnvironment(output_of_set): - """Extracts environment variables required for the toolchain to run from + """Extracts environment variables required for the toolchain to run from a textual dump output by the cmd.exe 'set' command.""" - envvars_to_save = ( - 'goma_.*', # TODO(scottmg): This is ugly, but needed for goma. - 'include', - 'lib', - 'libpath', - 'path', - 'pathext', - 'systemroot', - 'temp', - 'tmp', - ) - env = {} - # This occasionally happens and leads to misleading SYSTEMROOT error messages - # if not caught here. - if output_of_set.count('=') == 0: - raise Exception('Invalid output_of_set. Value is:\n%s' % output_of_set) - for line in output_of_set.splitlines(): - for envvar in envvars_to_save: - if re.match(envvar + '=', line.lower()): - var, setting = line.split('=', 1) - if envvar == 'path': - # Our own rules (for running gyp-win-tool) and other actions in - # Chromium rely on python being in the path. Add the path to this - # python here so that if it's not in the path when ninja is run - # later, python will still be found. - setting = os.path.dirname(sys.executable) + os.pathsep + setting - env[var.upper()] = setting - break - for required in ('SYSTEMROOT', 'TEMP', 'TMP'): - if required not in env: - raise Exception('Environment variable "%s" ' - 'required to be set to valid path' % required) - return env + envvars_to_save = ( + "goma_.*", # TODO(scottmg): This is ugly, but needed for goma. + "include", + "lib", + "libpath", + "path", + "pathext", + "systemroot", + "temp", + "tmp", + ) + env = {} + # This occasionally happens and leads to misleading SYSTEMROOT error messages + # if not caught here. + if output_of_set.count("=") == 0: + raise Exception("Invalid output_of_set. Value is:\n%s" % output_of_set) + for line in output_of_set.splitlines(): + for envvar in envvars_to_save: + if re.match(envvar + "=", line.lower()): + var, setting = line.split("=", 1) + if envvar == "path": + # Our own rules (for running gyp-win-tool) and other actions in + # Chromium rely on python being in the path. Add the path to this + # python here so that if it's not in the path when ninja is run + # later, python will still be found. + setting = os.path.dirname(sys.executable) + os.pathsep + setting + env[var.upper()] = setting + break + for required in ("SYSTEMROOT", "TEMP", "TMP"): + if required not in env: + raise Exception( + 'Environment variable "%s" ' + "required to be set to valid path" % required + ) + return env + def _FormatAsEnvironmentBlock(envvar_dict): - """Format as an 'environment block' directly suitable for CreateProcess. + """Format as an 'environment block' directly suitable for CreateProcess. Briefly this is a list of key=value\0, terminated by an additional \0. See CreateProcess documentation for more details.""" - block = '' - nul = '\0' - for key, value in envvar_dict.items(): - block += key + '=' + value + nul - block += nul - return block + block = "" + nul = "\0" + for key, value in envvar_dict.items(): + block += key + "=" + value + nul + block += nul + return block + def _ExtractCLPath(output_of_where): - """Gets the path to cl.exe based on the output of calling the environment + """Gets the path to cl.exe based on the output of calling the environment setup batch file, followed by the equivalent of `where`.""" - # Take the first line, as that's the first found in the PATH. - for line in output_of_where.strip().splitlines(): - if line.startswith('LOC:'): - return line[len('LOC:'):].strip() - -def GenerateEnvironmentFiles(toplevel_build_dir, generator_flags, - system_includes, open_out): - """It's not sufficient to have the absolute path to the compiler, linker, + # Take the first line, as that's the first found in the PATH. + for line in output_of_where.strip().splitlines(): + if line.startswith("LOC:"): + return line[len("LOC:") :].strip() + + +def GenerateEnvironmentFiles( + toplevel_build_dir, generator_flags, system_includes, open_out +): + """It's not sufficient to have the absolute path to the compiler, linker, etc. on Windows, as those tools rely on .dlls being in the PATH. We also need to support both x86 and x64 compilers within the same build (to support msvs_target_platform hackery). Different architectures require a different @@ -1043,80 +1176,86 @@ def GenerateEnvironmentFiles(toplevel_build_dir, generator_flags, meet your requirement (e.g. for custom toolchains), you can pass "-G ninja_use_custom_environment_files" to the gyp to suppress file generation and use custom environment files prepared by yourself.""" - archs = ('x86', 'x64') - if generator_flags.get('ninja_use_custom_environment_files', 0): + archs = ("x86", "x64") + if generator_flags.get("ninja_use_custom_environment_files", 0): + cl_paths = {} + for arch in archs: + cl_paths[arch] = "cl.exe" + return cl_paths + vs = GetVSVersion(generator_flags) cl_paths = {} for arch in archs: - cl_paths[arch] = 'cl.exe' + # Extract environment variables for subprocesses. + args = vs.SetupScript(arch) + args.extend(("&&", "set")) + popen = subprocess.Popen( + args, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT + ) + variables, _ = popen.communicate() + if PY3: + variables = variables.decode("utf-8") + if popen.returncode != 0: + raise Exception('"%s" failed with error %d' % (args, popen.returncode)) + env = _ExtractImportantEnvironment(variables) + + # Inject system includes from gyp files into INCLUDE. + if system_includes: + system_includes = system_includes | OrderedSet( + env.get("INCLUDE", "").split(";") + ) + env["INCLUDE"] = ";".join(system_includes) + + env_block = _FormatAsEnvironmentBlock(env) + f = open_out(os.path.join(toplevel_build_dir, "environment." + arch), "w") + f.write(env_block) + f.close() + + # Find cl.exe location for this architecture. + args = vs.SetupScript(arch) + args.extend( + ("&&", "for", "%i", "in", "(cl.exe)", "do", "@echo", "LOC:%~$PATH:i") + ) + popen = subprocess.Popen(args, shell=True, stdout=subprocess.PIPE) + output, _ = popen.communicate() + if PY3: + output = output.decode("utf-8") + cl_paths[arch] = _ExtractCLPath(output) return cl_paths - vs = GetVSVersion(generator_flags) - cl_paths = {} - for arch in archs: - # Extract environment variables for subprocesses. - args = vs.SetupScript(arch) - args.extend(('&&', 'set')) - popen = subprocess.Popen( - args, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) - variables, _ = popen.communicate() - if PY3: - variables = variables.decode('utf-8') - if popen.returncode != 0: - raise Exception('"%s" failed with error %d' % (args, popen.returncode)) - env = _ExtractImportantEnvironment(variables) - - # Inject system includes from gyp files into INCLUDE. - if system_includes: - system_includes = system_includes | OrderedSet( - env.get('INCLUDE', '').split(';')) - env['INCLUDE'] = ';'.join(system_includes) - - env_block = _FormatAsEnvironmentBlock(env) - f = open_out(os.path.join(toplevel_build_dir, 'environment.' + arch), 'wb') - f.write(env_block) - f.close() - - # Find cl.exe location for this architecture. - args = vs.SetupScript(arch) - args.extend(('&&', - 'for', '%i', 'in', '(cl.exe)', 'do', '@echo', 'LOC:%~$PATH:i')) - popen = subprocess.Popen(args, shell=True, stdout=subprocess.PIPE) - output, _ = popen.communicate() - if PY3: - output = output.decode('utf-8') - cl_paths[arch] = _ExtractCLPath(output) - return cl_paths + def VerifyMissingSources(sources, build_dir, generator_flags, gyp_to_ninja): - """Emulate behavior of msvs_error_on_missing_sources present in the msvs + """Emulate behavior of msvs_error_on_missing_sources present in the msvs generator: Check that all regular source files, i.e. not created at run time, exist on disk. Missing files cause needless recompilation when building via VS, and we want this check to match for people/bots that build using ninja, so they're not surprised when the VS build fails.""" - if int(generator_flags.get('msvs_error_on_missing_sources', 0)): - no_specials = filter(lambda x: '$' not in x, sources) - relative = [os.path.join(build_dir, gyp_to_ninja(s)) for s in no_specials] - missing = filter(lambda x: not os.path.exists(x), relative) - if missing: - # They'll look like out\Release\..\..\stuff\things.cc, so normalize the - # path for a slightly less crazy looking output. - cleaned_up = [os.path.normpath(x) for x in missing] - raise Exception('Missing input files:\n%s' % '\n'.join(cleaned_up)) + if int(generator_flags.get("msvs_error_on_missing_sources", 0)): + no_specials = filter(lambda x: "$" not in x, sources) + relative = [os.path.join(build_dir, gyp_to_ninja(s)) for s in no_specials] + missing = [x for x in relative if not os.path.exists(x)] + if missing: + # They'll look like out\Release\..\..\stuff\things.cc, so normalize the + # path for a slightly less crazy looking output. + cleaned_up = [os.path.normpath(x) for x in missing] + raise Exception("Missing input files:\n%s" % "\n".join(cleaned_up)) + # Sets some values in default_variables, which are required for many # generators, run on Windows. def CalculateCommonVariables(default_variables, params): - generator_flags = params.get('generator_flags', {}) - - # Set a variable so conditions can be based on msvs_version. - msvs_version = gyp.msvs_emulation.GetVSVersion(generator_flags) - default_variables['MSVS_VERSION'] = msvs_version.ShortName() - - # To determine processor word size on Windows, in addition to checking - # PROCESSOR_ARCHITECTURE (which reflects the word size of the current - # process), it is also necessary to check PROCESSOR_ARCHITEW6432 (which - # contains the actual word size of the system when running thru WOW64). - if ('64' in os.environ.get('PROCESSOR_ARCHITECTURE', '') or - '64' in os.environ.get('PROCESSOR_ARCHITEW6432', '')): - default_variables['MSVS_OS_BITS'] = 64 - else: - default_variables['MSVS_OS_BITS'] = 32 + generator_flags = params.get("generator_flags", {}) + + # Set a variable so conditions can be based on msvs_version. + msvs_version = gyp.msvs_emulation.GetVSVersion(generator_flags) + default_variables["MSVS_VERSION"] = msvs_version.ShortName() + + # To determine processor word size on Windows, in addition to checking + # PROCESSOR_ARCHITECTURE (which reflects the word size of the current + # process), it is also necessary to check PROCESSOR_ARCHITEW6432 (which + # contains the actual word size of the system when running thru WOW64). + if "64" in os.environ.get("PROCESSOR_ARCHITECTURE", "") or "64" in os.environ.get( + "PROCESSOR_ARCHITEW6432", "" + ): + default_variables["MSVS_OS_BITS"] = 64 + else: + default_variables["MSVS_OS_BITS"] = 32 diff --git a/gyp/pylib/gyp/ninja_syntax.py b/gyp/pylib/gyp/ninja_syntax.py index d2948f06c0..1421235808 100644 --- a/gyp/pylib/gyp/ninja_syntax.py +++ b/gyp/pylib/gyp/ninja_syntax.py @@ -10,10 +10,11 @@ """ import textwrap -import re + def escape_path(word): - return word.replace('$ ','$$ ').replace(' ','$ ').replace(':', '$:') + return word.replace("$ ", "$$ ").replace(" ", "$ ").replace(":", "$:") + class Writer(object): def __init__(self, output, width=78): @@ -21,47 +22,58 @@ def __init__(self, output, width=78): self.width = width def newline(self): - self.output.write('\n') + self.output.write("\n") def comment(self, text): for line in textwrap.wrap(text, self.width - 2): - self.output.write('# ' + line + '\n') + self.output.write("# " + line + "\n") def variable(self, key, value, indent=0): if value is None: return if isinstance(value, list): - value = ' '.join(filter(None, value)) # Filter out empty strings. - self._line('%s = %s' % (key, value), indent) + value = " ".join(filter(None, value)) # Filter out empty strings. + self._line("%s = %s" % (key, value), indent) def pool(self, name, depth): - self._line('pool %s' % name) - self.variable('depth', depth, indent=1) - - def rule(self, name, command, description=None, depfile=None, - generator=False, pool=None, restat=False, rspfile=None, - rspfile_content=None, deps=None): - self._line('rule %s' % name) - self.variable('command', command, indent=1) + self._line("pool %s" % name) + self.variable("depth", depth, indent=1) + + def rule( + self, + name, + command, + description=None, + depfile=None, + generator=False, + pool=None, + restat=False, + rspfile=None, + rspfile_content=None, + deps=None, + ): + self._line("rule %s" % name) + self.variable("command", command, indent=1) if description: - self.variable('description', description, indent=1) + self.variable("description", description, indent=1) if depfile: - self.variable('depfile', depfile, indent=1) + self.variable("depfile", depfile, indent=1) if generator: - self.variable('generator', '1', indent=1) + self.variable("generator", "1", indent=1) if pool: - self.variable('pool', pool, indent=1) + self.variable("pool", pool, indent=1) if restat: - self.variable('restat', '1', indent=1) + self.variable("restat", "1", indent=1) if rspfile: - self.variable('rspfile', rspfile, indent=1) + self.variable("rspfile", rspfile, indent=1) if rspfile_content: - self.variable('rspfile_content', rspfile_content, indent=1) + self.variable("rspfile_content", rspfile_content, indent=1) if deps: - self.variable('deps', deps, indent=1) + self.variable("deps", deps, indent=1) - def build(self, outputs, rule, inputs=None, implicit=None, order_only=None, - variables=None): + def build( + self, outputs, rule, inputs=None, implicit=None, order_only=None, variables=None + ): outputs = self._as_list(outputs) all_inputs = self._as_list(inputs)[:] out_outputs = list(map(escape_path, outputs)) @@ -69,15 +81,16 @@ def build(self, outputs, rule, inputs=None, implicit=None, order_only=None, if implicit: implicit = map(escape_path, self._as_list(implicit)) - all_inputs.append('|') + all_inputs.append("|") all_inputs.extend(implicit) if order_only: order_only = map(escape_path, self._as_list(order_only)) - all_inputs.append('||') + all_inputs.append("||") all_inputs.extend(order_only) - self._line('build %s: %s' % (' '.join(out_outputs), - ' '.join([rule] + all_inputs))) + self._line( + "build %s: %s" % (" ".join(out_outputs), " ".join([rule] + all_inputs)) + ) if variables: if isinstance(variables, dict): @@ -91,58 +104,59 @@ def build(self, outputs, rule, inputs=None, implicit=None, order_only=None, return outputs def include(self, path): - self._line('include %s' % path) + self._line("include %s" % path) def subninja(self, path): - self._line('subninja %s' % path) + self._line("subninja %s" % path) def default(self, paths): - self._line('default %s' % ' '.join(self._as_list(paths))) + self._line("default %s" % " ".join(self._as_list(paths))) def _count_dollars_before_index(self, s, i): - """Returns the number of '$' characters right in front of s[i].""" - dollar_count = 0 - dollar_index = i - 1 - while dollar_index > 0 and s[dollar_index] == '$': - dollar_count += 1 - dollar_index -= 1 - return dollar_count + """Returns the number of '$' characters right in front of s[i].""" + dollar_count = 0 + dollar_index = i - 1 + while dollar_index > 0 and s[dollar_index] == "$": + dollar_count += 1 + dollar_index -= 1 + return dollar_count def _line(self, text, indent=0): """Write 'text' word-wrapped at self.width characters.""" - leading_space = ' ' * indent + leading_space = " " * indent while len(leading_space) + len(text) > self.width: # The text is too wide; wrap if possible. # Find the rightmost space that would obey our width constraint and # that's not an escaped space. - available_space = self.width - len(leading_space) - len(' $') + available_space = self.width - len(leading_space) - len(" $") space = available_space while True: - space = text.rfind(' ', 0, space) - if space < 0 or \ - self._count_dollars_before_index(text, space) % 2 == 0: - break + space = text.rfind(" ", 0, space) + if space < 0 or self._count_dollars_before_index(text, space) % 2 == 0: + break if space < 0: # No such space; just use the first unescaped space we can find. space = available_space - 1 while True: - space = text.find(' ', space + 1) - if space < 0 or \ - self._count_dollars_before_index(text, space) % 2 == 0: - break + space = text.find(" ", space + 1) + if ( + space < 0 + or self._count_dollars_before_index(text, space) % 2 == 0 + ): + break if space < 0: # Give up on breaking. break - self.output.write(leading_space + text[0:space] + ' $\n') - text = text[space+1:] + self.output.write(leading_space + text[0:space] + " $\n") + text = text[space + 1 :] # Subsequent lines are continuations, so indent them. - leading_space = ' ' * (indent+2) + leading_space = " " * (indent + 2) - self.output.write(leading_space + text + '\n') + self.output.write(leading_space + text + "\n") def _as_list(self, input): if input is None: @@ -155,6 +169,6 @@ def _as_list(self, input): def escape(string): """Escape a string such that it can be embedded into a Ninja file without further interpretation.""" - assert '\n' not in string, 'Ninja syntax does not allow newlines' + assert "\n" not in string, "Ninja syntax does not allow newlines" # We only have one special metacharacter: '$'. - return string.replace('$', '$$') + return string.replace("$", "$$") diff --git a/gyp/pylib/gyp/simple_copy.py b/gyp/pylib/gyp/simple_copy.py index 94a6f17dab..e01106f9c4 100644 --- a/gyp/pylib/gyp/simple_copy.py +++ b/gyp/pylib/gyp/simple_copy.py @@ -7,44 +7,58 @@ because gyp copies so large structure that small copy overhead ends up taking seconds in a project the size of Chromium.""" + class Error(Exception): - pass + pass + __all__ = ["Error", "deepcopy"] + def deepcopy(x): - """Deep copy operation on gyp objects such as strings, ints, dicts + """Deep copy operation on gyp objects such as strings, ints, dicts and lists. More than twice as fast as copy.deepcopy but much less generic.""" - try: - return _deepcopy_dispatch[type(x)](x) - except KeyError: - raise Error('Unsupported type %s for deepcopy. Use copy.deepcopy ' + - 'or expand simple_copy support.' % type(x)) + try: + return _deepcopy_dispatch[type(x)](x) + except KeyError: + raise Error( + "Unsupported type %s for deepcopy. Use copy.deepcopy " + + "or expand simple_copy support." % type(x) + ) + _deepcopy_dispatch = d = {} + def _deepcopy_atomic(x): - return x + return x + try: - types = bool, float, int, str, type, type(None), long, unicode + types = bool, float, int, str, type, type(None), long, unicode except NameError: # Python 3 - types = bool, float, int, str, type, type(None) + types = bool, float, int, str, type, type(None) for x in types: - d[x] = _deepcopy_atomic + d[x] = _deepcopy_atomic + def _deepcopy_list(x): - return [deepcopy(a) for a in x] + return [deepcopy(a) for a in x] + + d[list] = _deepcopy_list + def _deepcopy_dict(x): - y = {} - for key, value in x.items(): - y[deepcopy(key)] = deepcopy(value) - return y + y = {} + for key, value in x.items(): + y[deepcopy(key)] = deepcopy(value) + return y + + d[dict] = _deepcopy_dict del d diff --git a/gyp/pylib/gyp/win_tool.py b/gyp/pylib/gyp/win_tool.py index cfdacb0d7c..758e9f5c45 100755 --- a/gyp/pylib/gyp/win_tool.py +++ b/gyp/pylib/gyp/win_tool.py @@ -24,312 +24,362 @@ # A regex matching an argument corresponding to the output filename passed to # link.exe. -_LINK_EXE_OUT_ARG = re.compile('/OUT:(?P.+)$', re.IGNORECASE) +_LINK_EXE_OUT_ARG = re.compile("/OUT:(?P.+)$", re.IGNORECASE) + def main(args): - executor = WinTool() - exit_code = executor.Dispatch(args) - if exit_code is not None: - sys.exit(exit_code) + executor = WinTool() + exit_code = executor.Dispatch(args) + if exit_code is not None: + sys.exit(exit_code) class WinTool(object): - """This class performs all the Windows tooling steps. The methods can either + """This class performs all the Windows tooling steps. The methods can either be executed directly, or dispatched from an argument list.""" - def _UseSeparateMspdbsrv(self, env, args): - """Allows to use a unique instance of mspdbsrv.exe per linker instead of a + def _UseSeparateMspdbsrv(self, env, args): + """Allows to use a unique instance of mspdbsrv.exe per linker instead of a shared one.""" - if len(args) < 1: - raise Exception("Not enough arguments") - - if args[0] != 'link.exe': - return - - # Use the output filename passed to the linker to generate an endpoint name - # for mspdbsrv.exe. - endpoint_name = None - for arg in args: - m = _LINK_EXE_OUT_ARG.match(arg) - if m: - endpoint_name = re.sub(r'\W+', '', - '%s_%d' % (m.group('out'), os.getpid())) - break - - if endpoint_name is None: - return - - # Adds the appropriate environment variable. This will be read by link.exe - # to know which instance of mspdbsrv.exe it should connect to (if it's - # not set then the default endpoint is used). - env['_MSPDBSRV_ENDPOINT_'] = endpoint_name - - def Dispatch(self, args): - """Dispatches a string command to a method.""" - if len(args) < 1: - raise Exception("Not enough arguments") - - method = "Exec%s" % self._CommandifyName(args[0]) - return getattr(self, method)(*args[1:]) - - def _CommandifyName(self, name_string): - """Transforms a tool name like recursive-mirror to RecursiveMirror.""" - return name_string.title().replace('-', '') - - def _GetEnv(self, arch): - """Gets the saved environment from a file for a given architecture.""" - # The environment is saved as an "environment block" (see CreateProcess - # and msvs_emulation for details). We convert to a dict here. - # Drop last 2 NULs, one for list terminator, one for trailing vs. separator. - pairs = open(arch).read()[:-2].split('\0') - kvs = [item.split('=', 1) for item in pairs] - return dict(kvs) - - def ExecStamp(self, path): - """Simple stamp command.""" - open(path, 'w').close() - - def ExecRecursiveMirror(self, source, dest): - """Emulation of rm -rf out && cp -af in out.""" - if os.path.exists(dest): - if os.path.isdir(dest): - def _on_error(fn, path, excinfo): - # The operation failed, possibly because the file is set to - # read-only. If that's why, make it writable and try the op again. - if not os.access(path, os.W_OK): - os.chmod(path, stat.S_IWRITE) - fn(path) - shutil.rmtree(dest, onerror=_on_error) - else: - if not os.access(dest, os.W_OK): - # Attempt to make the file writable before deleting it. - os.chmod(dest, stat.S_IWRITE) - os.unlink(dest) - - if os.path.isdir(source): - shutil.copytree(source, dest) - else: - shutil.copy2(source, dest) - - def ExecLinkWrapper(self, arch, use_separate_mspdbsrv, *args): - """Filter diagnostic output from link that looks like: + if len(args) < 1: + raise Exception("Not enough arguments") + + if args[0] != "link.exe": + return + + # Use the output filename passed to the linker to generate an endpoint name + # for mspdbsrv.exe. + endpoint_name = None + for arg in args: + m = _LINK_EXE_OUT_ARG.match(arg) + if m: + endpoint_name = re.sub( + r"\W+", "", "%s_%d" % (m.group("out"), os.getpid()) + ) + break + + if endpoint_name is None: + return + + # Adds the appropriate environment variable. This will be read by link.exe + # to know which instance of mspdbsrv.exe it should connect to (if it's + # not set then the default endpoint is used). + env["_MSPDBSRV_ENDPOINT_"] = endpoint_name + + def Dispatch(self, args): + """Dispatches a string command to a method.""" + if len(args) < 1: + raise Exception("Not enough arguments") + + method = "Exec%s" % self._CommandifyName(args[0]) + return getattr(self, method)(*args[1:]) + + def _CommandifyName(self, name_string): + """Transforms a tool name like recursive-mirror to RecursiveMirror.""" + return name_string.title().replace("-", "") + + def _GetEnv(self, arch): + """Gets the saved environment from a file for a given architecture.""" + # The environment is saved as an "environment block" (see CreateProcess + # and msvs_emulation for details). We convert to a dict here. + # Drop last 2 NULs, one for list terminator, one for trailing vs. separator. + pairs = open(arch).read()[:-2].split("\0") + kvs = [item.split("=", 1) for item in pairs] + return dict(kvs) + + def ExecStamp(self, path): + """Simple stamp command.""" + open(path, "w").close() + + def ExecRecursiveMirror(self, source, dest): + """Emulation of rm -rf out && cp -af in out.""" + if os.path.exists(dest): + if os.path.isdir(dest): + + def _on_error(fn, path, excinfo): + # The operation failed, possibly because the file is set to + # read-only. If that's why, make it writable and try the op again. + if not os.access(path, os.W_OK): + os.chmod(path, stat.S_IWRITE) + fn(path) + + shutil.rmtree(dest, onerror=_on_error) + else: + if not os.access(dest, os.W_OK): + # Attempt to make the file writable before deleting it. + os.chmod(dest, stat.S_IWRITE) + os.unlink(dest) + + if os.path.isdir(source): + shutil.copytree(source, dest) + else: + shutil.copy2(source, dest) + + def ExecLinkWrapper(self, arch, use_separate_mspdbsrv, *args): + """Filter diagnostic output from link that looks like: ' Creating library ui.dll.lib and object ui.dll.exp' This happens when there are exports from the dll or exe. """ - env = self._GetEnv(arch) - if use_separate_mspdbsrv == 'True': - self._UseSeparateMspdbsrv(env, args) - if sys.platform == 'win32': - args = list(args) # *args is a tuple by default, which is read-only. - args[0] = args[0].replace('/', '\\') - # https://docs.python.org/2/library/subprocess.html: - # "On Unix with shell=True [...] if args is a sequence, the first item - # specifies the command string, and any additional items will be treated as - # additional arguments to the shell itself. That is to say, Popen does the - # equivalent of: - # Popen(['/bin/sh', '-c', args[0], args[1], ...])" - # For that reason, since going through the shell doesn't seem necessary on - # non-Windows don't do that there. - link = subprocess.Popen(args, shell=sys.platform == 'win32', env=env, - stdout=subprocess.PIPE, stderr=subprocess.STDOUT) - out, _ = link.communicate() - if PY3: - out = out.decode('utf-8') - for line in out.splitlines(): - if (not line.startswith(' Creating library ') and - not line.startswith('Generating code') and - not line.startswith('Finished generating code')): - print(line) - return link.returncode - - def ExecLinkWithManifests(self, arch, embed_manifest, out, ldcmd, resname, - mt, rc, intermediate_manifest, *manifests): - """A wrapper for handling creating a manifest resource and then executing + env = self._GetEnv(arch) + if use_separate_mspdbsrv == "True": + self._UseSeparateMspdbsrv(env, args) + if sys.platform == "win32": + args = list(args) # *args is a tuple by default, which is read-only. + args[0] = args[0].replace("/", "\\") + # https://docs.python.org/2/library/subprocess.html: + # "On Unix with shell=True [...] if args is a sequence, the first item + # specifies the command string, and any additional items will be treated as + # additional arguments to the shell itself. That is to say, Popen does the + # equivalent of: + # Popen(['/bin/sh', '-c', args[0], args[1], ...])" + # For that reason, since going through the shell doesn't seem necessary on + # non-Windows don't do that there. + link = subprocess.Popen( + args, + shell=sys.platform == "win32", + env=env, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + ) + out, _ = link.communicate() + if PY3: + out = out.decode("utf-8") + for line in out.splitlines(): + if ( + not line.startswith(" Creating library ") + and not line.startswith("Generating code") + and not line.startswith("Finished generating code") + ): + print(line) + return link.returncode + + def ExecLinkWithManifests( + self, + arch, + embed_manifest, + out, + ldcmd, + resname, + mt, + rc, + intermediate_manifest, + *manifests + ): + """A wrapper for handling creating a manifest resource and then executing a link command.""" - # The 'normal' way to do manifests is to have link generate a manifest - # based on gathering dependencies from the object files, then merge that - # manifest with other manifests supplied as sources, convert the merged - # manifest to a resource, and then *relink*, including the compiled - # version of the manifest resource. This breaks incremental linking, and - # is generally overly complicated. Instead, we merge all the manifests - # provided (along with one that includes what would normally be in the - # linker-generated one, see msvs_emulation.py), and include that into the - # first and only link. We still tell link to generate a manifest, but we - # only use that to assert that our simpler process did not miss anything. - variables = { - 'python': sys.executable, - 'arch': arch, - 'out': out, - 'ldcmd': ldcmd, - 'resname': resname, - 'mt': mt, - 'rc': rc, - 'intermediate_manifest': intermediate_manifest, - 'manifests': ' '.join(manifests), - } - add_to_ld = '' - if manifests: - subprocess.check_call( - '%(python)s gyp-win-tool manifest-wrapper %(arch)s %(mt)s -nologo ' - '-manifest %(manifests)s -out:%(out)s.manifest' % variables) - if embed_manifest == 'True': - subprocess.check_call( - '%(python)s gyp-win-tool manifest-to-rc %(arch)s %(out)s.manifest' - ' %(out)s.manifest.rc %(resname)s' % variables) - subprocess.check_call( - '%(python)s gyp-win-tool rc-wrapper %(arch)s %(rc)s ' - '%(out)s.manifest.rc' % variables) - add_to_ld = ' %(out)s.manifest.res' % variables - subprocess.check_call(ldcmd + add_to_ld) - - # Run mt.exe on the theoretically complete manifest we generated, merging - # it with the one the linker generated to confirm that the linker - # generated one does not add anything. This is strictly unnecessary for - # correctness, it's only to verify that e.g. /MANIFESTDEPENDENCY was not - # used in a #pragma comment. - if manifests: - # Merge the intermediate one with ours to .assert.manifest, then check - # that .assert.manifest is identical to ours. - subprocess.check_call( - '%(python)s gyp-win-tool manifest-wrapper %(arch)s %(mt)s -nologo ' - '-manifest %(out)s.manifest %(intermediate_manifest)s ' - '-out:%(out)s.assert.manifest' % variables) - assert_manifest = '%(out)s.assert.manifest' % variables - our_manifest = '%(out)s.manifest' % variables - # Load and normalize the manifests. mt.exe sometimes removes whitespace, - # and sometimes doesn't unfortunately. - with open(our_manifest, 'rb') as our_f: - with open(assert_manifest, 'rb') as assert_f: - our_data = our_f.read().translate(None, string.whitespace) - assert_data = assert_f.read().translate(None, string.whitespace) - if our_data != assert_data: - os.unlink(out) - def dump(filename): - sys.stderr.write('%s\n-----\n' % filename) - with open(filename, 'rb') as f: - sys.stderr.write(f.read() + '\n-----\n') - dump(intermediate_manifest) - dump(our_manifest) - dump(assert_manifest) - sys.stderr.write( - 'Linker generated manifest "%s" added to final manifest "%s" ' - '(result in "%s"). ' - 'Were /MANIFEST switches used in #pragma statements? ' % ( - intermediate_manifest, our_manifest, assert_manifest)) - return 1 - - def ExecManifestWrapper(self, arch, *args): - """Run manifest tool with environment set. Strip out undesirable warning + # The 'normal' way to do manifests is to have link generate a manifest + # based on gathering dependencies from the object files, then merge that + # manifest with other manifests supplied as sources, convert the merged + # manifest to a resource, and then *relink*, including the compiled + # version of the manifest resource. This breaks incremental linking, and + # is generally overly complicated. Instead, we merge all the manifests + # provided (along with one that includes what would normally be in the + # linker-generated one, see msvs_emulation.py), and include that into the + # first and only link. We still tell link to generate a manifest, but we + # only use that to assert that our simpler process did not miss anything. + variables = { + "python": sys.executable, + "arch": arch, + "out": out, + "ldcmd": ldcmd, + "resname": resname, + "mt": mt, + "rc": rc, + "intermediate_manifest": intermediate_manifest, + "manifests": " ".join(manifests), + } + add_to_ld = "" + if manifests: + subprocess.check_call( + "%(python)s gyp-win-tool manifest-wrapper %(arch)s %(mt)s -nologo " + "-manifest %(manifests)s -out:%(out)s.manifest" % variables + ) + if embed_manifest == "True": + subprocess.check_call( + "%(python)s gyp-win-tool manifest-to-rc %(arch)s %(out)s.manifest" + " %(out)s.manifest.rc %(resname)s" % variables + ) + subprocess.check_call( + "%(python)s gyp-win-tool rc-wrapper %(arch)s %(rc)s " + "%(out)s.manifest.rc" % variables + ) + add_to_ld = " %(out)s.manifest.res" % variables + subprocess.check_call(ldcmd + add_to_ld) + + # Run mt.exe on the theoretically complete manifest we generated, merging + # it with the one the linker generated to confirm that the linker + # generated one does not add anything. This is strictly unnecessary for + # correctness, it's only to verify that e.g. /MANIFESTDEPENDENCY was not + # used in a #pragma comment. + if manifests: + # Merge the intermediate one with ours to .assert.manifest, then check + # that .assert.manifest is identical to ours. + subprocess.check_call( + "%(python)s gyp-win-tool manifest-wrapper %(arch)s %(mt)s -nologo " + "-manifest %(out)s.manifest %(intermediate_manifest)s " + "-out:%(out)s.assert.manifest" % variables + ) + assert_manifest = "%(out)s.assert.manifest" % variables + our_manifest = "%(out)s.manifest" % variables + # Load and normalize the manifests. mt.exe sometimes removes whitespace, + # and sometimes doesn't unfortunately. + with open(our_manifest, "r") as our_f: + with open(assert_manifest, "r") as assert_f: + our_data = our_f.read().translate(None, string.whitespace) + assert_data = assert_f.read().translate(None, string.whitespace) + if our_data != assert_data: + os.unlink(out) + + def dump(filename): + print(filename, file=sys.stderr) + print("-----", file=sys.stderr) + with open(filename, "r") as f: + print(f.read(), file=sys.stderr) + print("-----", file=sys.stderr) + + dump(intermediate_manifest) + dump(our_manifest) + dump(assert_manifest) + sys.stderr.write( + 'Linker generated manifest "%s" added to final manifest "%s" ' + '(result in "%s"). ' + "Were /MANIFEST switches used in #pragma statements? " + % (intermediate_manifest, our_manifest, assert_manifest) + ) + return 1 + + def ExecManifestWrapper(self, arch, *args): + """Run manifest tool with environment set. Strip out undesirable warning (some XML blocks are recognized by the OS loader, but not the manifest tool).""" - env = self._GetEnv(arch) - popen = subprocess.Popen(args, shell=True, env=env, - stdout=subprocess.PIPE, stderr=subprocess.STDOUT) - out, _ = popen.communicate() - if PY3: - out = out.decode('utf-8') - for line in out.splitlines(): - if line and 'manifest authoring warning 81010002' not in line: - print(line) - return popen.returncode - - def ExecManifestToRc(self, arch, *args): - """Creates a resource file pointing a SxS assembly manifest. + env = self._GetEnv(arch) + popen = subprocess.Popen( + args, shell=True, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT + ) + out, _ = popen.communicate() + if PY3: + out = out.decode("utf-8") + for line in out.splitlines(): + if line and "manifest authoring warning 81010002" not in line: + print(line) + return popen.returncode + + def ExecManifestToRc(self, arch, *args): + """Creates a resource file pointing a SxS assembly manifest. |args| is tuple containing path to resource file, path to manifest file and resource name which can be "1" (for executables) or "2" (for DLLs).""" - manifest_path, resource_path, resource_name = args - with open(resource_path, 'wb') as output: - output.write('#include \n%s RT_MANIFEST "%s"' % ( - resource_name, - os.path.abspath(manifest_path).replace('\\', '/'))) - - def ExecMidlWrapper(self, arch, outdir, tlb, h, dlldata, iid, proxy, idl, - *flags): - """Filter noisy filenames output from MIDL compile step that isn't + manifest_path, resource_path, resource_name = args + with open(resource_path, "w") as output: + output.write( + '#include \n%s RT_MANIFEST "%s"' + % (resource_name, os.path.abspath(manifest_path).replace("\\", "/")) + ) + + def ExecMidlWrapper(self, arch, outdir, tlb, h, dlldata, iid, proxy, idl, *flags): + """Filter noisy filenames output from MIDL compile step that isn't quietable via command line flags. """ - args = ['midl', '/nologo'] + list(flags) + [ - '/out', outdir, - '/tlb', tlb, - '/h', h, - '/dlldata', dlldata, - '/iid', iid, - '/proxy', proxy, - idl] - env = self._GetEnv(arch) - popen = subprocess.Popen(args, shell=True, env=env, - stdout=subprocess.PIPE, stderr=subprocess.STDOUT) - out, _ = popen.communicate() - if PY3: - out = out.decode('utf-8') - # Filter junk out of stdout, and write filtered versions. Output we want - # to filter is pairs of lines that look like this: - # Processing C:\Program Files (x86)\Microsoft SDKs\...\include\objidl.idl - # objidl.idl - lines = out.splitlines() - prefixes = ('Processing ', '64 bit Processing ') - processing = set(os.path.basename(x) - for x in lines if x.startswith(prefixes)) - for line in lines: - if not line.startswith(prefixes) and line not in processing: - print(line) - return popen.returncode - - def ExecAsmWrapper(self, arch, *args): - """Filter logo banner from invocations of asm.exe.""" - env = self._GetEnv(arch) - popen = subprocess.Popen(args, shell=True, env=env, - stdout=subprocess.PIPE, stderr=subprocess.STDOUT) - out, _ = popen.communicate() - if PY3: - out = out.decode('utf-8') - for line in out.splitlines(): - if (not line.startswith('Copyright (C) Microsoft Corporation') and - not line.startswith('Microsoft (R) Macro Assembler') and - not line.startswith(' Assembling: ') and - line): - print(line) - return popen.returncode - - def ExecRcWrapper(self, arch, *args): - """Filter logo banner from invocations of rc.exe. Older versions of RC + args = ( + ["midl", "/nologo"] + + list(flags) + + [ + "/out", + outdir, + "/tlb", + tlb, + "/h", + h, + "/dlldata", + dlldata, + "/iid", + iid, + "/proxy", + proxy, + idl, + ] + ) + env = self._GetEnv(arch) + popen = subprocess.Popen( + args, shell=True, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT + ) + out, _ = popen.communicate() + if PY3: + out = out.decode("utf-8") + # Filter junk out of stdout, and write filtered versions. Output we want + # to filter is pairs of lines that look like this: + # Processing C:\Program Files (x86)\Microsoft SDKs\...\include\objidl.idl + # objidl.idl + lines = out.splitlines() + prefixes = ("Processing ", "64 bit Processing ") + processing = set(os.path.basename(x) for x in lines if x.startswith(prefixes)) + for line in lines: + if not line.startswith(prefixes) and line not in processing: + print(line) + return popen.returncode + + def ExecAsmWrapper(self, arch, *args): + """Filter logo banner from invocations of asm.exe.""" + env = self._GetEnv(arch) + popen = subprocess.Popen( + args, shell=True, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT + ) + out, _ = popen.communicate() + if PY3: + out = out.decode("utf-8") + for line in out.splitlines(): + if ( + not line.startswith("Copyright (C) Microsoft Corporation") + and not line.startswith("Microsoft (R) Macro Assembler") + and not line.startswith(" Assembling: ") + and line + ): + print(line) + return popen.returncode + + def ExecRcWrapper(self, arch, *args): + """Filter logo banner from invocations of rc.exe. Older versions of RC don't support the /nologo flag.""" - env = self._GetEnv(arch) - popen = subprocess.Popen(args, shell=True, env=env, - stdout=subprocess.PIPE, stderr=subprocess.STDOUT) - out, _ = popen.communicate() - if PY3: - out = out.decode('utf-8') - for line in out.splitlines(): - if (not line.startswith('Microsoft (R) Windows (R) Resource Compiler') and - not line.startswith('Copyright (C) Microsoft Corporation') and - line): - print(line) - return popen.returncode - - def ExecActionWrapper(self, arch, rspfile, *dir): - """Runs an action command line from a response file using the environment + env = self._GetEnv(arch) + popen = subprocess.Popen( + args, shell=True, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT + ) + out, _ = popen.communicate() + if PY3: + out = out.decode("utf-8") + for line in out.splitlines(): + if ( + not line.startswith("Microsoft (R) Windows (R) Resource Compiler") + and not line.startswith("Copyright (C) Microsoft Corporation") + and line + ): + print(line) + return popen.returncode + + def ExecActionWrapper(self, arch, rspfile, *dir): + """Runs an action command line from a response file using the environment for |arch|. If |dir| is supplied, use that as the working directory.""" - env = self._GetEnv(arch) - # TODO(scottmg): This is a temporary hack to get some specific variables - # through to actions that are set after gyp-time. http://crbug.com/333738. - for k, v in os.environ.items(): - if k not in env: - env[k] = v - args = open(rspfile).read() - dir = dir[0] if dir else None - return subprocess.call(args, shell=True, env=env, cwd=dir) - - def ExecClCompile(self, project_dir, selected_files): - """Executed by msvs-ninja projects when the 'ClCompile' target is used to + env = self._GetEnv(arch) + # TODO(scottmg): This is a temporary hack to get some specific variables + # through to actions that are set after gyp-time. http://crbug.com/333738. + for k, v in os.environ.items(): + if k not in env: + env[k] = v + args = open(rspfile).read() + dir = dir[0] if dir else None + return subprocess.call(args, shell=True, env=env, cwd=dir) + + def ExecClCompile(self, project_dir, selected_files): + """Executed by msvs-ninja projects when the 'ClCompile' target is used to build selected C/C++ files.""" - project_dir = os.path.relpath(project_dir, BASE_DIR) - selected_files = selected_files.split(';') - ninja_targets = [os.path.join(project_dir, filename) + '^^' - for filename in selected_files] - cmd = ['ninja.exe'] - cmd.extend(ninja_targets) - return subprocess.call(cmd, shell=True, cwd=BASE_DIR) - -if __name__ == '__main__': - sys.exit(main(sys.argv[1:])) + project_dir = os.path.relpath(project_dir, BASE_DIR) + selected_files = selected_files.split(";") + ninja_targets = [ + os.path.join(project_dir, filename) + "^^" for filename in selected_files + ] + cmd = ["ninja.exe"] + cmd.extend(ninja_targets) + return subprocess.call(cmd, shell=True, cwd=BASE_DIR) + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py index c3daba5fb8..9a717d9095 100644 --- a/gyp/pylib/gyp/xcode_emulation.py +++ b/gyp/pylib/gyp/xcode_emulation.py @@ -17,7 +17,6 @@ import shlex import subprocess import sys -import tempfile from gyp.common import GypError PY3 = bytes != str @@ -33,71 +32,72 @@ def XcodeArchsVariableMapping(archs, archs_including_64_bit=None): - """Constructs a dictionary with expansion for $(ARCHS_STANDARD) variable, + """Constructs a dictionary with expansion for $(ARCHS_STANDARD) variable, and optionally for $(ARCHS_STANDARD_INCLUDING_64_BIT).""" - mapping = {'$(ARCHS_STANDARD)': archs} - if archs_including_64_bit: - mapping['$(ARCHS_STANDARD_INCLUDING_64_BIT)'] = archs_including_64_bit - return mapping + mapping = {"$(ARCHS_STANDARD)": archs} + if archs_including_64_bit: + mapping["$(ARCHS_STANDARD_INCLUDING_64_BIT)"] = archs_including_64_bit + return mapping + class XcodeArchsDefault(object): - """A class to resolve ARCHS variable from xcode_settings, resolving Xcode + """A class to resolve ARCHS variable from xcode_settings, resolving Xcode macros and implementing filtering by VALID_ARCHS. The expansion of macros depends on the SDKROOT used ("macosx", "iphoneos", "iphonesimulator") and on the version of Xcode. """ - # Match variable like $(ARCHS_STANDARD). - variable_pattern = re.compile(r'\$\([a-zA-Z_][a-zA-Z0-9_]*\)$') + # Match variable like $(ARCHS_STANDARD). + variable_pattern = re.compile(r"\$\([a-zA-Z_][a-zA-Z0-9_]*\)$") - def __init__(self, default, mac, iphonesimulator, iphoneos): - self._default = (default,) - self._archs = {'mac': mac, 'ios': iphoneos, 'iossim': iphonesimulator} + def __init__(self, default, mac, iphonesimulator, iphoneos): + self._default = (default,) + self._archs = {"mac": mac, "ios": iphoneos, "iossim": iphonesimulator} - def _VariableMapping(self, sdkroot): - """Returns the dictionary of variable mapping depending on the SDKROOT.""" - sdkroot = sdkroot.lower() - if 'iphoneos' in sdkroot: - return self._archs['ios'] - elif 'iphonesimulator' in sdkroot: - return self._archs['iossim'] - else: - return self._archs['mac'] - - def _ExpandArchs(self, archs, sdkroot): - """Expands variables references in ARCHS, and remove duplicates.""" - variable_mapping = self._VariableMapping(sdkroot) - expanded_archs = [] - for arch in archs: - if self.variable_pattern.match(arch): - variable = arch - try: - variable_expansion = variable_mapping[variable] - for arch in variable_expansion: - if arch not in expanded_archs: - expanded_archs.append(arch) - except KeyError as e: - print('Warning: Ignoring unsupported variable "%s".' % variable) - elif arch not in expanded_archs: - expanded_archs.append(arch) - return expanded_archs - - def ActiveArchs(self, archs, valid_archs, sdkroot): - """Expands variables references in ARCHS, and filter by VALID_ARCHS if it + def _VariableMapping(self, sdkroot): + """Returns the dictionary of variable mapping depending on the SDKROOT.""" + sdkroot = sdkroot.lower() + if "iphoneos" in sdkroot: + return self._archs["ios"] + elif "iphonesimulator" in sdkroot: + return self._archs["iossim"] + else: + return self._archs["mac"] + + def _ExpandArchs(self, archs, sdkroot): + """Expands variables references in ARCHS, and remove duplicates.""" + variable_mapping = self._VariableMapping(sdkroot) + expanded_archs = [] + for arch in archs: + if self.variable_pattern.match(arch): + variable = arch + try: + variable_expansion = variable_mapping[variable] + for arch in variable_expansion: + if arch not in expanded_archs: + expanded_archs.append(arch) + except KeyError: + print('Warning: Ignoring unsupported variable "%s".' % variable) + elif arch not in expanded_archs: + expanded_archs.append(arch) + return expanded_archs + + def ActiveArchs(self, archs, valid_archs, sdkroot): + """Expands variables references in ARCHS, and filter by VALID_ARCHS if it is defined (if not set, Xcode accept any value in ARCHS, otherwise, only values present in VALID_ARCHS are kept).""" - expanded_archs = self._ExpandArchs(archs or self._default, sdkroot or '') - if valid_archs: - filtered_archs = [] - for arch in expanded_archs: - if arch in valid_archs: - filtered_archs.append(arch) - expanded_archs = filtered_archs - return expanded_archs + expanded_archs = self._ExpandArchs(archs or self._default, sdkroot or "") + if valid_archs: + filtered_archs = [] + for arch in expanded_archs: + if arch in valid_archs: + filtered_archs.append(arch) + expanded_archs = filtered_archs + return expanded_archs def GetXcodeArchsDefault(): - """Returns the |XcodeArchsDefault| object to use to expand ARCHS for the + """Returns the |XcodeArchsDefault| object to use to expand ARCHS for the installed version of Xcode. The default values used by Xcode for ARCHS and the expansion of the variables depends on the version of Xcode used. @@ -116,738 +116,786 @@ def GetXcodeArchsDefault(): All thoses rules are coded in the construction of the |XcodeArchsDefault| object to use depending on the version of Xcode detected. The object is for performance reason.""" - global XCODE_ARCHS_DEFAULT_CACHE - if XCODE_ARCHS_DEFAULT_CACHE: + global XCODE_ARCHS_DEFAULT_CACHE + if XCODE_ARCHS_DEFAULT_CACHE: + return XCODE_ARCHS_DEFAULT_CACHE + xcode_version, _ = XcodeVersion() + if xcode_version < "0500": + XCODE_ARCHS_DEFAULT_CACHE = XcodeArchsDefault( + "$(ARCHS_STANDARD)", + XcodeArchsVariableMapping(["i386"]), + XcodeArchsVariableMapping(["i386"]), + XcodeArchsVariableMapping(["armv7"]), + ) + elif xcode_version < "0510": + XCODE_ARCHS_DEFAULT_CACHE = XcodeArchsDefault( + "$(ARCHS_STANDARD_INCLUDING_64_BIT)", + XcodeArchsVariableMapping(["x86_64"], ["x86_64"]), + XcodeArchsVariableMapping(["i386"], ["i386", "x86_64"]), + XcodeArchsVariableMapping( + ["armv7", "armv7s"], ["armv7", "armv7s", "arm64"] + ), + ) + else: + XCODE_ARCHS_DEFAULT_CACHE = XcodeArchsDefault( + "$(ARCHS_STANDARD)", + XcodeArchsVariableMapping(["x86_64"], ["x86_64"]), + XcodeArchsVariableMapping(["i386", "x86_64"], ["i386", "x86_64"]), + XcodeArchsVariableMapping( + ["armv7", "armv7s", "arm64"], ["armv7", "armv7s", "arm64"] + ), + ) return XCODE_ARCHS_DEFAULT_CACHE - xcode_version, _ = XcodeVersion() - if xcode_version < '0500': - XCODE_ARCHS_DEFAULT_CACHE = XcodeArchsDefault( - '$(ARCHS_STANDARD)', - XcodeArchsVariableMapping(['i386']), - XcodeArchsVariableMapping(['i386']), - XcodeArchsVariableMapping(['armv7'])) - elif xcode_version < '0510': - XCODE_ARCHS_DEFAULT_CACHE = XcodeArchsDefault( - '$(ARCHS_STANDARD_INCLUDING_64_BIT)', - XcodeArchsVariableMapping(['x86_64'], ['x86_64']), - XcodeArchsVariableMapping(['i386'], ['i386', 'x86_64']), - XcodeArchsVariableMapping( - ['armv7', 'armv7s'], - ['armv7', 'armv7s', 'arm64'])) - else: - XCODE_ARCHS_DEFAULT_CACHE = XcodeArchsDefault( - '$(ARCHS_STANDARD)', - XcodeArchsVariableMapping(['x86_64'], ['x86_64']), - XcodeArchsVariableMapping(['i386', 'x86_64'], ['i386', 'x86_64']), - XcodeArchsVariableMapping( - ['armv7', 'armv7s', 'arm64'], - ['armv7', 'armv7s', 'arm64'])) - return XCODE_ARCHS_DEFAULT_CACHE class XcodeSettings(object): - """A class that understands the gyp 'xcode_settings' object.""" - - # Populated lazily by _SdkPath(). Shared by all XcodeSettings, so cached - # at class-level for efficiency. - _sdk_path_cache = {} - _platform_path_cache = {} - _sdk_root_cache = {} - - # Populated lazily by GetExtraPlistItems(). Shared by all XcodeSettings, so - # cached at class-level for efficiency. - _plist_cache = {} - - # Populated lazily by GetIOSPostbuilds. Shared by all XcodeSettings, so - # cached at class-level for efficiency. - _codesigning_key_cache = {} - - def __init__(self, spec): - self.spec = spec - - self.isIOS = False - self.mac_toolchain_dir = None - self.header_map_path = None - - # Per-target 'xcode_settings' are pushed down into configs earlier by gyp. - # This means self.xcode_settings[config] always contains all settings - # for that config -- the per-target settings as well. Settings that are - # the same for all configs are implicitly per-target settings. - self.xcode_settings = {} - configs = spec['configurations'] - for configname, config in configs.items(): - self.xcode_settings[configname] = config.get('xcode_settings', {}) - self._ConvertConditionalKeys(configname) - if self.xcode_settings[configname].get('IPHONEOS_DEPLOYMENT_TARGET', - None): - self.isIOS = True - - # This is only non-None temporarily during the execution of some methods. - self.configname = None - - # Used by _AdjustLibrary to match .a and .dylib entries in libraries. - self.library_re = re.compile(r'^lib([^/]+)\.(a|dylib)$') - - def _ConvertConditionalKeys(self, configname): - """Converts or warns on conditional keys. Xcode supports conditional keys, + """A class that understands the gyp 'xcode_settings' object.""" + + # Populated lazily by _SdkPath(). Shared by all XcodeSettings, so cached + # at class-level for efficiency. + _sdk_path_cache = {} + _platform_path_cache = {} + _sdk_root_cache = {} + + # Populated lazily by GetExtraPlistItems(). Shared by all XcodeSettings, so + # cached at class-level for efficiency. + _plist_cache = {} + + # Populated lazily by GetIOSPostbuilds. Shared by all XcodeSettings, so + # cached at class-level for efficiency. + _codesigning_key_cache = {} + + def __init__(self, spec): + self.spec = spec + + self.isIOS = False + self.mac_toolchain_dir = None + self.header_map_path = None + + # Per-target 'xcode_settings' are pushed down into configs earlier by gyp. + # This means self.xcode_settings[config] always contains all settings + # for that config -- the per-target settings as well. Settings that are + # the same for all configs are implicitly per-target settings. + self.xcode_settings = {} + configs = spec["configurations"] + for configname, config in configs.items(): + self.xcode_settings[configname] = config.get("xcode_settings", {}) + self._ConvertConditionalKeys(configname) + if self.xcode_settings[configname].get("IPHONEOS_DEPLOYMENT_TARGET", None): + self.isIOS = True + + # This is only non-None temporarily during the execution of some methods. + self.configname = None + + # Used by _AdjustLibrary to match .a and .dylib entries in libraries. + self.library_re = re.compile(r"^lib([^/]+)\.(a|dylib)$") + + def _ConvertConditionalKeys(self, configname): + """Converts or warns on conditional keys. Xcode supports conditional keys, such as CODE_SIGN_IDENTITY[sdk=iphoneos*]. This is a partial implementation with some keys converted while the rest force a warning.""" - settings = self.xcode_settings[configname] - conditional_keys = [key for key in settings if key.endswith(']')] - for key in conditional_keys: - # If you need more, speak up at http://crbug.com/122592 - if key.endswith("[sdk=iphoneos*]"): - if configname.endswith("iphoneos"): - new_key = key.split("[")[0] - settings[new_key] = settings[key] - else: - print('Warning: Conditional keys not implemented, ignoring:', - ' '.join(conditional_keys)) - del settings[key] - - def _Settings(self): - assert self.configname - return self.xcode_settings[self.configname] - - def _Test(self, test_key, cond_key, default): - return self._Settings().get(test_key, default) == cond_key - - def _Appendf(self, lst, test_key, format_str, default=None): - if test_key in self._Settings(): - lst.append(format_str % str(self._Settings()[test_key])) - elif default: - lst.append(format_str % str(default)) - - def _WarnUnimplemented(self, test_key): - if test_key in self._Settings(): - print('Warning: Ignoring not yet implemented key "%s".' % test_key) - - def IsBinaryOutputFormat(self, configname): - default = "binary" if self.isIOS else "xml" - format = self.xcode_settings[configname].get('INFOPLIST_OUTPUT_FORMAT', - default) - return format == "binary" - - def IsIosFramework(self): - return self.spec['type'] == 'shared_library' and self._IsBundle() and \ - self.isIOS - - def _IsBundle(self): - return int(self.spec.get('mac_bundle', 0)) != 0 or self._IsXCTest() or \ - self._IsXCUiTest() - - def _IsXCTest(self): - return int(self.spec.get('mac_xctest_bundle', 0)) != 0 - - def _IsXCUiTest(self): - return int(self.spec.get('mac_xcuitest_bundle', 0)) != 0 - - def _IsIosAppExtension(self): - return int(self.spec.get('ios_app_extension', 0)) != 0 - - def _IsIosWatchKitExtension(self): - return int(self.spec.get('ios_watchkit_extension', 0)) != 0 - - def _IsIosWatchApp(self): - return int(self.spec.get('ios_watch_app', 0)) != 0 - - def GetFrameworkVersion(self): - """Returns the framework version of the current target. Only valid for + settings = self.xcode_settings[configname] + conditional_keys = [key for key in settings if key.endswith("]")] + for key in conditional_keys: + # If you need more, speak up at http://crbug.com/122592 + if key.endswith("[sdk=iphoneos*]"): + if configname.endswith("iphoneos"): + new_key = key.split("[")[0] + settings[new_key] = settings[key] + else: + print( + "Warning: Conditional keys not implemented, ignoring:", + " ".join(conditional_keys), + ) + del settings[key] + + def _Settings(self): + assert self.configname + return self.xcode_settings[self.configname] + + def _Test(self, test_key, cond_key, default): + return self._Settings().get(test_key, default) == cond_key + + def _Appendf(self, lst, test_key, format_str, default=None): + if test_key in self._Settings(): + lst.append(format_str % str(self._Settings()[test_key])) + elif default: + lst.append(format_str % str(default)) + + def _WarnUnimplemented(self, test_key): + if test_key in self._Settings(): + print('Warning: Ignoring not yet implemented key "%s".' % test_key) + + def IsBinaryOutputFormat(self, configname): + default = "binary" if self.isIOS else "xml" + format = self.xcode_settings[configname].get("INFOPLIST_OUTPUT_FORMAT", default) + return format == "binary" + + def IsIosFramework(self): + return self.spec["type"] == "shared_library" and self._IsBundle() and self.isIOS + + def _IsBundle(self): + return ( + int(self.spec.get("mac_bundle", 0)) != 0 + or self._IsXCTest() + or self._IsXCUiTest() + ) + + def _IsXCTest(self): + return int(self.spec.get("mac_xctest_bundle", 0)) != 0 + + def _IsXCUiTest(self): + return int(self.spec.get("mac_xcuitest_bundle", 0)) != 0 + + def _IsIosAppExtension(self): + return int(self.spec.get("ios_app_extension", 0)) != 0 + + def _IsIosWatchKitExtension(self): + return int(self.spec.get("ios_watchkit_extension", 0)) != 0 + + def _IsIosWatchApp(self): + return int(self.spec.get("ios_watch_app", 0)) != 0 + + def GetFrameworkVersion(self): + """Returns the framework version of the current target. Only valid for bundles.""" - assert self._IsBundle() - return self.GetPerTargetSetting('FRAMEWORK_VERSION', default='A') + assert self._IsBundle() + return self.GetPerTargetSetting("FRAMEWORK_VERSION", default="A") - def GetWrapperExtension(self): - """Returns the bundle extension (.app, .framework, .plugin, etc). Only + def GetWrapperExtension(self): + """Returns the bundle extension (.app, .framework, .plugin, etc). Only valid for bundles.""" - assert self._IsBundle() - if self.spec['type'] in ('loadable_module', 'shared_library'): - default_wrapper_extension = { - 'loadable_module': 'bundle', - 'shared_library': 'framework', - }[self.spec['type']] - wrapper_extension = self.GetPerTargetSetting( - 'WRAPPER_EXTENSION', default=default_wrapper_extension) - return '.' + self.spec.get('product_extension', wrapper_extension) - elif self.spec['type'] == 'executable': - if self._IsIosAppExtension() or self._IsIosWatchKitExtension(): - return '.' + self.spec.get('product_extension', 'appex') - else: - return '.' + self.spec.get('product_extension', 'app') - else: - assert False, "Don't know extension for '%s', target '%s'" % ( - self.spec['type'], self.spec['target_name']) - - def GetProductName(self): - """Returns PRODUCT_NAME.""" - return self.spec.get('product_name', self.spec['target_name']) - - def GetFullProductName(self): - """Returns FULL_PRODUCT_NAME.""" - if self._IsBundle(): - return self.GetWrapperName() - else: - return self._GetStandaloneBinaryPath() + assert self._IsBundle() + if self.spec["type"] in ("loadable_module", "shared_library"): + default_wrapper_extension = { + "loadable_module": "bundle", + "shared_library": "framework", + }[self.spec["type"]] + wrapper_extension = self.GetPerTargetSetting( + "WRAPPER_EXTENSION", default=default_wrapper_extension + ) + return "." + self.spec.get("product_extension", wrapper_extension) + elif self.spec["type"] == "executable": + if self._IsIosAppExtension() or self._IsIosWatchKitExtension(): + return "." + self.spec.get("product_extension", "appex") + else: + return "." + self.spec.get("product_extension", "app") + else: + assert False, "Don't know extension for '%s', target '%s'" % ( + self.spec["type"], + self.spec["target_name"], + ) + + def GetProductName(self): + """Returns PRODUCT_NAME.""" + return self.spec.get("product_name", self.spec["target_name"]) + + def GetFullProductName(self): + """Returns FULL_PRODUCT_NAME.""" + if self._IsBundle(): + return self.GetWrapperName() + else: + return self._GetStandaloneBinaryPath() - def GetWrapperName(self): - """Returns the directory name of the bundle represented by this target. + def GetWrapperName(self): + """Returns the directory name of the bundle represented by this target. Only valid for bundles.""" - assert self._IsBundle() - return self.GetProductName() + self.GetWrapperExtension() + assert self._IsBundle() + return self.GetProductName() + self.GetWrapperExtension() - def GetBundleContentsFolderPath(self): - """Returns the qualified path to the bundle's contents folder. E.g. + def GetBundleContentsFolderPath(self): + """Returns the qualified path to the bundle's contents folder. E.g. Chromium.app/Contents or Foo.bundle/Versions/A. Only valid for bundles.""" - if self.isIOS: - return self.GetWrapperName() - assert self._IsBundle() - if self.spec['type'] == 'shared_library': - return os.path.join( - self.GetWrapperName(), 'Versions', self.GetFrameworkVersion()) - else: - # loadable_modules have a 'Contents' folder like executables. - return os.path.join(self.GetWrapperName(), 'Contents') + if self.isIOS: + return self.GetWrapperName() + assert self._IsBundle() + if self.spec["type"] == "shared_library": + return os.path.join( + self.GetWrapperName(), "Versions", self.GetFrameworkVersion() + ) + else: + # loadable_modules have a 'Contents' folder like executables. + return os.path.join(self.GetWrapperName(), "Contents") - def GetBundleResourceFolder(self): - """Returns the qualified path to the bundle's resource folder. E.g. + def GetBundleResourceFolder(self): + """Returns the qualified path to the bundle's resource folder. E.g. Chromium.app/Contents/Resources. Only valid for bundles.""" - assert self._IsBundle() - if self.isIOS: - return self.GetBundleContentsFolderPath() - return os.path.join(self.GetBundleContentsFolderPath(), 'Resources') + assert self._IsBundle() + if self.isIOS: + return self.GetBundleContentsFolderPath() + return os.path.join(self.GetBundleContentsFolderPath(), "Resources") - def GetBundleExecutableFolderPath(self): - """Returns the qualified path to the bundle's executables folder. E.g. + def GetBundleExecutableFolderPath(self): + """Returns the qualified path to the bundle's executables folder. E.g. Chromium.app/Contents/MacOS. Only valid for bundles.""" - assert self._IsBundle() - if self.spec['type'] in ('shared_library') or self.isIOS: - return self.GetBundleContentsFolderPath() - elif self.spec['type'] in ('executable', 'loadable_module'): - return os.path.join(self.GetBundleContentsFolderPath(), 'MacOS') - - def GetBundleJavaFolderPath(self): - """Returns the qualified path to the bundle's Java resource folder. + assert self._IsBundle() + if self.spec["type"] in ("shared_library") or self.isIOS: + return self.GetBundleContentsFolderPath() + elif self.spec["type"] in ("executable", "loadable_module"): + return os.path.join(self.GetBundleContentsFolderPath(), "MacOS") + + def GetBundleJavaFolderPath(self): + """Returns the qualified path to the bundle's Java resource folder. E.g. Chromium.app/Contents/Resources/Java. Only valid for bundles.""" - assert self._IsBundle() - return os.path.join(self.GetBundleResourceFolder(), 'Java') + assert self._IsBundle() + return os.path.join(self.GetBundleResourceFolder(), "Java") - def GetBundleFrameworksFolderPath(self): - """Returns the qualified path to the bundle's frameworks folder. E.g, + def GetBundleFrameworksFolderPath(self): + """Returns the qualified path to the bundle's frameworks folder. E.g, Chromium.app/Contents/Frameworks. Only valid for bundles.""" - assert self._IsBundle() - return os.path.join(self.GetBundleContentsFolderPath(), 'Frameworks') + assert self._IsBundle() + return os.path.join(self.GetBundleContentsFolderPath(), "Frameworks") - def GetBundleSharedFrameworksFolderPath(self): - """Returns the qualified path to the bundle's frameworks folder. E.g, + def GetBundleSharedFrameworksFolderPath(self): + """Returns the qualified path to the bundle's frameworks folder. E.g, Chromium.app/Contents/SharedFrameworks. Only valid for bundles.""" - assert self._IsBundle() - return os.path.join(self.GetBundleContentsFolderPath(), - 'SharedFrameworks') + assert self._IsBundle() + return os.path.join(self.GetBundleContentsFolderPath(), "SharedFrameworks") - def GetBundleSharedSupportFolderPath(self): - """Returns the qualified path to the bundle's shared support folder. E.g, + def GetBundleSharedSupportFolderPath(self): + """Returns the qualified path to the bundle's shared support folder. E.g, Chromium.app/Contents/SharedSupport. Only valid for bundles.""" - assert self._IsBundle() - if self.spec['type'] == 'shared_library': - return self.GetBundleResourceFolder() - else: - return os.path.join(self.GetBundleContentsFolderPath(), - 'SharedSupport') + assert self._IsBundle() + if self.spec["type"] == "shared_library": + return self.GetBundleResourceFolder() + else: + return os.path.join(self.GetBundleContentsFolderPath(), "SharedSupport") - def GetBundlePlugInsFolderPath(self): - """Returns the qualified path to the bundle's plugins folder. E.g, + def GetBundlePlugInsFolderPath(self): + """Returns the qualified path to the bundle's plugins folder. E.g, Chromium.app/Contents/PlugIns. Only valid for bundles.""" - assert self._IsBundle() - return os.path.join(self.GetBundleContentsFolderPath(), 'PlugIns') + assert self._IsBundle() + return os.path.join(self.GetBundleContentsFolderPath(), "PlugIns") - def GetBundleXPCServicesFolderPath(self): - """Returns the qualified path to the bundle's XPC services folder. E.g, + def GetBundleXPCServicesFolderPath(self): + """Returns the qualified path to the bundle's XPC services folder. E.g, Chromium.app/Contents/XPCServices. Only valid for bundles.""" - assert self._IsBundle() - return os.path.join(self.GetBundleContentsFolderPath(), 'XPCServices') + assert self._IsBundle() + return os.path.join(self.GetBundleContentsFolderPath(), "XPCServices") - def GetBundlePlistPath(self): - """Returns the qualified path to the bundle's plist file. E.g. + def GetBundlePlistPath(self): + """Returns the qualified path to the bundle's plist file. E.g. Chromium.app/Contents/Info.plist. Only valid for bundles.""" - assert self._IsBundle() - if self.spec['type'] in ('executable', 'loadable_module') or \ - self.IsIosFramework(): - return os.path.join(self.GetBundleContentsFolderPath(), 'Info.plist') - else: - return os.path.join(self.GetBundleContentsFolderPath(), - 'Resources', 'Info.plist') - - def GetProductType(self): - """Returns the PRODUCT_TYPE of this target.""" - if self._IsIosAppExtension(): - assert self._IsBundle(), ('ios_app_extension flag requires mac_bundle ' - '(target %s)' % self.spec['target_name']) - return 'com.apple.product-type.app-extension' - if self._IsIosWatchKitExtension(): - assert self._IsBundle(), ('ios_watchkit_extension flag requires ' - 'mac_bundle (target %s)' % self.spec['target_name']) - return 'com.apple.product-type.watchkit-extension' - if self._IsIosWatchApp(): - assert self._IsBundle(), ('ios_watch_app flag requires mac_bundle ' - '(target %s)' % self.spec['target_name']) - return 'com.apple.product-type.application.watchapp' - if self._IsXCUiTest(): - assert self._IsBundle(), ('mac_xcuitest_bundle flag requires mac_bundle ' - '(target %s)' % self.spec['target_name']) - return 'com.apple.product-type.bundle.ui-testing' - if self._IsBundle(): - return { - 'executable': 'com.apple.product-type.application', - 'loadable_module': 'com.apple.product-type.bundle', - 'shared_library': 'com.apple.product-type.framework', - }[self.spec['type']] - else: - return { - 'executable': 'com.apple.product-type.tool', - 'loadable_module': 'com.apple.product-type.library.dynamic', - 'shared_library': 'com.apple.product-type.library.dynamic', - 'static_library': 'com.apple.product-type.library.static', - }[self.spec['type']] - - def GetMachOType(self): - """Returns the MACH_O_TYPE of this target.""" - # Weird, but matches Xcode. - if not self._IsBundle() and self.spec['type'] == 'executable': - return '' - return { - 'executable': 'mh_execute', - 'static_library': 'staticlib', - 'shared_library': 'mh_dylib', - 'loadable_module': 'mh_bundle', - }[self.spec['type']] - - def _GetBundleBinaryPath(self): - """Returns the name of the bundle binary of by this target. + assert self._IsBundle() + if ( + self.spec["type"] in ("executable", "loadable_module") + or self.IsIosFramework() + ): + return os.path.join(self.GetBundleContentsFolderPath(), "Info.plist") + else: + return os.path.join( + self.GetBundleContentsFolderPath(), "Resources", "Info.plist" + ) + + def GetProductType(self): + """Returns the PRODUCT_TYPE of this target.""" + if self._IsIosAppExtension(): + assert self._IsBundle(), ( + "ios_app_extension flag requires mac_bundle " + "(target %s)" % self.spec["target_name"] + ) + return "com.apple.product-type.app-extension" + if self._IsIosWatchKitExtension(): + assert self._IsBundle(), ( + "ios_watchkit_extension flag requires " + "mac_bundle (target %s)" % self.spec["target_name"] + ) + return "com.apple.product-type.watchkit-extension" + if self._IsIosWatchApp(): + assert self._IsBundle(), ( + "ios_watch_app flag requires mac_bundle " + "(target %s)" % self.spec["target_name"] + ) + return "com.apple.product-type.application.watchapp" + if self._IsXCUiTest(): + assert self._IsBundle(), ( + "mac_xcuitest_bundle flag requires mac_bundle " + "(target %s)" % self.spec["target_name"] + ) + return "com.apple.product-type.bundle.ui-testing" + if self._IsBundle(): + return { + "executable": "com.apple.product-type.application", + "loadable_module": "com.apple.product-type.bundle", + "shared_library": "com.apple.product-type.framework", + }[self.spec["type"]] + else: + return { + "executable": "com.apple.product-type.tool", + "loadable_module": "com.apple.product-type.library.dynamic", + "shared_library": "com.apple.product-type.library.dynamic", + "static_library": "com.apple.product-type.library.static", + }[self.spec["type"]] + + def GetMachOType(self): + """Returns the MACH_O_TYPE of this target.""" + # Weird, but matches Xcode. + if not self._IsBundle() and self.spec["type"] == "executable": + return "" + return { + "executable": "mh_execute", + "static_library": "staticlib", + "shared_library": "mh_dylib", + "loadable_module": "mh_bundle", + }[self.spec["type"]] + + def _GetBundleBinaryPath(self): + """Returns the name of the bundle binary of by this target. E.g. Chromium.app/Contents/MacOS/Chromium. Only valid for bundles.""" - assert self._IsBundle() - return os.path.join(self.GetBundleExecutableFolderPath(), \ - self.GetExecutableName()) - - def _GetStandaloneExecutableSuffix(self): - if 'product_extension' in self.spec: - return '.' + self.spec['product_extension'] - return { - 'executable': '', - 'static_library': '.a', - 'shared_library': '.dylib', - 'loadable_module': '.so', - }[self.spec['type']] - - def _GetStandaloneExecutablePrefix(self): - return self.spec.get('product_prefix', { - 'executable': '', - 'static_library': 'lib', - 'shared_library': 'lib', - # Non-bundled loadable_modules are called foo.so for some reason - # (that is, .so and no prefix) with the xcode build -- match that. - 'loadable_module': '', - }[self.spec['type']]) - - def _GetStandaloneBinaryPath(self): - """Returns the name of the non-bundle binary represented by this target. + assert self._IsBundle() + return os.path.join( + self.GetBundleExecutableFolderPath(), self.GetExecutableName() + ) + + def _GetStandaloneExecutableSuffix(self): + if "product_extension" in self.spec: + return "." + self.spec["product_extension"] + return { + "executable": "", + "static_library": ".a", + "shared_library": ".dylib", + "loadable_module": ".so", + }[self.spec["type"]] + + def _GetStandaloneExecutablePrefix(self): + return self.spec.get( + "product_prefix", + { + "executable": "", + "static_library": "lib", + "shared_library": "lib", + # Non-bundled loadable_modules are called foo.so for some reason + # (that is, .so and no prefix) with the xcode build -- match that. + "loadable_module": "", + }[self.spec["type"]], + ) + + def _GetStandaloneBinaryPath(self): + """Returns the name of the non-bundle binary represented by this target. E.g. hello_world. Only valid for non-bundles.""" - assert not self._IsBundle() - assert self.spec['type'] in ( - 'executable', 'shared_library', 'static_library', 'loadable_module'), ( - 'Unexpected type %s' % self.spec['type']) - target = self.spec['target_name'] - if self.spec['type'] == 'static_library': - if target[:3] == 'lib': - target = target[3:] - elif self.spec['type'] in ('loadable_module', 'shared_library'): - if target[:3] == 'lib': - target = target[3:] - - target_prefix = self._GetStandaloneExecutablePrefix() - target = self.spec.get('product_name', target) - target_ext = self._GetStandaloneExecutableSuffix() - return target_prefix + target + target_ext - - def GetExecutableName(self): - """Returns the executable name of the bundle represented by this target. + assert not self._IsBundle() + assert self.spec["type"] in ( + "executable", + "shared_library", + "static_library", + "loadable_module", + ), ("Unexpected type %s" % self.spec["type"]) + target = self.spec["target_name"] + if self.spec["type"] == "static_library": + if target[:3] == "lib": + target = target[3:] + elif self.spec["type"] in ("loadable_module", "shared_library"): + if target[:3] == "lib": + target = target[3:] + + target_prefix = self._GetStandaloneExecutablePrefix() + target = self.spec.get("product_name", target) + target_ext = self._GetStandaloneExecutableSuffix() + return target_prefix + target + target_ext + + def GetExecutableName(self): + """Returns the executable name of the bundle represented by this target. E.g. Chromium.""" - if self._IsBundle(): - return self.spec.get('product_name', self.spec['target_name']) - else: - return self._GetStandaloneBinaryPath() + if self._IsBundle(): + return self.spec.get("product_name", self.spec["target_name"]) + else: + return self._GetStandaloneBinaryPath() - def GetExecutablePath(self): - """Returns the qualified path to the primary executable of the bundle + def GetExecutablePath(self): + """Returns the qualified path to the primary executable of the bundle represented by this target. E.g. Chromium.app/Contents/MacOS/Chromium.""" - if self._IsBundle(): - return self._GetBundleBinaryPath() - else: - return self._GetStandaloneBinaryPath() - - def GetActiveArchs(self, configname): - """Returns the architectures this target should be built for.""" - config_settings = self.xcode_settings[configname] - xcode_archs_default = GetXcodeArchsDefault() - return xcode_archs_default.ActiveArchs( - config_settings.get('ARCHS'), - config_settings.get('VALID_ARCHS'), - config_settings.get('SDKROOT')) - - def _GetSdkVersionInfoItem(self, sdk, infoitem): - # xcodebuild requires Xcode and can't run on Command Line Tools-only - # systems from 10.7 onward. - # Since the CLT has no SDK paths anyway, returning None is the - # most sensible route and should still do the right thing. - try: - return GetStdoutQuiet(['xcrun', '--sdk', sdk, infoitem]) - except GypError: - pass - - def _SdkRoot(self, configname): - if configname is None: - configname = self.configname - return self.GetPerConfigSetting('SDKROOT', configname, default='') - - def _XcodePlatformPath(self, configname=None): - sdk_root = self._SdkRoot(configname) - if sdk_root not in XcodeSettings._platform_path_cache: - platform_path = self._GetSdkVersionInfoItem(sdk_root, - '--show-sdk-platform-path') - XcodeSettings._platform_path_cache[sdk_root] = platform_path - return XcodeSettings._platform_path_cache[sdk_root] - - def _SdkPath(self, configname=None): - sdk_root = self._SdkRoot(configname) - if sdk_root.startswith('/'): - return sdk_root - return self._XcodeSdkPath(sdk_root) - - def _XcodeSdkPath(self, sdk_root): - if sdk_root not in XcodeSettings._sdk_path_cache: - sdk_path = self._GetSdkVersionInfoItem(sdk_root, '--show-sdk-path') - XcodeSettings._sdk_path_cache[sdk_root] = sdk_path - if sdk_root: - XcodeSettings._sdk_root_cache[sdk_path] = sdk_root - return XcodeSettings._sdk_path_cache[sdk_root] - - def _AppendPlatformVersionMinFlags(self, lst): - self._Appendf(lst, 'MACOSX_DEPLOYMENT_TARGET', '-mmacosx-version-min=%s') - if 'IPHONEOS_DEPLOYMENT_TARGET' in self._Settings(): - # TODO: Implement this better? - sdk_path_basename = os.path.basename(self._SdkPath()) - if sdk_path_basename.lower().startswith('iphonesimulator'): - self._Appendf(lst, 'IPHONEOS_DEPLOYMENT_TARGET', - '-mios-simulator-version-min=%s') - else: - self._Appendf(lst, 'IPHONEOS_DEPLOYMENT_TARGET', - '-miphoneos-version-min=%s') - - def GetCflags(self, configname, arch=None): - """Returns flags that need to be added to .c, .cc, .m, and .mm + if self._IsBundle(): + return self._GetBundleBinaryPath() + else: + return self._GetStandaloneBinaryPath() + + def GetActiveArchs(self, configname): + """Returns the architectures this target should be built for.""" + config_settings = self.xcode_settings[configname] + xcode_archs_default = GetXcodeArchsDefault() + return xcode_archs_default.ActiveArchs( + config_settings.get("ARCHS"), + config_settings.get("VALID_ARCHS"), + config_settings.get("SDKROOT"), + ) + + def _GetSdkVersionInfoItem(self, sdk, infoitem): + # xcodebuild requires Xcode and can't run on Command Line Tools-only + # systems from 10.7 onward. + # Since the CLT has no SDK paths anyway, returning None is the + # most sensible route and should still do the right thing. + try: + return GetStdoutQuiet(["xcrun", "--sdk", sdk, infoitem]) + except GypError: + pass + + def _SdkRoot(self, configname): + if configname is None: + configname = self.configname + return self.GetPerConfigSetting("SDKROOT", configname, default="") + + def _XcodePlatformPath(self, configname=None): + sdk_root = self._SdkRoot(configname) + if sdk_root not in XcodeSettings._platform_path_cache: + platform_path = self._GetSdkVersionInfoItem( + sdk_root, "--show-sdk-platform-path" + ) + XcodeSettings._platform_path_cache[sdk_root] = platform_path + return XcodeSettings._platform_path_cache[sdk_root] + + def _SdkPath(self, configname=None): + sdk_root = self._SdkRoot(configname) + if sdk_root.startswith("/"): + return sdk_root + return self._XcodeSdkPath(sdk_root) + + def _XcodeSdkPath(self, sdk_root): + if sdk_root not in XcodeSettings._sdk_path_cache: + sdk_path = self._GetSdkVersionInfoItem(sdk_root, "--show-sdk-path") + XcodeSettings._sdk_path_cache[sdk_root] = sdk_path + if sdk_root: + XcodeSettings._sdk_root_cache[sdk_path] = sdk_root + return XcodeSettings._sdk_path_cache[sdk_root] + + def _AppendPlatformVersionMinFlags(self, lst): + self._Appendf(lst, "MACOSX_DEPLOYMENT_TARGET", "-mmacosx-version-min=%s") + if "IPHONEOS_DEPLOYMENT_TARGET" in self._Settings(): + # TODO: Implement this better? + sdk_path_basename = os.path.basename(self._SdkPath()) + if sdk_path_basename.lower().startswith("iphonesimulator"): + self._Appendf( + lst, "IPHONEOS_DEPLOYMENT_TARGET", "-mios-simulator-version-min=%s" + ) + else: + self._Appendf( + lst, "IPHONEOS_DEPLOYMENT_TARGET", "-miphoneos-version-min=%s" + ) + + def GetCflags(self, configname, arch=None): + """Returns flags that need to be added to .c, .cc, .m, and .mm compilations.""" - # This functions (and the similar ones below) do not offer complete - # emulation of all xcode_settings keys. They're implemented on demand. + # This functions (and the similar ones below) do not offer complete + # emulation of all xcode_settings keys. They're implemented on demand. - self.configname = configname - cflags = [] + self.configname = configname + cflags = [] - sdk_root = self._SdkPath() - if 'SDKROOT' in self._Settings() and sdk_root: - cflags.append('-isysroot %s' % sdk_root) + sdk_root = self._SdkPath() + if "SDKROOT" in self._Settings() and sdk_root: + cflags.append("-isysroot %s" % sdk_root) - if self.header_map_path: - cflags.append('-I%s' % self.header_map_path) + if self.header_map_path: + cflags.append("-I%s" % self.header_map_path) - if self._Test('CLANG_WARN_CONSTANT_CONVERSION', 'YES', default='NO'): - cflags.append('-Wconstant-conversion') + if self._Test("CLANG_WARN_CONSTANT_CONVERSION", "YES", default="NO"): + cflags.append("-Wconstant-conversion") - if self._Test('GCC_CHAR_IS_UNSIGNED_CHAR', 'YES', default='NO'): - cflags.append('-funsigned-char') + if self._Test("GCC_CHAR_IS_UNSIGNED_CHAR", "YES", default="NO"): + cflags.append("-funsigned-char") - if self._Test('GCC_CW_ASM_SYNTAX', 'YES', default='YES'): - cflags.append('-fasm-blocks') + if self._Test("GCC_CW_ASM_SYNTAX", "YES", default="YES"): + cflags.append("-fasm-blocks") - if 'GCC_DYNAMIC_NO_PIC' in self._Settings(): - if self._Settings()['GCC_DYNAMIC_NO_PIC'] == 'YES': - cflags.append('-mdynamic-no-pic') - else: - pass - # TODO: In this case, it depends on the target. xcode passes - # mdynamic-no-pic by default for executable and possibly static lib - # according to mento - - if self._Test('GCC_ENABLE_PASCAL_STRINGS', 'YES', default='YES'): - cflags.append('-mpascal-strings') - - self._Appendf(cflags, 'GCC_OPTIMIZATION_LEVEL', '-O%s', default='s') - - if self._Test('GCC_GENERATE_DEBUGGING_SYMBOLS', 'YES', default='YES'): - dbg_format = self._Settings().get('DEBUG_INFORMATION_FORMAT', 'dwarf') - if dbg_format == 'dwarf': - cflags.append('-gdwarf-2') - elif dbg_format == 'stabs': - raise NotImplementedError('stabs debug format is not supported yet.') - elif dbg_format == 'dwarf-with-dsym': - cflags.append('-gdwarf-2') - else: - raise NotImplementedError('Unknown debug format %s' % dbg_format) - - if self._Settings().get('GCC_STRICT_ALIASING') == 'YES': - cflags.append('-fstrict-aliasing') - elif self._Settings().get('GCC_STRICT_ALIASING') == 'NO': - cflags.append('-fno-strict-aliasing') - - if self._Test('GCC_SYMBOLS_PRIVATE_EXTERN', 'YES', default='NO'): - cflags.append('-fvisibility=hidden') - - if self._Test('GCC_TREAT_WARNINGS_AS_ERRORS', 'YES', default='NO'): - cflags.append('-Werror') - - if self._Test('GCC_WARN_ABOUT_MISSING_NEWLINE', 'YES', default='NO'): - cflags.append('-Wnewline-eof') - - # In Xcode, this is only activated when GCC_COMPILER_VERSION is clang or - # llvm-gcc. It also requires a fairly recent libtool, and - # if the system clang isn't used, DYLD_LIBRARY_PATH needs to contain the - # path to the libLTO.dylib that matches the used clang. - if self._Test('LLVM_LTO', 'YES', default='NO'): - cflags.append('-flto') - - self._AppendPlatformVersionMinFlags(cflags) - - # TODO: - if self._Test('COPY_PHASE_STRIP', 'YES', default='NO'): - self._WarnUnimplemented('COPY_PHASE_STRIP') - self._WarnUnimplemented('GCC_DEBUGGING_SYMBOLS') - self._WarnUnimplemented('GCC_ENABLE_OBJC_EXCEPTIONS') - - # TODO: This is exported correctly, but assigning to it is not supported. - self._WarnUnimplemented('MACH_O_TYPE') - self._WarnUnimplemented('PRODUCT_TYPE') - - if arch is not None: - archs = [arch] - else: - assert self.configname - archs = self.GetActiveArchs(self.configname) - if len(archs) != 1: - # TODO: Supporting fat binaries will be annoying. - self._WarnUnimplemented('ARCHS') - archs = ['i386'] - cflags.append('-arch ' + archs[0]) - - if archs[0] in ('i386', 'x86_64'): - if self._Test('GCC_ENABLE_SSE3_EXTENSIONS', 'YES', default='NO'): - cflags.append('-msse3') - if self._Test('GCC_ENABLE_SUPPLEMENTAL_SSE3_INSTRUCTIONS', 'YES', - default='NO'): - cflags.append('-mssse3') # Note 3rd 's'. - if self._Test('GCC_ENABLE_SSE41_EXTENSIONS', 'YES', default='NO'): - cflags.append('-msse4.1') - if self._Test('GCC_ENABLE_SSE42_EXTENSIONS', 'YES', default='NO'): - cflags.append('-msse4.2') - - cflags += self._Settings().get('WARNING_CFLAGS', []) - - if self._IsXCTest(): - platform_root = self._XcodePlatformPath(configname) - if platform_root: - cflags.append('-F' + platform_root + '/Developer/Library/Frameworks/') - - if sdk_root: - framework_root = sdk_root - else: - framework_root = '' - config = self.spec['configurations'][self.configname] - framework_dirs = config.get('mac_framework_dirs', []) - for directory in framework_dirs: - cflags.append('-F' + directory.replace('$(SDKROOT)', framework_root)) - - self.configname = None - return cflags - - def GetCflagsC(self, configname): - """Returns flags that need to be added to .c, and .m compilations.""" - self.configname = configname - cflags_c = [] - if self._Settings().get('GCC_C_LANGUAGE_STANDARD', '') == 'ansi': - cflags_c.append('-ansi') - else: - self._Appendf(cflags_c, 'GCC_C_LANGUAGE_STANDARD', '-std=%s') - cflags_c += self._Settings().get('OTHER_CFLAGS', []) - self.configname = None - return cflags_c - - def GetCflagsCC(self, configname): - """Returns flags that need to be added to .cc, and .mm compilations.""" - self.configname = configname - cflags_cc = [] - - clang_cxx_language_standard = self._Settings().get( - 'CLANG_CXX_LANGUAGE_STANDARD') - # Note: Don't make c++0x to c++11 so that c++0x can be used with older - # clangs that don't understand c++11 yet (like Xcode 4.2's). - if clang_cxx_language_standard: - cflags_cc.append('-std=%s' % clang_cxx_language_standard) - - self._Appendf(cflags_cc, 'CLANG_CXX_LIBRARY', '-stdlib=%s') - - if self._Test('GCC_ENABLE_CPP_RTTI', 'NO', default='YES'): - cflags_cc.append('-fno-rtti') - if self._Test('GCC_ENABLE_CPP_EXCEPTIONS', 'NO', default='YES'): - cflags_cc.append('-fno-exceptions') - if self._Test('GCC_INLINES_ARE_PRIVATE_EXTERN', 'YES', default='NO'): - cflags_cc.append('-fvisibility-inlines-hidden') - if self._Test('GCC_THREADSAFE_STATICS', 'NO', default='YES'): - cflags_cc.append('-fno-threadsafe-statics') - # Note: This flag is a no-op for clang, it only has an effect for gcc. - if self._Test('GCC_WARN_ABOUT_INVALID_OFFSETOF_MACRO', 'NO', default='YES'): - cflags_cc.append('-Wno-invalid-offsetof') - - other_ccflags = [] - - for flag in self._Settings().get('OTHER_CPLUSPLUSFLAGS', ['$(inherited)']): - # TODO: More general variable expansion. Missing in many other places too. - if flag in ('$inherited', '$(inherited)', '${inherited}'): - flag = '$OTHER_CFLAGS' - if flag in ('$OTHER_CFLAGS', '$(OTHER_CFLAGS)', '${OTHER_CFLAGS}'): - other_ccflags += self._Settings().get('OTHER_CFLAGS', []) - else: - other_ccflags.append(flag) - cflags_cc += other_ccflags - - self.configname = None - return cflags_cc - - def _AddObjectiveCGarbageCollectionFlags(self, flags): - gc_policy = self._Settings().get('GCC_ENABLE_OBJC_GC', 'unsupported') - if gc_policy == 'supported': - flags.append('-fobjc-gc') - elif gc_policy == 'required': - flags.append('-fobjc-gc-only') - - def _AddObjectiveCARCFlags(self, flags): - if self._Test('CLANG_ENABLE_OBJC_ARC', 'YES', default='NO'): - flags.append('-fobjc-arc') - - def _AddObjectiveCMissingPropertySynthesisFlags(self, flags): - if self._Test('CLANG_WARN_OBJC_MISSING_PROPERTY_SYNTHESIS', - 'YES', default='NO'): - flags.append('-Wobjc-missing-property-synthesis') - - def GetCflagsObjC(self, configname): - """Returns flags that need to be added to .m compilations.""" - self.configname = configname - cflags_objc = [] - self._AddObjectiveCGarbageCollectionFlags(cflags_objc) - self._AddObjectiveCARCFlags(cflags_objc) - self._AddObjectiveCMissingPropertySynthesisFlags(cflags_objc) - self.configname = None - return cflags_objc - - def GetCflagsObjCC(self, configname): - """Returns flags that need to be added to .mm compilations.""" - self.configname = configname - cflags_objcc = [] - self._AddObjectiveCGarbageCollectionFlags(cflags_objcc) - self._AddObjectiveCARCFlags(cflags_objcc) - self._AddObjectiveCMissingPropertySynthesisFlags(cflags_objcc) - if self._Test('GCC_OBJC_CALL_CXX_CDTORS', 'YES', default='NO'): - cflags_objcc.append('-fobjc-call-cxx-cdtors') - self.configname = None - return cflags_objcc - - def GetInstallNameBase(self): - """Return DYLIB_INSTALL_NAME_BASE for this target.""" - # Xcode sets this for shared_libraries, and for nonbundled loadable_modules. - if (self.spec['type'] != 'shared_library' and - (self.spec['type'] != 'loadable_module' or self._IsBundle())): - return None - install_base = self.GetPerTargetSetting( - 'DYLIB_INSTALL_NAME_BASE', - default='/Library/Frameworks' if self._IsBundle() else '/usr/local/lib') - return install_base - - def _StandardizePath(self, path): - """Do :standardizepath processing for path.""" - # I'm not quite sure what :standardizepath does. Just call normpath(), - # but don't let @executable_path/../foo collapse to foo. - if '/' in path: - prefix, rest = '', path - if path.startswith('@'): - prefix, rest = path.split('/', 1) - rest = os.path.normpath(rest) # :standardizepath - path = os.path.join(prefix, rest) - return path - - def GetInstallName(self): - """Return LD_DYLIB_INSTALL_NAME for this target.""" - # Xcode sets this for shared_libraries, and for nonbundled loadable_modules. - if (self.spec['type'] != 'shared_library' and - (self.spec['type'] != 'loadable_module' or self._IsBundle())): - return None - - default_install_name = \ - '$(DYLIB_INSTALL_NAME_BASE:standardizepath)/$(EXECUTABLE_PATH)' - install_name = self.GetPerTargetSetting( - 'LD_DYLIB_INSTALL_NAME', default=default_install_name) - - # Hardcode support for the variables used in chromium for now, to - # unblock people using the make build. - if '$' in install_name: - assert install_name in ('$(DYLIB_INSTALL_NAME_BASE:standardizepath)/' - '$(WRAPPER_NAME)/$(PRODUCT_NAME)', default_install_name), ( - 'Variables in LD_DYLIB_INSTALL_NAME are not generally supported ' - 'yet in target \'%s\' (got \'%s\')' % - (self.spec['target_name'], install_name)) - - install_name = install_name.replace( - '$(DYLIB_INSTALL_NAME_BASE:standardizepath)', - self._StandardizePath(self.GetInstallNameBase())) - if self._IsBundle(): - # These are only valid for bundles, hence the |if|. - install_name = install_name.replace( - '$(WRAPPER_NAME)', self.GetWrapperName()) - install_name = install_name.replace( - '$(PRODUCT_NAME)', self.GetProductName()) - else: - assert '$(WRAPPER_NAME)' not in install_name - assert '$(PRODUCT_NAME)' not in install_name - - install_name = install_name.replace( - '$(EXECUTABLE_PATH)', self.GetExecutablePath()) - return install_name - - def _MapLinkerFlagFilename(self, ldflag, gyp_to_build_path): - """Checks if ldflag contains a filename and if so remaps it from + if "GCC_DYNAMIC_NO_PIC" in self._Settings(): + if self._Settings()["GCC_DYNAMIC_NO_PIC"] == "YES": + cflags.append("-mdynamic-no-pic") + else: + pass + # TODO: In this case, it depends on the target. xcode passes + # mdynamic-no-pic by default for executable and possibly static lib + # according to mento + + if self._Test("GCC_ENABLE_PASCAL_STRINGS", "YES", default="YES"): + cflags.append("-mpascal-strings") + + self._Appendf(cflags, "GCC_OPTIMIZATION_LEVEL", "-O%s", default="s") + + if self._Test("GCC_GENERATE_DEBUGGING_SYMBOLS", "YES", default="YES"): + dbg_format = self._Settings().get("DEBUG_INFORMATION_FORMAT", "dwarf") + if dbg_format == "dwarf": + cflags.append("-gdwarf-2") + elif dbg_format == "stabs": + raise NotImplementedError("stabs debug format is not supported yet.") + elif dbg_format == "dwarf-with-dsym": + cflags.append("-gdwarf-2") + else: + raise NotImplementedError("Unknown debug format %s" % dbg_format) + + if self._Settings().get("GCC_STRICT_ALIASING") == "YES": + cflags.append("-fstrict-aliasing") + elif self._Settings().get("GCC_STRICT_ALIASING") == "NO": + cflags.append("-fno-strict-aliasing") + + if self._Test("GCC_SYMBOLS_PRIVATE_EXTERN", "YES", default="NO"): + cflags.append("-fvisibility=hidden") + + if self._Test("GCC_TREAT_WARNINGS_AS_ERRORS", "YES", default="NO"): + cflags.append("-Werror") + + if self._Test("GCC_WARN_ABOUT_MISSING_NEWLINE", "YES", default="NO"): + cflags.append("-Wnewline-eof") + + # In Xcode, this is only activated when GCC_COMPILER_VERSION is clang or + # llvm-gcc. It also requires a fairly recent libtool, and + # if the system clang isn't used, DYLD_LIBRARY_PATH needs to contain the + # path to the libLTO.dylib that matches the used clang. + if self._Test("LLVM_LTO", "YES", default="NO"): + cflags.append("-flto") + + self._AppendPlatformVersionMinFlags(cflags) + + # TODO: + if self._Test("COPY_PHASE_STRIP", "YES", default="NO"): + self._WarnUnimplemented("COPY_PHASE_STRIP") + self._WarnUnimplemented("GCC_DEBUGGING_SYMBOLS") + self._WarnUnimplemented("GCC_ENABLE_OBJC_EXCEPTIONS") + + # TODO: This is exported correctly, but assigning to it is not supported. + self._WarnUnimplemented("MACH_O_TYPE") + self._WarnUnimplemented("PRODUCT_TYPE") + + if arch is not None: + archs = [arch] + else: + assert self.configname + archs = self.GetActiveArchs(self.configname) + if len(archs) != 1: + # TODO: Supporting fat binaries will be annoying. + self._WarnUnimplemented("ARCHS") + archs = ["i386"] + cflags.append("-arch " + archs[0]) + + if archs[0] in ("i386", "x86_64"): + if self._Test("GCC_ENABLE_SSE3_EXTENSIONS", "YES", default="NO"): + cflags.append("-msse3") + if self._Test( + "GCC_ENABLE_SUPPLEMENTAL_SSE3_INSTRUCTIONS", "YES", default="NO" + ): + cflags.append("-mssse3") # Note 3rd 's'. + if self._Test("GCC_ENABLE_SSE41_EXTENSIONS", "YES", default="NO"): + cflags.append("-msse4.1") + if self._Test("GCC_ENABLE_SSE42_EXTENSIONS", "YES", default="NO"): + cflags.append("-msse4.2") + + cflags += self._Settings().get("WARNING_CFLAGS", []) + + if self._IsXCTest(): + platform_root = self._XcodePlatformPath(configname) + if platform_root: + cflags.append("-F" + platform_root + "/Developer/Library/Frameworks/") + + if sdk_root: + framework_root = sdk_root + else: + framework_root = "" + config = self.spec["configurations"][self.configname] + framework_dirs = config.get("mac_framework_dirs", []) + for directory in framework_dirs: + cflags.append("-F" + directory.replace("$(SDKROOT)", framework_root)) + + self.configname = None + return cflags + + def GetCflagsC(self, configname): + """Returns flags that need to be added to .c, and .m compilations.""" + self.configname = configname + cflags_c = [] + if self._Settings().get("GCC_C_LANGUAGE_STANDARD", "") == "ansi": + cflags_c.append("-ansi") + else: + self._Appendf(cflags_c, "GCC_C_LANGUAGE_STANDARD", "-std=%s") + cflags_c += self._Settings().get("OTHER_CFLAGS", []) + self.configname = None + return cflags_c + + def GetCflagsCC(self, configname): + """Returns flags that need to be added to .cc, and .mm compilations.""" + self.configname = configname + cflags_cc = [] + + clang_cxx_language_standard = self._Settings().get( + "CLANG_CXX_LANGUAGE_STANDARD" + ) + # Note: Don't make c++0x to c++11 so that c++0x can be used with older + # clangs that don't understand c++11 yet (like Xcode 4.2's). + if clang_cxx_language_standard: + cflags_cc.append("-std=%s" % clang_cxx_language_standard) + + self._Appendf(cflags_cc, "CLANG_CXX_LIBRARY", "-stdlib=%s") + + if self._Test("GCC_ENABLE_CPP_RTTI", "NO", default="YES"): + cflags_cc.append("-fno-rtti") + if self._Test("GCC_ENABLE_CPP_EXCEPTIONS", "NO", default="YES"): + cflags_cc.append("-fno-exceptions") + if self._Test("GCC_INLINES_ARE_PRIVATE_EXTERN", "YES", default="NO"): + cflags_cc.append("-fvisibility-inlines-hidden") + if self._Test("GCC_THREADSAFE_STATICS", "NO", default="YES"): + cflags_cc.append("-fno-threadsafe-statics") + # Note: This flag is a no-op for clang, it only has an effect for gcc. + if self._Test("GCC_WARN_ABOUT_INVALID_OFFSETOF_MACRO", "NO", default="YES"): + cflags_cc.append("-Wno-invalid-offsetof") + + other_ccflags = [] + + for flag in self._Settings().get("OTHER_CPLUSPLUSFLAGS", ["$(inherited)"]): + # TODO: More general variable expansion. Missing in many other places too. + if flag in ("$inherited", "$(inherited)", "${inherited}"): + flag = "$OTHER_CFLAGS" + if flag in ("$OTHER_CFLAGS", "$(OTHER_CFLAGS)", "${OTHER_CFLAGS}"): + other_ccflags += self._Settings().get("OTHER_CFLAGS", []) + else: + other_ccflags.append(flag) + cflags_cc += other_ccflags + + self.configname = None + return cflags_cc + + def _AddObjectiveCGarbageCollectionFlags(self, flags): + gc_policy = self._Settings().get("GCC_ENABLE_OBJC_GC", "unsupported") + if gc_policy == "supported": + flags.append("-fobjc-gc") + elif gc_policy == "required": + flags.append("-fobjc-gc-only") + + def _AddObjectiveCARCFlags(self, flags): + if self._Test("CLANG_ENABLE_OBJC_ARC", "YES", default="NO"): + flags.append("-fobjc-arc") + + def _AddObjectiveCMissingPropertySynthesisFlags(self, flags): + if self._Test( + "CLANG_WARN_OBJC_MISSING_PROPERTY_SYNTHESIS", "YES", default="NO" + ): + flags.append("-Wobjc-missing-property-synthesis") + + def GetCflagsObjC(self, configname): + """Returns flags that need to be added to .m compilations.""" + self.configname = configname + cflags_objc = [] + self._AddObjectiveCGarbageCollectionFlags(cflags_objc) + self._AddObjectiveCARCFlags(cflags_objc) + self._AddObjectiveCMissingPropertySynthesisFlags(cflags_objc) + self.configname = None + return cflags_objc + + def GetCflagsObjCC(self, configname): + """Returns flags that need to be added to .mm compilations.""" + self.configname = configname + cflags_objcc = [] + self._AddObjectiveCGarbageCollectionFlags(cflags_objcc) + self._AddObjectiveCARCFlags(cflags_objcc) + self._AddObjectiveCMissingPropertySynthesisFlags(cflags_objcc) + if self._Test("GCC_OBJC_CALL_CXX_CDTORS", "YES", default="NO"): + cflags_objcc.append("-fobjc-call-cxx-cdtors") + self.configname = None + return cflags_objcc + + def GetInstallNameBase(self): + """Return DYLIB_INSTALL_NAME_BASE for this target.""" + # Xcode sets this for shared_libraries, and for nonbundled loadable_modules. + if self.spec["type"] != "shared_library" and ( + self.spec["type"] != "loadable_module" or self._IsBundle() + ): + return None + install_base = self.GetPerTargetSetting( + "DYLIB_INSTALL_NAME_BASE", + default="/Library/Frameworks" if self._IsBundle() else "/usr/local/lib", + ) + return install_base + + def _StandardizePath(self, path): + """Do :standardizepath processing for path.""" + # I'm not quite sure what :standardizepath does. Just call normpath(), + # but don't let @executable_path/../foo collapse to foo. + if "/" in path: + prefix, rest = "", path + if path.startswith("@"): + prefix, rest = path.split("/", 1) + rest = os.path.normpath(rest) # :standardizepath + path = os.path.join(prefix, rest) + return path + + def GetInstallName(self): + """Return LD_DYLIB_INSTALL_NAME for this target.""" + # Xcode sets this for shared_libraries, and for nonbundled loadable_modules. + if self.spec["type"] != "shared_library" and ( + self.spec["type"] != "loadable_module" or self._IsBundle() + ): + return None + + default_install_name = ( + "$(DYLIB_INSTALL_NAME_BASE:standardizepath)/$(EXECUTABLE_PATH)" + ) + install_name = self.GetPerTargetSetting( + "LD_DYLIB_INSTALL_NAME", default=default_install_name + ) + + # Hardcode support for the variables used in chromium for now, to + # unblock people using the make build. + if "$" in install_name: + assert install_name in ( + "$(DYLIB_INSTALL_NAME_BASE:standardizepath)/" + "$(WRAPPER_NAME)/$(PRODUCT_NAME)", + default_install_name, + ), ( + "Variables in LD_DYLIB_INSTALL_NAME are not generally supported " + "yet in target '%s' (got '%s')" + % (self.spec["target_name"], install_name) + ) + + install_name = install_name.replace( + "$(DYLIB_INSTALL_NAME_BASE:standardizepath)", + self._StandardizePath(self.GetInstallNameBase()), + ) + if self._IsBundle(): + # These are only valid for bundles, hence the |if|. + install_name = install_name.replace( + "$(WRAPPER_NAME)", self.GetWrapperName() + ) + install_name = install_name.replace( + "$(PRODUCT_NAME)", self.GetProductName() + ) + else: + assert "$(WRAPPER_NAME)" not in install_name + assert "$(PRODUCT_NAME)" not in install_name + + install_name = install_name.replace( + "$(EXECUTABLE_PATH)", self.GetExecutablePath() + ) + return install_name + + def _MapLinkerFlagFilename(self, ldflag, gyp_to_build_path): + """Checks if ldflag contains a filename and if so remaps it from gyp-directory-relative to build-directory-relative.""" - # This list is expanded on demand. - # They get matched as: - # -exported_symbols_list file - # -Wl,exported_symbols_list file - # -Wl,exported_symbols_list,file - LINKER_FILE = r'(\S+)' - WORD = r'\S+' - linker_flags = [ - ['-exported_symbols_list', LINKER_FILE], # Needed for NaCl. - ['-unexported_symbols_list', LINKER_FILE], - ['-reexported_symbols_list', LINKER_FILE], - ['-sectcreate', WORD, WORD, LINKER_FILE], # Needed for remoting. - ] - for flag_pattern in linker_flags: - regex = re.compile('(?:-Wl,)?' + '[ ,]'.join(flag_pattern)) - m = regex.match(ldflag) - if m: - ldflag = ldflag[:m.start(1)] + gyp_to_build_path(m.group(1)) + \ - ldflag[m.end(1):] - # Required for ffmpeg (no idea why they don't use LIBRARY_SEARCH_PATHS, - # TODO(thakis): Update ffmpeg.gyp): - if ldflag.startswith('-L'): - ldflag = '-L' + gyp_to_build_path(ldflag[len('-L'):]) - return ldflag - - def GetLdflags(self, configname, product_dir, gyp_to_build_path, arch=None): - """Returns flags that need to be passed to the linker. + # This list is expanded on demand. + # They get matched as: + # -exported_symbols_list file + # -Wl,exported_symbols_list file + # -Wl,exported_symbols_list,file + LINKER_FILE = r"(\S+)" + WORD = r"\S+" + linker_flags = [ + ["-exported_symbols_list", LINKER_FILE], # Needed for NaCl. + ["-unexported_symbols_list", LINKER_FILE], + ["-reexported_symbols_list", LINKER_FILE], + ["-sectcreate", WORD, WORD, LINKER_FILE], # Needed for remoting. + ] + for flag_pattern in linker_flags: + regex = re.compile("(?:-Wl,)?" + "[ ,]".join(flag_pattern)) + m = regex.match(ldflag) + if m: + ldflag = ( + ldflag[: m.start(1)] + + gyp_to_build_path(m.group(1)) + + ldflag[m.end(1) :] + ) + # Required for ffmpeg (no idea why they don't use LIBRARY_SEARCH_PATHS, + # TODO(thakis): Update ffmpeg.gyp): + if ldflag.startswith("-L"): + ldflag = "-L" + gyp_to_build_path(ldflag[len("-L") :]) + return ldflag + + def GetLdflags(self, configname, product_dir, gyp_to_build_path, arch=None): + """Returns flags that need to be passed to the linker. Args: configname: The name of the configuration to get ld flags for. @@ -856,431 +904,470 @@ def GetLdflags(self, configname, product_dir, gyp_to_build_path, arch=None): gyp_to_build_path: A function that converts paths relative to the current gyp file to paths relative to the build directory. """ - self.configname = configname - ldflags = [] + self.configname = configname + ldflags = [] - # The xcode build is relative to a gyp file's directory, and OTHER_LDFLAGS - # can contain entries that depend on this. Explicitly absolutify these. - for ldflag in self._Settings().get('OTHER_LDFLAGS', []): - ldflags.append(self._MapLinkerFlagFilename(ldflag, gyp_to_build_path)) + # The xcode build is relative to a gyp file's directory, and OTHER_LDFLAGS + # can contain entries that depend on this. Explicitly absolutify these. + for ldflag in self._Settings().get("OTHER_LDFLAGS", []): + ldflags.append(self._MapLinkerFlagFilename(ldflag, gyp_to_build_path)) - if self._Test('DEAD_CODE_STRIPPING', 'YES', default='NO'): - ldflags.append('-Wl,-dead_strip') + if self._Test("DEAD_CODE_STRIPPING", "YES", default="NO"): + ldflags.append("-Wl,-dead_strip") - if self._Test('PREBINDING', 'YES', default='NO'): - ldflags.append('-Wl,-prebind') + if self._Test("PREBINDING", "YES", default="NO"): + ldflags.append("-Wl,-prebind") - self._Appendf( - ldflags, 'DYLIB_COMPATIBILITY_VERSION', '-compatibility_version %s') - self._Appendf( - ldflags, 'DYLIB_CURRENT_VERSION', '-current_version %s') + self._Appendf( + ldflags, "DYLIB_COMPATIBILITY_VERSION", "-compatibility_version %s" + ) + self._Appendf(ldflags, "DYLIB_CURRENT_VERSION", "-current_version %s") - self._AppendPlatformVersionMinFlags(ldflags) + self._AppendPlatformVersionMinFlags(ldflags) - if 'SDKROOT' in self._Settings() and self._SdkPath(): - ldflags.append('-isysroot ' + self._SdkPath()) + if "SDKROOT" in self._Settings() and self._SdkPath(): + ldflags.append("-isysroot " + self._SdkPath()) - for library_path in self._Settings().get('LIBRARY_SEARCH_PATHS', []): - ldflags.append('-L' + gyp_to_build_path(library_path)) + for library_path in self._Settings().get("LIBRARY_SEARCH_PATHS", []): + ldflags.append("-L" + gyp_to_build_path(library_path)) - if 'ORDER_FILE' in self._Settings(): - ldflags.append('-Wl,-order_file ' + - '-Wl,' + gyp_to_build_path( - self._Settings()['ORDER_FILE'])) + if "ORDER_FILE" in self._Settings(): + ldflags.append( + "-Wl,-order_file " + + "-Wl," + + gyp_to_build_path(self._Settings()["ORDER_FILE"]) + ) - if arch is not None: - archs = [arch] - else: - assert self.configname - archs = self.GetActiveArchs(self.configname) - if len(archs) != 1: - # TODO: Supporting fat binaries will be annoying. - self._WarnUnimplemented('ARCHS') - archs = ['i386'] - ldflags.append('-arch ' + archs[0]) - - # Xcode adds the product directory by default. - # Rewrite -L. to -L./ to work around http://www.openradar.me/25313838 - ldflags.append('-L' + (product_dir if product_dir != '.' else './')) - - install_name = self.GetInstallName() - if install_name and self.spec['type'] != 'loadable_module': - ldflags.append('-install_name ' + install_name.replace(' ', r'\ ')) - - for rpath in self._Settings().get('LD_RUNPATH_SEARCH_PATHS', []): - ldflags.append('-Wl,-rpath,' + rpath) - - sdk_root = self._SdkPath() - if not sdk_root: - sdk_root = '' - config = self.spec['configurations'][self.configname] - framework_dirs = config.get('mac_framework_dirs', []) - for directory in framework_dirs: - ldflags.append('-F' + directory.replace('$(SDKROOT)', sdk_root)) - - if self._IsXCTest(): - platform_root = self._XcodePlatformPath(configname) - if sdk_root and platform_root: - ldflags.append('-F' + platform_root + '/Developer/Library/Frameworks/') - ldflags.append('-framework XCTest') - - is_extension = self._IsIosAppExtension() or self._IsIosWatchKitExtension() - if sdk_root and is_extension: - # Adds the link flags for extensions. These flags are common for all - # extensions and provide loader and main function. - # These flags reflect the compilation options used by xcode to compile - # extensions. - xcode_version, _ = XcodeVersion() - if xcode_version < '0900': - ldflags.append('-lpkstart') - ldflags.append(sdk_root + - '/System/Library/PrivateFrameworks/PlugInKit.framework/PlugInKit') - else: - ldflags.append('-e _NSExtensionMain') - ldflags.append('-fapplication-extension') - - self._Appendf(ldflags, 'CLANG_CXX_LIBRARY', '-stdlib=%s') - - self.configname = None - return ldflags - - def GetLibtoolflags(self, configname): - """Returns flags that need to be passed to the static linker. + if arch is not None: + archs = [arch] + else: + assert self.configname + archs = self.GetActiveArchs(self.configname) + if len(archs) != 1: + # TODO: Supporting fat binaries will be annoying. + self._WarnUnimplemented("ARCHS") + archs = ["i386"] + ldflags.append("-arch " + archs[0]) + + # Xcode adds the product directory by default. + # Rewrite -L. to -L./ to work around http://www.openradar.me/25313838 + ldflags.append("-L" + (product_dir if product_dir != "." else "./")) + + install_name = self.GetInstallName() + if install_name and self.spec["type"] != "loadable_module": + ldflags.append("-install_name " + install_name.replace(" ", r"\ ")) + + for rpath in self._Settings().get("LD_RUNPATH_SEARCH_PATHS", []): + ldflags.append("-Wl,-rpath," + rpath) + + sdk_root = self._SdkPath() + if not sdk_root: + sdk_root = "" + config = self.spec["configurations"][self.configname] + framework_dirs = config.get("mac_framework_dirs", []) + for directory in framework_dirs: + ldflags.append("-F" + directory.replace("$(SDKROOT)", sdk_root)) + + if self._IsXCTest(): + platform_root = self._XcodePlatformPath(configname) + if sdk_root and platform_root: + ldflags.append("-F" + platform_root + "/Developer/Library/Frameworks/") + ldflags.append("-framework XCTest") + + is_extension = self._IsIosAppExtension() or self._IsIosWatchKitExtension() + if sdk_root and is_extension: + # Adds the link flags for extensions. These flags are common for all + # extensions and provide loader and main function. + # These flags reflect the compilation options used by xcode to compile + # extensions. + xcode_version, _ = XcodeVersion() + if xcode_version < "0900": + ldflags.append("-lpkstart") + ldflags.append( + sdk_root + + "/System/Library/PrivateFrameworks/PlugInKit.framework/PlugInKit" + ) + else: + ldflags.append("-e _NSExtensionMain") + ldflags.append("-fapplication-extension") + + self._Appendf(ldflags, "CLANG_CXX_LIBRARY", "-stdlib=%s") + + self.configname = None + return ldflags + + def GetLibtoolflags(self, configname): + """Returns flags that need to be passed to the static linker. Args: configname: The name of the configuration to get ld flags for. """ - self.configname = configname - libtoolflags = [] + self.configname = configname + libtoolflags = [] - for libtoolflag in self._Settings().get('OTHER_LDFLAGS', []): - libtoolflags.append(libtoolflag) - # TODO(thakis): ARCHS? + for libtoolflag in self._Settings().get("OTHER_LDFLAGS", []): + libtoolflags.append(libtoolflag) + # TODO(thakis): ARCHS? - self.configname = None - return libtoolflags + self.configname = None + return libtoolflags - def GetPerTargetSettings(self): - """Gets a list of all the per-target settings. This will only fetch keys + def GetPerTargetSettings(self): + """Gets a list of all the per-target settings. This will only fetch keys whose values are the same across all configurations.""" - first_pass = True - result = {} - for configname in sorted(self.xcode_settings.keys()): - if first_pass: - result = dict(self.xcode_settings[configname]) - first_pass = False - else: - for key, value in self.xcode_settings[configname].items(): - if key not in result: - continue - elif result[key] != value: - del result[key] - return result - - def GetPerConfigSetting(self, setting, configname, default=None): - if configname in self.xcode_settings: - return self.xcode_settings[configname].get(setting, default) - else: - return self.GetPerTargetSetting(setting, default) + first_pass = True + result = {} + for configname in sorted(self.xcode_settings.keys()): + if first_pass: + result = dict(self.xcode_settings[configname]) + first_pass = False + else: + for key, value in self.xcode_settings[configname].items(): + if key not in result: + continue + elif result[key] != value: + del result[key] + return result + + def GetPerConfigSetting(self, setting, configname, default=None): + if configname in self.xcode_settings: + return self.xcode_settings[configname].get(setting, default) + else: + return self.GetPerTargetSetting(setting, default) - def GetPerTargetSetting(self, setting, default=None): - """Tries to get xcode_settings.setting from spec. Assumes that the setting + def GetPerTargetSetting(self, setting, default=None): + """Tries to get xcode_settings.setting from spec. Assumes that the setting has the same value in all configurations and throws otherwise.""" - is_first_pass = True - result = None - for configname in sorted(self.xcode_settings.keys()): - if is_first_pass: - result = self.xcode_settings[configname].get(setting, None) - is_first_pass = False - else: - assert result == self.xcode_settings[configname].get(setting, None), ( - "Expected per-target setting for '%s', got per-config setting " - "(target %s)" % (setting, self.spec['target_name'])) - if result is None: - return default - return result - - def _GetStripPostbuilds(self, configname, output_binary, quiet): - """Returns a list of shell commands that contain the shell commands + is_first_pass = True + result = None + for configname in sorted(self.xcode_settings.keys()): + if is_first_pass: + result = self.xcode_settings[configname].get(setting, None) + is_first_pass = False + else: + assert result == self.xcode_settings[configname].get(setting, None), ( + "Expected per-target setting for '%s', got per-config setting " + "(target %s)" % (setting, self.spec["target_name"]) + ) + if result is None: + return default + return result + + def _GetStripPostbuilds(self, configname, output_binary, quiet): + """Returns a list of shell commands that contain the shell commands necessary to strip this target's binary. These should be run as postbuilds before the actual postbuilds run.""" - self.configname = configname - - result = [] - if (self._Test('DEPLOYMENT_POSTPROCESSING', 'YES', default='NO') and - self._Test('STRIP_INSTALLED_PRODUCT', 'YES', default='NO')): - - default_strip_style = 'debugging' - if ((self.spec['type'] == 'loadable_module' or self._IsIosAppExtension()) - and self._IsBundle()): - default_strip_style = 'non-global' - elif self.spec['type'] == 'executable': - default_strip_style = 'all' - - strip_style = self._Settings().get('STRIP_STYLE', default_strip_style) - strip_flags = { - 'all': '', - 'non-global': '-x', - 'debugging': '-S', - }[strip_style] - - explicit_strip_flags = self._Settings().get('STRIPFLAGS', '') - if explicit_strip_flags: - strip_flags += ' ' + _NormalizeEnvVarReferences(explicit_strip_flags) - - if not quiet: - result.append('echo STRIP\\(%s\\)' % self.spec['target_name']) - result.append('strip %s %s' % (strip_flags, output_binary)) - - self.configname = None - return result - - def _GetDebugInfoPostbuilds(self, configname, output, output_binary, quiet): - """Returns a list of shell commands that contain the shell commands + self.configname = configname + + result = [] + if self._Test("DEPLOYMENT_POSTPROCESSING", "YES", default="NO") and self._Test( + "STRIP_INSTALLED_PRODUCT", "YES", default="NO" + ): + + default_strip_style = "debugging" + if ( + self.spec["type"] == "loadable_module" or self._IsIosAppExtension() + ) and self._IsBundle(): + default_strip_style = "non-global" + elif self.spec["type"] == "executable": + default_strip_style = "all" + + strip_style = self._Settings().get("STRIP_STYLE", default_strip_style) + strip_flags = {"all": "", "non-global": "-x", "debugging": "-S"}[ + strip_style + ] + + explicit_strip_flags = self._Settings().get("STRIPFLAGS", "") + if explicit_strip_flags: + strip_flags += " " + _NormalizeEnvVarReferences(explicit_strip_flags) + + if not quiet: + result.append("echo STRIP\\(%s\\)" % self.spec["target_name"]) + result.append("strip %s %s" % (strip_flags, output_binary)) + + self.configname = None + return result + + def _GetDebugInfoPostbuilds(self, configname, output, output_binary, quiet): + """Returns a list of shell commands that contain the shell commands necessary to massage this target's debug information. These should be run as postbuilds before the actual postbuilds run.""" - self.configname = configname - - # For static libraries, no dSYMs are created. - result = [] - if (self._Test('GCC_GENERATE_DEBUGGING_SYMBOLS', 'YES', default='YES') and - self._Test( - 'DEBUG_INFORMATION_FORMAT', 'dwarf-with-dsym', default='dwarf') and - self.spec['type'] != 'static_library'): - if not quiet: - result.append('echo DSYMUTIL\\(%s\\)' % self.spec['target_name']) - result.append('dsymutil %s -o %s' % (output_binary, output + '.dSYM')) - - self.configname = None - return result - - def _GetTargetPostbuilds(self, configname, output, output_binary, - quiet=False): - """Returns a list of shell commands that contain the shell commands + self.configname = configname + + # For static libraries, no dSYMs are created. + result = [] + if ( + self._Test("GCC_GENERATE_DEBUGGING_SYMBOLS", "YES", default="YES") + and self._Test( + "DEBUG_INFORMATION_FORMAT", "dwarf-with-dsym", default="dwarf" + ) + and self.spec["type"] != "static_library" + ): + if not quiet: + result.append("echo DSYMUTIL\\(%s\\)" % self.spec["target_name"]) + result.append("dsymutil %s -o %s" % (output_binary, output + ".dSYM")) + + self.configname = None + return result + + def _GetTargetPostbuilds(self, configname, output, output_binary, quiet=False): + """Returns a list of shell commands that contain the shell commands to run as postbuilds for this target, before the actual postbuilds.""" - # dSYMs need to build before stripping happens. - return ( - self._GetDebugInfoPostbuilds(configname, output, output_binary, quiet) + - self._GetStripPostbuilds(configname, output_binary, quiet)) + # dSYMs need to build before stripping happens. + return self._GetDebugInfoPostbuilds( + configname, output, output_binary, quiet + ) + self._GetStripPostbuilds(configname, output_binary, quiet) - def _GetIOSPostbuilds(self, configname, output_binary): - """Return a shell command to codesign the iOS output binary so it can + def _GetIOSPostbuilds(self, configname, output_binary): + """Return a shell command to codesign the iOS output binary so it can be deployed to a device. This should be run as the very last step of the build.""" - if not (self.isIOS and - (self.spec['type'] == 'executable' or self._IsXCTest()) or - self.IsIosFramework()): - return [] - - postbuilds = [] - product_name = self.GetFullProductName() - settings = self.xcode_settings[configname] - - # Xcode expects XCTests to be copied into the TEST_HOST dir. - if self._IsXCTest(): - source = os.path.join("${BUILT_PRODUCTS_DIR}", product_name) - test_host = os.path.dirname(settings.get('TEST_HOST')) - xctest_destination = os.path.join(test_host, 'PlugIns', product_name) - postbuilds.extend(['ditto %s %s' % (source, xctest_destination)]) - - key = self._GetIOSCodeSignIdentityKey(settings) - if not key: - return postbuilds - - # Warn for any unimplemented signing xcode keys. - unimpl = ['OTHER_CODE_SIGN_FLAGS'] - unimpl = set(unimpl) & set(self.xcode_settings[configname].keys()) - if unimpl: - print('Warning: Some codesign keys not implemented, ignoring: %s' % - ', '.join(sorted(unimpl))) - - if self._IsXCTest(): - # For device xctests, Xcode copies two extra frameworks into $TEST_HOST. - test_host = os.path.dirname(settings.get('TEST_HOST')) - frameworks_dir = os.path.join(test_host, 'Frameworks') - platform_root = self._XcodePlatformPath(configname) - frameworks = \ - ['Developer/Library/PrivateFrameworks/IDEBundleInjection.framework', - 'Developer/Library/Frameworks/XCTest.framework'] - for framework in frameworks: - source = os.path.join(platform_root, framework) - destination = os.path.join(frameworks_dir, os.path.basename(framework)) - postbuilds.extend(['ditto %s %s' % (source, destination)]) - - # Then re-sign everything with 'preserve=True' - postbuilds.extend(['%s code-sign-bundle "%s" "%s" "%s" "%s" %s' % ( - os.path.join('${TARGET_BUILD_DIR}', 'gyp-mac-tool'), key, - settings.get('CODE_SIGN_ENTITLEMENTS', ''), - settings.get('PROVISIONING_PROFILE', ''), destination, True) - ]) - plugin_dir = os.path.join(test_host, 'PlugIns') - targets = [os.path.join(plugin_dir, product_name), test_host] - for target in targets: - postbuilds.extend(['%s code-sign-bundle "%s" "%s" "%s" "%s" %s' % ( - os.path.join('${TARGET_BUILD_DIR}', 'gyp-mac-tool'), key, - settings.get('CODE_SIGN_ENTITLEMENTS', ''), - settings.get('PROVISIONING_PROFILE', ''), target, True) - ]) - - postbuilds.extend(['%s code-sign-bundle "%s" "%s" "%s" "%s" %s' % ( - os.path.join('${TARGET_BUILD_DIR}', 'gyp-mac-tool'), key, - settings.get('CODE_SIGN_ENTITLEMENTS', ''), - settings.get('PROVISIONING_PROFILE', ''), - os.path.join("${BUILT_PRODUCTS_DIR}", product_name), False) - ]) - return postbuilds - - def _GetIOSCodeSignIdentityKey(self, settings): - identity = settings.get('CODE_SIGN_IDENTITY') - if not identity: - return None - if identity not in XcodeSettings._codesigning_key_cache: - output = subprocess.check_output( - ['security', 'find-identity', '-p', 'codesigning', '-v']) - for line in output.splitlines(): - if identity in line: - fingerprint = line.split()[1] - cache = XcodeSettings._codesigning_key_cache - assert identity not in cache or fingerprint == cache[identity], ( - "Multiple codesigning fingerprints for identity: %s" % identity) - XcodeSettings._codesigning_key_cache[identity] = fingerprint - return XcodeSettings._codesigning_key_cache.get(identity, '') - - def AddImplicitPostbuilds(self, configname, output, output_binary, - postbuilds=[], quiet=False): - """Returns a list of shell commands that should run before and after + if not ( + self.isIOS + and (self.spec["type"] == "executable" or self._IsXCTest()) + or self.IsIosFramework() + ): + return [] + + postbuilds = [] + product_name = self.GetFullProductName() + settings = self.xcode_settings[configname] + + # Xcode expects XCTests to be copied into the TEST_HOST dir. + if self._IsXCTest(): + source = os.path.join("${BUILT_PRODUCTS_DIR}", product_name) + test_host = os.path.dirname(settings.get("TEST_HOST")) + xctest_destination = os.path.join(test_host, "PlugIns", product_name) + postbuilds.extend(["ditto %s %s" % (source, xctest_destination)]) + + key = self._GetIOSCodeSignIdentityKey(settings) + if not key: + return postbuilds + + # Warn for any unimplemented signing xcode keys. + unimpl = ["OTHER_CODE_SIGN_FLAGS"] + unimpl = set(unimpl) & set(self.xcode_settings[configname].keys()) + if unimpl: + print( + "Warning: Some codesign keys not implemented, ignoring: %s" + % ", ".join(sorted(unimpl)) + ) + + if self._IsXCTest(): + # For device xctests, Xcode copies two extra frameworks into $TEST_HOST. + test_host = os.path.dirname(settings.get("TEST_HOST")) + frameworks_dir = os.path.join(test_host, "Frameworks") + platform_root = self._XcodePlatformPath(configname) + frameworks = [ + "Developer/Library/PrivateFrameworks/IDEBundleInjection.framework", + "Developer/Library/Frameworks/XCTest.framework", + ] + for framework in frameworks: + source = os.path.join(platform_root, framework) + destination = os.path.join(frameworks_dir, os.path.basename(framework)) + postbuilds.extend(["ditto %s %s" % (source, destination)]) + + # Then re-sign everything with 'preserve=True' + postbuilds.extend( + [ + '%s code-sign-bundle "%s" "%s" "%s" "%s" %s' + % ( + os.path.join("${TARGET_BUILD_DIR}", "gyp-mac-tool"), + key, + settings.get("CODE_SIGN_ENTITLEMENTS", ""), + settings.get("PROVISIONING_PROFILE", ""), + destination, + True, + ) + ] + ) + plugin_dir = os.path.join(test_host, "PlugIns") + targets = [os.path.join(plugin_dir, product_name), test_host] + for target in targets: + postbuilds.extend( + [ + '%s code-sign-bundle "%s" "%s" "%s" "%s" %s' + % ( + os.path.join("${TARGET_BUILD_DIR}", "gyp-mac-tool"), + key, + settings.get("CODE_SIGN_ENTITLEMENTS", ""), + settings.get("PROVISIONING_PROFILE", ""), + target, + True, + ) + ] + ) + + postbuilds.extend( + [ + '%s code-sign-bundle "%s" "%s" "%s" "%s" %s' + % ( + os.path.join("${TARGET_BUILD_DIR}", "gyp-mac-tool"), + key, + settings.get("CODE_SIGN_ENTITLEMENTS", ""), + settings.get("PROVISIONING_PROFILE", ""), + os.path.join("${BUILT_PRODUCTS_DIR}", product_name), + False, + ) + ] + ) + return postbuilds + + def _GetIOSCodeSignIdentityKey(self, settings): + identity = settings.get("CODE_SIGN_IDENTITY") + if not identity: + return None + if identity not in XcodeSettings._codesigning_key_cache: + output = subprocess.check_output( + ["security", "find-identity", "-p", "codesigning", "-v"] + ) + for line in output.splitlines(): + if identity in line: + fingerprint = line.split()[1] + cache = XcodeSettings._codesigning_key_cache + assert identity not in cache or fingerprint == cache[identity], ( + "Multiple codesigning fingerprints for identity: %s" % identity + ) + XcodeSettings._codesigning_key_cache[identity] = fingerprint + return XcodeSettings._codesigning_key_cache.get(identity, "") + + def AddImplicitPostbuilds( + self, configname, output, output_binary, postbuilds=[], quiet=False + ): + """Returns a list of shell commands that should run before and after |postbuilds|.""" - assert output_binary is not None - pre = self._GetTargetPostbuilds(configname, output, output_binary, quiet) - post = self._GetIOSPostbuilds(configname, output_binary) - return pre + postbuilds + post - - def _AdjustLibrary(self, library, config_name=None): - if library.endswith('.framework'): - l = '-framework ' + os.path.splitext(os.path.basename(library))[0] - else: - m = self.library_re.match(library) - if m: - l = '-l' + m.group(1) - else: - l = library - - sdk_root = self._SdkPath(config_name) - if not sdk_root: - sdk_root = '' - # Xcode 7 started shipping with ".tbd" (text based stubs) files instead of - # ".dylib" without providing a real support for them. What it does, for - # "/usr/lib" libraries, is do "-L/usr/lib -lname" which is dependent on the - # library order and cause collision when building Chrome. - # - # Instead substitute ".tbd" to ".dylib" in the generated project when the - # following conditions are both true: - # - library is referenced in the gyp file as "$(SDKROOT)/**/*.dylib", - # - the ".dylib" file does not exists but a ".tbd" file do. - library = l.replace('$(SDKROOT)', sdk_root) - if l.startswith('$(SDKROOT)'): - basename, ext = os.path.splitext(library) - if ext == '.dylib' and not os.path.exists(library): - tbd_library = basename + '.tbd' - if os.path.exists(tbd_library): - library = tbd_library - return library - - def AdjustLibraries(self, libraries, config_name=None): - """Transforms entries like 'Cocoa.framework' in libraries into entries like + assert output_binary is not None + pre = self._GetTargetPostbuilds(configname, output, output_binary, quiet) + post = self._GetIOSPostbuilds(configname, output_binary) + return pre + postbuilds + post + + def _AdjustLibrary(self, library, config_name=None): + if library.endswith(".framework"): + l_flag = "-framework " + os.path.splitext(os.path.basename(library))[0] + else: + m = self.library_re.match(library) + if m: + l_flag = "-l" + m.group(1) + else: + l_flag = library + + sdk_root = self._SdkPath(config_name) + if not sdk_root: + sdk_root = "" + # Xcode 7 started shipping with ".tbd" (text based stubs) files instead of + # ".dylib" without providing a real support for them. What it does, for + # "/usr/lib" libraries, is do "-L/usr/lib -lname" which is dependent on the + # library order and cause collision when building Chrome. + # + # Instead substitute ".tbd" to ".dylib" in the generated project when the + # following conditions are both true: + # - library is referenced in the gyp file as "$(SDKROOT)/**/*.dylib", + # - the ".dylib" file does not exists but a ".tbd" file do. + library = l_flag.replace("$(SDKROOT)", sdk_root) + if l_flag.startswith("$(SDKROOT)"): + basename, ext = os.path.splitext(library) + if ext == ".dylib" and not os.path.exists(library): + tbd_library = basename + ".tbd" + if os.path.exists(tbd_library): + library = tbd_library + return library + + def AdjustLibraries(self, libraries, config_name=None): + """Transforms entries like 'Cocoa.framework' in libraries into entries like '-framework Cocoa', 'libcrypto.dylib' into '-lcrypto', etc. """ - libraries = [self._AdjustLibrary(library, config_name) - for library in libraries] - return libraries - - def _BuildMachineOSBuild(self): - return GetStdout(['sw_vers', '-buildVersion']) - - def _XcodeIOSDeviceFamily(self, configname): - family = self.xcode_settings[configname].get('TARGETED_DEVICE_FAMILY', '1') - return [int(x) for x in family.split(',')] - - def GetExtraPlistItems(self, configname=None): - """Returns a dictionary with extra items to insert into Info.plist.""" - if configname not in XcodeSettings._plist_cache: - cache = {} - cache['BuildMachineOSBuild'] = self._BuildMachineOSBuild() - - xcode_version, xcode_build = XcodeVersion() - cache['DTXcode'] = xcode_version - cache['DTXcodeBuild'] = xcode_build - compiler = self.xcode_settings[configname].get('GCC_VERSION') - if compiler is not None: - cache['DTCompiler'] = compiler - - sdk_root = self._SdkRoot(configname) - if not sdk_root: - sdk_root = self._DefaultSdkRoot() - sdk_version = self._GetSdkVersionInfoItem(sdk_root, '--show-sdk-version') - cache['DTSDKName'] = sdk_root + (sdk_version or '') - if xcode_version >= '0720': - cache['DTSDKBuild'] = self._GetSdkVersionInfoItem( - sdk_root, '--show-sdk-build-version') - elif xcode_version >= '0430': - cache['DTSDKBuild'] = sdk_version - else: - cache['DTSDKBuild'] = cache['BuildMachineOSBuild'] - - if self.isIOS: - cache['MinimumOSVersion'] = self.xcode_settings[configname].get( - 'IPHONEOS_DEPLOYMENT_TARGET') - cache['DTPlatformName'] = sdk_root - cache['DTPlatformVersion'] = sdk_version - - if configname.endswith("iphoneos"): - cache['CFBundleSupportedPlatforms'] = ['iPhoneOS'] - cache['DTPlatformBuild'] = cache['DTSDKBuild'] - else: - cache['CFBundleSupportedPlatforms'] = ['iPhoneSimulator'] - # This is weird, but Xcode sets DTPlatformBuild to an empty field - # for simulator builds. - cache['DTPlatformBuild'] = "" - XcodeSettings._plist_cache[configname] = cache - - # Include extra plist items that are per-target, not per global - # XcodeSettings. - items = dict(XcodeSettings._plist_cache[configname]) - if self.isIOS: - items['UIDeviceFamily'] = self._XcodeIOSDeviceFamily(configname) - return items - - def _DefaultSdkRoot(self): - """Returns the default SDKROOT to use. + libraries = [self._AdjustLibrary(library, config_name) for library in libraries] + return libraries + + def _BuildMachineOSBuild(self): + return GetStdout(["sw_vers", "-buildVersion"]) + + def _XcodeIOSDeviceFamily(self, configname): + family = self.xcode_settings[configname].get("TARGETED_DEVICE_FAMILY", "1") + return [int(x) for x in family.split(",")] + + def GetExtraPlistItems(self, configname=None): + """Returns a dictionary with extra items to insert into Info.plist.""" + if configname not in XcodeSettings._plist_cache: + cache = {} + cache["BuildMachineOSBuild"] = self._BuildMachineOSBuild() + + xcode_version, xcode_build = XcodeVersion() + cache["DTXcode"] = xcode_version + cache["DTXcodeBuild"] = xcode_build + compiler = self.xcode_settings[configname].get("GCC_VERSION") + if compiler is not None: + cache["DTCompiler"] = compiler + + sdk_root = self._SdkRoot(configname) + if not sdk_root: + sdk_root = self._DefaultSdkRoot() + sdk_version = self._GetSdkVersionInfoItem(sdk_root, "--show-sdk-version") + cache["DTSDKName"] = sdk_root + (sdk_version or "") + if xcode_version >= "0720": + cache["DTSDKBuild"] = self._GetSdkVersionInfoItem( + sdk_root, "--show-sdk-build-version" + ) + elif xcode_version >= "0430": + cache["DTSDKBuild"] = sdk_version + else: + cache["DTSDKBuild"] = cache["BuildMachineOSBuild"] + + if self.isIOS: + cache["MinimumOSVersion"] = self.xcode_settings[configname].get( + "IPHONEOS_DEPLOYMENT_TARGET" + ) + cache["DTPlatformName"] = sdk_root + cache["DTPlatformVersion"] = sdk_version + + if configname.endswith("iphoneos"): + cache["CFBundleSupportedPlatforms"] = ["iPhoneOS"] + cache["DTPlatformBuild"] = cache["DTSDKBuild"] + else: + cache["CFBundleSupportedPlatforms"] = ["iPhoneSimulator"] + # This is weird, but Xcode sets DTPlatformBuild to an empty field + # for simulator builds. + cache["DTPlatformBuild"] = "" + XcodeSettings._plist_cache[configname] = cache + + # Include extra plist items that are per-target, not per global + # XcodeSettings. + items = dict(XcodeSettings._plist_cache[configname]) + if self.isIOS: + items["UIDeviceFamily"] = self._XcodeIOSDeviceFamily(configname) + return items + + def _DefaultSdkRoot(self): + """Returns the default SDKROOT to use. Prior to version 5.0.0, if SDKROOT was not explicitly set in the Xcode project, then the environment variable was empty. Starting with this version, Xcode uses the name of the newest SDK installed. """ - xcode_version, _ = XcodeVersion() - if xcode_version < '0500': - return '' - default_sdk_path = self._XcodeSdkPath('') - default_sdk_root = XcodeSettings._sdk_root_cache.get(default_sdk_path) - if default_sdk_root: - return default_sdk_root - try: - all_sdks = GetStdout(['xcodebuild', '-showsdks']) - except GypError: - # If xcodebuild fails, there will be no valid SDKs - return '' - for line in all_sdks.splitlines(): - items = line.split() - if len(items) >= 3 and items[-2] == '-sdk': - sdk_root = items[-1] - sdk_path = self._XcodeSdkPath(sdk_root) - if sdk_path == default_sdk_path: - return sdk_root - return '' + xcode_version, _ = XcodeVersion() + if xcode_version < "0500": + return "" + default_sdk_path = self._XcodeSdkPath("") + default_sdk_root = XcodeSettings._sdk_root_cache.get(default_sdk_path) + if default_sdk_root: + return default_sdk_root + try: + all_sdks = GetStdout(["xcodebuild", "-showsdks"]) + except GypError: + # If xcodebuild fails, there will be no valid SDKs + return "" + for line in all_sdks.splitlines(): + items = line.split() + if len(items) >= 3 and items[-2] == "-sdk": + sdk_root = items[-1] + sdk_path = self._XcodeSdkPath(sdk_root) + if sdk_path == default_sdk_path: + return sdk_root + return "" class MacPrefixHeader(object): - """A class that helps with emulating Xcode's GCC_PREFIX_HEADER feature. + """A class that helps with emulating Xcode's GCC_PREFIX_HEADER feature. This feature consists of several pieces: * If GCC_PREFIX_HEADER is present, all compilations in that project get an @@ -1301,9 +1388,11 @@ class MacPrefixHeader(object): system for writing dependencies to the gch files, for writing build commands for the gch files, and for figuring out the location of the gch files. """ - def __init__(self, xcode_settings, - gyp_path_to_build_path, gyp_path_to_build_output): - """If xcode_settings is None, all methods on this class are no-ops. + + def __init__( + self, xcode_settings, gyp_path_to_build_path, gyp_path_to_build_output + ): + """If xcode_settings is None, all methods on this class are no-ops. Args: gyp_path_to_build_path: A function that takes a gyp-relative path, @@ -1313,201 +1402,210 @@ def __init__(self, xcode_settings, to where the output of precompiling that path for that language should be placed (without the trailing '.gch'). """ - # This doesn't support per-configuration prefix headers. Good enough - # for now. - self.header = None - self.compile_headers = False - if xcode_settings: - self.header = xcode_settings.GetPerTargetSetting('GCC_PREFIX_HEADER') - self.compile_headers = xcode_settings.GetPerTargetSetting( - 'GCC_PRECOMPILE_PREFIX_HEADER', default='NO') != 'NO' - self.compiled_headers = {} - if self.header: - if self.compile_headers: - for lang in ['c', 'cc', 'm', 'mm']: - self.compiled_headers[lang] = gyp_path_to_build_output( - self.header, lang) - self.header = gyp_path_to_build_path(self.header) - - def _CompiledHeader(self, lang, arch): - assert self.compile_headers - h = self.compiled_headers[lang] - if arch: - h += '.' + arch - return h - - def GetInclude(self, lang, arch=None): - """Gets the cflags to include the prefix header for language |lang|.""" - if self.compile_headers and lang in self.compiled_headers: - return '-include %s' % self._CompiledHeader(lang, arch) - elif self.header: - return '-include %s' % self.header - else: - return '' + # This doesn't support per-configuration prefix headers. Good enough + # for now. + self.header = None + self.compile_headers = False + if xcode_settings: + self.header = xcode_settings.GetPerTargetSetting("GCC_PREFIX_HEADER") + self.compile_headers = ( + xcode_settings.GetPerTargetSetting( + "GCC_PRECOMPILE_PREFIX_HEADER", default="NO" + ) + != "NO" + ) + self.compiled_headers = {} + if self.header: + if self.compile_headers: + for lang in ["c", "cc", "m", "mm"]: + self.compiled_headers[lang] = gyp_path_to_build_output( + self.header, lang + ) + self.header = gyp_path_to_build_path(self.header) + + def _CompiledHeader(self, lang, arch): + assert self.compile_headers + h = self.compiled_headers[lang] + if arch: + h += "." + arch + return h + + def GetInclude(self, lang, arch=None): + """Gets the cflags to include the prefix header for language |lang|.""" + if self.compile_headers and lang in self.compiled_headers: + return "-include %s" % self._CompiledHeader(lang, arch) + elif self.header: + return "-include %s" % self.header + else: + return "" - def _Gch(self, lang, arch): - """Returns the actual file name of the prefix header for language |lang|.""" - assert self.compile_headers - return self._CompiledHeader(lang, arch) + '.gch' + def _Gch(self, lang, arch): + """Returns the actual file name of the prefix header for language |lang|.""" + assert self.compile_headers + return self._CompiledHeader(lang, arch) + ".gch" - def GetObjDependencies(self, sources, objs, arch=None): - """Given a list of source files and the corresponding object files, returns + def GetObjDependencies(self, sources, objs, arch=None): + """Given a list of source files and the corresponding object files, returns a list of (source, object, gch) tuples, where |gch| is the build-directory relative path to the gch file each object file depends on. |compilable[i]| has to be the source file belonging to |objs[i]|.""" - if not self.header or not self.compile_headers: - return [] - - result = [] - for source, obj in zip(sources, objs): - ext = os.path.splitext(source)[1] - lang = { - '.c': 'c', - '.cpp': 'cc', '.cc': 'cc', '.cxx': 'cc', - '.m': 'm', - '.mm': 'mm', - }.get(ext, None) - if lang: - result.append((source, obj, self._Gch(lang, arch))) - return result - - def GetPchBuildCommands(self, arch=None): - """Returns [(path_to_gch, language_flag, language, header)]. + if not self.header or not self.compile_headers: + return [] + + result = [] + for source, obj in zip(sources, objs): + ext = os.path.splitext(source)[1] + lang = { + ".c": "c", + ".cpp": "cc", + ".cc": "cc", + ".cxx": "cc", + ".m": "m", + ".mm": "mm", + }.get(ext, None) + if lang: + result.append((source, obj, self._Gch(lang, arch))) + return result + + def GetPchBuildCommands(self, arch=None): + """Returns [(path_to_gch, language_flag, language, header)]. |path_to_gch| and |header| are relative to the build directory. """ - if not self.header or not self.compile_headers: - return [] - return [ - (self._Gch('c', arch), '-x c-header', 'c', self.header), - (self._Gch('cc', arch), '-x c++-header', 'cc', self.header), - (self._Gch('m', arch), '-x objective-c-header', 'm', self.header), - (self._Gch('mm', arch), '-x objective-c++-header', 'mm', self.header), - ] + if not self.header or not self.compile_headers: + return [] + return [ + (self._Gch("c", arch), "-x c-header", "c", self.header), + (self._Gch("cc", arch), "-x c++-header", "cc", self.header), + (self._Gch("m", arch), "-x objective-c-header", "m", self.header), + (self._Gch("mm", arch), "-x objective-c++-header", "mm", self.header), + ] def XcodeVersion(): - """Returns a tuple of version and build version of installed Xcode.""" - # `xcodebuild -version` output looks like - # Xcode 4.6.3 - # Build version 4H1503 - # or like - # Xcode 3.2.6 - # Component versions: DevToolsCore-1809.0; DevToolsSupport-1806.0 - # BuildVersion: 10M2518 - # Convert that to ('0463', '4H1503') or ('0326', '10M2518'). - global XCODE_VERSION_CACHE - if XCODE_VERSION_CACHE: + """Returns a tuple of version and build version of installed Xcode.""" + # `xcodebuild -version` output looks like + # Xcode 4.6.3 + # Build version 4H1503 + # or like + # Xcode 3.2.6 + # Component versions: DevToolsCore-1809.0; DevToolsSupport-1806.0 + # BuildVersion: 10M2518 + # Convert that to ('0463', '4H1503') or ('0326', '10M2518'). + global XCODE_VERSION_CACHE + if XCODE_VERSION_CACHE: + return XCODE_VERSION_CACHE + version = "" + build = "" + try: + version_list = GetStdoutQuiet(["xcodebuild", "-version"]).splitlines() + # In some circumstances xcodebuild exits 0 but doesn't return + # the right results; for example, a user on 10.7 or 10.8 with + # a bogus path set via xcode-select + # In that case this may be a CLT-only install so fall back to + # checking that version. + if len(version_list) < 2: + raise GypError("xcodebuild returned unexpected results") + version = version_list[0].split()[-1] # Last word on first line + build = version_list[-1].split()[-1] # Last word on last line + except GypError: # Xcode not installed so look for XCode Command Line Tools + version = CLTVersion() # macOS Catalina returns 11.0.0.0.1.1567737322 + if not version: + raise GypError("No Xcode or CLT version detected!") + # Be careful to convert "4.2.3" to "0423" and "11.0.0" to "1100": + version = version.split(".")[:3] # Just major, minor, micro + version[0] = version[0].zfill(2) # Add a leading zero if major is one digit + version = ("".join(version) + "00")[:4] # Limit to exactly four characters + XCODE_VERSION_CACHE = (version, build) return XCODE_VERSION_CACHE - version = "" - build = "" - try: - version_list = GetStdoutQuiet(['xcodebuild', '-version']).splitlines() - # In some circumstances xcodebuild exits 0 but doesn't return - # the right results; for example, a user on 10.7 or 10.8 with - # a bogus path set via xcode-select - # In that case this may be a CLT-only install so fall back to - # checking that version. - if len(version_list) < 2: - raise GypError("xcodebuild returned unexpected results") - version = version_list[0].split()[-1] # Last word on first line - build = version_list[-1].split()[-1] # Last word on last line - except GypError: # Xcode not installed so look for XCode Command Line Tools - version = CLTVersion() # macOS Catalina returns 11.0.0.0.1.1567737322 - if not version: - raise GypError("No Xcode or CLT version detected!") - # Be careful to convert "4.2.3" to "0423" and "11.0.0" to "1100": - version = version.split(".")[:3] # Just major, minor, micro - version[0] = version[0].zfill(2) # Add a leading zero if major is one digit - version = ("".join(version) + "00")[:4] # Limit to exactly four characters - XCODE_VERSION_CACHE = (version, build) - return XCODE_VERSION_CACHE # This function ported from the logic in Homebrew's CLT version check def CLTVersion(): - """Returns the version of command-line tools from pkgutil.""" - # pkgutil output looks like - # package-id: com.apple.pkg.CLTools_Executables - # version: 5.0.1.0.1.1382131676 - # volume: / - # location: / - # install-time: 1382544035 - # groups: com.apple.FindSystemFiles.pkg-group com.apple.DevToolsBoth.pkg-group com.apple.DevToolsNonRelocatableShared.pkg-group - STANDALONE_PKG_ID = "com.apple.pkg.DeveloperToolsCLILeo" - FROM_XCODE_PKG_ID = "com.apple.pkg.DeveloperToolsCLI" - MAVERICKS_PKG_ID = "com.apple.pkg.CLTools_Executables" - - regex = re.compile('version: (?P.+)') - for key in [MAVERICKS_PKG_ID, STANDALONE_PKG_ID, FROM_XCODE_PKG_ID]: - try: - output = GetStdout(['/usr/sbin/pkgutil', '--pkg-info', key]) - return re.search(regex, output).groupdict()['version'] - except GypError: - continue + """Returns the version of command-line tools from pkgutil.""" + # pkgutil output looks like + # package-id: com.apple.pkg.CLTools_Executables + # version: 5.0.1.0.1.1382131676 + # volume: / + # location: / + # install-time: 1382544035 + # groups: com.apple.FindSystemFiles.pkg-group com.apple.DevToolsBoth.pkg-group com.apple.DevToolsNonRelocatableShared.pkg-group + STANDALONE_PKG_ID = "com.apple.pkg.DeveloperToolsCLILeo" + FROM_XCODE_PKG_ID = "com.apple.pkg.DeveloperToolsCLI" + MAVERICKS_PKG_ID = "com.apple.pkg.CLTools_Executables" + + regex = re.compile("version: (?P.+)") + for key in [MAVERICKS_PKG_ID, STANDALONE_PKG_ID, FROM_XCODE_PKG_ID]: + try: + output = GetStdout(["/usr/sbin/pkgutil", "--pkg-info", key]) + return re.search(regex, output).groupdict()["version"] + except GypError: + continue def GetStdoutQuiet(cmdlist): - """Returns the content of standard output returned by invoking |cmdlist|. + """Returns the content of standard output returned by invoking |cmdlist|. Ignores the stderr. Raises |GypError| if the command return with a non-zero return code.""" - job = subprocess.Popen(cmdlist, stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - out = job.communicate()[0] - if PY3: - out = out.decode("utf-8") - if job.returncode != 0: - raise GypError('Error %d running %s' % (job.returncode, cmdlist[0])) - return out.rstrip('\n') + job = subprocess.Popen(cmdlist, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + out = job.communicate()[0] + if PY3: + out = out.decode("utf-8") + if job.returncode != 0: + raise GypError("Error %d running %s" % (job.returncode, cmdlist[0])) + return out.rstrip("\n") def GetStdout(cmdlist): - """Returns the content of standard output returned by invoking |cmdlist|. + """Returns the content of standard output returned by invoking |cmdlist|. Raises |GypError| if the command return with a non-zero return code.""" - job = subprocess.Popen(cmdlist, stdout=subprocess.PIPE) - out = job.communicate()[0] - if PY3: - out = out.decode("utf-8") - if job.returncode != 0: - sys.stderr.write(out + '\n') - raise GypError('Error %d running %s' % (job.returncode, cmdlist[0])) - return out.rstrip('\n') + job = subprocess.Popen(cmdlist, stdout=subprocess.PIPE) + out = job.communicate()[0] + if PY3: + out = out.decode("utf-8") + if job.returncode != 0: + sys.stderr.write(out + "\n") + raise GypError("Error %d running %s" % (job.returncode, cmdlist[0])) + return out.rstrip("\n") def MergeGlobalXcodeSettingsToSpec(global_dict, spec): - """Merges the global xcode_settings dictionary into each configuration of the + """Merges the global xcode_settings dictionary into each configuration of the target represented by spec. For keys that are both in the global and the local xcode_settings dict, the local key gets precedence. """ - # The xcode generator special-cases global xcode_settings and does something - # that amounts to merging in the global xcode_settings into each local - # xcode_settings dict. - global_xcode_settings = global_dict.get('xcode_settings', {}) - for config in spec['configurations'].values(): - if 'xcode_settings' in config: - new_settings = global_xcode_settings.copy() - new_settings.update(config['xcode_settings']) - config['xcode_settings'] = new_settings + # The xcode generator special-cases global xcode_settings and does something + # that amounts to merging in the global xcode_settings into each local + # xcode_settings dict. + global_xcode_settings = global_dict.get("xcode_settings", {}) + for config in spec["configurations"].values(): + if "xcode_settings" in config: + new_settings = global_xcode_settings.copy() + new_settings.update(config["xcode_settings"]) + config["xcode_settings"] = new_settings def IsMacBundle(flavor, spec): - """Returns if |spec| should be treated as a bundle. + """Returns if |spec| should be treated as a bundle. Bundles are directories with a certain subdirectory structure, instead of just a single file. Bundle rules do not produce a binary but also package resources into that directory.""" - is_mac_bundle = int(spec.get('mac_xctest_bundle', 0)) != 0 or \ - int(spec.get('mac_xcuitest_bundle', 0)) != 0 or \ - (int(spec.get('mac_bundle', 0)) != 0 and flavor == 'mac') + is_mac_bundle = ( + int(spec.get("mac_xctest_bundle", 0)) != 0 + or int(spec.get("mac_xcuitest_bundle", 0)) != 0 + or (int(spec.get("mac_bundle", 0)) != 0 and flavor == "mac") + ) - if is_mac_bundle: - assert spec['type'] != 'none', ( - 'mac_bundle targets cannot have type none (target "%s")' % - spec['target_name']) - return is_mac_bundle + if is_mac_bundle: + assert spec["type"] != "none", ( + 'mac_bundle targets cannot have type none (target "%s")' + % spec["target_name"] + ) + return is_mac_bundle def GetMacBundleResources(product_dir, xcode_settings, resources): - """Yields (output, resource) pairs for every resource in |resources|. + """Yields (output, resource) pairs for every resource in |resources|. Only call this for mac bundle targets. Args: @@ -1516,38 +1614,36 @@ def GetMacBundleResources(product_dir, xcode_settings, resources): xcode_settings: The XcodeSettings of the current target. resources: A list of bundle resources, relative to the build directory. """ - dest = os.path.join(product_dir, - xcode_settings.GetBundleResourceFolder()) - for res in resources: - output = dest + dest = os.path.join(product_dir, xcode_settings.GetBundleResourceFolder()) + for res in resources: + output = dest - # The make generator doesn't support it, so forbid it everywhere - # to keep the generators more interchangeable. - assert ' ' not in res, ( - "Spaces in resource filenames not supported (%s)" % res) + # The make generator doesn't support it, so forbid it everywhere + # to keep the generators more interchangeable. + assert " " not in res, "Spaces in resource filenames not supported (%s)" % res - # Split into (path,file). - res_parts = os.path.split(res) + # Split into (path,file). + res_parts = os.path.split(res) - # Now split the path into (prefix,maybe.lproj). - lproj_parts = os.path.split(res_parts[0]) - # If the resource lives in a .lproj bundle, add that to the destination. - if lproj_parts[1].endswith('.lproj'): - output = os.path.join(output, lproj_parts[1]) + # Now split the path into (prefix,maybe.lproj). + lproj_parts = os.path.split(res_parts[0]) + # If the resource lives in a .lproj bundle, add that to the destination. + if lproj_parts[1].endswith(".lproj"): + output = os.path.join(output, lproj_parts[1]) - output = os.path.join(output, res_parts[1]) - # Compiled XIB files are referred to by .nib. - if output.endswith('.xib'): - output = os.path.splitext(output)[0] + '.nib' - # Compiled storyboard files are referred to by .storyboardc. - if output.endswith('.storyboard'): - output = os.path.splitext(output)[0] + '.storyboardc' + output = os.path.join(output, res_parts[1]) + # Compiled XIB files are referred to by .nib. + if output.endswith(".xib"): + output = os.path.splitext(output)[0] + ".nib" + # Compiled storyboard files are referred to by .storyboardc. + if output.endswith(".storyboard"): + output = os.path.splitext(output)[0] + ".storyboardc" - yield output, res + yield output, res def GetMacInfoPlist(product_dir, xcode_settings, gyp_path_to_build_path): - """Returns (info_plist, dest_plist, defines, extra_env), where: + """Returns (info_plist, dest_plist, defines, extra_env), where: * |info_plist| is the source plist path, relative to the build directory, * |dest_plist| is the destination plist path, relative to the @@ -1566,36 +1662,43 @@ def GetMacInfoPlist(product_dir, xcode_settings, gyp_path_to_build_path): gyp_to_build_path: A function that converts paths relative to the current gyp file to paths relative to the build directory. """ - info_plist = xcode_settings.GetPerTargetSetting('INFOPLIST_FILE') - if not info_plist: - return None, None, [], {} - - # The make generator doesn't support it, so forbid it everywhere - # to keep the generators more interchangeable. - assert ' ' not in info_plist, ( - "Spaces in Info.plist filenames not supported (%s)" % info_plist) - - info_plist = gyp_path_to_build_path(info_plist) + info_plist = xcode_settings.GetPerTargetSetting("INFOPLIST_FILE") + if not info_plist: + return None, None, [], {} - # If explicitly set to preprocess the plist, invoke the C preprocessor and - # specify any defines as -D flags. - if xcode_settings.GetPerTargetSetting( - 'INFOPLIST_PREPROCESS', default='NO') == 'YES': - # Create an intermediate file based on the path. - defines = shlex.split(xcode_settings.GetPerTargetSetting( - 'INFOPLIST_PREPROCESSOR_DEFINITIONS', default='')) - else: - defines = [] + # The make generator doesn't support it, so forbid it everywhere + # to keep the generators more interchangeable. + assert " " not in info_plist, ( + "Spaces in Info.plist filenames not supported (%s)" % info_plist + ) + + info_plist = gyp_path_to_build_path(info_plist) + + # If explicitly set to preprocess the plist, invoke the C preprocessor and + # specify any defines as -D flags. + if ( + xcode_settings.GetPerTargetSetting("INFOPLIST_PREPROCESS", default="NO") + == "YES" + ): + # Create an intermediate file based on the path. + defines = shlex.split( + xcode_settings.GetPerTargetSetting( + "INFOPLIST_PREPROCESSOR_DEFINITIONS", default="" + ) + ) + else: + defines = [] - dest_plist = os.path.join(product_dir, xcode_settings.GetBundlePlistPath()) - extra_env = xcode_settings.GetPerTargetSettings() + dest_plist = os.path.join(product_dir, xcode_settings.GetBundlePlistPath()) + extra_env = xcode_settings.GetPerTargetSettings() - return info_plist, dest_plist, defines, extra_env + return info_plist, dest_plist, defines, extra_env -def _GetXcodeEnv(xcode_settings, built_products_dir, srcroot, configuration, - additional_settings=None): - """Return the environment variables that Xcode would set. See +def _GetXcodeEnv( + xcode_settings, built_products_dir, srcroot, configuration, additional_settings=None +): + """Return the environment variables that Xcode would set. See http://developer.apple.com/library/mac/#documentation/DeveloperTools/Reference/XcodeBuildSettingRef/1-Build_Setting_Reference/build_setting_ref.html#//apple_ref/doc/uid/TP40003931-CH3-SW153 for a full list. @@ -1609,209 +1712,220 @@ def _GetXcodeEnv(xcode_settings, built_products_dir, srcroot, configuration, result. """ - if not xcode_settings: return {} - - # This function is considered a friend of XcodeSettings, so let it reach into - # its implementation details. - spec = xcode_settings.spec - - # These are filled in on an as-needed basis. - env = { - 'BUILT_FRAMEWORKS_DIR' : built_products_dir, - 'BUILT_PRODUCTS_DIR' : built_products_dir, - 'CONFIGURATION' : configuration, - 'PRODUCT_NAME' : xcode_settings.GetProductName(), - # See /Developer/Platforms/MacOSX.platform/Developer/Library/Xcode/Specifications/MacOSX\ Product\ Types.xcspec for FULL_PRODUCT_NAME - 'SRCROOT' : srcroot, - 'SOURCE_ROOT': '${SRCROOT}', - # This is not true for static libraries, but currently the env is only - # written for bundles: - 'TARGET_BUILD_DIR' : built_products_dir, - 'TEMP_DIR' : '${TMPDIR}', - 'XCODE_VERSION_ACTUAL' : XcodeVersion()[0], - } - if xcode_settings.GetPerConfigSetting('SDKROOT', configuration): - env['SDKROOT'] = xcode_settings._SdkPath(configuration) - else: - env['SDKROOT'] = '' - - if xcode_settings.mac_toolchain_dir: - env['DEVELOPER_DIR'] = xcode_settings.mac_toolchain_dir - - if spec['type'] in ( - 'executable', 'static_library', 'shared_library', 'loadable_module'): - env['EXECUTABLE_NAME'] = xcode_settings.GetExecutableName() - env['EXECUTABLE_PATH'] = xcode_settings.GetExecutablePath() - env['FULL_PRODUCT_NAME'] = xcode_settings.GetFullProductName() - mach_o_type = xcode_settings.GetMachOType() - if mach_o_type: - env['MACH_O_TYPE'] = mach_o_type - env['PRODUCT_TYPE'] = xcode_settings.GetProductType() - if xcode_settings._IsBundle(): - # xcodeproj_file.py sets the same Xcode subfolder value for this as for - # FRAMEWORKS_FOLDER_PATH so Xcode builds will actually use FFP's value. - env['BUILT_FRAMEWORKS_DIR'] = \ - os.path.join(built_products_dir + os.sep \ - + xcode_settings.GetBundleFrameworksFolderPath()) - env['CONTENTS_FOLDER_PATH'] = \ - xcode_settings.GetBundleContentsFolderPath() - env['EXECUTABLE_FOLDER_PATH'] = \ - xcode_settings.GetBundleExecutableFolderPath() - env['UNLOCALIZED_RESOURCES_FOLDER_PATH'] = \ - xcode_settings.GetBundleResourceFolder() - env['JAVA_FOLDER_PATH'] = xcode_settings.GetBundleJavaFolderPath() - env['FRAMEWORKS_FOLDER_PATH'] = \ - xcode_settings.GetBundleFrameworksFolderPath() - env['SHARED_FRAMEWORKS_FOLDER_PATH'] = \ - xcode_settings.GetBundleSharedFrameworksFolderPath() - env['SHARED_SUPPORT_FOLDER_PATH'] = \ - xcode_settings.GetBundleSharedSupportFolderPath() - env['PLUGINS_FOLDER_PATH'] = xcode_settings.GetBundlePlugInsFolderPath() - env['XPCSERVICES_FOLDER_PATH'] = \ - xcode_settings.GetBundleXPCServicesFolderPath() - env['INFOPLIST_PATH'] = xcode_settings.GetBundlePlistPath() - env['WRAPPER_NAME'] = xcode_settings.GetWrapperName() - - install_name = xcode_settings.GetInstallName() - if install_name: - env['LD_DYLIB_INSTALL_NAME'] = install_name - install_name_base = xcode_settings.GetInstallNameBase() - if install_name_base: - env['DYLIB_INSTALL_NAME_BASE'] = install_name_base - xcode_version, _ = XcodeVersion() - if xcode_version >= '0500' and not env.get('SDKROOT'): - sdk_root = xcode_settings._SdkRoot(configuration) - if not sdk_root: - sdk_root = xcode_settings._XcodeSdkPath('') - if sdk_root is None: - sdk_root = '' - env['SDKROOT'] = sdk_root - - if not additional_settings: - additional_settings = {} - else: - # Flatten lists to strings. - for k in additional_settings: - if not isinstance(additional_settings[k], str): - additional_settings[k] = ' '.join(additional_settings[k]) - additional_settings.update(env) + if not xcode_settings: + return {} + + # This function is considered a friend of XcodeSettings, so let it reach into + # its implementation details. + spec = xcode_settings.spec + + # These are filled in on an as-needed basis. + env = { + "BUILT_FRAMEWORKS_DIR": built_products_dir, + "BUILT_PRODUCTS_DIR": built_products_dir, + "CONFIGURATION": configuration, + "PRODUCT_NAME": xcode_settings.GetProductName(), + # See /Developer/Platforms/MacOSX.platform/Developer/Library/Xcode/Specifications/MacOSX\ Product\ Types.xcspec for FULL_PRODUCT_NAME + "SRCROOT": srcroot, + "SOURCE_ROOT": "${SRCROOT}", + # This is not true for static libraries, but currently the env is only + # written for bundles: + "TARGET_BUILD_DIR": built_products_dir, + "TEMP_DIR": "${TMPDIR}", + "XCODE_VERSION_ACTUAL": XcodeVersion()[0], + } + if xcode_settings.GetPerConfigSetting("SDKROOT", configuration): + env["SDKROOT"] = xcode_settings._SdkPath(configuration) + else: + env["SDKROOT"] = "" + + if xcode_settings.mac_toolchain_dir: + env["DEVELOPER_DIR"] = xcode_settings.mac_toolchain_dir + + if spec["type"] in ( + "executable", + "static_library", + "shared_library", + "loadable_module", + ): + env["EXECUTABLE_NAME"] = xcode_settings.GetExecutableName() + env["EXECUTABLE_PATH"] = xcode_settings.GetExecutablePath() + env["FULL_PRODUCT_NAME"] = xcode_settings.GetFullProductName() + mach_o_type = xcode_settings.GetMachOType() + if mach_o_type: + env["MACH_O_TYPE"] = mach_o_type + env["PRODUCT_TYPE"] = xcode_settings.GetProductType() + if xcode_settings._IsBundle(): + # xcodeproj_file.py sets the same Xcode subfolder value for this as for + # FRAMEWORKS_FOLDER_PATH so Xcode builds will actually use FFP's value. + env["BUILT_FRAMEWORKS_DIR"] = os.path.join( + built_products_dir + os.sep + xcode_settings.GetBundleFrameworksFolderPath() + ) + env["CONTENTS_FOLDER_PATH"] = xcode_settings.GetBundleContentsFolderPath() + env["EXECUTABLE_FOLDER_PATH"] = xcode_settings.GetBundleExecutableFolderPath() + env[ + "UNLOCALIZED_RESOURCES_FOLDER_PATH" + ] = xcode_settings.GetBundleResourceFolder() + env["JAVA_FOLDER_PATH"] = xcode_settings.GetBundleJavaFolderPath() + env["FRAMEWORKS_FOLDER_PATH"] = xcode_settings.GetBundleFrameworksFolderPath() + env[ + "SHARED_FRAMEWORKS_FOLDER_PATH" + ] = xcode_settings.GetBundleSharedFrameworksFolderPath() + env[ + "SHARED_SUPPORT_FOLDER_PATH" + ] = xcode_settings.GetBundleSharedSupportFolderPath() + env["PLUGINS_FOLDER_PATH"] = xcode_settings.GetBundlePlugInsFolderPath() + env["XPCSERVICES_FOLDER_PATH"] = xcode_settings.GetBundleXPCServicesFolderPath() + env["INFOPLIST_PATH"] = xcode_settings.GetBundlePlistPath() + env["WRAPPER_NAME"] = xcode_settings.GetWrapperName() + + install_name = xcode_settings.GetInstallName() + if install_name: + env["LD_DYLIB_INSTALL_NAME"] = install_name + install_name_base = xcode_settings.GetInstallNameBase() + if install_name_base: + env["DYLIB_INSTALL_NAME_BASE"] = install_name_base + xcode_version, _ = XcodeVersion() + if xcode_version >= "0500" and not env.get("SDKROOT"): + sdk_root = xcode_settings._SdkRoot(configuration) + if not sdk_root: + sdk_root = xcode_settings._XcodeSdkPath("") + if sdk_root is None: + sdk_root = "" + env["SDKROOT"] = sdk_root + + if not additional_settings: + additional_settings = {} + else: + # Flatten lists to strings. + for k in additional_settings: + if not isinstance(additional_settings[k], str): + additional_settings[k] = " ".join(additional_settings[k]) + additional_settings.update(env) - for k in additional_settings: - additional_settings[k] = _NormalizeEnvVarReferences(additional_settings[k]) + for k in additional_settings: + additional_settings[k] = _NormalizeEnvVarReferences(additional_settings[k]) - return additional_settings + return additional_settings def _NormalizeEnvVarReferences(str): - """Takes a string containing variable references in the form ${FOO}, $(FOO), + """Takes a string containing variable references in the form ${FOO}, $(FOO), or $FOO, and returns a string with all variable references in the form ${FOO}. """ - # $FOO -> ${FOO} - str = re.sub(r'\$([a-zA-Z_][a-zA-Z0-9_]*)', r'${\1}', str) + # $FOO -> ${FOO} + str = re.sub(r"\$([a-zA-Z_][a-zA-Z0-9_]*)", r"${\1}", str) - # $(FOO) -> ${FOO} - matches = re.findall(r'(\$\(([a-zA-Z0-9\-_]+)\))', str) - for match in matches: - to_replace, variable = match - assert '$(' not in match, '$($(FOO)) variables not supported: ' + match - str = str.replace(to_replace, '${' + variable + '}') + # $(FOO) -> ${FOO} + matches = re.findall(r"(\$\(([a-zA-Z0-9\-_]+)\))", str) + for match in matches: + to_replace, variable = match + assert "$(" not in match, "$($(FOO)) variables not supported: " + match + str = str.replace(to_replace, "${" + variable + "}") - return str + return str def ExpandEnvVars(string, expansions): - """Expands ${VARIABLES}, $(VARIABLES), and $VARIABLES in string per the + """Expands ${VARIABLES}, $(VARIABLES), and $VARIABLES in string per the expansions list. If the variable expands to something that references another variable, this variable is expanded as well if it's in env -- until no variables present in env are left.""" - for k, v in reversed(expansions): - string = string.replace('${' + k + '}', v) - string = string.replace('$(' + k + ')', v) - string = string.replace('$' + k, v) - return string + for k, v in reversed(expansions): + string = string.replace("${" + k + "}", v) + string = string.replace("$(" + k + ")", v) + string = string.replace("$" + k, v) + return string def _TopologicallySortedEnvVarKeys(env): - """Takes a dict |env| whose values are strings that can refer to other keys, + """Takes a dict |env| whose values are strings that can refer to other keys, for example env['foo'] = '$(bar) and $(baz)'. Returns a list L of all keys of env such that key2 is after key1 in L if env[key2] refers to env[key1]. Throws an Exception in case of dependency cycles. """ - # Since environment variables can refer to other variables, the evaluation - # order is important. Below is the logic to compute the dependency graph - # and sort it. - regex = re.compile(r'\$\{([a-zA-Z0-9\-_]+)\}') - def GetEdges(node): - # Use a definition of edges such that user_of_variable -> used_varible. - # This happens to be easier in this case, since a variable's - # definition contains all variables it references in a single string. - # We can then reverse the result of the topological sort at the end. - # Since: reverse(topsort(DAG)) = topsort(reverse_edges(DAG)) - matches = set([v for v in regex.findall(env[node]) if v in env]) - for dependee in matches: - assert '${' not in dependee, 'Nested variables not supported: ' + dependee - return matches - - try: - # Topologically sort, and then reverse, because we used an edge definition - # that's inverted from the expected result of this function (see comment - # above). - order = gyp.common.TopologicallySorted(env.keys(), GetEdges) - order.reverse() - return order - except gyp.common.CycleError as e: - raise GypError( - 'Xcode environment variables are cyclically dependent: ' + str(e.nodes)) - - -def GetSortedXcodeEnv(xcode_settings, built_products_dir, srcroot, - configuration, additional_settings=None): - env = _GetXcodeEnv(xcode_settings, built_products_dir, srcroot, configuration, - additional_settings) - return [(key, env[key]) for key in _TopologicallySortedEnvVarKeys(env)] + # Since environment variables can refer to other variables, the evaluation + # order is important. Below is the logic to compute the dependency graph + # and sort it. + regex = re.compile(r"\$\{([a-zA-Z0-9\-_]+)\}") + + def GetEdges(node): + # Use a definition of edges such that user_of_variable -> used_varible. + # This happens to be easier in this case, since a variable's + # definition contains all variables it references in a single string. + # We can then reverse the result of the topological sort at the end. + # Since: reverse(topsort(DAG)) = topsort(reverse_edges(DAG)) + matches = set([v for v in regex.findall(env[node]) if v in env]) + for dependee in matches: + assert "${" not in dependee, "Nested variables not supported: " + dependee + return matches + + try: + # Topologically sort, and then reverse, because we used an edge definition + # that's inverted from the expected result of this function (see comment + # above). + order = gyp.common.TopologicallySorted(env.keys(), GetEdges) + order.reverse() + return order + except gyp.common.CycleError as e: + raise GypError( + "Xcode environment variables are cyclically dependent: " + str(e.nodes) + ) + + +def GetSortedXcodeEnv( + xcode_settings, built_products_dir, srcroot, configuration, additional_settings=None +): + env = _GetXcodeEnv( + xcode_settings, built_products_dir, srcroot, configuration, additional_settings + ) + return [(key, env[key]) for key in _TopologicallySortedEnvVarKeys(env)] def GetSpecPostbuildCommands(spec, quiet=False): - """Returns the list of postbuilds explicitly defined on |spec|, in a form + """Returns the list of postbuilds explicitly defined on |spec|, in a form executable by a shell.""" - postbuilds = [] - for postbuild in spec.get('postbuilds', []): - if not quiet: - postbuilds.append('echo POSTBUILD\\(%s\\) %s' % ( - spec['target_name'], postbuild['postbuild_name'])) - postbuilds.append(gyp.common.EncodePOSIXShellList(postbuild['action'])) - return postbuilds + postbuilds = [] + for postbuild in spec.get("postbuilds", []): + if not quiet: + postbuilds.append( + "echo POSTBUILD\\(%s\\) %s" + % (spec["target_name"], postbuild["postbuild_name"]) + ) + postbuilds.append(gyp.common.EncodePOSIXShellList(postbuild["action"])) + return postbuilds def _HasIOSTarget(targets): - """Returns true if any target contains the iOS specific key + """Returns true if any target contains the iOS specific key IPHONEOS_DEPLOYMENT_TARGET.""" - for target_dict in targets.values(): - for config in target_dict['configurations'].values(): - if config.get('xcode_settings', {}).get('IPHONEOS_DEPLOYMENT_TARGET'): - return True - return False + for target_dict in targets.values(): + for config in target_dict["configurations"].values(): + if config.get("xcode_settings", {}).get("IPHONEOS_DEPLOYMENT_TARGET"): + return True + return False def _AddIOSDeviceConfigurations(targets): - """Clone all targets and append -iphoneos to the name. Configure these targets + """Clone all targets and append -iphoneos to the name. Configure these targets to build for iOS devices and use correct architectures for those builds.""" - for target_dict in targets.values(): - toolset = target_dict['toolset'] - configs = target_dict['configurations'] - for config_name, simulator_config_dict in dict(configs).items(): - iphoneos_config_dict = copy.deepcopy(simulator_config_dict) - configs[config_name + '-iphoneos'] = iphoneos_config_dict - configs[config_name + '-iphonesimulator'] = simulator_config_dict - if toolset == 'target': - simulator_config_dict['xcode_settings']['SDKROOT'] = 'iphonesimulator' - iphoneos_config_dict['xcode_settings']['SDKROOT'] = 'iphoneos' - return targets + for target_dict in targets.values(): + toolset = target_dict["toolset"] + configs = target_dict["configurations"] + for config_name, simulator_config_dict in dict(configs).items(): + iphoneos_config_dict = copy.deepcopy(simulator_config_dict) + configs[config_name + "-iphoneos"] = iphoneos_config_dict + configs[config_name + "-iphonesimulator"] = simulator_config_dict + if toolset == "target": + simulator_config_dict["xcode_settings"]["SDKROOT"] = "iphonesimulator" + iphoneos_config_dict["xcode_settings"]["SDKROOT"] = "iphoneos" + return targets + def CloneConfigurationForDeviceAndEmulator(target_dicts): - """If |target_dicts| contains any iOS targets, automatically create -iphoneos + """If |target_dicts| contains any iOS targets, automatically create -iphoneos targets for iOS device builds.""" - if _HasIOSTarget(target_dicts): - return _AddIOSDeviceConfigurations(target_dicts) - return target_dicts + if _HasIOSTarget(target_dicts): + return _AddIOSDeviceConfigurations(target_dicts) + return target_dicts diff --git a/gyp/pylib/gyp/xcode_ninja.py b/gyp/pylib/gyp/xcode_ninja.py index 2bc2143340..10ddcbccd0 100644 --- a/gyp/pylib/gyp/xcode_ninja.py +++ b/gyp/pylib/gyp/xcode_ninja.py @@ -20,116 +20,122 @@ def _WriteWorkspace(main_gyp, sources_gyp, params): - """ Create a workspace to wrap main and sources gyp paths. """ - (build_file_root, build_file_ext) = os.path.splitext(main_gyp) - workspace_path = build_file_root + '.xcworkspace' - options = params['options'] - if options.generator_output: - workspace_path = os.path.join(options.generator_output, workspace_path) - try: - os.makedirs(workspace_path) - except OSError as e: - if e.errno != errno.EEXIST: - raise - output_string = '\n' + \ - '\n' - for gyp_name in [main_gyp, sources_gyp]: - name = os.path.splitext(os.path.basename(gyp_name))[0] + '.xcodeproj' - name = xml.sax.saxutils.quoteattr("group:" + name) - output_string += ' \n' % name - output_string += '\n' - - workspace_file = os.path.join(workspace_path, "contents.xcworkspacedata") - - try: - with open(workspace_file, 'r') as input_file: - input_string = input_file.read() - if input_string == output_string: - return - except IOError: - # Ignore errors if the file doesn't exist. - pass - - with open(workspace_file, 'w') as output_file: - output_file.write(output_string) + """ Create a workspace to wrap main and sources gyp paths. """ + (build_file_root, build_file_ext) = os.path.splitext(main_gyp) + workspace_path = build_file_root + ".xcworkspace" + options = params["options"] + if options.generator_output: + workspace_path = os.path.join(options.generator_output, workspace_path) + try: + os.makedirs(workspace_path) + except OSError as e: + if e.errno != errno.EEXIST: + raise + output_string = ( + '\n' + '\n' + ) + for gyp_name in [main_gyp, sources_gyp]: + name = os.path.splitext(os.path.basename(gyp_name))[0] + ".xcodeproj" + name = xml.sax.saxutils.quoteattr("group:" + name) + output_string += " \n" % name + output_string += "\n" + + workspace_file = os.path.join(workspace_path, "contents.xcworkspacedata") + + try: + with open(workspace_file, "r") as input_file: + input_string = input_file.read() + if input_string == output_string: + return + except IOError: + # Ignore errors if the file doesn't exist. + pass + + with open(workspace_file, "w") as output_file: + output_file.write(output_string) + def _TargetFromSpec(old_spec, params): - """ Create fake target for xcode-ninja wrapper. """ - # Determine ninja top level build dir (e.g. /path/to/out). - ninja_toplevel = None - jobs = 0 - if params: - options = params['options'] - ninja_toplevel = \ - os.path.join(options.toplevel_dir, - gyp.generator.ninja.ComputeOutputDir(params)) - jobs = params.get('generator_flags', {}).get('xcode_ninja_jobs', 0) - - target_name = old_spec.get('target_name') - product_name = old_spec.get('product_name', target_name) - product_extension = old_spec.get('product_extension') - - ninja_target = {} - ninja_target['target_name'] = target_name - ninja_target['product_name'] = product_name - if product_extension: - ninja_target['product_extension'] = product_extension - ninja_target['toolset'] = old_spec.get('toolset') - ninja_target['default_configuration'] = old_spec.get('default_configuration') - ninja_target['configurations'] = {} - - # Tell Xcode to look in |ninja_toplevel| for build products. - new_xcode_settings = {} - if ninja_toplevel: - new_xcode_settings['CONFIGURATION_BUILD_DIR'] = \ - "%s/$(CONFIGURATION)$(EFFECTIVE_PLATFORM_NAME)" % ninja_toplevel - - if 'configurations' in old_spec: - for config in old_spec['configurations']: - old_xcode_settings = \ - old_spec['configurations'][config].get('xcode_settings', {}) - if 'IPHONEOS_DEPLOYMENT_TARGET' in old_xcode_settings: - new_xcode_settings['CODE_SIGNING_REQUIRED'] = "NO" - new_xcode_settings['IPHONEOS_DEPLOYMENT_TARGET'] = \ - old_xcode_settings['IPHONEOS_DEPLOYMENT_TARGET'] - for key in ['BUNDLE_LOADER', 'TEST_HOST']: - if key in old_xcode_settings: - new_xcode_settings[key] = old_xcode_settings[key] - - ninja_target['configurations'][config] = {} - ninja_target['configurations'][config]['xcode_settings'] = \ - new_xcode_settings - - ninja_target['mac_bundle'] = old_spec.get('mac_bundle', 0) - ninja_target['mac_xctest_bundle'] = old_spec.get('mac_xctest_bundle', 0) - ninja_target['ios_app_extension'] = old_spec.get('ios_app_extension', 0) - ninja_target['ios_watchkit_extension'] = \ - old_spec.get('ios_watchkit_extension', 0) - ninja_target['ios_watchkit_app'] = old_spec.get('ios_watchkit_app', 0) - ninja_target['type'] = old_spec['type'] - if ninja_toplevel: - ninja_target['actions'] = [ - { - 'action_name': 'Compile and copy %s via ninja' % target_name, - 'inputs': [], - 'outputs': [], - 'action': [ - 'env', - 'PATH=%s' % os.environ['PATH'], - 'ninja', - '-C', - new_xcode_settings['CONFIGURATION_BUILD_DIR'], - target_name, - ], - 'message': 'Compile and copy %s via ninja' % target_name, - }, - ] - if jobs > 0: - ninja_target['actions'][0]['action'].extend(('-j', jobs)) - return ninja_target + """ Create fake target for xcode-ninja wrapper. """ + # Determine ninja top level build dir (e.g. /path/to/out). + ninja_toplevel = None + jobs = 0 + if params: + options = params["options"] + ninja_toplevel = os.path.join( + options.toplevel_dir, gyp.generator.ninja.ComputeOutputDir(params) + ) + jobs = params.get("generator_flags", {}).get("xcode_ninja_jobs", 0) + + target_name = old_spec.get("target_name") + product_name = old_spec.get("product_name", target_name) + product_extension = old_spec.get("product_extension") + + ninja_target = {} + ninja_target["target_name"] = target_name + ninja_target["product_name"] = product_name + if product_extension: + ninja_target["product_extension"] = product_extension + ninja_target["toolset"] = old_spec.get("toolset") + ninja_target["default_configuration"] = old_spec.get("default_configuration") + ninja_target["configurations"] = {} + + # Tell Xcode to look in |ninja_toplevel| for build products. + new_xcode_settings = {} + if ninja_toplevel: + new_xcode_settings["CONFIGURATION_BUILD_DIR"] = ( + "%s/$(CONFIGURATION)$(EFFECTIVE_PLATFORM_NAME)" % ninja_toplevel + ) + + if "configurations" in old_spec: + for config in old_spec["configurations"]: + old_xcode_settings = old_spec["configurations"][config].get( + "xcode_settings", {} + ) + if "IPHONEOS_DEPLOYMENT_TARGET" in old_xcode_settings: + new_xcode_settings["CODE_SIGNING_REQUIRED"] = "NO" + new_xcode_settings["IPHONEOS_DEPLOYMENT_TARGET"] = old_xcode_settings[ + "IPHONEOS_DEPLOYMENT_TARGET" + ] + for key in ["BUNDLE_LOADER", "TEST_HOST"]: + if key in old_xcode_settings: + new_xcode_settings[key] = old_xcode_settings[key] + + ninja_target["configurations"][config] = {} + ninja_target["configurations"][config][ + "xcode_settings" + ] = new_xcode_settings + + ninja_target["mac_bundle"] = old_spec.get("mac_bundle", 0) + ninja_target["mac_xctest_bundle"] = old_spec.get("mac_xctest_bundle", 0) + ninja_target["ios_app_extension"] = old_spec.get("ios_app_extension", 0) + ninja_target["ios_watchkit_extension"] = old_spec.get("ios_watchkit_extension", 0) + ninja_target["ios_watchkit_app"] = old_spec.get("ios_watchkit_app", 0) + ninja_target["type"] = old_spec["type"] + if ninja_toplevel: + ninja_target["actions"] = [ + { + "action_name": "Compile and copy %s via ninja" % target_name, + "inputs": [], + "outputs": [], + "action": [ + "env", + "PATH=%s" % os.environ["PATH"], + "ninja", + "-C", + new_xcode_settings["CONFIGURATION_BUILD_DIR"], + target_name, + ], + "message": "Compile and copy %s via ninja" % target_name, + }, + ] + if jobs > 0: + ninja_target["actions"][0]["action"].extend(("-j", jobs)) + return ninja_target + def IsValidTargetForWrapper(target_extras, executable_target_pattern, spec): - """Limit targets for Xcode wrapper. + """Limit targets for Xcode wrapper. Xcode sometimes performs poorly with too many targets, so only include proper executable targets, with filters to customize. @@ -138,25 +144,27 @@ def IsValidTargetForWrapper(target_extras, executable_target_pattern, spec): executable_target_pattern: Regular expression limiting executable targets. spec: Specifications for target. """ - target_name = spec.get('target_name') - # Always include targets matching target_extras. - if target_extras is not None and re.search(target_extras, target_name): - return True - - # Otherwise just show executable targets and xc_tests. - if (int(spec.get('mac_xctest_bundle', 0)) != 0 or - (spec.get('type', '') == 'executable' and - spec.get('product_extension', '') != 'bundle')): - - # If there is a filter and the target does not match, exclude the target. - if executable_target_pattern is not None: - if not re.search(executable_target_pattern, target_name): - return False - return True - return False + target_name = spec.get("target_name") + # Always include targets matching target_extras. + if target_extras is not None and re.search(target_extras, target_name): + return True + + # Otherwise just show executable targets and xc_tests. + if int(spec.get("mac_xctest_bundle", 0)) != 0 or ( + spec.get("type", "") == "executable" + and spec.get("product_extension", "") != "bundle" + ): + + # If there is a filter and the target does not match, exclude the target. + if executable_target_pattern is not None: + if not re.search(executable_target_pattern, target_name): + return False + return True + return False + def CreateWrapper(target_list, target_dicts, data, params): - """Initialize targets for the ninja wrapper. + """Initialize targets for the ninja wrapper. This sets up the necessary variables in the targets to generate Xcode projects that use ninja as an external builder. @@ -166,124 +174,129 @@ def CreateWrapper(target_list, target_dicts, data, params): data: Dict of flattened build files keyed on gyp path. params: Dict of global options for gyp. """ - orig_gyp = params['build_files'][0] - for gyp_name, gyp_dict in data.items(): - if gyp_name == orig_gyp: - depth = gyp_dict['_DEPTH'] - - # Check for custom main gyp name, otherwise use the default CHROMIUM_GYP_FILE - # and prepend .ninja before the .gyp extension. - generator_flags = params.get('generator_flags', {}) - main_gyp = generator_flags.get('xcode_ninja_main_gyp', None) - if main_gyp is None: - (build_file_root, build_file_ext) = os.path.splitext(orig_gyp) - main_gyp = build_file_root + ".ninja" + build_file_ext - - # Create new |target_list|, |target_dicts| and |data| data structures. - new_target_list = [] - new_target_dicts = {} - new_data = {} - - # Set base keys needed for |data|. - new_data[main_gyp] = {} - new_data[main_gyp]['included_files'] = [] - new_data[main_gyp]['targets'] = [] - new_data[main_gyp]['xcode_settings'] = \ - data[orig_gyp].get('xcode_settings', {}) - - # Normally the xcode-ninja generator includes only valid executable targets. - # If |xcode_ninja_executable_target_pattern| is set, that list is reduced to - # executable targets that match the pattern. (Default all) - executable_target_pattern = \ - generator_flags.get('xcode_ninja_executable_target_pattern', None) - - # For including other non-executable targets, add the matching target name - # to the |xcode_ninja_target_pattern| regular expression. (Default none) - target_extras = generator_flags.get('xcode_ninja_target_pattern', None) - - for old_qualified_target in target_list: - spec = target_dicts[old_qualified_target] - if IsValidTargetForWrapper(target_extras, executable_target_pattern, spec): - # Add to new_target_list. - target_name = spec.get('target_name') - new_target_name = '%s:%s#target' % (main_gyp, target_name) - new_target_list.append(new_target_name) - - # Add to new_target_dicts. - new_target_dicts[new_target_name] = _TargetFromSpec(spec, params) - - # Add to new_data. - for old_target in data[old_qualified_target.split(':')[0]]['targets']: - if old_target['target_name'] == target_name: - new_data_target = {} - new_data_target['target_name'] = old_target['target_name'] - new_data_target['toolset'] = old_target['toolset'] - new_data[main_gyp]['targets'].append(new_data_target) - - # Create sources target. - sources_target_name = 'sources_for_indexing' - sources_target = _TargetFromSpec( - { 'target_name' : sources_target_name, - 'toolset': 'target', - 'default_configuration': 'Default', - 'mac_bundle': '0', - 'type': 'executable' - }, None) - - # Tell Xcode to look everywhere for headers. - sources_target['configurations'] = {'Default': { 'include_dirs': [ depth ] } } - - # Put excluded files into the sources target so they can be opened in Xcode. - skip_excluded_files = \ - not generator_flags.get('xcode_ninja_list_excluded_files', True) - - sources = [] - for target, target_dict in target_dicts.items(): - base = os.path.dirname(target) - files = target_dict.get('sources', []) + \ - target_dict.get('mac_bundle_resources', []) - - if not skip_excluded_files: - files.extend(target_dict.get('sources_excluded', []) + - target_dict.get('mac_bundle_resources_excluded', [])) - - for action in target_dict.get('actions', []): - files.extend(action.get('inputs', [])) - - if not skip_excluded_files: - files.extend(action.get('inputs_excluded', [])) - - # Remove files starting with $. These are mostly intermediate files for the - # build system. - files = [ file for file in files if not file.startswith('$')] - - # Make sources relative to root build file. - relative_path = os.path.dirname(main_gyp) - sources += [ os.path.relpath(os.path.join(base, file), relative_path) - for file in files ] - - sources_target['sources'] = sorted(set(sources)) - - # Put sources_to_index in it's own gyp. - sources_gyp = \ - os.path.join(os.path.dirname(main_gyp), sources_target_name + ".gyp") - fully_qualified_target_name = \ - '%s:%s#target' % (sources_gyp, sources_target_name) - - # Add to new_target_list, new_target_dicts and new_data. - new_target_list.append(fully_qualified_target_name) - new_target_dicts[fully_qualified_target_name] = sources_target - new_data_target = {} - new_data_target['target_name'] = sources_target['target_name'] - new_data_target['_DEPTH'] = depth - new_data_target['toolset'] = "target" - new_data[sources_gyp] = {} - new_data[sources_gyp]['targets'] = [] - new_data[sources_gyp]['included_files'] = [] - new_data[sources_gyp]['xcode_settings'] = \ - data[orig_gyp].get('xcode_settings', {}) - new_data[sources_gyp]['targets'].append(new_data_target) - - # Write workspace to file. - _WriteWorkspace(main_gyp, sources_gyp, params) - return (new_target_list, new_target_dicts, new_data) + orig_gyp = params["build_files"][0] + for gyp_name, gyp_dict in data.items(): + if gyp_name == orig_gyp: + depth = gyp_dict["_DEPTH"] + + # Check for custom main gyp name, otherwise use the default CHROMIUM_GYP_FILE + # and prepend .ninja before the .gyp extension. + generator_flags = params.get("generator_flags", {}) + main_gyp = generator_flags.get("xcode_ninja_main_gyp", None) + if main_gyp is None: + (build_file_root, build_file_ext) = os.path.splitext(orig_gyp) + main_gyp = build_file_root + ".ninja" + build_file_ext + + # Create new |target_list|, |target_dicts| and |data| data structures. + new_target_list = [] + new_target_dicts = {} + new_data = {} + + # Set base keys needed for |data|. + new_data[main_gyp] = {} + new_data[main_gyp]["included_files"] = [] + new_data[main_gyp]["targets"] = [] + new_data[main_gyp]["xcode_settings"] = data[orig_gyp].get("xcode_settings", {}) + + # Normally the xcode-ninja generator includes only valid executable targets. + # If |xcode_ninja_executable_target_pattern| is set, that list is reduced to + # executable targets that match the pattern. (Default all) + executable_target_pattern = generator_flags.get( + "xcode_ninja_executable_target_pattern", None + ) + + # For including other non-executable targets, add the matching target name + # to the |xcode_ninja_target_pattern| regular expression. (Default none) + target_extras = generator_flags.get("xcode_ninja_target_pattern", None) + + for old_qualified_target in target_list: + spec = target_dicts[old_qualified_target] + if IsValidTargetForWrapper(target_extras, executable_target_pattern, spec): + # Add to new_target_list. + target_name = spec.get("target_name") + new_target_name = "%s:%s#target" % (main_gyp, target_name) + new_target_list.append(new_target_name) + + # Add to new_target_dicts. + new_target_dicts[new_target_name] = _TargetFromSpec(spec, params) + + # Add to new_data. + for old_target in data[old_qualified_target.split(":")[0]]["targets"]: + if old_target["target_name"] == target_name: + new_data_target = {} + new_data_target["target_name"] = old_target["target_name"] + new_data_target["toolset"] = old_target["toolset"] + new_data[main_gyp]["targets"].append(new_data_target) + + # Create sources target. + sources_target_name = "sources_for_indexing" + sources_target = _TargetFromSpec( + { + "target_name": sources_target_name, + "toolset": "target", + "default_configuration": "Default", + "mac_bundle": "0", + "type": "executable", + }, + None, + ) + + # Tell Xcode to look everywhere for headers. + sources_target["configurations"] = {"Default": {"include_dirs": [depth]}} + + # Put excluded files into the sources target so they can be opened in Xcode. + skip_excluded_files = not generator_flags.get( + "xcode_ninja_list_excluded_files", True + ) + + sources = [] + for target, target_dict in target_dicts.items(): + base = os.path.dirname(target) + files = target_dict.get("sources", []) + target_dict.get( + "mac_bundle_resources", [] + ) + + if not skip_excluded_files: + files.extend( + target_dict.get("sources_excluded", []) + + target_dict.get("mac_bundle_resources_excluded", []) + ) + + for action in target_dict.get("actions", []): + files.extend(action.get("inputs", [])) + + if not skip_excluded_files: + files.extend(action.get("inputs_excluded", [])) + + # Remove files starting with $. These are mostly intermediate files for the + # build system. + files = [file for file in files if not file.startswith("$")] + + # Make sources relative to root build file. + relative_path = os.path.dirname(main_gyp) + sources += [ + os.path.relpath(os.path.join(base, file), relative_path) for file in files + ] + + sources_target["sources"] = sorted(set(sources)) + + # Put sources_to_index in it's own gyp. + sources_gyp = os.path.join(os.path.dirname(main_gyp), sources_target_name + ".gyp") + fully_qualified_target_name = "%s:%s#target" % (sources_gyp, sources_target_name) + + # Add to new_target_list, new_target_dicts and new_data. + new_target_list.append(fully_qualified_target_name) + new_target_dicts[fully_qualified_target_name] = sources_target + new_data_target = {} + new_data_target["target_name"] = sources_target["target_name"] + new_data_target["_DEPTH"] = depth + new_data_target["toolset"] = "target" + new_data[sources_gyp] = {} + new_data[sources_gyp]["targets"] = [] + new_data[sources_gyp]["included_files"] = [] + new_data[sources_gyp]["xcode_settings"] = data[orig_gyp].get("xcode_settings", {}) + new_data[sources_gyp]["targets"].append(new_data_target) + + # Write workspace to file. + _WriteWorkspace(main_gyp, sources_gyp, params) + return (new_target_list, new_target_dicts, new_data) diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py index 1e950dce8f..cde4f055f9 100644 --- a/gyp/pylib/gyp/xcodeproj_file.py +++ b/gyp/pylib/gyp/xcodeproj_file.py @@ -145,11 +145,12 @@ import sys try: - basestring, cmp, unicode + basestring, cmp, unicode except NameError: # Python 3 - basestring = unicode = str - def cmp(x, y): - return (x > y) - (x < y) + basestring = unicode = str + + def cmp(x, y): + return (x > y) - (x < y) # See XCObject._EncodeString. This pattern is used to determine when a string @@ -158,11 +159,11 @@ def cmp(x, y): # transformed to be properly encoded. Note that this expression matches the # characters listed with "+", for 1 or more occurrences: if a string is empty, # it must not match this pattern, because it needs to be encoded as "". -_unquoted = re.compile('^[A-Za-z0-9$./_]+$') +_unquoted = re.compile("^[A-Za-z0-9$./_]+$") # Strings that match this pattern are quoted regardless of what _unquoted says. # Oddly, Xcode will quote any string with a run of three or more underscores. -_quoted = re.compile('___') +_quoted = re.compile("___") # This pattern should match any character that needs to be escaped by # XCObject._EncodeString. See that function. @@ -170,10 +171,11 @@ def cmp(x, y): # Used by SourceTreeAndPathFromPath -_path_leading_variable = re.compile(r'^\$\((.*?)\)(/(.*))?$') +_path_leading_variable = re.compile(r"^\$\((.*?)\)(/(.*))?$") + def SourceTreeAndPathFromPath(input_path): - """Given input_path, returns a tuple with sourceTree and path values. + """Given input_path, returns a tuple with sourceTree and path values. Examples: input_path (source_tree, output_path) @@ -182,21 +184,23 @@ def SourceTreeAndPathFromPath(input_path): 'path' (None, 'path') """ - source_group_match = _path_leading_variable.match(input_path) - if source_group_match: - source_tree = source_group_match.group(1) - output_path = source_group_match.group(3) # This may be None. - else: - source_tree = None - output_path = input_path + source_group_match = _path_leading_variable.match(input_path) + if source_group_match: + source_tree = source_group_match.group(1) + output_path = source_group_match.group(3) # This may be None. + else: + source_tree = None + output_path = input_path + + return (source_tree, output_path) - return (source_tree, output_path) def ConvertVariablesToShellSyntax(input_string): - return re.sub(r'\$\((.*?)\)', '${\\1}', input_string) + return re.sub(r"\$\((.*?)\)", "${\\1}", input_string) + class XCObject(object): - """The abstract base of all class types used in Xcode project files. + """The abstract base of all class types used in Xcode project files. Class variables: _schema: A dictionary defining the properties of this class. The keys to @@ -263,45 +267,45 @@ class XCObject(object): described by its class' _schema variable. """ - _schema = {} - _should_print_single_line = False - - # See _EncodeString. - _encode_transforms = [] - i = 0 - while i < ord(' '): - _encode_transforms.append('\\U%04x' % i) - i = i + 1 - _encode_transforms[7] = '\\a' - _encode_transforms[8] = '\\b' - _encode_transforms[9] = '\\t' - _encode_transforms[10] = '\\n' - _encode_transforms[11] = '\\v' - _encode_transforms[12] = '\\f' - _encode_transforms[13] = '\\n' - - _alternate_encode_transforms = list(_encode_transforms) - _alternate_encode_transforms[9] = chr(9) - _alternate_encode_transforms[10] = chr(10) - _alternate_encode_transforms[11] = chr(11) - - def __init__(self, properties=None, id=None, parent=None): - self.id = id - self.parent = parent - self._properties = {} - self._hashables = [] - self._SetDefaultsFromSchema() - self.UpdateProperties(properties) - - def __repr__(self): - try: - name = self.Name() - except NotImplementedError: - return '<%s at 0x%x>' % (self.__class__.__name__, id(self)) - return '<%s %r at 0x%x>' % (self.__class__.__name__, name, id(self)) - - def Copy(self): - """Make a copy of this object. + _schema = {} + _should_print_single_line = False + + # See _EncodeString. + _encode_transforms = [] + i = 0 + while i < ord(" "): + _encode_transforms.append("\\U%04x" % i) + i = i + 1 + _encode_transforms[7] = "\\a" + _encode_transforms[8] = "\\b" + _encode_transforms[9] = "\\t" + _encode_transforms[10] = "\\n" + _encode_transforms[11] = "\\v" + _encode_transforms[12] = "\\f" + _encode_transforms[13] = "\\n" + + _alternate_encode_transforms = list(_encode_transforms) + _alternate_encode_transforms[9] = chr(9) + _alternate_encode_transforms[10] = chr(10) + _alternate_encode_transforms[11] = chr(11) + + def __init__(self, properties=None, id=None, parent=None): + self.id = id + self.parent = parent + self._properties = {} + self._hashables = [] + self._SetDefaultsFromSchema() + self.UpdateProperties(properties) + + def __repr__(self): + try: + name = self.Name() + except NotImplementedError: + return "<%s at 0x%x>" % (self.__class__.__name__, id(self)) + return "<%s %r at 0x%x>" % (self.__class__.__name__, name, id(self)) + + def Copy(self): + """Make a copy of this object. The new object will have its own copy of lists and dicts. Any XCObject objects owned by this object (marked "strong") will be copied in the @@ -310,62 +314,70 @@ def Copy(self): object without making a copy. """ - that = self.__class__(id=self.id, parent=self.parent) - for key, value in self._properties.items(): - is_strong = self._schema[key][2] - - if isinstance(value, XCObject): - if is_strong: - new_value = value.Copy() - new_value.parent = that - that._properties[key] = new_value - else: - that._properties[key] = value - elif isinstance(value, (basestring, int)): - that._properties[key] = value - elif isinstance(value, list): - if is_strong: - # If is_strong is True, each element is an XCObject, so it's safe to - # call Copy. - that._properties[key] = [] - for item in value: - new_item = item.Copy() - new_item.parent = that - that._properties[key].append(new_item) - else: - that._properties[key] = value[:] - elif isinstance(value, dict): - # dicts are never strong. - if is_strong: - raise TypeError('Strong dict for key ' + key + ' in ' + \ - self.__class__.__name__) - else: - that._properties[key] = value.copy() - else: - raise TypeError('Unexpected type ' + value.__class__.__name__ + \ - ' for key ' + key + ' in ' + self.__class__.__name__) - - return that - - def Name(self): - """Return the name corresponding to an object. + that = self.__class__(id=self.id, parent=self.parent) + for key, value in self._properties.items(): + is_strong = self._schema[key][2] + + if isinstance(value, XCObject): + if is_strong: + new_value = value.Copy() + new_value.parent = that + that._properties[key] = new_value + else: + that._properties[key] = value + elif isinstance(value, (basestring, int)): + that._properties[key] = value + elif isinstance(value, list): + if is_strong: + # If is_strong is True, each element is an XCObject, so it's safe to + # call Copy. + that._properties[key] = [] + for item in value: + new_item = item.Copy() + new_item.parent = that + that._properties[key].append(new_item) + else: + that._properties[key] = value[:] + elif isinstance(value, dict): + # dicts are never strong. + if is_strong: + raise TypeError( + "Strong dict for key " + key + " in " + self.__class__.__name__ + ) + else: + that._properties[key] = value.copy() + else: + raise TypeError( + "Unexpected type " + + value.__class__.__name__ + + " for key " + + key + + " in " + + self.__class__.__name__ + ) + + return that + + def Name(self): + """Return the name corresponding to an object. Not all objects necessarily need to be nameable, and not all that do have a "name" property. Override as needed. """ - # If the schema indicates that "name" is required, try to access the - # property even if it doesn't exist. This will result in a KeyError - # being raised for the property that should be present, which seems more - # appropriate than NotImplementedError in this case. - if 'name' in self._properties or \ - ('name' in self._schema and self._schema['name'][3]): - return self._properties['name'] + # If the schema indicates that "name" is required, try to access the + # property even if it doesn't exist. This will result in a KeyError + # being raised for the property that should be present, which seems more + # appropriate than NotImplementedError in this case. + if "name" in self._properties or ( + "name" in self._schema and self._schema["name"][3] + ): + return self._properties["name"] - raise NotImplementedError(self.__class__.__name__ + ' must implement Name') + raise NotImplementedError(self.__class__.__name__ + " must implement Name") - def Comment(self): - """Return a comment string for the object. + def Comment(self): + """Return a comment string for the object. Most objects just use their name as the comment, but PBXProject uses different values. @@ -374,24 +386,24 @@ def Comment(self): strings applied to it. """ - return self.Name() + return self.Name() - def Hashables(self): - hashables = [self.__class__.__name__] + def Hashables(self): + hashables = [self.__class__.__name__] - name = self.Name() - if name != None: - hashables.append(name) + name = self.Name() + if name is not None: + hashables.append(name) - hashables.extend(self._hashables) + hashables.extend(self._hashables) - return hashables + return hashables - def HashablesForChild(self): - return None + def HashablesForChild(self): + return None - def ComputeIDs(self, recursive=True, overwrite=True, seed_hash=None): - """Set "id" properties deterministically. + def ComputeIDs(self, recursive=True, overwrite=True, seed_hash=None): + """Set "id" properties deterministically. An object's "id" property is set based on a hash of its class type and name, as well as the class type and name of all ancestor objects. As @@ -405,8 +417,8 @@ def ComputeIDs(self, recursive=True, overwrite=True, seed_hash=None): replaced. """ - def _HashUpdate(hash, data): - """Update hash with data's length and contents. + def _HashUpdate(hash, data): + """Update hash with data's length and contents. If the hash were updated only with the value of data, it would be possible for clowns to induce collisions by manipulating the names of @@ -414,161 +426,166 @@ def _HashUpdate(hash, data): ID collisions will be encountered, intentionally or not. """ - hash.update(struct.pack('>i', len(data))) - hash.update(data) - - if seed_hash is None: - seed_hash = hashlib.sha1() - - hash = seed_hash.copy() - - hashables = self.Hashables() - assert len(hashables) > 0 - for hashable in hashables: - _HashUpdate(hash, hashable) - - if recursive: - hashables_for_child = self.HashablesForChild() - if hashables_for_child is None: - child_hash = hash - else: - assert len(hashables_for_child) > 0 - child_hash = seed_hash.copy() - for hashable in hashables_for_child: - _HashUpdate(child_hash, hashable) - - for child in self.Children(): - child.ComputeIDs(recursive, overwrite, child_hash) - - if overwrite or self.id is None: - # Xcode IDs are only 96 bits (24 hex characters), but a SHA-1 digest is - # is 160 bits. Instead of throwing out 64 bits of the digest, xor them - # into the portion that gets used. - assert hash.digest_size % 4 == 0 - digest_int_count = hash.digest_size / 4 - digest_ints = struct.unpack('>' + 'I' * digest_int_count, hash.digest()) - id_ints = [0, 0, 0] - for index in range(0, digest_int_count): - id_ints[index % 3] ^= digest_ints[index] - self.id = '%08X%08X%08X' % tuple(id_ints) - - def EnsureNoIDCollisions(self): - """Verifies that no two objects have the same ID. Checks all descendants. + hash.update(struct.pack(">i", len(data))) + hash.update(data) + + if seed_hash is None: + seed_hash = hashlib.sha1() + + hash = seed_hash.copy() + + hashables = self.Hashables() + assert len(hashables) > 0 + for hashable in hashables: + _HashUpdate(hash, hashable) + + if recursive: + hashables_for_child = self.HashablesForChild() + if hashables_for_child is None: + child_hash = hash + else: + assert len(hashables_for_child) > 0 + child_hash = seed_hash.copy() + for hashable in hashables_for_child: + _HashUpdate(child_hash, hashable) + + for child in self.Children(): + child.ComputeIDs(recursive, overwrite, child_hash) + + if overwrite or self.id is None: + # Xcode IDs are only 96 bits (24 hex characters), but a SHA-1 digest is + # is 160 bits. Instead of throwing out 64 bits of the digest, xor them + # into the portion that gets used. + assert hash.digest_size % 4 == 0 + digest_int_count = hash.digest_size // 4 + digest_ints = struct.unpack(">" + "I" * digest_int_count, hash.digest()) + id_ints = [0, 0, 0] + for index in range(0, digest_int_count): + id_ints[index % 3] ^= digest_ints[index] + self.id = "%08X%08X%08X" % tuple(id_ints) + + def EnsureNoIDCollisions(self): + """Verifies that no two objects have the same ID. Checks all descendants. """ - ids = {} - descendants = self.Descendants() - for descendant in descendants: - if descendant.id in ids: - other = ids[descendant.id] - raise KeyError( - 'Duplicate ID %s, objects "%s" and "%s" in "%s"' % \ - (descendant.id, str(descendant._properties), - str(other._properties), self._properties['rootObject'].Name())) - ids[descendant.id] = descendant - - def Children(self): - """Returns a list of all of this object's owned (strong) children.""" - - children = [] - for property, attributes in self._schema.items(): - (is_list, property_type, is_strong) = attributes[0:3] - if is_strong and property in self._properties: - if not is_list: - children.append(self._properties[property]) - else: - children.extend(self._properties[property]) - return children - - def Descendants(self): - """Returns a list of all of this object's descendants, including this + ids = {} + descendants = self.Descendants() + for descendant in descendants: + if descendant.id in ids: + other = ids[descendant.id] + raise KeyError( + 'Duplicate ID %s, objects "%s" and "%s" in "%s"' + % ( + descendant.id, + str(descendant._properties), + str(other._properties), + self._properties["rootObject"].Name(), + ) + ) + ids[descendant.id] = descendant + + def Children(self): + """Returns a list of all of this object's owned (strong) children.""" + + children = [] + for property, attributes in self._schema.items(): + (is_list, property_type, is_strong) = attributes[0:3] + if is_strong and property in self._properties: + if not is_list: + children.append(self._properties[property]) + else: + children.extend(self._properties[property]) + return children + + def Descendants(self): + """Returns a list of all of this object's descendants, including this object. """ - children = self.Children() - descendants = [self] - for child in children: - descendants.extend(child.Descendants()) - return descendants + children = self.Children() + descendants = [self] + for child in children: + descendants.extend(child.Descendants()) + return descendants - def PBXProjectAncestor(self): - # The base case for recursion is defined at PBXProject.PBXProjectAncestor. - if self.parent: - return self.parent.PBXProjectAncestor() - return None + def PBXProjectAncestor(self): + # The base case for recursion is defined at PBXProject.PBXProjectAncestor. + if self.parent: + return self.parent.PBXProjectAncestor() + return None - def _EncodeComment(self, comment): - """Encodes a comment to be placed in the project file output, mimicing + def _EncodeComment(self, comment): + """Encodes a comment to be placed in the project file output, mimicing Xcode behavior. """ - # This mimics Xcode behavior by wrapping the comment in "/*" and "*/". If - # the string already contains a "*/", it is turned into "(*)/". This keeps - # the file writer from outputting something that would be treated as the - # end of a comment in the middle of something intended to be entirely a - # comment. - - return '/* ' + comment.replace('*/', '(*)/') + ' */' - - def _EncodeTransform(self, match): - # This function works closely with _EncodeString. It will only be called - # by re.sub with match.group(0) containing a character matched by the - # the _escaped expression. - char = match.group(0) - - # Backslashes (\) and quotation marks (") are always replaced with a - # backslash-escaped version of the same. Everything else gets its - # replacement from the class' _encode_transforms array. - if char == '\\': - return '\\\\' - if char == '"': - return '\\"' - return self._encode_transforms[ord(char)] - - def _EncodeString(self, value): - """Encodes a string to be placed in the project file output, mimicing + # This mimics Xcode behavior by wrapping the comment in "/*" and "*/". If + # the string already contains a "*/", it is turned into "(*)/". This keeps + # the file writer from outputting something that would be treated as the + # end of a comment in the middle of something intended to be entirely a + # comment. + + return "/* " + comment.replace("*/", "(*)/") + " */" + + def _EncodeTransform(self, match): + # This function works closely with _EncodeString. It will only be called + # by re.sub with match.group(0) containing a character matched by the + # the _escaped expression. + char = match.group(0) + + # Backslashes (\) and quotation marks (") are always replaced with a + # backslash-escaped version of the same. Everything else gets its + # replacement from the class' _encode_transforms array. + if char == "\\": + return "\\\\" + if char == '"': + return '\\"' + return self._encode_transforms[ord(char)] + + def _EncodeString(self, value): + """Encodes a string to be placed in the project file output, mimicing Xcode behavior. """ - # Use quotation marks when any character outside of the range A-Z, a-z, 0-9, - # $ (dollar sign), . (period), and _ (underscore) is present. Also use - # quotation marks to represent empty strings. - # - # Escape " (double-quote) and \ (backslash) by preceding them with a - # backslash. - # - # Some characters below the printable ASCII range are encoded specially: - # 7 ^G BEL is encoded as "\a" - # 8 ^H BS is encoded as "\b" - # 11 ^K VT is encoded as "\v" - # 12 ^L NP is encoded as "\f" - # 127 ^? DEL is passed through as-is without escaping - # - In PBXFileReference and PBXBuildFile objects: - # 9 ^I HT is passed through as-is without escaping - # 10 ^J NL is passed through as-is without escaping - # 13 ^M CR is passed through as-is without escaping - # - In other objects: - # 9 ^I HT is encoded as "\t" - # 10 ^J NL is encoded as "\n" - # 13 ^M CR is encoded as "\n" rendering it indistinguishable from - # 10 ^J NL - # All other characters within the ASCII control character range (0 through - # 31 inclusive) are encoded as "\U001f" referring to the Unicode code point - # in hexadecimal. For example, character 14 (^N SO) is encoded as "\U000e". - # Characters above the ASCII range are passed through to the output encoded - # as UTF-8 without any escaping. These mappings are contained in the - # class' _encode_transforms list. - - if _unquoted.search(value) and not _quoted.search(value): - return value - - return '"' + _escaped.sub(self._EncodeTransform, value) + '"' - - def _XCPrint(self, file, tabs, line): - file.write('\t' * tabs + line) - - def _XCPrintableValue(self, tabs, value, flatten_list=False): - """Returns a representation of value that may be printed in a project file, + # Use quotation marks when any character outside of the range A-Z, a-z, 0-9, + # $ (dollar sign), . (period), and _ (underscore) is present. Also use + # quotation marks to represent empty strings. + # + # Escape " (double-quote) and \ (backslash) by preceding them with a + # backslash. + # + # Some characters below the printable ASCII range are encoded specially: + # 7 ^G BEL is encoded as "\a" + # 8 ^H BS is encoded as "\b" + # 11 ^K VT is encoded as "\v" + # 12 ^L NP is encoded as "\f" + # 127 ^? DEL is passed through as-is without escaping + # - In PBXFileReference and PBXBuildFile objects: + # 9 ^I HT is passed through as-is without escaping + # 10 ^J NL is passed through as-is without escaping + # 13 ^M CR is passed through as-is without escaping + # - In other objects: + # 9 ^I HT is encoded as "\t" + # 10 ^J NL is encoded as "\n" + # 13 ^M CR is encoded as "\n" rendering it indistinguishable from + # 10 ^J NL + # All other characters within the ASCII control character range (0 through + # 31 inclusive) are encoded as "\U001f" referring to the Unicode code point + # in hexadecimal. For example, character 14 (^N SO) is encoded as "\U000e". + # Characters above the ASCII range are passed through to the output encoded + # as UTF-8 without any escaping. These mappings are contained in the + # class' _encode_transforms list. + + if _unquoted.search(value) and not _quoted.search(value): + return value + + return '"' + _escaped.sub(self._EncodeTransform, value) + '"' + + def _XCPrint(self, file, tabs, line): + file.write("\t" * tabs + line) + + def _XCPrintableValue(self, tabs, value, flatten_list=False): + """Returns a representation of value that may be printed in a project file, mimicing Xcode's behavior. _XCPrintableValue can handle str and int values, XCObjects (which are @@ -582,58 +599,65 @@ def _XCPrintableValue(self, tabs, value, flatten_list=False): strings. """ - printable = '' - comment = None + printable = "" + comment = None - if self._should_print_single_line: - sep = ' ' - element_tabs = '' - end_tabs = '' - else: - sep = '\n' - element_tabs = '\t' * (tabs + 1) - end_tabs = '\t' * tabs - - if isinstance(value, XCObject): - printable += value.id - comment = value.Comment() - elif isinstance(value, str): - printable += self._EncodeString(value) - elif isinstance(value, unicode): - printable += self._EncodeString(value.encode('utf-8')) - elif isinstance(value, int): - printable += str(value) - elif isinstance(value, list): - if flatten_list and len(value) <= 1: - if len(value) == 0: - printable += self._EncodeString('') + if self._should_print_single_line: + sep = " " + element_tabs = "" + end_tabs = "" else: - printable += self._EncodeString(value[0]) - else: - printable = '(' + sep - for item in value: - printable += element_tabs + \ - self._XCPrintableValue(tabs + 1, item, flatten_list) + \ - ',' + sep - printable += end_tabs + ')' - elif isinstance(value, dict): - printable = '{' + sep - for item_key, item_value in sorted(value.items()): - printable += element_tabs + \ - self._XCPrintableValue(tabs + 1, item_key, flatten_list) + ' = ' + \ - self._XCPrintableValue(tabs + 1, item_value, flatten_list) + ';' + \ - sep - printable += end_tabs + '}' - else: - raise TypeError("Can't make " + value.__class__.__name__ + ' printable') + sep = "\n" + element_tabs = "\t" * (tabs + 1) + end_tabs = "\t" * tabs + + if isinstance(value, XCObject): + printable += value.id + comment = value.Comment() + elif isinstance(value, str): + printable += self._EncodeString(value) + elif isinstance(value, basestring): + printable += self._EncodeString(value.encode("utf-8")) + elif isinstance(value, int): + printable += str(value) + elif isinstance(value, list): + if flatten_list and len(value) <= 1: + if len(value) == 0: + printable += self._EncodeString("") + else: + printable += self._EncodeString(value[0]) + else: + printable = "(" + sep + for item in value: + printable += ( + element_tabs + + self._XCPrintableValue(tabs + 1, item, flatten_list) + + "," + + sep + ) + printable += end_tabs + ")" + elif isinstance(value, dict): + printable = "{" + sep + for item_key, item_value in sorted(value.items()): + printable += ( + element_tabs + + self._XCPrintableValue(tabs + 1, item_key, flatten_list) + + " = " + + self._XCPrintableValue(tabs + 1, item_value, flatten_list) + + ";" + + sep + ) + printable += end_tabs + "}" + else: + raise TypeError("Can't make " + value.__class__.__name__ + " printable") - if comment != None: - printable += ' ' + self._EncodeComment(comment) + if comment: + printable += " " + self._EncodeComment(comment) - return printable + return printable - def _XCKVPrint(self, file, tabs, key, value): - """Prints a key and value, members of an XCObject's _properties dictionary, + def _XCKVPrint(self, file, tabs, key, value): + """Prints a key and value, members of an XCObject's _properties dictionary, to file. tabs is an int identifying the indentation level. If the class' @@ -641,99 +665,100 @@ def _XCKVPrint(self, file, tabs, key, value): key-value pair will be followed by a space insead of a newline. """ - if self._should_print_single_line: - printable = '' - after_kv = ' ' - else: - printable = '\t' * tabs - after_kv = '\n' - - # Xcode usually prints remoteGlobalIDString values in PBXContainerItemProxy - # objects without comments. Sometimes it prints them with comments, but - # the majority of the time, it doesn't. To avoid unnecessary changes to - # the project file after Xcode opens it, don't write comments for - # remoteGlobalIDString. This is a sucky hack and it would certainly be - # cleaner to extend the schema to indicate whether or not a comment should - # be printed, but since this is the only case where the problem occurs and - # Xcode itself can't seem to make up its mind, the hack will suffice. - # - # Also see PBXContainerItemProxy._schema['remoteGlobalIDString']. - if key == 'remoteGlobalIDString' and isinstance(self, - PBXContainerItemProxy): - value_to_print = value.id - else: - value_to_print = value + if self._should_print_single_line: + printable = "" + after_kv = " " + else: + printable = "\t" * tabs + after_kv = "\n" + + # Xcode usually prints remoteGlobalIDString values in PBXContainerItemProxy + # objects without comments. Sometimes it prints them with comments, but + # the majority of the time, it doesn't. To avoid unnecessary changes to + # the project file after Xcode opens it, don't write comments for + # remoteGlobalIDString. This is a sucky hack and it would certainly be + # cleaner to extend the schema to indicate whether or not a comment should + # be printed, but since this is the only case where the problem occurs and + # Xcode itself can't seem to make up its mind, the hack will suffice. + # + # Also see PBXContainerItemProxy._schema['remoteGlobalIDString']. + if key == "remoteGlobalIDString" and isinstance(self, PBXContainerItemProxy): + value_to_print = value.id + else: + value_to_print = value - # PBXBuildFile's settings property is represented in the output as a dict, - # but a hack here has it represented as a string. Arrange to strip off the - # quotes so that it shows up in the output as expected. - if key == 'settings' and isinstance(self, PBXBuildFile): - strip_value_quotes = True - else: - strip_value_quotes = False + # PBXBuildFile's settings property is represented in the output as a dict, + # but a hack here has it represented as a string. Arrange to strip off the + # quotes so that it shows up in the output as expected. + if key == "settings" and isinstance(self, PBXBuildFile): + strip_value_quotes = True + else: + strip_value_quotes = False - # In another one-off, let's set flatten_list on buildSettings properties - # of XCBuildConfiguration objects, because that's how Xcode treats them. - if key == 'buildSettings' and isinstance(self, XCBuildConfiguration): - flatten_list = True - else: - flatten_list = False - - try: - printable_key = self._XCPrintableValue(tabs, key, flatten_list) - printable_value = self._XCPrintableValue(tabs, value_to_print, - flatten_list) - if strip_value_quotes and len(printable_value) > 1 and \ - printable_value[0] == '"' and printable_value[-1] == '"': - printable_value = printable_value[1:-1] - printable += printable_key + ' = ' + printable_value + ';' + after_kv - except TypeError as e: - gyp.common.ExceptionAppend(e, - 'while printing key "%s"' % key) - raise - - self._XCPrint(file, 0, printable) - - def Print(self, file=sys.stdout): - """Prints a reprentation of this object to file, adhering to Xcode output + # In another one-off, let's set flatten_list on buildSettings properties + # of XCBuildConfiguration objects, because that's how Xcode treats them. + if key == "buildSettings" and isinstance(self, XCBuildConfiguration): + flatten_list = True + else: + flatten_list = False + + try: + printable_key = self._XCPrintableValue(tabs, key, flatten_list) + printable_value = self._XCPrintableValue(tabs, value_to_print, flatten_list) + if ( + strip_value_quotes + and len(printable_value) > 1 + and printable_value[0] == '"' + and printable_value[-1] == '"' + ): + printable_value = printable_value[1:-1] + printable += printable_key + " = " + printable_value + ";" + after_kv + except TypeError as e: + gyp.common.ExceptionAppend(e, 'while printing key "%s"' % key) + raise + + self._XCPrint(file, 0, printable) + + def Print(self, file=sys.stdout): + """Prints a reprentation of this object to file, adhering to Xcode output formatting. """ - self.VerifyHasRequiredProperties() - - if self._should_print_single_line: - # When printing an object in a single line, Xcode doesn't put any space - # between the beginning of a dictionary (or presumably a list) and the - # first contained item, so you wind up with snippets like - # ...CDEF = {isa = PBXFileReference; fileRef = 0123... - # If it were me, I would have put a space in there after the opening - # curly, but I guess this is just another one of those inconsistencies - # between how Xcode prints PBXFileReference and PBXBuildFile objects as - # compared to other objects. Mimic Xcode's behavior here by using an - # empty string for sep. - sep = '' - end_tabs = 0 - else: - sep = '\n' - end_tabs = 2 + self.VerifyHasRequiredProperties() + + if self._should_print_single_line: + # When printing an object in a single line, Xcode doesn't put any space + # between the beginning of a dictionary (or presumably a list) and the + # first contained item, so you wind up with snippets like + # ...CDEF = {isa = PBXFileReference; fileRef = 0123... + # If it were me, I would have put a space in there after the opening + # curly, but I guess this is just another one of those inconsistencies + # between how Xcode prints PBXFileReference and PBXBuildFile objects as + # compared to other objects. Mimic Xcode's behavior here by using an + # empty string for sep. + sep = "" + end_tabs = 0 + else: + sep = "\n" + end_tabs = 2 - # Start the object. For example, '\t\tPBXProject = {\n'. - self._XCPrint(file, 2, self._XCPrintableValue(2, self) + ' = {' + sep) + # Start the object. For example, '\t\tPBXProject = {\n'. + self._XCPrint(file, 2, self._XCPrintableValue(2, self) + " = {" + sep) - # "isa" isn't in the _properties dictionary, it's an intrinsic property - # of the class which the object belongs to. Xcode always outputs "isa" - # as the first element of an object dictionary. - self._XCKVPrint(file, 3, 'isa', self.__class__.__name__) + # "isa" isn't in the _properties dictionary, it's an intrinsic property + # of the class which the object belongs to. Xcode always outputs "isa" + # as the first element of an object dictionary. + self._XCKVPrint(file, 3, "isa", self.__class__.__name__) - # The remaining elements of an object dictionary are sorted alphabetically. - for property, value in sorted(self._properties.items()): - self._XCKVPrint(file, 3, property, value) + # The remaining elements of an object dictionary are sorted alphabetically. + for property, value in sorted(self._properties.items()): + self._XCKVPrint(file, 3, property, value) - # End the object. - self._XCPrint(file, end_tabs, '};\n') + # End the object. + self._XCPrint(file, end_tabs, "};\n") - def UpdateProperties(self, properties, do_copy=False): - """Merge the supplied properties into the _properties dictionary. + def UpdateProperties(self, properties, do_copy=False): + """Merge the supplied properties into the _properties dictionary. The input properties must adhere to the class schema or a KeyError or TypeError exception will be raised. If adding an object of an XCObject @@ -745,206 +770,244 @@ def UpdateProperties(self, properties, do_copy=False): references added. """ - if properties is None: - return - - for property, value in properties.items(): - # Make sure the property is in the schema. - if not property in self._schema: - raise KeyError(property + ' not in ' + self.__class__.__name__) - - # Make sure the property conforms to the schema. - (is_list, property_type, is_strong) = self._schema[property][0:3] - if is_list: - if value.__class__ != list: - raise TypeError( - property + ' of ' + self.__class__.__name__ + \ - ' must be list, not ' + value.__class__.__name__) - for item in value: - if not isinstance(item, property_type) and \ - not (item.__class__ == unicode and property_type == str): - # Accept unicode where str is specified. str is treated as - # UTF-8-encoded. - raise TypeError( - 'item of ' + property + ' of ' + self.__class__.__name__ + \ - ' must be ' + property_type.__name__ + ', not ' + \ - item.__class__.__name__) - elif not isinstance(value, property_type) and \ - not (value.__class__ == unicode and property_type == str): - # Accept unicode where str is specified. str is treated as - # UTF-8-encoded. - raise TypeError( - property + ' of ' + self.__class__.__name__ + ' must be ' + \ - property_type.__name__ + ', not ' + value.__class__.__name__) - - # Checks passed, perform the assignment. - if do_copy: - if isinstance(value, XCObject): - if is_strong: - self._properties[property] = value.Copy() - else: - self._properties[property] = value - elif isinstance(value, (basestring, int)): - self._properties[property] = value - elif isinstance(value, list): - if is_strong: - # If is_strong is True, each element is an XCObject, so it's safe - # to call Copy. - self._properties[property] = [] - for item in value: - self._properties[property].append(item.Copy()) - else: - self._properties[property] = value[:] - elif isinstance(value, dict): - self._properties[property] = value.copy() - else: - raise TypeError("Don't know how to copy a " + \ - value.__class__.__name__ + ' object for ' + \ - property + ' in ' + self.__class__.__name__) - else: - self._properties[property] = value - - # Set up the child's back-reference to this object. Don't use |value| - # any more because it may not be right if do_copy is true. - if is_strong: + if properties is None: + return + + for property, value in properties.items(): + # Make sure the property is in the schema. + if property not in self._schema: + raise KeyError(property + " not in " + self.__class__.__name__) + + # Make sure the property conforms to the schema. + (is_list, property_type, is_strong) = self._schema[property][0:3] + if is_list: + if value.__class__ != list: + raise TypeError( + property + + " of " + + self.__class__.__name__ + + " must be list, not " + + value.__class__.__name__ + ) + for item in value: + if not isinstance(item, property_type) and not ( + isinstance(item, basestring) and property_type == str + ): + # Accept unicode where str is specified. str is treated as + # UTF-8-encoded. + raise TypeError( + "item of " + + property + + " of " + + self.__class__.__name__ + + " must be " + + property_type.__name__ + + ", not " + + item.__class__.__name__ + ) + elif not isinstance(value, property_type) and not ( + isinstance(value, basestring) and property_type == str + ): + # Accept unicode where str is specified. str is treated as + # UTF-8-encoded. + raise TypeError( + property + + " of " + + self.__class__.__name__ + + " must be " + + property_type.__name__ + + ", not " + + value.__class__.__name__ + ) + + # Checks passed, perform the assignment. + if do_copy: + if isinstance(value, XCObject): + if is_strong: + self._properties[property] = value.Copy() + else: + self._properties[property] = value + elif isinstance(value, (basestring, int)): + self._properties[property] = value + elif isinstance(value, list): + if is_strong: + # If is_strong is True, each element is an XCObject, so it's safe + # to call Copy. + self._properties[property] = [] + for item in value: + self._properties[property].append(item.Copy()) + else: + self._properties[property] = value[:] + elif isinstance(value, dict): + self._properties[property] = value.copy() + else: + raise TypeError( + "Don't know how to copy a " + + value.__class__.__name__ + + " object for " + + property + + " in " + + self.__class__.__name__ + ) + else: + self._properties[property] = value + + # Set up the child's back-reference to this object. Don't use |value| + # any more because it may not be right if do_copy is true. + if is_strong: + if not is_list: + self._properties[property].parent = self + else: + for item in self._properties[property]: + item.parent = self + + def HasProperty(self, key): + return key in self._properties + + def GetProperty(self, key): + return self._properties[key] + + def SetProperty(self, key, value): + self.UpdateProperties({key: value}) + + def DelProperty(self, key): + if key in self._properties: + del self._properties[key] + + def AppendProperty(self, key, value): + # TODO(mark): Support ExtendProperty too (and make this call that)? + + # Schema validation. + if key not in self._schema: + raise KeyError(key + " not in " + self.__class__.__name__) + + (is_list, property_type, is_strong) = self._schema[key][0:3] if not is_list: - self._properties[property].parent = self - else: - for item in self._properties[property]: - item.parent = self - - def HasProperty(self, key): - return key in self._properties - - def GetProperty(self, key): - return self._properties[key] - - def SetProperty(self, key, value): - self.UpdateProperties({key: value}) - - def DelProperty(self, key): - if key in self._properties: - del self._properties[key] - - def AppendProperty(self, key, value): - # TODO(mark): Support ExtendProperty too (and make this call that)? - - # Schema validation. - if not key in self._schema: - raise KeyError(key + ' not in ' + self.__class__.__name__) - - (is_list, property_type, is_strong) = self._schema[key][0:3] - if not is_list: - raise TypeError(key + ' of ' + self.__class__.__name__ + ' must be list') - if not isinstance(value, property_type): - raise TypeError('item of ' + key + ' of ' + self.__class__.__name__ + \ - ' must be ' + property_type.__name__ + ', not ' + \ - value.__class__.__name__) - - # If the property doesn't exist yet, create a new empty list to receive the - # item. - if not key in self._properties: - self._properties[key] = [] - - # Set up the ownership link. - if is_strong: - value.parent = self + raise TypeError(key + " of " + self.__class__.__name__ + " must be list") + if not isinstance(value, property_type): + raise TypeError( + "item of " + + key + + " of " + + self.__class__.__name__ + + " must be " + + property_type.__name__ + + ", not " + + value.__class__.__name__ + ) + + # If the property doesn't exist yet, create a new empty list to receive the + # item. + self._properties[key] = self._properties.get(key, []) + + # Set up the ownership link. + if is_strong: + value.parent = self - # Store the item. - self._properties[key].append(value) + # Store the item. + self._properties[key].append(value) - def VerifyHasRequiredProperties(self): - """Ensure that all properties identified as required by the schema are + def VerifyHasRequiredProperties(self): + """Ensure that all properties identified as required by the schema are set. """ - # TODO(mark): A stronger verification mechanism is needed. Some - # subclasses need to perform validation beyond what the schema can enforce. - for property, attributes in self._schema.items(): - (is_list, property_type, is_strong, is_required) = attributes[0:4] - if is_required and not property in self._properties: - raise KeyError(self.__class__.__name__ + ' requires ' + property) + # TODO(mark): A stronger verification mechanism is needed. Some + # subclasses need to perform validation beyond what the schema can enforce. + for property, attributes in self._schema.items(): + (is_list, property_type, is_strong, is_required) = attributes[0:4] + if is_required and property not in self._properties: + raise KeyError(self.__class__.__name__ + " requires " + property) - def _SetDefaultsFromSchema(self): - """Assign object default values according to the schema. This will not + def _SetDefaultsFromSchema(self): + """Assign object default values according to the schema. This will not overwrite properties that have already been set.""" - defaults = {} - for property, attributes in self._schema.items(): - (is_list, property_type, is_strong, is_required) = attributes[0:4] - if is_required and len(attributes) >= 5 and \ - not property in self._properties: - default = attributes[4] + defaults = {} + for property, attributes in self._schema.items(): + (is_list, property_type, is_strong, is_required) = attributes[0:4] + if ( + is_required + and len(attributes) >= 5 + and property not in self._properties + ): + default = attributes[4] - defaults[property] = default + defaults[property] = default - if len(defaults) > 0: - # Use do_copy=True so that each new object gets its own copy of strong - # objects, lists, and dicts. - self.UpdateProperties(defaults, do_copy=True) + if len(defaults) > 0: + # Use do_copy=True so that each new object gets its own copy of strong + # objects, lists, and dicts. + self.UpdateProperties(defaults, do_copy=True) class XCHierarchicalElement(XCObject): - """Abstract base for PBXGroup and PBXFileReference. Not represented in a + """Abstract base for PBXGroup and PBXFileReference. Not represented in a project file.""" - # TODO(mark): Do name and path belong here? Probably so. - # If path is set and name is not, name may have a default value. Name will - # be set to the basename of path, if the basename of path is different from - # the full value of path. If path is already just a leaf name, name will - # not be set. - _schema = XCObject._schema.copy() - _schema.update({ - 'comments': [0, str, 0, 0], - 'fileEncoding': [0, str, 0, 0], - 'includeInIndex': [0, int, 0, 0], - 'indentWidth': [0, int, 0, 0], - 'lineEnding': [0, int, 0, 0], - 'sourceTree': [0, str, 0, 1, ''], - 'tabWidth': [0, int, 0, 0], - 'usesTabs': [0, int, 0, 0], - 'wrapsLines': [0, int, 0, 0], - }) - - def __init__(self, properties=None, id=None, parent=None): - # super - XCObject.__init__(self, properties, id, parent) - if 'path' in self._properties and not 'name' in self._properties: - path = self._properties['path'] - name = posixpath.basename(path) - if name != '' and path != name: - self.SetProperty('name', name) - - if 'path' in self._properties and \ - (not 'sourceTree' in self._properties or \ - self._properties['sourceTree'] == ''): - # If the pathname begins with an Xcode variable like "$(SDKROOT)/", take - # the variable out and make the path be relative to that variable by - # assigning the variable name as the sourceTree. - (source_tree, path) = SourceTreeAndPathFromPath(self._properties['path']) - if source_tree != None: - self._properties['sourceTree'] = source_tree - if path != None: - self._properties['path'] = path - if source_tree != None and path is None and \ - not 'name' in self._properties: - # The path was of the form "$(SDKROOT)" with no path following it. - # This object is now relative to that variable, so it has no path - # attribute of its own. It does, however, keep a name. - del self._properties['path'] - self._properties['name'] = source_tree - - def Name(self): - if 'name' in self._properties: - return self._properties['name'] - elif 'path' in self._properties: - return self._properties['path'] - else: - # This happens in the case of the root PBXGroup. - return None + # TODO(mark): Do name and path belong here? Probably so. + # If path is set and name is not, name may have a default value. Name will + # be set to the basename of path, if the basename of path is different from + # the full value of path. If path is already just a leaf name, name will + # not be set. + _schema = XCObject._schema.copy() + _schema.update( + { + "comments": [0, str, 0, 0], + "fileEncoding": [0, str, 0, 0], + "includeInIndex": [0, int, 0, 0], + "indentWidth": [0, int, 0, 0], + "lineEnding": [0, int, 0, 0], + "sourceTree": [0, str, 0, 1, ""], + "tabWidth": [0, int, 0, 0], + "usesTabs": [0, int, 0, 0], + "wrapsLines": [0, int, 0, 0], + } + ) + + def __init__(self, properties=None, id=None, parent=None): + # super + XCObject.__init__(self, properties, id, parent) + if "path" in self._properties and "name" not in self._properties: + path = self._properties["path"] + name = posixpath.basename(path) + if name != "" and path != name: + self.SetProperty("name", name) + + if "path" in self._properties and ( + "sourceTree" not in self._properties + or self._properties["sourceTree"] == "" + ): + # If the pathname begins with an Xcode variable like "$(SDKROOT)/", take + # the variable out and make the path be relative to that variable by + # assigning the variable name as the sourceTree. + (source_tree, path) = SourceTreeAndPathFromPath(self._properties["path"]) + if source_tree is not None: + self._properties["sourceTree"] = source_tree + if path is not None: + self._properties["path"] = path + if ( + source_tree is not None + and path is None + and "name" not in self._properties + ): + # The path was of the form "$(SDKROOT)" with no path following it. + # This object is now relative to that variable, so it has no path + # attribute of its own. It does, however, keep a name. + del self._properties["path"] + self._properties["name"] = source_tree + + def Name(self): + if "name" in self._properties: + return self._properties["name"] + elif "path" in self._properties: + return self._properties["path"] + else: + # This happens in the case of the root PBXGroup. + return None - def Hashables(self): - """Custom hashables for XCHierarchicalElements. + def Hashables(self): + """Custom hashables for XCHierarchicalElements. XCHierarchicalElements are special. Generally, their hashes shouldn't change if the paths don't change. The normal XCObject implementation of @@ -968,128 +1031,134 @@ def Hashables(self): is not considered a problem because there can be only one main group. """ - if self == self.PBXProjectAncestor()._properties['mainGroup']: - # super - return XCObject.Hashables(self) - - hashables = [] - - # Put the name in first, ensuring that if TakeOverOnlyChild collapses - # children into a top-level group like "Source", the name always goes - # into the list of hashables without interfering with path components. - if 'name' in self._properties: - # Make it less likely for people to manipulate hashes by following the - # pattern of always pushing an object type value onto the list first. - hashables.append(self.__class__.__name__ + '.name') - hashables.append(self._properties['name']) - - # NOTE: This still has the problem that if an absolute path is encountered, - # including paths with a sourceTree, they'll still inherit their parents' - # hashables, even though the paths aren't relative to their parents. This - # is not expected to be much of a problem in practice. - path = self.PathFromSourceTreeAndPath() - if path != None: - components = path.split(posixpath.sep) - for component in components: - hashables.append(self.__class__.__name__ + '.path') - hashables.append(component) - - hashables.extend(self._hashables) - - return hashables - - def Compare(self, other): - # Allow comparison of these types. PBXGroup has the highest sort rank; - # PBXVariantGroup is treated as equal to PBXFileReference. - valid_class_types = { - PBXFileReference: 'file', - PBXGroup: 'group', - PBXVariantGroup: 'file', - } - self_type = valid_class_types[self.__class__] - other_type = valid_class_types[other.__class__] - - if self_type == other_type: - # If the two objects are of the same sort rank, compare their names. - return cmp(self.Name(), other.Name()) - - # Otherwise, sort groups before everything else. - if self_type == 'group': - return -1 - return 1 - - def CompareRootGroup(self, other): - # This function should be used only to compare direct children of the - # containing PBXProject's mainGroup. These groups should appear in the - # listed order. - # TODO(mark): "Build" is used by gyp.generator.xcode, perhaps the - # generator should have a way of influencing this list rather than having - # to hardcode for the generator here. - order = ['Source', 'Intermediates', 'Projects', 'Frameworks', 'Products', - 'Build'] - - # If the groups aren't in the listed order, do a name comparison. - # Otherwise, groups in the listed order should come before those that - # aren't. - self_name = self.Name() - other_name = other.Name() - self_in = isinstance(self, PBXGroup) and self_name in order - other_in = isinstance(self, PBXGroup) and other_name in order - if not self_in and not other_in: - return self.Compare(other) - if self_name in order and not other_name in order: - return -1 - if other_name in order and not self_name in order: - return 1 - - # If both groups are in the listed order, go by the defined order. - self_index = order.index(self_name) - other_index = order.index(other_name) - if self_index < other_index: - return -1 - if self_index > other_index: - return 1 - return 0 - - def PathFromSourceTreeAndPath(self): - # Turn the object's sourceTree and path properties into a single flat - # string of a form comparable to the path parameter. If there's a - # sourceTree property other than "", wrap it in $(...) for the - # comparison. - components = [] - if self._properties['sourceTree'] != '': - components.append('$(' + self._properties['sourceTree'] + ')') - if 'path' in self._properties: - components.append(self._properties['path']) - - if len(components) > 0: - return posixpath.join(*components) - - return None - - def FullPath(self): - # Returns a full path to self relative to the project file, or relative - # to some other source tree. Start with self, and walk up the chain of - # parents prepending their paths, if any, until no more parents are - # available (project-relative path) or until a path relative to some - # source tree is found. - xche = self - path = None - while isinstance(xche, XCHierarchicalElement) and \ - (path is None or \ - (not path.startswith('/') and not path.startswith('$'))): - this_path = xche.PathFromSourceTreeAndPath() - if this_path != None and path != None: - path = posixpath.join(this_path, path) - elif this_path != None: - path = this_path - xche = xche.parent - - return path + if self == self.PBXProjectAncestor()._properties["mainGroup"]: + # super + return XCObject.Hashables(self) + + hashables = [] + + # Put the name in first, ensuring that if TakeOverOnlyChild collapses + # children into a top-level group like "Source", the name always goes + # into the list of hashables without interfering with path components. + if "name" in self._properties: + # Make it less likely for people to manipulate hashes by following the + # pattern of always pushing an object type value onto the list first. + hashables.append(self.__class__.__name__ + ".name") + hashables.append(self._properties["name"]) + + # NOTE: This still has the problem that if an absolute path is encountered, + # including paths with a sourceTree, they'll still inherit their parents' + # hashables, even though the paths aren't relative to their parents. This + # is not expected to be much of a problem in practice. + path = self.PathFromSourceTreeAndPath() + if path is not None: + components = path.split(posixpath.sep) + for component in components: + hashables.append(self.__class__.__name__ + ".path") + hashables.append(component) + + hashables.extend(self._hashables) + + return hashables + + def Compare(self, other): + # Allow comparison of these types. PBXGroup has the highest sort rank; + # PBXVariantGroup is treated as equal to PBXFileReference. + valid_class_types = { + PBXFileReference: "file", + PBXGroup: "group", + PBXVariantGroup: "file", + } + self_type = valid_class_types[self.__class__] + other_type = valid_class_types[other.__class__] + + if self_type == other_type: + # If the two objects are of the same sort rank, compare their names. + return cmp(self.Name(), other.Name()) + + # Otherwise, sort groups before everything else. + if self_type == "group": + return -1 + return 1 + + def CompareRootGroup(self, other): + # This function should be used only to compare direct children of the + # containing PBXProject's mainGroup. These groups should appear in the + # listed order. + # TODO(mark): "Build" is used by gyp.generator.xcode, perhaps the + # generator should have a way of influencing this list rather than having + # to hardcode for the generator here. + order = [ + "Source", + "Intermediates", + "Projects", + "Frameworks", + "Products", + "Build", + ] + + # If the groups aren't in the listed order, do a name comparison. + # Otherwise, groups in the listed order should come before those that + # aren't. + self_name = self.Name() + other_name = other.Name() + self_in = isinstance(self, PBXGroup) and self_name in order + other_in = isinstance(self, PBXGroup) and other_name in order + if not self_in and not other_in: + return self.Compare(other) + if self_name in order and other_name not in order: + return -1 + if other_name in order and self_name not in order: + return 1 + + # If both groups are in the listed order, go by the defined order. + self_index = order.index(self_name) + other_index = order.index(other_name) + if self_index < other_index: + return -1 + if self_index > other_index: + return 1 + return 0 + + def PathFromSourceTreeAndPath(self): + # Turn the object's sourceTree and path properties into a single flat + # string of a form comparable to the path parameter. If there's a + # sourceTree property other than "", wrap it in $(...) for the + # comparison. + components = [] + if self._properties["sourceTree"] != "": + components.append("$(" + self._properties["sourceTree"] + ")") + if "path" in self._properties: + components.append(self._properties["path"]) + + if len(components) > 0: + return posixpath.join(*components) + + return None + + def FullPath(self): + # Returns a full path to self relative to the project file, or relative + # to some other source tree. Start with self, and walk up the chain of + # parents prepending their paths, if any, until no more parents are + # available (project-relative path) or until a path relative to some + # source tree is found. + xche = self + path = None + while isinstance(xche, XCHierarchicalElement) and ( + path is None or (not path.startswith("/") and not path.startswith("$")) + ): + this_path = xche.PathFromSourceTreeAndPath() + if this_path is not None and path is not None: + path = posixpath.join(this_path, path) + elif this_path is not None: + path = this_path + xche = xche.parent + + return path class PBXGroup(XCHierarchicalElement): - """ + """ Attributes: _children_by_path: Maps pathnames of children of this PBXGroup to the actual child XCHierarchicalElement objects. @@ -1097,116 +1166,122 @@ class PBXGroup(XCHierarchicalElement): PBXVariantGroup children to the actual child PBXVariantGroup objects. """ - _schema = XCHierarchicalElement._schema.copy() - _schema.update({ - 'children': [1, XCHierarchicalElement, 1, 1, []], - 'name': [0, str, 0, 0], - 'path': [0, str, 0, 0], - }) - - def __init__(self, properties=None, id=None, parent=None): - # super - XCHierarchicalElement.__init__(self, properties, id, parent) - self._children_by_path = {} - self._variant_children_by_name_and_path = {} - for child in self._properties.get('children', []): - self._AddChildToDicts(child) - - def Hashables(self): - # super - hashables = XCHierarchicalElement.Hashables(self) - - # It is not sufficient to just rely on name and parent to build a unique - # hashable : a node could have two child PBXGroup sharing a common name. - # To add entropy the hashable is enhanced with the names of all its - # children. - for child in self._properties.get('children', []): - child_name = child.Name() - if child_name != None: - hashables.append(child_name) - - return hashables - - def HashablesForChild(self): - # To avoid a circular reference the hashables used to compute a child id do - # not include the child names. - return XCHierarchicalElement.Hashables(self) - - def _AddChildToDicts(self, child): - # Sets up this PBXGroup object's dicts to reference the child properly. - child_path = child.PathFromSourceTreeAndPath() - if child_path: - if child_path in self._children_by_path: - raise ValueError('Found multiple children with path ' + child_path) - self._children_by_path[child_path] = child - - if isinstance(child, PBXVariantGroup): - child_name = child._properties.get('name', None) - key = (child_name, child_path) - if key in self._variant_children_by_name_and_path: - raise ValueError('Found multiple PBXVariantGroup children with ' + \ - 'name ' + str(child_name) + ' and path ' + \ - str(child_path)) - self._variant_children_by_name_and_path[key] = child - - def AppendChild(self, child): - # Callers should use this instead of calling - # AppendProperty('children', child) directly because this function - # maintains the group's dicts. - self.AppendProperty('children', child) - self._AddChildToDicts(child) - - def GetChildByName(self, name): - # This is not currently optimized with a dict as GetChildByPath is because - # it has few callers. Most callers probably want GetChildByPath. This - # function is only useful to get children that have names but no paths, - # which is rare. The children of the main group ("Source", "Products", - # etc.) is pretty much the only case where this likely to come up. - # - # TODO(mark): Maybe this should raise an error if more than one child is - # present with the same name. - if not 'children' in self._properties: - return None + _schema = XCHierarchicalElement._schema.copy() + _schema.update( + { + "children": [1, XCHierarchicalElement, 1, 1, []], + "name": [0, str, 0, 0], + "path": [0, str, 0, 0], + } + ) + + def __init__(self, properties=None, id=None, parent=None): + # super + XCHierarchicalElement.__init__(self, properties, id, parent) + self._children_by_path = {} + self._variant_children_by_name_and_path = {} + for child in self._properties.get("children", []): + self._AddChildToDicts(child) + + def Hashables(self): + # super + hashables = XCHierarchicalElement.Hashables(self) + + # It is not sufficient to just rely on name and parent to build a unique + # hashable : a node could have two child PBXGroup sharing a common name. + # To add entropy the hashable is enhanced with the names of all its + # children. + for child in self._properties.get("children", []): + child_name = child.Name() + if child_name is not None: + hashables.append(child_name) + + return hashables + + def HashablesForChild(self): + # To avoid a circular reference the hashables used to compute a child id do + # not include the child names. + return XCHierarchicalElement.Hashables(self) + + def _AddChildToDicts(self, child): + # Sets up this PBXGroup object's dicts to reference the child properly. + child_path = child.PathFromSourceTreeAndPath() + if child_path: + if child_path in self._children_by_path: + raise ValueError("Found multiple children with path " + child_path) + self._children_by_path[child_path] = child + + if isinstance(child, PBXVariantGroup): + child_name = child._properties.get("name", None) + key = (child_name, child_path) + if key in self._variant_children_by_name_and_path: + raise ValueError( + "Found multiple PBXVariantGroup children with " + + "name " + + str(child_name) + + " and path " + + str(child_path) + ) + self._variant_children_by_name_and_path[key] = child + + def AppendChild(self, child): + # Callers should use this instead of calling + # AppendProperty('children', child) directly because this function + # maintains the group's dicts. + self.AppendProperty("children", child) + self._AddChildToDicts(child) + + def GetChildByName(self, name): + # This is not currently optimized with a dict as GetChildByPath is because + # it has few callers. Most callers probably want GetChildByPath. This + # function is only useful to get children that have names but no paths, + # which is rare. The children of the main group ("Source", "Products", + # etc.) is pretty much the only case where this likely to come up. + # + # TODO(mark): Maybe this should raise an error if more than one child is + # present with the same name. + if "children" not in self._properties: + return None - for child in self._properties['children']: - if child.Name() == name: - return child + for child in self._properties["children"]: + if child.Name() == name: + return child - return None + return None - def GetChildByPath(self, path): - if not path: - return None + def GetChildByPath(self, path): + if not path: + return None - if path in self._children_by_path: - return self._children_by_path[path] + if path in self._children_by_path: + return self._children_by_path[path] - return None + return None - def GetChildByRemoteObject(self, remote_object): - # This method is a little bit esoteric. Given a remote_object, which - # should be a PBXFileReference in another project file, this method will - # return this group's PBXReferenceProxy object serving as a local proxy - # for the remote PBXFileReference. - # - # This function might benefit from a dict optimization as GetChildByPath - # for some workloads, but profiling shows that it's not currently a - # problem. - if not 'children' in self._properties: - return None + def GetChildByRemoteObject(self, remote_object): + # This method is a little bit esoteric. Given a remote_object, which + # should be a PBXFileReference in another project file, this method will + # return this group's PBXReferenceProxy object serving as a local proxy + # for the remote PBXFileReference. + # + # This function might benefit from a dict optimization as GetChildByPath + # for some workloads, but profiling shows that it's not currently a + # problem. + if "children" not in self._properties: + return None - for child in self._properties['children']: - if not isinstance(child, PBXReferenceProxy): - continue + for child in self._properties["children"]: + if not isinstance(child, PBXReferenceProxy): + continue - container_proxy = child._properties['remoteRef'] - if container_proxy._properties['remoteGlobalIDString'] == remote_object: - return child + container_proxy = child._properties["remoteRef"] + if container_proxy._properties["remoteGlobalIDString"] == remote_object: + return child - return None + return None - def AddOrGetFileByPath(self, path, hierarchical): - """Returns an existing or new file reference corresponding to path. + def AddOrGetFileByPath(self, path, hierarchical): + """Returns an existing or new file reference corresponding to path. If hierarchical is True, this method will create or use the necessary hierarchical group structure corresponding to path. Otherwise, it will @@ -1223,83 +1298,88 @@ def AddOrGetFileByPath(self, path, hierarchical): all other paths, a "normal" PBXFileReference will be returned. """ - # Adding or getting a directory? Directories end with a trailing slash. - is_dir = False - if path.endswith('/'): - is_dir = True - path = posixpath.normpath(path) - if is_dir: - path = path + '/' - - # Adding or getting a variant? Variants are files inside directories - # with an ".lproj" extension. Xcode uses variants for localization. For - # a variant path/to/Language.lproj/MainMenu.nib, put a variant group named - # MainMenu.nib inside path/to, and give it a variant named Language. In - # this example, grandparent would be set to path/to and parent_root would - # be set to Language. - variant_name = None - parent = posixpath.dirname(path) - grandparent = posixpath.dirname(parent) - parent_basename = posixpath.basename(parent) - (parent_root, parent_ext) = posixpath.splitext(parent_basename) - if parent_ext == '.lproj': - variant_name = parent_root - if grandparent == '': - grandparent = None - - # Putting a directory inside a variant group is not currently supported. - assert not is_dir or variant_name is None - - path_split = path.split(posixpath.sep) - if len(path_split) == 1 or \ - ((is_dir or variant_name != None) and len(path_split) == 2) or \ - not hierarchical: - # The PBXFileReference or PBXVariantGroup will be added to or gotten from - # this PBXGroup, no recursion necessary. - if variant_name is None: - # Add or get a PBXFileReference. - file_ref = self.GetChildByPath(path) - if file_ref != None: - assert file_ref.__class__ == PBXFileReference + # Adding or getting a directory? Directories end with a trailing slash. + is_dir = False + if path.endswith("/"): + is_dir = True + path = posixpath.normpath(path) + if is_dir: + path = path + "/" + + # Adding or getting a variant? Variants are files inside directories + # with an ".lproj" extension. Xcode uses variants for localization. For + # a variant path/to/Language.lproj/MainMenu.nib, put a variant group named + # MainMenu.nib inside path/to, and give it a variant named Language. In + # this example, grandparent would be set to path/to and parent_root would + # be set to Language. + variant_name = None + parent = posixpath.dirname(path) + grandparent = posixpath.dirname(parent) + parent_basename = posixpath.basename(parent) + (parent_root, parent_ext) = posixpath.splitext(parent_basename) + if parent_ext == ".lproj": + variant_name = parent_root + if grandparent == "": + grandparent = None + + # Putting a directory inside a variant group is not currently supported. + assert not is_dir or variant_name is None + + path_split = path.split(posixpath.sep) + if ( + len(path_split) == 1 + or ((is_dir or variant_name is not None) and len(path_split) == 2) + or not hierarchical + ): + # The PBXFileReference or PBXVariantGroup will be added to or gotten from + # this PBXGroup, no recursion necessary. + if variant_name is None: + # Add or get a PBXFileReference. + file_ref = self.GetChildByPath(path) + if file_ref is not None: + assert file_ref.__class__ == PBXFileReference + else: + file_ref = PBXFileReference({"path": path}) + self.AppendChild(file_ref) + else: + # Add or get a PBXVariantGroup. The variant group name is the same + # as the basename (MainMenu.nib in the example above). grandparent + # specifies the path to the variant group itself, and path_split[-2:] + # is the path of the specific variant relative to its group. + variant_group_name = posixpath.basename(path) + variant_group_ref = self.AddOrGetVariantGroupByNameAndPath( + variant_group_name, grandparent + ) + variant_path = posixpath.sep.join(path_split[-2:]) + variant_ref = variant_group_ref.GetChildByPath(variant_path) + if variant_ref is not None: + assert variant_ref.__class__ == PBXFileReference + else: + variant_ref = PBXFileReference( + {"name": variant_name, "path": variant_path} + ) + variant_group_ref.AppendChild(variant_ref) + # The caller is interested in the variant group, not the specific + # variant file. + file_ref = variant_group_ref + return file_ref else: - file_ref = PBXFileReference({'path': path}) - self.AppendChild(file_ref) - else: - # Add or get a PBXVariantGroup. The variant group name is the same - # as the basename (MainMenu.nib in the example above). grandparent - # specifies the path to the variant group itself, and path_split[-2:] - # is the path of the specific variant relative to its group. - variant_group_name = posixpath.basename(path) - variant_group_ref = self.AddOrGetVariantGroupByNameAndPath( - variant_group_name, grandparent) - variant_path = posixpath.sep.join(path_split[-2:]) - variant_ref = variant_group_ref.GetChildByPath(variant_path) - if variant_ref != None: - assert variant_ref.__class__ == PBXFileReference - else: - variant_ref = PBXFileReference({'name': variant_name, - 'path': variant_path}) - variant_group_ref.AppendChild(variant_ref) - # The caller is interested in the variant group, not the specific - # variant file. - file_ref = variant_group_ref - return file_ref - else: - # Hierarchical recursion. Add or get a PBXGroup corresponding to the - # outermost path component, and then recurse into it, chopping off that - # path component. - next_dir = path_split[0] - group_ref = self.GetChildByPath(next_dir) - if group_ref != None: - assert group_ref.__class__ == PBXGroup - else: - group_ref = PBXGroup({'path': next_dir}) - self.AppendChild(group_ref) - return group_ref.AddOrGetFileByPath(posixpath.sep.join(path_split[1:]), - hierarchical) - - def AddOrGetVariantGroupByNameAndPath(self, name, path): - """Returns an existing or new PBXVariantGroup for name and path. + # Hierarchical recursion. Add or get a PBXGroup corresponding to the + # outermost path component, and then recurse into it, chopping off that + # path component. + next_dir = path_split[0] + group_ref = self.GetChildByPath(next_dir) + if group_ref is not None: + assert group_ref.__class__ == PBXGroup + else: + group_ref = PBXGroup({"path": next_dir}) + self.AppendChild(group_ref) + return group_ref.AddOrGetFileByPath( + posixpath.sep.join(path_split[1:]), hierarchical + ) + + def AddOrGetVariantGroupByNameAndPath(self, name, path): + """Returns an existing or new PBXVariantGroup for name and path. If a PBXVariantGroup identified by the name and path arguments is already present as a child of this object, it is returned. Otherwise, a new @@ -1311,22 +1391,22 @@ def AddOrGetVariantGroupByNameAndPath(self, name, path): passed to it. """ - key = (name, path) - if key in self._variant_children_by_name_and_path: - variant_group_ref = self._variant_children_by_name_and_path[key] - assert variant_group_ref.__class__ == PBXVariantGroup - return variant_group_ref + key = (name, path) + if key in self._variant_children_by_name_and_path: + variant_group_ref = self._variant_children_by_name_and_path[key] + assert variant_group_ref.__class__ == PBXVariantGroup + return variant_group_ref - variant_group_properties = {'name': name} - if path != None: - variant_group_properties['path'] = path - variant_group_ref = PBXVariantGroup(variant_group_properties) - self.AppendChild(variant_group_ref) + variant_group_properties = {"name": name} + if path is not None: + variant_group_properties["path"] = path + variant_group_ref = PBXVariantGroup(variant_group_properties) + self.AppendChild(variant_group_ref) - return variant_group_ref + return variant_group_ref - def TakeOverOnlyChild(self, recurse=False): - """If this PBXGroup has only one child and it's also a PBXGroup, take + def TakeOverOnlyChild(self, recurse=False): + """If this PBXGroup has only one child and it's also a PBXGroup, take it over by making all of its children this object's children. This function will continue to take over only children when those children @@ -1341,210 +1421,226 @@ def TakeOverOnlyChild(self, recurse=False): a group for a/b/c containing a group for d3/e. """ - # At this stage, check that child class types are PBXGroup exactly, - # instead of using isinstance. The only subclass of PBXGroup, - # PBXVariantGroup, should not participate in reparenting in the same way: - # reparenting by merging different object types would be wrong. - while len(self._properties['children']) == 1 and \ - self._properties['children'][0].__class__ == PBXGroup: - # Loop to take over the innermost only-child group possible. - - child = self._properties['children'][0] - - # Assume the child's properties, including its children. Save a copy - # of this object's old properties, because they'll still be needed. - # This object retains its existing id and parent attributes. - old_properties = self._properties - self._properties = child._properties - self._children_by_path = child._children_by_path - - if not 'sourceTree' in self._properties or \ - self._properties['sourceTree'] == '': - # The child was relative to its parent. Fix up the path. Note that - # children with a sourceTree other than "" are not relative to - # their parents, so no path fix-up is needed in that case. - if 'path' in old_properties: - if 'path' in self._properties: - # Both the original parent and child have paths set. - self._properties['path'] = posixpath.join(old_properties['path'], - self._properties['path']) - else: - # Only the original parent has a path, use it. - self._properties['path'] = old_properties['path'] - if 'sourceTree' in old_properties: - # The original parent had a sourceTree set, use it. - self._properties['sourceTree'] = old_properties['sourceTree'] - - # If the original parent had a name set, keep using it. If the original - # parent didn't have a name but the child did, let the child's name - # live on. If the name attribute seems unnecessary now, get rid of it. - if 'name' in old_properties and old_properties['name'] != None and \ - old_properties['name'] != self.Name(): - self._properties['name'] = old_properties['name'] - if 'name' in self._properties and 'path' in self._properties and \ - self._properties['name'] == self._properties['path']: - del self._properties['name'] - - # Notify all children of their new parent. - for child in self._properties['children']: - child.parent = self - - # If asked to recurse, recurse. - if recurse: - for child in self._properties['children']: - if child.__class__ == PBXGroup: - child.TakeOverOnlyChild(recurse) - - def SortGroup(self): - self._properties['children'] = \ - sorted(self._properties['children'], cmp=lambda x,y: x.Compare(y)) - - # Recurse. - for child in self._properties['children']: - if isinstance(child, PBXGroup): - child.SortGroup() + # At this stage, check that child class types are PBXGroup exactly, + # instead of using isinstance. The only subclass of PBXGroup, + # PBXVariantGroup, should not participate in reparenting in the same way: + # reparenting by merging different object types would be wrong. + while ( + len(self._properties["children"]) == 1 + and self._properties["children"][0].__class__ == PBXGroup + ): + # Loop to take over the innermost only-child group possible. + + child = self._properties["children"][0] + + # Assume the child's properties, including its children. Save a copy + # of this object's old properties, because they'll still be needed. + # This object retains its existing id and parent attributes. + old_properties = self._properties + self._properties = child._properties + self._children_by_path = child._children_by_path + + if ( + "sourceTree" not in self._properties + or self._properties["sourceTree"] == "" + ): + # The child was relative to its parent. Fix up the path. Note that + # children with a sourceTree other than "" are not relative to + # their parents, so no path fix-up is needed in that case. + if "path" in old_properties: + if "path" in self._properties: + # Both the original parent and child have paths set. + self._properties["path"] = posixpath.join( + old_properties["path"], self._properties["path"] + ) + else: + # Only the original parent has a path, use it. + self._properties["path"] = old_properties["path"] + if "sourceTree" in old_properties: + # The original parent had a sourceTree set, use it. + self._properties["sourceTree"] = old_properties["sourceTree"] + + # If the original parent had a name set, keep using it. If the original + # parent didn't have a name but the child did, let the child's name + # live on. If the name attribute seems unnecessary now, get rid of it. + if "name" in old_properties and old_properties["name"] not in ( + None, + self.Name(), + ): + self._properties["name"] = old_properties["name"] + if ( + "name" in self._properties + and "path" in self._properties + and self._properties["name"] == self._properties["path"] + ): + del self._properties["name"] + + # Notify all children of their new parent. + for child in self._properties["children"]: + child.parent = self + + # If asked to recurse, recurse. + if recurse: + for child in self._properties["children"]: + if child.__class__ == PBXGroup: + child.TakeOverOnlyChild(recurse) + + def SortGroup(self): + self._properties["children"] = sorted( + self._properties["children"], cmp=lambda x, y: x.Compare(y) + ) + + # Recurse. + for child in self._properties["children"]: + if isinstance(child, PBXGroup): + child.SortGroup() class XCFileLikeElement(XCHierarchicalElement): - # Abstract base for objects that can be used as the fileRef property of - # PBXBuildFile. - - def PathHashables(self): - # A PBXBuildFile that refers to this object will call this method to - # obtain additional hashables specific to this XCFileLikeElement. Don't - # just use this object's hashables, they're not specific and unique enough - # on their own (without access to the parent hashables.) Instead, provide - # hashables that identify this object by path by getting its hashables as - # well as the hashables of ancestor XCHierarchicalElement objects. - - hashables = [] - xche = self - while xche != None and isinstance(xche, XCHierarchicalElement): - xche_hashables = xche.Hashables() - for index in range(0, len(xche_hashables)): - hashables.insert(index, xche_hashables[index]) - xche = xche.parent - return hashables + # Abstract base for objects that can be used as the fileRef property of + # PBXBuildFile. + + def PathHashables(self): + # A PBXBuildFile that refers to this object will call this method to + # obtain additional hashables specific to this XCFileLikeElement. Don't + # just use this object's hashables, they're not specific and unique enough + # on their own (without access to the parent hashables.) Instead, provide + # hashables that identify this object by path by getting its hashables as + # well as the hashables of ancestor XCHierarchicalElement objects. + + hashables = [] + xche = self + while isinstance(xche, XCHierarchicalElement): + xche_hashables = xche.Hashables() + for index, xche_hashable in enumerate(xche_hashables): + hashables.insert(index, xche_hashable) + xche = xche.parent + return hashables class XCContainerPortal(XCObject): - # Abstract base for objects that can be used as the containerPortal property - # of PBXContainerItemProxy. - pass + # Abstract base for objects that can be used as the containerPortal property + # of PBXContainerItemProxy. + pass class XCRemoteObject(XCObject): - # Abstract base for objects that can be used as the remoteGlobalIDString - # property of PBXContainerItemProxy. - pass + # Abstract base for objects that can be used as the remoteGlobalIDString + # property of PBXContainerItemProxy. + pass class PBXFileReference(XCFileLikeElement, XCContainerPortal, XCRemoteObject): - _schema = XCFileLikeElement._schema.copy() - _schema.update({ - 'explicitFileType': [0, str, 0, 0], - 'lastKnownFileType': [0, str, 0, 0], - 'name': [0, str, 0, 0], - 'path': [0, str, 0, 1], - }) - - # Weird output rules for PBXFileReference. - _should_print_single_line = True - # super - _encode_transforms = XCFileLikeElement._alternate_encode_transforms - - def __init__(self, properties=None, id=None, parent=None): + _schema = XCFileLikeElement._schema.copy() + _schema.update( + { + "explicitFileType": [0, str, 0, 0], + "lastKnownFileType": [0, str, 0, 0], + "name": [0, str, 0, 0], + "path": [0, str, 0, 1], + } + ) + + # Weird output rules for PBXFileReference. + _should_print_single_line = True # super - XCFileLikeElement.__init__(self, properties, id, parent) - if 'path' in self._properties and self._properties['path'].endswith('/'): - self._properties['path'] = self._properties['path'][:-1] - is_dir = True - else: - is_dir = False - - if 'path' in self._properties and \ - not 'lastKnownFileType' in self._properties and \ - not 'explicitFileType' in self._properties: - # TODO(mark): This is the replacement for a replacement for a quick hack. - # It is no longer incredibly sucky, but this list needs to be extended. - extension_map = { - 'a': 'archive.ar', - 'app': 'wrapper.application', - 'bdic': 'file', - 'bundle': 'wrapper.cfbundle', - 'c': 'sourcecode.c.c', - 'cc': 'sourcecode.cpp.cpp', - 'cpp': 'sourcecode.cpp.cpp', - 'css': 'text.css', - 'cxx': 'sourcecode.cpp.cpp', - 'dart': 'sourcecode', - 'dylib': 'compiled.mach-o.dylib', - 'framework': 'wrapper.framework', - 'gyp': 'sourcecode', - 'gypi': 'sourcecode', - 'h': 'sourcecode.c.h', - 'hxx': 'sourcecode.cpp.h', - 'icns': 'image.icns', - 'java': 'sourcecode.java', - 'js': 'sourcecode.javascript', - 'kext': 'wrapper.kext', - 'm': 'sourcecode.c.objc', - 'mm': 'sourcecode.cpp.objcpp', - 'nib': 'wrapper.nib', - 'o': 'compiled.mach-o.objfile', - 'pdf': 'image.pdf', - 'pl': 'text.script.perl', - 'plist': 'text.plist.xml', - 'pm': 'text.script.perl', - 'png': 'image.png', - 'py': 'text.script.python', - 'r': 'sourcecode.rez', - 'rez': 'sourcecode.rez', - 's': 'sourcecode.asm', - 'storyboard': 'file.storyboard', - 'strings': 'text.plist.strings', - 'swift': 'sourcecode.swift', - 'ttf': 'file', - 'xcassets': 'folder.assetcatalog', - 'xcconfig': 'text.xcconfig', - 'xcdatamodel': 'wrapper.xcdatamodel', - 'xcdatamodeld':'wrapper.xcdatamodeld', - 'xib': 'file.xib', - 'y': 'sourcecode.yacc', - } - - prop_map = { - 'dart': 'explicitFileType', - 'gyp': 'explicitFileType', - 'gypi': 'explicitFileType', - } - - if is_dir: - file_type = 'folder' - prop_name = 'lastKnownFileType' - else: - basename = posixpath.basename(self._properties['path']) - (root, ext) = posixpath.splitext(basename) - # Check the map using a lowercase extension. - # TODO(mark): Maybe it should try with the original case first and fall - # back to lowercase, in case there are any instances where case - # matters. There currently aren't. - if ext != '': - ext = ext[1:].lower() - - # TODO(mark): "text" is the default value, but "file" is appropriate - # for unrecognized files not containing text. Xcode seems to choose - # based on content. - file_type = extension_map.get(ext, 'text') - prop_name = prop_map.get(ext, 'lastKnownFileType') - - self._properties[prop_name] = file_type + _encode_transforms = XCFileLikeElement._alternate_encode_transforms + + def __init__(self, properties=None, id=None, parent=None): + # super + XCFileLikeElement.__init__(self, properties, id, parent) + if "path" in self._properties and self._properties["path"].endswith("/"): + self._properties["path"] = self._properties["path"][:-1] + is_dir = True + else: + is_dir = False + + if ( + "path" in self._properties + and "lastKnownFileType" not in self._properties + and "explicitFileType" not in self._properties + ): + # TODO(mark): This is the replacement for a replacement for a quick hack. + # It is no longer incredibly sucky, but this list needs to be extended. + extension_map = { + "a": "archive.ar", + "app": "wrapper.application", + "bdic": "file", + "bundle": "wrapper.cfbundle", + "c": "sourcecode.c.c", + "cc": "sourcecode.cpp.cpp", + "cpp": "sourcecode.cpp.cpp", + "css": "text.css", + "cxx": "sourcecode.cpp.cpp", + "dart": "sourcecode", + "dylib": "compiled.mach-o.dylib", + "framework": "wrapper.framework", + "gyp": "sourcecode", + "gypi": "sourcecode", + "h": "sourcecode.c.h", + "hxx": "sourcecode.cpp.h", + "icns": "image.icns", + "java": "sourcecode.java", + "js": "sourcecode.javascript", + "kext": "wrapper.kext", + "m": "sourcecode.c.objc", + "mm": "sourcecode.cpp.objcpp", + "nib": "wrapper.nib", + "o": "compiled.mach-o.objfile", + "pdf": "image.pdf", + "pl": "text.script.perl", + "plist": "text.plist.xml", + "pm": "text.script.perl", + "png": "image.png", + "py": "text.script.python", + "r": "sourcecode.rez", + "rez": "sourcecode.rez", + "s": "sourcecode.asm", + "storyboard": "file.storyboard", + "strings": "text.plist.strings", + "swift": "sourcecode.swift", + "ttf": "file", + "xcassets": "folder.assetcatalog", + "xcconfig": "text.xcconfig", + "xcdatamodel": "wrapper.xcdatamodel", + "xcdatamodeld": "wrapper.xcdatamodeld", + "xib": "file.xib", + "y": "sourcecode.yacc", + } + + prop_map = { + "dart": "explicitFileType", + "gyp": "explicitFileType", + "gypi": "explicitFileType", + } + + if is_dir: + file_type = "folder" + prop_name = "lastKnownFileType" + else: + basename = posixpath.basename(self._properties["path"]) + (root, ext) = posixpath.splitext(basename) + # Check the map using a lowercase extension. + # TODO(mark): Maybe it should try with the original case first and fall + # back to lowercase, in case there are any instances where case + # matters. There currently aren't. + if ext != "": + ext = ext[1:].lower() + + # TODO(mark): "text" is the default value, but "file" is appropriate + # for unrecognized files not containing text. Xcode seems to choose + # based on content. + file_type = extension_map.get(ext, "text") + prop_name = prop_map.get(ext, "lastKnownFileType") + + self._properties[prop_name] = file_type class PBXVariantGroup(PBXGroup, XCFileLikeElement): - """PBXVariantGroup is used by Xcode to represent localizations.""" - # No additions to the schema relative to PBXGroup. - pass + """PBXVariantGroup is used by Xcode to represent localizations.""" + + # No additions to the schema relative to PBXGroup. + pass # PBXReferenceProxy is also an XCFileLikeElement subclass. It is defined below @@ -1552,65 +1648,77 @@ class PBXVariantGroup(PBXGroup, XCFileLikeElement): class XCBuildConfiguration(XCObject): - _schema = XCObject._schema.copy() - _schema.update({ - 'baseConfigurationReference': [0, PBXFileReference, 0, 0], - 'buildSettings': [0, dict, 0, 1, {}], - 'name': [0, str, 0, 1], - }) + _schema = XCObject._schema.copy() + _schema.update( + { + "baseConfigurationReference": [0, PBXFileReference, 0, 0], + "buildSettings": [0, dict, 0, 1, {}], + "name": [0, str, 0, 1], + } + ) + + def HasBuildSetting(self, key): + return key in self._properties["buildSettings"] - def HasBuildSetting(self, key): - return key in self._properties['buildSettings'] + def GetBuildSetting(self, key): + return self._properties["buildSettings"][key] - def GetBuildSetting(self, key): - return self._properties['buildSettings'][key] + def SetBuildSetting(self, key, value): + # TODO(mark): If a list, copy? + self._properties["buildSettings"][key] = value - def SetBuildSetting(self, key, value): - # TODO(mark): If a list, copy? - self._properties['buildSettings'][key] = value + def AppendBuildSetting(self, key, value): + if key not in self._properties["buildSettings"]: + self._properties["buildSettings"][key] = [] + self._properties["buildSettings"][key].append(value) - def AppendBuildSetting(self, key, value): - if not key in self._properties['buildSettings']: - self._properties['buildSettings'][key] = [] - self._properties['buildSettings'][key].append(value) + def DelBuildSetting(self, key): + if key in self._properties["buildSettings"]: + del self._properties["buildSettings"][key] - def DelBuildSetting(self, key): - if key in self._properties['buildSettings']: - del self._properties['buildSettings'][key] + def SetBaseConfiguration(self, value): + self._properties["baseConfigurationReference"] = value - def SetBaseConfiguration(self, value): - self._properties['baseConfigurationReference'] = value class XCConfigurationList(XCObject): - # _configs is the default list of configurations. - _configs = [ XCBuildConfiguration({'name': 'Debug'}), - XCBuildConfiguration({'name': 'Release'}) ] - - _schema = XCObject._schema.copy() - _schema.update({ - 'buildConfigurations': [1, XCBuildConfiguration, 1, 1, _configs], - 'defaultConfigurationIsVisible': [0, int, 0, 1, 1], - 'defaultConfigurationName': [0, str, 0, 1, 'Release'], - }) - - def Name(self): - return 'Build configuration list for ' + \ - self.parent.__class__.__name__ + ' "' + self.parent.Name() + '"' - - def ConfigurationNamed(self, name): - """Convenience accessor to obtain an XCBuildConfiguration by name.""" - for configuration in self._properties['buildConfigurations']: - if configuration._properties['name'] == name: - return configuration - - raise KeyError(name) - - def DefaultConfiguration(self): - """Convenience accessor to obtain the default XCBuildConfiguration.""" - return self.ConfigurationNamed(self._properties['defaultConfigurationName']) - - def HasBuildSetting(self, key): - """Determines the state of a build setting in all XCBuildConfiguration + # _configs is the default list of configurations. + _configs = [ + XCBuildConfiguration({"name": "Debug"}), + XCBuildConfiguration({"name": "Release"}), + ] + + _schema = XCObject._schema.copy() + _schema.update( + { + "buildConfigurations": [1, XCBuildConfiguration, 1, 1, _configs], + "defaultConfigurationIsVisible": [0, int, 0, 1, 1], + "defaultConfigurationName": [0, str, 0, 1, "Release"], + } + ) + + def Name(self): + return ( + "Build configuration list for " + + self.parent.__class__.__name__ + + ' "' + + self.parent.Name() + + '"' + ) + + def ConfigurationNamed(self, name): + """Convenience accessor to obtain an XCBuildConfiguration by name.""" + for configuration in self._properties["buildConfigurations"]: + if configuration._properties["name"] == name: + return configuration + + raise KeyError(name) + + def DefaultConfiguration(self): + """Convenience accessor to obtain the default XCBuildConfiguration.""" + return self.ConfigurationNamed(self._properties["defaultConfigurationName"]) + + def HasBuildSetting(self, key): + """Determines the state of a build setting in all XCBuildConfiguration child objects. If all child objects have key in their build settings, and the value is the @@ -1622,112 +1730,114 @@ def HasBuildSetting(self, key): or if any children have different values for the key, returns -1. """ - has = None - value = None - for configuration in self._properties['buildConfigurations']: - configuration_has = configuration.HasBuildSetting(key) - if has is None: - has = configuration_has - elif has != configuration_has: - return -1 + has = None + value = None + for configuration in self._properties["buildConfigurations"]: + configuration_has = configuration.HasBuildSetting(key) + if has is None: + has = configuration_has + elif has != configuration_has: + return -1 - if configuration_has: - configuration_value = configuration.GetBuildSetting(key) - if value is None: - value = configuration_value - elif value != configuration_value: - return -1 + if configuration_has: + configuration_value = configuration.GetBuildSetting(key) + if value is None: + value = configuration_value + elif value != configuration_value: + return -1 - if not has: - return 0 + if not has: + return 0 - return 1 + return 1 - def GetBuildSetting(self, key): - """Gets the build setting for key. + def GetBuildSetting(self, key): + """Gets the build setting for key. All child XCConfiguration objects must have the same value set for the setting, or a ValueError will be raised. """ - # TODO(mark): This is wrong for build settings that are lists. The list - # contents should be compared (and a list copy returned?) + # TODO(mark): This is wrong for build settings that are lists. The list + # contents should be compared (and a list copy returned?) - value = None - for configuration in self._properties['buildConfigurations']: - configuration_value = configuration.GetBuildSetting(key) - if value is None: - value = configuration_value - else: - if value != configuration_value: - raise ValueError('Variant values for ' + key) + value = None + for configuration in self._properties["buildConfigurations"]: + configuration_value = configuration.GetBuildSetting(key) + if value is None: + value = configuration_value + else: + if value != configuration_value: + raise ValueError("Variant values for " + key) - return value + return value - def SetBuildSetting(self, key, value): - """Sets the build setting for key to value in all child + def SetBuildSetting(self, key, value): + """Sets the build setting for key to value in all child XCBuildConfiguration objects. """ - for configuration in self._properties['buildConfigurations']: - configuration.SetBuildSetting(key, value) + for configuration in self._properties["buildConfigurations"]: + configuration.SetBuildSetting(key, value) - def AppendBuildSetting(self, key, value): - """Appends value to the build setting for key, which is treated as a list, + def AppendBuildSetting(self, key, value): + """Appends value to the build setting for key, which is treated as a list, in all child XCBuildConfiguration objects. """ - for configuration in self._properties['buildConfigurations']: - configuration.AppendBuildSetting(key, value) + for configuration in self._properties["buildConfigurations"]: + configuration.AppendBuildSetting(key, value) - def DelBuildSetting(self, key): - """Deletes the build setting key from all child XCBuildConfiguration + def DelBuildSetting(self, key): + """Deletes the build setting key from all child XCBuildConfiguration objects. """ - for configuration in self._properties['buildConfigurations']: - configuration.DelBuildSetting(key) + for configuration in self._properties["buildConfigurations"]: + configuration.DelBuildSetting(key) - def SetBaseConfiguration(self, value): - """Sets the build configuration in all child XCBuildConfiguration objects. + def SetBaseConfiguration(self, value): + """Sets the build configuration in all child XCBuildConfiguration objects. """ - for configuration in self._properties['buildConfigurations']: - configuration.SetBaseConfiguration(value) + for configuration in self._properties["buildConfigurations"]: + configuration.SetBaseConfiguration(value) class PBXBuildFile(XCObject): - _schema = XCObject._schema.copy() - _schema.update({ - 'fileRef': [0, XCFileLikeElement, 0, 1], - 'settings': [0, str, 0, 0], # hack, it's a dict - }) + _schema = XCObject._schema.copy() + _schema.update( + { + "fileRef": [0, XCFileLikeElement, 0, 1], + "settings": [0, str, 0, 0], # hack, it's a dict + } + ) - # Weird output rules for PBXBuildFile. - _should_print_single_line = True - _encode_transforms = XCObject._alternate_encode_transforms + # Weird output rules for PBXBuildFile. + _should_print_single_line = True + _encode_transforms = XCObject._alternate_encode_transforms - def Name(self): - # Example: "main.cc in Sources" - return self._properties['fileRef'].Name() + ' in ' + self.parent.Name() + def Name(self): + # Example: "main.cc in Sources" + return self._properties["fileRef"].Name() + " in " + self.parent.Name() - def Hashables(self): - # super - hashables = XCObject.Hashables(self) + def Hashables(self): + # super + hashables = XCObject.Hashables(self) - # It is not sufficient to just rely on Name() to get the - # XCFileLikeElement's name, because that is not a complete pathname. - # PathHashables returns hashables unique enough that no two - # PBXBuildFiles should wind up with the same set of hashables, unless - # someone adds the same file multiple times to the same target. That - # would be considered invalid anyway. - hashables.extend(self._properties['fileRef'].PathHashables()) + # It is not sufficient to just rely on Name() to get the + # XCFileLikeElement's name, because that is not a complete pathname. + # PathHashables returns hashables unique enough that no two + # PBXBuildFiles should wind up with the same set of hashables, unless + # someone adds the same file multiple times to the same target. That + # would be considered invalid anyway. + hashables.extend(self._properties["fileRef"].PathHashables()) - return hashables + return hashables class XCBuildPhase(XCObject): - """Abstract base for build phase classes. Not represented in a project + """Abstract base for build phase classes. Not represented in a project file. Attributes: @@ -1737,51 +1847,52 @@ class XCBuildPhase(XCObject): to the corresponding PBXBuildFile children (values). """ - # TODO(mark): Some build phase types, like PBXShellScriptBuildPhase, don't - # actually have a "files" list. XCBuildPhase should not have "files" but - # another abstract subclass of it should provide this, and concrete build - # phase types that do have "files" lists should be derived from that new - # abstract subclass. XCBuildPhase should only provide buildActionMask and - # runOnlyForDeploymentPostprocessing, and not files or the various - # file-related methods and attributes. - - _schema = XCObject._schema.copy() - _schema.update({ - 'buildActionMask': [0, int, 0, 1, 0x7fffffff], - 'files': [1, PBXBuildFile, 1, 1, []], - 'runOnlyForDeploymentPostprocessing': [0, int, 0, 1, 0], - }) - - def __init__(self, properties=None, id=None, parent=None): - # super - XCObject.__init__(self, properties, id, parent) + # TODO(mark): Some build phase types, like PBXShellScriptBuildPhase, don't + # actually have a "files" list. XCBuildPhase should not have "files" but + # another abstract subclass of it should provide this, and concrete build + # phase types that do have "files" lists should be derived from that new + # abstract subclass. XCBuildPhase should only provide buildActionMask and + # runOnlyForDeploymentPostprocessing, and not files or the various + # file-related methods and attributes. + + _schema = XCObject._schema.copy() + _schema.update( + { + "buildActionMask": [0, int, 0, 1, 0x7FFFFFFF], + "files": [1, PBXBuildFile, 1, 1, []], + "runOnlyForDeploymentPostprocessing": [0, int, 0, 1, 0], + } + ) - self._files_by_path = {} - self._files_by_xcfilelikeelement = {} - for pbxbuildfile in self._properties.get('files', []): - self._AddBuildFileToDicts(pbxbuildfile) + def __init__(self, properties=None, id=None, parent=None): + # super + XCObject.__init__(self, properties, id, parent) - def FileGroup(self, path): - # Subclasses must override this by returning a two-element tuple. The - # first item in the tuple should be the PBXGroup to which "path" should be - # added, either as a child or deeper descendant. The second item should - # be a boolean indicating whether files should be added into hierarchical - # groups or one single flat group. - raise NotImplementedError( - self.__class__.__name__ + ' must implement FileGroup') + self._files_by_path = {} + self._files_by_xcfilelikeelement = {} + for pbxbuildfile in self._properties.get("files", []): + self._AddBuildFileToDicts(pbxbuildfile) - def _AddPathToDict(self, pbxbuildfile, path): - """Adds path to the dict tracking paths belonging to this build phase. + def FileGroup(self, path): + # Subclasses must override this by returning a two-element tuple. The + # first item in the tuple should be the PBXGroup to which "path" should be + # added, either as a child or deeper descendant. The second item should + # be a boolean indicating whether files should be added into hierarchical + # groups or one single flat group. + raise NotImplementedError(self.__class__.__name__ + " must implement FileGroup") + + def _AddPathToDict(self, pbxbuildfile, path): + """Adds path to the dict tracking paths belonging to this build phase. If the path is already a member of this build phase, raises an exception. """ - if path in self._files_by_path: - raise ValueError('Found multiple build files with path ' + path) - self._files_by_path[path] = pbxbuildfile + if path in self._files_by_path: + raise ValueError("Found multiple build files with path " + path) + self._files_by_path[path] = pbxbuildfile - def _AddBuildFileToDicts(self, pbxbuildfile, path=None): - """Maintains the _files_by_path and _files_by_xcfilelikeelement dicts. + def _AddBuildFileToDicts(self, pbxbuildfile, path=None): + """Maintains the _files_by_path and _files_by_xcfilelikeelement dicts. If path is specified, then it is the path that is being added to the phase, and pbxbuildfile must contain either a PBXFileReference directly @@ -1806,751 +1917,823 @@ def _AddBuildFileToDicts(self, pbxbuildfile, path=None): the PBXBuildFile if it is already present in the list of children. """ - xcfilelikeelement = pbxbuildfile._properties['fileRef'] + xcfilelikeelement = pbxbuildfile._properties["fileRef"] - paths = [] - if path != None: - # It's best when the caller provides the path. - if isinstance(xcfilelikeelement, PBXVariantGroup): - paths.append(path) - else: - # If the caller didn't provide a path, there can be either multiple - # paths (PBXVariantGroup) or one. - if isinstance(xcfilelikeelement, PBXVariantGroup): - for variant in xcfilelikeelement._properties['children']: - paths.append(variant.FullPath()) - else: - paths.append(xcfilelikeelement.FullPath()) - - # Add the paths first, because if something's going to raise, the - # messages provided by _AddPathToDict are more useful owing to its - # having access to a real pathname and not just an object's Name(). - for a_path in paths: - self._AddPathToDict(pbxbuildfile, a_path) - - # If another PBXBuildFile references this XCFileLikeElement, there's a - # problem. - if xcfilelikeelement in self._files_by_xcfilelikeelement and \ - self._files_by_xcfilelikeelement[xcfilelikeelement] != pbxbuildfile: - raise ValueError('Found multiple build files for ' + \ - xcfilelikeelement.Name()) - self._files_by_xcfilelikeelement[xcfilelikeelement] = pbxbuildfile - - def AppendBuildFile(self, pbxbuildfile, path=None): - # Callers should use this instead of calling - # AppendProperty('files', pbxbuildfile) directly because this function - # maintains the object's dicts. Better yet, callers can just call AddFile - # with a pathname and not worry about building their own PBXBuildFile - # objects. - self.AppendProperty('files', pbxbuildfile) - self._AddBuildFileToDicts(pbxbuildfile, path) - - def AddFile(self, path, settings=None): - (file_group, hierarchical) = self.FileGroup(path) - file_ref = file_group.AddOrGetFileByPath(path, hierarchical) - - if file_ref in self._files_by_xcfilelikeelement and \ - isinstance(file_ref, PBXVariantGroup): - # There's already a PBXBuildFile in this phase corresponding to the - # PBXVariantGroup. path just provides a new variant that belongs to - # the group. Add the path to the dict. - pbxbuildfile = self._files_by_xcfilelikeelement[file_ref] - self._AddBuildFileToDicts(pbxbuildfile, path) - else: - # Add a new PBXBuildFile to get file_ref into the phase. - if settings is None: - pbxbuildfile = PBXBuildFile({'fileRef': file_ref}) - else: - pbxbuildfile = PBXBuildFile({'fileRef': file_ref, 'settings': settings}) - self.AppendBuildFile(pbxbuildfile, path) + paths = [] + if path is not None: + # It's best when the caller provides the path. + if isinstance(xcfilelikeelement, PBXVariantGroup): + paths.append(path) + else: + # If the caller didn't provide a path, there can be either multiple + # paths (PBXVariantGroup) or one. + if isinstance(xcfilelikeelement, PBXVariantGroup): + for variant in xcfilelikeelement._properties["children"]: + paths.append(variant.FullPath()) + else: + paths.append(xcfilelikeelement.FullPath()) + + # Add the paths first, because if something's going to raise, the + # messages provided by _AddPathToDict are more useful owing to its + # having access to a real pathname and not just an object's Name(). + for a_path in paths: + self._AddPathToDict(pbxbuildfile, a_path) + + # If another PBXBuildFile references this XCFileLikeElement, there's a + # problem. + if ( + xcfilelikeelement in self._files_by_xcfilelikeelement + and self._files_by_xcfilelikeelement[xcfilelikeelement] != pbxbuildfile + ): + raise ValueError( + "Found multiple build files for " + xcfilelikeelement.Name() + ) + self._files_by_xcfilelikeelement[xcfilelikeelement] = pbxbuildfile + + def AppendBuildFile(self, pbxbuildfile, path=None): + # Callers should use this instead of calling + # AppendProperty('files', pbxbuildfile) directly because this function + # maintains the object's dicts. Better yet, callers can just call AddFile + # with a pathname and not worry about building their own PBXBuildFile + # objects. + self.AppendProperty("files", pbxbuildfile) + self._AddBuildFileToDicts(pbxbuildfile, path) + + def AddFile(self, path, settings=None): + (file_group, hierarchical) = self.FileGroup(path) + file_ref = file_group.AddOrGetFileByPath(path, hierarchical) + + if file_ref in self._files_by_xcfilelikeelement and isinstance( + file_ref, PBXVariantGroup + ): + # There's already a PBXBuildFile in this phase corresponding to the + # PBXVariantGroup. path just provides a new variant that belongs to + # the group. Add the path to the dict. + pbxbuildfile = self._files_by_xcfilelikeelement[file_ref] + self._AddBuildFileToDicts(pbxbuildfile, path) + else: + # Add a new PBXBuildFile to get file_ref into the phase. + if settings is None: + pbxbuildfile = PBXBuildFile({"fileRef": file_ref}) + else: + pbxbuildfile = PBXBuildFile({"fileRef": file_ref, "settings": settings}) + self.AppendBuildFile(pbxbuildfile, path) class PBXHeadersBuildPhase(XCBuildPhase): - # No additions to the schema relative to XCBuildPhase. + # No additions to the schema relative to XCBuildPhase. - def Name(self): - return 'Headers' + def Name(self): + return "Headers" - def FileGroup(self, path): - return self.PBXProjectAncestor().RootGroupForPath(path) + def FileGroup(self, path): + return self.PBXProjectAncestor().RootGroupForPath(path) class PBXResourcesBuildPhase(XCBuildPhase): - # No additions to the schema relative to XCBuildPhase. + # No additions to the schema relative to XCBuildPhase. - def Name(self): - return 'Resources' + def Name(self): + return "Resources" - def FileGroup(self, path): - return self.PBXProjectAncestor().RootGroupForPath(path) + def FileGroup(self, path): + return self.PBXProjectAncestor().RootGroupForPath(path) class PBXSourcesBuildPhase(XCBuildPhase): - # No additions to the schema relative to XCBuildPhase. + # No additions to the schema relative to XCBuildPhase. - def Name(self): - return 'Sources' + def Name(self): + return "Sources" - def FileGroup(self, path): - return self.PBXProjectAncestor().RootGroupForPath(path) + def FileGroup(self, path): + return self.PBXProjectAncestor().RootGroupForPath(path) class PBXFrameworksBuildPhase(XCBuildPhase): - # No additions to the schema relative to XCBuildPhase. - - def Name(self): - return 'Frameworks' - - def FileGroup(self, path): - (root, ext) = posixpath.splitext(path) - if ext != '': - ext = ext[1:].lower() - if ext == 'o': - # .o files are added to Xcode Frameworks phases, but conceptually aren't - # frameworks, they're more like sources or intermediates. Redirect them - # to show up in one of those other groups. - return self.PBXProjectAncestor().RootGroupForPath(path) - else: - return (self.PBXProjectAncestor().FrameworksGroup(), False) + # No additions to the schema relative to XCBuildPhase. + + def Name(self): + return "Frameworks" + + def FileGroup(self, path): + (root, ext) = posixpath.splitext(path) + if ext != "": + ext = ext[1:].lower() + if ext == "o": + # .o files are added to Xcode Frameworks phases, but conceptually aren't + # frameworks, they're more like sources or intermediates. Redirect them + # to show up in one of those other groups. + return self.PBXProjectAncestor().RootGroupForPath(path) + else: + return (self.PBXProjectAncestor().FrameworksGroup(), False) class PBXShellScriptBuildPhase(XCBuildPhase): - _schema = XCBuildPhase._schema.copy() - _schema.update({ - 'inputPaths': [1, str, 0, 1, []], - 'name': [0, str, 0, 0], - 'outputPaths': [1, str, 0, 1, []], - 'shellPath': [0, str, 0, 1, '/bin/sh'], - 'shellScript': [0, str, 0, 1], - 'showEnvVarsInLog': [0, int, 0, 0], - }) + _schema = XCBuildPhase._schema.copy() + _schema.update( + { + "inputPaths": [1, str, 0, 1, []], + "name": [0, str, 0, 0], + "outputPaths": [1, str, 0, 1, []], + "shellPath": [0, str, 0, 1, "/bin/sh"], + "shellScript": [0, str, 0, 1], + "showEnvVarsInLog": [0, int, 0, 0], + } + ) - def Name(self): - if 'name' in self._properties: - return self._properties['name'] + def Name(self): + if "name" in self._properties: + return self._properties["name"] - return 'ShellScript' + return "ShellScript" class PBXCopyFilesBuildPhase(XCBuildPhase): - _schema = XCBuildPhase._schema.copy() - _schema.update({ - 'dstPath': [0, str, 0, 1], - 'dstSubfolderSpec': [0, int, 0, 1], - 'name': [0, str, 0, 0], - }) - - # path_tree_re matches "$(DIR)/path", "$(DIR)/$(DIR2)/path" or just "$(DIR)". - # Match group 1 is "DIR", group 3 is "path" or "$(DIR2") or "$(DIR2)/path" - # or None. If group 3 is "path", group 4 will be None otherwise group 4 is - # "DIR2" and group 6 is "path". - path_tree_re = re.compile(r'^\$\((.*?)\)(/(\$\((.*?)\)(/(.*)|)|(.*)|)|)$') - - # path_tree_{first,second}_to_subfolder map names of Xcode variables to the - # associated dstSubfolderSpec property value used in a PBXCopyFilesBuildPhase - # object. - path_tree_first_to_subfolder = { - # Types that can be chosen via the Xcode UI. - 'BUILT_PRODUCTS_DIR': 16, # Products Directory - 'BUILT_FRAMEWORKS_DIR': 10, # Not an official Xcode macro. - # Existed before support for the - # names below was added. Maps to - # "Frameworks". - } - - path_tree_second_to_subfolder = { - 'WRAPPER_NAME': 1, # Wrapper - # Although Xcode's friendly name is "Executables", the destination - # is demonstrably the value of the build setting - # EXECUTABLE_FOLDER_PATH not EXECUTABLES_FOLDER_PATH. - 'EXECUTABLE_FOLDER_PATH': 6, # Executables. - 'UNLOCALIZED_RESOURCES_FOLDER_PATH': 7, # Resources - 'JAVA_FOLDER_PATH': 15, # Java Resources - 'FRAMEWORKS_FOLDER_PATH': 10, # Frameworks - 'SHARED_FRAMEWORKS_FOLDER_PATH': 11, # Shared Frameworks - 'SHARED_SUPPORT_FOLDER_PATH': 12, # Shared Support - 'PLUGINS_FOLDER_PATH': 13, # PlugIns - # For XPC Services, Xcode sets both dstPath and dstSubfolderSpec. - # Note that it re-uses the BUILT_PRODUCTS_DIR value for - # dstSubfolderSpec. dstPath is set below. - 'XPCSERVICES_FOLDER_PATH': 16, # XPC Services. - } - - def Name(self): - if 'name' in self._properties: - return self._properties['name'] - - return 'CopyFiles' - - def FileGroup(self, path): - return self.PBXProjectAncestor().RootGroupForPath(path) - - def SetDestination(self, path): - """Set the dstSubfolderSpec and dstPath properties from path. + _schema = XCBuildPhase._schema.copy() + _schema.update( + { + "dstPath": [0, str, 0, 1], + "dstSubfolderSpec": [0, int, 0, 1], + "name": [0, str, 0, 0], + } + ) + + # path_tree_re matches "$(DIR)/path", "$(DIR)/$(DIR2)/path" or just "$(DIR)". + # Match group 1 is "DIR", group 3 is "path" or "$(DIR2") or "$(DIR2)/path" + # or None. If group 3 is "path", group 4 will be None otherwise group 4 is + # "DIR2" and group 6 is "path". + path_tree_re = re.compile(r"^\$\((.*?)\)(/(\$\((.*?)\)(/(.*)|)|(.*)|)|)$") + + # path_tree_{first,second}_to_subfolder map names of Xcode variables to the + # associated dstSubfolderSpec property value used in a PBXCopyFilesBuildPhase + # object. + path_tree_first_to_subfolder = { + # Types that can be chosen via the Xcode UI. + "BUILT_PRODUCTS_DIR": 16, # Products Directory + "BUILT_FRAMEWORKS_DIR": 10, # Not an official Xcode macro. + # Existed before support for the + # names below was added. Maps to + # "Frameworks". + } + + path_tree_second_to_subfolder = { + "WRAPPER_NAME": 1, # Wrapper + # Although Xcode's friendly name is "Executables", the destination + # is demonstrably the value of the build setting + # EXECUTABLE_FOLDER_PATH not EXECUTABLES_FOLDER_PATH. + "EXECUTABLE_FOLDER_PATH": 6, # Executables. + "UNLOCALIZED_RESOURCES_FOLDER_PATH": 7, # Resources + "JAVA_FOLDER_PATH": 15, # Java Resources + "FRAMEWORKS_FOLDER_PATH": 10, # Frameworks + "SHARED_FRAMEWORKS_FOLDER_PATH": 11, # Shared Frameworks + "SHARED_SUPPORT_FOLDER_PATH": 12, # Shared Support + "PLUGINS_FOLDER_PATH": 13, # PlugIns + # For XPC Services, Xcode sets both dstPath and dstSubfolderSpec. + # Note that it re-uses the BUILT_PRODUCTS_DIR value for + # dstSubfolderSpec. dstPath is set below. + "XPCSERVICES_FOLDER_PATH": 16, # XPC Services. + } + + def Name(self): + if "name" in self._properties: + return self._properties["name"] + + return "CopyFiles" + + def FileGroup(self, path): + return self.PBXProjectAncestor().RootGroupForPath(path) + + def SetDestination(self, path): + """Set the dstSubfolderSpec and dstPath properties from path. path may be specified in the same notation used for XCHierarchicalElements, specifically, "$(DIR)/path". """ - path_tree_match = self.path_tree_re.search(path) - if path_tree_match: - path_tree = path_tree_match.group(1) - if path_tree in self.path_tree_first_to_subfolder: - subfolder = self.path_tree_first_to_subfolder[path_tree] - relative_path = path_tree_match.group(3) - if relative_path is None: - relative_path = '' - - if subfolder == 16 and path_tree_match.group(4) is not None: - # BUILT_PRODUCTS_DIR (16) is the first element in a path whose - # second element is possibly one of the variable names in - # path_tree_second_to_subfolder. Xcode sets the values of all these - # variables to relative paths so .gyp files must prefix them with - # BUILT_PRODUCTS_DIR, e.g. - # $(BUILT_PRODUCTS_DIR)/$(PLUGINS_FOLDER_PATH). Then - # xcode_emulation.py can export these variables with the same values - # as Xcode yet make & ninja files can determine the absolute path - # to the target. Xcode uses the dstSubfolderSpec value set here - # to determine the full path. - # - # An alternative of xcode_emulation.py setting the values to absolute - # paths when exporting these variables has been ruled out because - # then the values would be different depending on the build tool. - # - # Another alternative is to invent new names for the variables used - # to match to the subfolder indices in the second table. .gyp files - # then will not need to prepend $(BUILT_PRODUCTS_DIR) because - # xcode_emulation.py can set the values of those variables to - # the absolute paths when exporting. This is possibly the thinking - # behind BUILT_FRAMEWORKS_DIR which is used in exactly this manner. - # - # Requiring prepending BUILT_PRODUCTS_DIR has been chosen because - # this same way could be used to specify destinations in .gyp files - # that pre-date this addition to GYP. However they would only work - # with the Xcode generator. The previous version of xcode_emulation.py - # does not export these variables. Such files will get the benefit - # of the Xcode UI showing the proper destination name simply by - # regenerating the projects with this version of GYP. - path_tree = path_tree_match.group(4) - relative_path = path_tree_match.group(6) - separator = '/' - - if path_tree in self.path_tree_second_to_subfolder: - subfolder = self.path_tree_second_to_subfolder[path_tree] - if relative_path is None: - relative_path = '' - separator = '' - if path_tree == 'XPCSERVICES_FOLDER_PATH': - relative_path = '$(CONTENTS_FOLDER_PATH)/XPCServices' \ - + separator + relative_path - else: - # subfolder = 16 from above - # The second element of the path is an unrecognized variable. - # Include it and any remaining elements in relative_path. - relative_path = path_tree_match.group(3) - - else: - # The path starts with an unrecognized Xcode variable - # name like $(SRCROOT). Xcode will still handle this - # as an "absolute path" that starts with the variable. - subfolder = 0 - relative_path = path - elif path.startswith('/'): - # Special case. Absolute paths are in dstSubfolderSpec 0. - subfolder = 0 - relative_path = path[1:] - else: - raise ValueError('Can\'t use path %s in a %s' % \ - (path, self.__class__.__name__)) + path_tree_match = self.path_tree_re.search(path) + if path_tree_match: + path_tree = path_tree_match.group(1) + if path_tree in self.path_tree_first_to_subfolder: + subfolder = self.path_tree_first_to_subfolder[path_tree] + relative_path = path_tree_match.group(3) + if relative_path is None: + relative_path = "" + + if subfolder == 16 and path_tree_match.group(4) is not None: + # BUILT_PRODUCTS_DIR (16) is the first element in a path whose + # second element is possibly one of the variable names in + # path_tree_second_to_subfolder. Xcode sets the values of all these + # variables to relative paths so .gyp files must prefix them with + # BUILT_PRODUCTS_DIR, e.g. + # $(BUILT_PRODUCTS_DIR)/$(PLUGINS_FOLDER_PATH). Then + # xcode_emulation.py can export these variables with the same values + # as Xcode yet make & ninja files can determine the absolute path + # to the target. Xcode uses the dstSubfolderSpec value set here + # to determine the full path. + # + # An alternative of xcode_emulation.py setting the values to absolute + # paths when exporting these variables has been ruled out because + # then the values would be different depending on the build tool. + # + # Another alternative is to invent new names for the variables used + # to match to the subfolder indices in the second table. .gyp files + # then will not need to prepend $(BUILT_PRODUCTS_DIR) because + # xcode_emulation.py can set the values of those variables to + # the absolute paths when exporting. This is possibly the thinking + # behind BUILT_FRAMEWORKS_DIR which is used in exactly this manner. + # + # Requiring prepending BUILT_PRODUCTS_DIR has been chosen because + # this same way could be used to specify destinations in .gyp files + # that pre-date this addition to GYP. However they would only work + # with the Xcode generator. The previous version of xcode_emulation.py + # does not export these variables. Such files will get the benefit + # of the Xcode UI showing the proper destination name simply by + # regenerating the projects with this version of GYP. + path_tree = path_tree_match.group(4) + relative_path = path_tree_match.group(6) + separator = "/" + + if path_tree in self.path_tree_second_to_subfolder: + subfolder = self.path_tree_second_to_subfolder[path_tree] + if relative_path is None: + relative_path = "" + separator = "" + if path_tree == "XPCSERVICES_FOLDER_PATH": + relative_path = ( + "$(CONTENTS_FOLDER_PATH)/XPCServices" + + separator + + relative_path + ) + else: + # subfolder = 16 from above + # The second element of the path is an unrecognized variable. + # Include it and any remaining elements in relative_path. + relative_path = path_tree_match.group(3) + + else: + # The path starts with an unrecognized Xcode variable + # name like $(SRCROOT). Xcode will still handle this + # as an "absolute path" that starts with the variable. + subfolder = 0 + relative_path = path + elif path.startswith("/"): + # Special case. Absolute paths are in dstSubfolderSpec 0. + subfolder = 0 + relative_path = path[1:] + else: + raise ValueError( + "Can't use path %s in a %s" % (path, self.__class__.__name__) + ) - self._properties['dstPath'] = relative_path - self._properties['dstSubfolderSpec'] = subfolder + self._properties["dstPath"] = relative_path + self._properties["dstSubfolderSpec"] = subfolder class PBXBuildRule(XCObject): - _schema = XCObject._schema.copy() - _schema.update({ - 'compilerSpec': [0, str, 0, 1], - 'filePatterns': [0, str, 0, 0], - 'fileType': [0, str, 0, 1], - 'isEditable': [0, int, 0, 1, 1], - 'outputFiles': [1, str, 0, 1, []], - 'script': [0, str, 0, 0], - }) - - def Name(self): - # Not very inspired, but it's what Xcode uses. - return self.__class__.__name__ - - def Hashables(self): - # super - hashables = XCObject.Hashables(self) + _schema = XCObject._schema.copy() + _schema.update( + { + "compilerSpec": [0, str, 0, 1], + "filePatterns": [0, str, 0, 0], + "fileType": [0, str, 0, 1], + "isEditable": [0, int, 0, 1, 1], + "outputFiles": [1, str, 0, 1, []], + "script": [0, str, 0, 0], + } + ) - # Use the hashables of the weak objects that this object refers to. - hashables.append(self._properties['fileType']) - if 'filePatterns' in self._properties: - hashables.append(self._properties['filePatterns']) - return hashables + def Name(self): + # Not very inspired, but it's what Xcode uses. + return self.__class__.__name__ + + def Hashables(self): + # super + hashables = XCObject.Hashables(self) + + # Use the hashables of the weak objects that this object refers to. + hashables.append(self._properties["fileType"]) + if "filePatterns" in self._properties: + hashables.append(self._properties["filePatterns"]) + return hashables class PBXContainerItemProxy(XCObject): - # When referencing an item in this project file, containerPortal is the - # PBXProject root object of this project file. When referencing an item in - # another project file, containerPortal is a PBXFileReference identifying - # the other project file. - # - # When serving as a proxy to an XCTarget (in this project file or another), - # proxyType is 1. When serving as a proxy to a PBXFileReference (in another - # project file), proxyType is 2. Type 2 is used for references to the - # producs of the other project file's targets. - # - # Xcode is weird about remoteGlobalIDString. Usually, it's printed without - # a comment, indicating that it's tracked internally simply as a string, but - # sometimes it's printed with a comment (usually when the object is initially - # created), indicating that it's tracked as a project file object at least - # sometimes. This module always tracks it as an object, but contains a hack - # to prevent it from printing the comment in the project file output. See - # _XCKVPrint. - _schema = XCObject._schema.copy() - _schema.update({ - 'containerPortal': [0, XCContainerPortal, 0, 1], - 'proxyType': [0, int, 0, 1], - 'remoteGlobalIDString': [0, XCRemoteObject, 0, 1], - 'remoteInfo': [0, str, 0, 1], - }) - - def __repr__(self): - props = self._properties - name = '%s.gyp:%s' % (props['containerPortal'].Name(), props['remoteInfo']) - return '<%s %r at 0x%x>' % (self.__class__.__name__, name, id(self)) - - def Name(self): - # Admittedly not the best name, but it's what Xcode uses. - return self.__class__.__name__ - - def Hashables(self): - # super - hashables = XCObject.Hashables(self) + # When referencing an item in this project file, containerPortal is the + # PBXProject root object of this project file. When referencing an item in + # another project file, containerPortal is a PBXFileReference identifying + # the other project file. + # + # When serving as a proxy to an XCTarget (in this project file or another), + # proxyType is 1. When serving as a proxy to a PBXFileReference (in another + # project file), proxyType is 2. Type 2 is used for references to the + # producs of the other project file's targets. + # + # Xcode is weird about remoteGlobalIDString. Usually, it's printed without + # a comment, indicating that it's tracked internally simply as a string, but + # sometimes it's printed with a comment (usually when the object is initially + # created), indicating that it's tracked as a project file object at least + # sometimes. This module always tracks it as an object, but contains a hack + # to prevent it from printing the comment in the project file output. See + # _XCKVPrint. + _schema = XCObject._schema.copy() + _schema.update( + { + "containerPortal": [0, XCContainerPortal, 0, 1], + "proxyType": [0, int, 0, 1], + "remoteGlobalIDString": [0, XCRemoteObject, 0, 1], + "remoteInfo": [0, str, 0, 1], + } + ) + + def __repr__(self): + props = self._properties + name = "%s.gyp:%s" % (props["containerPortal"].Name(), props["remoteInfo"]) + return "<%s %r at 0x%x>" % (self.__class__.__name__, name, id(self)) + + def Name(self): + # Admittedly not the best name, but it's what Xcode uses. + return self.__class__.__name__ + + def Hashables(self): + # super + hashables = XCObject.Hashables(self) - # Use the hashables of the weak objects that this object refers to. - hashables.extend(self._properties['containerPortal'].Hashables()) - hashables.extend(self._properties['remoteGlobalIDString'].Hashables()) - return hashables + # Use the hashables of the weak objects that this object refers to. + hashables.extend(self._properties["containerPortal"].Hashables()) + hashables.extend(self._properties["remoteGlobalIDString"].Hashables()) + return hashables class PBXTargetDependency(XCObject): - # The "target" property accepts an XCTarget object, and obviously not - # NoneType. But XCTarget is defined below, so it can't be put into the - # schema yet. The definition of PBXTargetDependency can't be moved below - # XCTarget because XCTarget's own schema references PBXTargetDependency. - # Python doesn't deal well with this circular relationship, and doesn't have - # a real way to do forward declarations. To work around, the type of - # the "target" property is reset below, after XCTarget is defined. - # - # At least one of "name" and "target" is required. - _schema = XCObject._schema.copy() - _schema.update({ - 'name': [0, str, 0, 0], - 'target': [0, None.__class__, 0, 0], - 'targetProxy': [0, PBXContainerItemProxy, 1, 1], - }) - - def __repr__(self): - name = self._properties.get('name') or self._properties['target'].Name() - return '<%s %r at 0x%x>' % (self.__class__.__name__, name, id(self)) - - def Name(self): - # Admittedly not the best name, but it's what Xcode uses. - return self.__class__.__name__ - - def Hashables(self): - # super - hashables = XCObject.Hashables(self) + # The "target" property accepts an XCTarget object, and obviously not + # NoneType. But XCTarget is defined below, so it can't be put into the + # schema yet. The definition of PBXTargetDependency can't be moved below + # XCTarget because XCTarget's own schema references PBXTargetDependency. + # Python doesn't deal well with this circular relationship, and doesn't have + # a real way to do forward declarations. To work around, the type of + # the "target" property is reset below, after XCTarget is defined. + # + # At least one of "name" and "target" is required. + _schema = XCObject._schema.copy() + _schema.update( + { + "name": [0, str, 0, 0], + "target": [0, None.__class__, 0, 0], + "targetProxy": [0, PBXContainerItemProxy, 1, 1], + } + ) + + def __repr__(self): + name = self._properties.get("name") or self._properties["target"].Name() + return "<%s %r at 0x%x>" % (self.__class__.__name__, name, id(self)) + + def Name(self): + # Admittedly not the best name, but it's what Xcode uses. + return self.__class__.__name__ - # Use the hashables of the weak objects that this object refers to. - hashables.extend(self._properties['targetProxy'].Hashables()) - return hashables + def Hashables(self): + # super + hashables = XCObject.Hashables(self) + + # Use the hashables of the weak objects that this object refers to. + hashables.extend(self._properties["targetProxy"].Hashables()) + return hashables class PBXReferenceProxy(XCFileLikeElement): - _schema = XCFileLikeElement._schema.copy() - _schema.update({ - 'fileType': [0, str, 0, 1], - 'path': [0, str, 0, 1], - 'remoteRef': [0, PBXContainerItemProxy, 1, 1], - }) + _schema = XCFileLikeElement._schema.copy() + _schema.update( + { + "fileType": [0, str, 0, 1], + "path": [0, str, 0, 1], + "remoteRef": [0, PBXContainerItemProxy, 1, 1], + } + ) class XCTarget(XCRemoteObject): - # An XCTarget is really just an XCObject, the XCRemoteObject thing is just - # to allow PBXProject to be used in the remoteGlobalIDString property of - # PBXContainerItemProxy. - # - # Setting a "name" property at instantiation may also affect "productName", - # which may in turn affect the "PRODUCT_NAME" build setting in children of - # "buildConfigurationList". See __init__ below. - _schema = XCRemoteObject._schema.copy() - _schema.update({ - 'buildConfigurationList': [0, XCConfigurationList, 1, 1, - XCConfigurationList()], - 'buildPhases': [1, XCBuildPhase, 1, 1, []], - 'dependencies': [1, PBXTargetDependency, 1, 1, []], - 'name': [0, str, 0, 1], - 'productName': [0, str, 0, 1], - }) - - def __init__(self, properties=None, id=None, parent=None, - force_outdir=None, force_prefix=None, force_extension=None): - # super - XCRemoteObject.__init__(self, properties, id, parent) - - # Set up additional defaults not expressed in the schema. If a "name" - # property was supplied, set "productName" if it is not present. Also set - # the "PRODUCT_NAME" build setting in each configuration, but only if - # the setting is not present in any build configuration. - if 'name' in self._properties: - if not 'productName' in self._properties: - self.SetProperty('productName', self._properties['name']) - - if 'productName' in self._properties: - if 'buildConfigurationList' in self._properties: - configs = self._properties['buildConfigurationList'] - if configs.HasBuildSetting('PRODUCT_NAME') == 0: - configs.SetBuildSetting('PRODUCT_NAME', - self._properties['productName']) - - def AddDependency(self, other): - pbxproject = self.PBXProjectAncestor() - other_pbxproject = other.PBXProjectAncestor() - if pbxproject == other_pbxproject: - # Add a dependency to another target in the same project file. - container = PBXContainerItemProxy({'containerPortal': pbxproject, - 'proxyType': 1, - 'remoteGlobalIDString': other, - 'remoteInfo': other.Name()}) - dependency = PBXTargetDependency({'target': other, - 'targetProxy': container}) - self.AppendProperty('dependencies', dependency) - else: - # Add a dependency to a target in a different project file. - other_project_ref = \ - pbxproject.AddOrGetProjectReference(other_pbxproject)[1] - container = PBXContainerItemProxy({ - 'containerPortal': other_project_ref, - 'proxyType': 1, - 'remoteGlobalIDString': other, - 'remoteInfo': other.Name(), - }) - dependency = PBXTargetDependency({'name': other.Name(), - 'targetProxy': container}) - self.AppendProperty('dependencies', dependency) + # An XCTarget is really just an XCObject, the XCRemoteObject thing is just + # to allow PBXProject to be used in the remoteGlobalIDString property of + # PBXContainerItemProxy. + # + # Setting a "name" property at instantiation may also affect "productName", + # which may in turn affect the "PRODUCT_NAME" build setting in children of + # "buildConfigurationList". See __init__ below. + _schema = XCRemoteObject._schema.copy() + _schema.update( + { + "buildConfigurationList": [ + 0, + XCConfigurationList, + 1, + 1, + XCConfigurationList(), + ], + "buildPhases": [1, XCBuildPhase, 1, 1, []], + "dependencies": [1, PBXTargetDependency, 1, 1, []], + "name": [0, str, 0, 1], + "productName": [0, str, 0, 1], + } + ) + + def __init__( + self, + properties=None, + id=None, + parent=None, + force_outdir=None, + force_prefix=None, + force_extension=None, + ): + # super + XCRemoteObject.__init__(self, properties, id, parent) + + # Set up additional defaults not expressed in the schema. If a "name" + # property was supplied, set "productName" if it is not present. Also set + # the "PRODUCT_NAME" build setting in each configuration, but only if + # the setting is not present in any build configuration. + if "name" in self._properties: + if "productName" not in self._properties: + self.SetProperty("productName", self._properties["name"]) + + if "productName" in self._properties: + if "buildConfigurationList" in self._properties: + configs = self._properties["buildConfigurationList"] + if configs.HasBuildSetting("PRODUCT_NAME") == 0: + configs.SetBuildSetting( + "PRODUCT_NAME", self._properties["productName"] + ) + + def AddDependency(self, other): + pbxproject = self.PBXProjectAncestor() + other_pbxproject = other.PBXProjectAncestor() + if pbxproject == other_pbxproject: + # Add a dependency to another target in the same project file. + container = PBXContainerItemProxy( + { + "containerPortal": pbxproject, + "proxyType": 1, + "remoteGlobalIDString": other, + "remoteInfo": other.Name(), + } + ) + dependency = PBXTargetDependency( + {"target": other, "targetProxy": container} + ) + self.AppendProperty("dependencies", dependency) + else: + # Add a dependency to a target in a different project file. + other_project_ref = pbxproject.AddOrGetProjectReference(other_pbxproject)[1] + container = PBXContainerItemProxy( + { + "containerPortal": other_project_ref, + "proxyType": 1, + "remoteGlobalIDString": other, + "remoteInfo": other.Name(), + } + ) + dependency = PBXTargetDependency( + {"name": other.Name(), "targetProxy": container} + ) + self.AppendProperty("dependencies", dependency) - # Proxy all of these through to the build configuration list. + # Proxy all of these through to the build configuration list. - def ConfigurationNamed(self, name): - return self._properties['buildConfigurationList'].ConfigurationNamed(name) + def ConfigurationNamed(self, name): + return self._properties["buildConfigurationList"].ConfigurationNamed(name) - def DefaultConfiguration(self): - return self._properties['buildConfigurationList'].DefaultConfiguration() + def DefaultConfiguration(self): + return self._properties["buildConfigurationList"].DefaultConfiguration() - def HasBuildSetting(self, key): - return self._properties['buildConfigurationList'].HasBuildSetting(key) + def HasBuildSetting(self, key): + return self._properties["buildConfigurationList"].HasBuildSetting(key) - def GetBuildSetting(self, key): - return self._properties['buildConfigurationList'].GetBuildSetting(key) + def GetBuildSetting(self, key): + return self._properties["buildConfigurationList"].GetBuildSetting(key) - def SetBuildSetting(self, key, value): - return self._properties['buildConfigurationList'].SetBuildSetting(key, \ - value) + def SetBuildSetting(self, key, value): + return self._properties["buildConfigurationList"].SetBuildSetting(key, value) - def AppendBuildSetting(self, key, value): - return self._properties['buildConfigurationList'].AppendBuildSetting(key, \ - value) + def AppendBuildSetting(self, key, value): + return self._properties["buildConfigurationList"].AppendBuildSetting(key, value) - def DelBuildSetting(self, key): - return self._properties['buildConfigurationList'].DelBuildSetting(key) + def DelBuildSetting(self, key): + return self._properties["buildConfigurationList"].DelBuildSetting(key) # Redefine the type of the "target" property. See PBXTargetDependency._schema # above. -PBXTargetDependency._schema['target'][1] = XCTarget +PBXTargetDependency._schema["target"][1] = XCTarget class PBXNativeTarget(XCTarget): - # buildPhases is overridden in the schema to be able to set defaults. - # - # NOTE: Contrary to most objects, it is advisable to set parent when - # constructing PBXNativeTarget. A parent of an XCTarget must be a PBXProject - # object. A parent reference is required for a PBXNativeTarget during - # construction to be able to set up the target defaults for productReference, - # because a PBXBuildFile object must be created for the target and it must - # be added to the PBXProject's mainGroup hierarchy. - _schema = XCTarget._schema.copy() - _schema.update({ - 'buildPhases': [1, XCBuildPhase, 1, 1, - [PBXSourcesBuildPhase(), PBXFrameworksBuildPhase()]], - 'buildRules': [1, PBXBuildRule, 1, 1, []], - 'productReference': [0, PBXFileReference, 0, 1], - 'productType': [0, str, 0, 1], - }) - - # Mapping from Xcode product-types to settings. The settings are: - # filetype : used for explicitFileType in the project file - # prefix : the prefix for the file name - # suffix : the suffix for the file name - _product_filetypes = { - 'com.apple.product-type.application': ['wrapper.application', - '', '.app'], - 'com.apple.product-type.application.watchapp': ['wrapper.application', - '', '.app'], - 'com.apple.product-type.watchkit-extension': ['wrapper.app-extension', - '', '.appex'], - 'com.apple.product-type.app-extension': ['wrapper.app-extension', - '', '.appex'], - 'com.apple.product-type.bundle': ['wrapper.cfbundle', - '', '.bundle'], - 'com.apple.product-type.framework': ['wrapper.framework', - '', '.framework'], - 'com.apple.product-type.library.dynamic': ['compiled.mach-o.dylib', - 'lib', '.dylib'], - 'com.apple.product-type.library.static': ['archive.ar', - 'lib', '.a'], - 'com.apple.product-type.tool': ['compiled.mach-o.executable', - '', ''], - 'com.apple.product-type.bundle.unit-test': ['wrapper.cfbundle', - '', '.xctest'], - 'com.apple.product-type.bundle.ui-testing': ['wrapper.cfbundle', - '', '.xctest'], - 'com.googlecode.gyp.xcode.bundle': ['compiled.mach-o.dylib', - '', '.so'], - 'com.apple.product-type.kernel-extension': ['wrapper.kext', - '', '.kext'], - } - - def __init__(self, properties=None, id=None, parent=None, - force_outdir=None, force_prefix=None, force_extension=None): - # super - XCTarget.__init__(self, properties, id, parent) - - if 'productName' in self._properties and \ - 'productType' in self._properties and \ - not 'productReference' in self._properties and \ - self._properties['productType'] in self._product_filetypes: - products_group = None - pbxproject = self.PBXProjectAncestor() - if pbxproject != None: - products_group = pbxproject.ProductsGroup() - - if products_group != None: - (filetype, prefix, suffix) = \ - self._product_filetypes[self._properties['productType']] - # Xcode does not have a distinct type for loadable modules that are - # pure BSD targets (not in a bundle wrapper). GYP allows such modules - # to be specified by setting a target type to loadable_module without - # having mac_bundle set. These are mapped to the pseudo-product type - # com.googlecode.gyp.xcode.bundle. - # - # By picking up this special type and converting it to a dynamic - # library (com.apple.product-type.library.dynamic) with fix-ups, - # single-file loadable modules can be produced. - # - # MACH_O_TYPE is changed to mh_bundle to produce the proper file type - # (as opposed to mh_dylib). In order for linking to succeed, - # DYLIB_CURRENT_VERSION and DYLIB_COMPATIBILITY_VERSION must be - # cleared. They are meaningless for type mh_bundle. - # - # Finally, the .so extension is forcibly applied over the default - # (.dylib), unless another forced extension is already selected. - # .dylib is plainly wrong, and .bundle is used by loadable_modules in - # bundle wrappers (com.apple.product-type.bundle). .so seems an odd - # choice because it's used as the extension on many other systems that - # don't distinguish between linkable shared libraries and non-linkable - # loadable modules, but there's precedent: Python loadable modules on - # Mac OS X use an .so extension. - if self._properties['productType'] == 'com.googlecode.gyp.xcode.bundle': - self._properties['productType'] = \ - 'com.apple.product-type.library.dynamic' - self.SetBuildSetting('MACH_O_TYPE', 'mh_bundle') - self.SetBuildSetting('DYLIB_CURRENT_VERSION', '') - self.SetBuildSetting('DYLIB_COMPATIBILITY_VERSION', '') - if force_extension is None: - force_extension = suffix[1:] - - if self._properties['productType'] == \ - 'com.apple.product-type-bundle.unit.test' or \ - self._properties['productType'] == \ - 'com.apple.product-type-bundle.ui-testing': - if force_extension is None: - force_extension = suffix[1:] - - if force_extension is not None: - # If it's a wrapper (bundle), set WRAPPER_EXTENSION. - # Extension override. - suffix = '.' + force_extension - if filetype.startswith('wrapper.'): - self.SetBuildSetting('WRAPPER_EXTENSION', force_extension) - else: - self.SetBuildSetting('EXECUTABLE_EXTENSION', force_extension) - - if filetype.startswith('compiled.mach-o.executable'): - product_name = self._properties['productName'] - product_name += suffix - suffix = '' - self.SetProperty('productName', product_name) - self.SetBuildSetting('PRODUCT_NAME', product_name) - - # Xcode handles most prefixes based on the target type, however there - # are exceptions. If a "BSD Dynamic Library" target is added in the - # Xcode UI, Xcode sets EXECUTABLE_PREFIX. This check duplicates that - # behavior. - if force_prefix is not None: - prefix = force_prefix - if filetype.startswith('wrapper.'): - self.SetBuildSetting('WRAPPER_PREFIX', prefix) - else: - self.SetBuildSetting('EXECUTABLE_PREFIX', prefix) - - if force_outdir is not None: - self.SetBuildSetting('TARGET_BUILD_DIR', force_outdir) - - # TODO(tvl): Remove the below hack. - # http://code.google.com/p/gyp/issues/detail?id=122 - - # Some targets include the prefix in the target_name. These targets - # really should just add a product_name setting that doesn't include - # the prefix. For example: - # target_name = 'libevent', product_name = 'event' - # This check cleans up for them. - product_name = self._properties['productName'] - prefix_len = len(prefix) - if prefix_len and (product_name[:prefix_len] == prefix): - product_name = product_name[prefix_len:] - self.SetProperty('productName', product_name) - self.SetBuildSetting('PRODUCT_NAME', product_name) - - ref_props = { - 'explicitFileType': filetype, - 'includeInIndex': 0, - 'path': prefix + product_name + suffix, - 'sourceTree': 'BUILT_PRODUCTS_DIR', + # buildPhases is overridden in the schema to be able to set defaults. + # + # NOTE: Contrary to most objects, it is advisable to set parent when + # constructing PBXNativeTarget. A parent of an XCTarget must be a PBXProject + # object. A parent reference is required for a PBXNativeTarget during + # construction to be able to set up the target defaults for productReference, + # because a PBXBuildFile object must be created for the target and it must + # be added to the PBXProject's mainGroup hierarchy. + _schema = XCTarget._schema.copy() + _schema.update( + { + "buildPhases": [ + 1, + XCBuildPhase, + 1, + 1, + [PBXSourcesBuildPhase(), PBXFrameworksBuildPhase()], + ], + "buildRules": [1, PBXBuildRule, 1, 1, []], + "productReference": [0, PBXFileReference, 0, 1], + "productType": [0, str, 0, 1], } - file_ref = PBXFileReference(ref_props) - products_group.AppendChild(file_ref) - self.SetProperty('productReference', file_ref) - - def GetBuildPhaseByType(self, type): - if not 'buildPhases' in self._properties: - return None - - the_phase = None - for phase in self._properties['buildPhases']: - if isinstance(phase, type): - # Some phases may be present in multiples in a well-formed project file, - # but phases like PBXSourcesBuildPhase may only be present singly, and - # this function is intended as an aid to GetBuildPhaseByType. Loop - # over the entire list of phases and assert if more than one of the - # desired type is found. - assert the_phase is None - the_phase = phase - - return the_phase - - def HeadersPhase(self): - headers_phase = self.GetBuildPhaseByType(PBXHeadersBuildPhase) - if headers_phase is None: - headers_phase = PBXHeadersBuildPhase() - - # The headers phase should come before the resources, sources, and - # frameworks phases, if any. - insert_at = len(self._properties['buildPhases']) - for index in range(0, len(self._properties['buildPhases'])): - phase = self._properties['buildPhases'][index] - if isinstance(phase, PBXResourcesBuildPhase) or \ - isinstance(phase, PBXSourcesBuildPhase) or \ - isinstance(phase, PBXFrameworksBuildPhase): - insert_at = index - break - - self._properties['buildPhases'].insert(insert_at, headers_phase) - headers_phase.parent = self - - return headers_phase - - def ResourcesPhase(self): - resources_phase = self.GetBuildPhaseByType(PBXResourcesBuildPhase) - if resources_phase is None: - resources_phase = PBXResourcesBuildPhase() - - # The resources phase should come before the sources and frameworks - # phases, if any. - insert_at = len(self._properties['buildPhases']) - for index in range(0, len(self._properties['buildPhases'])): - phase = self._properties['buildPhases'][index] - if isinstance(phase, PBXSourcesBuildPhase) or \ - isinstance(phase, PBXFrameworksBuildPhase): - insert_at = index - break - - self._properties['buildPhases'].insert(insert_at, resources_phase) - resources_phase.parent = self - - return resources_phase - - def SourcesPhase(self): - sources_phase = self.GetBuildPhaseByType(PBXSourcesBuildPhase) - if sources_phase is None: - sources_phase = PBXSourcesBuildPhase() - self.AppendProperty('buildPhases', sources_phase) - - return sources_phase - - def FrameworksPhase(self): - frameworks_phase = self.GetBuildPhaseByType(PBXFrameworksBuildPhase) - if frameworks_phase is None: - frameworks_phase = PBXFrameworksBuildPhase() - self.AppendProperty('buildPhases', frameworks_phase) - - return frameworks_phase - - def AddDependency(self, other): - # super - XCTarget.AddDependency(self, other) - - static_library_type = 'com.apple.product-type.library.static' - shared_library_type = 'com.apple.product-type.library.dynamic' - framework_type = 'com.apple.product-type.framework' - if isinstance(other, PBXNativeTarget) and \ - 'productType' in self._properties and \ - self._properties['productType'] != static_library_type and \ - 'productType' in other._properties and \ - (other._properties['productType'] == static_library_type or \ - ((other._properties['productType'] == shared_library_type or \ - other._properties['productType'] == framework_type) and \ - ((not other.HasBuildSetting('MACH_O_TYPE')) or - other.GetBuildSetting('MACH_O_TYPE') != 'mh_bundle'))): - - file_ref = other.GetProperty('productReference') - - pbxproject = self.PBXProjectAncestor() - other_pbxproject = other.PBXProjectAncestor() - if pbxproject != other_pbxproject: - other_project_product_group = \ - pbxproject.AddOrGetProjectReference(other_pbxproject)[0] - file_ref = other_project_product_group.GetChildByRemoteObject(file_ref) - - self.FrameworksPhase().AppendProperty('files', - PBXBuildFile({'fileRef': file_ref})) + ) + + # Mapping from Xcode product-types to settings. The settings are: + # filetype : used for explicitFileType in the project file + # prefix : the prefix for the file name + # suffix : the suffix for the file name + _product_filetypes = { + "com.apple.product-type.application": ["wrapper.application", "", ".app"], + "com.apple.product-type.application.watchapp": [ + "wrapper.application", + "", + ".app", + ], + "com.apple.product-type.watchkit-extension": [ + "wrapper.app-extension", + "", + ".appex", + ], + "com.apple.product-type.app-extension": ["wrapper.app-extension", "", ".appex"], + "com.apple.product-type.bundle": ["wrapper.cfbundle", "", ".bundle"], + "com.apple.product-type.framework": ["wrapper.framework", "", ".framework"], + "com.apple.product-type.library.dynamic": [ + "compiled.mach-o.dylib", + "lib", + ".dylib", + ], + "com.apple.product-type.library.static": ["archive.ar", "lib", ".a"], + "com.apple.product-type.tool": ["compiled.mach-o.executable", "", ""], + "com.apple.product-type.bundle.unit-test": ["wrapper.cfbundle", "", ".xctest"], + "com.apple.product-type.bundle.ui-testing": ["wrapper.cfbundle", "", ".xctest"], + "com.googlecode.gyp.xcode.bundle": ["compiled.mach-o.dylib", "", ".so"], + "com.apple.product-type.kernel-extension": ["wrapper.kext", "", ".kext"], + } + + def __init__( + self, + properties=None, + id=None, + parent=None, + force_outdir=None, + force_prefix=None, + force_extension=None, + ): + # super + XCTarget.__init__(self, properties, id, parent) + + if ( + "productName" in self._properties + and "productType" in self._properties + and "productReference" not in self._properties + and self._properties["productType"] in self._product_filetypes + ): + products_group = None + pbxproject = self.PBXProjectAncestor() + if pbxproject is not None: + products_group = pbxproject.ProductsGroup() + + if products_group is not None: + (filetype, prefix, suffix) = self._product_filetypes[ + self._properties["productType"] + ] + # Xcode does not have a distinct type for loadable modules that are + # pure BSD targets (not in a bundle wrapper). GYP allows such modules + # to be specified by setting a target type to loadable_module without + # having mac_bundle set. These are mapped to the pseudo-product type + # com.googlecode.gyp.xcode.bundle. + # + # By picking up this special type and converting it to a dynamic + # library (com.apple.product-type.library.dynamic) with fix-ups, + # single-file loadable modules can be produced. + # + # MACH_O_TYPE is changed to mh_bundle to produce the proper file type + # (as opposed to mh_dylib). In order for linking to succeed, + # DYLIB_CURRENT_VERSION and DYLIB_COMPATIBILITY_VERSION must be + # cleared. They are meaningless for type mh_bundle. + # + # Finally, the .so extension is forcibly applied over the default + # (.dylib), unless another forced extension is already selected. + # .dylib is plainly wrong, and .bundle is used by loadable_modules in + # bundle wrappers (com.apple.product-type.bundle). .so seems an odd + # choice because it's used as the extension on many other systems that + # don't distinguish between linkable shared libraries and non-linkable + # loadable modules, but there's precedent: Python loadable modules on + # Mac OS X use an .so extension. + if self._properties["productType"] == "com.googlecode.gyp.xcode.bundle": + self._properties[ + "productType" + ] = "com.apple.product-type.library.dynamic" + self.SetBuildSetting("MACH_O_TYPE", "mh_bundle") + self.SetBuildSetting("DYLIB_CURRENT_VERSION", "") + self.SetBuildSetting("DYLIB_COMPATIBILITY_VERSION", "") + if force_extension is None: + force_extension = suffix[1:] + + if ( + self._properties["productType"] + == "com.apple.product-type-bundle.unit.test" + or self._properties["productType"] + == "com.apple.product-type-bundle.ui-testing" + ): + if force_extension is None: + force_extension = suffix[1:] + + if force_extension is not None: + # If it's a wrapper (bundle), set WRAPPER_EXTENSION. + # Extension override. + suffix = "." + force_extension + if filetype.startswith("wrapper."): + self.SetBuildSetting("WRAPPER_EXTENSION", force_extension) + else: + self.SetBuildSetting("EXECUTABLE_EXTENSION", force_extension) + + if filetype.startswith("compiled.mach-o.executable"): + product_name = self._properties["productName"] + product_name += suffix + suffix = "" + self.SetProperty("productName", product_name) + self.SetBuildSetting("PRODUCT_NAME", product_name) + + # Xcode handles most prefixes based on the target type, however there + # are exceptions. If a "BSD Dynamic Library" target is added in the + # Xcode UI, Xcode sets EXECUTABLE_PREFIX. This check duplicates that + # behavior. + if force_prefix is not None: + prefix = force_prefix + if filetype.startswith("wrapper."): + self.SetBuildSetting("WRAPPER_PREFIX", prefix) + else: + self.SetBuildSetting("EXECUTABLE_PREFIX", prefix) + + if force_outdir is not None: + self.SetBuildSetting("TARGET_BUILD_DIR", force_outdir) + + # TODO(tvl): Remove the below hack. + # http://code.google.com/p/gyp/issues/detail?id=122 + + # Some targets include the prefix in the target_name. These targets + # really should just add a product_name setting that doesn't include + # the prefix. For example: + # target_name = 'libevent', product_name = 'event' + # This check cleans up for them. + product_name = self._properties["productName"] + prefix_len = len(prefix) + if prefix_len and (product_name[:prefix_len] == prefix): + product_name = product_name[prefix_len:] + self.SetProperty("productName", product_name) + self.SetBuildSetting("PRODUCT_NAME", product_name) + + ref_props = { + "explicitFileType": filetype, + "includeInIndex": 0, + "path": prefix + product_name + suffix, + "sourceTree": "BUILT_PRODUCTS_DIR", + } + file_ref = PBXFileReference(ref_props) + products_group.AppendChild(file_ref) + self.SetProperty("productReference", file_ref) + + def GetBuildPhaseByType(self, type): + if "buildPhases" not in self._properties: + return None + + the_phase = None + for phase in self._properties["buildPhases"]: + if isinstance(phase, type): + # Some phases may be present in multiples in a well-formed project file, + # but phases like PBXSourcesBuildPhase may only be present singly, and + # this function is intended as an aid to GetBuildPhaseByType. Loop + # over the entire list of phases and assert if more than one of the + # desired type is found. + assert the_phase is None + the_phase = phase + + return the_phase + + def HeadersPhase(self): + headers_phase = self.GetBuildPhaseByType(PBXHeadersBuildPhase) + if headers_phase is None: + headers_phase = PBXHeadersBuildPhase() + + # The headers phase should come before the resources, sources, and + # frameworks phases, if any. + insert_at = len(self._properties["buildPhases"]) + for index, phase in enumerate(self._properties["buildPhases"]): + if ( + isinstance(phase, PBXResourcesBuildPhase) + or isinstance(phase, PBXSourcesBuildPhase) + or isinstance(phase, PBXFrameworksBuildPhase) + ): + insert_at = index + break + + self._properties["buildPhases"].insert(insert_at, headers_phase) + headers_phase.parent = self + + return headers_phase + + def ResourcesPhase(self): + resources_phase = self.GetBuildPhaseByType(PBXResourcesBuildPhase) + if resources_phase is None: + resources_phase = PBXResourcesBuildPhase() + + # The resources phase should come before the sources and frameworks + # phases, if any. + insert_at = len(self._properties["buildPhases"]) + for index, phase in enumerate(self._properties["buildPhases"]): + if isinstance(phase, PBXSourcesBuildPhase) or isinstance( + phase, PBXFrameworksBuildPhase + ): + insert_at = index + break + + self._properties["buildPhases"].insert(insert_at, resources_phase) + resources_phase.parent = self + + return resources_phase + + def SourcesPhase(self): + sources_phase = self.GetBuildPhaseByType(PBXSourcesBuildPhase) + if sources_phase is None: + sources_phase = PBXSourcesBuildPhase() + self.AppendProperty("buildPhases", sources_phase) + + return sources_phase + + def FrameworksPhase(self): + frameworks_phase = self.GetBuildPhaseByType(PBXFrameworksBuildPhase) + if frameworks_phase is None: + frameworks_phase = PBXFrameworksBuildPhase() + self.AppendProperty("buildPhases", frameworks_phase) + + return frameworks_phase + + def AddDependency(self, other): + # super + XCTarget.AddDependency(self, other) + + static_library_type = "com.apple.product-type.library.static" + shared_library_type = "com.apple.product-type.library.dynamic" + framework_type = "com.apple.product-type.framework" + if ( + isinstance(other, PBXNativeTarget) + and "productType" in self._properties + and self._properties["productType"] != static_library_type + and "productType" in other._properties + and ( + other._properties["productType"] == static_library_type + or ( + ( + other._properties["productType"] == shared_library_type + or other._properties["productType"] == framework_type + ) + and ( + (not other.HasBuildSetting("MACH_O_TYPE")) + or other.GetBuildSetting("MACH_O_TYPE") != "mh_bundle" + ) + ) + ) + ): + + file_ref = other.GetProperty("productReference") + + pbxproject = self.PBXProjectAncestor() + other_pbxproject = other.PBXProjectAncestor() + if pbxproject != other_pbxproject: + other_project_product_group = pbxproject.AddOrGetProjectReference( + other_pbxproject + )[0] + file_ref = other_project_product_group.GetChildByRemoteObject(file_ref) + + self.FrameworksPhase().AppendProperty( + "files", PBXBuildFile({"fileRef": file_ref}) + ) class PBXAggregateTarget(XCTarget): - pass + pass class PBXProject(XCContainerPortal): - # A PBXProject is really just an XCObject, the XCContainerPortal thing is - # just to allow PBXProject to be used in the containerPortal property of - # PBXContainerItemProxy. - """ + # A PBXProject is really just an XCObject, the XCContainerPortal thing is + # just to allow PBXProject to be used in the containerPortal property of + # PBXContainerItemProxy. + """ Attributes: path: "sample.xcodeproj". TODO(mark) Document me! @@ -2560,91 +2743,98 @@ class PBXProject(XCContainerPortal): PBXProject. """ - _schema = XCContainerPortal._schema.copy() - _schema.update({ - 'attributes': [0, dict, 0, 0], - 'buildConfigurationList': [0, XCConfigurationList, 1, 1, - XCConfigurationList()], - 'compatibilityVersion': [0, str, 0, 1, 'Xcode 3.2'], - 'hasScannedForEncodings': [0, int, 0, 1, 1], - 'mainGroup': [0, PBXGroup, 1, 1, PBXGroup()], - 'projectDirPath': [0, str, 0, 1, ''], - 'projectReferences': [1, dict, 0, 0], - 'projectRoot': [0, str, 0, 1, ''], - 'targets': [1, XCTarget, 1, 1, []], - }) - - def __init__(self, properties=None, id=None, parent=None, path=None): - self.path = path - self._other_pbxprojects = {} - # super - return XCContainerPortal.__init__(self, properties, id, parent) + _schema = XCContainerPortal._schema.copy() + _schema.update( + { + "attributes": [0, dict, 0, 0], + "buildConfigurationList": [ + 0, + XCConfigurationList, + 1, + 1, + XCConfigurationList(), + ], + "compatibilityVersion": [0, str, 0, 1, "Xcode 3.2"], + "hasScannedForEncodings": [0, int, 0, 1, 1], + "mainGroup": [0, PBXGroup, 1, 1, PBXGroup()], + "projectDirPath": [0, str, 0, 1, ""], + "projectReferences": [1, dict, 0, 0], + "projectRoot": [0, str, 0, 1, ""], + "targets": [1, XCTarget, 1, 1, []], + } + ) - def Name(self): - name = self.path - if name[-10:] == '.xcodeproj': - name = name[:-10] - return posixpath.basename(name) + def __init__(self, properties=None, id=None, parent=None, path=None): + self.path = path + self._other_pbxprojects = {} + # super + return XCContainerPortal.__init__(self, properties, id, parent) - def Path(self): - return self.path + def Name(self): + name = self.path + if name[-10:] == ".xcodeproj": + name = name[:-10] + return posixpath.basename(name) - def Comment(self): - return 'Project object' + def Path(self): + return self.path - def Children(self): - # super - children = XCContainerPortal.Children(self) + def Comment(self): + return "Project object" + + def Children(self): + # super + children = XCContainerPortal.Children(self) - # Add children that the schema doesn't know about. Maybe there's a more - # elegant way around this, but this is the only case where we need to own - # objects in a dictionary (that is itself in a list), and three lines for - # a one-off isn't that big a deal. - if 'projectReferences' in self._properties: - for reference in self._properties['projectReferences']: - children.append(reference['ProductGroup']) + # Add children that the schema doesn't know about. Maybe there's a more + # elegant way around this, but this is the only case where we need to own + # objects in a dictionary (that is itself in a list), and three lines for + # a one-off isn't that big a deal. + if "projectReferences" in self._properties: + for reference in self._properties["projectReferences"]: + children.append(reference["ProductGroup"]) - return children + return children - def PBXProjectAncestor(self): - return self + def PBXProjectAncestor(self): + return self - def _GroupByName(self, name): - if not 'mainGroup' in self._properties: - self.SetProperty('mainGroup', PBXGroup()) + def _GroupByName(self, name): + if "mainGroup" not in self._properties: + self.SetProperty("mainGroup", PBXGroup()) - main_group = self._properties['mainGroup'] - group = main_group.GetChildByName(name) - if group is None: - group = PBXGroup({'name': name}) - main_group.AppendChild(group) + main_group = self._properties["mainGroup"] + group = main_group.GetChildByName(name) + if group is None: + group = PBXGroup({"name": name}) + main_group.AppendChild(group) - return group + return group - # SourceGroup and ProductsGroup are created by default in Xcode's own - # templates. - def SourceGroup(self): - return self._GroupByName('Source') + # SourceGroup and ProductsGroup are created by default in Xcode's own + # templates. + def SourceGroup(self): + return self._GroupByName("Source") - def ProductsGroup(self): - return self._GroupByName('Products') + def ProductsGroup(self): + return self._GroupByName("Products") - # IntermediatesGroup is used to collect source-like files that are generated - # by rules or script phases and are placed in intermediate directories such - # as DerivedSources. - def IntermediatesGroup(self): - return self._GroupByName('Intermediates') + # IntermediatesGroup is used to collect source-like files that are generated + # by rules or script phases and are placed in intermediate directories such + # as DerivedSources. + def IntermediatesGroup(self): + return self._GroupByName("Intermediates") - # FrameworksGroup and ProjectsGroup are top-level groups used to collect - # frameworks and projects. - def FrameworksGroup(self): - return self._GroupByName('Frameworks') + # FrameworksGroup and ProjectsGroup are top-level groups used to collect + # frameworks and projects. + def FrameworksGroup(self): + return self._GroupByName("Frameworks") - def ProjectsGroup(self): - return self._GroupByName('Projects') + def ProjectsGroup(self): + return self._GroupByName("Projects") - def RootGroupForPath(self, path): - """Returns a PBXGroup child of this object to which path should be added. + def RootGroupForPath(self, path): + """Returns a PBXGroup child of this object to which path should be added. This method is intended to choose between SourceGroup and IntermediatesGroup on the basis of whether path is present in a source @@ -2658,83 +2848,84 @@ def RootGroupForPath(self, path): organized hierarchically (True) or as a single flat list (False). """ - # TODO(mark): make this a class variable and bind to self on call? - # Also, this list is nowhere near exhaustive. - # INTERMEDIATE_DIR and SHARED_INTERMEDIATE_DIR are used by - # gyp.generator.xcode. There should probably be some way for that module - # to push the names in, rather than having to hard-code them here. - source_tree_groups = { - 'DERIVED_FILE_DIR': (self.IntermediatesGroup, True), - 'INTERMEDIATE_DIR': (self.IntermediatesGroup, True), - 'PROJECT_DERIVED_FILE_DIR': (self.IntermediatesGroup, True), - 'SHARED_INTERMEDIATE_DIR': (self.IntermediatesGroup, True), - } + # TODO(mark): make this a class variable and bind to self on call? + # Also, this list is nowhere near exhaustive. + # INTERMEDIATE_DIR and SHARED_INTERMEDIATE_DIR are used by + # gyp.generator.xcode. There should probably be some way for that module + # to push the names in, rather than having to hard-code them here. + source_tree_groups = { + "DERIVED_FILE_DIR": (self.IntermediatesGroup, True), + "INTERMEDIATE_DIR": (self.IntermediatesGroup, True), + "PROJECT_DERIVED_FILE_DIR": (self.IntermediatesGroup, True), + "SHARED_INTERMEDIATE_DIR": (self.IntermediatesGroup, True), + } - (source_tree, path) = SourceTreeAndPathFromPath(path) - if source_tree != None and source_tree in source_tree_groups: - (group_func, hierarchical) = source_tree_groups[source_tree] - group = group_func() - return (group, hierarchical) + (source_tree, path) = SourceTreeAndPathFromPath(path) + if source_tree is not None and source_tree in source_tree_groups: + (group_func, hierarchical) = source_tree_groups[source_tree] + group = group_func() + return (group, hierarchical) - # TODO(mark): make additional choices based on file extension. + # TODO(mark): make additional choices based on file extension. - return (self.SourceGroup(), True) + return (self.SourceGroup(), True) - def AddOrGetFileInRootGroup(self, path): - """Returns a PBXFileReference corresponding to path in the correct group + def AddOrGetFileInRootGroup(self, path): + """Returns a PBXFileReference corresponding to path in the correct group according to RootGroupForPath's heuristics. If an existing PBXFileReference for path exists, it will be returned. Otherwise, one will be created and returned. """ - (group, hierarchical) = self.RootGroupForPath(path) - return group.AddOrGetFileByPath(path, hierarchical) - - def RootGroupsTakeOverOnlyChildren(self, recurse=False): - """Calls TakeOverOnlyChild for all groups in the main group.""" - - for group in self._properties['mainGroup']._properties['children']: - if isinstance(group, PBXGroup): - group.TakeOverOnlyChild(recurse) - - def SortGroups(self): - # Sort the children of the mainGroup (like "Source" and "Products") - # according to their defined order. - self._properties['mainGroup']._properties['children'] = \ - sorted(self._properties['mainGroup']._properties['children'], - cmp=lambda x,y: x.CompareRootGroup(y)) - - # Sort everything else by putting group before files, and going - # alphabetically by name within sections of groups and files. SortGroup - # is recursive. - for group in self._properties['mainGroup']._properties['children']: - if not isinstance(group, PBXGroup): - continue - - if group.Name() == 'Products': - # The Products group is a special case. Instead of sorting - # alphabetically, sort things in the order of the targets that - # produce the products. To do this, just build up a new list of - # products based on the targets. - products = [] - for target in self._properties['targets']: - if not isinstance(target, PBXNativeTarget): - continue - product = target._properties['productReference'] - # Make sure that the product is already in the products group. - assert product in group._properties['children'] - products.append(product) - - # Make sure that this process doesn't miss anything that was already - # in the products group. - assert len(products) == len(group._properties['children']) - group._properties['children'] = products - else: - group.SortGroup() - - def AddOrGetProjectReference(self, other_pbxproject): - """Add a reference to another project file (via PBXProject object) to this + (group, hierarchical) = self.RootGroupForPath(path) + return group.AddOrGetFileByPath(path, hierarchical) + + def RootGroupsTakeOverOnlyChildren(self, recurse=False): + """Calls TakeOverOnlyChild for all groups in the main group.""" + + for group in self._properties["mainGroup"]._properties["children"]: + if isinstance(group, PBXGroup): + group.TakeOverOnlyChild(recurse) + + def SortGroups(self): + # Sort the children of the mainGroup (like "Source" and "Products") + # according to their defined order. + self._properties["mainGroup"]._properties["children"] = sorted( + self._properties["mainGroup"]._properties["children"], + cmp=lambda x, y: x.CompareRootGroup(y), + ) + + # Sort everything else by putting group before files, and going + # alphabetically by name within sections of groups and files. SortGroup + # is recursive. + for group in self._properties["mainGroup"]._properties["children"]: + if not isinstance(group, PBXGroup): + continue + + if group.Name() == "Products": + # The Products group is a special case. Instead of sorting + # alphabetically, sort things in the order of the targets that + # produce the products. To do this, just build up a new list of + # products based on the targets. + products = [] + for target in self._properties["targets"]: + if not isinstance(target, PBXNativeTarget): + continue + product = target._properties["productReference"] + # Make sure that the product is already in the products group. + assert product in group._properties["children"] + products.append(product) + + # Make sure that this process doesn't miss anything that was already + # in the products group. + assert len(products) == len(group._properties["children"]) + group._properties["children"] = products + else: + group.SortGroup() + + def AddOrGetProjectReference(self, other_pbxproject): + """Add a reference to another project file (via PBXProject object) to this one. Returns [ProductGroup, ProjectRef]. ProductGroup is a PBXGroup object in @@ -2747,243 +2938,259 @@ def AddOrGetProjectReference(self, other_pbxproject): still be updated if necessary. """ - if not 'projectReferences' in self._properties: - self._properties['projectReferences'] = [] - - product_group = None - project_ref = None - - if not other_pbxproject in self._other_pbxprojects: - # This project file isn't yet linked to the other one. Establish the - # link. - product_group = PBXGroup({'name': 'Products'}) - - # ProductGroup is strong. - product_group.parent = self - - # There's nothing unique about this PBXGroup, and if left alone, it will - # wind up with the same set of hashables as all other PBXGroup objects - # owned by the projectReferences list. Add the hashables of the - # remote PBXProject that it's related to. - product_group._hashables.extend(other_pbxproject.Hashables()) - - # The other project reports its path as relative to the same directory - # that this project's path is relative to. The other project's path - # is not necessarily already relative to this project. Figure out the - # pathname that this project needs to use to refer to the other one. - this_path = posixpath.dirname(self.Path()) - projectDirPath = self.GetProperty('projectDirPath') - if projectDirPath: - if posixpath.isabs(projectDirPath[0]): - this_path = projectDirPath + if "projectReferences" not in self._properties: + self._properties["projectReferences"] = [] + + product_group = None + project_ref = None + + if other_pbxproject not in self._other_pbxprojects: + # This project file isn't yet linked to the other one. Establish the + # link. + product_group = PBXGroup({"name": "Products"}) + + # ProductGroup is strong. + product_group.parent = self + + # There's nothing unique about this PBXGroup, and if left alone, it will + # wind up with the same set of hashables as all other PBXGroup objects + # owned by the projectReferences list. Add the hashables of the + # remote PBXProject that it's related to. + product_group._hashables.extend(other_pbxproject.Hashables()) + + # The other project reports its path as relative to the same directory + # that this project's path is relative to. The other project's path + # is not necessarily already relative to this project. Figure out the + # pathname that this project needs to use to refer to the other one. + this_path = posixpath.dirname(self.Path()) + projectDirPath = self.GetProperty("projectDirPath") + if projectDirPath: + if posixpath.isabs(projectDirPath[0]): + this_path = projectDirPath + else: + this_path = posixpath.join(this_path, projectDirPath) + other_path = gyp.common.RelativePath(other_pbxproject.Path(), this_path) + + # ProjectRef is weak (it's owned by the mainGroup hierarchy). + project_ref = PBXFileReference( + { + "lastKnownFileType": "wrapper.pb-project", + "path": other_path, + "sourceTree": "SOURCE_ROOT", + } + ) + self.ProjectsGroup().AppendChild(project_ref) + + ref_dict = {"ProductGroup": product_group, "ProjectRef": project_ref} + self._other_pbxprojects[other_pbxproject] = ref_dict + self.AppendProperty("projectReferences", ref_dict) + + # Xcode seems to sort this list case-insensitively + self._properties["projectReferences"] = sorted( + self._properties["projectReferences"], + cmp=lambda x, y: cmp( + x["ProjectRef"].Name().lower(), y["ProjectRef"].Name().lower() + ), + ) else: - this_path = posixpath.join(this_path, projectDirPath) - other_path = gyp.common.RelativePath(other_pbxproject.Path(), this_path) - - # ProjectRef is weak (it's owned by the mainGroup hierarchy). - project_ref = PBXFileReference({ - 'lastKnownFileType': 'wrapper.pb-project', - 'path': other_path, - 'sourceTree': 'SOURCE_ROOT', - }) - self.ProjectsGroup().AppendChild(project_ref) - - ref_dict = {'ProductGroup': product_group, 'ProjectRef': project_ref} - self._other_pbxprojects[other_pbxproject] = ref_dict - self.AppendProperty('projectReferences', ref_dict) - - # Xcode seems to sort this list case-insensitively - self._properties['projectReferences'] = \ - sorted(self._properties['projectReferences'], cmp=lambda x,y: - cmp(x['ProjectRef'].Name().lower(), - y['ProjectRef'].Name().lower())) - else: - # The link already exists. Pull out the relevnt data. - project_ref_dict = self._other_pbxprojects[other_pbxproject] - product_group = project_ref_dict['ProductGroup'] - project_ref = project_ref_dict['ProjectRef'] - - self._SetUpProductReferences(other_pbxproject, product_group, project_ref) - - inherit_unique_symroot = self._AllSymrootsUnique(other_pbxproject, False) - targets = other_pbxproject.GetProperty('targets') - if all(self._AllSymrootsUnique(t, inherit_unique_symroot) for t in targets): - dir_path = project_ref._properties['path'] - product_group._hashables.extend(dir_path) - - return [product_group, project_ref] - - def _AllSymrootsUnique(self, target, inherit_unique_symroot): - # Returns True if all configurations have a unique 'SYMROOT' attribute. - # The value of inherit_unique_symroot decides, if a configuration is assumed - # to inherit a unique 'SYMROOT' attribute from its parent, if it doesn't - # define an explicit value for 'SYMROOT'. - symroots = self._DefinedSymroots(target) - for s in self._DefinedSymroots(target): - if (s is not None and not self._IsUniqueSymrootForTarget(s) or - s is None and not inherit_unique_symroot): + # The link already exists. Pull out the relevnt data. + project_ref_dict = self._other_pbxprojects[other_pbxproject] + product_group = project_ref_dict["ProductGroup"] + project_ref = project_ref_dict["ProjectRef"] + + self._SetUpProductReferences(other_pbxproject, product_group, project_ref) + + inherit_unique_symroot = self._AllSymrootsUnique(other_pbxproject, False) + targets = other_pbxproject.GetProperty("targets") + if all(self._AllSymrootsUnique(t, inherit_unique_symroot) for t in targets): + dir_path = project_ref._properties["path"] + product_group._hashables.extend(dir_path) + + return [product_group, project_ref] + + def _AllSymrootsUnique(self, target, inherit_unique_symroot): + # Returns True if all configurations have a unique 'SYMROOT' attribute. + # The value of inherit_unique_symroot decides, if a configuration is assumed + # to inherit a unique 'SYMROOT' attribute from its parent, if it doesn't + # define an explicit value for 'SYMROOT'. + symroots = self._DefinedSymroots(target) + for s in self._DefinedSymroots(target): + if ( + s is not None + and not self._IsUniqueSymrootForTarget(s) + or s is None + and not inherit_unique_symroot + ): + return False + return True if symroots else inherit_unique_symroot + + def _DefinedSymroots(self, target): + # Returns all values for the 'SYMROOT' attribute defined in all + # configurations for this target. If any configuration doesn't define the + # 'SYMROOT' attribute, None is added to the returned set. If all + # configurations don't define the 'SYMROOT' attribute, an empty set is + # returned. + config_list = target.GetProperty("buildConfigurationList") + symroots = set() + for config in config_list.GetProperty("buildConfigurations"): + setting = config.GetProperty("buildSettings") + if "SYMROOT" in setting: + symroots.add(setting["SYMROOT"]) + else: + symroots.add(None) + if len(symroots) == 1 and None in symroots: + return set() + return symroots + + def _IsUniqueSymrootForTarget(self, symroot): + # This method returns True if all configurations in target contain a + # 'SYMROOT' attribute that is unique for the given target. A value is + # unique, if the Xcode macro '$SRCROOT' appears in it in any form. + uniquifier = ["$SRCROOT", "$(SRCROOT)"] + if any(x in symroot for x in uniquifier): + return True return False - return True if symroots else inherit_unique_symroot - - def _DefinedSymroots(self, target): - # Returns all values for the 'SYMROOT' attribute defined in all - # configurations for this target. If any configuration doesn't define the - # 'SYMROOT' attribute, None is added to the returned set. If all - # configurations don't define the 'SYMROOT' attribute, an empty set is - # returned. - config_list = target.GetProperty('buildConfigurationList') - symroots = set() - for config in config_list.GetProperty('buildConfigurations'): - setting = config.GetProperty('buildSettings') - if 'SYMROOT' in setting: - symroots.add(setting['SYMROOT']) - else: - symroots.add(None) - if len(symroots) == 1 and None in symroots: - return set() - return symroots - - def _IsUniqueSymrootForTarget(self, symroot): - # This method returns True if all configurations in target contain a - # 'SYMROOT' attribute that is unique for the given target. A value is - # unique, if the Xcode macro '$SRCROOT' appears in it in any form. - uniquifier = ['$SRCROOT', '$(SRCROOT)'] - if any(x in symroot for x in uniquifier): - return True - return False - - def _SetUpProductReferences(self, other_pbxproject, product_group, - project_ref): - # TODO(mark): This only adds references to products in other_pbxproject - # when they don't exist in this pbxproject. Perhaps it should also - # remove references from this pbxproject that are no longer present in - # other_pbxproject. Perhaps it should update various properties if they - # change. - for target in other_pbxproject._properties['targets']: - if not isinstance(target, PBXNativeTarget): - continue - - other_fileref = target._properties['productReference'] - if product_group.GetChildByRemoteObject(other_fileref) is None: - # Xcode sets remoteInfo to the name of the target and not the name - # of its product, despite this proxy being a reference to the product. - container_item = PBXContainerItemProxy({ - 'containerPortal': project_ref, - 'proxyType': 2, - 'remoteGlobalIDString': other_fileref, - 'remoteInfo': target.Name() - }) - # TODO(mark): Does sourceTree get copied straight over from the other - # project? Can the other project ever have lastKnownFileType here - # instead of explicitFileType? (Use it if so?) Can path ever be - # unset? (I don't think so.) Can other_fileref have name set, and - # does it impact the PBXReferenceProxy if so? These are the questions - # that perhaps will be answered one day. - reference_proxy = PBXReferenceProxy({ - 'fileType': other_fileref._properties['explicitFileType'], - 'path': other_fileref._properties['path'], - 'sourceTree': other_fileref._properties['sourceTree'], - 'remoteRef': container_item, - }) - - product_group.AppendChild(reference_proxy) - - def SortRemoteProductReferences(self): - # For each remote project file, sort the associated ProductGroup in the - # same order that the targets are sorted in the remote project file. This - # is the sort order used by Xcode. - - def CompareProducts(x, y, remote_products): - # x and y are PBXReferenceProxy objects. Go through their associated - # PBXContainerItem to get the remote PBXFileReference, which will be - # present in the remote_products list. - x_remote = x._properties['remoteRef']._properties['remoteGlobalIDString'] - y_remote = y._properties['remoteRef']._properties['remoteGlobalIDString'] - x_index = remote_products.index(x_remote) - y_index = remote_products.index(y_remote) - - # Use the order of each remote PBXFileReference in remote_products to - # determine the sort order. - return cmp(x_index, y_index) - - for other_pbxproject, ref_dict in self._other_pbxprojects.items(): - # Build up a list of products in the remote project file, ordered the - # same as the targets that produce them. - remote_products = [] - for target in other_pbxproject._properties['targets']: - if not isinstance(target, PBXNativeTarget): - continue - remote_products.append(target._properties['productReference']) - - # Sort the PBXReferenceProxy children according to the list of remote - # products. - product_group = ref_dict['ProductGroup'] - product_group._properties['children'] = sorted( - product_group._properties['children'], - cmp=lambda x, y, rp=remote_products: CompareProducts(x, y, rp)) + + def _SetUpProductReferences(self, other_pbxproject, product_group, project_ref): + # TODO(mark): This only adds references to products in other_pbxproject + # when they don't exist in this pbxproject. Perhaps it should also + # remove references from this pbxproject that are no longer present in + # other_pbxproject. Perhaps it should update various properties if they + # change. + for target in other_pbxproject._properties["targets"]: + if not isinstance(target, PBXNativeTarget): + continue + + other_fileref = target._properties["productReference"] + if product_group.GetChildByRemoteObject(other_fileref) is None: + # Xcode sets remoteInfo to the name of the target and not the name + # of its product, despite this proxy being a reference to the product. + container_item = PBXContainerItemProxy( + { + "containerPortal": project_ref, + "proxyType": 2, + "remoteGlobalIDString": other_fileref, + "remoteInfo": target.Name(), + } + ) + # TODO(mark): Does sourceTree get copied straight over from the other + # project? Can the other project ever have lastKnownFileType here + # instead of explicitFileType? (Use it if so?) Can path ever be + # unset? (I don't think so.) Can other_fileref have name set, and + # does it impact the PBXReferenceProxy if so? These are the questions + # that perhaps will be answered one day. + reference_proxy = PBXReferenceProxy( + { + "fileType": other_fileref._properties["explicitFileType"], + "path": other_fileref._properties["path"], + "sourceTree": other_fileref._properties["sourceTree"], + "remoteRef": container_item, + } + ) + + product_group.AppendChild(reference_proxy) + + def SortRemoteProductReferences(self): + # For each remote project file, sort the associated ProductGroup in the + # same order that the targets are sorted in the remote project file. This + # is the sort order used by Xcode. + + def CompareProducts(x, y, remote_products): + # x and y are PBXReferenceProxy objects. Go through their associated + # PBXContainerItem to get the remote PBXFileReference, which will be + # present in the remote_products list. + x_remote = x._properties["remoteRef"]._properties["remoteGlobalIDString"] + y_remote = y._properties["remoteRef"]._properties["remoteGlobalIDString"] + x_index = remote_products.index(x_remote) + y_index = remote_products.index(y_remote) + + # Use the order of each remote PBXFileReference in remote_products to + # determine the sort order. + return cmp(x_index, y_index) + + for other_pbxproject, ref_dict in self._other_pbxprojects.items(): + # Build up a list of products in the remote project file, ordered the + # same as the targets that produce them. + remote_products = [] + for target in other_pbxproject._properties["targets"]: + if not isinstance(target, PBXNativeTarget): + continue + remote_products.append(target._properties["productReference"]) + + # Sort the PBXReferenceProxy children according to the list of remote + # products. + product_group = ref_dict["ProductGroup"] + product_group._properties["children"] = sorted( + product_group._properties["children"], + cmp=lambda x, y, rp=remote_products: CompareProducts(x, y, rp), + ) class XCProjectFile(XCObject): - _schema = XCObject._schema.copy() - _schema.update({ - 'archiveVersion': [0, int, 0, 1, 1], - 'classes': [0, dict, 0, 1, {}], - 'objectVersion': [0, int, 0, 1, 46], - 'rootObject': [0, PBXProject, 1, 1], - }) - - def ComputeIDs(self, recursive=True, overwrite=True, hash=None): - # Although XCProjectFile is implemented here as an XCObject, it's not a - # proper object in the Xcode sense, and it certainly doesn't have its own - # ID. Pass through an attempt to update IDs to the real root object. - if recursive: - self._properties['rootObject'].ComputeIDs(recursive, overwrite, hash) - - def Print(self, file=sys.stdout): - self.VerifyHasRequiredProperties() - - # Add the special "objects" property, which will be caught and handled - # separately during printing. This structure allows a fairly standard - # loop do the normal printing. - self._properties['objects'] = {} - self._XCPrint(file, 0, '// !$*UTF8*$!\n') - if self._should_print_single_line: - self._XCPrint(file, 0, '{ ') - else: - self._XCPrint(file, 0, '{\n') - for property, value in sorted(self._properties.items(), - cmp=lambda x, y: cmp(x, y)): - if property == 'objects': - self._PrintObjects(file) - else: - self._XCKVPrint(file, 1, property, value) - self._XCPrint(file, 0, '}\n') - del self._properties['objects'] - - def _PrintObjects(self, file): - if self._should_print_single_line: - self._XCPrint(file, 0, 'objects = {') - else: - self._XCPrint(file, 1, 'objects = {\n') - - objects_by_class = {} - for object in self.Descendants(): - if object == self: - continue - class_name = object.__class__.__name__ - if not class_name in objects_by_class: - objects_by_class[class_name] = [] - objects_by_class[class_name].append(object) - - for class_name in sorted(objects_by_class): - self._XCPrint(file, 0, '\n') - self._XCPrint(file, 0, '/* Begin ' + class_name + ' section */\n') - for object in sorted(objects_by_class[class_name], - cmp=lambda x, y: cmp(x.id, y.id)): - object.Print(file) - self._XCPrint(file, 0, '/* End ' + class_name + ' section */\n') - - if self._should_print_single_line: - self._XCPrint(file, 0, '}; ') - else: - self._XCPrint(file, 1, '};\n') + _schema = XCObject._schema.copy() + _schema.update( + { + "archiveVersion": [0, int, 0, 1, 1], + "classes": [0, dict, 0, 1, {}], + "objectVersion": [0, int, 0, 1, 46], + "rootObject": [0, PBXProject, 1, 1], + } + ) + + def ComputeIDs(self, recursive=True, overwrite=True, hash=None): + # Although XCProjectFile is implemented here as an XCObject, it's not a + # proper object in the Xcode sense, and it certainly doesn't have its own + # ID. Pass through an attempt to update IDs to the real root object. + if recursive: + self._properties["rootObject"].ComputeIDs(recursive, overwrite, hash) + + def Print(self, file=sys.stdout): + self.VerifyHasRequiredProperties() + + # Add the special "objects" property, which will be caught and handled + # separately during printing. This structure allows a fairly standard + # loop do the normal printing. + self._properties["objects"] = {} + self._XCPrint(file, 0, "// !$*UTF8*$!\n") + if self._should_print_single_line: + self._XCPrint(file, 0, "{ ") + else: + self._XCPrint(file, 0, "{\n") + for property, value in sorted( + self._properties.items(), cmp=lambda x, y: cmp(x, y) + ): + if property == "objects": + self._PrintObjects(file) + else: + self._XCKVPrint(file, 1, property, value) + self._XCPrint(file, 0, "}\n") + del self._properties["objects"] + + def _PrintObjects(self, file): + if self._should_print_single_line: + self._XCPrint(file, 0, "objects = {") + else: + self._XCPrint(file, 1, "objects = {\n") + + objects_by_class = {} + for object in self.Descendants(): + if object == self: + continue + class_name = object.__class__.__name__ + if class_name not in objects_by_class: + objects_by_class[class_name] = [] + objects_by_class[class_name].append(object) + + for class_name in sorted(objects_by_class): + self._XCPrint(file, 0, "\n") + self._XCPrint(file, 0, "/* Begin " + class_name + " section */\n") + for object in sorted( + objects_by_class[class_name], cmp=lambda x, y: cmp(x.id, y.id) + ): + object.Print(file) + self._XCPrint(file, 0, "/* End " + class_name + " section */\n") + + if self._should_print_single_line: + self._XCPrint(file, 0, "}; ") + else: + self._XCPrint(file, 1, "};\n") diff --git a/gyp/pylib/gyp/xml_fix.py b/gyp/pylib/gyp/xml_fix.py index 5de848158d..0a945322b4 100644 --- a/gyp/pylib/gyp/xml_fix.py +++ b/gyp/pylib/gyp/xml_fix.py @@ -14,56 +14,52 @@ def _Replacement_write_data(writer, data, is_attrib=False): - """Writes datachars to writer.""" - data = data.replace("&", "&").replace("<", "<") - data = data.replace("\"", """).replace(">", ">") - if is_attrib: - data = data.replace( - "\r", " ").replace( - "\n", " ").replace( - "\t", " ") - writer.write(data) + """Writes datachars to writer.""" + data = data.replace("&", "&").replace("<", "<") + data = data.replace('"', """).replace(">", ">") + if is_attrib: + data = data.replace("\r", " ").replace("\n", " ").replace("\t", " ") + writer.write(data) def _Replacement_writexml(self, writer, indent="", addindent="", newl=""): - # indent = current indentation - # addindent = indentation to add to higher levels - # newl = newline string - writer.write(indent+"<" + self.tagName) + # indent = current indentation + # addindent = indentation to add to higher levels + # newl = newline string + writer.write(indent + "<" + self.tagName) - attrs = self._get_attributes() - a_names = attrs.keys() - a_names.sort() + attrs = self._get_attributes() + a_names = sorted(attrs.keys()) - for a_name in a_names: - writer.write(" %s=\"" % a_name) - _Replacement_write_data(writer, attrs[a_name].value, is_attrib=True) - writer.write("\"") - if self.childNodes: - writer.write(">%s" % newl) - for node in self.childNodes: - node.writexml(writer, indent + addindent, addindent, newl) - writer.write("%s%s" % (indent, self.tagName, newl)) - else: - writer.write("/>%s" % newl) + for a_name in a_names: + writer.write(' %s="' % a_name) + _Replacement_write_data(writer, attrs[a_name].value, is_attrib=True) + writer.write('"') + if self.childNodes: + writer.write(">%s" % newl) + for node in self.childNodes: + node.writexml(writer, indent + addindent, addindent, newl) + writer.write("%s%s" % (indent, self.tagName, newl)) + else: + writer.write("/>%s" % newl) class XmlFix(object): - """Object to manage temporary patching of xml.dom.minidom.""" + """Object to manage temporary patching of xml.dom.minidom.""" - def __init__(self): - # Preserve current xml.dom.minidom functions. - self.write_data = xml.dom.minidom._write_data - self.writexml = xml.dom.minidom.Element.writexml - # Inject replacement versions of a function and a method. - xml.dom.minidom._write_data = _Replacement_write_data - xml.dom.minidom.Element.writexml = _Replacement_writexml + def __init__(self): + # Preserve current xml.dom.minidom functions. + self.write_data = xml.dom.minidom._write_data + self.writexml = xml.dom.minidom.Element.writexml + # Inject replacement versions of a function and a method. + xml.dom.minidom._write_data = _Replacement_write_data + xml.dom.minidom.Element.writexml = _Replacement_writexml - def Cleanup(self): - if self.write_data: - xml.dom.minidom._write_data = self.write_data - xml.dom.minidom.Element.writexml = self.writexml - self.write_data = None + def Cleanup(self): + if self.write_data: + xml.dom.minidom._write_data = self.write_data + xml.dom.minidom.Element.writexml = self.writexml + self.write_data = None - def __del__(self): - self.Cleanup() + def __del__(self): + self.Cleanup() diff --git a/gyp/requirements_dev.txt b/gyp/requirements_dev.txt new file mode 100644 index 0000000000..28ecacab60 --- /dev/null +++ b/gyp/requirements_dev.txt @@ -0,0 +1,2 @@ +flake8 +pytest diff --git a/gyp/samples/samples b/gyp/samples/samples deleted file mode 100755 index 804b618998..0000000000 --- a/gyp/samples/samples +++ /dev/null @@ -1,81 +0,0 @@ -#!/usr/bin/python - -# Copyright (c) 2009 Google Inc. All rights reserved. -# Use of this source code is governed by a BSD-style license that can be -# found in the LICENSE file. - -import os.path -import shutil -import sys - - -gyps = [ - 'app/app.gyp', - 'base/base.gyp', - 'build/temp_gyp/googleurl.gyp', - 'build/all.gyp', - 'build/common.gypi', - 'build/external_code.gypi', - 'chrome/test/security_tests/security_tests.gyp', - 'chrome/third_party/hunspell/hunspell.gyp', - 'chrome/chrome.gyp', - 'media/media.gyp', - 'net/net.gyp', - 'printing/printing.gyp', - 'sdch/sdch.gyp', - 'skia/skia.gyp', - 'testing/gmock.gyp', - 'testing/gtest.gyp', - 'third_party/bzip2/bzip2.gyp', - 'third_party/icu38/icu38.gyp', - 'third_party/libevent/libevent.gyp', - 'third_party/libjpeg/libjpeg.gyp', - 'third_party/libpng/libpng.gyp', - 'third_party/libxml/libxml.gyp', - 'third_party/libxslt/libxslt.gyp', - 'third_party/lzma_sdk/lzma_sdk.gyp', - 'third_party/modp_b64/modp_b64.gyp', - 'third_party/npapi/npapi.gyp', - 'third_party/sqlite/sqlite.gyp', - 'third_party/zlib/zlib.gyp', - 'v8/tools/gyp/v8.gyp', - 'webkit/activex_shim/activex_shim.gyp', - 'webkit/activex_shim_dll/activex_shim_dll.gyp', - 'webkit/build/action_csspropertynames.py', - 'webkit/build/action_cssvaluekeywords.py', - 'webkit/build/action_jsconfig.py', - 'webkit/build/action_makenames.py', - 'webkit/build/action_maketokenizer.py', - 'webkit/build/action_useragentstylesheets.py', - 'webkit/build/rule_binding.py', - 'webkit/build/rule_bison.py', - 'webkit/build/rule_gperf.py', - 'webkit/tools/test_shell/test_shell.gyp', - 'webkit/webkit.gyp', -] - - -def Main(argv): - if len(argv) != 3 or argv[1] not in ['push', 'pull']: - print 'Usage: %s push/pull PATH_TO_CHROME' % argv[0] - return 1 - - path_to_chrome = argv[2] - - for g in gyps: - chrome_file = os.path.join(path_to_chrome, g) - local_file = os.path.join(os.path.dirname(argv[0]), os.path.split(g)[1]) - if argv[1] == 'push': - print 'Copying %s to %s' % (local_file, chrome_file) - shutil.copyfile(local_file, chrome_file) - elif argv[1] == 'pull': - print 'Copying %s to %s' % (chrome_file, local_file) - shutil.copyfile(chrome_file, local_file) - else: - assert False - - return 0 - - -if __name__ == '__main__': - sys.exit(Main(sys.argv)) diff --git a/gyp/samples/samples.bat b/gyp/samples/samples.bat deleted file mode 100644 index 778d9c90f0..0000000000 --- a/gyp/samples/samples.bat +++ /dev/null @@ -1,5 +0,0 @@ -@rem Copyright (c) 2009 Google Inc. All rights reserved. -@rem Use of this source code is governed by a BSD-style license that can be -@rem found in the LICENSE file. - -@python %~dp0/samples %* diff --git a/gyp/setup.py b/gyp/setup.py index 75a42558d8..bf2f03f7f9 100755 --- a/gyp/setup.py +++ b/gyp/setup.py @@ -4,16 +4,41 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. +from os import path + from setuptools import setup +here = path.abspath(path.dirname(__file__)) +# Get the long description from the README file +with open(path.join(here, "README.md")) as in_file: + long_description = in_file.read() + setup( - name='gyp', - version='0.1', - description='Generate Your Projects', - author='Chromium Authors', - author_email='chromium-dev@googlegroups.com', - url='http://code.google.com/p/gyp', - package_dir = {'': 'pylib'}, - packages=['gyp', 'gyp.generator'], - entry_points = {'console_scripts': ['gyp=gyp:script_main'] } + name="gyp-next", + version="0.2.0", + description="A fork of the GYP build system for use in the Node.js projects", + long_description=long_description, + long_description_content_type="text/markdown", + author="Node.js contributors", + author_email="ryzokuken@disroot.org", + url="https://github.com/nodejs/gyp-next", + package_dir={"": "pylib"}, + packages=["gyp", "gyp.generator"], + entry_points={"console_scripts": ["gyp=gyp:script_main"]}, + python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*", + classifiers=[ + 'Development Status :: 3 - Alpha', + 'Environment :: Console', + 'Intended Audience :: Developers', + 'License :: OSI Approved :: BSD License', + 'Natural Language :: English', + 'Programming Language :: Python', + 'Programming Language :: Python :: 2', + 'Programming Language :: Python :: 2.7', + 'Programming Language :: Python :: 3', + 'Programming Language :: Python :: 3.5', + 'Programming Language :: Python :: 3.6', + 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', + ], ) diff --git a/gyp/test_gyp.py b/gyp/test_gyp.py new file mode 100755 index 0000000000..5975f8d4c6 --- /dev/null +++ b/gyp/test_gyp.py @@ -0,0 +1,269 @@ +#!/usr/bin/env python +# Copyright (c) 2012 Google Inc. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +"""gyptest.py -- test runner for GYP tests.""" + +from __future__ import print_function + +import argparse +import os +import platform +import subprocess +import sys +import time + + +def is_test_name(f): + return f.startswith("gyptest") and f.endswith(".py") + + +def find_all_gyptest_files(directory): + result = [] + for root, dirs, files in os.walk(directory): + result.extend([os.path.join(root, f) for f in files if is_test_name(f)]) + result.sort() + return result + + +def main(argv=None): + if argv is None: + argv = sys.argv + + parser = argparse.ArgumentParser() + parser.add_argument("-a", "--all", action="store_true", help="run all tests") + parser.add_argument("-C", "--chdir", action="store", help="change to directory") + parser.add_argument( + "-f", + "--format", + action="store", + default="", + help="run tests with the specified formats", + ) + parser.add_argument( + "-G", + "--gyp_option", + action="append", + default=[], + help="Add -G options to the gyp command line", + ) + parser.add_argument( + "-l", "--list", action="store_true", help="list available tests and exit" + ) + parser.add_argument( + "-n", + "--no-exec", + action="store_true", + help="no execute, just print the command line", + ) + parser.add_argument( + "--path", action="append", default=[], help="additional $PATH directory" + ) + parser.add_argument( + "-q", + "--quiet", + action="store_true", + help="quiet, don't print anything unless there are failures", + ) + parser.add_argument( + "-v", + "--verbose", + action="store_true", + help="print configuration info and test results.", + ) + parser.add_argument("tests", nargs="*") + args = parser.parse_args(argv[1:]) + + if args.chdir: + os.chdir(args.chdir) + + if args.path: + extra_path = [os.path.abspath(p) for p in args.path] + extra_path = os.pathsep.join(extra_path) + os.environ["PATH"] = extra_path + os.pathsep + os.environ["PATH"] + + if not args.tests: + if not args.all: + sys.stderr.write("Specify -a to get all tests.\n") + return 1 + args.tests = ["test"] + + tests = [] + for arg in args.tests: + if os.path.isdir(arg): + tests.extend(find_all_gyptest_files(os.path.normpath(arg))) + else: + if not is_test_name(os.path.basename(arg)): + print(arg, "is not a valid gyp test name.", file=sys.stderr) + sys.exit(1) + tests.append(arg) + + if args.list: + for test in tests: + print(test) + sys.exit(0) + + os.environ["PYTHONPATH"] = os.path.abspath("test/lib") + + if args.verbose: + print_configuration_info() + + if args.gyp_option and not args.quiet: + print("Extra Gyp options: %s\n" % args.gyp_option) + + if args.format: + format_list = args.format.split(",") + else: + format_list = { + "aix5": ["make"], + "freebsd7": ["make"], + "freebsd8": ["make"], + "openbsd5": ["make"], + "cygwin": ["msvs"], + "win32": ["msvs", "ninja"], + "linux": ["make", "ninja"], + "linux2": ["make", "ninja"], + "linux3": ["make", "ninja"], + # TODO: Re-enable xcode-ninja. + # https://bugs.chromium.org/p/gyp/issues/detail?id=530 + # 'darwin': ['make', 'ninja', 'xcode', 'xcode-ninja'], + "darwin": ["make", "ninja", "xcode"], + }[sys.platform] + + gyp_options = [] + for option in args.gyp_option: + gyp_options += ["-G", option] + + runner = Runner(format_list, tests, gyp_options, args.verbose) + runner.run() + + if not args.quiet: + runner.print_results() + + if runner.failures: + return 1 + else: + return 0 + + +def print_configuration_info(): + print("Test configuration:") + if sys.platform == "darwin": + sys.path.append(os.path.abspath("test/lib")) + import TestMac + + print(" Mac %s %s" % (platform.mac_ver()[0], platform.mac_ver()[2])) + print(" Xcode %s" % TestMac.Xcode.Version()) + elif sys.platform == "win32": + sys.path.append(os.path.abspath("pylib")) + import gyp.MSVSVersion + + print(" Win %s %s\n" % platform.win32_ver()[0:2]) + print(" MSVS %s" % gyp.MSVSVersion.SelectVisualStudioVersion().Description()) + elif sys.platform in ("linux", "linux2"): + print(" Linux %s" % " ".join(platform.linux_distribution())) + print(" Python %s" % platform.python_version()) + print(" PYTHONPATH=%s" % os.environ["PYTHONPATH"]) + print() + + +class Runner(object): + def __init__(self, formats, tests, gyp_options, verbose): + self.formats = formats + self.tests = tests + self.verbose = verbose + self.gyp_options = gyp_options + self.failures = [] + self.num_tests = len(formats) * len(tests) + num_digits = len(str(self.num_tests)) + self.fmt_str = "[%%%dd/%%%dd] (%%s) %%s" % (num_digits, num_digits) + self.isatty = sys.stdout.isatty() and not self.verbose + self.env = os.environ.copy() + self.hpos = 0 + + def run(self): + run_start = time.time() + + i = 1 + for fmt in self.formats: + for test in self.tests: + self.run_test(test, fmt, i) + i += 1 + + if self.isatty: + self.erase_current_line() + + self.took = time.time() - run_start + + def run_test(self, test, fmt, i): + if self.isatty: + self.erase_current_line() + + msg = self.fmt_str % (i, self.num_tests, fmt, test) + self.print_(msg) + + start = time.time() + cmd = [sys.executable, test] + self.gyp_options + self.env["TESTGYP_FORMAT"] = fmt + proc = subprocess.Popen( + cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=self.env + ) + proc.wait() + took = time.time() - start + + stdout = proc.stdout.read().decode("utf8") + if proc.returncode == 2: + res = "skipped" + elif proc.returncode: + res = "failed" + self.failures.append("(%s) %s" % (test, fmt)) + else: + res = "passed" + res_msg = " %s %.3fs" % (res, took) + self.print_(res_msg) + + if ( + stdout + and not stdout.endswith("PASSED\n") + and not (stdout.endswith("NO RESULT\n")) + ): + print() + for l in stdout.splitlines(): + print(" %s" % l) + elif not self.isatty: + print() + + def print_(self, msg): + print(msg, end="") + index = msg.rfind("\n") + if index == -1: + self.hpos += len(msg) + else: + self.hpos = len(msg) - index + sys.stdout.flush() + + def erase_current_line(self): + print("\b" * self.hpos + " " * self.hpos + "\b" * self.hpos, end="") + sys.stdout.flush() + self.hpos = 0 + + def print_results(self): + num_failures = len(self.failures) + if num_failures: + print() + if num_failures == 1: + print("Failed the following test:") + else: + print("Failed the following %d tests:" % num_failures) + print("\t" + "\n\t".join(sorted(self.failures))) + print() + print( + "Ran %d tests in %.3fs, %d failed." + % (self.num_tests, self.took, num_failures) + ) + print() + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/gyp/tools/graphviz.py b/gyp/tools/graphviz.py index 538b059da4..1f3acf37fc 100755 --- a/gyp/tools/graphviz.py +++ b/gyp/tools/graphviz.py @@ -16,87 +16,88 @@ def ParseTarget(target): - target, _, suffix = target.partition('#') - filename, _, target = target.partition(':') - return filename, target, suffix + target, _, suffix = target.partition("#") + filename, _, target = target.partition(":") + return filename, target, suffix def LoadEdges(filename, targets): - """Load the edges map from the dump file, and filter it to only + """Load the edges map from the dump file, and filter it to only show targets in |targets| and their depedendents.""" - file = open('dump.json') - edges = json.load(file) - file.close() + file = open("dump.json") + edges = json.load(file) + file.close() - # Copy out only the edges we're interested in from the full edge list. - target_edges = {} - to_visit = targets[:] - while to_visit: - src = to_visit.pop() - if src in target_edges: - continue - target_edges[src] = edges[src] - to_visit.extend(edges[src]) + # Copy out only the edges we're interested in from the full edge list. + target_edges = {} + to_visit = targets[:] + while to_visit: + src = to_visit.pop() + if src in target_edges: + continue + target_edges[src] = edges[src] + to_visit.extend(edges[src]) - return target_edges + return target_edges def WriteGraph(edges): - """Print a graphviz graph to stdout. + """Print a graphviz graph to stdout. |edges| is a map of target to a list of other targets it depends on.""" - # Bucket targets by file. - files = collections.defaultdict(list) - for src, dst in edges.items(): - build_file, target_name, toolset = ParseTarget(src) - files[build_file].append(src) - - print('digraph D {') - print(' fontsize=8') # Used by subgraphs. - print(' node [fontsize=8]') - - # Output nodes by file. We must first write out each node within - # its file grouping before writing out any edges that may refer - # to those nodes. - for filename, targets in files.items(): - if len(targets) == 1: - # If there's only one node for this file, simplify - # the display by making it a box without an internal node. - target = targets[0] - build_file, target_name, toolset = ParseTarget(target) - print(' "%s" [shape=box, label="%s\\n%s"]' % (target, filename, - target_name)) - else: - # Group multiple nodes together in a subgraph. - print(' subgraph "cluster_%s" {' % filename) - print(' label = "%s"' % filename) - for target in targets: - build_file, target_name, toolset = ParseTarget(target) - print(' "%s" [label="%s"]' % (target, target_name)) - print(' }') - - # Now that we've placed all the nodes within subgraphs, output all - # the edges between nodes. - for src, dsts in edges.items(): - for dst in dsts: - print(' "%s" -> "%s"' % (src, dst)) - - print('}') + # Bucket targets by file. + files = collections.defaultdict(list) + for src, dst in edges.items(): + build_file, target_name, toolset = ParseTarget(src) + files[build_file].append(src) + + print("digraph D {") + print(" fontsize=8") # Used by subgraphs. + print(" node [fontsize=8]") + + # Output nodes by file. We must first write out each node within + # its file grouping before writing out any edges that may refer + # to those nodes. + for filename, targets in files.items(): + if len(targets) == 1: + # If there's only one node for this file, simplify + # the display by making it a box without an internal node. + target = targets[0] + build_file, target_name, toolset = ParseTarget(target) + print( + ' "%s" [shape=box, label="%s\\n%s"]' % (target, filename, target_name) + ) + else: + # Group multiple nodes together in a subgraph. + print(' subgraph "cluster_%s" {' % filename) + print(' label = "%s"' % filename) + for target in targets: + build_file, target_name, toolset = ParseTarget(target) + print(' "%s" [label="%s"]' % (target, target_name)) + print(" }") + + # Now that we've placed all the nodes within subgraphs, output all + # the edges between nodes. + for src, dsts in edges.items(): + for dst in dsts: + print(' "%s" -> "%s"' % (src, dst)) + + print("}") def main(): - if len(sys.argv) < 2: - print(__doc__, file=sys.stderr) - print(file=sys.stderr) - print('usage: %s target1 target2...' % (sys.argv[0]), file=sys.stderr) - return 1 + if len(sys.argv) < 2: + print(__doc__, file=sys.stderr) + print(file=sys.stderr) + print("usage: %s target1 target2..." % (sys.argv[0]), file=sys.stderr) + return 1 - edges = LoadEdges('dump.json', sys.argv[1:]) + edges = LoadEdges("dump.json", sys.argv[1:]) - WriteGraph(edges) - return 0 + WriteGraph(edges) + return 0 -if __name__ == '__main__': - sys.exit(main()) +if __name__ == "__main__": + sys.exit(main()) diff --git a/gyp/tools/pretty_gyp.py b/gyp/tools/pretty_gyp.py index 633048a59a..7313b4fe1b 100755 --- a/gyp/tools/pretty_gyp.py +++ b/gyp/tools/pretty_gyp.py @@ -13,7 +13,7 @@ # Regex to remove comments when we're counting braces. -COMMENT_RE = re.compile(r'\s*#.*') +COMMENT_RE = re.compile(r"\s*#.*") # Regex to remove quoted strings when we're counting braces. # It takes into account quoted quotes, and makes sure that the quotes match. @@ -24,45 +24,47 @@ def comment_replace(matchobj): - return matchobj.group(1) + matchobj.group(2) + '#' * len(matchobj.group(3)) + return matchobj.group(1) + matchobj.group(2) + "#" * len(matchobj.group(3)) def mask_comments(input): - """Mask the quoted strings so we skip braces inside quoted strings.""" - search_re = re.compile(r'(.*?)(#)(.*)') - return [search_re.sub(comment_replace, line) for line in input] + """Mask the quoted strings so we skip braces inside quoted strings.""" + search_re = re.compile(r"(.*?)(#)(.*)") + return [search_re.sub(comment_replace, line) for line in input] def quote_replace(matchobj): - return "%s%s%s%s" % (matchobj.group(1), - matchobj.group(2), - 'x'*len(matchobj.group(3)), - matchobj.group(2)) + return "%s%s%s%s" % ( + matchobj.group(1), + matchobj.group(2), + "x" * len(matchobj.group(3)), + matchobj.group(2), + ) def mask_quotes(input): - """Mask the quoted strings so we skip braces inside quoted strings.""" - search_re = re.compile(r'(.*?)' + QUOTE_RE_STR) - return [search_re.sub(quote_replace, line) for line in input] + """Mask the quoted strings so we skip braces inside quoted strings.""" + search_re = re.compile(r"(.*?)" + QUOTE_RE_STR) + return [search_re.sub(quote_replace, line) for line in input] def do_split(input, masked_input, search_re): - output = [] - mask_output = [] - for (line, masked_line) in zip(input, masked_input): - m = search_re.match(masked_line) - while m: - split = len(m.group(1)) - line = line[:split] + r'\n' + line[split:] - masked_line = masked_line[:split] + r'\n' + masked_line[split:] - m = search_re.match(masked_line) - output.extend(line.split(r'\n')) - mask_output.extend(masked_line.split(r'\n')) - return (output, mask_output) + output = [] + mask_output = [] + for (line, masked_line) in zip(input, masked_input): + m = search_re.match(masked_line) + while m: + split = len(m.group(1)) + line = line[:split] + r"\n" + line[split:] + masked_line = masked_line[:split] + r"\n" + masked_line[split:] + m = search_re.match(masked_line) + output.extend(line.split(r"\n")) + mask_output.extend(masked_line.split(r"\n")) + return (output, mask_output) def split_double_braces(input): - """Masks out the quotes and comments, and then splits appropriate + """Masks out the quotes and comments, and then splits appropriate lines (lines that matche the double_*_brace re's above) before indenting them below. @@ -70,88 +72,86 @@ def split_double_braces(input): that the indentation looks prettier when all laid out (e.g. closing braces make a nice diagonal line). """ - double_open_brace_re = re.compile(r'(.*?[\[\{\(,])(\s*)([\[\{\(])') - double_close_brace_re = re.compile(r'(.*?[\]\}\)],?)(\s*)([\]\}\)])') + double_open_brace_re = re.compile(r"(.*?[\[\{\(,])(\s*)([\[\{\(])") + double_close_brace_re = re.compile(r"(.*?[\]\}\)],?)(\s*)([\]\}\)])") - masked_input = mask_quotes(input) - masked_input = mask_comments(masked_input) + masked_input = mask_quotes(input) + masked_input = mask_comments(masked_input) - (output, mask_output) = do_split(input, masked_input, double_open_brace_re) - (output, mask_output) = do_split(output, mask_output, double_close_brace_re) + (output, mask_output) = do_split(input, masked_input, double_open_brace_re) + (output, mask_output) = do_split(output, mask_output, double_close_brace_re) - return output + return output def count_braces(line): - """keeps track of the number of braces on a given line and returns the result. + """keeps track of the number of braces on a given line and returns the result. It starts at zero and subtracts for closed braces, and adds for open braces. """ - open_braces = ['[', '(', '{'] - close_braces = [']', ')', '}'] - closing_prefix_re = re.compile(r'(.*?[^\s\]\}\)]+.*?)([\]\}\)],?)\s*$') - cnt = 0 - stripline = COMMENT_RE.sub(r'', line) - stripline = QUOTE_RE.sub(r"''", stripline) - for char in stripline: - for brace in open_braces: - if char == brace: - cnt += 1 - for brace in close_braces: - if char == brace: - cnt -= 1 - - after = False - if cnt > 0: - after = True - - # This catches the special case of a closing brace having something - # other than just whitespace ahead of it -- we don't want to - # unindent that until after this line is printed so it stays with - # the previous indentation level. - if cnt < 0 and closing_prefix_re.match(stripline): - after = True - return (cnt, after) + open_braces = ["[", "(", "{"] + close_braces = ["]", ")", "}"] + closing_prefix_re = re.compile(r"(.*?[^\s\]\}\)]+.*?)([\]\}\)],?)\s*$") + cnt = 0 + stripline = COMMENT_RE.sub(r"", line) + stripline = QUOTE_RE.sub(r"''", stripline) + for char in stripline: + for brace in open_braces: + if char == brace: + cnt += 1 + for brace in close_braces: + if char == brace: + cnt -= 1 + + after = False + if cnt > 0: + after = True + + # This catches the special case of a closing brace having something + # other than just whitespace ahead of it -- we don't want to + # unindent that until after this line is printed so it stays with + # the previous indentation level. + if cnt < 0 and closing_prefix_re.match(stripline): + after = True + return (cnt, after) def prettyprint_input(lines): - """Does the main work of indenting the input based on the brace counts.""" - indent = 0 - basic_offset = 2 - last_line = "" - for line in lines: - if COMMENT_RE.match(line): - print(line) - else: - line = line.strip('\r\n\t ') # Otherwise doesn't strip \r on Unix. - if len(line) > 0: - (brace_diff, after) = count_braces(line) - if brace_diff != 0: - if after: - print(" " * (basic_offset * indent) + line) - indent += brace_diff - else: - indent += brace_diff - print(" " * (basic_offset * indent) + line) + """Does the main work of indenting the input based on the brace counts.""" + indent = 0 + basic_offset = 2 + for line in lines: + if COMMENT_RE.match(line): + print(line) else: - print(" " * (basic_offset * indent) + line) - else: - print("") - last_line = line + line = line.strip("\r\n\t ") # Otherwise doesn't strip \r on Unix. + if len(line) > 0: + (brace_diff, after) = count_braces(line) + if brace_diff != 0: + if after: + print(" " * (basic_offset * indent) + line) + indent += brace_diff + else: + indent += brace_diff + print(" " * (basic_offset * indent) + line) + else: + print(" " * (basic_offset * indent) + line) + else: + print("") def main(): - if len(sys.argv) > 1: - data = open(sys.argv[1]).read().splitlines() - else: - data = sys.stdin.read().splitlines() - # Split up the double braces. - lines = split_double_braces(data) + if len(sys.argv) > 1: + data = open(sys.argv[1]).read().splitlines() + else: + data = sys.stdin.read().splitlines() + # Split up the double braces. + lines = split_double_braces(data) - # Indent and print the output. - prettyprint_input(lines) - return 0 + # Indent and print the output. + prettyprint_input(lines) + return 0 -if __name__ == '__main__': - sys.exit(main()) +if __name__ == "__main__": + sys.exit(main()) diff --git a/gyp/tools/pretty_sln.py b/gyp/tools/pretty_sln.py index 196566fb9e..2b1cb1de74 100755 --- a/gyp/tools/pretty_sln.py +++ b/gyp/tools/pretty_sln.py @@ -19,153 +19,164 @@ import sys import pretty_vcproj -__author__ = 'nsylvain (Nicolas Sylvain)' +__author__ = "nsylvain (Nicolas Sylvain)" + def BuildProject(project, built, projects, deps): - # if all dependencies are done, we can build it, otherwise we try to build the - # dependency. - # This is not infinite-recursion proof. - for dep in deps[project]: - if dep not in built: - BuildProject(dep, built, projects, deps) - print(project) - built.append(project) + # if all dependencies are done, we can build it, otherwise we try to build the + # dependency. + # This is not infinite-recursion proof. + for dep in deps[project]: + if dep not in built: + BuildProject(dep, built, projects, deps) + print(project) + built.append(project) + def ParseSolution(solution_file): - # All projects, their clsid and paths. - projects = dict() - - # A list of dependencies associated with a project. - dependencies = dict() - - # Regular expressions that matches the SLN format. - # The first line of a project definition. - begin_project = re.compile(r'^Project\("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942' - r'}"\) = "(.*)", "(.*)", "(.*)"$') - # The last line of a project definition. - end_project = re.compile('^EndProject$') - # The first line of a dependency list. - begin_dep = re.compile( - r'ProjectSection\(ProjectDependencies\) = postProject$') - # The last line of a dependency list. - end_dep = re.compile('EndProjectSection$') - # A line describing a dependency. - dep_line = re.compile(' *({.*}) = ({.*})$') - - in_deps = False - solution = open(solution_file) - for line in solution: - results = begin_project.search(line) - if results: - # Hack to remove icu because the diff is too different. - if results.group(1).find('icu') != -1: - continue - # We remove "_gyp" from the names because it helps to diff them. - current_project = results.group(1).replace('_gyp', '') - projects[current_project] = [results.group(2).replace('_gyp', ''), - results.group(3), - results.group(2)] - dependencies[current_project] = [] - continue - - results = end_project.search(line) - if results: - current_project = None - continue - - results = begin_dep.search(line) - if results: - in_deps = True - continue - - results = end_dep.search(line) - if results: - in_deps = False - continue - - results = dep_line.search(line) - if results and in_deps and current_project: - dependencies[current_project].append(results.group(1)) - continue - - # Change all dependencies clsid to name instead. - for project in dependencies: - # For each dependencies in this project - new_dep_array = [] - for dep in dependencies[project]: - # Look for the project name matching this cldis - for project_info in projects: - if projects[project_info][1] == dep: - new_dep_array.append(project_info) - dependencies[project] = sorted(new_dep_array) - - return (projects, dependencies) + # All projects, their clsid and paths. + projects = dict() + + # A list of dependencies associated with a project. + dependencies = dict() + + # Regular expressions that matches the SLN format. + # The first line of a project definition. + begin_project = re.compile( + r'^Project\("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942' + r'}"\) = "(.*)", "(.*)", "(.*)"$' + ) + # The last line of a project definition. + end_project = re.compile("^EndProject$") + # The first line of a dependency list. + begin_dep = re.compile(r"ProjectSection\(ProjectDependencies\) = postProject$") + # The last line of a dependency list. + end_dep = re.compile("EndProjectSection$") + # A line describing a dependency. + dep_line = re.compile(" *({.*}) = ({.*})$") + + in_deps = False + solution = open(solution_file) + for line in solution: + results = begin_project.search(line) + if results: + # Hack to remove icu because the diff is too different. + if results.group(1).find("icu") != -1: + continue + # We remove "_gyp" from the names because it helps to diff them. + current_project = results.group(1).replace("_gyp", "") + projects[current_project] = [ + results.group(2).replace("_gyp", ""), + results.group(3), + results.group(2), + ] + dependencies[current_project] = [] + continue + + results = end_project.search(line) + if results: + current_project = None + continue + + results = begin_dep.search(line) + if results: + in_deps = True + continue + + results = end_dep.search(line) + if results: + in_deps = False + continue + + results = dep_line.search(line) + if results and in_deps and current_project: + dependencies[current_project].append(results.group(1)) + continue + + # Change all dependencies clsid to name instead. + for project in dependencies: + # For each dependencies in this project + new_dep_array = [] + for dep in dependencies[project]: + # Look for the project name matching this cldis + for project_info in projects: + if projects[project_info][1] == dep: + new_dep_array.append(project_info) + dependencies[project] = sorted(new_dep_array) + + return (projects, dependencies) + def PrintDependencies(projects, deps): - print("---------------------------------------") - print("Dependencies for all projects") - print("---------------------------------------") - print("-- --") + print("---------------------------------------") + print("Dependencies for all projects") + print("---------------------------------------") + print("-- --") + + for (project, dep_list) in sorted(deps.items()): + print("Project : %s" % project) + print("Path : %s" % projects[project][0]) + if dep_list: + for dep in dep_list: + print(" - %s" % dep) + print("") - for (project, dep_list) in sorted(deps.items()): - print("Project : %s" % project) - print("Path : %s" % projects[project][0]) - if dep_list: - for dep in dep_list: - print(" - %s" % dep) - print("") + print("-- --") - print("-- --") def PrintBuildOrder(projects, deps): - print("---------------------------------------") - print("Build order ") - print("---------------------------------------") - print("-- --") + print("---------------------------------------") + print("Build order ") + print("---------------------------------------") + print("-- --") - built = [] - for (project, _) in sorted(deps.items()): - if project not in built: - BuildProject(project, built, projects, deps) + built = [] + for (project, _) in sorted(deps.items()): + if project not in built: + BuildProject(project, built, projects, deps) - print("-- --") + print("-- --") -def PrintVCProj(projects): - for project in projects: - print("-------------------------------------") - print("-------------------------------------") - print(project) - print(project) - print(project) - print("-------------------------------------") - print("-------------------------------------") +def PrintVCProj(projects): - project_path = os.path.abspath(os.path.join(os.path.dirname(sys.argv[1]), - projects[project][2])) + for project in projects: + print("-------------------------------------") + print("-------------------------------------") + print(project) + print(project) + print(project) + print("-------------------------------------") + print("-------------------------------------") + + project_path = os.path.abspath( + os.path.join(os.path.dirname(sys.argv[1]), projects[project][2]) + ) + + pretty = pretty_vcproj + argv = [ + "", + project_path, + "$(SolutionDir)=%s\\" % os.path.dirname(sys.argv[1]), + ] + argv.extend(sys.argv[3:]) + pretty.main(argv) - pretty = pretty_vcproj - argv = [ '', - project_path, - '$(SolutionDir)=%s\\' % os.path.dirname(sys.argv[1]), - ] - argv.extend(sys.argv[3:]) - pretty.main(argv) def main(): - # check if we have exactly 1 parameter. - if len(sys.argv) < 2: - print('Usage: %s "c:\\path\\to\\project.sln"' % sys.argv[0]) - return 1 + # check if we have exactly 1 parameter. + if len(sys.argv) < 2: + print('Usage: %s "c:\\path\\to\\project.sln"' % sys.argv[0]) + return 1 - (projects, deps) = ParseSolution(sys.argv[1]) - PrintDependencies(projects, deps) - PrintBuildOrder(projects, deps) + (projects, deps) = ParseSolution(sys.argv[1]) + PrintDependencies(projects, deps) + PrintBuildOrder(projects, deps) - if '--recursive' in sys.argv: - PrintVCProj(projects) - return 0 + if "--recursive" in sys.argv: + PrintVCProj(projects) + return 0 -if __name__ == '__main__': - sys.exit(main()) +if __name__ == "__main__": + sys.exit(main()) diff --git a/gyp/tools/pretty_vcproj.py b/gyp/tools/pretty_vcproj.py index 3212626004..38fbf3fecf 100755 --- a/gyp/tools/pretty_vcproj.py +++ b/gyp/tools/pretty_vcproj.py @@ -20,318 +20,326 @@ from xml.dom.minidom import parse from xml.dom.minidom import Node -__author__ = 'nsylvain (Nicolas Sylvain)' +__author__ = "nsylvain (Nicolas Sylvain)" try: - cmp + cmp except NameError: - def cmp(x, y): - return (x > y) - (x < y) + + def cmp(x, y): + return (x > y) - (x < y) + REPLACEMENTS = dict() ARGUMENTS = None class CmpTuple(object): - """Compare function between 2 tuple.""" - def __call__(self, x, y): - return cmp(x[0], y[0]) + """Compare function between 2 tuple.""" + + def __call__(self, x, y): + return cmp(x[0], y[0]) class CmpNode(object): - """Compare function between 2 xml nodes.""" + """Compare function between 2 xml nodes.""" - def __call__(self, x, y): - def get_string(node): - node_string = "node" - node_string += node.nodeName - if node.nodeValue: - node_string += node.nodeValue + def __call__(self, x, y): + def get_string(node): + node_string = "node" + node_string += node.nodeName + if node.nodeValue: + node_string += node.nodeValue - if node.attributes: - # We first sort by name, if present. - node_string += node.getAttribute("Name") + if node.attributes: + # We first sort by name, if present. + node_string += node.getAttribute("Name") - all_nodes = [] - for (name, value) in node.attributes.items(): - all_nodes.append((name, value)) + all_nodes = [] + for (name, value) in node.attributes.items(): + all_nodes.append((name, value)) - all_nodes.sort(CmpTuple()) - for (name, value) in all_nodes: - node_string += name - node_string += value + all_nodes.sort(CmpTuple()) + for (name, value) in all_nodes: + node_string += name + node_string += value - return node_string + return node_string - return cmp(get_string(x), get_string(y)) + return cmp(get_string(x), get_string(y)) def PrettyPrintNode(node, indent=0): - if node.nodeType == Node.TEXT_NODE: - if node.data.strip(): - print('%s%s' % (' '*indent, node.data.strip())) - return - - if node.childNodes: - node.normalize() - # Get the number of attributes - attr_count = 0 - if node.attributes: - attr_count = node.attributes.length - - # Print the main tag - if attr_count == 0: - print('%s<%s>' % (' '*indent, node.nodeName)) - else: - print('%s<%s' % (' '*indent, node.nodeName)) - - all_attributes = [] - for (name, value) in node.attributes.items(): - all_attributes.append((name, value)) - all_attributes.sort(CmpTuple()) - for (name, value) in all_attributes: - print('%s %s="%s"' % (' '*indent, name, value)) - print('%s>' % (' '*indent)) - if node.nodeValue: - print('%s %s' % (' '*indent, node.nodeValue)) - - for sub_node in node.childNodes: - PrettyPrintNode(sub_node, indent=indent+2) - print('%s' % (' '*indent, node.nodeName)) + if node.nodeType == Node.TEXT_NODE: + if node.data.strip(): + print("%s%s" % (" " * indent, node.data.strip())) + return + + if node.childNodes: + node.normalize() + # Get the number of attributes + attr_count = 0 + if node.attributes: + attr_count = node.attributes.length + + # Print the main tag + if attr_count == 0: + print("%s<%s>" % (" " * indent, node.nodeName)) + else: + print("%s<%s" % (" " * indent, node.nodeName)) + + all_attributes = [] + for (name, value) in node.attributes.items(): + all_attributes.append((name, value)) + all_attributes.sort(CmpTuple()) + for (name, value) in all_attributes: + print('%s %s="%s"' % (" " * indent, name, value)) + print("%s>" % (" " * indent)) + if node.nodeValue: + print("%s %s" % (" " * indent, node.nodeValue)) + + for sub_node in node.childNodes: + PrettyPrintNode(sub_node, indent=indent + 2) + print("%s" % (" " * indent, node.nodeName)) def FlattenFilter(node): - """Returns a list of all the node and sub nodes.""" - node_list = [] + """Returns a list of all the node and sub nodes.""" + node_list = [] - if (node.attributes and - node.getAttribute('Name') == '_excluded_files'): - # We don't add the "_excluded_files" filter. - return [] + if node.attributes and node.getAttribute("Name") == "_excluded_files": + # We don't add the "_excluded_files" filter. + return [] - for current in node.childNodes: - if current.nodeName == 'Filter': - node_list.extend(FlattenFilter(current)) - else: - node_list.append(current) + for current in node.childNodes: + if current.nodeName == "Filter": + node_list.extend(FlattenFilter(current)) + else: + node_list.append(current) - return node_list + return node_list def FixFilenames(filenames, current_directory): - new_list = [] - for filename in filenames: - if filename: - for key in REPLACEMENTS: - filename = filename.replace(key, REPLACEMENTS[key]) - os.chdir(current_directory) - filename = filename.strip('"\' ') - if filename.startswith('$'): - new_list.append(filename) - else: - new_list.append(os.path.abspath(filename)) - return new_list + new_list = [] + for filename in filenames: + if filename: + for key in REPLACEMENTS: + filename = filename.replace(key, REPLACEMENTS[key]) + os.chdir(current_directory) + filename = filename.strip("\"' ") + if filename.startswith("$"): + new_list.append(filename) + else: + new_list.append(os.path.abspath(filename)) + return new_list def AbsoluteNode(node): - """Makes all the properties we know about in this node absolute.""" - if node.attributes: - for (name, value) in node.attributes.items(): - if name in ['InheritedPropertySheets', 'RelativePath', - 'AdditionalIncludeDirectories', - 'IntermediateDirectory', 'OutputDirectory', - 'AdditionalLibraryDirectories']: - # We want to fix up these paths - path_list = value.split(';') - new_list = FixFilenames(path_list, os.path.dirname(ARGUMENTS[1])) - node.setAttribute(name, ';'.join(new_list)) - if not value: - node.removeAttribute(name) + """Makes all the properties we know about in this node absolute.""" + if node.attributes: + for (name, value) in node.attributes.items(): + if name in [ + "InheritedPropertySheets", + "RelativePath", + "AdditionalIncludeDirectories", + "IntermediateDirectory", + "OutputDirectory", + "AdditionalLibraryDirectories", + ]: + # We want to fix up these paths + path_list = value.split(";") + new_list = FixFilenames(path_list, os.path.dirname(ARGUMENTS[1])) + node.setAttribute(name, ";".join(new_list)) + if not value: + node.removeAttribute(name) def CleanupVcproj(node): - """For each sub node, we call recursively this function.""" - for sub_node in node.childNodes: - AbsoluteNode(sub_node) - CleanupVcproj(sub_node) - - # Normalize the node, and remove all extranous whitespaces. - for sub_node in node.childNodes: - if sub_node.nodeType == Node.TEXT_NODE: - sub_node.data = sub_node.data.replace("\r", "") - sub_node.data = sub_node.data.replace("\n", "") - sub_node.data = sub_node.data.rstrip() - - # Fix all the semicolon separated attributes to be sorted, and we also - # remove the dups. - if node.attributes: - for (name, value) in node.attributes.items(): - sorted_list = sorted(value.split(';')) - unique_list = [] - for i in sorted_list: - if not unique_list.count(i): - unique_list.append(i) - node.setAttribute(name, ';'.join(unique_list)) - if not value: - node.removeAttribute(name) - - if node.childNodes: - node.normalize() - - # For each node, take a copy, and remove it from the list. - node_array = [] - while node.childNodes and node.childNodes[0]: - # Take a copy of the node and remove it from the list. - current = node.childNodes[0] - node.removeChild(current) - - # If the child is a filter, we want to append all its children - # to this same list. - if current.nodeName == 'Filter': - node_array.extend(FlattenFilter(current)) - else: - node_array.append(current) - - - # Sort the list. - node_array.sort(CmpNode()) - - # Insert the nodes in the correct order. - for new_node in node_array: - # But don't append empty tool node. - if new_node.nodeName == 'Tool': - if new_node.attributes and new_node.attributes.length == 1: - # This one was empty. - continue - if new_node.nodeName == 'UserMacro': - continue - node.appendChild(new_node) + """For each sub node, we call recursively this function.""" + for sub_node in node.childNodes: + AbsoluteNode(sub_node) + CleanupVcproj(sub_node) + + # Normalize the node, and remove all extranous whitespaces. + for sub_node in node.childNodes: + if sub_node.nodeType == Node.TEXT_NODE: + sub_node.data = sub_node.data.replace("\r", "") + sub_node.data = sub_node.data.replace("\n", "") + sub_node.data = sub_node.data.rstrip() + + # Fix all the semicolon separated attributes to be sorted, and we also + # remove the dups. + if node.attributes: + for (name, value) in node.attributes.items(): + sorted_list = sorted(value.split(";")) + unique_list = [] + for i in sorted_list: + if not unique_list.count(i): + unique_list.append(i) + node.setAttribute(name, ";".join(unique_list)) + if not value: + node.removeAttribute(name) + + if node.childNodes: + node.normalize() + + # For each node, take a copy, and remove it from the list. + node_array = [] + while node.childNodes and node.childNodes[0]: + # Take a copy of the node and remove it from the list. + current = node.childNodes[0] + node.removeChild(current) + + # If the child is a filter, we want to append all its children + # to this same list. + if current.nodeName == "Filter": + node_array.extend(FlattenFilter(current)) + else: + node_array.append(current) + + # Sort the list. + node_array.sort(CmpNode()) + + # Insert the nodes in the correct order. + for new_node in node_array: + # But don't append empty tool node. + if new_node.nodeName == "Tool": + if new_node.attributes and new_node.attributes.length == 1: + # This one was empty. + continue + if new_node.nodeName == "UserMacro": + continue + node.appendChild(new_node) def GetConfiguationNodes(vcproj): - #TODO(nsylvain): Find a better way to navigate the xml. - nodes = [] - for node in vcproj.childNodes: - if node.nodeName == "Configurations": - for sub_node in node.childNodes: - if sub_node.nodeName == "Configuration": - nodes.append(sub_node) + # TODO(nsylvain): Find a better way to navigate the xml. + nodes = [] + for node in vcproj.childNodes: + if node.nodeName == "Configurations": + for sub_node in node.childNodes: + if sub_node.nodeName == "Configuration": + nodes.append(sub_node) - return nodes + return nodes def GetChildrenVsprops(filename): - dom = parse(filename) - if dom.documentElement.attributes: - vsprops = dom.documentElement.getAttribute('InheritedPropertySheets') - return FixFilenames(vsprops.split(';'), os.path.dirname(filename)) - return [] + dom = parse(filename) + if dom.documentElement.attributes: + vsprops = dom.documentElement.getAttribute("InheritedPropertySheets") + return FixFilenames(vsprops.split(";"), os.path.dirname(filename)) + return [] -def SeekToNode(node1, child2): - # A text node does not have properties. - if child2.nodeType == Node.TEXT_NODE: - return None - # Get the name of the current node. - current_name = child2.getAttribute("Name") - if not current_name: - # There is no name. We don't know how to merge. +def SeekToNode(node1, child2): + # A text node does not have properties. + if child2.nodeType == Node.TEXT_NODE: + return None + + # Get the name of the current node. + current_name = child2.getAttribute("Name") + if not current_name: + # There is no name. We don't know how to merge. + return None + + # Look through all the nodes to find a match. + for sub_node in node1.childNodes: + if sub_node.nodeName == child2.nodeName: + name = sub_node.getAttribute("Name") + if name == current_name: + return sub_node + + # No match. We give up. return None - # Look through all the nodes to find a match. - for sub_node in node1.childNodes: - if sub_node.nodeName == child2.nodeName: - name = sub_node.getAttribute("Name") - if name == current_name: - return sub_node - - # No match. We give up. - return None - def MergeAttributes(node1, node2): - # No attributes to merge? - if not node2.attributes: - return - - for (name, value2) in node2.attributes.items(): - # Don't merge the 'Name' attribute. - if name == 'Name': - continue - value1 = node1.getAttribute(name) - if value1: - # The attribute exist in the main node. If it's equal, we leave it - # untouched, otherwise we concatenate it. - if value1 != value2: - node1.setAttribute(name, ';'.join([value1, value2])) - else: - # The attribute does not exist in the main node. We append this one. - node1.setAttribute(name, value2) - - # If the attribute was a property sheet attributes, we remove it, since - # they are useless. - if name == 'InheritedPropertySheets': - node1.removeAttribute(name) + # No attributes to merge? + if not node2.attributes: + return + + for (name, value2) in node2.attributes.items(): + # Don't merge the 'Name' attribute. + if name == "Name": + continue + value1 = node1.getAttribute(name) + if value1: + # The attribute exist in the main node. If it's equal, we leave it + # untouched, otherwise we concatenate it. + if value1 != value2: + node1.setAttribute(name, ";".join([value1, value2])) + else: + # The attribute does not exist in the main node. We append this one. + node1.setAttribute(name, value2) + + # If the attribute was a property sheet attributes, we remove it, since + # they are useless. + if name == "InheritedPropertySheets": + node1.removeAttribute(name) def MergeProperties(node1, node2): - MergeAttributes(node1, node2) - for child2 in node2.childNodes: - child1 = SeekToNode(node1, child2) - if child1: - MergeProperties(child1, child2) - else: - node1.appendChild(child2.cloneNode(True)) + MergeAttributes(node1, node2) + for child2 in node2.childNodes: + child1 = SeekToNode(node1, child2) + if child1: + MergeProperties(child1, child2) + else: + node1.appendChild(child2.cloneNode(True)) def main(argv): - """Main function of this vcproj prettifier.""" - global ARGUMENTS - ARGUMENTS = argv - - # check if we have exactly 1 parameter. - if len(argv) < 2: - print('Usage: %s "c:\\path\\to\\vcproj.vcproj" [key1=value1] ' - '[key2=value2]' % argv[0]) - return 1 - - # Parse the keys - for i in range(2, len(argv)): - (key, value) = argv[i].split('=') - REPLACEMENTS[key] = value - - # Open the vcproj and parse the xml. - dom = parse(argv[1]) - - # First thing we need to do is find the Configuration Node and merge them - # with the vsprops they include. - for configuration_node in GetConfiguationNodes(dom.documentElement): - # Get the property sheets associated with this configuration. - vsprops = configuration_node.getAttribute('InheritedPropertySheets') - - # Fix the filenames to be absolute. - vsprops_list = FixFilenames(vsprops.strip().split(';'), - os.path.dirname(argv[1])) - - # Extend the list of vsprops with all vsprops contained in the current - # vsprops. - for current_vsprops in vsprops_list: - vsprops_list.extend(GetChildrenVsprops(current_vsprops)) - - # Now that we have all the vsprops, we need to merge them. - for current_vsprops in vsprops_list: - MergeProperties(configuration_node, - parse(current_vsprops).documentElement) - - # Now that everything is merged, we need to cleanup the xml. - CleanupVcproj(dom.documentElement) - - # Finally, we use the prett xml function to print the vcproj back to the - # user. - #print dom.toprettyxml(newl="\n") - PrettyPrintNode(dom.documentElement) - return 0 - - -if __name__ == '__main__': - sys.exit(main(sys.argv)) + """Main function of this vcproj prettifier.""" + global ARGUMENTS + ARGUMENTS = argv + + # check if we have exactly 1 parameter. + if len(argv) < 2: + print( + 'Usage: %s "c:\\path\\to\\vcproj.vcproj" [key1=value1] ' + "[key2=value2]" % argv[0] + ) + return 1 + + # Parse the keys + for i in range(2, len(argv)): + (key, value) = argv[i].split("=") + REPLACEMENTS[key] = value + + # Open the vcproj and parse the xml. + dom = parse(argv[1]) + + # First thing we need to do is find the Configuration Node and merge them + # with the vsprops they include. + for configuration_node in GetConfiguationNodes(dom.documentElement): + # Get the property sheets associated with this configuration. + vsprops = configuration_node.getAttribute("InheritedPropertySheets") + + # Fix the filenames to be absolute. + vsprops_list = FixFilenames( + vsprops.strip().split(";"), os.path.dirname(argv[1]) + ) + + # Extend the list of vsprops with all vsprops contained in the current + # vsprops. + for current_vsprops in vsprops_list: + vsprops_list.extend(GetChildrenVsprops(current_vsprops)) + + # Now that we have all the vsprops, we need to merge them. + for current_vsprops in vsprops_list: + MergeProperties(configuration_node, parse(current_vsprops).documentElement) + + # Now that everything is merged, we need to cleanup the xml. + CleanupVcproj(dom.documentElement) + + # Finally, we use the prett xml function to print the vcproj back to the + # user. + # print dom.toprettyxml(newl="\n") + PrettyPrintNode(dom.documentElement) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) From e529f3309d41111d5d6d5d08b4fbde610cd0efff Mon Sep 17 00:00:00 2001 From: Ujjwal Sharma Date: Tue, 14 Apr 2020 12:12:52 +0530 Subject: [PATCH 158/551] doc: update README to reflect upgrade to gyp-next PR-URL: https://github.com/nodejs/node-gyp/pull/2092 Reviewed-By: Rod Vagg --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index e3fab86ca6..fd7e6a117c 100644 --- a/README.md +++ b/README.md @@ -4,9 +4,9 @@ [![Build Status](https://github.com/nodejs/node-gyp/workflows/Python_tests/badge.svg)](https://github.com/nodejs/node-gyp/actions?workflow=Python_tests) `node-gyp` is a cross-platform command-line tool written in Node.js for -compiling native addon modules for Node.js. It contains a fork of the -[gyp](https://gyp.gsrc.io) project that was previously used by the Chromium -team, extended to support the development of Node.js native addons. +compiling native addon modules for Node.js. It contains a vendored copy of the +[gyp-next](https://github.com/nodejs/gyp-next) project that was previously used +by the Chromium team, extended to support the development of Node.js native addons. Note that `node-gyp` is _not_ used to build Node.js itself. From a6b76a8b488c3ca3ee39f4138644593075c25306 Mon Sep 17 00:00:00 2001 From: Ujjwal Sharma Date: Sun, 17 May 2020 23:49:58 +0530 Subject: [PATCH 159/551] gyp: update gyp to 0.2.1 PR-URL: https://github.com/nodejs/node-gyp/pull/2092 Reviewed-By: Rod Vagg --- gyp/.github/workflows/nodejs-windows.yml | 27 ++++++++++++++++++++++++ gyp/LICENSE | 2 +- gyp/pylib/gyp/common.py | 2 +- gyp/setup.py | 2 +- 4 files changed, 30 insertions(+), 3 deletions(-) create mode 100644 gyp/.github/workflows/nodejs-windows.yml diff --git a/gyp/.github/workflows/nodejs-windows.yml b/gyp/.github/workflows/nodejs-windows.yml new file mode 100644 index 0000000000..48a42372c2 --- /dev/null +++ b/gyp/.github/workflows/nodejs-windows.yml @@ -0,0 +1,27 @@ +name: Node.js Windows integration + +on: [push, pull_request] + +jobs: + build-windows: + runs-on: windows-latest + steps: + - name: Clone node-gyp + uses: actions/checkout@v2 + with: + path: gyp-next + - name: Clone nodejs/node + uses: actions/checkout@v2 + with: + repository: nodejs/node + path: node + - name: Install deps + run: choco install nasm + - name: Replace gyp in Node.js + run: | + rm -Recurse node/tools/gyp + cp -Recurse gyp-next node/tools/gyp + - name: Build Node.js + run: | + cd node + ./vcbuild.bat diff --git a/gyp/LICENSE b/gyp/LICENSE index 372b8a9bc0..c6944c5e4e 100644 --- a/gyp/LICENSE +++ b/gyp/LICENSE @@ -1,4 +1,4 @@ -Copyright (c) 2019 Ujjwal Sharma. All rights reserved. +Copyright (c) 2020 Node.js contributors. All rights reserved. Copyright (c) 2009 Google Inc. All rights reserved. Redistribution and use in source and binary forms, with or without diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py index 3f2329bda4..bfe546f867 100644 --- a/gyp/pylib/gyp/common.py +++ b/gyp/pylib/gyp/common.py @@ -364,7 +364,7 @@ def __init__(self): dir=base_temp_dir, ) try: - self.tmp_file = os.fdopen(tmp_fd, "w") + self.tmp_file = os.fdopen(tmp_fd, "wb") except Exception: # Don't leave turds behind. os.unlink(self.tmp_path) diff --git a/gyp/setup.py b/gyp/setup.py index bf2f03f7f9..0781c59184 100755 --- a/gyp/setup.py +++ b/gyp/setup.py @@ -15,7 +15,7 @@ setup( name="gyp-next", - version="0.2.0", + version="0.2.1", description="A fork of the GYP build system for use in the Node.js projects", long_description=long_description, long_description_content_type="text/markdown", From 4937722cf597ccd1953628f3d5e2ab5204280051 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Wed, 13 May 2020 11:42:33 +1000 Subject: [PATCH 160/551] deps: replace mkdirp with {recursive} mkdir only supported on Node.js 10+ Closes: #2084 PR-URL: https://github.com/nodejs/node-gyp/pull/2123 Reviewed-By: Richard Lau Reviewed-By: Jiawen Geng --- lib/configure.js | 3 +-- lib/install.js | 5 ++--- package.json | 1 - test/test-configure-python.js | 3 ++- 4 files changed, 5 insertions(+), 7 deletions(-) diff --git a/lib/configure.js b/lib/configure.js index 564564eea4..c164284e7d 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -4,7 +4,6 @@ const fs = require('graceful-fs') const path = require('path') const log = require('npmlog') const os = require('os') -const mkdirp = require('mkdirp') const processRelease = require('./process-release') const win = process.platform === 'win32' const findNodeDirectory = require('./find-node-directory') @@ -73,7 +72,7 @@ function configure (gyp, argv, callback) { function createBuildDir () { log.verbose('build dir', 'attempting to create "build" dir: %s', buildDir) - mkdirp(buildDir, function (err, isNew) { + fs.mkdir(buildDir, { recursive: true }, function (err, isNew) { if (err) { return callback(err) } diff --git a/lib/install.js b/lib/install.js index c919c10588..f9fa2b34bd 100644 --- a/lib/install.js +++ b/lib/install.js @@ -8,7 +8,6 @@ const crypto = require('crypto') const log = require('npmlog') const semver = require('semver') const request = require('request') -const mkdir = require('mkdirp') const processRelease = require('./process-release') const win = process.platform === 'win32' const getProxyFromURI = require('./proxy') @@ -114,7 +113,7 @@ function install (fs, gyp, argv, callback) { log.verbose('ensuring nodedir is created', devDir) // first create the dir for the node dev files - mkdir(devDir, function (err, created) { + fs.mkdir(devDir, { recursive: true }, function (err, created) { if (err) { if (err.code === 'EACCES') { eaccesFallback(err) @@ -310,7 +309,7 @@ function install (fs, gyp, argv, callback) { log.verbose(name, 'dir', dir) log.verbose(name, 'url', libUrl) - mkdir(dir, function (err) { + fs.mkdir(dir, { recursive: true }, function (err) { if (err) { return done(err) } diff --git a/package.json b/package.json index 707524e846..a71dd64370 100644 --- a/package.json +++ b/package.json @@ -25,7 +25,6 @@ "env-paths": "^2.2.0", "glob": "^7.1.4", "graceful-fs": "^4.2.3", - "mkdirp": "^0.5.1", "nopt": "^4.0.3", "npmlog": "^4.1.2", "request": "^2.88.2", diff --git a/test/test-configure-python.js b/test/test-configure-python.js index d08f9e5ed3..ac25f7972e 100644 --- a/test/test-configure-python.js +++ b/test/test-configure-python.js @@ -10,7 +10,8 @@ const configure = requireInject('../lib/configure', { openSync: function () { return 0 }, closeSync: function () { }, writeFile: function (file, data, cb) { cb() }, - stat: function (file, cb) { cb(null, {}) } + stat: function (file, cb) { cb(null, {}) }, + mkdir: function (dir, options, cb) { cb() } } }) From f7bfce96eddd77bdf56d25b9f113cc94ce93390a Mon Sep 17 00:00:00 2001 From: Dario Vladovic Date: Sat, 25 Apr 2020 01:29:45 +0200 Subject: [PATCH 161/551] doc: update acid test and introduce curl|bash test script PR-URL: https://github.com/nodejs/node-gyp/pull/2105 Reviewed-By: Rod Vagg --- macOS_Catalina.md | 13 ++++++------- macOS_Catalina_acid_test.sh | 21 +++++++++++++++++++++ 2 files changed, 27 insertions(+), 7 deletions(-) create mode 100644 macOS_Catalina_acid_test.sh diff --git a/macOS_Catalina.md b/macOS_Catalina.md index dbc8da4e7d..699fcfb7c8 100644 --- a/macOS_Catalina.md +++ b/macOS_Catalina.md @@ -21,14 +21,13 @@ If `ProductVersion` is less then `10.15` then this document is not for you. Norm ### The acid test To see if `Xcode Command Line Tools` is installed in a way that will work with `node-gyp`, run: -1. `/usr/sbin/pkgutil --packages | grep CL` - * `com.apple.pkg.CLTools_Executables` should be listed. If it isn't, this test failed. -2. `/usr/sbin/pkgutil --pkg-info com.apple.pkg.CLTools_Executables` - * `version: 11.0.0` (or later) should be listed. If it isn't, this test failed. - -If both tests succeeded, _you are done_! You should be ready to install `node-gyp`. +``` +curl -L https://github.com/nodejs/node-gyp/raw/master/macOS_Catalina_acid_test.sh | bash +``` + +If test succeeded, _you are done_! You should be ready to install `node-gyp`. -If either test failed, there is a problem with your Xcode Command Line Tools installation. [Continue to Solutions](#Solutions). +If test failed, there is a problem with your Xcode Command Line Tools installation. [Continue to Solutions](#Solutions). ### Solutions There are three ways to install the Xcode libraries `node-gyp` needs on macOS. People running Catalina have had success with some but not others in a way that has been unpredictable. diff --git a/macOS_Catalina_acid_test.sh b/macOS_Catalina_acid_test.sh new file mode 100644 index 0000000000..e1e98941a8 --- /dev/null +++ b/macOS_Catalina_acid_test.sh @@ -0,0 +1,21 @@ +#!/bin/bash + +pkgs=( + "com.apple.pkg.DeveloperToolsCLILeo" # standalone + "com.apple.pkg.DeveloperToolsCLI" # from XCode + "com.apple.pkg.CLTools_Executables" # Mavericks +) + +for pkg in "${pkgs[@]}"; do + output=$(/usr/sbin/pkgutil --pkg-info "$pkg" 2>/dev/null) + if [ "$output" ]; then + version=$(echo "$output" | grep 'version' | cut -d' ' -f2) + break + fi +done + +if [ "$version" ]; then + echo "Command Line Tools version: $version" +else + echo >&2 'Command Line Tools not found' +fi From ba4f34b7d6bb9e0b5f1d23f15b906af86a589ebd Mon Sep 17 00:00:00 2001 From: Dario Vladovic Date: Mon, 18 May 2020 02:39:58 +0200 Subject: [PATCH 162/551] doc: update catalina xcode clt download link PR-URL: https://github.com/nodejs/node-gyp/pull/2133 Reviewed-By: Rod Vagg Reviewed-By: Christian Clauss --- macOS_Catalina.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/macOS_Catalina.md b/macOS_Catalina.md index 699fcfb7c8..9ad2be5535 100644 --- a/macOS_Catalina.md +++ b/macOS_Catalina.md @@ -67,7 +67,7 @@ There are three ways to install the Xcode libraries `node-gyp` needs on macOS. P 10. Repeat step 5 above. Is the path different this time? Repeat the _acid test_. ### Installing `node-gyp` using the Xcode Command Line Tools via manual download -1. Download the appropriate version of the "Command Line Tools for Xcode" for your version of Catalina from . As of MacOS 10.15.3, that's [Command_Line_Tools_for_Xcode_11.3.1.dmg](https://download.developer.apple.com/Developer_Tools/Command_Line_Tools_for_Xcode_11.3.1/Command_Line_Tools_for_Xcode_11.3.1.dmg) +1. Download the appropriate version of the "Command Line Tools for Xcode" for your version of Catalina from . As of MacOS 10.15.5, that's [Command_Line_Tools_for_Xcode_11.5.dmg](https://download.developer.apple.com/Developer_Tools/Command_Line_Tools_for_Xcode_11.5/Command_Line_Tools_for_Xcode_11.5.dmg) 2. Install the package. 3. Run the [_acid test_ steps above](#The-acid-test). From 33affe2fbf96d05b2a16acd5d0ecdc2d97ac9376 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Wed, 13 May 2020 12:08:33 +1000 Subject: [PATCH 163/551] v7.0.0: bump version and update changelog PR-URL: https://github.com/nodejs/node-gyp/pull/2124 --- CHANGELOG.md | 28 ++++++++++++++++++++++++++++ package.json | 2 +- 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index eac853c3e3..bdea7ae4ba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,31 @@ +v7.0.0 2020-06-03 +================= + +* [[`e18a61afc1`](https://github.com/nodejs/node-gyp/commit/e18a61afc1)] - **build**: shrink bloated addon binaries on windows (Shelley Vohr) [#2060](https://github.com/nodejs/node-gyp/pull/2060) +* [[`4937722cf5`](https://github.com/nodejs/node-gyp/commit/4937722cf5)] - **(SEMVER-MAJOR)** **deps**: replace mkdirp with {recursive} mkdir (Rod Vagg) [#2123](https://github.com/nodejs/node-gyp/pull/2123) +* [[`d45438a047`](https://github.com/nodejs/node-gyp/commit/d45438a047)] - **(SEMVER-MAJOR)** **deps**: update deps, match to npm@7 (Rod Vagg) [#2126](https://github.com/nodejs/node-gyp/pull/2126) +* [[`ba4f34b7d6`](https://github.com/nodejs/node-gyp/commit/ba4f34b7d6)] - **doc**: update catalina xcode clt download link (Dario Vladovic) [#2133](https://github.com/nodejs/node-gyp/pull/2133) +* [[`f7bfce96ed`](https://github.com/nodejs/node-gyp/commit/f7bfce96ed)] - **doc**: update acid test and introduce curl|bash test script (Dario Vladovic) [#2105](https://github.com/nodejs/node-gyp/pull/2105) +* [[`e529f3309d`](https://github.com/nodejs/node-gyp/commit/e529f3309d)] - **doc**: update README to reflect upgrade to gyp-next (Ujjwal Sharma) [#2092](https://github.com/nodejs/node-gyp/pull/2092) +* [[`9aed6286a3`](https://github.com/nodejs/node-gyp/commit/9aed6286a3)] - **doc**: give more attention to Catalina issues doc (Matheus Marchini) [#2134](https://github.com/nodejs/node-gyp/pull/2134) +* [[`963f2a7b48`](https://github.com/nodejs/node-gyp/commit/963f2a7b48)] - **doc**: improve cataline discoverability for search engines (Matheus Marchini) [#2135](https://github.com/nodejs/node-gyp/pull/2135) +* [[`7b75af349b`](https://github.com/nodejs/node-gyp/commit/7b75af349b)] - **doc**: add macOS Catalina software update info (Karl Horky) [#2078](https://github.com/nodejs/node-gyp/pull/2078) +* [[`4f23c7bee2`](https://github.com/nodejs/node-gyp/commit/4f23c7bee2)] - **doc**: update link to the code of conduct (#2073) (Michaël Zasso) [#2073](https://github.com/nodejs/node-gyp/pull/2073) +* [[`473cfa283f`](https://github.com/nodejs/node-gyp/commit/473cfa283f)] - **doc**: note in README that Python 3.8 is supported (#2072) (Michaël Zasso) [#2072](https://github.com/nodejs/node-gyp/pull/2072) +* [[`e7402b4a7c`](https://github.com/nodejs/node-gyp/commit/e7402b4a7c)] - **doc**: update catalina xcode cli tools download link (#2044) (Dario Vladović) [#2044](https://github.com/nodejs/node-gyp/pull/2044) +* [[`35de45984f`](https://github.com/nodejs/node-gyp/commit/35de45984f)] - **doc**: update catalina xcode cli tools download link; formatting (Jonathan Hult) [#2034](https://github.com/nodejs/node-gyp/pull/2034) +* [[`48642191f5`](https://github.com/nodejs/node-gyp/commit/48642191f5)] - **doc**: add download link for Command Line Tools for Xcode (Przemysław Bitkowski) [#2029](https://github.com/nodejs/node-gyp/pull/2029) +* [[`ae5b150051`](https://github.com/nodejs/node-gyp/commit/ae5b150051)] - **doc**: Catalina suggestion: remove /Library/Developer/CommandLineTools (Christian Clauss) [#2022](https://github.com/nodejs/node-gyp/pull/2022) +* [[`d1dea13fe4`](https://github.com/nodejs/node-gyp/commit/d1dea13fe4)] - **doc**: fix changelog 6.1.0 release year to be 2020 (Quentin Vernot) [#2021](https://github.com/nodejs/node-gyp/pull/2021) +* [[`6356117b08`](https://github.com/nodejs/node-gyp/commit/6356117b08)] - **doc, bin**: stop suggesting opening node-gyp issues (Bartosz Sosnowski) [#2096](https://github.com/nodejs/node-gyp/pull/2096) +* [[`a6b76a8b48`](https://github.com/nodejs/node-gyp/commit/a6b76a8b48)] - **gyp**: update gyp to 0.2.1 (Ujjwal Sharma) [#2092](https://github.com/nodejs/node-gyp/pull/2092) +* [[`ebc34ec823`](https://github.com/nodejs/node-gyp/commit/ebc34ec823)] - **gyp**: update gyp to 0.2.0 (Ujjwal Sharma) [#2092](https://github.com/nodejs/node-gyp/pull/2092) +* [[`972780bde7`](https://github.com/nodejs/node-gyp/commit/972780bde7)] - **(SEMVER-MAJOR)** **gyp**: sync code base with nodejs repo (#1975) (Michaël Zasso) [#1975](https://github.com/nodejs/node-gyp/pull/1975) +* [[`c255ffbf6a`](https://github.com/nodejs/node-gyp/commit/c255ffbf6a)] - **lib**: drop "-2" flag for "py.exe" launcher (DeeDeeG) [#2131](https://github.com/nodejs/node-gyp/pull/2131) +* [[`1f7e1e93b5`](https://github.com/nodejs/node-gyp/commit/1f7e1e93b5)] - **lib**: ignore VS instances that cause COMExceptions (Andrew Casey) [#2018](https://github.com/nodejs/node-gyp/pull/2018) +* [[`741ab096d5`](https://github.com/nodejs/node-gyp/commit/741ab096d5)] - **test**: remove support for EOL versions of Node.js (Shelley Vohr) +* [[`ca86ef2539`](https://github.com/nodejs/node-gyp/commit/ca86ef2539)] - **test**: bump actions/checkout from v1 to v2 (BSKY) [#2063](https://github.com/nodejs/node-gyp/pull/2063) + v6.1.0 2020-01-08 ================= diff --git a/package.json b/package.json index a71dd64370..d78e8c3619 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "6.1.0", + "version": "7.0.0", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From 7857cb2eb19ebd709c69509040b23c3a0f39e95e Mon Sep 17 00:00:00 2001 From: DeeDeeG Date: Sun, 7 Jun 2020 15:10:43 -0400 Subject: [PATCH 164/551] deps: increase "engines" to "node" : ">= 10.12.0" Makes npm warn users if they are using an unsupported Node version. Refs: https://github.com/nodejs/node-gyp/pull/2123 PR-URL: https://github.com/nodejs/node-gyp/pull/2153 Reviewed-By: Jiawen Geng Reviewed-By: Rod Vagg --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index d78e8c3619..df236d345e 100644 --- a/package.json +++ b/package.json @@ -34,7 +34,7 @@ "which": "^2.0.2" }, "engines": { - "node": ">= 6.0.0" + "node": ">= 10.12.0" }, "devDependencies": { "bindings": "^1.5.0", From 4fc8ff179d9572984cfdd6c99726df66f8d7351d Mon Sep 17 00:00:00 2001 From: Chia Wei Ong Date: Fri, 5 Jun 2020 09:25:49 +0200 Subject: [PATCH 165/551] doc: silence curl for macOS Catalina acid test PR-URL: https://github.com/nodejs/node-gyp/pull/2150 Reviewed-By: Rod Vagg --- macOS_Catalina.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/macOS_Catalina.md b/macOS_Catalina.md index 9ad2be5535..338ee4a2cd 100644 --- a/macOS_Catalina.md +++ b/macOS_Catalina.md @@ -22,7 +22,7 @@ If `ProductVersion` is less then `10.15` then this document is not for you. Norm ### The acid test To see if `Xcode Command Line Tools` is installed in a way that will work with `node-gyp`, run: ``` -curl -L https://github.com/nodejs/node-gyp/raw/master/macOS_Catalina_acid_test.sh | bash +curl -sL https://github.com/nodejs/node-gyp/raw/master/macOS_Catalina_acid_test.sh | bash ``` If test succeeded, _you are done_! You should be ready to install `node-gyp`. From ee6fa7d3bc80d350fb8ed2651d6f56099e5edfdd Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Wed, 17 Jun 2020 11:52:16 +1000 Subject: [PATCH 166/551] docs: note that node-gyp@7 should solve Catalina CLT issues PR-URL: https://github.com/nodejs/node-gyp/pull/2156 Reviewed-By: Christian Clauss --- macOS_Catalina.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/macOS_Catalina.md b/macOS_Catalina.md index 338ee4a2cd..79bf6ff50b 100644 --- a/macOS_Catalina.md +++ b/macOS_Catalina.md @@ -8,6 +8,21 @@ _This document specifically refers to upgrades from previous versions of macOS t gyp: No Xcode or CLT version detected! ``` +## node-gyp v7 + +The newest release of `node-gyp` should solve this problem. If you are using `node-gyp` directly then you should be able to install v7 and use it as-is. + +If you need to use `node-gyp` from within `npm` (e.g. through `npm install`), you will have to install `node-gyp` (either globally with `-g` or to a predictable location) and tell `npm` where the new version is. Either use: + +* `npm config set node_gyp `; or +* run `npm` with an environment variable prefix: `npm_config_node_gyp= npm install` + +Where "path to node-gyp" is to the `node-gyp` executable which may be a symlink in your global bin directory (e.g. `/usr/local/bin/node-gyp`), or a path to the `node-gyp` installation directory and the `bin/node-gyp.js` file within it (e.g. `/usr/local/lib/node_modules/node-gyp/bin/node-gyp.js`). + +**If you use `npm config set` to change your global `node_gyp` you are responsible for keeping it up to date and can't rely on `npm` to give you a newer version when available.** Use `npm config delete node_gyp` to unset this configuration option. + +## Fixing Catalina for older versions of `node-gyp` + ### Is my Mac running macOS Catalina? Let's first make sure that your Mac is running Catalina: ``` @@ -30,7 +45,7 @@ If test succeeded, _you are done_! You should be ready to install `node-gyp`. If test failed, there is a problem with your Xcode Command Line Tools installation. [Continue to Solutions](#Solutions). ### Solutions -There are three ways to install the Xcode libraries `node-gyp` needs on macOS. People running Catalina have had success with some but not others in a way that has been unpredictable. +There are three ways to install the Xcode libraries `node-gyp` needs on macOS. People running Catalina have had success with some but not others in a way that has been unpredictable. 1. With the full Xcode (~7.6 GB download) from the `App Store` app. 2. With the _much_ smaller Xcode Command Line Tools via `xcode-select --install` From f461d56c5336d458eb9d8a70ff31774987bb14b6 Mon Sep 17 00:00:00 2001 From: Samuel Attard Date: Thu, 9 Jul 2020 14:31:23 -0700 Subject: [PATCH 167/551] build: support apple silicon (arm64 darwin) builds Reviewed-By: Rod Vagg Reviewed-By: Jiawen Geng PR-URL: https://github.com/nodejs/node-gyp/pull/2165 --- lib/configure.js | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/configure.js b/lib/configure.js index c164284e7d..d4342b9d76 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -132,6 +132,7 @@ function configure (gyp, argv, callback) { variables.target_arch = gyp.opts.arch || process.arch || 'ia32' if (variables.target_arch === 'arm64') { defaults.msvs_configuration_platform = 'ARM64' + defaults.xcode_configuration_platform = 'arm64' } // set the node development directory From 3baa4e4172c47052ee7e942cef542c1a7dea75aa Mon Sep 17 00:00:00 2001 From: Samuel Attard Date: Tue, 14 Jul 2020 19:41:49 -0700 Subject: [PATCH 168/551] gyp: update gyp to 0.4.0 Reviewed-By: Rod Vagg Reviewed-By: Jiawen Geng PR-URL: https://github.com/nodejs/node-gyp/pull/2165 --- gyp/.github/workflows/Python_tests.yml | 4 +- gyp/CHANGELOG.md | 41 +++++++++++ gyp/pylib/gyp/generator/android.py | 4 +- gyp/pylib/gyp/generator/make.py | 14 +++- gyp/pylib/gyp/generator/msvs.py | 96 ++++++++++++++++++-------- gyp/pylib/gyp/generator/ninja.py | 14 ++-- gyp/pylib/gyp/input.py | 18 ++--- gyp/pylib/gyp/xcode_emulation.py | 7 ++ gyp/setup.py | 28 ++++---- gyp/test_gyp.py | 3 +- 10 files changed, 162 insertions(+), 67 deletions(-) create mode 100644 gyp/CHANGELOG.md diff --git a/gyp/.github/workflows/Python_tests.yml b/gyp/.github/workflows/Python_tests.yml index 47c40343ad..a93b92f426 100644 --- a/gyp/.github/workflows/Python_tests.yml +++ b/gyp/.github/workflows/Python_tests.yml @@ -14,9 +14,9 @@ jobs: os: [macos-latest, ubuntu-latest] # , windows-latest] python-version: [2.7, 3.6, 3.7, 3.8] # 3.5, steps: - - uses: actions/checkout@v1 + - uses: actions/checkout@v2 - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v1 + uses: actions/setup-python@v2 with: python-version: ${{ matrix.python-version }} - name: Install dependencies diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md new file mode 100644 index 0000000000..8cbcdd3b72 --- /dev/null +++ b/gyp/CHANGELOG.md @@ -0,0 +1,41 @@ +# Changelog + +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). + +## [Unreleased] + +## [0.4.0] - 2020-07-14 + +### Added +- Added support for passing arbitrary architectures to Xcode builds, enables `arm64` builds. + +### Fixed +- Fixed a bug on Solaris where copying archives failed. + +## [0.3.0] - 2020-06-06 + +### Added +- Added support for MSVC cross-compilation. This allows compilation on x64 for + a Windows ARM target. + +### Fixed +- Fixed XCode CLT version detection on macOS Catalina. + +## [0.2.1] - 2020-05-05 + +### Fixed +- Relicensed to Node.js contributors. +- Fixed Windows bug introduced in v0.2.0. + +## [0.2.0] - 2020-04-06 + +This is the first release of this project, based on https://chromium.googlesource.com/external/gyp +with changes made over the years in Node.js and node-gyp. + +[Unreleased]: https://github.com/nodejs/gyp-next/compare/v0.4.0...HEAD +[0.4.0]: https://github.com/nodejs/gyp-next/compare/v0.3.0...v0.4.0 +[0.3.0]: https://github.com/nodejs/gyp-next/compare/v0.2.1...v0.3.0 +[0.2.1]: https://github.com/nodejs/gyp-next/compare/v0.2.0...v0.2.1 +[0.2.0]: https://github.com/nodejs/gyp-next/releases/tag/v0.2.0 diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py index 7dbbd579ca..3ac61008b9 100644 --- a/gyp/pylib/gyp/generator/android.py +++ b/gyp/pylib/gyp/generator/android.py @@ -981,9 +981,9 @@ def WriteList( """ values = "" if value_list: - value_list = [quoter(prefix + l) for l in value_list] + value_list = [quoter(prefix + value) for value in value_list] if local_pathify: - value_list = [self.LocalPathify(l) for l in value_list] + value_list = [self.LocalPathify(value) for value in value_list] values = " \\\n\t" + " \\\n\t".join(value_list) self.fp.write("%s :=%s\n\n" % (variable, values)) diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index 4a50b04339..6e1c5205cf 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -1342,7 +1342,7 @@ def WriteSources( ) if self.flavor == "mac": - cflags = self.xcode_settings.GetCflags(configname) + cflags = self.xcode_settings.GetCflags(configname, arch=config.get('xcode_configuration_platform')) cflags_c = self.xcode_settings.GetCflagsC(configname) cflags_cc = self.xcode_settings.GetCflagsCC(configname) cflags_objc = self.xcode_settings.GetCflagsObjC(configname) @@ -1637,6 +1637,7 @@ def WriteTarget( configname, generator_default_variables["PRODUCT_DIR"], lambda p: Sourceify(self.Absolutify(p)), + arch=config.get('xcode_configuration_platform') ) # TARGET_POSTBUILDS_$(BUILDTYPE) is added to postbuilds later on. @@ -1944,7 +1945,7 @@ def WriteList(self, value_list, variable=None, prefix="", quoter=QuoteIfNecessar """ values = "" if value_list: - value_list = [quoter(prefix + l) for l in value_list] + value_list = [quoter(prefix + value) for value in value_list] values = " \\\n\t" + " \\\n\t".join(value_list) self.fp.write("%s :=%s\n\n" % (variable, values)) @@ -2362,7 +2363,14 @@ def CalculateMakefilePath(build_file, base_name): } ) elif flavor == "solaris": - header_params.update({"flock": "./gyp-flock-tool flock", "flock_index": 2}) + copy_archive_arguments = "-pPRf@" + header_params.update( + { + "copy_archive_args": copy_archive_arguments, + "flock": "./gyp-flock-tool flock", + "flock_index": 2 + } + ) elif flavor == "freebsd": # Note: OpenBSD has sysutils/flock. lockf seems to be FreeBSD specific. header_params.update({"flock": "lockf"}) diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index 3a8aac0eb5..5bfceb05cb 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -39,6 +39,7 @@ # letters. VALID_MSVS_GUID_CHARS = re.compile(r"^[A-F0-9\-]+$") +generator_supports_multiple_toolsets = gyp.common.CrossCompileRequested() generator_default_variables = { "DRIVER_PREFIX": "", @@ -50,7 +51,7 @@ "STATIC_LIB_SUFFIX": ".lib", "SHARED_LIB_SUFFIX": ".dll", "INTERMEDIATE_DIR": "$(IntDir)", - "SHARED_INTERMEDIATE_DIR": "$(OutDir)obj/global_intermediate", + "SHARED_INTERMEDIATE_DIR": "$(OutDir)/obj/global_intermediate", "OS": "win", "PRODUCT_DIR": "$(OutDir)", "LIB_DIR": "$(OutDir)lib", @@ -1005,7 +1006,7 @@ def _GetMsbuildToolsetOfProject(proj_path, spec, version): return toolset -def _GenerateProject(project, options, version, generator_flags): +def _GenerateProject(project, options, version, generator_flags, spec): """Generates a vcproj file. Arguments: @@ -1023,7 +1024,7 @@ def _GenerateProject(project, options, version, generator_flags): return [] if version.UsesVcxproj(): - return _GenerateMSBuildProject(project, options, version, generator_flags) + return _GenerateMSBuildProject(project, options, version, generator_flags, spec) else: return _GenerateMSVSProject(project, options, version, generator_flags) @@ -1903,6 +1904,8 @@ def _GatherSolutionFolders(sln_projects, project_objects, flat): # Convert into a tree of dicts on path. for p in sln_projects: gyp_file, target = gyp.common.ParseQualifiedTarget(p)[0:2] + if p.endswith("#host"): + target += "_host" gyp_dir = os.path.dirname(gyp_file) path_dict = _GetPathDict(root, gyp_dir) path_dict[target + ".vcproj"] = project_objects[p] @@ -1921,9 +1924,10 @@ def _GetPathOfProject(qualified_target, spec, options, msvs_version): default_config = _GetDefaultConfiguration(spec) proj_filename = default_config.get("msvs_existing_vcproj") if not proj_filename: - proj_filename = ( - spec["target_name"] + options.suffix + msvs_version.ProjectExtension() - ) + proj_filename = spec["target_name"] + if spec["toolset"] == "host": + proj_filename += "_host" + proj_filename = proj_filename + options.suffix + msvs_version.ProjectExtension() build_file = gyp.common.BuildFile(qualified_target) proj_path = os.path.join(os.path.dirname(build_file), proj_filename) @@ -1948,6 +1952,8 @@ def _GetPlatformOverridesOfProject(spec): _ConfigBaseName(config_name, _ConfigPlatform(c)), platform, ) + if spec["toolset"] == "host" and generator_supports_multiple_toolsets: + fixed_config_fullname = "%s|x64" % (config_name,) config_platform_overrides[config_fullname] = fixed_config_fullname return config_platform_overrides @@ -1968,11 +1974,6 @@ def _CreateProjectObjects(target_list, target_dicts, options, msvs_version): projects = {} for qualified_target in target_list: spec = target_dicts[qualified_target] - if spec["toolset"] != "target": - raise GypError( - "Multiple toolsets not supported in msvs build (target %s)" - % qualified_target - ) proj_path, fixpath_prefix = _GetPathOfProject( qualified_target, spec, options, msvs_version ) @@ -1980,9 +1981,12 @@ def _CreateProjectObjects(target_list, target_dicts, options, msvs_version): overrides = _GetPlatformOverridesOfProject(spec) build_file = gyp.common.BuildFile(qualified_target) # Create object for this project. + target_name = spec["target_name"] + if spec["toolset"] == "host": + target_name += "_host" obj = MSVSNew.MSVSProject( proj_path, - name=spec["target_name"], + name=target_name, guid=guid, spec=spec, build_file=build_file, @@ -2161,7 +2165,10 @@ def GenerateOutput(target_list, target_dicts, data, params): for qualified_target in target_list: spec = target_dicts[qualified_target] for config_name, config in spec["configurations"].items(): - configs.add(_ConfigFullName(config_name, config)) + config_name = _ConfigFullName(config_name, config) + configs.add(config_name) + if config_name == "Release|arm64": + configs.add("Release|x64") configs = list(configs) # Figure out all the projects that will be generated and their guids @@ -2174,12 +2181,15 @@ def GenerateOutput(target_list, target_dicts, data, params): for project in project_objects.values(): fixpath_prefix = project.fixpath_prefix missing_sources.extend( - _GenerateProject(project, options, msvs_version, generator_flags) + _GenerateProject(project, options, msvs_version, generator_flags, spec) ) fixpath_prefix = None for build_file in data: # Validate build_file extension + target_only_configs = configs + if generator_supports_multiple_toolsets: + target_only_configs = [i for i in configs if i.endswith("arm64")] if not build_file.endswith(".gyp"): continue sln_path = os.path.splitext(build_file)[0] + options.suffix + ".sln" @@ -2196,7 +2206,7 @@ def GenerateOutput(target_list, target_dicts, data, params): sln = MSVSNew.MSVSSolution( sln_path, entries=root_entries, - variants=configs, + variants=target_only_configs, websiteProperties=False, version=msvs_version, ) @@ -2930,22 +2940,24 @@ def _GenerateMSBuildRuleXmlFile(xml_path, msbuild_rules): easy_xml.WriteXmlIfChanged(content, xml_path, pretty=True, win32=True) -def _GetConfigurationAndPlatform(name, settings): +def _GetConfigurationAndPlatform(name, settings, spec): configuration = name.rsplit("_", 1)[0] platform = settings.get("msvs_configuration_platform", "Win32") + if spec["toolset"] == "host" and platform == "arm64": + platform = "x64" # Host-only tools are always built for x64 return (configuration, platform) -def _GetConfigurationCondition(name, settings): +def _GetConfigurationCondition(name, settings, spec): return r"'$(Configuration)|$(Platform)'=='%s|%s'" % _GetConfigurationAndPlatform( - name, settings + name, settings, spec ) -def _GetMSBuildProjectConfigurations(configurations): +def _GetMSBuildProjectConfigurations(configurations, spec): group = ["ItemGroup", {"Label": "ProjectConfigurations"}] for (name, settings) in sorted(configurations.items()): - configuration, platform = _GetConfigurationAndPlatform(name, settings) + configuration, platform = _GetConfigurationAndPlatform(name, settings, spec) designation = "%s|%s" % (configuration, platform) group.append( [ @@ -3033,7 +3045,7 @@ def _GetMSBuildConfigurationDetails(spec, build_file): properties = {} for name, settings in spec["configurations"].items(): msbuild_attributes = _GetMSBuildAttributes(spec, settings, build_file) - condition = _GetConfigurationCondition(name, settings) + condition = _GetConfigurationCondition(name, settings, spec) character_set = msbuild_attributes.get("CharacterSet") config_type = msbuild_attributes.get("ConfigurationType") _AddConditionalProperty(properties, condition, "ConfigurationType", config_type) @@ -3064,12 +3076,12 @@ def _GetMSBuildLocalProperties(msbuild_toolset): return properties -def _GetMSBuildPropertySheets(configurations): +def _GetMSBuildPropertySheets(configurations, spec): user_props = r"$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" additional_props = {} props_specified = False for name, settings in sorted(configurations.items()): - configuration = _GetConfigurationCondition(name, settings) + configuration = _GetConfigurationCondition(name, settings, spec) if "msbuild_props" in settings: additional_props[configuration] = _FixPaths(settings["msbuild_props"]) props_specified = True @@ -3222,7 +3234,7 @@ def _GetMSBuildConfigurationGlobalProperties(spec, configurations, build_file): properties = {} for (name, configuration) in sorted(configurations.items()): - condition = _GetConfigurationCondition(name, configuration) + condition = _GetConfigurationCondition(name, configuration, spec) attributes = _GetMSBuildAttributes(spec, configuration, build_file) msbuild_settings = configuration["finalized_msbuild_settings"] _AddConditionalProperty( @@ -3345,7 +3357,7 @@ def _GetMSBuildToolSettingsSections(spec, configurations): msbuild_settings = configuration["finalized_msbuild_settings"] group = [ "ItemDefinitionGroup", - {"Condition": _GetConfigurationCondition(name, configuration)}, + {"Condition": _GetConfigurationCondition(name, configuration, spec)}, ] for tool_name, tool_settings in sorted(msbuild_settings.items()): # Skip the tool named '' which is a holder of global settings handled @@ -3625,7 +3637,7 @@ def _AddSources2( if precompiled_source == source: condition = _GetConfigurationCondition( - config_name, configuration + config_name, configuration, spec ) detail.append( ["PrecompiledHeader", {"Condition": condition}, "Create"] @@ -3652,7 +3664,21 @@ def _GetMSBuildProjectReferences(project): references = [] if project.dependencies: group = ["ItemGroup"] + added_dependency_set = set() for dependency in project.dependencies: + dependency_spec = dependency.spec + should_skip_dep = False + if project.spec["toolset"] == "target": + if dependency_spec["toolset"] == "host": + if dependency_spec["type"] == "static_library": + should_skip_dep = True + if dependency.name.startswith("run_"): + should_skip_dep = False + if should_skip_dep: + continue + + canonical_name = dependency.name.replace("_host", "") + added_dependency_set.add(canonical_name) guid = dependency.guid project_dir = os.path.split(project.path)[0] relative_path = gyp.common.RelativePath(dependency.path, project_dir) @@ -3675,7 +3701,7 @@ def _GetMSBuildProjectReferences(project): return references -def _GenerateMSBuildProject(project, options, version, generator_flags): +def _GenerateMSBuildProject(project, options, version, generator_flags, spec): spec = project.spec configurations = spec["configurations"] project_dir, project_file_name = os.path.split(project.path) @@ -3774,7 +3800,7 @@ def _GenerateMSBuildProject(project, options, version, generator_flags): }, ] - content += _GetMSBuildProjectConfigurations(configurations) + content += _GetMSBuildProjectConfigurations(configurations, spec) content += _GetMSBuildGlobalProperties( spec, version, project.guid, project_file_name ) @@ -3789,7 +3815,7 @@ def _GenerateMSBuildProject(project, options, version, generator_flags): if spec.get("msvs_enable_marmasm"): content += import_marmasm_props_section content += _GetMSBuildExtensions(props_files_of_rules) - content += _GetMSBuildPropertySheets(configurations) + content += _GetMSBuildPropertySheets(configurations, spec) content += macro_section content += _GetMSBuildConfigurationGlobalProperties( spec, configurations, project.build_file @@ -3893,15 +3919,27 @@ def _GenerateActionsForMSBuild(spec, actions_to_add): sources_handled_by_action = OrderedSet() actions_spec = [] for primary_input, actions in actions_to_add.items(): + if generator_supports_multiple_toolsets: + primary_input = primary_input.replace(".exe", "_host.exe") inputs = OrderedSet() outputs = OrderedSet() descriptions = [] commands = [] for action in actions: + + def fixup_host_exe(i): + if "$(OutDir)" in i: + i = i.replace(".exe", "_host.exe") + return i + + if generator_supports_multiple_toolsets: + action["inputs"] = [fixup_host_exe(i) for i in action["inputs"]] inputs.update(OrderedSet(action["inputs"])) outputs.update(OrderedSet(action["outputs"])) descriptions.append(action["description"]) cmd = action["command"] + if generator_supports_multiple_toolsets: + cmd = cmd.replace(".exe", "_host.exe") # For most actions, add 'call' so that actions that invoke batch files # return and continue executing. msbuild_use_call provides a way to # disable this but I have not seen any adverse effect from doing that diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py index 19e00319a7..384b252e73 100644 --- a/gyp/pylib/gyp/generator/ninja.py +++ b/gyp/pylib/gyp/generator/ninja.py @@ -78,7 +78,7 @@ def QuoteShellArgument(arg, flavor): """Quote a string such that it will be interpreted as a single argument by the shell.""" # Rather than attempting to enumerate the bad shell characters, just - # whitelist common OK ones and quote anything else. + # allow common OK ones and quote anything else. if re.match(r"^[a-zA-Z0-9_=.\\/-]+$", arg): return arg # No quoting necessary. if flavor == "win": @@ -1481,16 +1481,18 @@ def WriteLinkForArch( library_dirs = config.get("library_dirs", []) if self.flavor == "win": library_dirs = [ - self.msvs_settings.ConvertVSMacros(l, config_name) for l in library_dirs + self.msvs_settings.ConvertVSMacros(library_dir, config_name) + for library_dir in library_dirs ] library_dirs = [ - "/LIBPATH:" + QuoteShellArgument(self.GypPathToNinja(l), self.flavor) - for l in library_dirs + "/LIBPATH:" + + QuoteShellArgument(self.GypPathToNinja(library_dir), self.flavor) + for library_dir in library_dirs ] else: library_dirs = [ - QuoteShellArgument("-L" + self.GypPathToNinja(l), self.flavor) - for l in library_dirs + QuoteShellArgument("-L" + self.GypPathToNinja(library_dir), self.flavor) + for library_dir in library_dirs ] libraries = gyp.common.uniquer( diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index 139df75405..00c4ee1f96 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -1619,10 +1619,10 @@ def ExpandWildcardDependencies(targets, data): index = index + 1 -def Unify(l): - """Removes duplicate elements from l, keeping the first element.""" +def Unify(items): + """Removes duplicate elements from items, keeping the first element.""" seen = {} - return [seen.setdefault(e, e) for e in l if e not in seen] + return [seen.setdefault(e, e) for e in items if e not in seen] def RemoveDuplicateDependencies(targets): @@ -1635,10 +1635,10 @@ def RemoveDuplicateDependencies(targets): target_dict[dependency_key] = Unify(dependencies) -def Filter(l, item): - """Removes item from l.""" +def Filter(items, item): + """Removes item from items.""" res = {} - return [res.setdefault(e, e) for e in l if e != item] + return [res.setdefault(e, e) for e in items if e != item] def RemoveSelfDependencies(targets): @@ -2242,11 +2242,11 @@ def MergeLists(to, fro, to_file, fro_file, is_paths=False, append=True): def is_hashable(val): return val.__hash__ - # If x is hashable, returns whether x is in s. Else returns whether x is in l. - def is_in_set_or_list(x, s, l): + # If x is hashable, returns whether x is in s. Else returns whether x is in items. + def is_in_set_or_list(x, s, items): if is_hashable(x): return x in s - return x in l + return x in items prepend_index = 0 diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py index 9a717d9095..42a4ce47ed 100644 --- a/gyp/pylib/gyp/xcode_emulation.py +++ b/gyp/pylib/gyp/xcode_emulation.py @@ -1541,6 +1541,13 @@ def CLTVersion(): except GypError: continue + regex = re.compile(r'Command Line Tools for Xcode\s+(?P\S+)') + try: + output = GetStdout(["/usr/sbin/softwareupdate", "--history"]) + return re.search(regex, output).groupdict()["version"] + except GypError: + return None + def GetStdoutQuiet(cmdlist): """Returns the content of standard output returned by invoking |cmdlist|. diff --git a/gyp/setup.py b/gyp/setup.py index 0781c59184..0f75a99b18 100755 --- a/gyp/setup.py +++ b/gyp/setup.py @@ -15,7 +15,7 @@ setup( name="gyp-next", - version="0.2.1", + version="0.4.0", description="A fork of the GYP build system for use in the Node.js projects", long_description=long_description, long_description_content_type="text/markdown", @@ -27,18 +27,18 @@ entry_points={"console_scripts": ["gyp=gyp:script_main"]}, python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*", classifiers=[ - 'Development Status :: 3 - Alpha', - 'Environment :: Console', - 'Intended Audience :: Developers', - 'License :: OSI Approved :: BSD License', - 'Natural Language :: English', - 'Programming Language :: Python', - 'Programming Language :: Python :: 2', - 'Programming Language :: Python :: 2.7', - 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.5', - 'Programming Language :: Python :: 3.6', - 'Programming Language :: Python :: 3.7', - 'Programming Language :: Python :: 3.8', + "Development Status :: 3 - Alpha", + "Environment :: Console", + "Intended Audience :: Developers", + "License :: OSI Approved :: BSD License", + "Natural Language :: English", + "Programming Language :: Python", + "Programming Language :: Python :: 2", + "Programming Language :: Python :: 2.7", + "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3.5", + "Programming Language :: Python :: 3.6", + "Programming Language :: Python :: 3.7", + "Programming Language :: Python :: 3.8", ], ) diff --git a/gyp/test_gyp.py b/gyp/test_gyp.py index 5975f8d4c6..382e75272d 100755 --- a/gyp/test_gyp.py +++ b/gyp/test_gyp.py @@ -229,8 +229,7 @@ def run_test(self, test, fmt, i): and not (stdout.endswith("NO RESULT\n")) ): print() - for l in stdout.splitlines(): - print(" %s" % l) + print("\n".join(" %s" % line for line in stdout.splitlines())) elif not self.isatty: print() From aaf33c30296ddb71c12e2b587a5ec5add3f8ace0 Mon Sep 17 00:00:00 2001 From: Samuel Attard Date: Tue, 14 Jul 2020 20:03:24 -0700 Subject: [PATCH 169/551] build: add update-gyp script Co-authored-by: Christian Clauss Reviewed-By: Jiawen Geng Reviewed-By: Christian Clauss PR-URL: https://github.com/nodejs/node-gyp/pull/2167 --- update-gyp.py | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 update-gyp.py diff --git a/update-gyp.py b/update-gyp.py new file mode 100755 index 0000000000..aa2bcb9eb9 --- /dev/null +++ b/update-gyp.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python3 + +import argparse +import os +import shutil +import subprocess +import sys +import tarfile +import tempfile +import urllib.request + +BASE_URL = "https://github.com/nodejs/gyp-next/archive/" +CHECKOUT_PATH = os.path.dirname(os.path.realpath(__file__)) +CHECKOUT_GYP_PATH = os.path.join(CHECKOUT_PATH, 'gyp') + +parser = argparse.ArgumentParser() +parser.add_argument("tag", help="gyp tag to update to") +args = parser.parse_args() + +tar_url = BASE_URL + args.tag + ".tar.gz" + +changed_files = subprocess.check_output(["git", "diff", "--name-only"]).strip() +if changed_files: + raise Exception("Can't update gyp while you have uncommitted changes in node-gyp") + +with tempfile.TemporaryDirectory() as tmp_dir: + tar_file = os.path.join(tmp_dir, 'gyp.tar.gz') + unzip_target = os.path.join(tmp_dir, 'gyp') + with open(tar_file, 'wb') as f: + print("Downloading gyp-next@" + args.tag + " into temporary directory...") + print("From: " + tar_url) + with urllib.request.urlopen(tar_url) as in_file: + f.write(in_file.read()) + + print("Unzipping...") + with tarfile.open(tar_file, "r:gz") as tar_ref: + tar_ref.extractall(unzip_target) + + print("Moving to current checkout (" + CHECKOUT_PATH + ")...") + if os.path.exists(CHECKOUT_GYP_PATH): + shutil.rmtree(CHECKOUT_GYP_PATH) + shutil.move(os.path.join(unzip_target, os.listdir(unzip_target)[0]), CHECKOUT_GYP_PATH) + +subprocess.check_output(["git", "add", "gyp"], cwd=CHECKOUT_PATH) +subprocess.check_output(["git", "commit", "-m", "gyp: update gyp to " + args.tag]) From c60379690e0d0b34d4941d535a13f69d55d1a9ce Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Tue, 11 Aug 2020 15:41:31 +1000 Subject: [PATCH 170/551] v7.1.0: bump version and update changelog --- CHANGELOG.md | 10 ++++++++++ package.json | 2 +- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bdea7ae4ba..2605d61fbe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,13 @@ +v7.1.0 2020-08-12 +================= + +* [[`aaf33c3029`](https://github.com/nodejs/node-gyp/commit/aaf33c3029)] - **build**: add update-gyp script (Samuel Attard) [#2167](https://github.com/nodejs/node-gyp/pull/2167) +* * [[`3baa4e4172`](https://github.com/nodejs/node-gyp/commit/3baa4e4172)] - **(SEMVER-MINOR)** **gyp**: update gyp to 0.4.0 (Samuel Attard) [#2165](https://github.com/nodejs/node-gyp/pull/2165) +* * [[`f461d56c53`](https://github.com/nodejs/node-gyp/commit/f461d56c53)] - **(SEMVER-MINOR)** **build**: support apple silicon (arm64 darwin) builds (Samuel Attard) [#2165](https://github.com/nodejs/node-gyp/pull/2165) +* * [[`ee6fa7d3bc`](https://github.com/nodejs/node-gyp/commit/ee6fa7d3bc)] - **docs**: note that node-gyp@7 should solve Catalina CLT issues (Rod Vagg) [#2156](https://github.com/nodejs/node-gyp/pull/2156) +* * [[`4fc8ff179d`](https://github.com/nodejs/node-gyp/commit/4fc8ff179d)] - **doc**: silence curl for macOS Catalina acid test (Chia Wei Ong) [#2150](https://github.com/nodejs/node-gyp/pull/2150) +* * [[`7857cb2eb1`](https://github.com/nodejs/node-gyp/commit/7857cb2eb1)] - **deps**: increase "engines" to "node" : "\>= 10.12.0" (DeeDeeG) [#2153](https://github.com/nodejs/node-gyp/pull/2153) + v7.0.0 2020-06-03 ================= diff --git a/package.json b/package.json index df236d345e..fd286374aa 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "7.0.0", + "version": "7.1.0", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From 2cca9b74f78b5ecec7c9c01c3e99c8d30b4f1130 Mon Sep 17 00:00:00 2001 From: DeeDeeG Date: Sat, 29 Aug 2020 18:58:52 -0400 Subject: [PATCH 171/551] doc: drop the --production flag for installing windows-build-tools This isn't needed, and was probably copy-pasted from windows-build-tools' README.md, which has since been changed to drop the `--production` flag from the install instructions. PR-URL: https://github.com/nodejs/node-gyp/pull/2206 Reviewed-By: Richard Lau Reviewed-By: Christian Clauss Reviewed-By: Rod Vagg --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index fd7e6a117c..cea8015810 100644 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ Install the current version of Python from the [Microsoft Store package](https:/ #### Option 1 -Install all the required tools and configurations using Microsoft's [windows-build-tools](https://github.com/felixrieseberg/windows-build-tools) using `npm install --global --production windows-build-tools` from an elevated PowerShell or CMD.exe (run as Administrator). +Install all the required tools and configurations using Microsoft's [windows-build-tools](https://github.com/felixrieseberg/windows-build-tools) using `npm install --global windows-build-tools` from an elevated PowerShell or CMD.exe (run as Administrator). #### Option 2 From 2317dc400c6e11cba3ed63d1867ad9f46992ee93 Mon Sep 17 00:00:00 2001 From: Shelley Vohr Date: Wed, 13 May 2020 13:19:16 -0700 Subject: [PATCH 172/551] ci: switch to GitHub Actions Co-authored-by: Christian Clauss Co-authored-by: Matias Lopez PR-URL: https://github.com/nodejs/node-gyp/pull/2210 Closes: #2127 Closes: #2209 --- .github/workflows/Python_tests.yml | 40 ------------- .github/workflows/tests.yml | 51 ++++++++++++++++ .travis.yml | 93 ------------------------------ test/test-options.js | 16 +++-- 4 files changed, 62 insertions(+), 138 deletions(-) delete mode 100644 .github/workflows/Python_tests.yml create mode 100644 .github/workflows/tests.yml delete mode 100644 .travis.yml diff --git a/.github/workflows/Python_tests.yml b/.github/workflows/Python_tests.yml deleted file mode 100644 index 067294515d..0000000000 --- a/.github/workflows/Python_tests.yml +++ /dev/null @@ -1,40 +0,0 @@ -# TODO: Line 15, enable python-version: 3.5 -# TODO: Line 36, enable pytest --doctest-modules - -name: Python_tests -on: [push, pull_request] -jobs: - Python_tests: - runs-on: ${{ matrix.os }} - strategy: - fail-fast: false - max-parallel: 15 - matrix: - os: [macos-latest, ubuntu-latest, windows-latest] - python-version: [2.7, 3.6, 3.7, 3.8] # 3.5, - steps: - - uses: actions/checkout@v2 - - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v1 - with: - python-version: ${{ matrix.python-version }} - - name: Install dependencies - run: | - python -m pip install --upgrade pip - pip install flake8 pytest # -r requirements.txt - - name: Lint with flake8 - if: matrix.os == 'ubuntu-latest' - run: | - # stop the build if there are Python syntax errors or undefined names - flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics - # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide - flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics - - name: Test with pytest (Linux and macOS) - if: matrix.os != 'windows-latest' - run: pytest - - name: Test with pytest (Windows) - if: matrix.os == 'windows-latest' - shell: bash - run: GYP_MSVS_VERSION=2015 GYP_MSVS_OVERRIDE_PATH="C:\\Dummy" pytest - # - name: Run doctests with pytest - # run: pytest --doctest-modules diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml new file mode 100644 index 0000000000..729a8f05e9 --- /dev/null +++ b/.github/workflows/tests.yml @@ -0,0 +1,51 @@ +# TODO: Line 47, enable pytest --doctest-modules + +name: Tests +on: [push, pull_request] +jobs: + Tests: + strategy: + fail-fast: false + max-parallel: 15 + matrix: + node: [10.x, 12.x, 14.x] + python: [3.6, 3.7, 3.8] + os: [macos-latest, ubuntu-latest, windows-latest] + runs-on: ${{ matrix.os }} + steps: + - name: Checkout Repository + uses: actions/checkout@v2 + - name: Use Node.js ${{ matrix.node }} + uses: actions/setup-node@v1 + with: + node-version: ${{ matrix.node }} + - name: Use Python ${{ matrix.python }} + uses: actions/setup-python@v2 + with: + python-version: ${{ matrix.python }} + env: + PYTHON_VERSION: ${{ matrix.python }} + - name: Install Dependencies + run: | + npm install --no-progress + pip install flake8 pytest + - name: Set Windows environment + if: matrix.os == 'windows-latest' + run: + echo '::set-env name=GYP_MSVS_VERSION::2015' + echo '::set-env name=GYP_MSVS_OVERRIDE_PATH::C:\\Dummy' + - name: Lint Python + if: matrix.os == 'ubuntu-latest' + run: | + # stop the build if there are Python syntax errors or undefined names + flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics + # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide + flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics + - name: Run Python tests + run: | + python -m pytest + # - name: Run doctests with pytest + # run: python -m pytest --doctest-modules + - name: Run Node tests + run: | + npm test diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index ae691bed48..0000000000 --- a/.travis.yml +++ /dev/null @@ -1,93 +0,0 @@ -dist: xenial -language: python -cache: pip -addons: - homebrew: - update: true - packages: - - npm - - pyenv -jobs: - include: - - name: "Python 2.7 on Linux" - env: NODE_GYP_FORCE_PYTHON=python2 - python: 2.7 - - - name: "Node.js 10 & Python 3.8 on Linux" - python: 3.8 - env: NODE_GYP_FORCE_PYTHON=python3 - before_install: nvm install 10 - - - name: "Node.js 12 & Python 3.5 on Linux" - python: 3.5 - env: NODE_GYP_FORCE_PYTHON=python3 - before_install: nvm install 12 - - name: "Node.js 12 & Python 3.6 on Linux" - python: 3.6 - env: NODE_GYP_FORCE_PYTHON=python3 - before_install: nvm install 12 - - name: "Node.js 12 & Python 3.7 on Linux" - python: 3.7 - env: NODE_GYP_FORCE_PYTHON=python3 - before_install: nvm install 12 - - name: "Node.js 12 & Python 3.8 on Linux" - python: 3.8 - env: NODE_GYP_FORCE_PYTHON=python3 - before_install: nvm install 12 - - - name: "Python 2.7 on macOS" - os: osx - osx_image: xcode11.2 - language: shell # 'language: python' is not yet supported on macOS - env: NODE_GYP_FORCE_PYTHON=python2 PATH=$HOME/.pyenv/shims:$PATH PYENV_VERSION=2.7.17 - before_install: pyenv install $PYENV_VERSION - - name: "Python 3.8 on macOS" - os: osx - osx_image: xcode11.2 - language: shell # 'language: python' is not yet supported on macOS - env: NODE_GYP_FORCE_PYTHON=python3 PATH=$HOME/.pyenv/shims:$PATH PYENV_VERSION=3.8.0 - before_install: pyenv install $PYENV_VERSION - - - name: "Node.js 12 & Python 2.7 on Windows" - os: windows - language: node_js - node_js: 12 # node - env: >- - PATH=/c/Python27:/c/Python27/Scripts:$PATH - NODE_GYP_FORCE_PYTHON=/c/Python27/python.exe - before_install: choco install python2 - - - name: "Node.js 12 & Python 3.7 on Windows" - os: windows - language: node_js - node_js: 12 # node - env: >- - PATH=/c/Python37:/c/Python37/Scripts:$PATH - NODE_GYP_FORCE_PYTHON=/c/Python37/python.exe - before_install: choco install python --version=3.7.4 - - name: "Node.js 12 & Python 3.8 on Windows" - os: windows - language: node_js - node_js: 12 # node - env: >- - PATH=/c/Python38:/c/Python38/Scripts:$PATH - NODE_GYP_FORCE_PYTHON=/c/Python38/python.exe - before_install: choco install python - -install: - - python -m pip install --upgrade flake8 pytest==4.6.6 # pytest 5 no longer supports legacy Python -before_script: - - python -m flake8 --version - # stop the build if there are Python syntax errors or undefined names - - python -m flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics - # exit-zero treats all errors as warnings. Two space indentation is OK. The GitHub editor is 127 chars wide - - python -m flake8 . --count --exit-zero --ignore=E111,E114,W503 --max-complexity=10 --max-line-length=127 --statistics - - npm install - - npm list -script: - - node -e 'require("npmlog").level="verbose"; require("./lib/find-python")(null,()=>{})' - - npm test - - GYP_MSVS_VERSION=2015 GYP_MSVS_OVERRIDE_PATH="C:\\Dummy" python -m pytest -notifications: - on_success: change - on_failure: change # `always` will be the setting once code changes slow down diff --git a/test/test-options.js b/test/test-options.js index 252baa2035..b2ac62c874 100644 --- a/test/test-options.js +++ b/test/test-options.js @@ -3,13 +3,19 @@ const test = require('tap').test const gyp = require('../lib/node-gyp') -test('options in environment', function (t) { +test('options in environment', (t) => { t.plan(1) // `npm test` dumps a ton of npm_config_* variables in the environment. Object.keys(process.env) - .filter(function (key) { return /^npm_config_/.test(key) }) - .forEach(function (key) { delete process.env[key] }) + .filter((key) => /^npm_config_/.test(key)) + .forEach((key) => { delete process.env[key] }) + + // in some platforms, certain keys are stubborn and cannot be removed + const keys = Object.keys(process.env) + .filter((key) => /^npm_config_/.test(key)) + .map((key) => key.substring('npm_config_'.length)) + .concat('argv', 'x') // Zero-length keys should get filtered out. process.env.npm_config_ = '42' @@ -18,8 +24,8 @@ test('options in environment', function (t) { // Except loglevel. process.env.npm_config_loglevel = 'debug' - var g = gyp() + const g = gyp() g.parseArgv(['rebuild']) // Also sets opts.argv. - t.deepEqual(Object.keys(g.opts).sort(), ['argv', 'x']) + t.deepEqual(Object.keys(g.opts).sort(), keys.sort()) }) From 754996b9ec972a33ece95a233ef00cb3c9ab8afa Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Wed, 9 Sep 2020 13:48:09 +1000 Subject: [PATCH 173/551] doc: replace status badges with new Actions badge PR-URL: https://github.com/nodejs/node-gyp/pull/2218 Reviewed-By: Christian Clauss Reviewed-By: Matias Lopez Reviewed-By: Richard Lau --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index cea8015810..e06b01a739 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,6 @@ # `node-gyp` - Node.js native addon build tool -[![Travis CI](https://travis-ci.com/nodejs/node-gyp.svg?branch=master)](https://travis-ci.com/nodejs/node-gyp) -[![Build Status](https://github.com/nodejs/node-gyp/workflows/Python_tests/badge.svg)](https://github.com/nodejs/node-gyp/actions?workflow=Python_tests) +[![Build Status](https://github.com/nodejs/node-gyp/workflows/Tests/badge.svg?branch=master)](https://github.com/nodejs/node-gyp/actions?query=workflow%3ATests+branch%3Amaster) `node-gyp` is a cross-platform command-line tool written in Node.js for compiling native addon modules for Node.js. It contains a vendored copy of the From 7fb314339f74d020b5b03ef0ba2d0691a5fa1654 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Wed, 7 Oct 2020 18:33:14 +0200 Subject: [PATCH 174/551] test: GitHub Actions: Test on Python 3.9 From python: [3.6, 3.7, 3.8] --> python: [3.6, 3.8, 3.9] because if things work on Python 3.6 and 3.8 then they should work on 3.7. https://www.python.org/downloads/release/python-390/ PR-URL: https://github.com/nodejs/node-gyp/pull/2230 Reviewed-By: Shelley Vohr Reviewed-By: Richard Lau --- .github/workflows/tests.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 729a8f05e9..651b1a9f49 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -9,7 +9,7 @@ jobs: max-parallel: 15 matrix: node: [10.x, 12.x, 14.x] - python: [3.6, 3.7, 3.8] + python: [3.6, 3.8, 3.9] os: [macos-latest, ubuntu-latest, windows-latest] runs-on: ${{ matrix.os }} steps: From 3e7f8ccafc8c65fb2392e86b89c5ef7ad7fd5791 Mon Sep 17 00:00:00 2001 From: Martin Midtgaard Date: Tue, 29 Sep 2020 11:33:49 +0200 Subject: [PATCH 175/551] lib: better log message when ps fails PR-URL: https://github.com/nodejs/node-gyp/pull/2229 Reviewed-By: Bartosz Sosnowski Reviewed-By: Rod Vagg --- lib/find-visualstudio.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js index c5d26f9a20..9c6dad90f8 100644 --- a/lib/find-visualstudio.js +++ b/lib/find-visualstudio.js @@ -151,7 +151,7 @@ VisualStudioFinder.prototype = { const failPowershell = () => { this.addLog( - 'could not use PowerShell to find Visual Studio 2017 or newer') + 'could not use PowerShell to find Visual Studio 2017 or newer, try re-running with \'--loglevel silly\' for more details') cb(null) } From ee6a837cb71f465c0e689a81b4adc784bdf8a0c7 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Thu, 15 Oct 2020 10:31:04 +1100 Subject: [PATCH 176/551] gyp: update gyp to 0.6.1 Closes: https://github.com/nodejs/node-gyp/pull/2236 PR-URL: https://github.com/nodejs/node-gyp/pull/2238 Reviewed-By: Christian Clauss Reviewed-By: Richard Lau Reviewed-By: Myles Borins --- gyp/.flake8 | 4 +- gyp/.github/workflows/Python_tests.yml | 2 +- gyp/.github/workflows/node-gyp.yml | 40 +++++++++ gyp/.github/workflows/nodejs-windows.yml | 2 +- gyp/CHANGELOG.md | 28 +++++- gyp/README.md | 3 + gyp/pylib/gyp/MSVSSettings_test.py | 9 +- gyp/pylib/gyp/__init__.py | 17 ---- gyp/pylib/gyp/common.py | 24 +++-- gyp/pylib/gyp/generator/android.py | 2 +- .../gyp/generator/compile_commands_json.py | 4 +- gyp/pylib/gyp/generator/make.py | 88 +++++-------------- gyp/pylib/gyp/generator/msvs.py | 86 ++++++------------ gyp/pylib/gyp/generator/ninja.py | 34 +++---- gyp/pylib/gyp/generator/xcode.py | 15 ++-- gyp/pylib/gyp/input.py | 58 +++--------- gyp/pylib/gyp/msvs_emulation.py | 4 +- gyp/pylib/gyp/xcode_emulation.py | 9 +- gyp/pylib/gyp/xcodeproj_file.py | 20 +++-- gyp/setup.py | 2 +- gyp/tools/pretty_vcproj.py | 2 +- 21 files changed, 210 insertions(+), 243 deletions(-) create mode 100644 gyp/.github/workflows/node-gyp.yml diff --git a/gyp/.flake8 b/gyp/.flake8 index 139e952e7d..ea0c7680ef 100644 --- a/gyp/.flake8 +++ b/gyp/.flake8 @@ -1,4 +1,4 @@ [flake8] -max-complexity = 10 +max-complexity = 101 max-line-length = 88 -extend-ignore = E203,C901,E501 +extend-ignore = E203 # whitespace before ':' to agree with psf/black diff --git a/gyp/.github/workflows/Python_tests.yml b/gyp/.github/workflows/Python_tests.yml index a93b92f426..128654f312 100644 --- a/gyp/.github/workflows/Python_tests.yml +++ b/gyp/.github/workflows/Python_tests.yml @@ -12,7 +12,7 @@ jobs: max-parallel: 15 matrix: os: [macos-latest, ubuntu-latest] # , windows-latest] - python-version: [2.7, 3.6, 3.7, 3.8] # 3.5, + python-version: [2.7, 3.6, 3.7, 3.8, 3.9] steps: - uses: actions/checkout@v2 - name: Set up Python ${{ matrix.python-version }} diff --git a/gyp/.github/workflows/node-gyp.yml b/gyp/.github/workflows/node-gyp.yml new file mode 100644 index 0000000000..78fe502bda --- /dev/null +++ b/gyp/.github/workflows/node-gyp.yml @@ -0,0 +1,40 @@ +name: node-gyp integration + +on: [push, pull_request] + +jobs: + test: + strategy: + fail-fast: false + matrix: + os: [macos-latest, ubuntu-latest, windows-latest] + runs-on: ${{ matrix.os }} + steps: + - name: Clone gyp-next + uses: actions/checkout@v2 + with: + path: gyp-next + - name: Clone nodejs/node-gyp + uses: actions/checkout@v2 + with: + repository: nodejs/node-gyp + path: node-gyp + - uses: actions/setup-node@v1 + with: + node-version: 14.x + - uses: actions/setup-python@v2 + with: + python-version: 3.9 + - name: Install dependencies + run: | + cd node-gyp + npm install --no-progress + - name: Replace gyp in node-gyp + shell: bash + run: | + rm -rf node-gyp/gyp + cp -r gyp-next node-gyp/gyp + - name: Run tests + run: | + cd node-gyp + npm test diff --git a/gyp/.github/workflows/nodejs-windows.yml b/gyp/.github/workflows/nodejs-windows.yml index 48a42372c2..fffe96e33b 100644 --- a/gyp/.github/workflows/nodejs-windows.yml +++ b/gyp/.github/workflows/nodejs-windows.yml @@ -6,7 +6,7 @@ jobs: build-windows: runs-on: windows-latest steps: - - name: Clone node-gyp + - name: Clone gyp-next uses: actions/checkout@v2 with: path: gyp-next diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md index 8cbcdd3b72..ea632d388f 100644 --- a/gyp/CHANGELOG.md +++ b/gyp/CHANGELOG.md @@ -6,6 +6,29 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ## [Unreleased] +## [0.6.1] - 2020-10-14 + +### Fixed +- Correctly rename object files for absolute paths in MSVS generator. + +## [0.6.0] - 2020-10-13 + +### Added +- The Makefile generator will now output shared libraries directly to the product + directory on all platforms (previously only macOS). + +## [0.5.0] - 2020-09-30 + +### Added +- Extended compile_commands_json generator to consider more file extensions than + just `c` and `cc`. `cpp` and `cxx` are now supported. +- Source files with duplicate basenames are now supported. + +### Removed +- The `--no-duplicate-basename-check` option was removed. +- The `msvs_enable_marmasm` configuration option was removed in favor of + auto-inclusion of the "marmasm" sections for Windows on ARM. + ## [0.4.0] - 2020-07-14 ### Added @@ -34,7 +57,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). This is the first release of this project, based on https://chromium.googlesource.com/external/gyp with changes made over the years in Node.js and node-gyp. -[Unreleased]: https://github.com/nodejs/gyp-next/compare/v0.4.0...HEAD +[Unreleased]: https://github.com/nodejs/gyp-next/compare/v0.6.1...HEAD +[0.6.1]: https://github.com/nodejs/gyp-next/compare/v0.6.0...v0.6.1 +[0.6.0]: https://github.com/nodejs/gyp-next/compare/v0.5.0...v0.6.0 +[0.5.0]: https://github.com/nodejs/gyp-next/compare/v0.4.0...v0.5.0 [0.4.0]: https://github.com/nodejs/gyp-next/compare/v0.3.0...v0.4.0 [0.3.0]: https://github.com/nodejs/gyp-next/compare/v0.2.1...v0.3.0 [0.2.1]: https://github.com/nodejs/gyp-next/compare/v0.2.0...v0.2.1 diff --git a/gyp/README.md b/gyp/README.md index c0d73ac958..9ffc2b21e8 100644 --- a/gyp/README.md +++ b/gyp/README.md @@ -2,3 +2,6 @@ GYP can Generate Your Projects. =================================== Documents are available at [gyp.gsrc.io](https://gyp.gsrc.io), or you can check out ```md-pages``` branch to read those documents offline. + +__gyp-next__ is [released](https://github.com/nodejs/gyp-next/releases) to the [__Python Packaging Index__](https://pypi.org/project/gyp-next) (PyPI) and can be installed with the command: +* `python3 -m pip install gyp-next` diff --git a/gyp/pylib/gyp/MSVSSettings_test.py b/gyp/pylib/gyp/MSVSSettings_test.py index c53c88e824..99860c880e 100755 --- a/gyp/pylib/gyp/MSVSSettings_test.py +++ b/gyp/pylib/gyp/MSVSSettings_test.py @@ -678,7 +678,8 @@ def testConvertToMSBuildSettings_warnings(self): "MSBuild, index value (21) not in expected range [0, 3)", "Warning: while converting VCCLCompilerTool/UsePrecompiledHeader to " "MSBuild, index value (13) not in expected range [0, 3)", - "Warning: while converting VCCLCompilerTool/GeneratePreprocessedFile to " + "Warning: while converting " + "VCCLCompilerTool/GeneratePreprocessedFile to " "MSBuild, value must be one of [0, 1, 2]; got 14", "Warning: while converting VCLinkerTool/Driver to " "MSBuild, index value (10) not in expected range [0, 4)", @@ -1348,7 +1349,8 @@ def testConvertToMSBuildSettings_actual(self): "EmbedManifest": "false", "GenerateCatalogFiles": "true", "InputResourceManifests": "asfsfdafs", - "ManifestResourceFile": "$(IntDir)\\$(TargetFileName).embed.manifest.resfdsf", + "ManifestResourceFile": + "$(IntDir)\\$(TargetFileName).embed.manifest.resfdsf", "OutputManifestFile": "$(TargetPath).manifestdfs", "RegistrarScriptFile": "sdfsfd", "ReplacementsFile": "sdffsd", @@ -1532,7 +1534,8 @@ def testConvertToMSBuildSettings_actual(self): "LinkIncremental": "", }, "ManifestResourceCompile": { - "ResourceOutputFileName": "$(IntDir)$(TargetFileName).embed.manifest.resfdsf" + "ResourceOutputFileName": + "$(IntDir)$(TargetFileName).embed.manifest.resfdsf" }, } self.maxDiff = 9999 # on failure display a long diff diff --git a/gyp/pylib/gyp/__init__.py b/gyp/pylib/gyp/__init__.py index e249536429..f6ea625d40 100755 --- a/gyp/pylib/gyp/__init__.py +++ b/gyp/pylib/gyp/__init__.py @@ -68,7 +68,6 @@ def Load( params=None, check=False, circular_check=True, - duplicate_basename_check=True, ): """ Loads one or more specified build files. @@ -156,7 +155,6 @@ def Load( generator_input_info, check, circular_check, - duplicate_basename_check, params["parallel"], params["root_targets"], ) @@ -431,20 +429,6 @@ def gyp_main(args): regenerate=False, help="don't check for circular relationships between files", ) - # --no-duplicate-basename-check disables the check for duplicate basenames - # in a static_library/shared_library project. Visual C++ 2008 generator - # doesn't support this configuration. Libtool on Mac also generates warnings - # when duplicate basenames are passed into Make generator on Mac. - # TODO(yukawa): Remove this option when these legacy generators are - # deprecated. - parser.add_argument( - "--no-duplicate-basename-check", - dest="duplicate_basename_check", - action="store_false", - default=True, - regenerate=False, - help="don't check for duplicate basenames", - ) parser.add_argument( "--no-parallel", action="store_true", @@ -651,7 +635,6 @@ def gyp_main(args): params, options.check, options.circular_check, - options.duplicate_basename_check, ) # TODO(mark): Pass |data| for now because the generator needs a list of diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py index bfe546f867..a915643867 100644 --- a/gyp/pylib/gyp/common.py +++ b/gyp/pylib/gyp/common.py @@ -352,10 +352,14 @@ class Writer(object): """Wrapper around file which only covers the target if it differs.""" def __init__(self): - # On Cygwin remove the "dir" argument because `C:` prefixed paths are treated as relative, - # consequently ending up with current dir "/cygdrive/c/..." being prefixed to those, which was - # obviously a non-existent path, for example: "/cygdrive/c//C:\". - # See https://docs.python.org/2/library/tempfile.html#tempfile.mkstemp for more details + # On Cygwin remove the "dir" argument + # `C:` prefixed paths are treated as relative, + # consequently ending up with current dir "/cygdrive/c/..." + # being prefixed to those, which was + # obviously a non-existent path, + # for example: "/cygdrive/c//C:\". + # For more details see: + # https://docs.python.org/2/library/tempfile.html#tempfile.mkstemp base_temp_dir = "" if IsCygwin() else os.path.dirname(filename) # Pick temporary file. tmp_fd, self.tmp_path = tempfile.mkstemp( @@ -391,13 +395,15 @@ def close(self): # one. os.unlink(self.tmp_path) else: - # The new file is different from the old one, or there is no old one. + # The new file is different from the old one, + # or there is no old one. # Rename the new file to the permanent name. # # tempfile.mkstemp uses an overly restrictive mode, resulting in a # file that can only be read by the owner, regardless of the umask. - # There's no reason to not respect the umask here, which means that - # an extra hoop is required to fetch it and reset the new file's mode. + # There's no reason to not respect the umask here, + # which means that an extra hoop is required + # to fetch it and reset the new file's mode. # # No way to get the umask without setting a new one? Set a safe one # and then set it back to the old value. @@ -406,8 +412,8 @@ def close(self): os.chmod(self.tmp_path, 0o666 & ~umask) if sys.platform == "win32" and os.path.exists(filename): # NOTE: on windows (but not cygwin) rename will not replace an - # existing file, so it must be preceded with a remove. Sadly there - # is no way to make the switch atomic. + # existing file, so it must be preceded with a remove. + # Sadly there is no way to make the switch atomic. os.remove(filename) os.rename(self.tmp_path, filename) except Exception: diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py index 3ac61008b9..16728847c5 100644 --- a/gyp/pylib/gyp/generator/android.py +++ b/gyp/pylib/gyp/generator/android.py @@ -557,7 +557,7 @@ def WriteSources(self, spec, configs, extra_sources): These are source files necessary to build the current target. We need to handle shared_intermediate directory source files as a special case by copying them to the intermediate directory and - treating them as a genereated sources. Otherwise the Android build + treating them as a generated sources. Otherwise the Android build rules won't pick them up. Args: diff --git a/gyp/pylib/gyp/generator/compile_commands_json.py b/gyp/pylib/gyp/generator/compile_commands_json.py index 9bc21ff834..f330a04dea 100644 --- a/gyp/pylib/gyp/generator/compile_commands_json.py +++ b/gyp/pylib/gyp/generator/compile_commands_json.py @@ -61,8 +61,8 @@ def AddCommandsForTarget(cwd, target, params, per_config_commands): defines = ["-D" + s for s in defines] # TODO(bnoordhuis) Handle generated source files. - sources = target.get("sources", []) - sources = [s for s in sources if s.endswith(".c") or s.endswith(".cc")] + extensions = (".c", ".cc", ".cpp", ".cxx") + sources = [s for s in target.get("sources", []) if s.endswith(extensions)] def resolve(filename): return os.path.abspath(os.path.join(cwd, filename)) diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index 6e1c5205cf..d163ae3135 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -30,7 +30,6 @@ import gyp.common import gyp.xcode_emulation from gyp.common import GetEnvironFallback -from gyp.common import GypError import hashlib @@ -177,7 +176,7 @@ def CalculateGeneratorInputInfo(params): quiet_cmd_solink_module = SOLINK_MODULE($(TOOLSET)) $@ cmd_solink_module = $(LINK.$(TOOLSET)) -o $@ -shared $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -Wl,-soname=$(@F) -Wl,--start-group $(filter-out FORCE_DO_CMD, $^) -Wl,--end-group $(LIBS) -""" +""" # noqa: E501 LINK_COMMANDS_MAC = """\ quiet_cmd_alink = LIBTOOL-STATIC $@ @@ -191,7 +190,7 @@ def CalculateGeneratorInputInfo(params): quiet_cmd_solink_module = SOLINK_MODULE($(TOOLSET)) $@ cmd_solink_module = $(LINK.$(TOOLSET)) -bundle $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ $(filter-out FORCE_DO_CMD, $^) $(LIBS) -""" +""" # noqa: E501 LINK_COMMANDS_ANDROID = """\ quiet_cmd_alink = AR($(TOOLSET)) $@ @@ -218,7 +217,7 @@ def CalculateGeneratorInputInfo(params): cmd_solink_module = $(LINK.$(TOOLSET)) -shared $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -Wl,-soname=$(@F) -o $@ -Wl,--start-group $(filter-out FORCE_DO_CMD, $^) -Wl,--end-group $(LIBS) quiet_cmd_solink_module_host = SOLINK_MODULE($(TOOLSET)) $@ cmd_solink_module_host = $(LINK.$(TOOLSET)) -shared $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -Wl,-soname=$(@F) -o $@ $(filter-out FORCE_DO_CMD, $^) $(LIBS) -""" +""" # noqa: E501 LINK_COMMANDS_AIX = """\ @@ -236,7 +235,7 @@ def CalculateGeneratorInputInfo(params): quiet_cmd_solink_module = SOLINK_MODULE($(TOOLSET)) $@ cmd_solink_module = $(LINK.$(TOOLSET)) -shared $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ $(filter-out FORCE_DO_CMD, $^) $(LIBS) -""" +""" # noqa: E501 LINK_COMMANDS_OS390 = """\ @@ -254,8 +253,7 @@ def CalculateGeneratorInputInfo(params): quiet_cmd_solink_module = SOLINK_MODULE($(TOOLSET)) $@ cmd_solink_module = $(LINK.$(TOOLSET)) $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ $(filter-out FORCE_DO_CMD, $^) $(LIBS) -Wl,DLL - -""" +""" # noqa: E501 # Header of toplevel Makefile. @@ -404,7 +402,7 @@ def CalculateGeneratorInputInfo(params): cmd_copy = ln -f "$<" "$@" 2>/dev/null || (rm -rf "$@" && cp %(copy_archive_args)s "$<" "$@") %(link_commands)s -""" +""" # noqa: E501 r""" # Define an escape_quotes function to escape single quotes. # This allows us to handle quotes properly as long as we always use @@ -503,7 +501,7 @@ def CalculateGeneratorInputInfo(params): .PHONY: FORCE_DO_CMD FORCE_DO_CMD: -""" +""" # noqa: E501 ) SHARED_HEADER_MAC_COMMANDS = """ @@ -534,7 +532,7 @@ def CalculateGeneratorInputInfo(params): quiet_cmd_infoplist = INFOPLIST $@ cmd_infoplist = $(CC.$(TOOLSET)) -E -P -Wno-trigraphs -x c $(INFOPLIST_DEFINES) "$<" -o "$@" -""" +""" # noqa: E501 def WriteRootHeaderSuffixRules(writer): @@ -672,43 +670,6 @@ def SourceifyAndQuoteSpaces(path): return QuoteSpaces(Sourceify(path)) -# TODO: Avoid code duplication with _ValidateSourcesForMSVSProject in msvs.py. -def _ValidateSourcesForOSX(spec, all_sources): - """Makes sure if duplicate basenames are not specified in the source list. - - Arguments: - spec: The target dictionary containing the properties of the target. - """ - if spec.get("type", None) != "static_library": - return - - basenames = {} - for source in all_sources: - name, ext = os.path.splitext(source) - is_compiled_file = ext in [".c", ".cc", ".cpp", ".cxx", ".m", ".mm", ".s", ".S"] - if not is_compiled_file: - continue - basename = os.path.basename(name) # Don't include extension. - basenames.setdefault(basename, []).append(source) - - error = "" - for basename, files in basenames.items(): - if len(files) > 1: - error += " %s: %s\n" % (basename, " ".join(files)) - - if error: - print( - ( - "static library %s has several files with the same basename:\n" - % spec["target_name"] - ) - + error - + "libtool on OS X will generate" - + " warnings for them." - ) - raise GypError("Duplicate basenames in sources section, see list above") - - # Map from qualified target to path to output. target_outputs = {} # Map from qualified target to any linkable output. A subset @@ -867,10 +828,6 @@ def Write( # Sources. all_sources = spec.get("sources", []) + extra_sources if all_sources: - if self.flavor == "mac": - # libtool on OS X generates warnings for duplicate basenames in the same - # target. - _ValidateSourcesForOSX(spec, all_sources) self.WriteSources( configs, deps, @@ -1342,7 +1299,10 @@ def WriteSources( ) if self.flavor == "mac": - cflags = self.xcode_settings.GetCflags(configname, arch=config.get('xcode_configuration_platform')) + cflags = self.xcode_settings.GetCflags( + configname, + arch=config.get('xcode_configuration_platform') + ) cflags_c = self.xcode_settings.GetCflagsC(configname) cflags_cc = self.xcode_settings.GetCflagsCC(configname) cflags_objc = self.xcode_settings.GetCflagsObjC(configname) @@ -1659,12 +1619,10 @@ def WriteTarget( ldflags = config.get("ldflags", []) # Compute an rpath for this output if needed. if any(dep.endswith(".so") or ".so." in dep for dep in deps): - # We want to get the literal string "$ORIGIN" into the link command, - # so we need lots of escaping. - ldflags.append(r"-Wl,-rpath=\$$ORIGIN/lib.%s/" % self.toolset) - ldflags.append( - r"-Wl,-rpath-link=\$(builddir)/lib.%s/" % self.toolset - ) + # We want to get the literal string "$ORIGIN" + # into the link command, so we need lots of escaping. + ldflags.append(r"-Wl,-rpath=\$$ORIGIN/") + ldflags.append(r"-Wl,-rpath-link=\$(builddir)/") library_dirs = config.get("library_dirs", []) ldflags += [("-L%s" % library_dir) for library_dir in library_dirs] self.WriteList(ldflags, "LDFLAGS_%s" % configname) @@ -2212,14 +2170,16 @@ def ExpandInputRoot(self, template, expansion, dirname): def _InstallableTargetInstallPath(self): """Returns the location of the final output for an installable target.""" + # Functionality removed for all platforms to match Xcode and hoist + # shared libraries into PRODUCT_DIR for users: # Xcode puts shared_library results into PRODUCT_DIR, and some gyp files # rely on this. Emulate this behavior for mac. - if self.type == "shared_library" and ( - self.flavor != "mac" or self.toolset != "target" - ): - # Install all shared libs into a common directory (per toolset) for - # convenient access with LD_LIBRARY_PATH. - return "$(builddir)/lib.%s/%s" % (self.toolset, self.alias) + # if self.type == "shared_library" and ( + # self.flavor != "mac" or self.toolset != "target" + # ): + # # Install all shared libs into a common directory (per toolset) for + # # convenient access with LD_LIBRARY_PATH. + # return "$(builddir)/lib.%s/%s" % (self.toolset, self.alias) return "$(builddir)/" + self.alias diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index 5bfceb05cb..e05a8a78a9 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -84,7 +84,6 @@ "msvs_enable_winrt", "msvs_requires_importlibrary", "msvs_enable_winphone", - "msvs_enable_marmasm", "msvs_application_type_revision", "msvs_target_platform_version", "msvs_target_platform_minversion", @@ -183,7 +182,8 @@ def _FixPath(path): def _IsWindowsAbsPath(path): """ - On Cygwin systems Python needs a little help determining if a path is an absolute Windows path or not, so that + On Cygwin systems Python needs a little help determining if a path + is an absolute Windows path or not, so that it does not treat those as relative, which results in bad paths like: '..\\C:\\\\some_source_code_file.cc' """ @@ -753,7 +753,7 @@ def _EscapeEnvironmentVariableExpansion(s): Returns: The escaped string. - """ + """ # noqa: E731,E123,E501 s = s.replace("%", "%%") return s @@ -846,7 +846,7 @@ def _EscapeCppDefineForMSVS(s): s = _EscapeEnvironmentVariableExpansion(s) s = _EscapeCommandLineArgumentForMSVS(s) s = _EscapeVCProjCommandLineArgListItem(s) - # cl.exe replaces literal # characters with = in preprocesor definitions for + # cl.exe replaces literal # characters with = in preprocessor definitions for # some reason. Octal-encode to work around that. s = s.replace("#", "\\%03o" % ord("#")) return s @@ -885,7 +885,7 @@ def _EscapeCppDefineForMSBuild(s): s = _EscapeEnvironmentVariableExpansion(s) s = _EscapeCommandLineArgumentForMSBuild(s) s = _EscapeMSBuildSpecialCharacters(s) - # cl.exe replaces literal # characters with = in preprocesor definitions for + # cl.exe replaces literal # characters with = in preprocessor definitions for # some reason. Octal-encode to work around that. s = s.replace("#", "\\%03o" % ord("#")) return s @@ -1029,45 +1029,6 @@ def _GenerateProject(project, options, version, generator_flags, spec): return _GenerateMSVSProject(project, options, version, generator_flags) -# TODO: Avoid code duplication with _ValidateSourcesForOSX in make.py. -def _ValidateSourcesForMSVSProject(spec, version): - """Makes sure if duplicate basenames are not specified in the source list. - - Arguments: - spec: The target dictionary containing the properties of the target. - version: The VisualStudioVersion object. - """ - # This validation should not be applied to MSVC2010 and later. - assert not version.UsesVcxproj() - - # TODO: Check if MSVC allows this for loadable_module targets. - if spec.get("type", None) not in ("static_library", "shared_library"): - return - sources = spec.get("sources", []) - basenames = {} - for source in sources: - name, ext = os.path.splitext(source) - is_compiled_file = ext in [".c", ".cc", ".cpp", ".cxx", ".m", ".mm", ".s", ".S"] - if not is_compiled_file: - continue - basename = os.path.basename(name) # Don't include extension. - basenames.setdefault(basename, []).append(source) - - error = "" - for basename, files in basenames.items(): - if len(files) > 1: - error += " %s: %s\n" % (basename, " ".join(files)) - - if error: - print( - "static library %s has several files with the same basename:\n" - % spec["target_name"] - + error - + "MSVC08 cannot handle that." - ) - raise GypError("Duplicate basenames in sources section, see list above") - - def _GenerateMSVSProject(project, options, version, generator_flags): """Generates a .vcproj file. It may create .rules and .user files too. @@ -1094,11 +1055,6 @@ def _GenerateMSVSProject(project, options, version, generator_flags): for config_name, config in spec["configurations"].items(): _AddConfigurationToMSVSProject(p, spec, config_type, config_name, config) - # MSVC08 and prior version cannot handle duplicate basenames in the same - # target. - # TODO: Take excluded sources into consideration if possible. - _ValidateSourcesForMSVSProject(spec, version) - # Prepare list of sources and excluded sources. gyp_file = os.path.split(project.build_file)[1] sources, excluded_sources = _PrepareListOfSources(spec, generator_flags, gyp_file) @@ -1415,7 +1371,7 @@ def _GetOutputTargetExt(spec): def _GetDefines(config): - """Returns the list of preprocessor definitions for this configuation. + """Returns the list of preprocessor definitions for this configuration. Arguments: config: The dictionary that defines the special processing to be done @@ -2277,7 +2233,7 @@ def _AppendFiltersForMSBuild( sources: The hierarchy of filters and sources to process. extension_to_rule_name: A dictionary mapping file extensions to rules. filter_group: The list to which filter entries will be appended. - source_group: The list to which source entries will be appeneded. + source_group: The list to which source entries will be appended. """ for source in sources: if isinstance(source, MSVSProject.Filter): @@ -2374,8 +2330,8 @@ def _GenerateRulesForMSBuild( ): # MSBuild rules are implemented using three files: an XML file, a .targets # file and a .props file. - # See http://blogs.msdn.com/b/vcblog/archive/2010/04/21/quick-help-on-vs2010-custom-build-rule.aspx - # for more details. + # For more details see: + # https://devblogs.microsoft.com/cppblog/quick-help-on-vs2010-custom-build-rule/ rules = spec.get("rules", []) rules_native = [r for r in rules if not int(r.get("msvs_external_rule", 0))] rules_external = [r for r in rules if int(r.get("msvs_external_rule", 0))] @@ -3623,8 +3579,9 @@ def _AddSources2( if precompiled_source != "": precompiled_source = _FixPath(precompiled_source) if not extensions_excluded_from_precompile: - # If the precompiled header is generated by a C source, we must - # not try to use it for C++ sources, and vice versa. + # If the precompiled header is generated by a C source, + # we must not try to use it for C++ sources, + # and vice versa. basename, extension = os.path.splitext(precompiled_source) if extension == ".c": extensions_excluded_from_precompile = [ @@ -3657,6 +3614,17 @@ def _AddSources2( extension_to_rule_name, _GetUniquePlatforms(spec), ) + if group == "compile": + # Always add an value to support duplicate + # source file basenames. + file_name = os.path.splitext(source)[0] + ".obj" + if os.path.isabs(file_name): + file_name = os.path.splitdrive(file_name)[1] + elif file_name.startswith("..\\"): + file_name = re.sub(r"^(\.\.\\)+", "", file_name) + elif file_name.startswith("$("): + file_name = re.sub(r"^\$\([^)]+\)\\", "", file_name) + detail.append(["ObjectFileName", "$(IntDir)\\" + file_name]) grouped_sources[group].append([element, {"Include": source}] + detail) @@ -3704,6 +3672,7 @@ def _GetMSBuildProjectReferences(project): def _GenerateMSBuildProject(project, options, version, generator_flags, spec): spec = project.spec configurations = spec["configurations"] + toolset = spec["toolset"] project_dir, project_file_name = os.path.split(project.path) gyp.common.EnsureDirExists(project.path) # Prepare list of sources and excluded sources. @@ -3717,6 +3686,7 @@ def _GenerateMSBuildProject(project, options, version, generator_flags, spec): rule_dependencies = set() extension_to_rule_name = {} list_excluded = generator_flags.get("msvs_list_excluded_files", True) + platforms = _GetUniquePlatforms(spec) # Don't generate rules if we are using an external builder like ninja. if not spec.get("msvs_external_builder"): @@ -3759,7 +3729,7 @@ def _GenerateMSBuildProject(project, options, version, generator_flags, spec): sources, rule_dependencies, extension_to_rule_name, - _GetUniquePlatforms(spec), + platforms, ) missing_sources = _VerifySourcesExist(sources, project_dir) @@ -3812,7 +3782,7 @@ def _GenerateMSBuildProject(project, options, version, generator_flags, spec): content += _GetMSBuildLocalProperties(project.msbuild_toolset) content += import_cpp_props_section content += import_masm_props_section - if spec.get("msvs_enable_marmasm"): + if "arm64" in platforms and toolset == "target": content += import_marmasm_props_section content += _GetMSBuildExtensions(props_files_of_rules) content += _GetMSBuildPropertySheets(configurations, spec) @@ -3834,7 +3804,7 @@ def _GenerateMSBuildProject(project, options, version, generator_flags, spec): content += _GetMSBuildProjectReferences(project) content += import_cpp_targets_section content += import_masm_targets_section - if spec.get("msvs_enable_marmasm"): + if "arm64" in platforms and toolset == "target": content += import_marmasm_targets_section content += _GetMSBuildExtensionTargets(targets_files_of_rules) diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py index 384b252e73..e064bad7ed 100644 --- a/gyp/pylib/gyp/generator/ninja.py +++ b/gyp/pylib/gyp/generator/ninja.py @@ -90,7 +90,7 @@ def Define(d, flavor): """Takes a preprocessor define and returns a -D parameter that's ninja- and shell-escaped.""" if flavor == "win": - # cl.exe replaces literal # characters with = in preprocesor definitions for + # cl.exe replaces literal # characters with = in preprocessor definitions for # some reason. Octal-encode to work around that. d = d.replace("#", "\\%03o" % ord("#")) return QuoteShellArgument(ninja_syntax.escape("-D" + d), flavor) @@ -781,9 +781,10 @@ def cygwin_munge(path): rule.get("process_outputs_as_mac_bundle_resources", False) ): extra_mac_bundle_resources += outputs - # Note: This is n_resources * n_outputs_in_rule. Put to-be-removed - # items in a set and remove them all in a single pass if this becomes - # a performance issue. + # Note: This is n_resources * n_outputs_in_rule. + # Put to-be-removed items in a set and + # remove them all in a single pass + # if this becomes a performance issue. if was_mac_bundle_resource: mac_bundle_resources.remove(source) @@ -816,7 +817,7 @@ def cygwin_munge(path): outputs = [self.GypPathToNinja(o, env) for o in outputs] if self.flavor == "win": - # WriteNewNinjaRule uses unique_name for creating an rsp file on win. + # WriteNewNinjaRule uses unique_name to create a rsp file on win. extra_bindings.append( ("unique_name", hashlib.md5(outputs[0]).hexdigest()) ) @@ -853,11 +854,12 @@ def WriteCopies(self, copies, prebuild, mac_bundle_depends): outputs += self.ninja.build(dst, "copy", src, order_only=prebuild) if self.is_mac_bundle: # gyp has mac_bundle_resources to copy things into a bundle's - # Resources folder, but there's no built-in way to copy files to other - # places in the bundle. Hence, some targets use copies for this. Check - # if this file is copied into the current bundle, and if so add it to - # the bundle depends so that dependent targets get rebuilt if the copy - # input changes. + # Resources folder, but there's no built-in way to copy files + # to other places in the bundle. + # Hence, some targets use copies for this. + # Check if this file is copied into the current bundle, + # and if so add it to the bundle depends so + # that dependent targets get rebuilt if the copy input changes. if dst.startswith( self.xcode_settings.GetBundleContentsFolderPath() ): @@ -1513,8 +1515,8 @@ def WriteLinkForArch( if self.flavor != "win": link_file_list = output if self.is_mac_bundle: - # 'Dependency Framework.framework/Versions/A/Dependency Framework' -> - # 'Dependency Framework.framework.rsp' + # 'Dependency Framework.framework/Versions/A/Dependency Framework' + # -> 'Dependency Framework.framework.rsp' link_file_list = self.xcode_settings.GetWrapperName() if arch: link_file_list += "." + arch @@ -1628,7 +1630,8 @@ def WriteTarget(self, spec, config_name, config, link_deps, compile_deps): variables=variables, ) inputs.append(output) - # TODO: It's not clear if libtool_flags should be passed to the alink + # TODO: It's not clear if + # libtool_flags should be passed to the alink # call that combines single-arch .a files into a fat .a file. self.AppendPostbuildVariable( variables, spec, self.target.binary, self.target.binary @@ -2535,9 +2538,10 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name "solink", description="SOLINK $lib", restat=True, - command=mtime_preserving_solink_base % {"suffix": "@$link_file_list"}, + command=mtime_preserving_solink_base % {"suffix": "@$link_file_list"}, # noqa: E501 rspfile="$link_file_list", - rspfile_content="-Wl,--whole-archive $in $solibs -Wl,--no-whole-archive $libs", + rspfile_content=("-Wl,--whole-archive $in $solibs -Wl," + "--no-whole-archive $libs"), pool="link_pool", ) master_ninja.rule( diff --git a/gyp/pylib/gyp/generator/xcode.py b/gyp/pylib/gyp/generator/xcode.py index 535c5400ce..9e7e99e9e1 100644 --- a/gyp/pylib/gyp/generator/xcode.py +++ b/gyp/pylib/gyp/generator/xcode.py @@ -594,7 +594,7 @@ def ExpandXcodeVariables(string, expansions): def EscapeXcodeDefine(s): """We must escape the defines that we give to XCode so that it knows not to split on spaces and to respect backslash and quote literals. However, we - must not quote the define, or Xcode will incorrectly intepret variables + must not quote the define, or Xcode will incorrectly interpret variables especially $(inherited).""" return re.sub(_xcode_define_re, r"\\\1", s) @@ -735,7 +735,8 @@ def GenerateOutput(target_list, target_dicts, data, params): "loadable_module+xcuitest": "com.apple.product-type.bundle.ui-testing", "shared_library+bundle": "com.apple.product-type.framework", "executable+extension+bundle": "com.apple.product-type.app-extension", - "executable+watch+extension+bundle": "com.apple.product-type.watchkit-extension", + "executable+watch+extension+bundle": + "com.apple.product-type.watchkit-extension", "executable+watch+bundle": "com.apple.product-type.application.watchapp", "mac_kernel_extension+bundle": "com.apple.product-type.kernel-extension", } @@ -1026,7 +1027,7 @@ def GenerateOutput(target_list, target_dicts, data, params): for output in rule.get("outputs", []): # Fortunately, Xcode and make both use $(VAR) format for their # variables, so the expansion is the only transformation necessary. - # Any remaning $(VAR)-type variables in the string can be given + # Any remaining $(VAR)-type variables in the string can be given # directly to make, which will pick up the correct settings from # what Xcode puts into the environment. concrete_output = ExpandXcodeVariables(output, rule_input_dict) @@ -1151,8 +1152,8 @@ def GenerateOutput(target_list, target_dicts, data, params): '\t@mkdir -p "%s"\n' % '" "'.join(concrete_output_dirs) ) - # The rule message and action have already had the necessary variable - # substitutions performed. + # The rule message and action have already had + # the necessary variable substitutions performed. if message: # Mark it with note: so Xcode picks it up in build output. makefile.write("\t@echo note: %s\n" % message) @@ -1204,8 +1205,8 @@ def GenerateOutput(target_list, target_dicts, data, params): support_xct.AppendProperty("buildPhases", ssbp) else: # TODO(mark): this assumes too much knowledge of the internals of - # xcodeproj_file; some of these smarts should move into xcodeproj_file - # itself. + # xcodeproj_file; some of these smarts should move + # into xcodeproj_file itself. xct._properties["buildPhases"].insert(prebuild_index, ssbp) prebuild_index = prebuild_index + 1 diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index 00c4ee1f96..5504390c0b 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -66,7 +66,7 @@ def IsPathSection(section): if section in path_sections: return True - # Sections mathing the regexp '_(dir|file|path)s?$' are also + # Sections matching the regexp '_(dir|file|path)s?$' are also # considered PathSections. Using manual string matching since that # is much faster than the regexp and this can be called hundreds of # thousands of times so micro performance matters. @@ -232,8 +232,8 @@ def LoadOneBuildFile(build_file_path, data, aux_data, includes, is_target, check # to make sure platform specific newlines ('\r\n' or '\r') are converted to '\n' # which otherwise will fail eval() if sys.platform == "zos": - # On z/OS, universal-newlines mode treats the file as an ascii file. But since - # node-gyp produces ebcdic files, do not use that mode. + # On z/OS, universal-newlines mode treats the file as an ascii file. + # But since node-gyp produces ebcdic files, do not use that mode. build_file_contents = open(build_file_path, "r").read() else: build_file_contents = open(build_file_path, "rU").read() @@ -1731,10 +1731,10 @@ def ExtractNodeRef(node): node_dependent.dependencies, key=ExtractNodeRef ): if node_dependent_dependency.ref not in flat_list: - # The dependent one or more dependencies not in flat_list. There - # will be more chances to add it to flat_list when examining - # it again as a dependent of those other dependencies, provided - # that there are no cycles. + # The dependent one or more dependencies not in flat_list. + # There will be more chances to add it to flat_list + # when examining it again as a dependent of those other + # dependencies, provided that there are no cycles. is_in_degree_zero = False break @@ -2427,7 +2427,7 @@ def MergeDicts(to, fro, to_file, fro_file): def MergeConfigWithInheritance( new_configuration_dict, build_file, target_dict, configuration, visited ): - # Skip if previously visted. + # Skip if previously visited. if configuration in visited: return @@ -2641,10 +2641,10 @@ def ProcessListFiltersInDict(name, the_dict): pattern_re = re.compile(pattern) if action == "exclude": - # This item matches an exclude regex, so set its value to 0 (exclude). + # This item matches an exclude regex, set its value to 0 (exclude). action_value = 0 elif action == "include": - # This item matches an include regex, so set its value to 1 (include). + # This item matches an include regex, set its value to 1 (include). action_value = 1 else: # This is an action that doesn't make any sense. @@ -2659,8 +2659,8 @@ def ProcessListFiltersInDict(name, the_dict): for index, list_item in enumerate(the_list): if list_actions[index] == action_value: - # Even if the regex matches, nothing will change so continue (regex - # searches are expensive). + # Even if the regex matches, nothing will change so continue + # (regex searches are expensive). continue if pattern_re.search(list_item): # Regular expression match. @@ -2750,36 +2750,6 @@ def ValidateTargetType(target, target_dict): ) -def ValidateSourcesInTarget(target, target_dict, build_file, duplicate_basename_check): - if not duplicate_basename_check: - return - if target_dict.get("type", None) != "static_library": - return - sources = target_dict.get("sources", []) - basenames = {} - for source in sources: - name, ext = os.path.splitext(source) - is_compiled_file = ext in [".c", ".cc", ".cpp", ".cxx", ".m", ".mm", ".s", ".S"] - if not is_compiled_file: - continue - basename = os.path.basename(name) # Don't include extension. - basenames.setdefault(basename, []).append(source) - - error = "" - for basename, files in basenames.items(): - if len(files) > 1: - error += " %s: %s\n" % (basename, " ".join(files)) - - if error: - print( - "static library %s has several files with the same basename:\n" % target - + error - + "libtool on Mac cannot handle that. Use " - "--no-duplicate-basename-check to disable this validation." - ) - raise GypError("Duplicate basenames in sources section, see list above") - - def ValidateRulesInTarget(target, target_dict, extra_sources_for_rules): """Ensures that the rules sections in target_dict are valid and consistent, and determines which sources they apply to. @@ -3021,7 +2991,6 @@ def Load( generator_input_info, check, circular_check, - duplicate_basename_check, parallel, root_targets, ): @@ -3167,9 +3136,6 @@ def Load( target_dict = targets[target] build_file = gyp.common.BuildFile(target) ValidateTargetType(target, target_dict) - ValidateSourcesInTarget( - target, target_dict, build_file, duplicate_basename_check - ) ValidateRulesInTarget(target, target_dict, extra_sources_for_rules) ValidateRunAsInTarget(target, target_dict, build_file) ValidateActionsInTarget(target, target_dict, build_file) diff --git a/gyp/pylib/gyp/msvs_emulation.py b/gyp/pylib/gyp/msvs_emulation.py index c665c9bf90..1afc1d687e 100644 --- a/gyp/pylib/gyp/msvs_emulation.py +++ b/gyp/pylib/gyp/msvs_emulation.py @@ -814,8 +814,8 @@ def _GetLdManifestFlags( manifest_files = [] generated_manifest_outer = ( "" - "%s" - "" + "" + "%s" ) if enable_uac == "true": execution_level = self._Setting( diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py index 42a4ce47ed..8af2b39f9a 100644 --- a/gyp/pylib/gyp/xcode_emulation.py +++ b/gyp/pylib/gyp/xcode_emulation.py @@ -113,7 +113,7 @@ def GetXcodeArchsDefault(): of $(ARCHS_STANDARD_INCLUDING_64_BIT) from Xcode 5.0. From Xcode 5.1, they are also part of $(ARCHS_STANDARD). - All thoses rules are coded in the construction of the |XcodeArchsDefault| + All these rules are coded in the construction of the |XcodeArchsDefault| object to use depending on the version of Xcode detected. The object is for performance reason.""" global XCODE_ARCHS_DEFAULT_CACHE @@ -1528,7 +1528,9 @@ def CLTVersion(): # volume: / # location: / # install-time: 1382544035 - # groups: com.apple.FindSystemFiles.pkg-group com.apple.DevToolsBoth.pkg-group com.apple.DevToolsNonRelocatableShared.pkg-group + # groups: com.apple.FindSystemFiles.pkg-group + # com.apple.DevToolsBoth.pkg-group + # com.apple.DevToolsNonRelocatableShared.pkg-group STANDALONE_PKG_ID = "com.apple.pkg.DeveloperToolsCLILeo" FROM_XCODE_PKG_ID = "com.apple.pkg.DeveloperToolsCLI" MAVERICKS_PKG_ID = "com.apple.pkg.CLTools_Executables" @@ -1732,7 +1734,8 @@ def _GetXcodeEnv( "BUILT_PRODUCTS_DIR": built_products_dir, "CONFIGURATION": configuration, "PRODUCT_NAME": xcode_settings.GetProductName(), - # See /Developer/Platforms/MacOSX.platform/Developer/Library/Xcode/Specifications/MacOSX\ Product\ Types.xcspec for FULL_PRODUCT_NAME + # For FULL_PRODUCT_NAME see: + # /Developer/Platforms/MacOSX.platform/Developer/Library/Xcode/Specifications/MacOSX\ Product\ Types.xcspec # noqa: E501 "SRCROOT": srcroot, "SOURCE_ROOT": "${SRCROOT}", # This is not true for static libraries, but currently the env is only diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py index cde4f055f9..d90dd99dcc 100644 --- a/gyp/pylib/gyp/xcodeproj_file.py +++ b/gyp/pylib/gyp/xcodeproj_file.py @@ -515,7 +515,7 @@ def PBXProjectAncestor(self): return None def _EncodeComment(self, comment): - """Encodes a comment to be placed in the project file output, mimicing + """Encodes a comment to be placed in the project file output, mimicking Xcode behavior. """ @@ -543,7 +543,7 @@ def _EncodeTransform(self, match): return self._encode_transforms[ord(char)] def _EncodeString(self, value): - """Encodes a string to be placed in the project file output, mimicing + """Encodes a string to be placed in the project file output, mimicking Xcode behavior. """ @@ -586,7 +586,7 @@ def _XCPrint(self, file, tabs, line): def _XCPrintableValue(self, tabs, value, flatten_list=False): """Returns a representation of value that may be printed in a project file, - mimicing Xcode's behavior. + mimicking Xcode's behavior. _XCPrintableValue can handle str and int values, XCObjects (which are made printable by returning their id property), and list and dict objects @@ -831,8 +831,8 @@ def UpdateProperties(self, properties, do_copy=False): self._properties[property] = value elif isinstance(value, list): if is_strong: - # If is_strong is True, each element is an XCObject, so it's safe - # to call Copy. + # If is_strong is True, each element is an XCObject, + # so it's safe to call Copy. self._properties[property] = [] for item in value: self._properties[property].append(item.Copy()) @@ -2132,9 +2132,10 @@ def SetDestination(self, path): # to the target. Xcode uses the dstSubfolderSpec value set here # to determine the full path. # - # An alternative of xcode_emulation.py setting the values to absolute - # paths when exporting these variables has been ruled out because - # then the values would be different depending on the build tool. + # An alternative of xcode_emulation.py setting the values to + # absolute paths when exporting these variables has been + # ruled out because then the values would be different + # depending on the build tool. # # Another alternative is to invent new names for the variables used # to match to the subfolder indices in the second table. .gyp files @@ -2146,7 +2147,8 @@ def SetDestination(self, path): # Requiring prepending BUILT_PRODUCTS_DIR has been chosen because # this same way could be used to specify destinations in .gyp files # that pre-date this addition to GYP. However they would only work - # with the Xcode generator. The previous version of xcode_emulation.py + # with the Xcode generator. + # The previous version of xcode_emulation.py # does not export these variables. Such files will get the benefit # of the Xcode UI showing the proper destination name simply by # regenerating the projects with this version of GYP. diff --git a/gyp/setup.py b/gyp/setup.py index 0f75a99b18..befcdb22b5 100755 --- a/gyp/setup.py +++ b/gyp/setup.py @@ -15,7 +15,7 @@ setup( name="gyp-next", - version="0.4.0", + version="0.6.1", description="A fork of the GYP build system for use in the Node.js projects", long_description=long_description, long_description_content_type="text/markdown", diff --git a/gyp/tools/pretty_vcproj.py b/gyp/tools/pretty_vcproj.py index 38fbf3fecf..b171fae6cf 100755 --- a/gyp/tools/pretty_vcproj.py +++ b/gyp/tools/pretty_vcproj.py @@ -161,7 +161,7 @@ def CleanupVcproj(node): AbsoluteNode(sub_node) CleanupVcproj(sub_node) - # Normalize the node, and remove all extranous whitespaces. + # Normalize the node, and remove all extraneous whitespaces. for sub_node in node.childNodes: if sub_node.nodeType == Node.TEXT_NODE: sub_node.data = sub_node.data.replace("\r", "") From 18bf2d1d38de0098697e521a60c16c6800a4ca33 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Thu, 15 Oct 2020 13:54:13 +1100 Subject: [PATCH 177/551] deps: update deps to match npm@7 PR-URL: https://github.com/nodejs/node-gyp/pull/2240 Reviewed-By: Richard Lau --- package.json | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/package.json b/package.json index fd286374aa..b37b48199f 100644 --- a/package.json +++ b/package.json @@ -25,12 +25,12 @@ "env-paths": "^2.2.0", "glob": "^7.1.4", "graceful-fs": "^4.2.3", - "nopt": "^4.0.3", + "nopt": "^5.0.0", "npmlog": "^4.1.2", "request": "^2.88.2", - "rimraf": "^2.6.3", + "rimraf": "^3.0.2", "semver": "^7.3.2", - "tar": "^6.0.1", + "tar": "^6.0.2", "which": "^2.0.2" }, "engines": { @@ -38,7 +38,7 @@ }, "devDependencies": { "bindings": "^1.5.0", - "nan": "^2.14.1", + "nan": "^2.14.2", "require-inject": "^1.4.4", "standard": "^14.3.4", "tap": "^12.7.0" From b9e3ad25a64aa5783851b6c94eacea40f250663b Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Thu, 15 Oct 2020 11:20:23 +1100 Subject: [PATCH 178/551] v7.1.1: bump version and update changelog PR-URL: https://github.com/nodejs/node-gyp/pull/2239 Reviewed-By: Christian Clauss --- CHANGELOG.md | 18 ++++++++++++++++++ package.json | 2 +- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2605d61fbe..dd357e32f1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,21 @@ +v7.1.1 2020-10-15 +================= + +This release restores the location of shared library builds to the pre-v7 +location. In v7.0.0 until this release, shared library outputs were placed +in a lib.target subdirectory inside the build/{Release,Debug} directory for +builds using `make` (Linux, etc.). This is inconsistent with macOS (Xcode) +behavior and previous node-gyp behavior so has been reverted. +We consider this a bug-fix rather than semver-major change. + +* [[`18bf2d1d38`](https://github.com/nodejs/node-gyp/commit/18bf2d1d38)] - **deps**: update deps to match npm@7 (Rod Vagg) [#2240](https://github.com/nodejs/node-gyp/pull/2240) +* [[`ee6a837cb7`](https://github.com/nodejs/node-gyp/commit/ee6a837cb7)] - **gyp**: update gyp to 0.6.1 (Rod Vagg) [#2238](https://github.com/nodejs/node-gyp/pull/2238) +* [[`3e7f8ccafc`](https://github.com/nodejs/node-gyp/commit/3e7f8ccafc)] - **lib**: better log message when ps fails (Martin Midtgaard) [#2229](https://github.com/nodejs/node-gyp/pull/2229) +* [[`7fb314339f`](https://github.com/nodejs/node-gyp/commit/7fb314339f)] - **test**: GitHub Actions: Test on Python 3.9 (Christian Clauss) [#2230](https://github.com/nodejs/node-gyp/pull/2230) +* [[`754996b9ec`](https://github.com/nodejs/node-gyp/commit/754996b9ec)] - **doc**: replace status badges with new Actions badge (Rod Vagg) [#2218](https://github.com/nodejs/node-gyp/pull/2218) +* [[`2317dc400c`](https://github.com/nodejs/node-gyp/commit/2317dc400c)] - **ci**: switch to GitHub Actions (Shelley Vohr) [#2210](https://github.com/nodejs/node-gyp/pull/2210) +* [[`2cca9b74f7`](https://github.com/nodejs/node-gyp/commit/2cca9b74f7)] - **doc**: drop the --production flag for installing windows-build-tools (DeeDeeG) [#2206](https://github.com/nodejs/node-gyp/pull/2206) + v7.1.0 2020-08-12 ================= diff --git a/package.json b/package.json index b37b48199f..b50ceb42be 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "7.1.0", + "version": "7.1.1", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From 54f97cd2434b61d06516fe2a062dc34a93cbb901 Mon Sep 17 00:00:00 2001 From: Valera Rozuvan Date: Thu, 15 Oct 2020 17:31:17 +0300 Subject: [PATCH 179/551] doc: add cmd to reset `xcode-select` to initial state PR-URL: https://github.com/nodejs/node-gyp/pull/2235 Reviewed-By: Christian Clauss --- macOS_Catalina.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/macOS_Catalina.md b/macOS_Catalina.md index 79bf6ff50b..ca2fd23473 100644 --- a/macOS_Catalina.md +++ b/macOS_Catalina.md @@ -89,12 +89,13 @@ There are three ways to install the Xcode libraries `node-gyp` needs on macOS. P ### I did all that and the acid test still does not pass :-( 1. `sudo rm -rf $(xcode-select -print-path)` # Enter root password. No output is normal. 2. `sudo rm -rf /Library/Developer/CommandLineTools` # Enter root password. -2. `xcode-select --install` -3. If the [_acid test_ steps above](#The-acid-test) still does _not_ pass then... -4. `npm explore npm -g -- npm install node-gyp@latest` -5. `npm explore npm -g -- npm explore npm-lifecycle -- npm install node-gyp@latest` -6. If the _acid test_ still does _not_ pass then... -7. Add a comment to https://github.com/nodejs/node-gyp/issues/1927 so we can improve. +3. `xcode-select --reset` +4. `xcode-select --install` +5. If the [_acid test_ steps above](#The-acid-test) still does _not_ pass then... +6. `npm explore npm -g -- npm install node-gyp@latest` +7. `npm explore npm -g -- npm explore npm-lifecycle -- npm install node-gyp@latest` +8. If the _acid test_ still does _not_ pass then... +9. Add a comment to https://github.com/nodejs/node-gyp/issues/1927 so we can improve. Lessons learned from: * https://github.com/nodejs/node-gyp/issues/1779 From 096e3aded50f9d6eaef6f66510396f095847b361 Mon Sep 17 00:00:00 2001 From: Myles Borins Date: Fri, 16 Oct 2020 14:22:51 -0400 Subject: [PATCH 180/551] gyp: update gyp to 0.6.2 Refs: https://github.com/nodejs/gyp-next/releases/tag/v0.6.2 PR-URL: https://github.com/nodejs/node-gyp/pull/2241 Reviewed-By: Rod Vagg --- gyp/.github/workflows/release-please.yml | 16 +++++ gyp/CHANGELOG.md | 91 ++++++++++++------------ gyp/pylib/gyp/generator/msvs.py | 32 +++++---- gyp/setup.py | 2 +- 4 files changed, 84 insertions(+), 57 deletions(-) create mode 100644 gyp/.github/workflows/release-please.yml mode change 100755 => 100644 gyp/setup.py diff --git a/gyp/.github/workflows/release-please.yml b/gyp/.github/workflows/release-please.yml new file mode 100644 index 0000000000..a414c10e15 --- /dev/null +++ b/gyp/.github/workflows/release-please.yml @@ -0,0 +1,16 @@ +on: + push: + branches: + - master + +name: release-please +jobs: + release-please: + runs-on: ubuntu-latest + steps: + - uses: GoogleCloudPlatform/release-please-action@v2.5.6 + with: + token: ${{ secrets.GITHUB_TOKEN }} + release-type: python + package-name: gyp-next + bump-minor-pre-major: Yes diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md index ea632d388f..53c922b6c9 100644 --- a/gyp/CHANGELOG.md +++ b/gyp/CHANGELOG.md @@ -1,67 +1,70 @@ # Changelog -All notable changes to this project will be documented in this file. +### [0.6.2](https://www.github.com/nodejs/gyp-next/compare/v0.6.1...v0.6.2) (2020-10-16) -The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). -## [Unreleased] +### Bug Fixes -## [0.6.1] - 2020-10-14 +* do not rewrite absolute paths to avoid long paths ([#74](https://www.github.com/nodejs/gyp-next/issues/74)) ([c2ccc1a](https://www.github.com/nodejs/gyp-next/commit/c2ccc1a81f7f94433a94f4d01a2e820db4c4331a)) +* only include MARMASM when toolset is target ([5a2794a](https://www.github.com/nodejs/gyp-next/commit/5a2794aefb58f0c00404ff042b61740bc8b8d5cd)) -### Fixed -- Correctly rename object files for absolute paths in MSVS generator. +### [0.6.1](https://github.com/nodejs/gyp-next/compare/v0.6.0...v0.6.1) (2020-10-14) -## [0.6.0] - 2020-10-13 -### Added -- The Makefile generator will now output shared libraries directly to the product - directory on all platforms (previously only macOS). +### Bug Fixes -## [0.5.0] - 2020-09-30 +* Correctly rename object files for absolute paths in MSVS generator. -### Added -- Extended compile_commands_json generator to consider more file extensions than - just `c` and `cc`. `cpp` and `cxx` are now supported. -- Source files with duplicate basenames are now supported. +## [0.6.0](https://github.com/nodejs/gyp-next/compare/v0.5.0...v0.6.0) (2020-10-13) + + +### Features + +* The Makefile generator will now output shared libraries directly to the product directory on all platforms (previously only macOS). + +## [0.5.0](https://github.com/nodejs/gyp-next/compare/v0.4.0...v0.5.0) (2020-09-30) + + +### Features + +* Extended compile_commands_json generator to consider more file extensions than just `c` and `cc`. `cpp` and `cxx` are now supported. +* Source files with duplicate basenames are now supported. ### Removed -- The `--no-duplicate-basename-check` option was removed. -- The `msvs_enable_marmasm` configuration option was removed in favor of - auto-inclusion of the "marmasm" sections for Windows on ARM. -## [0.4.0] - 2020-07-14 +* The `--no-duplicate-basename-check` option was removed. +* The `msvs_enable_marmasm` configuration option was removed in favor of auto-inclusion of the "marmasm" sections for Windows on ARM. + +## [0.4.0](https://github.com/nodejs/gyp-next/compare/v0.3.0...v0.4.0) (2020-07-14) + + +### Features + +* Added support for passing arbitrary architectures to Xcode builds, enables `arm64` builds. + +### Bug Fixes + +* Fixed a bug on Solaris where copying archives failed. + +## [0.3.0](https://github.com/nodejs/gyp-next/compare/v0.2.1...v0.3.0) (2020-06-06) + -### Added -- Added support for passing arbitrary architectures to Xcode builds, enables `arm64` builds. +### Features -### Fixed -- Fixed a bug on Solaris where copying archives failed. +* Added support for MSVC cross-compilation. This allows compilation on x64 for a Windows ARM target. -## [0.3.0] - 2020-06-06 +### Bug Fixes -### Added -- Added support for MSVC cross-compilation. This allows compilation on x64 for - a Windows ARM target. +* Fixed XCode CLT version detection on macOS Catalina. -### Fixed -- Fixed XCode CLT version detection on macOS Catalina. +### [0.2.1](https://github.com/nodejs/gyp-next/compare/v0.2.0...v0.2.1) (2020-05-05) -## [0.2.1] - 2020-05-05 -### Fixed -- Relicensed to Node.js contributors. -- Fixed Windows bug introduced in v0.2.0. +### Bug Fixes -## [0.2.0] - 2020-04-06 +* Relicensed to Node.js contributors. +* Fixed Windows bug introduced in v0.2.0. -This is the first release of this project, based on https://chromium.googlesource.com/external/gyp -with changes made over the years in Node.js and node-gyp. +## [0.2.0](https://github.com/nodejs/gyp-next/releases/tag/v0.2.0) (2020-04-06) -[Unreleased]: https://github.com/nodejs/gyp-next/compare/v0.6.1...HEAD -[0.6.1]: https://github.com/nodejs/gyp-next/compare/v0.6.0...v0.6.1 -[0.6.0]: https://github.com/nodejs/gyp-next/compare/v0.5.0...v0.6.0 -[0.5.0]: https://github.com/nodejs/gyp-next/compare/v0.4.0...v0.5.0 -[0.4.0]: https://github.com/nodejs/gyp-next/compare/v0.3.0...v0.4.0 -[0.3.0]: https://github.com/nodejs/gyp-next/compare/v0.2.1...v0.3.0 -[0.2.1]: https://github.com/nodejs/gyp-next/compare/v0.2.0...v0.2.1 -[0.2.0]: https://github.com/nodejs/gyp-next/releases/tag/v0.2.0 +This is the first release of this project, based on https://chromium.googlesource.com/external/gyp with changes made over the years in Node.js and node-gyp. diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index e05a8a78a9..32bf4746a1 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -2177,7 +2177,12 @@ def GenerateOutput(target_list, target_dicts, data, params): def _GenerateMSBuildFiltersFile( - filters_path, source_files, rule_dependencies, extension_to_rule_name, platforms + filters_path, + source_files, + rule_dependencies, + extension_to_rule_name, + platforms, + toolset, ): """Generate the filters file. @@ -2197,6 +2202,7 @@ def _GenerateMSBuildFiltersFile( rule_dependencies, extension_to_rule_name, platforms, + toolset, filter_group, source_group, ) @@ -2222,6 +2228,7 @@ def _AppendFiltersForMSBuild( rule_dependencies, extension_to_rule_name, platforms, + toolset, filter_group, source_group, ): @@ -2257,13 +2264,14 @@ def _AppendFiltersForMSBuild( rule_dependencies, extension_to_rule_name, platforms, + toolset, filter_group, source_group, ) else: # It's a source. Create a source entry. _, element = _MapFileToMsBuildSourceType( - source, rule_dependencies, extension_to_rule_name, platforms + source, rule_dependencies, extension_to_rule_name, platforms, toolset ) source_entry = [element, {"Include": source}] # Specify the filter it is part of, if any. @@ -2273,7 +2281,7 @@ def _AppendFiltersForMSBuild( def _MapFileToMsBuildSourceType( - source, rule_dependencies, extension_to_rule_name, platforms + source, rule_dependencies, extension_to_rule_name, platforms, toolset ): """Returns the group and element type of the source file. @@ -2301,9 +2309,8 @@ def _MapFileToMsBuildSourceType( elif ext in [".s", ".asm"]: group = "masm" element = "MASM" - for platform in platforms: - if platform.lower() in ["arm", "arm64"]: - element = "MARMASM" + if "arm64" in platforms and toolset == "target": + element = "MARMASM" elif ext == ".idl": group = "midl" element = "Midl" @@ -3613,14 +3620,14 @@ def _AddSources2( rule_dependencies, extension_to_rule_name, _GetUniquePlatforms(spec), + spec["toolset"], ) - if group == "compile": - # Always add an value to support duplicate - # source file basenames. + if group == "compile" and not os.path.isabs(source): + # Add an value to support duplicate source + # file basenames, except for absolute paths to avoid paths + # with more than 260 characters. file_name = os.path.splitext(source)[0] + ".obj" - if os.path.isabs(file_name): - file_name = os.path.splitdrive(file_name)[1] - elif file_name.startswith("..\\"): + if file_name.startswith("..\\"): file_name = re.sub(r"^(\.\.\\)+", "", file_name) elif file_name.startswith("$("): file_name = re.sub(r"^\$\([^)]+\)\\", "", file_name) @@ -3730,6 +3737,7 @@ def _GenerateMSBuildProject(project, options, version, generator_flags, spec): rule_dependencies, extension_to_rule_name, platforms, + toolset, ) missing_sources = _VerifySourcesExist(sources, project_dir) diff --git a/gyp/setup.py b/gyp/setup.py old mode 100755 new mode 100644 index befcdb22b5..d1869c1b52 --- a/gyp/setup.py +++ b/gyp/setup.py @@ -15,7 +15,7 @@ setup( name="gyp-next", - version="0.6.1", + version="0.6.2", description="A fork of the GYP build system for use in the Node.js projects", long_description=long_description, long_description_content_type="text/markdown", From 19e0f3c6a0e0f6480b03d7843a82811f86dad6cd Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Sat, 17 Oct 2020 10:40:28 +1100 Subject: [PATCH 181/551] v7.1.1: bump version and update changelog PR-URL: https://github.com/nodejs/node-gyp/pull/2242 --- CHANGELOG.md | 6 ++++++ package.json | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dd357e32f1..733a4b5dd6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,9 @@ +v7.1.2 2020-10-17 +================= + +* [[`096e3aded5`](https://github.com/nodejs/node-gyp/commit/096e3aded5)] - **gyp**: update gyp to 0.6.2 (Myles Borins) [#2241](https://github.com/nodejs/node-gyp/pull/2241) +* [[`54f97cd243`](https://github.com/nodejs/node-gyp/commit/54f97cd243)] - **doc**: add cmd to reset `xcode-select` to initial state (Valera Rozuvan) [#2235](https://github.com/nodejs/node-gyp/pull/2235) + v7.1.1 2020-10-15 ================= diff --git a/package.json b/package.json index b50ceb42be..8e256f0170 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "7.1.1", + "version": "7.1.2", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From 66c0f0446749caa591ad841cd029b6d5b5c8da42 Mon Sep 17 00:00:00 2001 From: Karl Horky Date: Sat, 24 Oct 2020 15:10:37 +0200 Subject: [PATCH 182/551] doc: add missing `sudo` to Catalina doc PR-URL: https://github.com/nodejs/node-gyp/pull/2244 Reviewed-By: Rod Vagg --- macOS_Catalina.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/macOS_Catalina.md b/macOS_Catalina.md index ca2fd23473..42ac6f1d0b 100644 --- a/macOS_Catalina.md +++ b/macOS_Catalina.md @@ -89,7 +89,7 @@ There are three ways to install the Xcode libraries `node-gyp` needs on macOS. P ### I did all that and the acid test still does not pass :-( 1. `sudo rm -rf $(xcode-select -print-path)` # Enter root password. No output is normal. 2. `sudo rm -rf /Library/Developer/CommandLineTools` # Enter root password. -3. `xcode-select --reset` +3. `sudo xcode-select --reset` 4. `xcode-select --install` 5. If the [_acid test_ steps above](#The-acid-test) still does _not_ pass then... 6. `npm explore npm -g -- npm install node-gyp@latest` From 15a5c7d45b02c53d17501a30def244222e26b7af Mon Sep 17 00:00:00 2001 From: Jiawen Geng <3759816+gengjiawen@users.noreply.github.com> Date: Fri, 18 Dec 2020 17:25:28 +0800 Subject: [PATCH 183/551] ci: migrate deprecated grammar (#2285) PR-URL: https://github.com/nodejs/node-gyp/pull/2285 Reviewed-By: Richard Lau --- .github/workflows/tests.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 651b1a9f49..2fdf4d4c23 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -31,9 +31,9 @@ jobs: pip install flake8 pytest - name: Set Windows environment if: matrix.os == 'windows-latest' - run: - echo '::set-env name=GYP_MSVS_VERSION::2015' - echo '::set-env name=GYP_MSVS_OVERRIDE_PATH::C:\\Dummy' + run: | + echo 'GYP_MSVS_VERSION=2015' >> $Env:GITHUB_ENV + echo 'GYP_MSVS_OVERRIDE_PATH=C:\\Dummy' >> $Env:GITHUB_ENV - name: Lint Python if: matrix.os == 'ubuntu-latest' run: | From 6287118fc4407edfeda1d092c58f848d23bd6c2f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E0=AE=AE=E0=AE=A9=E0=AF=8B=E0=AE=9C=E0=AF=8D=E0=AE=95?= =?UTF-8?q?=E0=AF=81=E0=AE=AE=E0=AE=BE=E0=AE=B0=E0=AF=8D=20=E0=AE=AA?= =?UTF-8?q?=E0=AE=B4=E0=AE=A9=E0=AE=BF=E0=AE=9A=E0=AF=8D=E0=AE=9A=E0=AE=BE?= =?UTF-8?q?=E0=AE=AE=E0=AE=BF?= Date: Sat, 19 Dec 2020 18:53:43 +0530 Subject: [PATCH 184/551] doc: updated README.md to copy easily (#2281) PR-URL: https://github.com/nodejs/node-gyp/pull/2281 Reviewed-By: Jiawen Geng --- README.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index e06b01a739..64e82dbbe3 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ etc.), regardless of what version of Node.js is actually installed on your syste You can install `node-gyp` using `npm`: ``` bash -$ npm install -g node-gyp +npm install -g node-gyp ``` Depending on your operating system, you will need to install: @@ -71,7 +71,7 @@ version `node-gyp` should use in one of the following ways: 1. by setting the `--python` command-line option, e.g.: ``` bash -$ node-gyp --python /path/to/executable/python +node-gyp --python /path/to/executable/python ``` 2. If `node-gyp` is called by way of `npm`, *and* you have multiple versions of @@ -79,7 +79,7 @@ Python installed, then you can set `npm`'s 'python' config key to the appropriat value: ``` bash -$ npm config set python /path/to/executable/python +npm config set python /path/to/executable/python ``` 3. If the `PYTHON` environment variable is set to the path of a Python executable, @@ -95,20 +95,20 @@ searching will be done. To compile your native addon, first go to its root directory: ``` bash -$ cd my_node_addon +cd my_node_addon ``` The next step is to generate the appropriate project build files for the current platform. Use `configure` for that: ``` bash -$ node-gyp configure +node-gyp configure ``` Auto-detection fails for Visual C++ Build Tools 2015, so `--msvs_version=2015` needs to be added (not needed when run by npm as configured above): ``` bash -$ node-gyp configure --msvs_version=2015 +node-gyp configure --msvs_version=2015 ``` __Note__: The `configure` step looks for a `binding.gyp` file in the current @@ -118,7 +118,7 @@ Now you will have either a `Makefile` (on Unix platforms) or a `vcxproj` file (on Windows) in the `build/` directory. Next, invoke the `build` command: ``` bash -$ node-gyp build +node-gyp build ``` Now you have your compiled `.node` bindings file! The compiled bindings end up @@ -214,13 +214,13 @@ For example, to set `devdir` equal to `/tmp/.gyp`, you would: Run this on Unix: ```bash -$ export npm_config_devdir=/tmp/.gyp +export npm_config_devdir=/tmp/.gyp ``` Or this on Windows: ```console -> set npm_config_devdir=c:\temp\.gyp +set npm_config_devdir=c:\temp\.gyp ``` ### `npm` configuration @@ -230,7 +230,7 @@ Use the form `OPTION_NAME` for any of the command options listed above. For example, to set `devdir` equal to `/tmp/.gyp`, you would run: ```bash -$ npm config set [--global] devdir /tmp/.gyp +npm config set [--global] devdir /tmp/.gyp ``` **Note:** Configuration set via `npm` will only be used when `node-gyp` From 9e1397c52e429eb96a9013622cffffda56c78632 Mon Sep 17 00:00:00 2001 From: Jiawen Geng <3759816+gengjiawen@users.noreply.github.com> Date: Tue, 22 Dec 2020 08:19:25 +0800 Subject: [PATCH 185/551] gyp: update gyp to v0.7.0 (#2284) PR-URL: https://github.com/nodejs/node-gyp/pull/2284 Reviewed-By: Richard Lau Reviewed-By: Christian Clauss --- gyp/CHANGELOG.md | 19 +++++++++ gyp/pylib/gyp/generator/cmake.py | 4 +- gyp/pylib/gyp/generator/msvs.py | 8 +--- gyp/pylib/gyp/input.py | 2 +- gyp/pylib/gyp/xcode_emulation.py | 69 +++++++++++++++++--------------- gyp/setup.py | 2 +- 6 files changed, 61 insertions(+), 43 deletions(-) diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md index 53c922b6c9..7940367695 100644 --- a/gyp/CHANGELOG.md +++ b/gyp/CHANGELOG.md @@ -1,5 +1,24 @@ # Changelog +## [0.7.0](https://www.github.com/nodejs/gyp-next/compare/v0.6.2...v0.7.0) (2020-12-17) + + +### ⚠ BREAKING CHANGES + +* **msvs:** On Windows, arguments passed to the "action" commands are no longer transformed to replace slashes with backslashes. + +### Features + +* **xcode:** --cross-compiling overrides arch-specific settings ([973bae0](https://www.github.com/nodejs/gyp-next/commit/973bae0b7b08be7b680ecae9565fbd04b3e0787d)) + + +### Bug Fixes + +* **msvs:** do not fix paths in action command arguments ([fc22f83](https://www.github.com/nodejs/gyp-next/commit/fc22f8335e2016da4aae4f4233074bd651d2faea)) +* cmake on python 3 ([fd61f5f](https://www.github.com/nodejs/gyp-next/commit/fd61f5faa5275ec8fc98e3c7868c0dd46f109540)) +* ValueError: invalid mode: 'rU' while trying to load binding.gyp ([d0504e6](https://www.github.com/nodejs/gyp-next/commit/d0504e6700ce48f44957a4d5891b142a60be946f)) +* xcode cmake parsing ([eefe8d1](https://www.github.com/nodejs/gyp-next/commit/eefe8d10e99863bc4ac7e2ed32facd608d400d4b)) + ### [0.6.2](https://www.github.com/nodejs/gyp-next/compare/v0.6.1...v0.6.2) (2020-10-16) diff --git a/gyp/pylib/gyp/generator/cmake.py b/gyp/pylib/gyp/generator/cmake.py index f5ceacfca3..75f5822369 100644 --- a/gyp/pylib/gyp/generator/cmake.py +++ b/gyp/pylib/gyp/generator/cmake.py @@ -41,7 +41,7 @@ try: # maketrans moved to str in python3. _maketrans = string.maketrans -except NameError: +except (NameError, AttributeError): _maketrans = str.maketrans generator_default_variables = { @@ -1047,7 +1047,7 @@ def WriteTarget( # XCode settings xcode_settings = config.get("xcode_settings", {}) - for xcode_setting, xcode_value in xcode_settings.viewitems(): + for xcode_setting, xcode_value in xcode_settings.items(): SetTargetProperty( output, cmake_target_name, diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index 32bf4746a1..96283b2757 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -423,13 +423,7 @@ def _BuildCommandLineForRuleRaw( # file out of the raw command string, and some commands (like python) are # actually batch files themselves. command.insert(0, "call") - # Fix the paths - # TODO(quote): This is a really ugly heuristic, and will miss path fixing - # for arguments like "--arg=path" or "/opt:path". - # If the argument starts with a slash or dash, it's probably a command line - # switch - arguments = [i if (i[:1] in "/-") else _FixPath(i) for i in cmd[1:]] - arguments = [i.replace("$(InputDir)", "%INPUTDIR%") for i in arguments] + arguments = [i.replace("$(InputDir)", "%INPUTDIR%") for i in cmd[1:]] arguments = [MSVSSettings.FixVCMacroSlashes(i) for i in arguments] if quote_cmd: # Support a mode for using cmd directly. diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index 5504390c0b..9039776240 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -231,7 +231,7 @@ def LoadOneBuildFile(build_file_path, data, aux_data, includes, is_target, check # Open the build file for read ('r') with universal-newlines mode ('U') # to make sure platform specific newlines ('\r\n' or '\r') are converted to '\n' # which otherwise will fail eval() - if sys.platform == "zos": + if PY3 or sys.platform == "zos": # On z/OS, universal-newlines mode treats the file as an ascii file. # But since node-gyp produces ebcdic files, do not use that mode. build_file_contents = open(build_file_path, "r").read() diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py index 8af2b39f9a..a79aaa41fb 100644 --- a/gyp/pylib/gyp/xcode_emulation.py +++ b/gyp/pylib/gyp/xcode_emulation.py @@ -654,28 +654,32 @@ def GetCflags(self, configname, arch=None): self._WarnUnimplemented("MACH_O_TYPE") self._WarnUnimplemented("PRODUCT_TYPE") - if arch is not None: - archs = [arch] - else: - assert self.configname - archs = self.GetActiveArchs(self.configname) - if len(archs) != 1: - # TODO: Supporting fat binaries will be annoying. - self._WarnUnimplemented("ARCHS") - archs = ["i386"] - cflags.append("-arch " + archs[0]) - - if archs[0] in ("i386", "x86_64"): - if self._Test("GCC_ENABLE_SSE3_EXTENSIONS", "YES", default="NO"): - cflags.append("-msse3") - if self._Test( - "GCC_ENABLE_SUPPLEMENTAL_SSE3_INSTRUCTIONS", "YES", default="NO" - ): - cflags.append("-mssse3") # Note 3rd 's'. - if self._Test("GCC_ENABLE_SSE41_EXTENSIONS", "YES", default="NO"): - cflags.append("-msse4.1") - if self._Test("GCC_ENABLE_SSE42_EXTENSIONS", "YES", default="NO"): - cflags.append("-msse4.2") + # If GYP_CROSSCOMPILE (--cross-compiling), disable architecture-specific + # additions and assume these will be provided as required via CC_host, + # CXX_host, CC_target and CXX_target. + if not gyp.common.CrossCompileRequested(): + if arch is not None: + archs = [arch] + else: + assert self.configname + archs = self.GetActiveArchs(self.configname) + if len(archs) != 1: + # TODO: Supporting fat binaries will be annoying. + self._WarnUnimplemented("ARCHS") + archs = ["i386"] + cflags.append("-arch " + archs[0]) + + if archs[0] in ("i386", "x86_64"): + if self._Test("GCC_ENABLE_SSE3_EXTENSIONS", "YES", default="NO"): + cflags.append("-msse3") + if self._Test( + "GCC_ENABLE_SUPPLEMENTAL_SSE3_INSTRUCTIONS", "YES", default="NO" + ): + cflags.append("-mssse3") # Note 3rd 's'. + if self._Test("GCC_ENABLE_SSE41_EXTENSIONS", "YES", default="NO"): + cflags.append("-msse4.1") + if self._Test("GCC_ENABLE_SSE42_EXTENSIONS", "YES", default="NO"): + cflags.append("-msse4.2") cflags += self._Settings().get("WARNING_CFLAGS", []) @@ -938,16 +942,17 @@ def GetLdflags(self, configname, product_dir, gyp_to_build_path, arch=None): + gyp_to_build_path(self._Settings()["ORDER_FILE"]) ) - if arch is not None: - archs = [arch] - else: - assert self.configname - archs = self.GetActiveArchs(self.configname) - if len(archs) != 1: - # TODO: Supporting fat binaries will be annoying. - self._WarnUnimplemented("ARCHS") - archs = ["i386"] - ldflags.append("-arch " + archs[0]) + if not gyp.common.CrossCompileRequested(): + if arch is not None: + archs = [arch] + else: + assert self.configname + archs = self.GetActiveArchs(self.configname) + if len(archs) != 1: + # TODO: Supporting fat binaries will be annoying. + self._WarnUnimplemented("ARCHS") + archs = ["i386"] + ldflags.append("-arch " + archs[0]) # Xcode adds the product directory by default. # Rewrite -L. to -L./ to work around http://www.openradar.me/25313838 diff --git a/gyp/setup.py b/gyp/setup.py index d1869c1b52..766a7651ba 100644 --- a/gyp/setup.py +++ b/gyp/setup.py @@ -15,7 +15,7 @@ setup( name="gyp-next", - version="0.6.2", + version="0.7.0", description="A fork of the GYP build system for use in the Node.js projects", long_description=long_description, long_description_content_type="text/markdown", From cc1cbce056b0db0a98aa6041f9148c7d9326c531 Mon Sep 17 00:00:00 2001 From: iMrLopez <8272737+iMrLopez@users.noreply.github.com> Date: Tue, 5 Jan 2021 00:48:40 -0600 Subject: [PATCH 186/551] doc: update macOS_Catalina.md (#2293) PR-URL: https://github.com/nodejs/node-gyp/pull/2293 Reviewed-By: Jiawen Geng --- macOS_Catalina.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/macOS_Catalina.md b/macOS_Catalina.md index 42ac6f1d0b..4fe0f29b21 100644 --- a/macOS_Catalina.md +++ b/macOS_Catalina.md @@ -40,7 +40,7 @@ To see if `Xcode Command Line Tools` is installed in a way that will work with ` curl -sL https://github.com/nodejs/node-gyp/raw/master/macOS_Catalina_acid_test.sh | bash ``` -If test succeeded, _you are done_! You should be ready to install `node-gyp`. +If test succeeded, _you are done_! You should be ready to [install](https://github.com/nodejs/node-gyp#installation) `node-gyp`. If test failed, there is a problem with your Xcode Command Line Tools installation. [Continue to Solutions](#Solutions). From c3c510d89ede3a747eb679a49254052344ed8bc3 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Thu, 28 Jan 2021 08:11:53 +0100 Subject: [PATCH 187/551] gyp: update gyp to v0.8.0 (#2318) PR-URL: https://github.com/nodejs/node-gyp/pull/2318 Reviewed-By: Rod Vagg Reviewed-By: Jiawen Geng --- gyp/.github/workflows/Python_tests.yml | 5 +- gyp/CHANGELOG.md | 16 ++ gyp/gyp_main.py | 12 +- gyp/pylib/gyp/MSVSNew.py | 19 +- gyp/pylib/gyp/MSVSProject.py | 6 +- gyp/pylib/gyp/MSVSSettings.py | 26 +- gyp/pylib/gyp/MSVSSettings_test.py | 5 +- gyp/pylib/gyp/MSVSToolFile.py | 2 +- gyp/pylib/gyp/MSVSUserFile.py | 4 +- gyp/pylib/gyp/MSVSUtil.py | 4 +- gyp/pylib/gyp/MSVSVersion.py | 24 +- gyp/pylib/gyp/__init__.py | 12 +- gyp/pylib/gyp/common.py | 19 +- gyp/pylib/gyp/easy_xml.py | 8 +- gyp/pylib/gyp/easy_xml_test.py | 5 +- gyp/pylib/gyp/flock_tool.py | 2 +- gyp/pylib/gyp/generator/analyzer.py | 13 +- gyp/pylib/gyp/generator/android.py | 227 ++++++++------- gyp/pylib/gyp/generator/cmake.py | 24 +- .../gyp/generator/dump_dependency_json.py | 1 - gyp/pylib/gyp/generator/eclipse.py | 10 +- gyp/pylib/gyp/generator/gypsh.py | 2 +- gyp/pylib/gyp/generator/make.py | 264 +++++++++--------- gyp/pylib/gyp/generator/msvs.py | 47 ++-- gyp/pylib/gyp/generator/msvs_test.py | 5 +- gyp/pylib/gyp/generator/ninja.py | 206 +++++++------- gyp/pylib/gyp/generator/xcode.py | 13 +- gyp/pylib/gyp/input.py | 30 +- gyp/pylib/gyp/mac_tool.py | 15 +- gyp/pylib/gyp/msvs_emulation.py | 199 +++++++------ gyp/pylib/gyp/ninja_syntax.py | 6 +- gyp/pylib/gyp/simple_copy.py | 5 +- gyp/pylib/gyp/win_tool.py | 32 +-- gyp/pylib/gyp/xcode_emulation.py | 29 +- gyp/pylib/gyp/xcode_ninja.py | 8 +- gyp/pylib/gyp/xcodeproj_file.py | 32 +-- gyp/pylib/gyp/xml_fix.py | 4 +- gyp/setup.py | 8 +- gyp/test_gyp.py | 9 +- gyp/tools/README | 2 +- gyp/tools/graphviz.py | 7 +- gyp/tools/pretty_gyp.py | 3 +- gyp/tools/pretty_sln.py | 1 - gyp/tools/pretty_vcproj.py | 30 +- 44 files changed, 639 insertions(+), 762 deletions(-) diff --git a/gyp/.github/workflows/Python_tests.yml b/gyp/.github/workflows/Python_tests.yml index 128654f312..649251c8dd 100644 --- a/gyp/.github/workflows/Python_tests.yml +++ b/gyp/.github/workflows/Python_tests.yml @@ -1,5 +1,4 @@ # TODO: Enable os: windows-latest -# TODO: Enable python-version: 3.5 # TODO: Enable pytest --doctest-modules name: Python_tests @@ -9,10 +8,10 @@ jobs: runs-on: ${{ matrix.os }} strategy: fail-fast: false - max-parallel: 15 + max-parallel: 8 matrix: os: [macos-latest, ubuntu-latest] # , windows-latest] - python-version: [2.7, 3.6, 3.7, 3.8, 3.9] + python-version: [3.6, 3.7, 3.8, 3.9] steps: - uses: actions/checkout@v2 - name: Set up Python ${{ matrix.python-version }} diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md index 7940367695..52a07a29ac 100644 --- a/gyp/CHANGELOG.md +++ b/gyp/CHANGELOG.md @@ -1,5 +1,21 @@ # Changelog +## [0.8.0](https://www.github.com/nodejs/gyp-next/compare/v0.7.0...v0.8.0) (2021-01-15) + + +### ⚠ BREAKING CHANGES + +* remove support for Python 2 + +### Bug Fixes + +* revert posix build job ([#86](https://www.github.com/nodejs/gyp-next/issues/86)) ([39dc34f](https://www.github.com/nodejs/gyp-next/commit/39dc34f0799c074624005fb9bbccf6e028607f9d)) + + +### gyp + +* Remove support for Python 2 ([#88](https://www.github.com/nodejs/gyp-next/issues/88)) ([22e4654](https://www.github.com/nodejs/gyp-next/commit/22e465426fd892403c95534229af819a99c3f8dc)) + ## [0.7.0](https://www.github.com/nodejs/gyp-next/compare/v0.6.2...v0.7.0) (2020-12-17) diff --git a/gyp/gyp_main.py b/gyp/gyp_main.py index da696cfc4b..f1559089d1 100755 --- a/gyp/gyp_main.py +++ b/gyp/gyp_main.py @@ -8,8 +8,6 @@ import sys import subprocess -PY3 = bytes != str - def IsCygwin(): # Function copied from pylib/gyp/common.py @@ -17,10 +15,8 @@ def IsCygwin(): out = subprocess.Popen( "uname", stdout=subprocess.PIPE, stderr=subprocess.STDOUT ) - stdout, stderr = out.communicate() - if PY3: - stdout = stdout.decode("utf-8") - return "CYGWIN" in str(stdout) + stdout, _ = out.communicate() + return "CYGWIN" in stdout.decode("utf-8") except Exception: return False @@ -33,9 +29,7 @@ def UnixifyPath(path): ["cygpath", "-u", path], stdout=subprocess.PIPE, stderr=subprocess.STDOUT ) stdout, _ = out.communicate() - if PY3: - stdout = stdout.decode("utf-8") - return str(stdout) + return stdout.decode("utf-8") except Exception: return path diff --git a/gyp/pylib/gyp/MSVSNew.py b/gyp/pylib/gyp/MSVSNew.py index 04bbb3df71..d6b189760c 100644 --- a/gyp/pylib/gyp/MSVSNew.py +++ b/gyp/pylib/gyp/MSVSNew.py @@ -11,12 +11,9 @@ import gyp.common -try: - cmp -except NameError: - def cmp(x, y): - return (x > y) - (x < y) +def cmp(x, y): + return (x > y) - (x < y) # Initialize random number generator @@ -69,7 +66,7 @@ def MakeGuid(name, seed="msvs_new"): # ------------------------------------------------------------------------------ -class MSVSSolutionEntry(object): +class MSVSSolutionEntry: def __cmp__(self, other): # Sort by name then guid (so things are in order on vs2008). return cmp((self.name, self.get_guid()), (other.name, other.get_guid())) @@ -190,7 +187,7 @@ def set_msbuild_toolset(self, msbuild_toolset): # ------------------------------------------------------------------------------ -class MSVSSolution(object): +class MSVSSolution: """Visual Studio solution.""" def __init__( @@ -292,14 +289,14 @@ def Write(self, writer=gyp.common.WriteOnDiff): if e.items: f.write("\tProjectSection(SolutionItems) = preProject\r\n") for i in e.items: - f.write("\t\t%s = %s\r\n" % (i, i)) + f.write(f"\t\t{i} = {i}\r\n") f.write("\tEndProjectSection\r\n") if isinstance(e, MSVSProject): if e.dependencies: f.write("\tProjectSection(ProjectDependencies) = postProject\r\n") for d in e.dependencies: - f.write("\t\t%s = %s\r\n" % (d.get_guid(), d.get_guid())) + f.write(f"\t\t{d.get_guid()} = {d.get_guid()}\r\n") f.write("\tEndProjectSection\r\n") f.write("EndProject\r\n") @@ -310,7 +307,7 @@ def Write(self, writer=gyp.common.WriteOnDiff): # Configurations (variants) f.write("\tGlobalSection(SolutionConfigurationPlatforms) = preSolution\r\n") for v in self.variants: - f.write("\t\t%s = %s\r\n" % (v, v)) + f.write(f"\t\t{v} = {v}\r\n") f.write("\tEndGlobalSection\r\n") # Sort config guids for easier diffing of solution changes. @@ -362,7 +359,7 @@ def Write(self, writer=gyp.common.WriteOnDiff): if not isinstance(e, MSVSFolder): continue # Does not apply to projects, only folders for subentry in e.entries: - f.write("\t\t%s = %s\r\n" % (subentry.get_guid(), e.get_guid())) + f.write(f"\t\t{subentry.get_guid()} = {e.get_guid()}\r\n") f.write("\tEndGlobalSection\r\n") f.write("EndGlobal\r\n") diff --git a/gyp/pylib/gyp/MSVSProject.py b/gyp/pylib/gyp/MSVSProject.py index f953d52cd0..f0cfabe834 100644 --- a/gyp/pylib/gyp/MSVSProject.py +++ b/gyp/pylib/gyp/MSVSProject.py @@ -9,7 +9,7 @@ # ------------------------------------------------------------------------------ -class Tool(object): +class Tool: """Visual Studio tool.""" def __init__(self, name, attrs=None): @@ -31,7 +31,7 @@ def _GetSpecification(self): return ["Tool", self._attrs] -class Filter(object): +class Filter: """Visual Studio filter - that is, a virtual folder.""" def __init__(self, name, contents=None): @@ -48,7 +48,7 @@ def __init__(self, name, contents=None): # ------------------------------------------------------------------------------ -class Writer(object): +class Writer: """Visual Studio XML project writer.""" def __init__(self, project_path, version, name, guid=None, platforms=None): diff --git a/gyp/pylib/gyp/MSVSSettings.py b/gyp/pylib/gyp/MSVSSettings.py index 6ef16f2a0b..e89a971a3b 100644 --- a/gyp/pylib/gyp/MSVSSettings.py +++ b/gyp/pylib/gyp/MSVSSettings.py @@ -14,12 +14,8 @@ MSBuild install directory, e.g. c:\Program Files (x86)\MSBuild """ -from __future__ import print_function - -from gyp import string_types - -import sys import re +import sys # Dictionaries of settings validators. The key is the tool name, the value is # a dictionary mapping setting names to validation functions. @@ -36,7 +32,7 @@ _msbuild_name_of_tool = {} -class _Tool(object): +class _Tool: """Represents a tool used by MSVS or MSBuild. Attributes: @@ -68,7 +64,7 @@ def _GetMSBuildToolSettings(msbuild_settings, tool): return msbuild_settings.setdefault(tool.msbuild_name, {}) -class _Type(object): +class _Type: """Type of settings (Base class).""" def ValidateMSVS(self, value): @@ -110,11 +106,11 @@ class _String(_Type): """A setting that's just a string.""" def ValidateMSVS(self, value): - if not isinstance(value, string_types): + if not isinstance(value, str): raise ValueError("expected string; got %r" % value) def ValidateMSBuild(self, value): - if not isinstance(value, string_types): + if not isinstance(value, str): raise ValueError("expected string; got %r" % value) def ConvertToMSBuild(self, value): @@ -126,11 +122,11 @@ class _StringList(_Type): """A settings that's a list of strings.""" def ValidateMSVS(self, value): - if not isinstance(value, string_types) and not isinstance(value, list): + if not isinstance(value, (list, str)): raise ValueError("expected string list; got %r" % value) def ValidateMSBuild(self, value): - if not isinstance(value, string_types) and not isinstance(value, list): + if not isinstance(value, (list, str)): raise ValueError("expected string list; got %r" % value) def ConvertToMSBuild(self, value): @@ -195,7 +191,7 @@ class _Enumeration(_Type): def __init__(self, label_list, new=None): _Type.__init__(self) self._label_list = label_list - self._msbuild_values = set(value for value in label_list if value is not None) + self._msbuild_values = {value for value in label_list if value is not None} if new is not None: self._msbuild_values.update(new) @@ -342,7 +338,7 @@ def _Translate(value, msbuild_settings): if value == "true": tool_settings = _GetMSBuildToolSettings(msbuild_settings, tool) if "AdditionalOptions" in tool_settings: - new_flags = "%s %s" % (tool_settings["AdditionalOptions"], flag) + new_flags = "{} {}".format(tool_settings["AdditionalOptions"], flag) else: new_flags = flag tool_settings["AdditionalOptions"] = new_flags @@ -536,14 +532,14 @@ def _ValidateSettings(validators, settings, stderr): tool_validators[setting](value) except ValueError as e: print( - "Warning: for %s/%s, %s" % (tool_name, setting, e), + f"Warning: for {tool_name}/{setting}, {e}", file=stderr, ) else: _ValidateExclusionSetting( setting, tool_validators, - ("Warning: unrecognized setting %s/%s" % (tool_name, setting)), + (f"Warning: unrecognized setting {tool_name}/{setting}"), stderr, ) diff --git a/gyp/pylib/gyp/MSVSSettings_test.py b/gyp/pylib/gyp/MSVSSettings_test.py index 99860c880e..7665c68e52 100755 --- a/gyp/pylib/gyp/MSVSSettings_test.py +++ b/gyp/pylib/gyp/MSVSSettings_test.py @@ -9,10 +9,7 @@ import unittest import gyp.MSVSSettings as MSVSSettings -try: - from StringIO import StringIO # Python 2 -except ImportError: - from io import StringIO # Python 3 +from io import StringIO class TestSequenceFunctions(unittest.TestCase): diff --git a/gyp/pylib/gyp/MSVSToolFile.py b/gyp/pylib/gyp/MSVSToolFile.py index 2c08589e06..2e5c811bdd 100644 --- a/gyp/pylib/gyp/MSVSToolFile.py +++ b/gyp/pylib/gyp/MSVSToolFile.py @@ -7,7 +7,7 @@ import gyp.easy_xml as easy_xml -class Writer(object): +class Writer: """Visual Studio XML tool file writer.""" def __init__(self, tool_file_path, name): diff --git a/gyp/pylib/gyp/MSVSUserFile.py b/gyp/pylib/gyp/MSVSUserFile.py index de0896e693..e580c00fb7 100644 --- a/gyp/pylib/gyp/MSVSUserFile.py +++ b/gyp/pylib/gyp/MSVSUserFile.py @@ -53,7 +53,7 @@ def _QuoteWin32CommandLineArgs(args): return new_args -class Writer(object): +class Writer: """Visual Studio XML user user file writer.""" def __init__(self, user_file_path, version, name): @@ -93,7 +93,7 @@ def AddDebugSettings( abs_command = _FindCommandInPath(command[0]) if environment and isinstance(environment, dict): - env_list = ['%s="%s"' % (key, val) for (key, val) in environment.items()] + env_list = [f'{key}="{val}"' for (key, val) in environment.items()] environment = " ".join(env_list) else: environment = "" diff --git a/gyp/pylib/gyp/MSVSUtil.py b/gyp/pylib/gyp/MSVSUtil.py index 83a9c297ed..cb55305eae 100644 --- a/gyp/pylib/gyp/MSVSUtil.py +++ b/gyp/pylib/gyp/MSVSUtil.py @@ -55,7 +55,7 @@ def _SuffixName(name, suffix): Target name with suffix added (foo_suffix#target) """ parts = name.rsplit("#", 1) - parts[0] = "%s_%s" % (parts[0], suffix) + parts[0] = "{}_{}".format(parts[0], suffix) return "#".join(parts) @@ -160,7 +160,7 @@ def _GetPdbPath(target_dict, config_name, vars): return pdb_path pdb_base = target_dict.get("product_name", target_dict["target_name"]) - pdb_base = "%s.%s.pdb" % (pdb_base, TARGET_TYPE_EXT[target_dict["type"]]) + pdb_base = "{}.{}.pdb".format(pdb_base, TARGET_TYPE_EXT[target_dict["type"]]) pdb_path = vars["PRODUCT_DIR"] + "/" + pdb_base return pdb_path diff --git a/gyp/pylib/gyp/MSVSVersion.py b/gyp/pylib/gyp/MSVSVersion.py index 36b006aaa9..134b35557b 100644 --- a/gyp/pylib/gyp/MSVSVersion.py +++ b/gyp/pylib/gyp/MSVSVersion.py @@ -11,14 +11,12 @@ import sys import glob -PY3 = bytes != str - def JoinPath(*args): return os.path.normpath(os.path.join(*args)) -class VisualStudioVersion(object): +class VisualStudioVersion: """Information regarding a version of Visual Studio.""" def __init__( @@ -176,9 +174,7 @@ def _RegistryQueryBase(sysdir, key, value): p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) # Obtain the stdout from reg.exe, reading to the end so p.returncode is valid # Note that the error text may be in [1] in some cases - text = p.communicate()[0] - if PY3: - text = text.decode("utf-8") + text = p.communicate()[0].decode("utf-8") # Check return code from reg.exe; officially 0==success and 1==error if p.returncode: return None @@ -221,21 +217,15 @@ def _RegistryGetValueUsingWinReg(key, value): value: The particular registry value to read. Return: contents of the registry key's value, or None on failure. Throws - ImportError if _winreg is unavailable. + ImportError if winreg is unavailable. """ - try: - # Python 2 - from _winreg import HKEY_LOCAL_MACHINE, OpenKey, QueryValueEx - except ImportError: - # Python 3 - from winreg import HKEY_LOCAL_MACHINE, OpenKey, QueryValueEx - + from winreg import HKEY_LOCAL_MACHINE, OpenKey, QueryValueEx try: root, subkey = key.split("\\", 1) assert root == "HKLM" # Only need HKLM for now. with OpenKey(HKEY_LOCAL_MACHINE, subkey) as hkey: return QueryValueEx(hkey, value)[0] - except WindowsError: + except OSError: return None @@ -426,9 +416,7 @@ def _ConvertToCygpath(path): """Convert to cygwin path if we are using cygwin.""" if sys.platform == "cygwin": p = subprocess.Popen(["cygpath", path], stdout=subprocess.PIPE) - path = p.communicate()[0].strip() - if PY3: - path = path.decode("utf-8") + path = p.communicate()[0].decode("utf-8").strip() return path diff --git a/gyp/pylib/gyp/__init__.py b/gyp/pylib/gyp/__init__.py index f6ea625d40..52549fff11 100755 --- a/gyp/pylib/gyp/__init__.py +++ b/gyp/pylib/gyp/__init__.py @@ -4,7 +4,6 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. -from __future__ import print_function import copy import gyp.input @@ -16,13 +15,6 @@ import traceback from gyp.common import GypError -try: - # Python 2 - string_types = basestring -except NameError: - # Python 3 - string_types = str - # Default debug modes for GYP debug = {} @@ -193,7 +185,7 @@ def ShlexEnv(env_name): def FormatOpt(opt, value): if opt.startswith("--"): - return "%s=%s" % (opt, value) + return f"{opt}={value}" return opt + value @@ -524,7 +516,7 @@ def gyp_main(args): for option, value in sorted(options.__dict__.items()): if option[0] == "_": continue - if isinstance(value, string_types): + if isinstance(value, str): DebugOutput(DEBUG_GENERAL, " %s: '%s'", option, value) else: DebugOutput(DEBUG_GENERAL, " %s: %s", option, value) diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py index a915643867..ba310ce247 100644 --- a/gyp/pylib/gyp/common.py +++ b/gyp/pylib/gyp/common.py @@ -10,17 +10,12 @@ import sys import subprocess -try: - from collections.abc import MutableSet -except ImportError: - from collections import MutableSet - -PY3 = bytes != str +from collections.abc import MutableSet # A minimal memoizing decorator. It'll blow up if the args aren't immutable, # among other "problems". -class memoize(object): +class memoize: def __init__(self, func): self.func = func self.cache = {} @@ -348,7 +343,7 @@ def WriteOnDiff(filename): the target if it differs (on close). """ - class Writer(object): + class Writer: """Wrapper around file which only covers the target if it differs.""" def __init__(self): @@ -566,8 +561,8 @@ def pop(self, last=True): # pylint: disable=W0221 def __repr__(self): if not self: - return "%s()" % (self.__class__.__name__,) - return "%s(%r)" % (self.__class__.__name__, list(self)) + return f"{self.__class__.__name__}()" + return "{}({!r})".format(self.__class__.__name__, list(self)) def __eq__(self, other): if isinstance(other, OrderedSet): @@ -653,9 +648,7 @@ def IsCygwin(): out = subprocess.Popen( "uname", stdout=subprocess.PIPE, stderr=subprocess.STDOUT ) - stdout, stderr = out.communicate() - if PY3: - stdout = stdout.decode("utf-8") + stdout = out.communicate()[0].decode("utf-8") return "CYGWIN" in str(stdout) except Exception: return False diff --git a/gyp/pylib/gyp/easy_xml.py b/gyp/pylib/gyp/easy_xml.py index e0628ef4d8..e475b5530c 100644 --- a/gyp/pylib/gyp/easy_xml.py +++ b/gyp/pylib/gyp/easy_xml.py @@ -84,7 +84,7 @@ def _ConstructContentList(xml_parts, specification, pretty, level=0): rest = specification[1:] if rest and isinstance(rest[0], dict): for at, val in sorted(rest[0].items()): - xml_parts.append(' %s="%s"' % (at, _XmlEscape(val, attr=True))) + xml_parts.append(' {}="{}"'.format(at, _XmlEscape(val, attr=True))) rest = rest[1:] if rest: xml_parts.append(">") @@ -101,7 +101,7 @@ def _ConstructContentList(xml_parts, specification, pretty, level=0): _ConstructContentList(xml_parts, child_spec, pretty, level + 1) if multi_line and indentation: xml_parts.append(indentation) - xml_parts.append("%s" % (name, new_line)) + xml_parts.append(f"{new_line}") else: xml_parts.append("/>%s" % new_line) @@ -125,9 +125,9 @@ def WriteXmlIfChanged(content, path, encoding="utf-8", pretty=False, win32=False # Get the old content try: - with open(path, "r") as file: + with open(path) as file: existing = file.read() - except IOError: + except OSError: existing = None # It has changed, write it diff --git a/gyp/pylib/gyp/easy_xml_test.py b/gyp/pylib/gyp/easy_xml_test.py index 5bc795ddbe..c21e054246 100755 --- a/gyp/pylib/gyp/easy_xml_test.py +++ b/gyp/pylib/gyp/easy_xml_test.py @@ -9,10 +9,7 @@ import gyp.easy_xml as easy_xml import unittest -try: - from StringIO import StringIO # Python 2 -except ImportError: - from io import StringIO # Python 3 +from io import StringIO class TestSequenceFunctions(unittest.TestCase): diff --git a/gyp/pylib/gyp/flock_tool.py b/gyp/pylib/gyp/flock_tool.py index f9f89e520a..c649a89ef8 100755 --- a/gyp/pylib/gyp/flock_tool.py +++ b/gyp/pylib/gyp/flock_tool.py @@ -18,7 +18,7 @@ def main(args): executor.Dispatch(args) -class FlockTool(object): +class FlockTool: """This class emulates the 'flock' command.""" def Dispatch(self, args): diff --git a/gyp/pylib/gyp/generator/analyzer.py b/gyp/pylib/gyp/generator/analyzer.py index 7a393c1f93..f15df00c36 100644 --- a/gyp/pylib/gyp/generator/analyzer.py +++ b/gyp/pylib/gyp/generator/analyzer.py @@ -62,7 +62,6 @@ then the "all" target includes "b1" and "b2". """ -from __future__ import print_function import gyp.common import json @@ -216,7 +215,7 @@ def _ExtractSources(target, target_dict, toplevel_dir): return results -class Target(object): +class Target: """Holds information about a particular target: deps: set of Targets this Target depends upon. This is not recursive, only the direct dependent Targets. @@ -252,7 +251,7 @@ def __init__(self, name): self.is_or_has_linked_ancestor = False -class Config(object): +class Config: """Details what we're looking for files: set of files to search for targets: see file description for details.""" @@ -271,10 +270,10 @@ def Init(self, params): if not config_path: return try: - f = open(config_path, "r") + f = open(config_path) config = json.load(f) f.close() - except IOError: + except OSError: raise Exception("Unable to open file " + config_path) except ValueError as e: raise Exception("Unable to parse config file " + config_path + str(e)) @@ -586,7 +585,7 @@ def _WriteOutput(params, **values): f = open(output_path, "w") f.write(json.dumps(values) + "\n") f.close() - except IOError as e: + except OSError as e: print("Error writing to output file", output_path, str(e)) @@ -627,7 +626,7 @@ def CalculateVariables(default_variables, params): default_variables.setdefault("OS", operating_system) -class TargetCalculator(object): +class TargetCalculator: """Calculates the matching test_targets and matching compile_targets.""" def __init__( diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py index 16728847c5..040d8088a2 100644 --- a/gyp/pylib/gyp/generator/android.py +++ b/gyp/pylib/gyp/generator/android.py @@ -14,7 +14,6 @@ # variables set potentially clash with other Android build system variables. # Try to avoid setting global variables where possible. -from __future__ import print_function import gyp import gyp.common @@ -84,7 +83,7 @@ def IsCPPExtension(ext): def Sourceify(path): """Convert a path to its source directory form. The Android backend does not - support options.generator_output, so this function is a noop.""" + support options.generator_output, so this function is a noop.""" return path @@ -100,11 +99,11 @@ def Sourceify(path): target_link_deps = {} -class AndroidMkWriter(object): +class AndroidMkWriter: """AndroidMkWriter packages up the writing of one target-specific Android.mk. - Its only real entry point is Write(), and is mostly used for namespacing. - """ + Its only real entry point is Write(), and is mostly used for namespacing. + """ def __init__(self, android_top_dir): self.android_top_dir = android_top_dir @@ -123,18 +122,18 @@ def Write( ): """The main entry point: writes a .mk file for a single target. - Arguments: - qualified_target: target we're generating - relative_target: qualified target name relative to the root - base_path: path relative to source root we're building in, used to resolve - target-relative paths - output_filename: output .mk file name to write - spec, configs: gyp info - part_of_all: flag indicating this target is part of 'all' - write_alias_target: flag indicating whether to create short aliases for - this target - sdk_version: what to emit for LOCAL_SDK_VERSION in output - """ + Arguments: + qualified_target: target we're generating + relative_target: qualified target name relative to the root + base_path: path relative to source root we're building in, used to resolve + target-relative paths + output_filename: output .mk file name to write + spec, configs: gyp info + part_of_all: flag indicating this target is part of 'all' + write_alias_target: flag indicating whether to create short aliases for + this target + sdk_version: what to emit for LOCAL_SDK_VERSION in output + """ gyp.common.EnsureDirExists(output_filename) self.fp = open(output_filename, "w") @@ -254,15 +253,15 @@ def Write( def WriteActions(self, actions, extra_sources, extra_outputs): """Write Makefile code for any 'actions' from the gyp input. - extra_sources: a list that will be filled in with newly generated source - files, if any - extra_outputs: a list that will be filled in with any outputs of these - actions (used to make other pieces dependent on these - actions) - """ + extra_sources: a list that will be filled in with newly generated source + files, if any + extra_outputs: a list that will be filled in with any outputs of these + actions (used to make other pieces dependent on these + actions) + """ for action in actions: name = make.StringToMakefileVariable( - "%s_%s" % (self.relative_target, action["action_name"]) + "{}_{}".format(self.relative_target, action["action_name"]) ) self.WriteLn('### Rules for action "%s":' % action["action_name"]) inputs = action["inputs"] @@ -350,7 +349,7 @@ def WriteActions(self, actions, extra_sources, extra_outputs): for output in outputs[1:]: # Make each output depend on the main output, with an empty command # to force make to notice that the mtime has changed. - self.WriteLn("%s: %s ;" % (self.LocalPathify(output), main_output)) + self.WriteLn("{}: {} ;".format(self.LocalPathify(output), main_output)) extra_outputs += outputs self.WriteLn() @@ -360,11 +359,11 @@ def WriteActions(self, actions, extra_sources, extra_outputs): def WriteRules(self, rules, extra_sources, extra_outputs): """Write Makefile code for any 'rules' from the gyp input. - extra_sources: a list that will be filled in with newly generated source - files, if any - extra_outputs: a list that will be filled in with any outputs of these - rules (used to make other pieces dependent on these rules) - """ + extra_sources: a list that will be filled in with newly generated source + files, if any + extra_outputs: a list that will be filled in with any outputs of these + rules (used to make other pieces dependent on these rules) + """ if len(rules) == 0: return @@ -372,7 +371,7 @@ def WriteRules(self, rules, extra_sources, extra_outputs): if len(rule.get("rule_sources", [])) == 0: continue name = make.StringToMakefileVariable( - "%s_%s" % (self.relative_target, rule["rule_name"]) + "{}_{}".format(self.relative_target, rule["rule_name"]) ) self.WriteLn('\n### Generated for rule "%s":' % name) self.WriteLn('# "%s":' % rule) @@ -452,7 +451,7 @@ def WriteRules(self, rules, extra_sources, extra_outputs): for output in outputs[1:]: # Make each output depend on the main output, with an empty command # to force make to notice that the mtime has changed. - self.WriteLn("%s: %s ;" % (output, main_output)) + self.WriteLn(f"{output}: {main_output} ;") self.WriteLn() self.WriteLn() @@ -460,9 +459,9 @@ def WriteRules(self, rules, extra_sources, extra_outputs): def WriteCopies(self, copies, extra_outputs): """Write Makefile code for any 'copies' from the gyp input. - extra_outputs: a list that will be filled in with any outputs of this action - (used to make other pieces dependent on this action) - """ + extra_outputs: a list that will be filled in with any outputs of this action + (used to make other pieces dependent on this action) + """ self.WriteLn("### Generated for copy rule.") variable = make.StringToMakefileVariable(self.relative_target + "_copies") @@ -487,25 +486,25 @@ def WriteCopies(self, copies, extra_outputs): self.LocalPathify(os.path.join(copy["destination"], filename)) ) - self.WriteLn( - "%s: %s $(GYP_TARGET_DEPENDENCIES) | $(ACP)" % (output, path) - ) + self.WriteLn(f"{output}: {path} $(GYP_TARGET_DEPENDENCIES) | $(ACP)") self.WriteLn("\t@echo Copying: $@") self.WriteLn("\t$(hide) mkdir -p $(dir $@)") self.WriteLn("\t$(hide) $(ACP) -rpf $< $@") self.WriteLn() outputs.append(output) - self.WriteLn("%s = %s" % (variable, " ".join(map(make.QuoteSpaces, outputs)))) + self.WriteLn( + "{} = {}".format(variable, " ".join(map(make.QuoteSpaces, outputs))) + ) extra_outputs.append("$(%s)" % variable) self.WriteLn() def WriteSourceFlags(self, spec, configs): """Write out the flags and include paths used to compile source files for - the current target. + the current target. - Args: - spec, configs: input from gyp. - """ + Args: + spec, configs: input from gyp. + """ for configname, config in sorted(configs.items()): extracted_includes = [] @@ -554,16 +553,16 @@ def WriteSourceFlags(self, spec, configs): def WriteSources(self, spec, configs, extra_sources): """Write Makefile code for any 'sources' from the gyp input. - These are source files necessary to build the current target. - We need to handle shared_intermediate directory source files as - a special case by copying them to the intermediate directory and - treating them as a generated sources. Otherwise the Android build - rules won't pick them up. - - Args: - spec, configs: input from gyp. - extra_sources: Sources generated from Actions or Rules. - """ + These are source files necessary to build the current target. + We need to handle shared_intermediate directory source files as + a special case by copying them to the intermediate directory and + treating them as a generated sources. Otherwise the Android build + rules won't pick them up. + + Args: + spec, configs: input from gyp. + extra_sources: Sources generated from Actions or Rules. + """ sources = filter(make.Compilable, spec.get("sources", [])) generated_not_sources = [x for x in extra_sources if not make.Compilable(x)] extra_sources = filter(make.Compilable, extra_sources) @@ -617,7 +616,7 @@ def WriteSources(self, spec, configs, extra_sources): if IsCPPExtension(ext) and ext != local_cpp_extension: local_file = root + local_cpp_extension if local_file != source: - self.WriteLn("%s: %s" % (local_file, self.LocalPathify(source))) + self.WriteLn("{}: {}".format(local_file, self.LocalPathify(source))) self.WriteLn("\tmkdir -p $(@D); cp $< $@") origin_src_dirs.append(os.path.dirname(source)) final_generated_sources.append(local_file) @@ -640,10 +639,10 @@ def WriteSources(self, spec, configs, extra_sources): def ComputeAndroidModule(self, spec): """Return the Android module name used for a gyp spec. - We use the complete qualified target name to avoid collisions between - duplicate targets in different directories. We also add a suffix to - distinguish gyp-generated module names. - """ + We use the complete qualified target name to avoid collisions between + duplicate targets in different directories. We also add a suffix to + distinguish gyp-generated module names. + """ if int(spec.get("android_unmangled_name", 0)): assert self.type != "shared_library" or self.target.startswith("lib") @@ -662,7 +661,7 @@ def ComputeAndroidModule(self, spec): suffix = "_gyp" if self.path: - middle = make.StringToMakefileVariable("%s_%s" % (self.path, self.target)) + middle = make.StringToMakefileVariable(f"{self.path}_{self.target}") else: middle = make.StringToMakefileVariable(self.target) @@ -671,11 +670,11 @@ def ComputeAndroidModule(self, spec): def ComputeOutputParts(self, spec): """Return the 'output basename' of a gyp spec, split into filename + ext. - Android libraries must be named the same thing as their module name, - otherwise the linker can't find them, so product_name and so on must be - ignored if we are building a library, and the "lib" prepending is - not done for Android. - """ + Android libraries must be named the same thing as their module name, + otherwise the linker can't find them, so product_name and so on must be + ignored if we are building a library, and the "lib" prepending is + not done for Android. + """ assert self.type != "loadable_module" # TODO: not supported? target = spec["target_name"] @@ -711,17 +710,17 @@ def ComputeOutputParts(self, spec): def ComputeOutputBasename(self, spec): """Return the 'output basename' of a gyp spec. - E.g., the loadable module 'foobar' in directory 'baz' will produce - 'libfoobar.so' - """ + E.g., the loadable module 'foobar' in directory 'baz' will produce + 'libfoobar.so' + """ return "".join(self.ComputeOutputParts(spec)) def ComputeOutput(self, spec): """Return the 'output' (full output path) of a gyp spec. - E.g., the loadable module 'foobar' in directory 'baz' will produce - '$(obj)/baz/libfoobar.so' - """ + E.g., the loadable module 'foobar' in directory 'baz' will produce + '$(obj)/baz/libfoobar.so' + """ if self.type == "executable": # We install host executables into shared_intermediate_dir so they can be # run by gyp rules that refer to PRODUCT_DIR. @@ -740,7 +739,7 @@ def ComputeOutput(self, spec): % (self.android_class, self.android_module) ) else: - path = "$(call intermediates-dir-for,%s,%s,,,$(GYP_VAR_PREFIX))" % ( + path = "$(call intermediates-dir-for,{},{},,,$(GYP_VAR_PREFIX))".format( self.android_class, self.android_module, ) @@ -749,14 +748,14 @@ def ComputeOutput(self, spec): return os.path.join(path, self.ComputeOutputBasename(spec)) def NormalizeIncludePaths(self, include_paths): - """ Normalize include_paths. - Convert absolute paths to relative to the Android top directory. - - Args: - include_paths: A list of unprocessed include paths. - Returns: - A list of normalized include paths. - """ + """Normalize include_paths. + Convert absolute paths to relative to the Android top directory. + + Args: + include_paths: A list of unprocessed include paths. + Returns: + A list of normalized include paths. + """ normalized = [] for path in include_paths: if path[0] == "/": @@ -767,11 +766,11 @@ def NormalizeIncludePaths(self, include_paths): def ExtractIncludesFromCFlags(self, cflags): """Extract includes "-I..." out from cflags - Args: - cflags: A list of compiler flags, which may be mixed with "-I.." - Returns: - A tuple of lists: (clean_clfags, include_paths). "-I.." is trimmed. - """ + Args: + cflags: A list of compiler flags, which may be mixed with "-I.." + Returns: + A tuple of lists: (clean_clfags, include_paths). "-I.." is trimmed. + """ clean_cflags = [] include_paths = [] for flag in cflags: @@ -785,14 +784,14 @@ def ExtractIncludesFromCFlags(self, cflags): def FilterLibraries(self, libraries): """Filter the 'libraries' key to separate things that shouldn't be ldflags. - Library entries that look like filenames should be converted to android - module names instead of being passed to the linker as flags. + Library entries that look like filenames should be converted to android + module names instead of being passed to the linker as flags. - Args: - libraries: the value of spec.get('libraries') - Returns: - A tuple (static_lib_modules, dynamic_lib_modules, ldflags) - """ + Args: + libraries: the value of spec.get('libraries') + Returns: + A tuple (static_lib_modules, dynamic_lib_modules, ldflags) + """ static_lib_modules = [] dynamic_lib_modules = [] ldflags = [] @@ -823,10 +822,10 @@ def FilterLibraries(self, libraries): def ComputeDeps(self, spec): """Compute the dependencies of a gyp spec. - Returns a tuple (deps, link_deps), where each is a list of - filenames that will need to be put in front of make for either - building (deps) or linking (link_deps). - """ + Returns a tuple (deps, link_deps), where each is a list of + filenames that will need to be put in front of make for either + building (deps) or linking (link_deps). + """ deps = [] link_deps = [] if "dependencies" in spec: @@ -846,9 +845,9 @@ def ComputeDeps(self, spec): def WriteTargetFlags(self, spec, configs, link_deps): """Write Makefile code to specify the link flags and library dependencies. - spec, configs: input from gyp. - link_deps: link dependency list; see ComputeDeps() - """ + spec, configs: input from gyp. + link_deps: link dependency list; see ComputeDeps() + """ # Libraries (i.e. -lfoo) # These must be included even for static libraries as some of them provide # implicit include paths through the build system. @@ -891,12 +890,12 @@ def WriteTarget( ): """Write Makefile code to produce the final target of the gyp spec. - spec, configs: input from gyp. - deps, link_deps: dependency lists; see ComputeDeps() - part_of_all: flag indicating this target is part of 'all' - write_alias_target: flag indicating whether to create short aliases for this - target - """ + spec, configs: input from gyp. + deps, link_deps: dependency lists; see ComputeDeps() + part_of_all: flag indicating this target is part of 'all' + write_alias_target: flag indicating whether to create short aliases for this + target + """ self.WriteLn("### Rules for final target.") if self.type != "none": @@ -909,7 +908,7 @@ def WriteTarget( if isinstance(v, list): self.WriteList(v, k) else: - self.WriteLn("%s := %s" % (k, make.QuoteIfNecessary(v))) + self.WriteLn("{} := {}".format(k, make.QuoteIfNecessary(v))) self.WriteLn("") # Add to the set of targets which represent the gyp 'all' target. We use the @@ -928,7 +927,7 @@ def WriteTarget( if self.target != self.android_module and write_alias_target: self.WriteLn("# Alias gyp target name.") self.WriteLn(".PHONY: %s" % self.target) - self.WriteLn("%s: %s" % (self.target, self.android_module)) + self.WriteLn(f"{self.target}: {self.android_module}") self.WriteLn("") # Add the command to trigger build of the target type depending @@ -975,25 +974,25 @@ def WriteList( ): """Write a variable definition that is a list of values. - E.g. WriteList(['a','b'], 'foo', prefix='blah') writes out - foo = blaha blahb - but in a pretty-printed style. - """ + E.g. WriteList(['a','b'], 'foo', prefix='blah') writes out + foo = blaha blahb + but in a pretty-printed style. + """ values = "" if value_list: value_list = [quoter(prefix + value) for value in value_list] if local_pathify: value_list = [self.LocalPathify(value) for value in value_list] values = " \\\n\t" + " \\\n\t".join(value_list) - self.fp.write("%s :=%s\n\n" % (variable, values)) + self.fp.write(f"{variable} :={values}\n\n") def WriteLn(self, text=""): self.fp.write(text + "\n") def LocalPathify(self, path): """Convert a subdirectory-relative path into a normalized path which starts - with the make variable $(LOCAL_PATH) (i.e. the top of the project tree). - Absolute paths, or paths that contain variables, are just normalized.""" + with the make variable $(LOCAL_PATH) (i.e. the top of the project tree). + Absolute paths, or paths that contain variables, are just normalized.""" if "$(" in path or os.path.isabs(path): # path is not a file in the project tree in this case, but calling # normpath is still important for trimming trailing slashes. @@ -1006,7 +1005,7 @@ def LocalPathify(self, path): # so we don't look for a slash. assert local_path.startswith( "$(LOCAL_PATH)" - ), "Path %s attempts to escape from gyp path %s !)" % (path, self.path) + ), f"Path {path} attempts to escape from gyp path {self.path} !)" return local_path def ExpandInputRoot(self, template, expansion, dirname): diff --git a/gyp/pylib/gyp/generator/cmake.py b/gyp/pylib/gyp/generator/cmake.py index 75f5822369..c95d18415c 100644 --- a/gyp/pylib/gyp/generator/cmake.py +++ b/gyp/pylib/gyp/generator/cmake.py @@ -28,21 +28,15 @@ CMakeLists.txt file. """ -from __future__ import print_function import multiprocessing import os import signal -import string import subprocess import gyp.common import gyp.xcode_emulation -try: - # maketrans moved to str in python3. - _maketrans = string.maketrans -except (NameError, AttributeError): - _maketrans = str.maketrans +_maketrans = str.maketrans generator_default_variables = { "EXECUTABLE_PREFIX": "", @@ -223,7 +217,7 @@ def WriteVariable(output, variable_name, prepend=None): output.write("}") -class CMakeTargetType(object): +class CMakeTargetType: def __init__(self, command, modifier, property_modifier): self.command = command self.modifier = modifier @@ -263,7 +257,7 @@ def WriteActions(target_name, actions, extra_sources, extra_deps, path_to_gyp, o """ for action in actions: action_name = StringToCMakeTargetName(action["action_name"]) - action_target_name = "%s__%s" % (target_name, action_name) + action_target_name = f"{target_name}__{action_name}" inputs = action["inputs"] inputs_name = action_target_name + "__input" @@ -282,7 +276,7 @@ def WriteActions(target_name, actions, extra_sources, extra_deps, path_to_gyp, o # Build up a list of outputs. # Collect the output dirs we'll need. - dirs = set(dir for dir in (os.path.dirname(o) for o in outputs) if dir) + dirs = {dir for dir in (os.path.dirname(o) for o in outputs) if dir} if int(action.get("process_outputs_as_sources", False)): extra_sources.extend(zip(cmake_outputs, outputs)) @@ -377,7 +371,7 @@ def WriteRules(target_name, rules, extra_sources, extra_deps, path_to_gyp, outpu # Build up a list of outputs. # Collect the output dirs we'll need. - dirs = set(dir for dir in (os.path.dirname(o) for o in outputs) if dir) + dirs = {dir for dir in (os.path.dirname(o) for o in outputs) if dir} # Create variables for the output, as 'local' variable will be unset. these_outputs = [] @@ -478,7 +472,7 @@ def WriteCopies(target_name, copies, extra_deps, path_to_gyp, output): extra_deps.append(copy_name) return - class Copy(object): + class Copy: def __init__(self, ext, command): self.cmake_inputs = [] self.cmake_outputs = [] @@ -587,7 +581,7 @@ def CreateCMakeTargetFullName(qualified_target): return StringToCMakeTargetName(cmake_target_full_name) -class CMakeNamer(object): +class CMakeNamer: """Converts Gyp target names into CMake target names. CMake requires that target names be globally unique. One way to ensure @@ -1285,11 +1279,11 @@ def PerformBuild(data, configurations, params): os.path.join(generator_dir, output_dir, config_name) ) arguments = ["cmake", "-G", "Ninja"] - print("Generating [%s]: %s" % (config_name, arguments)) + print(f"Generating [{config_name}]: {arguments}") subprocess.check_call(arguments, cwd=build_dir) arguments = ["ninja", "-C", build_dir] - print("Building [%s]: %s" % (config_name, arguments)) + print(f"Building [{config_name}]: {arguments}") subprocess.check_call(arguments) diff --git a/gyp/pylib/gyp/generator/dump_dependency_json.py b/gyp/pylib/gyp/generator/dump_dependency_json.py index 46f68e0384..99d5c1fd69 100644 --- a/gyp/pylib/gyp/generator/dump_dependency_json.py +++ b/gyp/pylib/gyp/generator/dump_dependency_json.py @@ -2,7 +2,6 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. -from __future__ import print_function import os import gyp diff --git a/gyp/pylib/gyp/generator/eclipse.py b/gyp/pylib/gyp/generator/eclipse.py index 4bd49725dc..1ff0dc83ae 100644 --- a/gyp/pylib/gyp/generator/eclipse.py +++ b/gyp/pylib/gyp/generator/eclipse.py @@ -26,8 +26,6 @@ import shlex import xml.etree.cElementTree as ET -PY3 = bytes != str - generator_wants_static_library_dependencies_adjusted = False generator_default_variables = {} @@ -105,9 +103,7 @@ def GetAllIncludeDirectories( stdout=subprocess.PIPE, stderr=subprocess.PIPE, ) - output = proc.communicate()[1] - if PY3: - output = output.decode("utf-8") + output = proc.communicate()[1].decode("utf-8") # Extract the list of include dirs from the output, which has this format: # ... # #include "..." search starts here: @@ -245,9 +241,7 @@ def GetAllDefines(target_list, target_dicts, data, config_name, params, compiler cpp_proc = subprocess.Popen( args=command, cwd=".", stdin=subprocess.PIPE, stdout=subprocess.PIPE ) - cpp_output = cpp_proc.communicate()[0] - if PY3: - cpp_output = cpp_output.decode("utf-8") + cpp_output = cpp_proc.communicate()[0].decode("utf-8") cpp_lines = cpp_output.split("\n") for cpp_line in cpp_lines: if not cpp_line.strip(): diff --git a/gyp/pylib/gyp/generator/gypsh.py b/gyp/pylib/gyp/generator/gypsh.py index 2d8aba5d1c..82a07ddc65 100644 --- a/gyp/pylib/gyp/generator/gypsh.py +++ b/gyp/pylib/gyp/generator/gypsh.py @@ -49,7 +49,7 @@ def GenerateOutput(target_list, target_dicts, data, params): # Use a banner that looks like the stock Python one and like what # code.interact uses by default, but tack on something to indicate what # locals are available, and identify gypsh. - banner = "Python %s on %s\nlocals.keys() = %s\ngypsh" % ( + banner = "Python {} on {}\nlocals.keys() = {}\ngypsh".format( sys.version, sys.platform, repr(sorted(locals.keys())), diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index d163ae3135..857875a564 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -21,7 +21,6 @@ # toplevel Makefile. It may make sense to generate some .mk files on # the side to keep the files readable. -from __future__ import print_function import os import re @@ -108,7 +107,7 @@ def CalculateVariables(default_variables, params): def CalculateGeneratorInputInfo(params): """Calculate the generator specific info that gets fed to input (called by - gyp).""" + gyp).""" generator_flags = params.get("generator_flags", {}) android_ndk_version = generator_flags.get("android_ndk_version", None) # Android NDK requires a strict link order. @@ -615,15 +614,15 @@ def Target(filename): def EscapeShellArgument(s): """Quotes an argument so that it will be interpreted literally by a POSIX - shell. Taken from - http://stackoverflow.com/questions/35817/whats-the-best-way-to-escape-ossystem-calls-in-python - """ + shell. Taken from + http://stackoverflow.com/questions/35817/whats-the-best-way-to-escape-ossystem-calls-in-python + """ return "'" + s.replace("'", "'\\''") + "'" def EscapeMakeVariableExpansion(s): """Make has its own variable expansion syntax using $. We must escape it for - string to be interpreted literally.""" + string to be interpreted literally.""" return s.replace("$", "$$") @@ -638,7 +637,7 @@ def EscapeCppDefine(s): def QuoteIfNecessary(string): """TODO: Should this ideally be replaced with one or more of the above - functions?""" + functions?""" if '"' in string: string = '"' + string.replace('"', '\\"') + '"' return string @@ -679,11 +678,11 @@ def SourceifyAndQuoteSpaces(path): target_link_deps = {} -class MakefileWriter(object): +class MakefileWriter: """MakefileWriter packages up the writing of one target-specific foobar.mk. - Its only real entry point is Write(), and is mostly used for namespacing. - """ + Its only real entry point is Write(), and is mostly used for namespacing. + """ def __init__(self, generator_flags, flavor): self.generator_flags = generator_flags @@ -737,14 +736,14 @@ def Write( ): """The main entry point: writes a .mk file for a single target. - Arguments: - qualified_target: target we're generating - base_path: path relative to source root we're building in, used to resolve - target-relative paths - output_filename: output .mk file name to write - spec, configs: gyp info - part_of_all: flag indicating this target is part of 'all' - """ + Arguments: + qualified_target: target we're generating + base_path: path relative to source root we're building in, used to resolve + target-relative paths + output_filename: output .mk file name to write + spec, configs: gyp info + part_of_all: flag indicating this target is part of 'all' + """ gyp.common.EnsureDirExists(output_filename) self.fp = open(output_filename, "w") @@ -844,7 +843,7 @@ def Write( sources = [x for x in all_sources if Compilable(x)] if sources: self.WriteLn(SHARED_HEADER_SUFFIX_RULES_COMMENT1) - extensions = set([os.path.splitext(s)[1] for s in sources]) + extensions = {os.path.splitext(s)[1] for s in sources} for ext in extensions: if ext in self.suffix_rules_srcdir: self.WriteLn(self.suffix_rules_srcdir[ext]) @@ -888,15 +887,15 @@ def Write( def WriteSubMake(self, output_filename, makefile_path, targets, build_dir): """Write a "sub-project" Makefile. - This is a small, wrapper Makefile that calls the top-level Makefile to build - the targets from a single gyp file (i.e. a sub-project). + This is a small, wrapper Makefile that calls the top-level Makefile to build + the targets from a single gyp file (i.e. a sub-project). - Arguments: - output_filename: sub-project Makefile name to write - makefile_path: path to the top-level Makefile - targets: list of "all" targets for this sub-project - build_dir: build output directory, relative to the sub-project - """ + Arguments: + output_filename: sub-project Makefile name to write + makefile_path: path to the top-level Makefile + targets: list of "all" targets for this sub-project + build_dir: build output directory, relative to the sub-project + """ gyp.common.EnsureDirExists(output_filename) self.fp = open(output_filename, "w") self.fp.write(header) @@ -910,7 +909,7 @@ def WriteSubMake(self, output_filename, makefile_path, targets, build_dir): self.WriteLn("all:") if makefile_path: makefile_path = " -C " + makefile_path - self.WriteLn("\t$(MAKE)%s %s" % (makefile_path, " ".join(targets))) + self.WriteLn("\t$(MAKE){} {}".format(makefile_path, " ".join(targets))) self.fp.close() def WriteActions( @@ -923,17 +922,17 @@ def WriteActions( ): """Write Makefile code for any 'actions' from the gyp input. - extra_sources: a list that will be filled in with newly generated source - files, if any - extra_outputs: a list that will be filled in with any outputs of these - actions (used to make other pieces dependent on these - actions) - part_of_all: flag indicating this target is part of 'all' - """ + extra_sources: a list that will be filled in with newly generated source + files, if any + extra_outputs: a list that will be filled in with any outputs of these + actions (used to make other pieces dependent on these + actions) + part_of_all: flag indicating this target is part of 'all' + """ env = self.GetSortedXcodeEnv() for action in actions: name = StringToMakefileVariable( - "%s_%s" % (self.qualified_target, action["action_name"]) + "{}_{}".format(self.qualified_target, action["action_name"]) ) self.WriteLn('### Rules for action "%s":' % action["action_name"]) inputs = action["inputs"] @@ -960,9 +959,11 @@ def WriteActions( ] command = gyp.common.EncodePOSIXShellList(action_commands) if "message" in action: - self.WriteLn("quiet_cmd_%s = ACTION %s $@" % (name, action["message"])) + self.WriteLn( + "quiet_cmd_{} = ACTION {} $@".format(name, action["message"]) + ) else: - self.WriteLn("quiet_cmd_%s = ACTION %s $@" % (name, name)) + self.WriteLn(f"quiet_cmd_{name} = ACTION {name} $@") if len(dirs) > 0: command = "mkdir -p %s" % " ".join(dirs) + "; " + command @@ -1022,7 +1023,7 @@ def WriteActions( # Stuff the outputs in a variable so we can refer to them later. outputs_variable = "action_%s_outputs" % name - self.WriteLn("%s := %s" % (outputs_variable, " ".join(outputs))) + self.WriteLn("{} := {}".format(outputs_variable, " ".join(outputs))) extra_outputs.append("$(%s)" % outputs_variable) self.WriteLn() @@ -1038,16 +1039,16 @@ def WriteRules( ): """Write Makefile code for any 'rules' from the gyp input. - extra_sources: a list that will be filled in with newly generated source - files, if any - extra_outputs: a list that will be filled in with any outputs of these - rules (used to make other pieces dependent on these rules) - part_of_all: flag indicating this target is part of 'all' - """ + extra_sources: a list that will be filled in with newly generated source + files, if any + extra_outputs: a list that will be filled in with any outputs of these + rules (used to make other pieces dependent on these rules) + part_of_all: flag indicating this target is part of 'all' + """ env = self.GetSortedXcodeEnv() for rule in rules: name = StringToMakefileVariable( - "%s_%s" % (self.qualified_target, rule["rule_name"]) + "{}_{}".format(self.qualified_target, rule["rule_name"]) ) count = 0 self.WriteLn("### Generated for rule %s:" % name) @@ -1175,10 +1176,10 @@ def WriteRules( def WriteCopies(self, copies, extra_outputs, part_of_all): """Write Makefile code for any 'copies' from the gyp input. - extra_outputs: a list that will be filled in with any outputs of this action - (used to make other pieces dependent on this action) - part_of_all: flag indicating this target is part of 'all' - """ + extra_outputs: a list that will be filled in with any outputs of this action + (used to make other pieces dependent on this action) + part_of_all: flag indicating this target is part of 'all' + """ self.WriteLn("### Generated for copy rule.") variable = StringToMakefileVariable(self.qualified_target + "_copies") @@ -1206,7 +1207,9 @@ def WriteCopies(self, copies, extra_outputs, part_of_all): path = gyp.xcode_emulation.ExpandEnvVars(path, env) self.WriteDoCmd([output], [path], "copy", part_of_all) outputs.append(output) - self.WriteLn("%s = %s" % (variable, " ".join(QuoteSpaces(o) for o in outputs))) + self.WriteLn( + "{} = {}".format(variable, " ".join(QuoteSpaces(o) for o in outputs)) + ) extra_outputs.append("$(%s)" % variable) self.WriteLn() @@ -1278,15 +1281,15 @@ def WriteSources( precompiled_header, ): """Write Makefile code for any 'sources' from the gyp input. - These are source files necessary to build the current target. - - configs, deps, sources: input from gyp. - extra_outputs: a list of extra outputs this action should be dependent on; - used to serialize action/rules before compilation - extra_link_deps: a list that will be filled in with any outputs of - compilation (to be used in link lines) - part_of_all: flag indicating this target is part of 'all' - """ + These are source files necessary to build the current target. + + configs, deps, sources: input from gyp. + extra_outputs: a list of extra outputs this action should be dependent on; + used to serialize action/rules before compilation + extra_link_deps: a list that will be filled in with any outputs of + compilation (to be used in link lines) + part_of_all: flag indicating this target is part of 'all' + """ # Write configuration-specific variables for CFLAGS, etc. for configname in sorted(configs.keys()): @@ -1300,8 +1303,7 @@ def WriteSources( if self.flavor == "mac": cflags = self.xcode_settings.GetCflags( - configname, - arch=config.get('xcode_configuration_platform') + configname, arch=config.get("xcode_configuration_platform") ) cflags_c = self.xcode_settings.GetCflagsC(configname) cflags_cc = self.xcode_settings.GetCflagsCC(configname) @@ -1364,7 +1366,7 @@ def WriteSources( if pchdeps: self.WriteLn("# Dependencies from obj files to their precompiled headers") for source, obj, gch in pchdeps: - self.WriteLn("%s: %s" % (obj, gch)) + self.WriteLn(f"{obj}: {gch}") self.WriteLn("# End precompiled header dependencies") if objs: @@ -1436,12 +1438,12 @@ def WritePchTargets(self, pch_commands): "mm": "GYP_PCH_OBJCXXFLAGS", }[lang] self.WriteLn( - "%s: %s := %s " % (gch, var_name, lang_flag) + "$(DEFS_$(BUILDTYPE)) " + f"{gch}: {var_name} := {lang_flag} " + "$(DEFS_$(BUILDTYPE)) " "$(INCS_$(BUILDTYPE)) " "$(CFLAGS_$(BUILDTYPE)) " + extra_flags ) - self.WriteLn("%s: %s FORCE_DO_CMD" % (gch, input)) + self.WriteLn(f"{gch}: {input} FORCE_DO_CMD") self.WriteLn("\t@$(call do_cmd,pch_%s,1)" % lang) self.WriteLn("") assert " " not in gch, "Spaces in gch filenames not supported (%s)" % gch @@ -1451,9 +1453,9 @@ def WritePchTargets(self, pch_commands): def ComputeOutputBasename(self, spec): """Return the 'output basename' of a gyp spec. - E.g., the loadable module 'foobar' in directory 'baz' will produce - 'libfoobar.so' - """ + E.g., the loadable module 'foobar' in directory 'baz' will produce + 'libfoobar.so' + """ assert not self.is_mac_bundle if self.flavor == "mac" and self.type in ( @@ -1510,9 +1512,9 @@ def _InstallImmediately(self): def ComputeOutput(self, spec): """Return the 'output' (full output path) of a gyp spec. - E.g., the loadable module 'foobar' in directory 'baz' will produce - '$(obj)/baz/libfoobar.so' - """ + E.g., the loadable module 'foobar' in directory 'baz' will produce + '$(obj)/baz/libfoobar.so' + """ assert not self.is_mac_bundle path = os.path.join("$(obj)." + self.toolset, self.path) @@ -1535,10 +1537,10 @@ def ComputeMacBundleBinaryOutput(self, spec): def ComputeDeps(self, spec): """Compute the dependencies of a gyp spec. - Returns a tuple (deps, link_deps), where each is a list of - filenames that will need to be put in front of make for either - building (deps) or linking (link_deps). - """ + Returns a tuple (deps, link_deps), where each is a list of + filenames that will need to be put in front of make for either + building (deps) or linking (link_deps). + """ deps = [] link_deps = [] if "dependencies" in spec: @@ -1571,11 +1573,11 @@ def WriteTarget( ): """Write Makefile code to produce the final target of the gyp spec. - spec, configs: input from gyp. - deps, link_deps: dependency lists; see ComputeDeps() - extra_outputs: any extra outputs that our target should depend on - part_of_all: flag indicating this target is part of 'all' - """ + spec, configs: input from gyp. + deps, link_deps: dependency lists; see ComputeDeps() + extra_outputs: any extra outputs that our target should depend on + part_of_all: flag indicating this target is part of 'all' + """ self.WriteLn("### Rules for final target.") @@ -1597,7 +1599,7 @@ def WriteTarget( configname, generator_default_variables["PRODUCT_DIR"], lambda p: Sourceify(self.Absolutify(p)), - arch=config.get('xcode_configuration_platform') + arch=config.get("xcode_configuration_platform"), ) # TARGET_POSTBUILDS_$(BUILDTYPE) is added to postbuilds later on. @@ -1860,7 +1862,7 @@ def WriteTarget( and self.toolset == "target" ): # On mac, products are created in install_path immediately. - assert install_path == self.output, "%s != %s" % ( + assert install_path == self.output, "{} != {}".format( install_path, self.output, ) @@ -1897,24 +1899,24 @@ def WriteTarget( def WriteList(self, value_list, variable=None, prefix="", quoter=QuoteIfNecessary): """Write a variable definition that is a list of values. - E.g. WriteList(['a','b'], 'foo', prefix='blah') writes out - foo = blaha blahb - but in a pretty-printed style. - """ + E.g. WriteList(['a','b'], 'foo', prefix='blah') writes out + foo = blaha blahb + but in a pretty-printed style. + """ values = "" if value_list: value_list = [quoter(prefix + value) for value in value_list] values = " \\\n\t" + " \\\n\t".join(value_list) - self.fp.write("%s :=%s\n\n" % (variable, values)) + self.fp.write(f"{variable} :={values}\n\n") def WriteDoCmd( self, outputs, inputs, command, part_of_all, comment=None, postbuilds=False ): """Write a Makefile rule that uses do_cmd. - This makes the outputs dependent on the command line that was run, - as well as support the V= make command line flag. - """ + This makes the outputs dependent on the command line that was run, + as well as support the V= make command line flag. + """ suffix = "" if postbuilds: assert "," not in command @@ -1922,7 +1924,7 @@ def WriteDoCmd( self.WriteMakeRule( outputs, inputs, - actions=["$(call do_cmd,%s%s)" % (command, suffix)], + actions=[f"$(call do_cmd,{command}{suffix})"], comment=comment, command=command, force=True, @@ -1947,18 +1949,18 @@ def WriteMakeRule( ): """Write a Makefile rule, with some extra tricks. - outputs: a list of outputs for the rule (note: this is not directly - supported by make; see comments below) - inputs: a list of inputs for the rule - actions: a list of shell commands to run for the rule - comment: a comment to put in the Makefile above the rule (also useful - for making this Python script's code self-documenting) - order_only: if true, makes the dependency order-only - force: if true, include FORCE_DO_CMD as an order-only dep - phony: if true, the rule does not actually generate the named output, the - output is just a name to run the rule - command: (optional) command name to generate unambiguous labels - """ + outputs: a list of outputs for the rule (note: this is not directly + supported by make; see comments below) + inputs: a list of inputs for the rule + actions: a list of shell commands to run for the rule + comment: a comment to put in the Makefile above the rule (also useful + for making this Python script's code self-documenting) + order_only: if true, makes the dependency order-only + force: if true, include FORCE_DO_CMD as an order-only dep + phony: if true, the rule does not actually generate the named output, the + output is just a name to run the rule + command: (optional) command name to generate unambiguous labels + """ outputs = [QuoteSpaces(o) for o in outputs] inputs = [QuoteSpaces(i) for i in inputs] @@ -1974,11 +1976,11 @@ def WriteMakeRule( # Order only rule: Just write a simple rule. # TODO(evanm): just make order_only a list of deps instead of this hack. self.WriteLn( - "%s: | %s%s" % (" ".join(outputs), " ".join(inputs), force_append) + "{}: | {}{}".format(" ".join(outputs), " ".join(inputs), force_append) ) elif len(outputs) == 1: # Regular rule, one output: Just write a simple rule. - self.WriteLn("%s: %s%s" % (outputs[0], " ".join(inputs), force_append)) + self.WriteLn("{}: {}{}".format(outputs[0], " ".join(inputs), force_append)) else: # Regular rule, more than one output: Multiple outputs are tricky in # make. We will write three rules: @@ -1994,10 +1996,12 @@ def WriteMakeRule( (command or self.target).encode("utf-8") ).hexdigest() intermediate = "%s.intermediate" % cmddigest - self.WriteLn("%s: %s" % (" ".join(outputs), intermediate)) + self.WriteLn("{}: {}".format(" ".join(outputs), intermediate)) self.WriteLn("\t%s" % "@:") - self.WriteLn("%s: %s" % (".INTERMEDIATE", intermediate)) - self.WriteLn("%s: %s%s" % (intermediate, " ".join(inputs), force_append)) + self.WriteLn("{}: {}".format(".INTERMEDIATE", intermediate)) + self.WriteLn( + "{}: {}{}".format(intermediate, " ".join(inputs), force_append) + ) actions.insert(0, "$(call do_cmd,touch)") if actions: @@ -2008,16 +2012,16 @@ def WriteMakeRule( def WriteAndroidNdkModuleRule(self, module_name, all_sources, link_deps): """Write a set of LOCAL_XXX definitions for Android NDK. - These variable definitions will be used by Android NDK but do nothing for - non-Android applications. + These variable definitions will be used by Android NDK but do nothing for + non-Android applications. - Arguments: - module_name: Android NDK module name, which must be unique among all - module names. - all_sources: A list of source files (will be filtered by Compilable). - link_deps: A list of link dependencies, which must be sorted in - the order from dependencies to dependents. - """ + Arguments: + module_name: Android NDK module name, which must be unique among all + module names. + all_sources: A list of source files (will be filtered by Compilable). + link_deps: A list of link dependencies, which must be sorted in + the order from dependencies to dependents. + """ if self.type not in ("executable", "shared_library", "static_library"): return @@ -2129,14 +2133,14 @@ def WriteSortedXcodeEnv(self, target, env): # export foo := a\ b # it does not -- the backslash is written to the env as literal character. # So don't escape spaces in |env[k]|. - self.WriteLn("%s: export %s := %s" % (QuoteSpaces(target), k, v)) + self.WriteLn("{}: export {} := {}".format(QuoteSpaces(target), k, v)) def Objectify(self, path): """Convert a path to its output directory form.""" if "$(" in path: path = path.replace("$(obj)/", "$(obj).%s/$(TARGET)/" % self.toolset) if "$(obj)" not in path: - path = "$(obj).%s/$(TARGET)/%s" % (self.toolset, path) + path = f"$(obj).{self.toolset}/$(TARGET)/{path}" return path def Pchify(self, path, lang): @@ -2144,14 +2148,14 @@ def Pchify(self, path, lang): path = self.Absolutify(path) if "$(" in path: path = path.replace( - "$(obj)/", "$(obj).%s/$(TARGET)/pch-%s" % (self.toolset, lang) + "$(obj)/", f"$(obj).{self.toolset}/$(TARGET)/pch-{lang}" ) return path - return "$(obj).%s/$(TARGET)/pch-%s/%s" % (self.toolset, lang, path) + return f"$(obj).{self.toolset}/$(TARGET)/pch-{lang}/{path}" def Absolutify(self, path): """Convert a subdirectory-relative path into a base-relative path. - Skips over paths that contain variables.""" + Skips over paths that contain variables.""" if "$(" in path: # Don't call normpath in this case, as it might collapse the # path too aggressively if it features '..'. However it's still @@ -2219,7 +2223,7 @@ def PerformBuild(data, configurations, params): if options.toplevel_dir and options.toplevel_dir != ".": arguments += "-C", options.toplevel_dir arguments.append("BUILDTYPE=" + config) - print("Building [%s]: %s" % (config, arguments)) + print(f"Building [{config}]: {arguments}") subprocess.check_call(arguments) @@ -2253,7 +2257,7 @@ def CalculateMakefilePath(build_file, base_name): # away when we add verification that all targets have the # necessary configurations. default_configuration = None - toolsets = set([target_dicts[target]["toolset"] for target in target_list]) + toolsets = {target_dicts[target]["toolset"] for target in target_list} for target in target_list: spec = target_dicts[target] if spec["default_configuration"] != "Default": @@ -2328,7 +2332,7 @@ def CalculateMakefilePath(build_file, base_name): { "copy_archive_args": copy_archive_arguments, "flock": "./gyp-flock-tool flock", - "flock_index": 2 + "flock_index": 2, } ) elif flavor == "freebsd": @@ -2362,7 +2366,7 @@ def CalculateMakefilePath(build_file, base_name): value = "$(abspath %s)" % value wrapper = wrappers.get(key) if wrapper: - value = "%s %s" % (wrapper, value) + value = f"{wrapper} {value}" del wrappers[key] if key in ("CC", "CC.host", "CXX", "CXX.host"): make_global_settings += ( @@ -2372,10 +2376,10 @@ def CalculateMakefilePath(build_file, base_name): env_key = key.replace(".", "_") # CC.host -> CC_host if env_key in os.environ: value = os.environ[env_key] - make_global_settings += " %s = %s\n" % (key, value) + make_global_settings += f" {key} = {value}\n" make_global_settings += "endif\n" else: - make_global_settings += "%s ?= %s\n" % (key, value) + make_global_settings += f"{key} ?= {value}\n" # TODO(ukai): define cmd when only wrapper is specified in # make_global_settings. @@ -2413,8 +2417,8 @@ def CalculateMakefilePath(build_file, base_name): this_make_global_settings = data[build_file].get("make_global_settings", []) assert make_global_settings_array == this_make_global_settings, ( - "make_global_settings needs to be the same for all targets. %s vs. %s" - % (this_make_global_settings, make_global_settings) + "make_global_settings needs to be the same for all targets " + f"{this_make_global_settings} vs. {make_global_settings}" ) build_files.add(gyp.common.RelativePath(build_file, options.toplevel_dir)) diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index 96283b2757..e5f9dbb542 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -2,7 +2,6 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. -from __future__ import print_function import ntpath import os @@ -26,8 +25,6 @@ from gyp.common import GypError from gyp.common import OrderedSet -PY3 = bytes != str - # Regular expression for validating Visual Studio GUIDs. If the GUID # contains lowercase hex letters, MSVS will be fine. However, @@ -120,9 +117,7 @@ def _GetDomainAndUserName(): call = subprocess.Popen( ["net", "config", "Workstation"], stdout=subprocess.PIPE ) - config = call.communicate()[0] - if PY3: - config = config.decode("utf-8") + config = call.communicate()[0].decode("utf-8") username_re = re.compile(r"^User name\s+(\S+)", re.MULTILINE) username_match = username_re.search(config) if username_match: @@ -319,7 +314,7 @@ def _ConfigBaseName(config_name, platform_name): def _ConfigFullName(config_name, config_data): platform_name = _ConfigPlatform(config_data) - return "%s|%s" % (_ConfigBaseName(config_name, platform_name), platform_name) + return "{}|{}".format(_ConfigBaseName(config_name, platform_name), platform_name) def _ConfigWindowsTargetPlatformVersion(config_data, version): @@ -614,7 +609,7 @@ def _GenerateNativeRulesForMSVS(p, rules, output_dir, spec, options): spec: the project dict options: global generator options """ - rules_filename = "%s%s.rules" % (spec["target_name"], options.suffix) + rules_filename = "{}{}.rules".format(spec["target_name"], options.suffix) rules_file = MSVSToolFile.Writer( os.path.join(output_dir, rules_filename), spec["target_name"] ) @@ -660,7 +655,7 @@ def _GenerateExternalRules(rules, output_dir, spec, sources, options, actions_to options: global generator options actions_to_add: The list of actions we will add to. """ - filename = "%s_rules%s.mk" % (spec["target_name"], options.suffix) + filename = "{}_rules{}.mk".format(spec["target_name"], options.suffix) mk_file = gyp.common.WriteOnDiff(os.path.join(output_dir, filename)) # Find cygwin style versions of some paths. mk_file.write('OutDirCygwin:=$(shell cygpath -u "$(OutDir)")\n') @@ -703,7 +698,7 @@ def _GenerateExternalRules(rules, output_dir, spec, sources, options, actions_to cmd = ['"%s"' % i for i in cmd] cmd = " ".join(cmd) # Add it to the makefile. - mk_file.write("%s: %s\n" % (" ".join(outputs), " ".join(inputs))) + mk_file.write("{}: {}\n".format(" ".join(outputs), " ".join(inputs))) mk_file.write("\t%s\n\n" % cmd) # Close up the file. mk_file.close() @@ -1570,7 +1565,7 @@ def _AdjustSourcesAndConvertToFilterHierarchy( if version.UsesVcxproj(): while ( all([isinstance(s, MSVSProject.Filter) for s in sources]) - and len(set([s.name for s in sources])) == 1 + and len({s.name for s in sources}) == 1 ): assert all([len(s.contents) == 1 for s in sources]) sources = [s.contents[0] for s in sources] @@ -1776,8 +1771,8 @@ def _GetCopies(spec): base_dir = posixpath.split(src_bare)[0] outer_dir = posixpath.split(src_bare)[1] fixed_dst = _FixPath(dst) - full_dst = '"%s\\%s\\"' % (fixed_dst, outer_dir) - cmd = 'mkdir %s 2>nul & cd "%s" && xcopy /e /f /y "%s" %s' % ( + full_dst = f'"{fixed_dst}\\{outer_dir}\\"' + cmd = 'mkdir {} 2>nul & cd "{}" && xcopy /e /f /y "{}" {}'.format( full_dst, _FixPath(base_dir), outer_dir, @@ -1788,17 +1783,17 @@ def _GetCopies(spec): [src], ["dummy_copies", dst], cmd, - "Copying %s to %s" % (src, fixed_dst), + f"Copying {src} to {fixed_dst}", ) ) else: fix_dst = _FixPath(cpy["destination"]) - cmd = 'mkdir "%s" 2>nul & set ERRORLEVEL=0 & copy /Y "%s" "%s"' % ( + cmd = 'mkdir "{}" 2>nul & set ERRORLEVEL=0 & copy /Y "{}" "{}"'.format( fix_dst, _FixPath(src), _FixPath(dst), ) - copies.append(([src], [dst], cmd, "Copying %s to %s" % (src, fix_dst))) + copies.append(([src], [dst], cmd, f"Copying {src} to {fix_dst}")) return copies @@ -1898,12 +1893,12 @@ def _GetPlatformOverridesOfProject(spec): for config_name, c in spec["configurations"].items(): config_fullname = _ConfigFullName(config_name, c) platform = c.get("msvs_target_platform", _ConfigPlatform(c)) - fixed_config_fullname = "%s|%s" % ( + fixed_config_fullname = "{}|{}".format( _ConfigBaseName(config_name, _ConfigPlatform(c)), platform, ) if spec["toolset"] == "host" and generator_supports_multiple_toolsets: - fixed_config_fullname = "%s|x64" % (config_name,) + fixed_config_fullname = f"{config_name}|x64" config_platform_overrides[config_fullname] = fixed_config_fullname return config_platform_overrides @@ -2056,7 +2051,7 @@ def PerformBuild(data, configurations, params): for config in configurations: arguments = [devenv, sln_path, "/Build", config] - print("Building [%s]: %s" % (config, arguments)) + print(f"Building [{config}]: {arguments}") subprocess.check_call(arguments) @@ -2242,7 +2237,7 @@ def _AppendFiltersForMSBuild( if not parent_filter_name: filter_name = source.name else: - filter_name = "%s\\%s" % (parent_filter_name, source.name) + filter_name = f"{parent_filter_name}\\{source.name}" # Add the filter to the group. filter_group.append( [ @@ -2370,7 +2365,7 @@ def _GenerateRulesForMSBuild( _AdjustSourcesForRules(rules, sources, excluded_sources, True) -class MSBuildRule(object): +class MSBuildRule: """Used to store information used to generate an MSBuild rule. Attributes: @@ -2569,7 +2564,7 @@ def _GenerateMSBuildRuleTargetsFile(targets_path, msbuild_rules): "Condition": "'@(%s)' != '' and '%%(%s.ExcludedFromBuild)' != " "'true'" % (rule.tlog, rule.tlog), "File": "$(IntDir)$(ProjectName).read.1.tlog", - "Lines": "^%%(%s.Source);%%(%s.Inputs)" % (rule.tlog, rule.tlog), + "Lines": f"^%({rule.tlog}.Source);%({rule.tlog}.Inputs)", }, ] command_and_input_section = [ @@ -2915,7 +2910,7 @@ def _GetMSBuildProjectConfigurations(configurations, spec): group = ["ItemGroup", {"Label": "ProjectConfigurations"}] for (name, settings) in sorted(configurations.items()): configuration, platform = _GetConfigurationAndPlatform(name, settings, spec) - designation = "%s|%s" % (configuration, platform) + designation = f"{configuration}|{platform}" group.append( [ "ProjectConfiguration", @@ -3280,13 +3275,11 @@ def GetEdges(node): # Self references are ignored. Self reference is used in a few places to # append to the default value. I.e. PATH=$(PATH);other_path edges.update( - set( - [ + { v for v in MSVS_VARIABLE_REFERENCE.findall(value) if v in properties and v != node - ] - ) + } ) return edges diff --git a/gyp/pylib/gyp/generator/msvs_test.py b/gyp/pylib/gyp/generator/msvs_test.py index e001f417d5..69a5c7ec0e 100755 --- a/gyp/pylib/gyp/generator/msvs_test.py +++ b/gyp/pylib/gyp/generator/msvs_test.py @@ -8,10 +8,7 @@ import gyp.generator.msvs as msvs import unittest -try: - from StringIO import StringIO # Python 2 -except ImportError: - from io import StringIO # Python 3 +from io import StringIO class TestSequenceFunctions(unittest.TestCase): diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py index e064bad7ed..c57bec6abb 100644 --- a/gyp/pylib/gyp/generator/ninja.py +++ b/gyp/pylib/gyp/generator/ninja.py @@ -2,7 +2,6 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. -from __future__ import print_function import collections import copy @@ -20,10 +19,7 @@ import gyp.MSVSUtil as MSVSUtil import gyp.xcode_emulation -try: - from cStringIO import StringIO -except ImportError: - from io import StringIO +from io import StringIO from gyp.common import GetEnvironFallback import gyp.ninja_syntax as ninja_syntax @@ -76,7 +72,7 @@ def StripPrefix(arg, prefix): def QuoteShellArgument(arg, flavor): """Quote a string such that it will be interpreted as a single argument - by the shell.""" + by the shell.""" # Rather than attempting to enumerate the bad shell characters, just # allow common OK ones and quote anything else. if re.match(r"^[a-zA-Z0-9_=.\\/-]+$", arg): @@ -88,7 +84,7 @@ def QuoteShellArgument(arg, flavor): def Define(d, flavor): """Takes a preprocessor define and returns a -D parameter that's ninja- and - shell-escaped.""" + shell-escaped.""" if flavor == "win": # cl.exe replaces literal # characters with = in preprocessor definitions for # some reason. Octal-encode to work around that. @@ -99,32 +95,32 @@ def Define(d, flavor): def AddArch(output, arch): """Adds an arch string to an output path.""" output, extension = os.path.splitext(output) - return "%s.%s%s" % (output, arch, extension) + return f"{output}.{arch}{extension}" -class Target(object): +class Target: """Target represents the paths used within a single gyp target. - Conceptually, building a single target A is a series of steps: + Conceptually, building a single target A is a series of steps: - 1) actions/rules/copies generates source/resources/etc. - 2) compiles generates .o files - 3) link generates a binary (library/executable) - 4) bundle merges the above in a mac bundle + 1) actions/rules/copies generates source/resources/etc. + 2) compiles generates .o files + 3) link generates a binary (library/executable) + 4) bundle merges the above in a mac bundle - (Any of these steps can be optional.) + (Any of these steps can be optional.) - From a build ordering perspective, a dependent target B could just - depend on the last output of this series of steps. + From a build ordering perspective, a dependent target B could just + depend on the last output of this series of steps. - But some dependent commands sometimes need to reach inside the box. - For example, when linking B it needs to get the path to the static - library generated by A. + But some dependent commands sometimes need to reach inside the box. + For example, when linking B it needs to get the path to the static + library generated by A. - This object stores those paths. To keep things simple, member - variables only store concrete paths to single files, while methods - compute derived values like "the last output of the target". - """ + This object stores those paths. To keep things simple, member + variables only store concrete paths to single files, while methods + compute derived values like "the last output of the target". + """ def __init__(self, type): # Gyp type ("static_library", etc.) of this target. @@ -163,7 +159,7 @@ def Linkable(self): def UsesToc(self, flavor): """Return true if the target should produce a restat rule based on a TOC - file.""" + file.""" # For bundles, the .TOC should be produced for the binary, not for # FinalOutput(). But the naive approach would put the TOC file into the # bundle, so don't do this for bundles for now. @@ -173,19 +169,19 @@ def UsesToc(self, flavor): def PreActionInput(self, flavor): """Return the path, if any, that should be used as a dependency of - any dependent action step.""" + any dependent action step.""" if self.UsesToc(flavor): return self.FinalOutput() + ".TOC" return self.FinalOutput() or self.preaction_stamp def PreCompileInput(self): """Return the path, if any, that should be used as a dependency of - any dependent compile step.""" + any dependent compile step.""" return self.actions_stamp or self.precompile_stamp def FinalOutput(self): """Return the last output of the target, which depends on all prior - steps.""" + steps.""" return self.bundle or self.binary or self.actions_stamp @@ -214,7 +210,7 @@ def FinalOutput(self): # to the input file name as well as the output target name. -class NinjaWriter(object): +class NinjaWriter: def __init__( self, hash_for_rules, @@ -228,11 +224,11 @@ def __init__( toplevel_dir=None, ): """ - base_dir: path from source root to directory containing this gyp file, - by gyp semantics, all input paths are relative to this - build_dir: path from source root to build output - toplevel_dir: path to the toplevel directory - """ + base_dir: path from source root to directory containing this gyp file, + by gyp semantics, all input paths are relative to this + build_dir: path from source root to build output + toplevel_dir: path to the toplevel directory + """ self.hash_for_rules = hash_for_rules self.target_outputs = target_outputs @@ -263,10 +259,10 @@ def __init__( def ExpandSpecial(self, path, product_dir=None): """Expand specials like $!PRODUCT_DIR in |path|. - If |product_dir| is None, assumes the cwd is already the product - dir. Otherwise, |product_dir| is the relative path to the product - dir. - """ + If |product_dir| is None, assumes the cwd is already the product + dir. Otherwise, |product_dir| is the relative path to the product + dir. + """ PRODUCT_DIR = "$!PRODUCT_DIR" if PRODUCT_DIR in path: @@ -303,9 +299,9 @@ def ExpandRuleVariables(self, path, root, dirname, source, ext, name): def GypPathToNinja(self, path, env=None): """Translate a gyp path to a ninja path, optionally expanding environment - variable references in |path| with |env|. + variable references in |path| with |env|. - See the above discourse on path conversions.""" + See the above discourse on path conversions.""" if env: if self.flavor == "mac": path = gyp.xcode_emulation.ExpandEnvVars(path, env) @@ -324,11 +320,11 @@ def GypPathToNinja(self, path, env=None): def GypPathToUniqueOutput(self, path, qualified=True): """Translate a gyp path to a ninja path for writing output. - If qualified is True, qualify the resulting filename with the name - of the target. This is necessary when e.g. compiling the same - path twice for two separate output targets. + If qualified is True, qualify the resulting filename with the name + of the target. This is necessary when e.g. compiling the same + path twice for two separate output targets. - See the above discourse on path conversions.""" + See the above discourse on path conversions.""" path = self.ExpandSpecial(path) assert not path.startswith("$"), path @@ -361,9 +357,9 @@ def GypPathToUniqueOutput(self, path, qualified=True): def WriteCollapsedDependencies(self, name, targets, order_only=None): """Given a list of targets, return a path for a single file - representing the result of building all the targets or None. + representing the result of building all the targets or None. - Uses a stamp file if necessary.""" + Uses a stamp file if necessary.""" assert targets == [item for item in targets if item], targets if len(targets) == 0: @@ -377,14 +373,14 @@ def WriteCollapsedDependencies(self, name, targets, order_only=None): def _SubninjaNameForArch(self, arch): output_file_base = os.path.splitext(self.output_file_name)[0] - return "%s.%s.ninja" % (output_file_base, arch) + return f"{output_file_base}.{arch}.ninja" def WriteSpec(self, spec, config_name, generator_flags): """The main entry point for NinjaWriter: write the build rules for a spec. - Returns a Target object, which represents the output paths for this spec. - Returns None if there are no outputs (e.g. a settings-only 'none' type - target).""" + Returns a Target object, which represents the output paths for this spec. + Returns None if there are no outputs (e.g. a settings-only 'none' type + target).""" self.config_name = config_name self.name = spec["target_name"] @@ -418,20 +414,17 @@ def WriteSpec(self, spec, config_name, generator_flags): if self.flavor == "mac": self.archs = self.xcode_settings.GetActiveArchs(config_name) if len(self.archs) > 1: - self.arch_subninjas = dict( - ( - arch, - ninja_syntax.Writer( - OpenOutput( - os.path.join( - self.toplevel_build, self._SubninjaNameForArch(arch) - ), - "w", - ) - ), + self.arch_subninjas = { + arch: ninja_syntax.Writer( + OpenOutput( + os.path.join( + self.toplevel_build, self._SubninjaNameForArch(arch) + ), + "w", + ) ) for arch in self.archs - ) + } # Compute predepends for all rules. # actions_depends is the dependencies this target depends on before running @@ -558,7 +551,7 @@ def WriteSpec(self, spec, config_name, generator_flags): def _WinIdlRule(self, source, prebuild, outputs): """Handle the implicit VS .idl rule for one source file. Fills |outputs| - with files that are generated.""" + with files that are generated.""" outdir, output, vars, flags = self.msvs_settings.GetIdlBuildData( source, self.config_name ) @@ -595,7 +588,7 @@ def WriteActionsRulesCopies( self, spec, extra_sources, prebuild, mac_bundle_depends ): """Write out the Actions, Rules, and Copies steps. Return a path - representing the outputs of these steps.""" + representing the outputs of these steps.""" outputs = [] if self.is_mac_bundle: mac_bundle_resources = spec.get("mac_bundle_resources", [])[:] @@ -638,16 +631,16 @@ def WriteActionsRulesCopies( def GenerateDescription(self, verb, message, fallback): """Generate and return a description of a build step. - |verb| is the short summary, e.g. ACTION or RULE. - |message| is a hand-written description, or None if not available. - |fallback| is the gyp-level name of the step, usable as a fallback. - """ + |verb| is the short summary, e.g. ACTION or RULE. + |message| is a hand-written description, or None if not available. + |fallback| is the gyp-level name of the step, usable as a fallback. + """ if self.toolset != "target": verb += "(%s)" % self.toolset if message: - return "%s %s" % (verb, self.ExpandSpecial(message)) + return "{} {}".format(verb, self.ExpandSpecial(message)) else: - return "%s %s: %s" % (verb, self.name, fallback) + return f"{verb} {self.name}: {fallback}" def WriteActions( self, actions, extra_sources, prebuild, extra_mac_bundle_resources @@ -657,7 +650,7 @@ def WriteActions( all_outputs = [] for action in actions: # First write out a rule for the action. - name = "%s_%s" % (action["action_name"], self.hash_for_rules) + name = "{}_{}".format(action["action_name"], self.hash_for_rules) description = self.GenerateDescription( "ACTION", action.get("message", None), name ) @@ -706,7 +699,7 @@ def WriteRules( continue # First write out a rule for the rule action. - name = "%s_%s" % (rule["rule_name"], self.hash_for_rules) + name = "{}_{}".format(rule["rule_name"], self.hash_for_rules) args = rule["action"] description = self.GenerateDescription( @@ -731,7 +724,7 @@ def WriteRules( # must vary per source file. # Compute the list of variables we'll need to provide. special_locals = ("source", "root", "dirname", "ext", "name") - needed_variables = set(["source"]) + needed_variables = {"source"} for argument in args: for var in special_locals: if "${%s}" % var in argument: @@ -875,7 +868,7 @@ def WriteiOSFrameworkHeaders(self, spec, outputs, prebuild): output = self.GypPathToUniqueOutput("headers.hmap") self.xcode_settings.header_map_path = output all_headers = map( - self.GypPathToNinja, filter(lambda x: x.endswith((".h")), all_sources) + self.GypPathToNinja, filter(lambda x: x.endswith(".h"), all_sources) ) variables = [ ("framework", framework), @@ -925,11 +918,11 @@ def WriteMacBundleResources(self, resources, bundle_depends): def WriteMacXCassets(self, xcassets, bundle_depends): """Writes ninja edges for 'mac_bundle_resources' .xcassets files. - This add an invocation of 'actool' via the 'mac_tool.py' helper script. - It assumes that the assets catalogs define at least one imageset and - thus an Assets.car file will be generated in the application resources - directory. If this is not the case, then the build will probably be done - at each invocation of ninja.""" + This add an invocation of 'actool' via the 'mac_tool.py' helper script. + It assumes that the assets catalogs define at least one imageset and + thus an Assets.car file will be generated in the application resources + directory. If this is not the case, then the build will probably be done + at each invocation of ninja.""" if not xcassets: return @@ -1047,22 +1040,19 @@ def WriteSources( spec, ) else: - return dict( - ( - arch, - self.WriteSourcesForArch( - self.arch_subninjas[arch], - config_name, - config, - sources, - predepends, - precompiled_header, - spec, - arch=arch, - ), + return { + arch: self.WriteSourcesForArch( + self.arch_subninjas[arch], + config_name, + config, + sources, + predepends, + precompiled_header, + spec, + arch=arch, ) for arch in self.archs - ) + } def WriteSourcesForArch( self, @@ -1729,8 +1719,8 @@ def AppendPostbuildVariable( def GetPostbuildCommand(self, spec, output, output_binary, is_command_start): """Returns a shell command that runs all the postbuilds, and removes - |output| if any of them fails. If |is_command_start| is False, then the - returned string will start with ' && '.""" + |output| if any of them fails. If |is_command_start| is False, then the + returned string will start with ' && '.""" if not self.xcode_settings or spec["type"] == "none" or not output: return "" output = QuoteShellArgument(output, self.flavor) @@ -1776,8 +1766,8 @@ def GetPostbuildCommand(self, spec, output, output_binary, is_command_start): def ComputeExportEnvString(self, env): """Given an environment, returns a string looking like - 'export FOO=foo; export BAR="${FOO} bar;' - that exports |env| to the shell.""" + 'export FOO=foo; export BAR="${FOO} bar;' + that exports |env| to the shell.""" export_str = [] for k, v in env: export_str.append( @@ -1842,7 +1832,7 @@ def ComputeOutputFileName(self, spec, type=None): "shared_library", "executable", ): - return "%s%s%s" % (prefix, target, extension) + return f"{prefix}{target}{extension}" elif type == "none": return "%s.stamp" % target else: @@ -1909,8 +1899,8 @@ def WriteNewNinjaRule( ): """Write out a new ninja "rule" statement for a given command. - Returns the name of the new rule, and a copy of |args| with variables - expanded.""" + Returns the name of the new rule, and a copy of |args| with variables + expanded.""" if self.flavor == "win": args = [ @@ -2147,7 +2137,7 @@ class MEMORYSTATUSEX(ctypes.Structure): def _GetWinLinkRuleNameSuffix(embed_manifest): """Returns the suffix used to select an appropriate linking rule depending on - whether the manifest embedding is enabled.""" + whether the manifest embedding is enabled.""" return "_embed" if embed_manifest else "" @@ -2538,10 +2528,12 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name "solink", description="SOLINK $lib", restat=True, - command=mtime_preserving_solink_base % {"suffix": "@$link_file_list"}, # noqa: E501 + command=mtime_preserving_solink_base + % {"suffix": "@$link_file_list"}, # noqa: E501 rspfile="$link_file_list", - rspfile_content=("-Wl,--whole-archive $in $solibs -Wl," - "--no-whole-archive $libs"), + rspfile_content=( + "-Wl,--whole-archive $in $solibs -Wl," "--no-whole-archive $libs" + ), pool="link_pool", ) master_ninja.rule( @@ -2798,8 +2790,8 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name this_make_global_settings = data[build_file].get("make_global_settings", []) assert make_global_settings == this_make_global_settings, ( - "make_global_settings needs to be the same for all targets. %s vs. %s" - % (this_make_global_settings, make_global_settings) + "make_global_settings needs to be the same for all targets. " + f"{this_make_global_settings} vs. {make_global_settings}" ) spec = target_dicts[qualified_target] @@ -2891,7 +2883,7 @@ def PerformBuild(data, configurations, params): for config in configurations: builddir = os.path.join(options.toplevel_dir, "out", config) arguments = ["ninja", "-C", builddir] - print("Building [%s]: %s" % (config, arguments)) + print(f"Building [{config}]: {arguments}") subprocess.check_call(arguments) diff --git a/gyp/pylib/gyp/generator/xcode.py b/gyp/pylib/gyp/generator/xcode.py index 9e7e99e9e1..2f4d17e514 100644 --- a/gyp/pylib/gyp/generator/xcode.py +++ b/gyp/pylib/gyp/generator/xcode.py @@ -2,7 +2,6 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. -from __future__ import print_function import filecmp import gyp.common @@ -110,7 +109,7 @@ def CreateXCConfigurationList(configuration_names): return xccl -class XcodeProject(object): +class XcodeProject: def __init__(self, gyp_path, path, build_file_dict): self.gyp_path = gyp_path self.path = path @@ -613,7 +612,7 @@ def PerformBuild(data, configurations, params): for config in configurations: arguments = ["xcodebuild", "-project", xcodeproj_path] arguments += ["-configuration", config] - print("Building [%s]: %s" % (config, arguments)) + print(f"Building [{config}]: {arguments}") subprocess.check_call(arguments) @@ -1072,7 +1071,7 @@ def GenerateOutput(target_list, target_dicts, data, params): # TODO(mark): There's a possibility for collision here. Consider # target "t" rule "A_r" and target "t_A" rule "r". makefile_name = "%s.make" % re.sub( - "[^a-zA-Z0-9_]", "_", "%s_%s" % (target_name, rule["rule_name"]) + "[^a-zA-Z0-9_]", "_", "{}_{}".format(target_name, rule["rule_name"]) ) makefile_path = os.path.join( xcode_projects[build_file].path, makefile_name @@ -1102,7 +1101,7 @@ def GenerateOutput(target_list, target_dicts, data, params): eol = "" else: eol = " \\" - makefile.write(" %s%s\n" % (concrete_output, eol)) + makefile.write(f" {concrete_output}{eol}\n") for (rule_source, concrete_outputs, message, action) in zip( rule["rule_sources"], @@ -1123,7 +1122,7 @@ def GenerateOutput(target_list, target_dicts, data, params): bol = "" else: bol = " " - makefile.write("%s%s \\\n" % (bol, concrete_output)) + makefile.write(f"{bol}{concrete_output} \\\n") concrete_output_dir = posixpath.dirname(concrete_output) if ( @@ -1143,7 +1142,7 @@ def GenerateOutput(target_list, target_dicts, data, params): eol = "" else: eol = " \\" - makefile.write(" %s%s\n" % (prerequisite, eol)) + makefile.write(f" {prerequisite}{eol}\n") # Make sure that output directories exist before executing the rule # action. diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index 9039776240..ca7ce44eab 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -2,7 +2,6 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. -from __future__ import print_function import ast @@ -21,8 +20,6 @@ from gyp.common import GypError from gyp.common import OrderedSet -PY3 = bytes != str - # A list of types that are treated as linkable. linkable_types = [ "executable", @@ -228,17 +225,9 @@ def LoadOneBuildFile(build_file_path, data, aux_data, includes, is_target, check return data[build_file_path] if os.path.exists(build_file_path): - # Open the build file for read ('r') with universal-newlines mode ('U') - # to make sure platform specific newlines ('\r\n' or '\r') are converted to '\n' - # which otherwise will fail eval() - if PY3 or sys.platform == "zos": - # On z/OS, universal-newlines mode treats the file as an ascii file. - # But since node-gyp produces ebcdic files, do not use that mode. - build_file_contents = open(build_file_path, "r").read() - else: - build_file_contents = open(build_file_path, "rU").read() + build_file_contents = open(build_file_path).read() else: - raise GypError("%s not found (cwd: %s)" % (build_file_path, os.getcwd())) + raise GypError(f"{build_file_path} not found (cwd: {os.getcwd()})") build_file_data = None try: @@ -567,7 +556,7 @@ class ParallelProcessingError(Exception): pass -class ParallelState(object): +class ParallelState: """Class to keep track of state when processing input files in parallel. If build files are loaded in parallel, use this to keep track of @@ -987,9 +976,8 @@ def ExpandVariables(input, phase, variables, build_file): ) p_stdout, p_stderr = p.communicate("") - if PY3: - p_stdout = p_stdout.decode("utf-8") - p_stderr = p_stderr.decode("utf-8") + p_stdout = p_stdout.decode("utf-8") + p_stderr = p_stderr.decode("utf-8") if p.wait() != 0 or p_stderr: sys.stderr.write(p_stderr) @@ -1219,7 +1207,7 @@ def EvalSingleCondition(cond_expr, true_dict, false_dict, phase, variables, buil except NameError as e: gyp.common.ExceptionAppend( e, - "while evaluating condition '%s' in %s" % (cond_expr_expanded, build_file), + f"while evaluating condition '{cond_expr_expanded}' in {build_file}", ) raise GypError(e) @@ -1675,7 +1663,7 @@ def RemoveLinkDependenciesFromNoneTargets(targets): ) -class DependencyGraphNode(object): +class DependencyGraphNode: """ Attributes: @@ -2252,7 +2240,7 @@ def is_in_set_or_list(x, s, items): # Make membership testing of hashables in |to| (in particular, strings) # faster. - hashable_to_set = set(x for x in to if is_hashable(x)) + hashable_to_set = {x for x in to if is_hashable(x)} for item in fro: singleton = False if type(item) in (str, int): @@ -2772,7 +2760,7 @@ def ValidateRulesInTarget(target, target_dict, extra_sources_for_rules): rule_name = rule["rule_name"] if rule_name in rule_names: raise GypError( - "rule %s exists in duplicate, target %s" % (rule_name, target) + f"rule {rule_name} exists in duplicate, target {target}" ) rule_names[rule_name] = rule diff --git a/gyp/pylib/gyp/mac_tool.py b/gyp/pylib/gyp/mac_tool.py index 07412578d1..1cd8e3798f 100755 --- a/gyp/pylib/gyp/mac_tool.py +++ b/gyp/pylib/gyp/mac_tool.py @@ -8,7 +8,6 @@ These functions are executed via gyp-mac-tool when using the Makefile generator. """ -from __future__ import print_function import fcntl import fnmatch @@ -23,8 +22,6 @@ import sys import tempfile -PY3 = bytes != str - def main(args): executor = MacTool() @@ -33,7 +30,7 @@ def main(args): sys.exit(exit_code) -class MacTool(object): +class MacTool: """This class performs all the Mac tooling steps. The methods can either be executed directly, or dispatched from an argument list.""" @@ -179,7 +176,7 @@ def _DetectInputEncoding(self, file_name): def ExecCopyInfoPlist(self, source, dest, convert_to_binary, *keys): """Copies the |source| Info.plist to the destination directory |dest|.""" # Read the source Info.plist into memory. - with open(source, "r") as fd: + with open(source) as fd: lines = fd.read() # Insert synthesized key/value pairs (e.g. BuildMachineOSBuild). @@ -251,7 +248,7 @@ def _WritePkgInfo(self, info_plist): dest = os.path.join(os.path.dirname(info_plist), "PkgInfo") with open(dest, "w") as fp: - fp.write("%s%s" % (package_type, signature_code)) + fp.write(f"{package_type}{signature_code}") def ExecFlock(self, lockfile, *cmd_list): """Emulates the most basic behavior of Linux's flock(1).""" @@ -278,9 +275,7 @@ def ExecFilterLibtool(self, *cmd_list): # epoch=0, e.g. 1970-1-1 or 1969-12-31 depending on timezone. env["ZERO_AR_DATE"] = "1" libtoolout = subprocess.Popen(cmd_list, stderr=subprocess.PIPE, env=env) - _, err = libtoolout.communicate() - if PY3: - err = err.decode("utf-8") + err = libtoolout.communicate()[1].decode("utf-8") for line in err.splitlines(): if not libtool_re.match(line) and not libtool_re5.match(line): print(line, file=sys.stderr) @@ -540,7 +535,7 @@ def _FindProvisioningProfile(self, profile, bundle_identifier): "application-identifier", "" ) for team_identifier in profile_data.get("TeamIdentifier", []): - app_id = "%s.%s" % (team_identifier, bundle_identifier) + app_id = f"{team_identifier}.{bundle_identifier}" if fnmatch.fnmatch(app_id, app_id_pattern): valid_provisioning_profiles[app_id_pattern] = ( profile_path, diff --git a/gyp/pylib/gyp/msvs_emulation.py b/gyp/pylib/gyp/msvs_emulation.py index 1afc1d687e..f744e38df1 100644 --- a/gyp/pylib/gyp/msvs_emulation.py +++ b/gyp/pylib/gyp/msvs_emulation.py @@ -16,15 +16,13 @@ import gyp.MSVSUtil import gyp.MSVSVersion -PY3 = bytes != str - windows_quoter_regex = re.compile(r'(\\*)"') def QuoteForRspFile(arg): """Quote a command line argument so that it appears as one argument when - processed via cmd.exe and parsed by CommandLineToArgvW (as is typical for - Windows programs).""" + processed via cmd.exe and parsed by CommandLineToArgvW (as is typical for + Windows programs).""" # See http://goo.gl/cuFbX and http://goo.gl/dhPnp including the comment # threads. This is actually the quoting rules for CommandLineToArgvW, not # for the shell, because the shell doesn't do anything in Windows. This @@ -74,7 +72,7 @@ def EncodeRspFileList(args): def _GenericRetrieve(root, default, path): """Given a list of dictionary keys |path| and a tree of dicts |root|, find - value at path, or return |default| if any of the path doesn't exist.""" + value at path, or return |default| if any of the path doesn't exist.""" if not root: return default if not path: @@ -95,7 +93,7 @@ def _AddPrefix(element, prefix): def _DoRemapping(element, map): """If |element| then remap it through |map|. If |element| is iterable then - each item will be remapped. Any elements not found will be removed.""" + each item will be remapped. Any elements not found will be removed.""" if map is not None and element is not None: if not callable(map): map = map.get # Assume it's a dict, otherwise a callable to do the remap. @@ -108,8 +106,8 @@ def _DoRemapping(element, map): def _AppendOrReturn(append, element): """If |append| is None, simply return |element|. If |append| is not None, - then add |element| to it, adding each item in |element| if it's a list or - tuple.""" + then add |element| to it, adding each item in |element| if it's a list or + tuple.""" if append is not None and element is not None: if isinstance(element, list) or isinstance(element, tuple): append.extend(element) @@ -121,8 +119,8 @@ def _AppendOrReturn(append, element): def _FindDirectXInstallation(): """Try to find an installation location for the DirectX SDK. Check for the - standard environment variable, and if that doesn't exist, try to find - via the registry. May return None if not found in either location.""" + standard environment variable, and if that doesn't exist, try to find + via the registry. May return None if not found in either location.""" # Return previously calculated value, if there is one if hasattr(_FindDirectXInstallation, "dxsdk_dir"): return _FindDirectXInstallation.dxsdk_dir @@ -132,9 +130,7 @@ def _FindDirectXInstallation(): # Setup params to pass to and attempt to launch reg.exe. cmd = ["reg.exe", "query", r"HKLM\Software\Microsoft\DirectX", "/s"] p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) - stdout = p.communicate()[0] - if PY3: - stdout = stdout.decode("utf-8") + stdout = p.communicate()[0].decode("utf-8") for line in stdout.splitlines(): if "InstallPath" in line: dxsdk_dir = line.split(" ")[3] + "\\" @@ -146,7 +142,7 @@ def _FindDirectXInstallation(): def GetGlobalVSMacroEnv(vs_version): """Get a dict of variables mapping internal VS macro names to their gyp - equivalents. Returns all variables that are independent of the target.""" + equivalents. Returns all variables that are independent of the target.""" env = {} # '$(VSInstallDir)' and '$(VCInstallDir)' are available when and only when # Visual Studio is actually installed. @@ -167,7 +163,7 @@ def GetGlobalVSMacroEnv(vs_version): def ExtractSharedMSVSSystemIncludes(configs, generator_flags): """Finds msvs_system_include_dirs that are common to all targets, removes - them from all targets, and returns an OrderedSet containing them.""" + them from all targets, and returns an OrderedSet containing them.""" all_system_includes = OrderedSet(configs[0].get("msvs_system_include_dirs", [])) for config in configs[1:]: system_includes = config.get("msvs_system_include_dirs", []) @@ -193,10 +189,10 @@ def ExtractSharedMSVSSystemIncludes(configs, generator_flags): return expanded_system_includes -class MsvsSettings(object): +class MsvsSettings: """A class that understands the gyp 'msvs_...' values (especially the - msvs_settings field). They largely correpond to the VS2008 IDE DOM. This - class helps map those settings to command line options.""" + msvs_settings field). They largely correpond to the VS2008 IDE DOM. This + class helps map those settings to command line options.""" def __init__(self, spec, generator_flags): self.spec = spec @@ -229,7 +225,9 @@ def __init__(self, spec, generator_flags): for config in configs.values(): if field in config: unsupported += [ - "%s not supported (target %s)." % (field, spec["target_name"]) + "{} not supported (target {}).".format( + field, spec["target_name"] + ) ] if unsupported: raise Exception("\n".join(unsupported)) @@ -237,9 +235,9 @@ def __init__(self, spec, generator_flags): def GetExtension(self): """Returns the extension for the target, with no leading dot. - Uses 'product_extension' if specified, otherwise uses MSVS defaults based on - the target type. - """ + Uses 'product_extension' if specified, otherwise uses MSVS defaults based on + the target type. + """ ext = self.spec.get("product_extension", None) if ext: return ext @@ -247,7 +245,7 @@ def GetExtension(self): def GetVSMacroEnv(self, base_to_build=None, config=None): """Get a dict of variables mapping internal VS macro names to their gyp - equivalents.""" + equivalents.""" target_arch = self.GetArch(config) if target_arch == "x86": target_platform = "Win32" @@ -294,15 +292,15 @@ def AdjustLibraries(self, libraries): def _GetAndMunge(self, field, path, default, prefix, append, map): """Retrieve a value from |field| at |path| or return |default|. If - |append| is specified, and the item is found, it will be appended to that - object instead of returned. If |map| is specified, results will be - remapped through |map| before being returned or appended.""" + |append| is specified, and the item is found, it will be appended to that + object instead of returned. If |map| is specified, results will be + remapped through |map| before being returned or appended.""" result = _GenericRetrieve(field, default, path) result = _DoRemapping(result, map) result = _AddPrefix(result, prefix) return _AppendOrReturn(append, result) - class _GetWrapper(object): + class _GetWrapper: def __init__(self, parent, field, base_path, append=None): self.parent = parent self.field = field @@ -321,7 +319,7 @@ def __call__(self, name, map=None, prefix="", default=None): def GetArch(self, config): """Get architecture based on msvs_configuration_platform and - msvs_target_platform. Returns either 'x86' or 'x64'.""" + msvs_target_platform. Returns either 'x86' or 'x64'.""" configuration_platform = self.msvs_configuration_platform.get(config, "") platform = self.msvs_target_platform.get(config, "") if not platform: # If no specific override, use the configuration's. @@ -368,7 +366,7 @@ def _ConfigAttrib( def AdjustIncludeDirs(self, include_dirs, config): """Updates include_dirs to expand VS specific paths, and adds the system - include dirs used for platform SDK and similar.""" + include dirs used for platform SDK and similar.""" config = self._TargetConfig(config) includes = include_dirs + self.msvs_system_include_dirs[config] includes.extend( @@ -380,7 +378,7 @@ def AdjustIncludeDirs(self, include_dirs, config): def AdjustMidlIncludeDirs(self, midl_include_dirs, config): """Updates midl_include_dirs to expand VS specific paths, and adds the - system include dirs used for platform SDK and similar.""" + system include dirs used for platform SDK and similar.""" config = self._TargetConfig(config) includes = midl_include_dirs + self.msvs_system_include_dirs[config] includes.extend( @@ -392,7 +390,7 @@ def AdjustMidlIncludeDirs(self, midl_include_dirs, config): def GetComputedDefines(self, config): """Returns the set of defines that are injected to the defines list based - on other VS settings.""" + on other VS settings.""" config = self._TargetConfig(config) defines = [] if self._ConfigAttrib(["CharacterSet"], config) == "1": @@ -408,7 +406,7 @@ def GetComputedDefines(self, config): def GetCompilerPdbName(self, config, expand_special): """Get the pdb file name that should be used for compiler invocations, or - None if there's no explicit name specified.""" + None if there's no explicit name specified.""" config = self._TargetConfig(config) pdbname = self._Setting(("VCCLCompilerTool", "ProgramDataBaseFileName"), config) if pdbname: @@ -417,7 +415,7 @@ def GetCompilerPdbName(self, config, expand_special): def GetMapFileName(self, config, expand_special): """Gets the explicitly overridden map file name for a target or returns None - if it's not set.""" + if it's not set.""" config = self._TargetConfig(config) map_file = self._Setting(("VCLinkerTool", "MapFileName"), config) if map_file: @@ -426,7 +424,7 @@ def GetMapFileName(self, config, expand_special): def GetOutputName(self, config, expand_special): """Gets the explicitly overridden output name for a target or returns None - if it's not overridden.""" + if it's not overridden.""" config = self._TargetConfig(config) type = self.spec["type"] root = "VCLibrarianTool" if type == "static_library" else "VCLinkerTool" @@ -440,7 +438,7 @@ def GetOutputName(self, config, expand_special): def GetPDBName(self, config, expand_special, default): """Gets the explicitly overridden pdb name for a target or returns - default if it's not overridden, or if no pdb will be generated.""" + default if it's not overridden, or if no pdb will be generated.""" config = self._TargetConfig(config) output_file = self._Setting(("VCLinkerTool", "ProgramDatabaseFile"), config) generate_debug_info = self._Setting( @@ -456,7 +454,7 @@ def GetPDBName(self, config, expand_special, default): def GetNoImportLibrary(self, config): """If NoImportLibrary: true, ninja will not expect the output to include - an import library.""" + an import library.""" config = self._TargetConfig(config) noimplib = self._Setting(("NoImportLibrary",), config) return noimplib == "true" @@ -549,8 +547,7 @@ def GetCflags(self, config): return cflags def _GetPchFlags(self, config, extension): - """Get the flags to be added to the cflags for precompiled header support. - """ + """Get the flags to be added to the cflags for precompiled header support.""" config = self._TargetConfig(config) # The PCH is only built once by a particular source file. Usage of PCH must # only be for the same language (i.e. C vs. C++), so only include the pch @@ -575,7 +572,7 @@ def GetCflagsCC(self, config): def _GetAdditionalLibraryDirectories(self, root, config, gyp_to_build_path): """Get and normalize the list of paths in AdditionalLibraryDirectories - setting.""" + setting.""" config = self._TargetConfig(config) libpaths = self._Setting( (root, "AdditionalLibraryDirectories"), config, default=[] @@ -622,14 +619,14 @@ def GetDefFile(self, gyp_to_build_path): def _GetDefFileAsLdflags(self, ldflags, gyp_to_build_path): """.def files get implicitly converted to a ModuleDefinitionFile for the - linker in the VS generator. Emulate that behaviour here.""" + linker in the VS generator. Emulate that behaviour here.""" def_file = self.GetDefFile(gyp_to_build_path) if def_file: ldflags.append('/DEF:"%s"' % def_file) def GetPGDName(self, config, expand_special): """Gets the explicitly overridden pgd name for a target or returns None - if it's not overridden.""" + if it's not overridden.""" config = self._TargetConfig(config) output_file = self._Setting(("VCLinkerTool", "ProfileGuidedDatabase"), config) if output_file: @@ -649,7 +646,7 @@ def GetLdflags( build_dir, ): """Returns the flags that need to be added to link commands, and the - manifest files.""" + manifest files.""" config = self._TargetConfig(config) ldflags = [] ld = self._GetWrapper( @@ -709,7 +706,7 @@ def GetLdflags( ) if stack_commit_size: stack_commit_size = "," + stack_commit_size - ldflags.append("/STACK:%s%s" % (stack_reserve_size, stack_commit_size)) + ldflags.append(f"/STACK:{stack_reserve_size}{stack_commit_size}") ld("TerminalServerAware", map={"1": ":NO", "2": ""}, prefix="/TSAWARE") ld("LinkIncremental", map={"1": ":NO", "2": ""}, prefix="/INCREMENTAL") @@ -775,12 +772,12 @@ def _GetLdManifestFlags( self, config, name, gyp_to_build_path, allow_isolation, build_dir ): """Returns a 3-tuple: - - the set of flags that need to be added to the link to generate - a default manifest - - the intermediate manifest that the linker will generate that should be - used to assert it doesn't add anything to the merged one. - - the list of all the manifest files to be merged by the manifest tool and - included into the link.""" + - the set of flags that need to be added to the link to generate + a default manifest + - the intermediate manifest that the linker will generate that should be + used to assert it doesn't add anything to the merged one. + - the list of all the manifest files to be merged by the manifest tool and + included into the link.""" generate_manifest = self._Setting( ("VCLinkerTool", "GenerateManifest"), config, default="true" ) @@ -835,10 +832,10 @@ def _GetLdManifestFlags( - + -""" % ( +""".format( execution_level_map[execution_level], ui_access, ) @@ -867,7 +864,7 @@ def _GetLdManifestFlags( def _GetAdditionalManifestFiles(self, config, gyp_to_build_path): """Gets additional manifest files that are added to the default one - generated by the linker.""" + generated by the linker.""" files = self._Setting( ("VCManifestTool", "AdditionalManifestFiles"), config, default=[] ) @@ -880,7 +877,7 @@ def _GetAdditionalManifestFiles(self, config, gyp_to_build_path): def IsUseLibraryDependencyInputs(self, config): """Returns whether the target should be linked via Use Library Dependency - Inputs (using component .objs of a given .lib).""" + Inputs (using component .objs of a given .lib).""" config = self._TargetConfig(config) uldi = self._Setting(("VCLinkerTool", "UseLibraryDependencyInputs"), config) return uldi == "true" @@ -901,7 +898,7 @@ def IsLinkIncremental(self, config): def GetRcflags(self, config, gyp_to_ninja_path): """Returns the flags that need to be added to invocations of the resource - compiler.""" + compiler.""" config = self._TargetConfig(config) rcflags = [] rc = self._GetWrapper( @@ -916,13 +913,13 @@ def GetRcflags(self, config, gyp_to_ninja_path): def BuildCygwinBashCommandLine(self, args, path_to_base): """Build a command line that runs args via cygwin bash. We assume that all - incoming paths are in Windows normpath'd form, so they need to be - converted to posix style for the part of the command line that's passed to - bash. We also have to do some Visual Studio macro emulation here because - various rules use magic VS names for things. Also note that rules that - contain ninja variables cannot be fixed here (for example ${source}), so - the outer generator needs to make sure that the paths that are written out - are in posix style, if the command line will be used here.""" + incoming paths are in Windows normpath'd form, so they need to be + converted to posix style for the part of the command line that's passed to + bash. We also have to do some Visual Studio macro emulation here because + various rules use magic VS names for things. Also note that rules that + contain ninja variables cannot be fixed here (for example ${source}), so + the outer generator needs to make sure that the paths that are written out + are in posix style, if the command line will be used here.""" cygwin_dir = os.path.normpath( os.path.join(path_to_base, self.msvs_cygwin_dirs[0]) ) @@ -932,13 +929,13 @@ def BuildCygwinBashCommandLine(self, args, path_to_base): bash_cmd = " ".join(args) cmd = ( 'call "%s\\setup_env.bat" && set CYGWIN=nontsec && ' % cygwin_dir - + 'bash -c "%s ; %s"' % (cd, bash_cmd) + + f'bash -c "{cd} ; {bash_cmd}"' ) return cmd def IsRuleRunUnderCygwin(self, rule): """Determine if an action should be run under cygwin. If the variable is - unset, or set to 1 we use cygwin.""" + unset, or set to 1 we use cygwin.""" return ( int(rule.get("msvs_cygwin_shell", self.spec.get("msvs_cygwin_shell", 1))) != 0 @@ -959,19 +956,19 @@ def _HasExplicitIdlActions(self, spec): def HasExplicitIdlRulesOrActions(self, spec): """Determine if there's an explicit rule or action for idl files. When - there isn't we need to generate implicit rules to build MIDL .idl files.""" + there isn't we need to generate implicit rules to build MIDL .idl files.""" return self._HasExplicitRuleForExtension( spec, "idl" ) or self._HasExplicitIdlActions(spec) def HasExplicitAsmRules(self, spec): """Determine if there's an explicit rule for asm files. When there isn't we - need to generate implicit rules to assemble .asm files.""" + need to generate implicit rules to assemble .asm files.""" return self._HasExplicitRuleForExtension(spec, "asm") def GetIdlBuildData(self, source, config): """Determine the implicit outputs for an idl file. Returns output - directory, outputs, and variables and flags that are required.""" + directory, outputs, and variables and flags that are required.""" config = self._TargetConfig(config) midl_get = self._GetWrapper(self, self.msvs_settings[config], "VCMIDLTool") @@ -1010,10 +1007,10 @@ def _LanguageMatchesForPch(source_ext, pch_source_ext): ) -class PrecompiledHeader(object): +class PrecompiledHeader: """Helper to generate dependencies and build rules to handle generation of - precompiled headers. Interface matches the GCH handler in xcode_emulation.py. - """ + precompiled headers. Interface matches the GCH handler in xcode_emulation.py. + """ def __init__( self, settings, config, gyp_to_build_path, gyp_to_unique_output, obj_ext @@ -1027,14 +1024,14 @@ def __init__( def _PchHeader(self): """Get the header that will appear in an #include line for all source - files.""" + files.""" return self.settings.msvs_precompiled_header[self.config] def GetObjDependencies(self, sources, objs, arch): """Given a list of sources files and the corresponding object files, - returns a list of the pch files that should be depended upon. The - additional wrapping in the return value is for interface compatibility - with make.py on Mac, and xcode_emulation.py.""" + returns a list of the pch files that should be depended upon. The + additional wrapping in the return value is for interface compatibility + with make.py on Mac, and xcode_emulation.py.""" assert arch is None if not self._PchHeader(): return [] @@ -1046,14 +1043,14 @@ def GetObjDependencies(self, sources, objs, arch): def GetPchBuildCommands(self, arch): """Not used on Windows as there are no additional build steps required - (instead, existing steps are modified in GetFlagsModifications below).""" + (instead, existing steps are modified in GetFlagsModifications below).""" return [] def GetFlagsModifications( self, input, output, implicit, command, cflags_c, cflags_cc, expand_special ): """Get the modified cflags and implicit dependencies that should be used - for the pch compilation step.""" + for the pch compilation step.""" if input == self.pch_source: pch_output = ["/Yc" + self._PchHeader()] if command == "cxx": @@ -1090,7 +1087,7 @@ def _GetVsvarsSetupArgs(generator_flags, arch): def ExpandMacros(string, expansions): """Expand $(Variable) per expansions dict. See MsvsSettings.GetVSMacroEnv - for the canonical way to retrieve a suitable dict.""" + for the canonical way to retrieve a suitable dict.""" if "$" in string: for old, new in expansions.items(): assert "$(" not in new, new @@ -1100,7 +1097,7 @@ def ExpandMacros(string, expansions): def _ExtractImportantEnvironment(output_of_set): """Extracts environment variables required for the toolchain to run from - a textual dump output by the cmd.exe 'set' command.""" + a textual dump output by the cmd.exe 'set' command.""" envvars_to_save = ( "goma_.*", # TODO(scottmg): This is ugly, but needed for goma. "include", @@ -1140,8 +1137,8 @@ def _ExtractImportantEnvironment(output_of_set): def _FormatAsEnvironmentBlock(envvar_dict): """Format as an 'environment block' directly suitable for CreateProcess. - Briefly this is a list of key=value\0, terminated by an additional \0. See - CreateProcess documentation for more details.""" + Briefly this is a list of key=value\0, terminated by an additional \0. See + CreateProcess documentation for more details.""" block = "" nul = "\0" for key, value in envvar_dict.items(): @@ -1152,7 +1149,7 @@ def _FormatAsEnvironmentBlock(envvar_dict): def _ExtractCLPath(output_of_where): """Gets the path to cl.exe based on the output of calling the environment - setup batch file, followed by the equivalent of `where`.""" + setup batch file, followed by the equivalent of `where`.""" # Take the first line, as that's the first found in the PATH. for line in output_of_where.strip().splitlines(): if line.startswith("LOC:"): @@ -1163,19 +1160,19 @@ def GenerateEnvironmentFiles( toplevel_build_dir, generator_flags, system_includes, open_out ): """It's not sufficient to have the absolute path to the compiler, linker, - etc. on Windows, as those tools rely on .dlls being in the PATH. We also - need to support both x86 and x64 compilers within the same build (to support - msvs_target_platform hackery). Different architectures require a different - compiler binary, and different supporting environment variables (INCLUDE, - LIB, LIBPATH). So, we extract the environment here, wrap all invocations - of compiler tools (cl, link, lib, rc, midl, etc.) via win_tool.py which - sets up the environment, and then we do not prefix the compiler with - an absolute path, instead preferring something like "cl.exe" in the rule - which will then run whichever the environment setup has put in the path. - When the following procedure to generate environment files does not - meet your requirement (e.g. for custom toolchains), you can pass - "-G ninja_use_custom_environment_files" to the gyp to suppress file - generation and use custom environment files prepared by yourself.""" + etc. on Windows, as those tools rely on .dlls being in the PATH. We also + need to support both x86 and x64 compilers within the same build (to support + msvs_target_platform hackery). Different architectures require a different + compiler binary, and different supporting environment variables (INCLUDE, + LIB, LIBPATH). So, we extract the environment here, wrap all invocations + of compiler tools (cl, link, lib, rc, midl, etc.) via win_tool.py which + sets up the environment, and then we do not prefix the compiler with + an absolute path, instead preferring something like "cl.exe" in the rule + which will then run whichever the environment setup has put in the path. + When the following procedure to generate environment files does not + meet your requirement (e.g. for custom toolchains), you can pass + "-G ninja_use_custom_environment_files" to the gyp to suppress file + generation and use custom environment files prepared by yourself.""" archs = ("x86", "x64") if generator_flags.get("ninja_use_custom_environment_files", 0): cl_paths = {} @@ -1191,9 +1188,7 @@ def GenerateEnvironmentFiles( popen = subprocess.Popen( args, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT ) - variables, _ = popen.communicate() - if PY3: - variables = variables.decode("utf-8") + variables = popen.communicate()[0].decode("utf-8") if popen.returncode != 0: raise Exception('"%s" failed with error %d' % (args, popen.returncode)) env = _ExtractImportantEnvironment(variables) @@ -1216,19 +1211,17 @@ def GenerateEnvironmentFiles( ("&&", "for", "%i", "in", "(cl.exe)", "do", "@echo", "LOC:%~$PATH:i") ) popen = subprocess.Popen(args, shell=True, stdout=subprocess.PIPE) - output, _ = popen.communicate() - if PY3: - output = output.decode("utf-8") + output = popen.communicate()[0].decode("utf-8") cl_paths[arch] = _ExtractCLPath(output) return cl_paths def VerifyMissingSources(sources, build_dir, generator_flags, gyp_to_ninja): """Emulate behavior of msvs_error_on_missing_sources present in the msvs - generator: Check that all regular source files, i.e. not created at run time, - exist on disk. Missing files cause needless recompilation when building via - VS, and we want this check to match for people/bots that build using ninja, - so they're not surprised when the VS build fails.""" + generator: Check that all regular source files, i.e. not created at run time, + exist on disk. Missing files cause needless recompilation when building via + VS, and we want this check to match for people/bots that build using ninja, + so they're not surprised when the VS build fails.""" if int(generator_flags.get("msvs_error_on_missing_sources", 0)): no_specials = filter(lambda x: "$" not in x, sources) relative = [os.path.join(build_dir, gyp_to_ninja(s)) for s in no_specials] diff --git a/gyp/pylib/gyp/ninja_syntax.py b/gyp/pylib/gyp/ninja_syntax.py index 1421235808..0e3e86c743 100644 --- a/gyp/pylib/gyp/ninja_syntax.py +++ b/gyp/pylib/gyp/ninja_syntax.py @@ -16,7 +16,7 @@ def escape_path(word): return word.replace("$ ", "$$ ").replace(" ", "$ ").replace(":", "$:") -class Writer(object): +class Writer: def __init__(self, output, width=78): self.output = output self.width = width @@ -33,7 +33,7 @@ def variable(self, key, value, indent=0): return if isinstance(value, list): value = " ".join(filter(None, value)) # Filter out empty strings. - self._line("%s = %s" % (key, value), indent) + self._line(f"{key} = {value}", indent) def pool(self, name, depth): self._line("pool %s" % name) @@ -89,7 +89,7 @@ def build( all_inputs.extend(order_only) self._line( - "build %s: %s" % (" ".join(out_outputs), " ".join([rule] + all_inputs)) + "build {}: {}".format(" ".join(out_outputs), " ".join([rule] + all_inputs)) ) if variables: diff --git a/gyp/pylib/gyp/simple_copy.py b/gyp/pylib/gyp/simple_copy.py index e01106f9c4..729cec0636 100644 --- a/gyp/pylib/gyp/simple_copy.py +++ b/gyp/pylib/gyp/simple_copy.py @@ -36,10 +36,7 @@ def _deepcopy_atomic(x): return x -try: - types = bool, float, int, str, type, type(None), long, unicode -except NameError: # Python 3 - types = bool, float, int, str, type, type(None) +types = bool, float, int, str, type, type(None) for x in types: d[x] = _deepcopy_atomic diff --git a/gyp/pylib/gyp/win_tool.py b/gyp/pylib/gyp/win_tool.py index 758e9f5c45..7e85946720 100755 --- a/gyp/pylib/gyp/win_tool.py +++ b/gyp/pylib/gyp/win_tool.py @@ -9,7 +9,6 @@ These functions are executed via gyp-win-tool when using the ninja generator. """ -from __future__ import print_function import os import re @@ -20,7 +19,6 @@ import sys BASE_DIR = os.path.dirname(os.path.abspath(__file__)) -PY3 = bytes != str # A regex matching an argument corresponding to the output filename passed to # link.exe. @@ -34,7 +32,7 @@ def main(args): sys.exit(exit_code) -class WinTool(object): +class WinTool: """This class performs all the Windows tooling steps. The methods can either be executed directly, or dispatched from an argument list.""" @@ -141,9 +139,7 @@ def ExecLinkWrapper(self, arch, use_separate_mspdbsrv, *args): stdout=subprocess.PIPE, stderr=subprocess.STDOUT, ) - out, _ = link.communicate() - if PY3: - out = out.decode("utf-8") + out = link.communicate()[0].decode("utf-8") for line in out.splitlines(): if ( not line.startswith(" Creating library ") @@ -223,8 +219,8 @@ def ExecLinkWithManifests( our_manifest = "%(out)s.manifest" % variables # Load and normalize the manifests. mt.exe sometimes removes whitespace, # and sometimes doesn't unfortunately. - with open(our_manifest, "r") as our_f: - with open(assert_manifest, "r") as assert_f: + with open(our_manifest) as our_f: + with open(assert_manifest) as assert_f: our_data = our_f.read().translate(None, string.whitespace) assert_data = assert_f.read().translate(None, string.whitespace) if our_data != assert_data: @@ -233,7 +229,7 @@ def ExecLinkWithManifests( def dump(filename): print(filename, file=sys.stderr) print("-----", file=sys.stderr) - with open(filename, "r") as f: + with open(filename) as f: print(f.read(), file=sys.stderr) print("-----", file=sys.stderr) @@ -256,9 +252,7 @@ def ExecManifestWrapper(self, arch, *args): popen = subprocess.Popen( args, shell=True, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT ) - out, _ = popen.communicate() - if PY3: - out = out.decode("utf-8") + out = popen.communicate()[0].decode("utf-8") for line in out.splitlines(): if line and "manifest authoring warning 81010002" not in line: print(line) @@ -302,16 +296,14 @@ def ExecMidlWrapper(self, arch, outdir, tlb, h, dlldata, iid, proxy, idl, *flags popen = subprocess.Popen( args, shell=True, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT ) - out, _ = popen.communicate() - if PY3: - out = out.decode("utf-8") + out = popen.communicate()[0].decode("utf-8") # Filter junk out of stdout, and write filtered versions. Output we want # to filter is pairs of lines that look like this: # Processing C:\Program Files (x86)\Microsoft SDKs\...\include\objidl.idl # objidl.idl lines = out.splitlines() prefixes = ("Processing ", "64 bit Processing ") - processing = set(os.path.basename(x) for x in lines if x.startswith(prefixes)) + processing = {os.path.basename(x) for x in lines if x.startswith(prefixes)} for line in lines: if not line.startswith(prefixes) and line not in processing: print(line) @@ -323,9 +315,7 @@ def ExecAsmWrapper(self, arch, *args): popen = subprocess.Popen( args, shell=True, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT ) - out, _ = popen.communicate() - if PY3: - out = out.decode("utf-8") + out = popen.communicate()[0].decode("utf-8") for line in out.splitlines(): if ( not line.startswith("Copyright (C) Microsoft Corporation") @@ -343,9 +333,7 @@ def ExecRcWrapper(self, arch, *args): popen = subprocess.Popen( args, shell=True, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT ) - out, _ = popen.communicate() - if PY3: - out = out.decode("utf-8") + out = popen.communicate()[0].decode("utf-8") for line in out.splitlines(): if ( not line.startswith("Microsoft (R) Windows (R) Resource Compiler") diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py index a79aaa41fb..a75d8eeab7 100644 --- a/gyp/pylib/gyp/xcode_emulation.py +++ b/gyp/pylib/gyp/xcode_emulation.py @@ -7,7 +7,6 @@ other build systems, such as make and ninja. """ -from __future__ import print_function import copy import gyp.common @@ -19,8 +18,6 @@ import sys from gyp.common import GypError -PY3 = bytes != str - # Populated lazily by XcodeVersion, for efficiency, and to fix an issue when # "xcodebuild" is called too quickly (it has been found to return incorrect # version number). @@ -40,7 +37,7 @@ def XcodeArchsVariableMapping(archs, archs_including_64_bit=None): return mapping -class XcodeArchsDefault(object): +class XcodeArchsDefault: """A class to resolve ARCHS variable from xcode_settings, resolving Xcode macros and implementing filtering by VALID_ARCHS. The expansion of macros depends on the SDKROOT used ("macosx", "iphoneos", "iphonesimulator") and @@ -148,7 +145,7 @@ def GetXcodeArchsDefault(): return XCODE_ARCHS_DEFAULT_CACHE -class XcodeSettings(object): +class XcodeSettings: """A class that understands the gyp 'xcode_settings' object.""" # Populated lazily by _SdkPath(). Shared by all XcodeSettings, so cached @@ -281,7 +278,7 @@ def GetWrapperExtension(self): else: return "." + self.spec.get("product_extension", "app") else: - assert False, "Don't know extension for '%s', target '%s'" % ( + assert False, "Don't know extension for '{}', target '{}'".format( self.spec["type"], self.spec["target_name"], ) @@ -1088,7 +1085,7 @@ def _GetStripPostbuilds(self, configname, output_binary, quiet): if not quiet: result.append("echo STRIP\\(%s\\)" % self.spec["target_name"]) - result.append("strip %s %s" % (strip_flags, output_binary)) + result.append(f"strip {strip_flags} {output_binary}") self.configname = None return result @@ -1110,7 +1107,7 @@ def _GetDebugInfoPostbuilds(self, configname, output, output_binary, quiet): ): if not quiet: result.append("echo DSYMUTIL\\(%s\\)" % self.spec["target_name"]) - result.append("dsymutil %s -o %s" % (output_binary, output + ".dSYM")) + result.append("dsymutil {} -o {}".format(output_binary, output + ".dSYM")) self.configname = None return result @@ -1143,7 +1140,7 @@ def _GetIOSPostbuilds(self, configname, output_binary): source = os.path.join("${BUILT_PRODUCTS_DIR}", product_name) test_host = os.path.dirname(settings.get("TEST_HOST")) xctest_destination = os.path.join(test_host, "PlugIns", product_name) - postbuilds.extend(["ditto %s %s" % (source, xctest_destination)]) + postbuilds.extend([f"ditto {source} {xctest_destination}"]) key = self._GetIOSCodeSignIdentityKey(settings) if not key: @@ -1170,7 +1167,7 @@ def _GetIOSPostbuilds(self, configname, output_binary): for framework in frameworks: source = os.path.join(platform_root, framework) destination = os.path.join(frameworks_dir, os.path.basename(framework)) - postbuilds.extend(["ditto %s %s" % (source, destination)]) + postbuilds.extend([f"ditto {source} {destination}"]) # Then re-sign everything with 'preserve=True' postbuilds.extend( @@ -1371,7 +1368,7 @@ def _DefaultSdkRoot(self): return "" -class MacPrefixHeader(object): +class MacPrefixHeader: """A class that helps with emulating Xcode's GCC_PREFIX_HEADER feature. This feature consists of several pieces: @@ -1561,9 +1558,7 @@ def GetStdoutQuiet(cmdlist): Ignores the stderr. Raises |GypError| if the command return with a non-zero return code.""" job = subprocess.Popen(cmdlist, stdout=subprocess.PIPE, stderr=subprocess.PIPE) - out = job.communicate()[0] - if PY3: - out = out.decode("utf-8") + out = job.communicate()[0].decode("utf-8") if job.returncode != 0: raise GypError("Error %d running %s" % (job.returncode, cmdlist[0])) return out.rstrip("\n") @@ -1573,9 +1568,7 @@ def GetStdout(cmdlist): """Returns the content of standard output returned by invoking |cmdlist|. Raises |GypError| if the command return with a non-zero return code.""" job = subprocess.Popen(cmdlist, stdout=subprocess.PIPE) - out = job.communicate()[0] - if PY3: - out = out.decode("utf-8") + out = job.communicate()[0].decode("utf-8") if job.returncode != 0: sys.stderr.write(out + "\n") raise GypError("Error %d running %s" % (job.returncode, cmdlist[0])) @@ -1871,7 +1864,7 @@ def GetEdges(node): # definition contains all variables it references in a single string. # We can then reverse the result of the topological sort at the end. # Since: reverse(topsort(DAG)) = topsort(reverse_edges(DAG)) - matches = set([v for v in regex.findall(env[node]) if v in env]) + matches = {v for v in regex.findall(env[node]) if v in env} for dependee in matches: assert "${" not in dependee, "Nested variables not supported: " + dependee return matches diff --git a/gyp/pylib/gyp/xcode_ninja.py b/gyp/pylib/gyp/xcode_ninja.py index 10ddcbccd0..bb74eacbea 100644 --- a/gyp/pylib/gyp/xcode_ninja.py +++ b/gyp/pylib/gyp/xcode_ninja.py @@ -43,11 +43,11 @@ def _WriteWorkspace(main_gyp, sources_gyp, params): workspace_file = os.path.join(workspace_path, "contents.xcworkspacedata") try: - with open(workspace_file, "r") as input_file: + with open(workspace_file) as input_file: input_string = input_file.read() if input_string == output_string: return - except IOError: + except OSError: # Ignore errors if the file doesn't exist. pass @@ -214,7 +214,7 @@ def CreateWrapper(target_list, target_dicts, data, params): if IsValidTargetForWrapper(target_extras, executable_target_pattern, spec): # Add to new_target_list. target_name = spec.get("target_name") - new_target_name = "%s:%s#target" % (main_gyp, target_name) + new_target_name = f"{main_gyp}:{target_name}#target" new_target_list.append(new_target_name) # Add to new_target_dicts. @@ -282,7 +282,7 @@ def CreateWrapper(target_list, target_dicts, data, params): # Put sources_to_index in it's own gyp. sources_gyp = os.path.join(os.path.dirname(main_gyp), sources_target_name + ".gyp") - fully_qualified_target_name = "%s:%s#target" % (sources_gyp, sources_target_name) + fully_qualified_target_name = f"{sources_gyp}:{sources_target_name}#target" # Add to new_target_list, new_target_dicts and new_data. new_target_list.append(fully_qualified_target_name) diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py index d90dd99dcc..bca4fb79d0 100644 --- a/gyp/pylib/gyp/xcodeproj_file.py +++ b/gyp/pylib/gyp/xcodeproj_file.py @@ -144,13 +144,9 @@ import struct import sys -try: - basestring, cmp, unicode -except NameError: # Python 3 - basestring = unicode = str - def cmp(x, y): - return (x > y) - (x < y) +def cmp(x, y): + return (x > y) - (x < y) # See XCObject._EncodeString. This pattern is used to determine when a string @@ -199,7 +195,7 @@ def ConvertVariablesToShellSyntax(input_string): return re.sub(r"\$\((.*?)\)", "${\\1}", input_string) -class XCObject(object): +class XCObject: """The abstract base of all class types used in Xcode project files. Class variables: @@ -301,8 +297,8 @@ def __repr__(self): try: name = self.Name() except NotImplementedError: - return "<%s at 0x%x>" % (self.__class__.__name__, id(self)) - return "<%s %r at 0x%x>" % (self.__class__.__name__, name, id(self)) + return "<{} at 0x{:x}>".format(self.__class__.__name__, id(self)) + return "<{} {!r} at 0x{:x}>".format(self.__class__.__name__, name, id(self)) def Copy(self): """Make a copy of this object. @@ -325,7 +321,7 @@ def Copy(self): that._properties[key] = new_value else: that._properties[key] = value - elif isinstance(value, (basestring, int)): + elif isinstance(value, (str, int)): that._properties[key] = value elif isinstance(value, list): if is_strong: @@ -616,7 +612,7 @@ def _XCPrintableValue(self, tabs, value, flatten_list=False): comment = value.Comment() elif isinstance(value, str): printable += self._EncodeString(value) - elif isinstance(value, basestring): + elif isinstance(value, str): printable += self._EncodeString(value.encode("utf-8")) elif isinstance(value, int): printable += str(value) @@ -791,7 +787,7 @@ def UpdateProperties(self, properties, do_copy=False): ) for item in value: if not isinstance(item, property_type) and not ( - isinstance(item, basestring) and property_type == str + isinstance(item, str) and property_type == str ): # Accept unicode where str is specified. str is treated as # UTF-8-encoded. @@ -806,7 +802,7 @@ def UpdateProperties(self, properties, do_copy=False): + item.__class__.__name__ ) elif not isinstance(value, property_type) and not ( - isinstance(value, basestring) and property_type == str + isinstance(value, str) and property_type == str ): # Accept unicode where str is specified. str is treated as # UTF-8-encoded. @@ -827,7 +823,7 @@ def UpdateProperties(self, properties, do_copy=False): self._properties[property] = value.Copy() else: self._properties[property] = value - elif isinstance(value, (basestring, int)): + elif isinstance(value, (str, int)): self._properties[property] = value elif isinstance(value, list): if is_strong: @@ -2185,7 +2181,7 @@ def SetDestination(self, path): relative_path = path[1:] else: raise ValueError( - "Can't use path %s in a %s" % (path, self.__class__.__name__) + f"Can't use path {path} in a {self.__class__.__name__}" ) self._properties["dstPath"] = relative_path @@ -2250,8 +2246,8 @@ class PBXContainerItemProxy(XCObject): def __repr__(self): props = self._properties - name = "%s.gyp:%s" % (props["containerPortal"].Name(), props["remoteInfo"]) - return "<%s %r at 0x%x>" % (self.__class__.__name__, name, id(self)) + name = "{}.gyp:{}".format(props["containerPortal"].Name(), props["remoteInfo"]) + return "<{} {!r} at 0x{:x}>".format(self.__class__.__name__, name, id(self)) def Name(self): # Admittedly not the best name, but it's what Xcode uses. @@ -2288,7 +2284,7 @@ class PBXTargetDependency(XCObject): def __repr__(self): name = self._properties.get("name") or self._properties["target"].Name() - return "<%s %r at 0x%x>" % (self.__class__.__name__, name, id(self)) + return "<{} {!r} at 0x{:x}>".format(self.__class__.__name__, name, id(self)) def Name(self): # Admittedly not the best name, but it's what Xcode uses. diff --git a/gyp/pylib/gyp/xml_fix.py b/gyp/pylib/gyp/xml_fix.py index 0a945322b4..5301963669 100644 --- a/gyp/pylib/gyp/xml_fix.py +++ b/gyp/pylib/gyp/xml_fix.py @@ -39,12 +39,12 @@ def _Replacement_writexml(self, writer, indent="", addindent="", newl=""): writer.write(">%s" % newl) for node in self.childNodes: node.writexml(writer, indent + addindent, addindent, newl) - writer.write("%s%s" % (indent, self.tagName, newl)) + writer.write(f"{indent}{newl}") else: writer.write("/>%s" % newl) -class XmlFix(object): +class XmlFix: """Object to manage temporary patching of xml.dom.minidom.""" def __init__(self): diff --git a/gyp/setup.py b/gyp/setup.py index 766a7651ba..4ff447fc8b 100644 --- a/gyp/setup.py +++ b/gyp/setup.py @@ -15,7 +15,7 @@ setup( name="gyp-next", - version="0.7.0", + version="0.8.0", description="A fork of the GYP build system for use in the Node.js projects", long_description=long_description, long_description_content_type="text/markdown", @@ -25,7 +25,7 @@ package_dir={"": "pylib"}, packages=["gyp", "gyp.generator"], entry_points={"console_scripts": ["gyp=gyp:script_main"]}, - python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*", + python_requires=">=3.6", classifiers=[ "Development Status :: 3 - Alpha", "Environment :: Console", @@ -33,12 +33,10 @@ "License :: OSI Approved :: BSD License", "Natural Language :: English", "Programming Language :: Python", - "Programming Language :: Python :: 2", - "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", - "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", + "Programming Language :: Python :: 3.9", ], ) diff --git a/gyp/test_gyp.py b/gyp/test_gyp.py index 382e75272d..8ee2e48c1e 100755 --- a/gyp/test_gyp.py +++ b/gyp/test_gyp.py @@ -5,7 +5,6 @@ """gyptest.py -- test runner for GYP tests.""" -from __future__ import print_function import argparse import os @@ -153,7 +152,7 @@ def print_configuration_info(): sys.path.append(os.path.abspath("test/lib")) import TestMac - print(" Mac %s %s" % (platform.mac_ver()[0], platform.mac_ver()[2])) + print(" Mac {} {}".format(platform.mac_ver()[0], platform.mac_ver()[2])) print(" Xcode %s" % TestMac.Xcode.Version()) elif sys.platform == "win32": sys.path.append(os.path.abspath("pylib")) @@ -168,7 +167,7 @@ def print_configuration_info(): print() -class Runner(object): +class Runner: def __init__(self, formats, tests, gyp_options, verbose): self.formats = formats self.tests = tests @@ -217,10 +216,10 @@ def run_test(self, test, fmt, i): res = "skipped" elif proc.returncode: res = "failed" - self.failures.append("(%s) %s" % (test, fmt)) + self.failures.append(f"({test}) {fmt}") else: res = "passed" - res_msg = " %s %.3fs" % (res, took) + res_msg = f" {res} {took:.3f}s" self.print_(res_msg) if ( diff --git a/gyp/tools/README b/gyp/tools/README index 712e4efbb7..84a73d1521 100644 --- a/gyp/tools/README +++ b/gyp/tools/README @@ -5,7 +5,7 @@ pretty_vcproj: For example, if I want to diff the base.vcproj project: - pretty_vcproj.py z:\dev\src-chrome\src\base\build\base.vcproj "$(SolutionDir)=z:\dev\src-chrome\src\chrome\\" "$(CHROMIUM_BUILD)=" "$(CHROME_BUILD_TYPE)=" > orignal.txt + pretty_vcproj.py z:\dev\src-chrome\src\base\build\base.vcproj "$(SolutionDir)=z:\dev\src-chrome\src\chrome\\" "$(CHROMIUM_BUILD)=" "$(CHROME_BUILD_TYPE)=" > original.txt pretty_vcproj.py z:\dev\src-chrome\src\base\base_gyp.vcproj "$(SolutionDir)=z:\dev\src-chrome\src\chrome\\" "$(CHROMIUM_BUILD)=" "$(CHROME_BUILD_TYPE)=" > gyp.txt And you can use your favorite diff tool to see the changes. diff --git a/gyp/tools/graphviz.py b/gyp/tools/graphviz.py index 1f3acf37fc..9005d2c55a 100755 --- a/gyp/tools/graphviz.py +++ b/gyp/tools/graphviz.py @@ -8,7 +8,6 @@ generate input suitable for graphviz to render a dependency graph of targets.""" -from __future__ import print_function import collections import json @@ -66,7 +65,7 @@ def WriteGraph(edges): target = targets[0] build_file, target_name, toolset = ParseTarget(target) print( - ' "%s" [shape=box, label="%s\\n%s"]' % (target, filename, target_name) + f' "{target}" [shape=box, label="{filename}\\n{target_name}"]' ) else: # Group multiple nodes together in a subgraph. @@ -74,14 +73,14 @@ def WriteGraph(edges): print(' label = "%s"' % filename) for target in targets: build_file, target_name, toolset = ParseTarget(target) - print(' "%s" [label="%s"]' % (target, target_name)) + print(f' "{target}" [label="{target_name}"]') print(" }") # Now that we've placed all the nodes within subgraphs, output all # the edges between nodes. for src, dsts in edges.items(): for dst in dsts: - print(' "%s" -> "%s"' % (src, dst)) + print(f' "{src}" -> "{dst}"') print("}") diff --git a/gyp/tools/pretty_gyp.py b/gyp/tools/pretty_gyp.py index 7313b4fe1b..c8c7578fda 100755 --- a/gyp/tools/pretty_gyp.py +++ b/gyp/tools/pretty_gyp.py @@ -6,7 +6,6 @@ """Pretty-prints the contents of a GYP file.""" -from __future__ import print_function import sys import re @@ -34,7 +33,7 @@ def mask_comments(input): def quote_replace(matchobj): - return "%s%s%s%s" % ( + return "{}{}{}{}".format( matchobj.group(1), matchobj.group(2), "x" * len(matchobj.group(3)), diff --git a/gyp/tools/pretty_sln.py b/gyp/tools/pretty_sln.py index 2b1cb1de74..87c03b8880 100755 --- a/gyp/tools/pretty_sln.py +++ b/gyp/tools/pretty_sln.py @@ -12,7 +12,6 @@ Then it outputs a possible build order. """ -from __future__ import print_function import os import re diff --git a/gyp/tools/pretty_vcproj.py b/gyp/tools/pretty_vcproj.py index b171fae6cf..23ef7c9727 100755 --- a/gyp/tools/pretty_vcproj.py +++ b/gyp/tools/pretty_vcproj.py @@ -12,7 +12,6 @@ It outputs the resulting xml to stdout. """ -from __future__ import print_function import os import sys @@ -21,27 +20,22 @@ from xml.dom.minidom import Node __author__ = "nsylvain (Nicolas Sylvain)" - -try: - cmp -except NameError: - - def cmp(x, y): - return (x > y) - (x < y) +ARGUMENTS = None +REPLACEMENTS = dict() -REPLACEMENTS = dict() -ARGUMENTS = None +def cmp(x, y): + return (x > y) - (x < y) -class CmpTuple(object): +class CmpTuple: """Compare function between 2 tuple.""" def __call__(self, x, y): return cmp(x[0], y[0]) -class CmpNode(object): +class CmpNode: """Compare function between 2 xml nodes.""" def __call__(self, x, y): @@ -72,7 +66,7 @@ def get_string(node): def PrettyPrintNode(node, indent=0): if node.nodeType == Node.TEXT_NODE: if node.data.strip(): - print("%s%s" % (" " * indent, node.data.strip())) + print("{}{}".format(" " * indent, node.data.strip())) return if node.childNodes: @@ -84,23 +78,23 @@ def PrettyPrintNode(node, indent=0): # Print the main tag if attr_count == 0: - print("%s<%s>" % (" " * indent, node.nodeName)) + print("{}<{}>".format(" " * indent, node.nodeName)) else: - print("%s<%s" % (" " * indent, node.nodeName)) + print("{}<{}".format(" " * indent, node.nodeName)) all_attributes = [] for (name, value) in node.attributes.items(): all_attributes.append((name, value)) all_attributes.sort(CmpTuple()) for (name, value) in all_attributes: - print('%s %s="%s"' % (" " * indent, name, value)) + print('{} {}="{}"'.format(" " * indent, name, value)) print("%s>" % (" " * indent)) if node.nodeValue: - print("%s %s" % (" " * indent, node.nodeValue)) + print("{} {}".format(" " * indent, node.nodeValue)) for sub_node in node.childNodes: PrettyPrintNode(sub_node, indent=indent + 2) - print("%s" % (" " * indent, node.nodeName)) + print("{}".format(" " * indent, node.nodeName)) def FlattenFilter(node): From 392b7760b45af45fa958c16a4a34b873a03f2b3a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C3=ABl=20Zasso?= Date: Sun, 14 Feb 2021 05:47:47 +0100 Subject: [PATCH 188/551] lib: avoid changing process.config (#2322) PR-URL: https://github.com/nodejs/node-gyp/pull/2322 Refs: https://github.com/nodejs/node/pull/36902 Reviewed-By: Richard Lau Reviewed-By: Rod Vagg --- lib/configure.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/configure.js b/lib/configure.js index d4342b9d76..7992b658a4 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -96,7 +96,7 @@ function configure (gyp, argv, callback) { log.verbose('build/' + configFilename, 'creating config file') - var config = process.config || {} + var config = Object.assign({}, process.config) var defaults = config.target_defaults var variables = config.variables From a78b584236e92a9469f72916c55ba83e9819ddea Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Sun, 14 Feb 2021 05:51:19 +0100 Subject: [PATCH 189/551] gyp: remove support for Python 2 (#2300) PR-URL: https://github.com/nodejs/node-gyp/pull/2300 Reviewed-By: Jiawen Geng --- README.md | 14 ++++---------- gyp/pylib/gyp/MSVSSettings.py | 2 -- gyp/pylib/gyp/generator/android.py | 4 +++- test/fixtures/test-charmap.py | 7 +++---- 4 files changed, 10 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 64e82dbbe3..8c37b210db 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ Depending on your operating system, you will need to install: ### On Unix - * Python v2.7, v3.5, v3.6, v3.7, or v3.8 + * Python v3.6, v3.7, v3.8, or v3.9 * `make` * A proper C/C++ compiler toolchain, like [GCC](https://gcc.gnu.org) @@ -38,7 +38,7 @@ Depending on your operating system, you will need to install: **ATTENTION**: If your Mac has been _upgraded_ to macOS Catalina (10.15), please read [macOS_Catalina.md](macOS_Catalina.md). - * Python v2.7, v3.5, v3.6, v3.7, or v3.8 + * Python v3.6, v3.7, v3.8, or v3.9 * [Xcode](https://developer.apple.com/xcode/download/) * You also need to install the `XCode Command Line Tools` by running `xcode-select --install`. Alternatively, if you already have the full Xcode installed, you can find them under the menu `Xcode -> Open Developer Tool -> More Developer Tools...`. This step will install `clang`, `clang++`, and `make`. @@ -46,12 +46,6 @@ Depending on your operating system, you will need to install: Install the current version of Python from the [Microsoft Store package](https://docs.python.org/3/using/windows.html#the-microsoft-store-package). -#### Option 1 - -Install all the required tools and configurations using Microsoft's [windows-build-tools](https://github.com/felixrieseberg/windows-build-tools) using `npm install --global windows-build-tools` from an elevated PowerShell or CMD.exe (run as Administrator). - -#### Option 2 - Install tools and configuration manually: * Install Visual C++ Build Environment: [Visual Studio Build Tools](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) (using "Visual C++ build tools" workload) or [Visual Studio 2017 Community](https://visualstudio.microsoft.com/pl/thank-you-downloading-visual-studio/?sku=Community) @@ -64,8 +58,8 @@ Install tools and configuration manually: ### Configuring Python Dependency -`node-gyp` requires that you have installed a compatible version of Python, one of: v2.7, v3.5, v3.6, -v3.7, or v3.8. If you have multiple Python versions installed, you can identify which Python +`node-gyp` requires that you have installed a compatible version of Python, one of: v3.6, v3.7, +v3.8, or v3.9. If you have multiple Python versions installed, you can identify which Python version `node-gyp` should use in one of the following ways: 1. by setting the `--python` command-line option, e.g.: diff --git a/gyp/pylib/gyp/MSVSSettings.py b/gyp/pylib/gyp/MSVSSettings.py index e89a971a3b..87c20041e3 100644 --- a/gyp/pylib/gyp/MSVSSettings.py +++ b/gyp/pylib/gyp/MSVSSettings.py @@ -22,12 +22,10 @@ _msvs_validators = {} _msbuild_validators = {} - # A dictionary of settings converters. The key is the tool name, the value is # a dictionary mapping setting names to conversion functions. _msvs_to_msbuild_converters = {} - # Tool name mapping from MSVS to MSBuild. _msbuild_name_of_tool = {} diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py index 040d8088a2..fe1be3c858 100644 --- a/gyp/pylib/gyp/generator/android.py +++ b/gyp/pylib/gyp/generator/android.py @@ -486,7 +486,9 @@ def WriteCopies(self, copies, extra_outputs): self.LocalPathify(os.path.join(copy["destination"], filename)) ) - self.WriteLn(f"{output}: {path} $(GYP_TARGET_DEPENDENCIES) | $(ACP)") + self.WriteLn( + f"{output}: {path} $(GYP_TARGET_DEPENDENCIES) | $(ACP)" + ) self.WriteLn("\t@echo Copying: $@") self.WriteLn("\t$(hide) mkdir -p $(dir $@)") self.WriteLn("\t$(hide) $(ACP) -rpf $< $@") diff --git a/test/fixtures/test-charmap.py b/test/fixtures/test-charmap.py index b338f915bc..033eb9bcf4 100644 --- a/test/fixtures/test-charmap.py +++ b/test/fixtures/test-charmap.py @@ -1,4 +1,3 @@ -from __future__ import print_function import sys import locale @@ -18,9 +17,9 @@ def main(): pass textmap = { - 'cp936': u'\u4e2d\u6587', - 'cp1252': u'Lat\u012Bna', - 'cp932': u'\u306b\u307b\u3093\u3054' + 'cp936': '\u4e2d\u6587', + 'cp1252': 'Lat\u012Bna', + 'cp932': '\u306b\u307b\u3093\u3054' } if encoding in textmap: print(textmap[encoding]) From e81602ef55023cc8ffeebc5509a9a8649d60434d Mon Sep 17 00:00:00 2001 From: Matias Lopez Date: Wed, 17 Mar 2021 21:30:04 -0400 Subject: [PATCH 190/551] lib: migrate requests to fetch (#2220) PR-URL: https://github.com/nodejs/node-gyp/pull/2220 Reviewed-By: Jiawen Geng --- lib/install.js | 557 ++++++++++++++++++------------------------ lib/proxy.js | 92 ------- package.json | 2 +- test/test-download.js | 317 ++++++++++-------------- test/test-install.js | 54 ++-- 5 files changed, 403 insertions(+), 619 deletions(-) delete mode 100644 lib/proxy.js diff --git a/lib/install.js b/lib/install.js index f9fa2b34bd..99f6d8592a 100644 --- a/lib/install.js +++ b/lib/install.js @@ -4,55 +4,42 @@ const fs = require('graceful-fs') const os = require('os') const tar = require('tar') const path = require('path') +const util = require('util') +const stream = require('stream') const crypto = require('crypto') const log = require('npmlog') const semver = require('semver') -const request = require('request') +const fetch = require('make-fetch-happen') const processRelease = require('./process-release') const win = process.platform === 'win32' -const getProxyFromURI = require('./proxy') +const streamPipeline = util.promisify(stream.pipeline) -function install (fs, gyp, argv, callback) { - var release = processRelease(argv, gyp, process.version, process.release) +/** + * @param {typeof import('graceful-fs')} fs + */ - // ensure no double-callbacks happen - function cb (err) { - if (cb.done) { - return - } - cb.done = true - if (err) { - log.warn('install', 'got an error, rolling back install') - // roll-back the install if anything went wrong - gyp.commands.remove([release.versionDir], function () { - callback(err) - }) - } else { - callback(null, release.version) - } - } +async function install (fs, gyp, argv) { + const release = processRelease(argv, gyp, process.version, process.release) // Determine which node dev files version we are installing log.verbose('install', 'input version string %j', release.version) if (!release.semver) { // could not parse the version string with semver - return callback(new Error('Invalid version number: ' + release.version)) + throw new Error('Invalid version number: ' + release.version) } if (semver.lt(release.version, '0.8.0')) { - return callback(new Error('Minimum target version is `0.8.0` or greater. Got: ' + release.version)) + throw new Error('Minimum target version is `0.8.0` or greater. Got: ' + release.version) } // 0.x.y-pre versions are not published yet and cannot be installed. Bail. if (release.semver.prerelease[0] === 'pre') { log.verbose('detected "pre" node version', release.version) - if (gyp.opts.nodedir) { - log.verbose('--nodedir flag was passed; skipping install', gyp.opts.nodedir) - callback() - } else { - callback(new Error('"pre" versions of node cannot be installed, use the --nodedir flag instead')) + if (!gyp.opts.nodedir) { + throw new Error('"pre" versions of node cannot be installed, use the --nodedir flag instead') } + log.verbose('--nodedir flag was passed; skipping install', gyp.opts.nodedir) return } @@ -60,296 +47,225 @@ function install (fs, gyp, argv, callback) { log.verbose('install', 'installing version: %s', release.versionDir) // the directory where the dev files will be installed - var devDir = path.resolve(gyp.devDir, release.versionDir) + const devDir = path.resolve(gyp.devDir, release.versionDir) // If '--ensure' was passed, then don't *always* install the version; // check if it is already installed, and only install when needed if (gyp.opts.ensure) { log.verbose('install', '--ensure was passed, so won\'t reinstall if already installed') - fs.stat(devDir, function (err) { - if (err) { - if (err.code === 'ENOENT') { - log.verbose('install', 'version not already installed, continuing with install', release.version) - go() - } else if (err.code === 'EACCES') { - eaccesFallback(err) - } else { - cb(err) + try { + await fs.promises.stat(devDir) + } catch (err) { + if (err.code === 'ENOENT') { + log.verbose('install', 'version not already installed, continuing with install', release.version) + try { + return await go() + } catch (err) { + return rollback(err) } - return + } else if (err.code === 'EACCES') { + return eaccesFallback(err) } - log.verbose('install', 'version is already installed, need to check "installVersion"') - var installVersionFile = path.resolve(devDir, 'installVersion') - fs.readFile(installVersionFile, 'ascii', function (err, ver) { - if (err && err.code !== 'ENOENT') { - return cb(err) - } - var installVersion = parseInt(ver, 10) || 0 - log.verbose('got "installVersion"', installVersion) - log.verbose('needs "installVersion"', gyp.package.installVersion) - if (installVersion < gyp.package.installVersion) { - log.verbose('install', 'version is no good; reinstalling') - go() - } else { - log.verbose('install', 'version is good') - cb() - } - }) - }) + throw err + } + log.verbose('install', 'version is already installed, need to check "installVersion"') + const installVersionFile = path.resolve(devDir, 'installVersion') + let installVersion = 0 + try { + const ver = await fs.promises.readFile(installVersionFile, 'ascii') + installVersion = parseInt(ver, 10) || 0 + } catch (err) { + if (err.code !== 'ENOENT') { + throw err + } + } + log.verbose('got "installVersion"', installVersion) + log.verbose('needs "installVersion"', gyp.package.installVersion) + if (installVersion < gyp.package.installVersion) { + log.verbose('install', 'version is no good; reinstalling') + try { + return await go() + } catch (err) { + return rollback(err) + } + } + log.verbose('install', 'version is good') } else { - go() - } - - function getContentSha (res, callback) { - var shasum = crypto.createHash('sha256') - res.on('data', function (chunk) { - shasum.update(chunk) - }).on('end', function () { - callback(null, shasum.digest('hex')) - }) + try { + return await go() + } catch (err) { + return rollback(err) + } } - function go () { + async function go () { log.verbose('ensuring nodedir is created', devDir) // first create the dir for the node dev files - fs.mkdir(devDir, { recursive: true }, function (err, created) { - if (err) { - if (err.code === 'EACCES') { - eaccesFallback(err) - } else { - cb(err) - } - return - } + try { + const created = await fs.promises.mkdir(devDir, { recursive: true }) if (created) { log.verbose('created nodedir', created) } - - // now download the node tarball - var tarPath = gyp.opts.tarball - var badDownload = false - var extractCount = 0 - var contentShasums = {} - var expectShasums = {} - - // checks if a file to be extracted from the tarball is valid. - // only .h header files and the gyp files get extracted - function isValid (path) { - var isValid = valid(path) - if (isValid) { - log.verbose('extracted file from tarball', path) - extractCount++ - } else { - // invalid - log.silly('ignoring from tarball', path) - } - return isValid + } catch (err) { + if (err.code === 'EACCES') { + return eaccesFallback(err) } - // download the tarball and extract! - if (tarPath) { - return tar.extract({ - file: tarPath, - strip: 1, - filter: isValid, - cwd: devDir - }).then(afterTarball, cb) - } + throw err + } - try { - var req = download(gyp, process.env, release.tarballUrl) - } catch (e) { - return cb(e) + // now download the node tarball + const tarPath = gyp.opts.tarball + let extractCount = 0 + const contentShasums = {} + const expectShasums = {} + + // checks if a file to be extracted from the tarball is valid. + // only .h header files and the gyp files get extracted + function isValid (path) { + const isValid = valid(path) + if (isValid) { + log.verbose('extracted file from tarball', path) + extractCount++ + } else { + // invalid + log.silly('ignoring from tarball', path) } + return isValid + } - // something went wrong downloading the tarball? - req.on('error', function (err) { - if (err.code === 'ENOTFOUND') { - return cb(new Error('This is most likely not a problem with node-gyp or the package itself and\n' + - 'is related to network connectivity. In most cases you are behind a proxy or have bad \n' + - 'network settings.')) - } - badDownload = true - cb(err) - }) + // download the tarball and extract! - req.on('close', function () { - if (extractCount === 0) { - cb(new Error('Connection closed while downloading tarball file')) - } + if (tarPath) { + await tar.extract({ + file: tarPath, + strip: 1, + filter: isValid, + cwd: devDir }) + } else { + try { + const res = await download(gyp, release.tarballUrl) - req.on('response', function (res) { - if (res.statusCode !== 200) { - badDownload = true - cb(new Error(res.statusCode + ' response downloading ' + release.tarballUrl)) - return + if (res.status !== 200) { + throw new Error(`${res.status} response downloading ${release.tarballUrl}`) } - // content checksum - getContentSha(res, function (_, checksum) { - var filename = path.basename(release.tarballUrl).trim() - contentShasums[filename] = checksum - log.verbose('content checksum', filename, checksum) - }) - - // start unzipping and untaring - res.pipe(tar.extract({ - strip: 1, - cwd: devDir, - filter: isValid - }).on('close', afterTarball).on('error', cb)) - }) - // invoked after the tarball has finished being extracted - function afterTarball () { - if (badDownload) { - return - } - if (extractCount === 0) { - return cb(new Error('There was a fatal problem while downloading/extracting the tarball')) + await streamPipeline( + res.body, + // content checksum + new ShaSum((_, checksum) => { + const filename = path.basename(release.tarballUrl).trim() + contentShasums[filename] = checksum + log.verbose('content checksum', filename, checksum) + }), + tar.extract({ + strip: 1, + cwd: devDir, + filter: isValid + }) + ) + } catch (err) { + // something went wrong downloading the tarball? + if (err.code === 'ENOTFOUND') { + throw new Error('This is most likely not a problem with node-gyp or the package itself and\n' + + 'is related to network connectivity. In most cases you are behind a proxy or have bad \n' + + 'network settings.') } - log.verbose('tarball', 'done parsing tarball') - var async = 0 + throw err + } + } - if (win) { - // need to download node.lib - async++ - downloadNodeLib(deref) - } + // invoked after the tarball has finished being extracted + if (extractCount === 0) { + throw new Error('There was a fatal problem while downloading/extracting the tarball') + } - // write the "installVersion" file - async++ - var installVersionPath = path.resolve(devDir, 'installVersion') - fs.writeFile(installVersionPath, gyp.package.installVersion + '\n', deref) + log.verbose('tarball', 'done parsing tarball') + + const installVersionPath = path.resolve(devDir, 'installVersion') + await Promise.all([ + // need to download node.lib + ...(win ? downloadNodeLib() : []), + // write the "installVersion" file + fs.promises.writeFile(installVersionPath, gyp.package.installVersion + '\n'), + // Only download SHASUMS.txt if we downloaded something in need of SHA verification + ...(!tarPath || win ? [downloadShasums()] : []) + ]) + + log.verbose('download contents checksum', JSON.stringify(contentShasums)) + // check content shasums + for (const k in contentShasums) { + log.verbose('validating download checksum for ' + k, '(%s == %s)', contentShasums[k], expectShasums[k]) + if (contentShasums[k] !== expectShasums[k]) { + throw new Error(k + ' local checksum ' + contentShasums[k] + ' not match remote ' + expectShasums[k]) + } + } - // Only download SHASUMS.txt if we downloaded something in need of SHA verification - if (!tarPath || win) { - // download SHASUMS.txt - async++ - downloadShasums(deref) - } + async function downloadShasums () { + log.verbose('check download content checksum, need to download `SHASUMS256.txt`...') + log.verbose('checksum url', release.shasumsUrl) - if (async === 0) { - // no async tasks required - cb() - } + const res = await download(gyp, release.shasumsUrl) - function deref (err) { - if (err) { - return cb(err) - } + if (res.status !== 200) { + throw new Error(`${res.status} status code downloading checksum`) + } - async-- - if (!async) { - log.verbose('download contents checksum', JSON.stringify(contentShasums)) - // check content shasums - for (var k in contentShasums) { - log.verbose('validating download checksum for ' + k, '(%s == %s)', contentShasums[k], expectShasums[k]) - if (contentShasums[k] !== expectShasums[k]) { - cb(new Error(k + ' local checksum ' + contentShasums[k] + ' not match remote ' + expectShasums[k])) - return - } - } - cb() - } + for (const line of (await res.text()).trim().split('\n')) { + const items = line.trim().split(/\s+/) + if (items.length !== 2) { + return } + + // 0035d18e2dcf9aad669b1c7c07319e17abfe3762 ./node-v0.11.4.tar.gz + const name = items[1].replace(/^\.\//, '') + expectShasums[name] = items[0] } - function downloadShasums (done) { - log.verbose('check download content checksum, need to download `SHASUMS256.txt`...') - log.verbose('checksum url', release.shasumsUrl) - try { - var req = download(gyp, process.env, release.shasumsUrl) - } catch (e) { - return cb(e) - } + log.verbose('checksum data', JSON.stringify(expectShasums)) + } - req.on('error', done) - req.on('response', function (res) { - if (res.statusCode !== 200) { - done(new Error(res.statusCode + ' status code downloading checksum')) - return + function downloadNodeLib () { + log.verbose('on Windows; need to download `' + release.name + '.lib`...') + const archs = ['ia32', 'x64', 'arm64'] + return archs.map(async (arch) => { + const dir = path.resolve(devDir, arch) + const targetLibPath = path.resolve(dir, release.name + '.lib') + const { libUrl, libPath } = release[arch] + const name = `${arch} ${release.name}.lib` + log.verbose(name, 'dir', dir) + log.verbose(name, 'url', libUrl) + + await fs.promises.mkdir(dir, { recursive: true }) + log.verbose('streaming', name, 'to:', targetLibPath) + + const res = await download(gyp, libUrl) + + if (res.status === 403 || res.status === 404) { + if (arch === 'arm64') { + // Arm64 is a newer platform on Windows and not all node distributions provide it. + log.verbose(`${name} was not found in ${libUrl}`) + } else { + log.warn(`${name} was not found in ${libUrl}`) } + return + } else if (res.status !== 200) { + throw new Error(`${res.status} status code downloading ${name}`) + } - var chunks = [] - res.on('data', function (chunk) { - chunks.push(chunk) - }) - res.on('end', function () { - var lines = Buffer.concat(chunks).toString().trim().split('\n') - lines.forEach(function (line) { - var items = line.trim().split(/\s+/) - if (items.length !== 2) { - return - } - - // 0035d18e2dcf9aad669b1c7c07319e17abfe3762 ./node-v0.11.4.tar.gz - var name = items[1].replace(/^\.\//, '') - expectShasums[name] = items[0] - }) - - log.verbose('checksum data', JSON.stringify(expectShasums)) - done() - }) - }) - } - - function downloadNodeLib (done) { - log.verbose('on Windows; need to download `' + release.name + '.lib`...') - var archs = ['ia32', 'x64', 'arm64'] - var async = archs.length - archs.forEach(function (arch) { - var dir = path.resolve(devDir, arch) - var targetLibPath = path.resolve(dir, release.name + '.lib') - var libUrl = release[arch].libUrl - var libPath = release[arch].libPath - var name = arch + ' ' + release.name + '.lib' - log.verbose(name, 'dir', dir) - log.verbose(name, 'url', libUrl) - - fs.mkdir(dir, { recursive: true }, function (err) { - if (err) { - return done(err) - } - log.verbose('streaming', name, 'to:', targetLibPath) - - try { - var req = download(gyp, process.env, libUrl, cb) - } catch (e) { - return cb(e) - } - - req.on('error', done) - req.on('response', function (res) { - if (res.statusCode === 403 || res.statusCode === 404) { - if (arch === 'arm64') { - // Arm64 is a newer platform on Windows and not all node distributions provide it. - log.verbose(`${name} was not found in ${libUrl}`) - } else { - log.warn(`${name} was not found in ${libUrl}`) - } - return - } else if (res.statusCode !== 200) { - done(new Error(res.statusCode + ' status code downloading ' + name)) - return - } - - getContentSha(res, function (_, checksum) { - contentShasums[libPath] = checksum - log.verbose('content checksum', libPath, checksum) - }) - - var ws = fs.createWriteStream(targetLibPath) - ws.on('error', cb) - req.pipe(ws) - }) - req.on('end', function () { --async || done() }) - }) - }) - } // downloadNodeLib() - }) // mkdir() + return streamPipeline( + res.body, + new ShaSum((_, checksum) => { + contentShasums[libPath] = checksum + log.verbose('content checksum', libPath, checksum) + }), + fs.createWriteStream(targetLibPath) + ) + }) + } // downloadNodeLib() } // go() /** @@ -358,10 +274,17 @@ function install (fs, gyp, argv, callback) { function valid (file) { // header files - var extname = path.extname(file) + const extname = path.extname(file) return extname === '.h' || extname === '.gypi' } + async function rollback (err) { + log.warn('install', 'got an error, rolling back install') + // roll-back the install if anything went wrong + await util.promisify(gyp.commands.remove)([release.versionDir]) + throw err + } + /** * The EACCES fallback is a workaround for npm's `sudo` behavior, where * it drops the permissions before invoking any child processes (like @@ -371,14 +294,14 @@ function install (fs, gyp, argv, callback) { * the compilation will succeed... */ - function eaccesFallback (err) { - var noretry = '--node_gyp_internal_noretry' + async function eaccesFallback (err) { + const noretry = '--node_gyp_internal_noretry' if (argv.indexOf(noretry) !== -1) { - return cb(err) + throw err } - var tmpdir = os.tmpdir() + const tmpdir = os.tmpdir() gyp.devDir = path.resolve(tmpdir, '.node-gyp') - var userString = '' + let userString = '' try { // os.userInfo can fail on some systems, it's not critical here userString = ` ("${os.userInfo().username}")` @@ -389,59 +312,65 @@ function install (fs, gyp, argv, callback) { log.verbose('tmpdir == cwd', 'automatically will remove dev files after to save disk space') gyp.todo.push({ name: 'remove', args: argv }) } - gyp.commands.install([noretry].concat(argv), cb) + return util.promisify(gyp.commands.install)([noretry].concat(argv)) + } +} + +class ShaSum extends stream.Transform { + constructor (callback) { + super() + this._callback = callback + this._digester = crypto.createHash('sha256') + } + + _transform (chunk, _, callback) { + this._digester.update(chunk) + callback(null, chunk) + } + + _flush (callback) { + this._callback(null, this._digester.digest('hex')) + callback() } } -function download (gyp, env, url) { +async function download (gyp, url) { log.http('GET', url) - var requestOpts = { - uri: url, + const requestOpts = { headers: { - 'User-Agent': 'node-gyp v' + gyp.version + ' (node ' + process.version + ')', + 'User-Agent': `node-gyp v${gyp.version} (node ${process.version})`, Connection: 'keep-alive' - } + }, + proxy: gyp.opts.proxy, + noProxy: gyp.opts.noproxy } - var cafile = gyp.opts.cafile + const cafile = gyp.opts.cafile if (cafile) { - requestOpts.ca = readCAFile(cafile) - } - - // basic support for a proxy server - var proxyUrl = getProxyFromURI(gyp, env, url) - if (proxyUrl) { - if (/^https?:\/\//i.test(proxyUrl)) { - log.verbose('download', 'using proxy url: "%s"', proxyUrl) - requestOpts.proxy = proxyUrl - } else { - log.warn('download', 'ignoring invalid "proxy" config setting: "%s"', proxyUrl) - } + requestOpts.ca = await readCAFile(cafile) } - var req = request(requestOpts) - req.on('response', function (res) { - log.http(res.statusCode, url) - }) + const res = await fetch(url, requestOpts) + log.http(res.status, res.url) - return req + return res } -function readCAFile (filename) { +async function readCAFile (filename) { // The CA file can contain multiple certificates so split on certificate // boundaries. [\S\s]*? is used to match everything including newlines. - var ca = fs.readFileSync(filename, 'utf8') - var re = /(-----BEGIN CERTIFICATE-----[\S\s]*?-----END CERTIFICATE-----)/g + const ca = await fs.promises.readFile(filename, 'utf8') + const re = /(-----BEGIN CERTIFICATE-----[\S\s]*?-----END CERTIFICATE-----)/g return ca.match(re) } module.exports = function (gyp, argv, callback) { - return install(fs, gyp, argv, callback) + install(fs, gyp, argv).then(callback.bind(undefined, null), callback) } module.exports.test = { - download: download, - install: install, - readCAFile: readCAFile + download, + install, + readCAFile } module.exports.usage = 'Install node development files for the specified node version.' diff --git a/lib/proxy.js b/lib/proxy.js deleted file mode 100644 index 92d9ed2f7f..0000000000 --- a/lib/proxy.js +++ /dev/null @@ -1,92 +0,0 @@ -'use strict' -// Taken from https://github.com/request/request/blob/212570b/lib/getProxyFromURI.js - -const url = require('url') - -function formatHostname (hostname) { - // canonicalize the hostname, so that 'oogle.com' won't match 'google.com' - return hostname.replace(/^\.*/, '.').toLowerCase() -} - -function parseNoProxyZone (zone) { - zone = zone.trim().toLowerCase() - - var zoneParts = zone.split(':', 2) - var zoneHost = formatHostname(zoneParts[0]) - var zonePort = zoneParts[1] - var hasPort = zone.indexOf(':') > -1 - - return { hostname: zoneHost, port: zonePort, hasPort: hasPort } -} - -function uriInNoProxy (uri, noProxy) { - var port = uri.port || (uri.protocol === 'https:' ? '443' : '80') - var hostname = formatHostname(uri.hostname) - var noProxyList = noProxy.split(',') - - // iterate through the noProxyList until it finds a match. - return noProxyList.map(parseNoProxyZone).some(function (noProxyZone) { - var isMatchedAt = hostname.indexOf(noProxyZone.hostname) - var hostnameMatched = ( - isMatchedAt > -1 && - (isMatchedAt === hostname.length - noProxyZone.hostname.length) - ) - - if (noProxyZone.hasPort) { - return (port === noProxyZone.port) && hostnameMatched - } - - return hostnameMatched - }) -} - -function getProxyFromURI (gyp, env, uri) { - // If a string URI/URL was given, parse it into a URL object - if (typeof uri === 'string') { - // eslint-disable-next-line - uri = url.parse(uri) - } - - // Decide the proper request proxy to use based on the request URI object and the - // environmental variables (NO_PROXY, HTTP_PROXY, etc.) - // respect NO_PROXY environment variables (see: https://lynx.invisible-island.net/lynx2.8.7/breakout/lynx_help/keystrokes/environments.html) - - var noProxy = gyp.opts.noproxy || env.NO_PROXY || env.no_proxy || env.npm_config_noproxy || '' - - // if the noProxy is a wildcard then return null - - if (noProxy === '*') { - return null - } - - // if the noProxy is not empty and the uri is found return null - - if (noProxy !== '' && uriInNoProxy(uri, noProxy)) { - return null - } - - // Check for HTTP or HTTPS Proxy in environment Else default to null - - if (uri.protocol === 'http:') { - return gyp.opts.proxy || - env.HTTP_PROXY || - env.http_proxy || - env.npm_config_proxy || null - } - - if (uri.protocol === 'https:') { - return gyp.opts.proxy || - env.HTTPS_PROXY || - env.https_proxy || - env.HTTP_PROXY || - env.http_proxy || - env.npm_config_proxy || null - } - - // if none of that works, return null - // (What uri protocol are you using then?) - - return null -} - -module.exports = getProxyFromURI diff --git a/package.json b/package.json index 8e256f0170..1a4fdf1845 100644 --- a/package.json +++ b/package.json @@ -25,9 +25,9 @@ "env-paths": "^2.2.0", "glob": "^7.1.4", "graceful-fs": "^4.2.3", + "make-fetch-happen": "^8.0.9", "nopt": "^5.0.0", "npmlog": "^4.1.2", - "request": "^2.88.2", "rimraf": "^3.0.2", "semver": "^7.3.2", "tar": "^6.0.2", diff --git a/test/test-download.js b/test/test-download.js index fe373e3280..71a3c0d092 100644 --- a/test/test-download.js +++ b/test/test-download.js @@ -1,8 +1,9 @@ 'use strict' -const test = require('tap').test +const { test } = require('tap') const fs = require('fs') const path = require('path') +const util = require('util') const http = require('http') const https = require('https') const install = require('../lib/install') @@ -14,191 +15,142 @@ const log = require('npmlog') log.level = 'warn' -test('download over http', function (t) { +test('download over http', async (t) => { t.plan(2) - var server = http.createServer(function (req, res) { - t.strictEqual(req.headers['user-agent'], - 'node-gyp v42 (node ' + process.version + ')') + const server = http.createServer((req, res) => { + t.strictEqual(req.headers['user-agent'], `node-gyp v42 (node ${process.version})`) res.end('ok') - server.close() }) - var host = 'localhost' - server.listen(0, host, function () { - var port = this.address().port - var gyp = { - opts: {}, - version: '42' - } - var url = 'http://' + host + ':' + port - var req = install.test.download(gyp, {}, url) - req.on('response', function (res) { - var body = '' - res.setEncoding('utf8') - res.on('data', function (data) { - body += data - }) - res.on('end', function () { - t.strictEqual(body, 'ok') - }) - }) - }) + t.tearDown(() => new Promise((resolve) => server.close(resolve))) + + const host = 'localhost' + await new Promise((resolve) => server.listen(0, host, resolve)) + const { port } = server.address() + const gyp = { + opts: {}, + version: '42' + } + const url = `http://${host}:${port}` + const res = await install.test.download(gyp, url) + t.strictEqual(await res.text(), 'ok') }) -test('download over https with custom ca', function (t) { +test('download over https with custom ca', async (t) => { t.plan(3) - var cert = fs.readFileSync(path.join(__dirname, 'fixtures/server.crt'), 'utf8') - var key = fs.readFileSync(path.join(__dirname, 'fixtures/server.key'), 'utf8') + const cafile = path.join(__dirname, '/fixtures/ca.crt') + const [cert, key, ca] = await Promise.all([ + fs.promises.readFile(path.join(__dirname, 'fixtures/server.crt'), 'utf8'), + fs.promises.readFile(path.join(__dirname, 'fixtures/server.key'), 'utf8'), + install.test.readCAFile(cafile) + ]) - var cafile = path.join(__dirname, '/fixtures/ca.crt') - var ca = install.test.readCAFile(cafile) t.strictEqual(ca.length, 1) - var options = { ca: ca, cert: cert, key: key } - var server = https.createServer(options, function (req, res) { - t.strictEqual(req.headers['user-agent'], - 'node-gyp v42 (node ' + process.version + ')') + const options = { ca: ca, cert: cert, key: key } + const server = https.createServer(options, (req, res) => { + t.strictEqual(req.headers['user-agent'], `node-gyp v42 (node ${process.version})`) res.end('ok') - server.close() }) - server.on('clientError', function (err) { - throw err - }) + t.tearDown(() => new Promise((resolve) => server.close(resolve))) - var host = 'localhost' - server.listen(8000, host, function () { - var port = this.address().port - var gyp = { - opts: { cafile: cafile }, - version: '42' - } - var url = 'https://' + host + ':' + port - var req = install.test.download(gyp, {}, url) - req.on('response', function (res) { - var body = '' - res.setEncoding('utf8') - res.on('data', function (data) { - body += data - }) - res.on('end', function () { - t.strictEqual(body, 'ok') - }) - }) - }) + server.on('clientError', (err) => { throw err }) + + const host = 'localhost' + await new Promise((resolve) => server.listen(0, host, resolve)) + const { port } = server.address() + const gyp = { + opts: { cafile }, + version: '42' + } + const url = `https://${host}:${port}` + const res = await install.test.download(gyp, url) + t.strictEqual(await res.text(), 'ok') }) -test('download over http with proxy', function (t) { +test('download over http with proxy', async (t) => { t.plan(2) - var server = http.createServer(function (req, res) { - t.strictEqual(req.headers['user-agent'], - 'node-gyp v42 (node ' + process.version + ')') + const server = http.createServer((_, res) => { res.end('ok') - pserver.close(function () { - server.close() - }) }) - var pserver = http.createServer(function (req, res) { - t.strictEqual(req.headers['user-agent'], - 'node-gyp v42 (node ' + process.version + ')') + const pserver = http.createServer((req, res) => { + t.strictEqual(req.headers['user-agent'], `node-gyp v42 (node ${process.version})`) res.end('proxy ok') - server.close(function () { - pserver.close() - }) }) - var host = 'localhost' - server.listen(0, host, function () { - var port = this.address().port - pserver.listen(port + 1, host, function () { - var gyp = { - opts: { - proxy: 'http://' + host + ':' + (port + 1) - }, - version: '42' - } - var url = 'http://' + host + ':' + port - var req = install.test.download(gyp, {}, url) - req.on('response', function (res) { - var body = '' - res.setEncoding('utf8') - res.on('data', function (data) { - body += data - }) - res.on('end', function () { - t.strictEqual(body, 'proxy ok') - }) - }) - }) - }) + t.tearDown(() => Promise.all([ + new Promise((resolve) => server.close(resolve)), + new Promise((resolve) => pserver.close(resolve)) + ])) + + const host = 'localhost' + await new Promise((resolve) => server.listen(0, host, resolve)) + const { port } = server.address() + await new Promise((resolve) => pserver.listen(port + 1, host, resolve)) + const gyp = { + opts: { + proxy: `http://${host}:${port + 1}`, + noproxy: 'bad' + }, + version: '42' + } + const url = `http://${host}:${port}` + const res = await install.test.download(gyp, url) + t.strictEqual(await res.text(), 'proxy ok') }) -test('download over http with noproxy', function (t) { +test('download over http with noproxy', async (t) => { t.plan(2) - var server = http.createServer(function (req, res) { - t.strictEqual(req.headers['user-agent'], - 'node-gyp v42 (node ' + process.version + ')') + const server = http.createServer((req, res) => { + t.strictEqual(req.headers['user-agent'], `node-gyp v42 (node ${process.version})`) res.end('ok') - pserver.close(function () { - server.close() - }) }) - var pserver = http.createServer(function (req, res) { - t.strictEqual(req.headers['user-agent'], - 'node-gyp v42 (node ' + process.version + ')') + const pserver = http.createServer((_, res) => { res.end('proxy ok') - server.close(function () { - pserver.close() - }) }) - var host = 'localhost' - server.listen(0, host, function () { - var port = this.address().port - pserver.listen(port + 1, host, function () { - var gyp = { - opts: { - proxy: 'http://' + host + ':' + (port + 1), - noproxy: 'localhost' - }, - version: '42' - } - var url = 'http://' + host + ':' + port - var req = install.test.download(gyp, {}, url) - req.on('response', function (res) { - var body = '' - res.setEncoding('utf8') - res.on('data', function (data) { - body += data - }) - res.on('end', function () { - t.strictEqual(body, 'ok') - }) - }) - }) - }) + t.tearDown(() => Promise.all([ + new Promise((resolve) => server.close(resolve)), + new Promise((resolve) => pserver.close(resolve)) + ])) + + const host = 'localhost' + await new Promise((resolve) => server.listen(0, host, resolve)) + const { port } = server.address() + await new Promise((resolve) => pserver.listen(port + 1, host, resolve)) + const gyp = { + opts: { + proxy: `http://${host}:${port + 1}`, + noproxy: host + }, + version: '42' + } + const url = `http://${host}:${port}` + const res = await install.test.download(gyp, url) + t.strictEqual(await res.text(), 'ok') }) -test('download with missing cafile', function (t) { +test('download with missing cafile', async (t) => { t.plan(1) - var gyp = { + const gyp = { opts: { cafile: 'no.such.file' } } try { - install.test.download(gyp, {}, 'http://bad/') + await install.test.download(gyp, {}, 'http://bad/') } catch (e) { t.ok(/no.such.file/.test(e.message)) } }) -test('check certificate splitting', function (t) { - var cas = install.test.readCAFile(path.join(__dirname, 'fixtures/ca-bundle.crt')) +test('check certificate splitting', async (t) => { + const cas = await install.test.readCAFile(path.join(__dirname, 'fixtures/ca-bundle.crt')) t.plan(2) t.strictEqual(cas.length, 2) t.notStrictEqual(cas[0], cas[1]) @@ -206,7 +158,7 @@ test('check certificate splitting', function (t) { // only run this test if we are running a version of Node with predictable version path behavior -test('download headers (actual)', function (t) { +test('download headers (actual)', async (t) => { if (process.env.FAST_TEST || process.release.name !== 'node' || semver.prerelease(process.version) !== null || @@ -214,55 +166,42 @@ test('download headers (actual)', function (t) { return t.skip('Skipping actual download of headers due to test environment configuration') } - t.plan(17) + t.plan(12) const expectedDir = path.join(devDir, process.version.replace(/^v/, '')) - rimraf(expectedDir, (err) => { - t.ifError(err) - - const prog = gyp() - prog.parseArgv([]) - prog.devDir = devDir - log.level = 'warn' - install(prog, [], (err) => { - t.ifError(err) - - fs.readFile(path.join(expectedDir, 'installVersion'), 'utf8', (err, data) => { - t.ifError(err) - t.strictEqual(data, '9\n', 'correct installVersion') - }) - - fs.readdir(path.join(expectedDir, 'include/node'), (err, list) => { - t.ifError(err) - - t.ok(list.includes('common.gypi')) - t.ok(list.includes('config.gypi')) - t.ok(list.includes('node.h')) - t.ok(list.includes('node_version.h')) - t.ok(list.includes('openssl')) - t.ok(list.includes('uv')) - t.ok(list.includes('uv.h')) - t.ok(list.includes('v8-platform.h')) - t.ok(list.includes('v8.h')) - t.ok(list.includes('zlib.h')) - }) - - fs.readFile(path.join(expectedDir, 'include/node/node_version.h'), 'utf8', (err, contents) => { - t.ifError(err) - - const lines = contents.split('\n') - - // extract the 3 version parts from the defines to build a valid version string and - // and check them against our current env version - const version = ['major', 'minor', 'patch'].reduce((version, type) => { - const re = new RegExp(`^#define\\sNODE_${type.toUpperCase()}_VERSION`) - const line = lines.find((l) => re.test(l)) - const i = line ? parseInt(line.replace(/^[^0-9]+([0-9]+).*$/, '$1'), 10) : 'ERROR' - return `${version}${type !== 'major' ? '.' : 'v'}${i}` - }, '') - - t.strictEqual(version, process.version) - }) - }) - }) + await util.promisify(rimraf)(expectedDir) + + const prog = gyp() + prog.parseArgv([]) + prog.devDir = devDir + log.level = 'warn' + await util.promisify(install)(prog, []) + + const data = await fs.promises.readFile(path.join(expectedDir, 'installVersion'), 'utf8') + t.strictEqual(data, '9\n', 'correct installVersion') + + const list = await fs.promises.readdir(path.join(expectedDir, 'include/node')) + t.ok(list.includes('common.gypi')) + t.ok(list.includes('config.gypi')) + t.ok(list.includes('node.h')) + t.ok(list.includes('node_version.h')) + t.ok(list.includes('openssl')) + t.ok(list.includes('uv')) + t.ok(list.includes('uv.h')) + t.ok(list.includes('v8-platform.h')) + t.ok(list.includes('v8.h')) + t.ok(list.includes('zlib.h')) + + const lines = (await fs.promises.readFile(path.join(expectedDir, 'include/node/node_version.h'), 'utf8')).split('\n') + + // extract the 3 version parts from the defines to build a valid version string and + // and check them against our current env version + const version = ['major', 'minor', 'patch'].reduce((version, type) => { + const re = new RegExp(`^#define\\sNODE_${type.toUpperCase()}_VERSION`) + const line = lines.find((l) => re.test(l)) + const i = line ? parseInt(line.replace(/^[^0-9]+([0-9]+).*$/, '$1'), 10) : 'ERROR' + return `${version}${type !== 'major' ? '.' : 'v'}${i}` + }, '') + + t.strictEqual(version, process.version) }) diff --git a/test/test-install.js b/test/test-install.js index c3317155e0..5039dc992e 100644 --- a/test/test-install.js +++ b/test/test-install.js @@ -1,38 +1,46 @@ 'use strict' -const test = require('tap').test -const install = require('../lib/install').test.install +const { test } = require('tap') +const { test: { install } } = require('../lib/install') +const log = require('npmlog') -require('npmlog').level = 'error' // we expect a warning +log.level = 'error' // we expect a warning -test('EACCES retry once', function (t) { +test('EACCES retry once', async (t) => { t.plan(3) - var fs = {} - fs.stat = function (path, cb) { - var err = new Error() - err.code = 'EACCES' - cb(err) - t.ok(true) + const fs = { + promises: { + stat (_) { + const err = new Error() + err.code = 'EACCES' + t.ok(true) + throw err + } + } } - var gyp = {} - gyp.devDir = __dirname - gyp.opts = {} - gyp.opts.ensure = true - gyp.commands = {} - gyp.commands.install = function (argv, cb) { - install(fs, gyp, argv, cb) - } - gyp.commands.remove = function (argv, cb) { - cb() + const Gyp = { + devDir: __dirname, + opts: { + ensure: true + }, + commands: { + install (argv, cb) { + install(fs, Gyp, argv).then(cb, cb) + }, + remove (_, cb) { + cb() + } + } } - gyp.commands.install([], function (err) { + try { + await install(fs, Gyp, []) + } catch (err) { t.ok(true) if (/"pre" versions of node cannot be installed/.test(err.message)) { t.ok(true) - t.ok(true) } - }) + } }) From 1bd18f3e7705a5441ab1b8f22ec92e962651b39d Mon Sep 17 00:00:00 2001 From: DeeDeeG Date: Fri, 26 Mar 2021 07:57:06 -0400 Subject: [PATCH 191/551] lib: drop Python 2 support in find-python.js (#2333) Co-authored-by: Christian Clauss PR-URL: https://github.com/nodejs/node-gyp/pull/2333 Reviewed-By: Christian Clauss Reviewed-By: Jiawen Geng --- lib/find-python.js | 41 ++++++++++++++++++++++++++-------------- test/test-find-python.js | 22 +++++++++------------ 2 files changed, 36 insertions(+), 27 deletions(-) diff --git a/lib/find-python.js b/lib/find-python.js index af269de2fc..47a9aa6dd4 100644 --- a/lib/find-python.js +++ b/lib/find-python.js @@ -1,6 +1,5 @@ 'use strict' -const path = require('path') const log = require('npmlog') const semver = require('semver') const cp = require('child_process') @@ -8,6 +7,23 @@ const extend = require('util')._extend // eslint-disable-line const win = process.platform === 'win32' const logWithPrefix = require('./util').logWithPrefix +const systemDrive = process.env.SystemDrive || 'C:' +const username = process.env.USERNAME || process.env.USER || require('os').userInfo().username +const localAppData = process.env.LOCALAPPDATA || `${systemDrive}\\${username}\\AppData\\Local` +const programFiles = process.env.ProgramW6432 || process.env.ProgramFiles || `${systemDrive}\\Program Files` +const programFilesX86 = process.env['ProgramFiles(x86)'] || `${programFiles} (x86)` + +const winDefaultLocationsArray = [] +for (const majorMinor of ['39', '38', '37', '36']) { + winDefaultLocationsArray.push( + `${localAppData}\\Programs\\Python\\Python${majorMinor}\\python.exe`, + `${programFiles}\\Python${majorMinor}\\python.exe`, + `${localAppData}\\Programs\\Python\\Python${majorMinor}-32\\python.exe`, + `${programFiles}\\Python${majorMinor}-32\\python.exe`, + `${programFilesX86}\\Python${majorMinor}-32\\python.exe` + ) +} + function PythonFinder (configPython, callback) { this.callback = callback this.configPython = configPython @@ -18,17 +34,14 @@ PythonFinder.prototype = { log: logWithPrefix(log, 'find Python'), argsExecutable: ['-c', 'import sys; print(sys.executable);'], argsVersion: ['-c', 'import sys; print("%s.%s.%s" % sys.version_info[:3]);'], - semverRange: '2.7.x || >=3.5.0', + semverRange: '>=3.6.0', // These can be overridden for testing: execFile: cp.execFile, env: process.env, win: win, pyLauncher: 'py.exe', - winDefaultLocations: [ - path.join(process.env.SystemDrive || 'C:', 'Python37', 'python.exe'), - path.join(process.env.SystemDrive || 'C:', 'Python27', 'python.exe') - ], + winDefaultLocations: winDefaultLocationsArray, // Logs a message at verbose level, but also saves it to be displayed later // at error level if an error occurs. This should help diagnose the problem. @@ -96,11 +109,6 @@ PythonFinder.prototype = { before: () => { this.addLog('checking if "python" can be used') }, check: this.checkCommand, arg: 'python' - }, - { - before: () => { this.addLog('checking if "python2" can be used') }, - check: this.checkCommand, - arg: 'python2' } ] @@ -119,7 +127,7 @@ PythonFinder.prototype = { checks.push({ before: () => { this.addLog( - 'checking if the py launcher can be used to find Python') + 'checking if the py launcher can be used to find Python 3') }, check: this.checkPyLauncher }) @@ -188,10 +196,15 @@ PythonFinder.prototype = { // Distributions of Python on Windows by default install with the "py.exe" // Python launcher which is more likely to exist than the Python executable // being in the $PATH. + // Because the Python launcher supports Python 2 and Python 3, we should + // explicitly request a Python 3 version. This is done by supplying "-3" as + // the first command line argument. Since "py.exe -3" would be an invalid + // executable for "execFile", we have to use the launcher to figure out + // where the actual "python.exe" executable is located. checkPyLauncher: function checkPyLauncher (errorCallback) { this.log.verbose( - `- executing "${this.pyLauncher}" to get Python executable path`) - this.run(this.pyLauncher, this.argsExecutable, false, + `- executing "${this.pyLauncher}" to get Python 3 executable path`) + this.run(this.pyLauncher, ['-3', ...this.argsExecutable], false, function (err, execPath) { // Possible outcomes: same as checkCommand if (err) { diff --git a/test/test-find-python.js b/test/test-find-python.js index 6be887f7eb..67d0b2664f 100644 --- a/test/test-find-python.js +++ b/test/test-find-python.js @@ -16,13 +16,8 @@ test('find python', function (t) { t.strictEqual(err, null) var proc = execFile(found, ['-V'], function (err, stdout, stderr) { t.strictEqual(err, null) - if (/Python 2/.test(stderr)) { - t.strictEqual(stdout, '') - t.ok(/Python 2/.test(stderr)) - } else { - t.ok(/Python 3/.test(stdout)) - t.strictEqual(stderr, '') - } + t.ok(/Python 3/.test(stdout)) + t.strictEqual(stderr, '') }) proc.stdout.setEncoding('utf-8') proc.stderr.setEncoding('utf-8') @@ -66,7 +61,7 @@ test('find python - python', function (t) { poison(f, 'execFile') t.strictEqual(program, '/path/python') t.ok(/sys\.version_info/.test(args[1])) - cb(null, '2.7.15') + cb(null, '3.9.1') } t.strictEqual(program, process.platform === 'win32' ? '"python"' : 'python') @@ -146,13 +141,14 @@ test('find python - no python2, no python, unix', function (t) { }) test('find python - no python, use python launcher', function (t) { - t.plan(3) + t.plan(4) var f = new TestPythonFinder(null, done) f.win = true f.execFile = function (program, args, opts, cb) { if (program === 'py.exe') { + t.notEqual(args.indexOf('-3'), -1) t.notEqual(args.indexOf('-c'), -1) return cb(null, 'Z:\\snake.exe') } @@ -162,7 +158,7 @@ test('find python - no python, use python launcher', function (t) { cb(new Error('not found')) } else if (/sys\.version_info/.test(args[args.length - 1])) { if (program === 'Z:\\snake.exe') { - cb(null, '2.7.14') + cb(null, '3.9.0') } else { t.fail() } @@ -181,9 +177,9 @@ test('find python - no python, use python launcher', function (t) { test('find python - no python, no python launcher, good guess', function (t) { t.plan(2) - var re = /C:[\\/]Python37[\\/]python[.]exe/ var f = new TestPythonFinder(null, done) f.win = true + const expectedProgram = f.winDefaultLocations[0] f.execFile = function (program, args, opts, cb) { if (program === 'py.exe') { @@ -191,7 +187,7 @@ test('find python - no python, no python launcher, good guess', function (t) { } if (/sys\.executable/.test(args[args.length - 1])) { cb(new Error('not found')) - } else if (re.test(program) && + } else if (program === expectedProgram && /sys\.version_info/.test(args[args.length - 1])) { cb(null, '3.7.3') } else { @@ -202,7 +198,7 @@ test('find python - no python, no python launcher, good guess', function (t) { function done (err, python) { t.strictEqual(err, null) - t.ok(re.test(python)) + t.ok(python === expectedProgram) } }) From 0d8a6f1b1940dc0af549c0a3ca44a8059465e446 Mon Sep 17 00:00:00 2001 From: Sora Morimoto Date: Fri, 26 Mar 2021 20:59:28 +0900 Subject: [PATCH 192/551] ci: update actions/setup-node to v2 (#2302) PR-URL: https://github.com/nodejs/node-gyp/pull/2302 Reviewed-By: Christian Clauss Reviewed-By: Richard Lau Reviewed-By: Rod Vagg Reviewed-By: Jiawen Geng --- .github/workflows/tests.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 2fdf4d4c23..e4baee897f 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -5,7 +5,7 @@ on: [push, pull_request] jobs: Tests: strategy: - fail-fast: false + fail-fast: false max-parallel: 15 matrix: node: [10.x, 12.x, 14.x] @@ -16,7 +16,7 @@ jobs: - name: Checkout Repository uses: actions/checkout@v2 - name: Use Node.js ${{ matrix.node }} - uses: actions/setup-node@v1 + uses: actions/setup-node@v2 with: node-version: ${{ matrix.node }} - name: Use Python ${{ matrix.python }} From a5fd1f41e348a5995a3471c4613e1dcd79898995 Mon Sep 17 00:00:00 2001 From: Jiawen Geng <3759816+gengjiawen@users.noreply.github.com> Date: Mon, 29 Mar 2021 09:49:57 +0800 Subject: [PATCH 193/551] doc: add downloads badge (#2352) PR-URL: https://github.com/nodejs/node-gyp/pull/2352 Reviewed-By: Christian Clauss Reviewed-By: Richard Lau --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 8c37b210db..a192351318 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,7 @@ # `node-gyp` - Node.js native addon build tool [![Build Status](https://github.com/nodejs/node-gyp/workflows/Tests/badge.svg?branch=master)](https://github.com/nodejs/node-gyp/actions?query=workflow%3ATests+branch%3Amaster) +![npm](https://img.shields.io/npm/dm/node-gyp) `node-gyp` is a cross-platform command-line tool written in Node.js for compiling native addon modules for Node.js. It contains a vendored copy of the From 06ddde27f9ce96ba26d9d10b4711fbf0918826a3 Mon Sep 17 00:00:00 2001 From: DeeDeeG Date: Sat, 20 Mar 2021 13:23:35 -0400 Subject: [PATCH 194/551] deps: sync mutual dependencies with npm Sync with npm 7.7.0 PR-URL: https://github.com/nodejs/node-gyp/pull/2348 Reviewed-By: Rod Vagg Reviewed-By: Jiawen Geng --- package.json | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/package.json b/package.json index 1a4fdf1845..ee4e9534fb 100644 --- a/package.json +++ b/package.json @@ -24,13 +24,13 @@ "dependencies": { "env-paths": "^2.2.0", "glob": "^7.1.4", - "graceful-fs": "^4.2.3", - "make-fetch-happen": "^8.0.9", + "graceful-fs": "^4.2.6", + "make-fetch-happen": "^8.0.14", "nopt": "^5.0.0", "npmlog": "^4.1.2", "rimraf": "^3.0.2", - "semver": "^7.3.2", - "tar": "^6.0.2", + "semver": "^7.3.5", + "tar": "^6.1.0", "which": "^2.0.2" }, "engines": { From 0093ec8646a56856578f440b67d16ea0dafd8858 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Tue, 30 Mar 2021 08:55:13 +0200 Subject: [PATCH 195/551] gyp: Improve our flake8 linting tests PR-URL: https://github.com/nodejs/node-gyp/pull/2356 Reviewed-By: Jiawen Geng --- .github/workflows/tests.yml | 14 ++++-------- gyp/pylib/gyp/generator/msvs.py | 6 ++--- test/fixtures/test-charmap.py | 40 +++++++++++++++++---------------- update-gyp.py | 39 ++++++++++++++++---------------- 4 files changed, 48 insertions(+), 51 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index e4baee897f..3b710b9449 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -1,4 +1,4 @@ -# TODO: Line 47, enable pytest --doctest-modules +# TODO: Line 43, enable pytest --doctest-modules name: Tests on: [push, pull_request] @@ -36,16 +36,10 @@ jobs: echo 'GYP_MSVS_OVERRIDE_PATH=C:\\Dummy' >> $Env:GITHUB_ENV - name: Lint Python if: matrix.os == 'ubuntu-latest' - run: | - # stop the build if there are Python syntax errors or undefined names - flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics - # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide - flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics + run: flake8 . --ignore=E203,W503 --max-complexity=101 --max-line-length=88 --show-source --statistics - name: Run Python tests - run: | - python -m pytest + run: python -m pytest # - name: Run doctests with pytest # run: python -m pytest --doctest-modules - name: Run Node tests - run: | - npm test + run: npm test diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index e5f9dbb542..5435eb1e1f 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -3276,9 +3276,9 @@ def GetEdges(node): # append to the default value. I.e. PATH=$(PATH);other_path edges.update( { - v - for v in MSVS_VARIABLE_REFERENCE.findall(value) - if v in properties and v != node + v + for v in MSVS_VARIABLE_REFERENCE.findall(value) + if v in properties and v != node } ) return edges diff --git a/test/fixtures/test-charmap.py b/test/fixtures/test-charmap.py index 033eb9bcf4..63aa77bb48 100644 --- a/test/fixtures/test-charmap.py +++ b/test/fixtures/test-charmap.py @@ -2,28 +2,30 @@ import locale try: - reload(sys) + reload(sys) except NameError: # Python 3 - pass + pass + def main(): - encoding = locale.getdefaultlocale()[1] - if not encoding: - return False + encoding = locale.getdefaultlocale()[1] + if not encoding: + return False - try: - sys.setdefaultencoding(encoding) - except AttributeError: # Python 3 - pass + try: + sys.setdefaultencoding(encoding) + except AttributeError: # Python 3 + pass + + textmap = { + "cp936": "\u4e2d\u6587", + "cp1252": "Lat\u012Bna", + "cp932": "\u306b\u307b\u3093\u3054", + } + if encoding in textmap: + print(textmap[encoding]) + return True - textmap = { - 'cp936': '\u4e2d\u6587', - 'cp1252': 'Lat\u012Bna', - 'cp932': '\u306b\u307b\u3093\u3054' - } - if encoding in textmap: - print(textmap[encoding]) - return True -if __name__ == '__main__': - print(main()) +if __name__ == "__main__": + print(main()) diff --git a/update-gyp.py b/update-gyp.py index aa2bcb9eb9..6998718434 100755 --- a/update-gyp.py +++ b/update-gyp.py @@ -4,14 +4,13 @@ import os import shutil import subprocess -import sys import tarfile import tempfile import urllib.request BASE_URL = "https://github.com/nodejs/gyp-next/archive/" CHECKOUT_PATH = os.path.dirname(os.path.realpath(__file__)) -CHECKOUT_GYP_PATH = os.path.join(CHECKOUT_PATH, 'gyp') +CHECKOUT_GYP_PATH = os.path.join(CHECKOUT_PATH, "gyp") parser = argparse.ArgumentParser() parser.add_argument("tag", help="gyp tag to update to") @@ -21,25 +20,27 @@ changed_files = subprocess.check_output(["git", "diff", "--name-only"]).strip() if changed_files: - raise Exception("Can't update gyp while you have uncommitted changes in node-gyp") + raise Exception("Can't update gyp while you have uncommitted changes in node-gyp") with tempfile.TemporaryDirectory() as tmp_dir: - tar_file = os.path.join(tmp_dir, 'gyp.tar.gz') - unzip_target = os.path.join(tmp_dir, 'gyp') - with open(tar_file, 'wb') as f: - print("Downloading gyp-next@" + args.tag + " into temporary directory...") - print("From: " + tar_url) - with urllib.request.urlopen(tar_url) as in_file: - f.write(in_file.read()) - - print("Unzipping...") - with tarfile.open(tar_file, "r:gz") as tar_ref: - tar_ref.extractall(unzip_target) - - print("Moving to current checkout (" + CHECKOUT_PATH + ")...") - if os.path.exists(CHECKOUT_GYP_PATH): - shutil.rmtree(CHECKOUT_GYP_PATH) - shutil.move(os.path.join(unzip_target, os.listdir(unzip_target)[0]), CHECKOUT_GYP_PATH) + tar_file = os.path.join(tmp_dir, "gyp.tar.gz") + unzip_target = os.path.join(tmp_dir, "gyp") + with open(tar_file, "wb") as f: + print("Downloading gyp-next@" + args.tag + " into temporary directory...") + print("From: " + tar_url) + with urllib.request.urlopen(tar_url) as in_file: + f.write(in_file.read()) + + print("Unzipping...") + with tarfile.open(tar_file, "r:gz") as tar_ref: + tar_ref.extractall(unzip_target) + + print("Moving to current checkout (" + CHECKOUT_PATH + ")...") + if os.path.exists(CHECKOUT_GYP_PATH): + shutil.rmtree(CHECKOUT_GYP_PATH) + shutil.move( + os.path.join(unzip_target, os.listdir(unzip_target)[0]), CHECKOUT_GYP_PATH + ) subprocess.check_output(["git", "add", "gyp"], cwd=CHECKOUT_PATH) subprocess.check_output(["git", "commit", "-m", "gyp: update gyp to " + args.tag]) From 0da2e0140dbf4296d5ba7ea498fac05e74bb4bbb Mon Sep 17 00:00:00 2001 From: DeeDeeG Date: Wed, 31 Mar 2021 03:43:58 -0400 Subject: [PATCH 196/551] gyp: update gyp to v0.8.1 (#2355) PR-URL: https://github.com/nodejs/node-gyp/pull/2355 Reviewed-By: Jiawen Geng Reviewed-By: Christian Clauss --- gyp/.github/workflows/node-gyp.yml | 2 +- gyp/.github/workflows/release-please.yml | 2 +- gyp/CHANGELOG.md | 7 +++++++ gyp/gyp_main.py | 2 +- gyp/pylib/gyp/MSVSSettings.py | 2 ++ gyp/pylib/gyp/MSVSSettings_test.py | 2 +- gyp/pylib/gyp/__init__.py | 2 +- gyp/pylib/gyp/common_test.py | 2 +- gyp/pylib/gyp/easy_xml_test.py | 2 +- gyp/pylib/gyp/flock_tool.py | 2 +- gyp/pylib/gyp/generator/android.py | 4 +--- gyp/pylib/gyp/generator/msvs_test.py | 2 +- gyp/pylib/gyp/generator/ninja_test.py | 2 +- gyp/pylib/gyp/generator/xcode_test.py | 2 +- gyp/pylib/gyp/input_test.py | 2 +- gyp/pylib/gyp/mac_tool.py | 2 +- gyp/pylib/gyp/win_tool.py | 2 +- gyp/setup.py | 4 ++-- gyp/test_gyp.py | 2 +- gyp/tools/graphviz.py | 2 +- gyp/tools/pretty_gyp.py | 2 +- gyp/tools/pretty_sln.py | 2 +- gyp/tools/pretty_vcproj.py | 2 +- 23 files changed, 31 insertions(+), 24 deletions(-) diff --git a/gyp/.github/workflows/node-gyp.yml b/gyp/.github/workflows/node-gyp.yml index 78fe502bda..59d23fdffc 100644 --- a/gyp/.github/workflows/node-gyp.yml +++ b/gyp/.github/workflows/node-gyp.yml @@ -19,7 +19,7 @@ jobs: with: repository: nodejs/node-gyp path: node-gyp - - uses: actions/setup-node@v1 + - uses: actions/setup-node@v2 with: node-version: 14.x - uses: actions/setup-python@v2 diff --git a/gyp/.github/workflows/release-please.yml b/gyp/.github/workflows/release-please.yml index a414c10e15..c8e98fbb56 100644 --- a/gyp/.github/workflows/release-please.yml +++ b/gyp/.github/workflows/release-please.yml @@ -8,7 +8,7 @@ jobs: release-please: runs-on: ubuntu-latest steps: - - uses: GoogleCloudPlatform/release-please-action@v2.5.6 + - uses: GoogleCloudPlatform/release-please-action@v2 with: token: ${{ secrets.GITHUB_TOKEN }} release-type: python diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md index 52a07a29ac..2f8bbb6249 100644 --- a/gyp/CHANGELOG.md +++ b/gyp/CHANGELOG.md @@ -1,5 +1,12 @@ # Changelog +### [0.8.1](https://www.github.com/nodejs/gyp-next/compare/v0.8.0...v0.8.1) (2021-02-18) + + +### Bug Fixes + +* update shebang lines from python to python3 ([#94](https://www.github.com/nodejs/gyp-next/issues/94)) ([a1b0d41](https://www.github.com/nodejs/gyp-next/commit/a1b0d4171a8049a4ab7a614202063dec332f2df4)) + ## [0.8.0](https://www.github.com/nodejs/gyp-next/compare/v0.7.0...v0.8.0) (2021-01-15) diff --git a/gyp/gyp_main.py b/gyp/gyp_main.py index f1559089d1..f23dcdf882 100755 --- a/gyp/gyp_main.py +++ b/gyp/gyp_main.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Copyright (c) 2009 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be diff --git a/gyp/pylib/gyp/MSVSSettings.py b/gyp/pylib/gyp/MSVSSettings.py index 87c20041e3..e89a971a3b 100644 --- a/gyp/pylib/gyp/MSVSSettings.py +++ b/gyp/pylib/gyp/MSVSSettings.py @@ -22,10 +22,12 @@ _msvs_validators = {} _msbuild_validators = {} + # A dictionary of settings converters. The key is the tool name, the value is # a dictionary mapping setting names to conversion functions. _msvs_to_msbuild_converters = {} + # Tool name mapping from MSVS to MSBuild. _msbuild_name_of_tool = {} diff --git a/gyp/pylib/gyp/MSVSSettings_test.py b/gyp/pylib/gyp/MSVSSettings_test.py index 7665c68e52..6ca09687ad 100755 --- a/gyp/pylib/gyp/MSVSSettings_test.py +++ b/gyp/pylib/gyp/MSVSSettings_test.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be diff --git a/gyp/pylib/gyp/__init__.py b/gyp/pylib/gyp/__init__.py index 52549fff11..6790ef96a1 100755 --- a/gyp/pylib/gyp/__init__.py +++ b/gyp/pylib/gyp/__init__.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be diff --git a/gyp/pylib/gyp/common_test.py b/gyp/pylib/gyp/common_test.py index 0310fb266c..05344085ad 100755 --- a/gyp/pylib/gyp/common_test.py +++ b/gyp/pylib/gyp/common_test.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be diff --git a/gyp/pylib/gyp/easy_xml_test.py b/gyp/pylib/gyp/easy_xml_test.py index c21e054246..342f693a32 100755 --- a/gyp/pylib/gyp/easy_xml_test.py +++ b/gyp/pylib/gyp/easy_xml_test.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Copyright (c) 2011 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be diff --git a/gyp/pylib/gyp/flock_tool.py b/gyp/pylib/gyp/flock_tool.py index c649a89ef8..1cb9815263 100755 --- a/gyp/pylib/gyp/flock_tool.py +++ b/gyp/pylib/gyp/flock_tool.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Copyright (c) 2011 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py index fe1be3c858..040d8088a2 100644 --- a/gyp/pylib/gyp/generator/android.py +++ b/gyp/pylib/gyp/generator/android.py @@ -486,9 +486,7 @@ def WriteCopies(self, copies, extra_outputs): self.LocalPathify(os.path.join(copy["destination"], filename)) ) - self.WriteLn( - f"{output}: {path} $(GYP_TARGET_DEPENDENCIES) | $(ACP)" - ) + self.WriteLn(f"{output}: {path} $(GYP_TARGET_DEPENDENCIES) | $(ACP)") self.WriteLn("\t@echo Copying: $@") self.WriteLn("\t$(hide) mkdir -p $(dir $@)") self.WriteLn("\t$(hide) $(ACP) -rpf $< $@") diff --git a/gyp/pylib/gyp/generator/msvs_test.py b/gyp/pylib/gyp/generator/msvs_test.py index 69a5c7ec0e..e80b57f06a 100755 --- a/gyp/pylib/gyp/generator/msvs_test.py +++ b/gyp/pylib/gyp/generator/msvs_test.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. diff --git a/gyp/pylib/gyp/generator/ninja_test.py b/gyp/pylib/gyp/generator/ninja_test.py index abadcd9828..7d180685b2 100644 --- a/gyp/pylib/gyp/generator/ninja_test.py +++ b/gyp/pylib/gyp/generator/ninja_test.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be diff --git a/gyp/pylib/gyp/generator/xcode_test.py b/gyp/pylib/gyp/generator/xcode_test.py index 51fbca6a2a..49772d1f4d 100644 --- a/gyp/pylib/gyp/generator/xcode_test.py +++ b/gyp/pylib/gyp/generator/xcode_test.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Copyright (c) 2013 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be diff --git a/gyp/pylib/gyp/input_test.py b/gyp/pylib/gyp/input_test.py index 6672ddc014..a18f72e9eb 100755 --- a/gyp/pylib/gyp/input_test.py +++ b/gyp/pylib/gyp/input_test.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Copyright 2013 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be diff --git a/gyp/pylib/gyp/mac_tool.py b/gyp/pylib/gyp/mac_tool.py index 1cd8e3798f..59647c9a89 100755 --- a/gyp/pylib/gyp/mac_tool.py +++ b/gyp/pylib/gyp/mac_tool.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. diff --git a/gyp/pylib/gyp/win_tool.py b/gyp/pylib/gyp/win_tool.py index 7e85946720..4dbcda50a4 100755 --- a/gyp/pylib/gyp/win_tool.py +++ b/gyp/pylib/gyp/win_tool.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be diff --git a/gyp/setup.py b/gyp/setup.py index 4ff447fc8b..de0e3f5f61 100644 --- a/gyp/setup.py +++ b/gyp/setup.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Copyright (c) 2009 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be @@ -15,7 +15,7 @@ setup( name="gyp-next", - version="0.8.0", + version="0.8.1", description="A fork of the GYP build system for use in the Node.js projects", long_description=long_description, long_description_content_type="text/markdown", diff --git a/gyp/test_gyp.py b/gyp/test_gyp.py index 8ee2e48c1e..757d2fc0b0 100755 --- a/gyp/test_gyp.py +++ b/gyp/test_gyp.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. diff --git a/gyp/tools/graphviz.py b/gyp/tools/graphviz.py index 9005d2c55a..f19426b69f 100755 --- a/gyp/tools/graphviz.py +++ b/gyp/tools/graphviz.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Copyright (c) 2011 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be diff --git a/gyp/tools/pretty_gyp.py b/gyp/tools/pretty_gyp.py index c8c7578fda..4ffa444551 100755 --- a/gyp/tools/pretty_gyp.py +++ b/gyp/tools/pretty_gyp.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be diff --git a/gyp/tools/pretty_sln.py b/gyp/tools/pretty_sln.py index 87c03b8880..6ca0cd12a7 100755 --- a/gyp/tools/pretty_sln.py +++ b/gyp/tools/pretty_sln.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be diff --git a/gyp/tools/pretty_vcproj.py b/gyp/tools/pretty_vcproj.py index 23ef7c9727..00d32debda 100755 --- a/gyp/tools/pretty_vcproj.py +++ b/gyp/tools/pretty_vcproj.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Copyright (c) 2012 Google Inc. All rights reserved. # Use of this source code is governed by a BSD-style license that can be From 989abc7ec2a3f618c70405600e5f6380e331fb8a Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Thu, 18 Mar 2021 16:10:48 +1100 Subject: [PATCH 197/551] v8.0.0: bump version and update changelog --- CHANGELOG.md | 19 +++++++++++++++++++ package.json | 2 +- 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 733a4b5dd6..e1c56131e7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,22 @@ +v8.0.0 2021-03-19 +================= + +* [[`0d8a6f1b19`](https://github.com/nodejs/node-gyp/commit/0d8a6f1b19)] - **ci**: update actions/setup-node to v2 (#2302) (Sora Morimoto) [#2302](https://github.com/nodejs/node-gyp/pull/2302) +* [[`15a5c7d45b`](https://github.com/nodejs/node-gyp/commit/15a5c7d45b)] - **ci**: migrate deprecated grammar (#2285) (Jiawen Geng) [#2285](https://github.com/nodejs/node-gyp/pull/2285) +* [[`06ddde27f9`](https://github.com/nodejs/node-gyp/commit/06ddde27f9)] - **deps**: sync mutual dependencies with npm (DeeDeeG) [#2348](https://github.com/nodejs/node-gyp/pull/2348) +* [[`a5fd1f41e3`](https://github.com/nodejs/node-gyp/commit/a5fd1f41e3)] - **doc**: add downloads badge (#2352) (Jiawen Geng) [#2352](https://github.com/nodejs/node-gyp/pull/2352) +* [[`cc1cbce056`](https://github.com/nodejs/node-gyp/commit/cc1cbce056)] - **doc**: update macOS\_Catalina.md (#2293) (iMrLopez) [#2293](https://github.com/nodejs/node-gyp/pull/2293) +* [[`6287118fc4`](https://github.com/nodejs/node-gyp/commit/6287118fc4)] - **doc**: updated README.md to copy easily (#2281) (மனோஜ்குமார் பழனிச்சாமி) [#2281](https://github.com/nodejs/node-gyp/pull/2281) +* [[`66c0f04467`](https://github.com/nodejs/node-gyp/commit/66c0f04467)] - **doc**: add missing `sudo` to Catalina doc (Karl Horky) [#2244](https://github.com/nodejs/node-gyp/pull/2244) +* [[`0da2e0140d`](https://github.com/nodejs/node-gyp/commit/0da2e0140d)] - **gyp**: update gyp to v0.8.1 (#2355) (DeeDeeG) [#2355](https://github.com/nodejs/node-gyp/pull/2355) +* [[`0093ec8646`](https://github.com/nodejs/node-gyp/commit/0093ec8646)] - **gyp**: Improve our flake8 linting tests (Christian Clauss) [#2356](https://github.com/nodejs/node-gyp/pull/2356) +* [[`a78b584236`](https://github.com/nodejs/node-gyp/commit/a78b584236)] - **(SEMVER-MAJOR)** **gyp**: remove support for Python 2 (#2300) (Christian Clauss) [#2300](https://github.com/nodejs/node-gyp/pull/2300) +* [[`c3c510d89e`](https://github.com/nodejs/node-gyp/commit/c3c510d89e)] - **gyp**: update gyp to v0.8.0 (#2318) (Christian Clauss) [#2318](https://github.com/nodejs/node-gyp/pull/2318) +* [[`9e1397c52e`](https://github.com/nodejs/node-gyp/commit/9e1397c52e)] - **(SEMVER-MAJOR)** **gyp**: update gyp to v0.7.0 (#2284) (Jiawen Geng) [#2284](https://github.com/nodejs/node-gyp/pull/2284) +* [[`1bd18f3e77`](https://github.com/nodejs/node-gyp/commit/1bd18f3e77)] - **(SEMVER-MAJOR)** **lib**: drop Python 2 support in find-python.js (#2333) (DeeDeeG) [#2333](https://github.com/nodejs/node-gyp/pull/2333) +* [[`e81602ef55`](https://github.com/nodejs/node-gyp/commit/e81602ef55)] - **(SEMVER-MAJOR)** **lib**: migrate requests to fetch (#2220) (Matias Lopez) [#2220](https://github.com/nodejs/node-gyp/pull/2220) +* [[`392b7760b4`](https://github.com/nodejs/node-gyp/commit/392b7760b4)] - **lib**: avoid changing process.config (#2322) (Michaël Zasso) [#2322](https://github.com/nodejs/node-gyp/pull/2322) + v7.1.2 2020-10-17 ================= diff --git a/package.json b/package.json index ee4e9534fb..ee15c98c80 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "7.1.2", + "version": "8.0.0", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From 4b83c3de7300457919d53f26d96ea9ad6f6bedd8 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Sat, 3 Apr 2021 11:32:48 +1100 Subject: [PATCH 198/551] doc: fix v8.0.0 release date PR-URL: https://github.com/nodejs/node-gyp/pull/2346 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e1c56131e7..5447c6cd12 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,4 @@ -v8.0.0 2021-03-19 +v8.0.0 2021-04-03 ================= * [[`0d8a6f1b19`](https://github.com/nodejs/node-gyp/commit/0d8a6f1b19)] - **ci**: update actions/setup-node to v2 (#2302) (Sora Morimoto) [#2302](https://github.com/nodejs/node-gyp/pull/2302) From 07e9d7c7ee80ba119ea760c635f72fd8e7efe198 Mon Sep 17 00:00:00 2001 From: DeeDeeG Date: Wed, 19 May 2021 03:22:00 -0400 Subject: [PATCH 199/551] meta: add `release-please-action` for automated releases (#2395) Co-authored-by: gengjiawen --- .github/PULL_REQUEST_TEMPLATE.md | 2 +- .github/workflows/release-please.yml | 56 +++++++++++++ CHANGELOG.md | 120 +++++++++------------------ 3 files changed, 98 insertions(+), 80 deletions(-) create mode 100644 .github/workflows/release-please.yml diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 10156d89af..bcc4bb1a04 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -10,7 +10,7 @@ Contributor guide: https://github.com/nodejs/node/blob/master/CONTRIBUTING.md - [ ] `npm install && npm test` passes - [ ] tests are included - [ ] documentation is changed or added -- [ ] commit message follows [commit guidelines](https://github.com/nodejs/node/blob/master/doc/guides/contributing/pull-requests.md#commit-message-guidelines) +- [ ] commit message follows [commit guidelines](https://github.com/googleapis/release-please#how-should-i-write-my-commits) ##### Description of change diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml new file mode 100644 index 0000000000..7d3cf9dd45 --- /dev/null +++ b/.github/workflows/release-please.yml @@ -0,0 +1,56 @@ +name: release-please + +on: + push: + branches: + - master + +jobs: + release-please: + runs-on: ubuntu-latest + steps: + - uses: google-github-actions/release-please-action@v2 + id: release + with: + package-name: node-gyp + release-type: node + changelog-types: > + [{"type":"feat","section":"Features","hidden":false}, + {"type":"fix","section":"Bug Fixes","hidden":false}, + {"type":"bin","section":"Core","hidden":false}, + {"type":"gyp","section":"Core","hidden":false}, + {"type":"lib","section":"Core","hidden":false}, + {"type":"src","section":"Core","hidden":false}, + {"type":"test","section":"Tests","hidden":false}, + {"type":"build","section":"Core","hidden":false}, + {"type":"clean","section":"Core","hidden":false}, + {"type":"configure","section":"Core","hidden":false}, + {"type":"install","section":"Core","hidden":false}, + {"type":"list","section":"Core","hidden":false}, + {"type":"rebuild","section":"Core","hidden":false}, + {"type":"remove","section":"Core","hidden":false}, + {"type":"deps","section":"Core","hidden":false}, + {"type":"python","section":"Core","hidden":false}, + {"type":"lin","section":"Core","hidden":false}, + {"type":"linux","section":"Core","hidden":false}, + {"type":"mac","section":"Core","hidden":false}, + {"type":"macos","section":"Core","hidden":false}, + {"type":"win","section":"Core","hidden":false}, + {"type":"windows","section":"Core","hidden":false}, + {"type":"zos","section":"Core","hidden":false}, + {"type":"doc","section":"Doc","hidden":false}, + {"type":"docs","section":"Doc","hidden":false}, + {"type":"readme","section":"Doc","hidden":false}, + {"type":"chore","section":"Miscellaneous","hidden":false}, + {"type":"refactor","section":"Miscellaneous","hidden":false}, + {"type":"ci","section":"Miscellaneous","hidden":false}, + {"type":"meta","section":"Miscellaneous","hidden":false}] + + # Standard Conventional Commits: `feat` and `fix` + # node-gyp subdirectories: `bin`, `gyp`, `lib`, `src`, `test` + # node-gyp subcommands: `build`, `clean`, `configure`, `install`, `list`, `rebuild`, `remove` + # Core abstract category: `deps` + # Languages/platforms: `python`, `lin`, `linux`, `mac`, `macos`, `win`, `window`, `zos` + # Documentation: `doc`, `docs`, `readme` + # Standard Conventional Commits: `chore` (under "Miscellaneous") + # Miscellaneous abstract categories: `refactor`, `ci`, `meta` diff --git a/CHANGELOG.md b/CHANGELOG.md index 5447c6cd12..77a71506fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,6 @@ -v8.0.0 2021-04-03 -================= +# Changelog + +## v8.0.0 2021-04-03 * [[`0d8a6f1b19`](https://github.com/nodejs/node-gyp/commit/0d8a6f1b19)] - **ci**: update actions/setup-node to v2 (#2302) (Sora Morimoto) [#2302](https://github.com/nodejs/node-gyp/pull/2302) * [[`15a5c7d45b`](https://github.com/nodejs/node-gyp/commit/15a5c7d45b)] - **ci**: migrate deprecated grammar (#2285) (Jiawen Geng) [#2285](https://github.com/nodejs/node-gyp/pull/2285) @@ -17,14 +18,12 @@ v8.0.0 2021-04-03 * [[`e81602ef55`](https://github.com/nodejs/node-gyp/commit/e81602ef55)] - **(SEMVER-MAJOR)** **lib**: migrate requests to fetch (#2220) (Matias Lopez) [#2220](https://github.com/nodejs/node-gyp/pull/2220) * [[`392b7760b4`](https://github.com/nodejs/node-gyp/commit/392b7760b4)] - **lib**: avoid changing process.config (#2322) (Michaël Zasso) [#2322](https://github.com/nodejs/node-gyp/pull/2322) -v7.1.2 2020-10-17 -================= +## v7.1.2 2020-10-17 * [[`096e3aded5`](https://github.com/nodejs/node-gyp/commit/096e3aded5)] - **gyp**: update gyp to 0.6.2 (Myles Borins) [#2241](https://github.com/nodejs/node-gyp/pull/2241) * [[`54f97cd243`](https://github.com/nodejs/node-gyp/commit/54f97cd243)] - **doc**: add cmd to reset `xcode-select` to initial state (Valera Rozuvan) [#2235](https://github.com/nodejs/node-gyp/pull/2235) -v7.1.1 2020-10-15 -================= +## v7.1.1 2020-10-15 This release restores the location of shared library builds to the pre-v7 location. In v7.0.0 until this release, shared library outputs were placed @@ -41,8 +40,7 @@ We consider this a bug-fix rather than semver-major change. * [[`2317dc400c`](https://github.com/nodejs/node-gyp/commit/2317dc400c)] - **ci**: switch to GitHub Actions (Shelley Vohr) [#2210](https://github.com/nodejs/node-gyp/pull/2210) * [[`2cca9b74f7`](https://github.com/nodejs/node-gyp/commit/2cca9b74f7)] - **doc**: drop the --production flag for installing windows-build-tools (DeeDeeG) [#2206](https://github.com/nodejs/node-gyp/pull/2206) -v7.1.0 2020-08-12 -================= +## v7.1.0 2020-08-12 * [[`aaf33c3029`](https://github.com/nodejs/node-gyp/commit/aaf33c3029)] - **build**: add update-gyp script (Samuel Attard) [#2167](https://github.com/nodejs/node-gyp/pull/2167) * * [[`3baa4e4172`](https://github.com/nodejs/node-gyp/commit/3baa4e4172)] - **(SEMVER-MINOR)** **gyp**: update gyp to 0.4.0 (Samuel Attard) [#2165](https://github.com/nodejs/node-gyp/pull/2165) @@ -51,8 +49,7 @@ v7.1.0 2020-08-12 * * [[`4fc8ff179d`](https://github.com/nodejs/node-gyp/commit/4fc8ff179d)] - **doc**: silence curl for macOS Catalina acid test (Chia Wei Ong) [#2150](https://github.com/nodejs/node-gyp/pull/2150) * * [[`7857cb2eb1`](https://github.com/nodejs/node-gyp/commit/7857cb2eb1)] - **deps**: increase "engines" to "node" : "\>= 10.12.0" (DeeDeeG) [#2153](https://github.com/nodejs/node-gyp/pull/2153) -v7.0.0 2020-06-03 -================= +## v7.0.0 2020-06-03 * [[`e18a61afc1`](https://github.com/nodejs/node-gyp/commit/e18a61afc1)] - **build**: shrink bloated addon binaries on windows (Shelley Vohr) [#2060](https://github.com/nodejs/node-gyp/pull/2060) * [[`4937722cf5`](https://github.com/nodejs/node-gyp/commit/4937722cf5)] - **(SEMVER-MAJOR)** **deps**: replace mkdirp with {recursive} mkdir (Rod Vagg) [#2123](https://github.com/nodejs/node-gyp/pull/2123) @@ -79,8 +76,7 @@ v7.0.0 2020-06-03 * [[`741ab096d5`](https://github.com/nodejs/node-gyp/commit/741ab096d5)] - **test**: remove support for EOL versions of Node.js (Shelley Vohr) * [[`ca86ef2539`](https://github.com/nodejs/node-gyp/commit/ca86ef2539)] - **test**: bump actions/checkout from v1 to v2 (BSKY) [#2063](https://github.com/nodejs/node-gyp/pull/2063) -v6.1.0 2020-01-08 -================= +## v6.1.0 2020-01-08 * [[`9a7dd16b76`](https://github.com/nodejs/node-gyp/commit/9a7dd16b76)] - **doc**: remove backticks from Python version list (Rod Vagg) [#2011](https://github.com/nodejs/node-gyp/pull/2011) * [[`26cd6eaea6`](https://github.com/nodejs/node-gyp/commit/26cd6eaea6)] - **doc**: add GitHub Actions badge (#1994) (Rod Vagg) [#1994](https://github.com/nodejs/node-gyp/pull/1994) @@ -101,8 +97,7 @@ v6.1.0 2020-01-08 * [[`0670e5189d`](https://github.com/nodejs/node-gyp/commit/0670e5189d)] - **test**: add header download test (Rod Vagg) [#1796](https://github.com/nodejs/node-gyp/pull/1796) * [[`c506a6a150`](https://github.com/nodejs/node-gyp/commit/c506a6a150)] - **test**: configure proper devDir for invoking configure() (Rod Vagg) [#1796](https://github.com/nodejs/node-gyp/pull/1796) -v6.0.1 2019-11-01 -================= +## v6.0.1 2019-11-01 * [[`8ec2e681d5`](https://github.com/nodejs/node-gyp/commit/8ec2e681d5)] - **doc**: add macOS\_Catalina.md document (cclauss) [#1940](https://github.com/nodejs/node-gyp/pull/1940) * [[`1b11be63cc`](https://github.com/nodejs/node-gyp/commit/1b11be63cc)] - **gyp**: python3 fixes: utf8 decode, use of 'None' in eval (Wilfried Goesgens) [#1925](https://github.com/nodejs/node-gyp/pull/1925) @@ -119,8 +114,7 @@ v6.0.1 2019-11-01 * [[`032db2a2d0`](https://github.com/nodejs/node-gyp/commit/032db2a2d0)] - **lib,install**: always download SHA sums on Windows (Sam Hughes) [#1926](https://github.com/nodejs/node-gyp/pull/1926) * [[`5a83630c33`](https://github.com/nodejs/node-gyp/commit/5a83630c33)] - **travis**: add Windows + Python 3.8 to the mix (Rod Vagg) [#1921](https://github.com/nodejs/node-gyp/pull/1921) -v6.0.0 2019-10-04 -================= +## v6.0.0 2019-10-04 * [[`dd0e97ef0b`](https://github.com/nodejs/node-gyp/commit/dd0e97ef0b)] - **(SEMVER-MAJOR)** **lib**: try to find `python` after `python3` (Sam Roberts) [#1907](https://github.com/nodejs/node-gyp/pull/1907) * [[`f60ed47d14`](https://github.com/nodejs/node-gyp/commit/f60ed47d14)] - **travis**: add Python 3.5 and 3.6 tests on Linux (Christian Clauss) [#1903](https://github.com/nodejs/node-gyp/pull/1903) @@ -128,8 +122,7 @@ v6.0.0 2019-10-04 * [[`3d1c60ab81`](https://github.com/nodejs/node-gyp/commit/3d1c60ab81)] - **(SEMVER-MAJOR)** **lib**: accept Python 3 by default (João Reis) [#1844](https://github.com/nodejs/node-gyp/pull/1844) * [[`c6e3b65a23`](https://github.com/nodejs/node-gyp/commit/c6e3b65a23)] - **(SEMVER-MAJOR)** **lib**: raise the minimum Python version from 2.6 to 2.7 (cclauss) [#1818](https://github.com/nodejs/node-gyp/pull/1818) -v5.1.1 2020-05-25 -================= +## v5.1.1 2020-05-25 * [[`bdd3a79abe`](https://github.com/nodejs/node-gyp/commit/bdd3a79abe)] - **build**: shrink bloated addon binaries on windows (Shelley Vohr) [#2060](https://github.com/nodejs/node-gyp/pull/2060) * [[`1f2ba75bc0`](https://github.com/nodejs/node-gyp/commit/1f2ba75bc0)] - **doc**: add macOS Catalina software update info (Karl Horky) [#2078](https://github.com/nodejs/node-gyp/pull/2078) @@ -142,8 +135,7 @@ v5.1.1 2020-05-25 * [[`2b6fc3c8d6`](https://github.com/nodejs/node-gyp/commit/2b6fc3c8d6)] - **doc, bin**: stop suggesting opening node-gyp issues (Bartosz Sosnowski) [#2096](https://github.com/nodejs/node-gyp/pull/2096) * [[`a876ae58ad`](https://github.com/nodejs/node-gyp/commit/a876ae58ad)] - **test**: bump actions/checkout from v1 to v2 (BSKY) [#2063](https://github.com/nodejs/node-gyp/pull/2063) -v5.1.0 2020-02-05 -================= +## v5.1.0 2020-02-05 * [[`f37a8b40d0`](https://github.com/nodejs/node-gyp/commit/f37a8b40d0)] - **doc**: add GitHub Actions badge (#1994) (Rod Vagg) [#1994](https://github.com/nodejs/node-gyp/pull/1994) * [[`cb3f6aae5e`](https://github.com/nodejs/node-gyp/commit/cb3f6aae5e)] - **doc**: update macOS\_Catalina.md (#1992) (James Home) [#1992](https://github.com/nodejs/node-gyp/pull/1992) @@ -160,13 +152,11 @@ v5.1.0 2020-02-05 * [[`32c8744b34`](https://github.com/nodejs/node-gyp/commit/32c8744b34)] - **test**: fix macOS Travis on Python 2.7 & 3.7 (Christian Clauss) [#1979](https://github.com/nodejs/node-gyp/pull/1979) * [[`fd4b1351e4`](https://github.com/nodejs/node-gyp/commit/fd4b1351e4)] - **test**: initial Github Actions with Ubuntu & macOS (Christian Clauss) [#1985](https://github.com/nodejs/node-gyp/pull/1985) -v5.0.7 2019-12-16 -================= +## v5.0.7 2019-12-16 Republish of v5.0.6 with unnecessary tarball removed from pack file. -v5.0.6 2019-12-16 -================= +## v5.0.6 2019-12-16 * [[`cdec00286f`](https://github.com/nodejs/node-gyp/commit/cdec00286f)] - **doc**: adjustments to the README.md for new users (Dan Pike) [#1919](https://github.com/nodejs/node-gyp/pull/1919) * [[`b7c8233ef2`](https://github.com/nodejs/node-gyp/commit/b7c8233ef2)] - **test**: fix Python unittests (cclauss) [#1961](https://github.com/nodejs/node-gyp/pull/1961) @@ -187,8 +177,7 @@ v5.0.6 2019-12-16 * [[`7edf7658fa`](https://github.com/nodejs/node-gyp/commit/7edf7658fa)] - **lib,install**: always download SHA sums on Windows (Sam Hughes) [#1926](https://github.com/nodejs/node-gyp/pull/1926) * [[`69056d04fe`](https://github.com/nodejs/node-gyp/commit/69056d04fe)] - **travis**: add Windows + Python 3.8 to the mix (Rod Vagg) [#1921](https://github.com/nodejs/node-gyp/pull/1921) -v5.0.5 2019-10-04 -================= +## v5.0.5 2019-10-04 * [[`3891391746`](https://github.com/nodejs/node-gyp/commit/3891391746)] - **doc**: reconcile README with Python 3 compat changes (Rod Vagg) [#1911](https://github.com/nodejs/node-gyp/pull/1911) * [[`07f81f1920`](https://github.com/nodejs/node-gyp/commit/07f81f1920)] - **lib**: accept Python 3 after Python 2 (Sam Roberts) [#1910](https://github.com/nodejs/node-gyp/pull/1910) @@ -200,8 +189,7 @@ v5.0.5 2019-10-04 * [[`53ee7dfe89`](https://github.com/nodejs/node-gyp/commit/53ee7dfe89)] - **gyp**: fix undefined name: cflags --\> ldflags (Christian Clauss) [#1901](https://github.com/nodejs/node-gyp/pull/1901) * [[`5871dcf6c9`](https://github.com/nodejs/node-gyp/commit/5871dcf6c9)] - **src,win**: add support for fetching arm64 node.lib (Richard Townsend) [#1875](https://github.com/nodejs/node-gyp/pull/1875) -v5.0.4 2019-09-27 -================= +## v5.0.4 2019-09-27 * [[`1236869ffc`](https://github.com/nodejs/node-gyp/commit/1236869ffc)] - **gyp**: modify XcodeVersion() to convert "4.2" to "0420" and "10.0" to "1000" (Christian Clauss) [#1895](https://github.com/nodejs/node-gyp/pull/1895) * [[`36638afe48`](https://github.com/nodejs/node-gyp/commit/36638afe48)] - **gyp**: more decode stdout on Python 3 (cclauss) [#1894](https://github.com/nodejs/node-gyp/pull/1894) @@ -224,8 +212,7 @@ v5.0.4 2019-09-27 * [[`fa0ed4aa42`](https://github.com/nodejs/node-gyp/commit/fa0ed4aa42)] - **build**: more Python 3 compat, replace compile with ast (cclauss) [#1820](https://github.com/nodejs/node-gyp/pull/1820) * [[`18d5c7c9d0`](https://github.com/nodejs/node-gyp/commit/18d5c7c9d0)] - **win,src**: update win\_delay\_load\_hook.cc to work with /clr (Ivan Petrovic) [#1819](https://github.com/nodejs/node-gyp/pull/1819) -v5.0.3 2019-07-17 -================= +## v5.0.3 2019-07-17 * [[`66ad305775`](https://github.com/nodejs/node-gyp/commit/66ad305775)] - **python**: accept Python 3 conditionally (João Reis) [#1815](https://github.com/nodejs/node-gyp/pull/1815) * [[`7e7fce3fed`](https://github.com/nodejs/node-gyp/commit/7e7fce3fed)] - **python**: move Python detection to its own file (João Reis) [#1815](https://github.com/nodejs/node-gyp/pull/1815) @@ -236,8 +223,7 @@ v5.0.3 2019-07-17 * [[`24109148df`](https://github.com/nodejs/node-gyp/commit/24109148df)] - **test**: downgrade to tap@^12 for continued Node 6 support (Rod Vagg) [#1808](https://github.com/nodejs/node-gyp/pull/1808) * [[`656117cc4a`](https://github.com/nodejs/node-gyp/commit/656117cc4a)] - **win**: make VS path match case-insensitive (João Reis) [#1806](https://github.com/nodejs/node-gyp/pull/1806) -v5.0.2 2019-06-27 -================= +## v5.0.2 2019-06-27 * [[`2761afbf73`](https://github.com/nodejs/node-gyp/commit/2761afbf73)] - **build,test**: add duplicate symbol test (Gabriel Schulhof) [#1689](https://github.com/nodejs/node-gyp/pull/1689) * [[`82f129d6de`](https://github.com/nodejs/node-gyp/commit/82f129d6de)] - **gyp**: replace optparse to argparse (KiYugadgeter) [#1591](https://github.com/nodejs/node-gyp/pull/1591) @@ -256,14 +242,12 @@ v5.0.2 2019-06-27 * [[`1597c84aad`](https://github.com/nodejs/node-gyp/commit/1597c84aad)] - **test**: use Travis CI to run tests on every pull request (cclauss) [#1752](https://github.com/nodejs/node-gyp/pull/1752) * [[`dd9bf929ac`](https://github.com/nodejs/node-gyp/commit/dd9bf929ac)] - **zos**: update compiler options (Shuowang (Wayne) Zhang) [#1768](https://github.com/nodejs/node-gyp/pull/1768) -v5.0.1 2019-06-20 -================= +## v5.0.1 2019-06-20 * [[`e3861722ed`](https://github.com/nodejs/node-gyp/commit/e3861722ed)] - **doc**: document --jobs max (David Sanders) [#1770](https://github.com/nodejs/node-gyp/pull/1770) * [[`1cfdb28886`](https://github.com/nodejs/node-gyp/commit/1cfdb28886)] - **lib**: reintroduce support for iojs file naming for releases \>= 1 && \< 4 (Samuel Attard) [#1777](https://github.com/nodejs/node-gyp/pull/1777) -v5.0.0 2019-06-13 -================= +## v5.0.0 2019-06-13 * [[`8a83972743`](https://github.com/nodejs/node-gyp/commit/8a83972743)] - **(SEMVER-MAJOR)** **bin**: follow XDG OS conventions for storing data (Selwyn) [#1570](https://github.com/nodejs/node-gyp/pull/1570) * [[`9e46872ea3`](https://github.com/nodejs/node-gyp/commit/9e46872ea3)] - **bin,lib**: remove extra comments/lines/spaces (Jon Moss) [#1508](https://github.com/nodejs/node-gyp/pull/1508) @@ -301,16 +285,14 @@ v5.0.0 2019-06-13 * [[`721dc7d314`](https://github.com/nodejs/node-gyp/commit/721dc7d314)] - Add ARM64 to MSBuild /Platform logic (Jon Kunkee) [#1655](https://github.com/nodejs/node-gyp/pull/1655) * [[`a5b7410497`](https://github.com/nodejs/node-gyp/commit/a5b7410497)] - Add ESLint no-unused-vars rule (Jon Moss) [#1497](https://github.com/nodejs/node-gyp/pull/1497) -v4.0.0 2019-04-24 -================= +## v4.0.0 2019-04-24 * [[`ceed5cbe10`](https://github.com/nodejs/node-gyp/commit/ceed5cbe10)] - **deps**: updated tar package version to 4.4.8 (Pobegaylo Maksim) [#1713](https://github.com/nodejs/node-gyp/pull/1713) * [[`374519e066`](https://github.com/nodejs/node-gyp/commit/374519e066)] - **(SEMVER-MAJOR)** Upgrade to tar v3 (isaacs) [#1212](https://github.com/nodejs/node-gyp/pull/1212) * [[`e6699d13cd`](https://github.com/nodejs/node-gyp/commit/e6699d13cd)] - **test**: fix addon test for Node.js 12 and V8 7.4 (Richard Lau) [#1705](https://github.com/nodejs/node-gyp/pull/1705) * [[`0c6bf530a0`](https://github.com/nodejs/node-gyp/commit/0c6bf530a0)] - **lib**: use print() for python version detection (GreenAddress) [#1534](https://github.com/nodejs/node-gyp/pull/1534) -v3.8.0 2018-08-09 -================= +## v3.8.0 2018-08-09 * [[`c5929cb4fe`](https://github.com/nodejs/node-gyp/commit/c5929cb4fe)] - **doc**: update Xcode preferences tab name. (Ivan Daniluk) [#1330](https://github.com/nodejs/node-gyp/pull/1330) * [[`8b488da8b9`](https://github.com/nodejs/node-gyp/commit/8b488da8b9)] - **doc**: update link to commit guidelines (Jonas Hermsmeier) [#1456](https://github.com/nodejs/node-gyp/pull/1456) @@ -329,8 +311,7 @@ v3.8.0 2018-08-09 * [[`969447c5bd`](https://github.com/nodejs/node-gyp/commit/969447c5bd)] - **deps**: bump request to 2.8.7, fixes heok/hawk issues (Rohit Hazra) [#1492](https://github.com/nodejs/node-gyp/pull/1492) * [[`340403ccfe`](https://github.com/nodejs/node-gyp/commit/340403ccfe)] - **win**: improve parsing of SDK version (Alessandro Vergani) [#1516](https://github.com/nodejs/node-gyp/pull/1516) -v3.7.0 2018-06-08 -================= +## v3.7.0 2018-06-08 * [[`84cea7b30d`](https://github.com/nodejs/node-gyp/commit/84cea7b30d)] - Remove unused gyp test scripts. (Ben Noordhuis) [#1458](https://github.com/nodejs/node-gyp/pull/1458) * [[`0540e4ec63`](https://github.com/nodejs/node-gyp/commit/0540e4ec63)] - **gyp**: escape spaces in filenames in make generator (Jeff Senn) [#1436](https://github.com/nodejs/node-gyp/pull/1436) @@ -354,14 +335,12 @@ v3.7.0 2018-06-08 * [[`f27599193a`](https://github.com/nodejs/node-gyp/commit/f27599193a)] - **gyp**: update xml string encoding conversion (Liu Chao) [#1203](https://github.com/nodejs/node-gyp/pull/1203) * [[`0a07e481f7`](https://github.com/nodejs/node-gyp/commit/0a07e481f7)] - **configure**: don't set ensure if tarball is set (Gibson Fahnestock) [#1220](https://github.com/nodejs/node-gyp/pull/1220) -v3.6.3 2018-06-08 -================= +## v3.6.3 2018-06-08 * [[`90cd2e8da9`](https://github.com/nodejs/node-gyp/commit/90cd2e8da9)] - **gyp**: fix regex to match multi-digit versions (Jonas Hermsmeier) [#1455](https://github.com/nodejs/node-gyp/pull/1455) * [[`7900122337`](https://github.com/nodejs/node-gyp/commit/7900122337)] - deps: pin `request` version range (Refael Ackerman) [#1300](https://github.com/nodejs/node-gyp/pull/1300) -v3.6.2 2017-06-01 -================= +## v3.6.2 2017-06-01 * [[`72afdd62cd`](https://github.com/nodejs/node-gyp/commit/72afdd62cd)] - **build**: rename copyNodeLib() to doBuild() (Liu Chao) [#1206](https://github.com/nodejs/node-gyp/pull/1206) * [[`bad903ac70`](https://github.com/nodejs/node-gyp/commit/bad903ac70)] - **win**: more robust parsing of SDK version (Refael Ackermann) [#1198](https://github.com/nodejs/node-gyp/pull/1198) @@ -370,8 +349,7 @@ v3.6.2 2017-06-01 * [[`0913b2dd99`](https://github.com/nodejs/node-gyp/commit/0913b2dd99)] - **build, win**: use target_arch to link with node.lib (Pavel Medvedev) [#964](https://github.com/nodejs/node-gyp/pull/964) * [[`c307b302f7`](https://github.com/nodejs/node-gyp/commit/c307b302f7)] - **doc**: blorb about setting `npm_config_OPTION_NAME` (Refael Ackermann) [#1185](https://github.com/nodejs/node-gyp/pull/1185) -v3.6.1 2017-04-30 -================= +## v3.6.1 2017-04-30 * [[`49801716c2`](https://github.com/nodejs/node-gyp/commit/49801716c2)] - **test**: fix test-find-python on v0.10.x buildbot. (Ben Noordhuis) [#1172](https://github.com/nodejs/node-gyp/pull/1172) * [[`a83a3801fc`](https://github.com/nodejs/node-gyp/commit/a83a3801fc)] - **test**: fix test/test-configure-python on AIX (Richard Lau) [#1131](https://github.com/nodejs/node-gyp/pull/1131) @@ -379,8 +357,7 @@ v3.6.1 2017-04-30 * [[`c09cf7671e`](https://github.com/nodejs/node-gyp/commit/c09cf7671e)] - **doc**: add a note for using `configure` on Windows (Vse Mozhet Byt) [#1152](https://github.com/nodejs/node-gyp/pull/1152) * [[`da9cb5f411`](https://github.com/nodejs/node-gyp/commit/da9cb5f411)] - Delete superfluous .patch files. (Ben Noordhuis) [#1122](https://github.com/nodejs/node-gyp/pull/1122) -v3.6.0 2017-03-16 -================= +## v3.6.0 2017-03-16 * [[`ae141e1906`](https://github.com/nodejs/node-gyp/commit/ae141e1906)] - **win**: find and setup for VS2017 (Refael Ackermann) [#1130](https://github.com/nodejs/node-gyp/pull/1130) * [[`ec5fc36a80`](https://github.com/nodejs/node-gyp/commit/ec5fc36a80)] - Add support to build node.js with chakracore for ARM. (Kunal Pathak) [#873](https://github.com/nodejs/node-gyp/pull/873) @@ -388,9 +365,7 @@ v3.6.0 2017-03-16 * [[`93d7fa83c8`](https://github.com/nodejs/node-gyp/commit/93d7fa83c8)] - Upgrade semver dependency. (Ben Noordhuis) [#1107](https://github.com/nodejs/node-gyp/pull/1107) * [[`ff9a6fadfd`](https://github.com/nodejs/node-gyp/commit/ff9a6fadfd)] - Update link of gyp as Google code is shutting down (Peter Dave Hello) [#1061](https://github.com/nodejs/node-gyp/pull/1061) - -v3.5.0 2017-01-10 -================= +## v3.5.0 2017-01-10 * [[`762d19a39e`](https://github.com/nodejs/node-gyp/commit/762d19a39e)] - \[doc\] merge History.md and CHANGELOG.md (Rod Vagg) * [[`80fc5c3d31`](https://github.com/nodejs/node-gyp/commit/80fc5c3d31)] - Fix deprecated dependency warning (Simone Primarosa) [#1069](https://github.com/nodejs/node-gyp/pull/1069) @@ -404,8 +379,7 @@ v3.5.0 2017-01-10 * [[`9c8d275526`](https://github.com/nodejs/node-gyp/commit/9c8d275526)] - Add --devdir flag. (Ben Noordhuis) [#916](https://github.com/nodejs/node-gyp/pull/916) * [[`f6eab1f9e4`](https://github.com/nodejs/node-gyp/commit/f6eab1f9e4)] - **doc**: add windows-build-tools to readme (Felix Rieseberg) [#970](https://github.com/nodejs/node-gyp/pull/970) -v3.4.0 2016-06-28 -================= +## v3.4.0 2016-06-28 * [[`ce5fd04e94`](https://github.com/nodejs/node-gyp/commit/ce5fd04e94)] - **deps**: update minimatch version (delphiactual) [#961](https://github.com/nodejs/node-gyp/pull/961) * [[`77383ddd85`](https://github.com/nodejs/node-gyp/commit/77383ddd85)] - Replace fs.accessSync call to fs.statSync (Richard Lau) [#955](https://github.com/nodejs/node-gyp/pull/955) @@ -425,13 +399,11 @@ v3.4.0 2016-06-28 * [[`625c1515f9`](https://github.com/nodejs/node-gyp/commit/625c1515f9)] - **gyp**: inherit CC/CXX for CC/CXX.host (Johan Bergström) [#908](https://github.com/nodejs/node-gyp/pull/908) * [[`3bcb1720e4`](https://github.com/nodejs/node-gyp/commit/3bcb1720e4)] - Add support for the Python launcher on Windows (Patrick Westerhoff) [#894](https://github.com/nodejs/node-gyp/pull/894 -v3.3.1 2016-03-04 -================= +## v3.3.1 2016-03-04 * [[`a981ef847a`](https://github.com/nodejs/node-gyp/commit/a981ef847a)] - **gyp**: fix android generator (Robert Chiras) [#889](https://github.com/nodejs/node-gyp/pull/889) -v3.3.0 2016-02-16 -================= +## v3.3.0 2016-02-16 * [[`818d854a4d`](https://github.com/nodejs/node-gyp/commit/818d854a4d)] - Introduce NODEJS_ORG_MIRROR and IOJS_ORG_MIRROR (Rod Vagg) [#878](https://github.com/nodejs/node-gyp/pull/878) * [[`d1e4cc4b62`](https://github.com/nodejs/node-gyp/commit/d1e4cc4b62)] - **(SEMVER-MINOR)** Download headers tarball for ~0.12.10 || ~0.10.42 (Rod Vagg) [#877](https://github.com/nodejs/node-gyp/pull/877) @@ -440,14 +412,12 @@ v3.3.0 2016-02-16 * [[`8c4b0ffa50`](https://github.com/nodejs/node-gyp/commit/8c4b0ffa50)] - **(SEMVER-MINOR)** Add --cafile command line option. (Ben Noordhuis) [#837](https://github.com/nodejs/node-gyp/pull/837) * [[`b3ad43498e`](https://github.com/nodejs/node-gyp/commit/b3ad43498e)] - **(SEMVER-MINOR)** Make download() function testable. (Ben Noordhuis) [#837](https://github.com/nodejs/node-gyp/pull/837) -v3.2.1 2015-12-03 -================= +## v3.2.1 2015-12-03 * [[`ab89b477c4`](https://github.com/nodejs/node-gyp/commit/ab89b477c4)] - Upgrade gyp to b3cef02. (Ben Noordhuis) [#831](https://github.com/nodejs/node-gyp/pull/831) * [[`90078ecb17`](https://github.com/nodejs/node-gyp/commit/90078ecb17)] - Define WIN32_LEAN_AND_MEAN conditionally. (Ben Noordhuis) [#824](https://github.com/nodejs/node-gyp/pull/824) -v3.2.0 2015-11-25 -================= +## v3.2.0 2015-11-25 * [[`268f1ca4c7`](https://github.com/nodejs/node-gyp/commit/268f1ca4c7)] - Use result of `which` when searching for python. (Refael Ackermann) [#668](https://github.com/nodejs/node-gyp/pull/668) * [[`817ed9bd78`](https://github.com/nodejs/node-gyp/commit/817ed9bd78)] - Add test for python executable search logic. (Ben Noordhuis) [#756](https://github.com/nodejs/node-gyp/pull/756) @@ -456,8 +426,7 @@ v3.2.0 2015-11-25 * [[`a8d441a0a2`](https://github.com/nodejs/node-gyp/commit/a8d441a0a2)] - Update README for Windows 10 support. (Jason Williams) [#766](https://github.com/nodejs/node-gyp/pull/766) * [[`d1d6015276`](https://github.com/nodejs/node-gyp/commit/d1d6015276)] - Update broken links and switch to HTTPS. (andrew morton) -v3.1.0 2015-11-14 -================= +## v3.1.0 2015-11-14 * [[`9049241f91`](https://github.com/nodejs/node-gyp/commit/9049241f91)] - **gyp**: don't use links at all, just copy the files instead (Nathan Zadoks) * [[`8ef90348d1`](https://github.com/nodejs/node-gyp/commit/8ef90348d1)] - **gyp**: apply https://codereview.chromium.org/11361103/ (Nathan Rajlich) @@ -467,24 +436,20 @@ v3.1.0 2015-11-14 * [[`2ac7de02c4`](https://github.com/nodejs/node-gyp/commit/2ac7de02c4)] - Fix infinite loop with zero-length options. (Ben Noordhuis) [#745](https://github.com/nodejs/node-gyp/pull/745) * [[`101bed639b`](https://github.com/nodejs/node-gyp/commit/101bed639b)] - This platform value came from debian package, and now the value (Jérémy Lal) [#738](https://github.com/nodejs/node-gyp/pull/738) -v3.0.3 2015-09-14 -================= +## v3.0.3 2015-09-14 * [[`ad827cda30`](https://github.com/nodejs/node-gyp/commit/ad827cda30)] - tarballUrl global and && when checking for iojs (Lars-Magnus Skog) [#729](https://github.com/nodejs/node-gyp/pull/729) -v3.0.2 2015-09-12 -================= +## v3.0.2 2015-09-12 * [[`6e8c3bf3c6`](https://github.com/nodejs/node-gyp/commit/6e8c3bf3c6)] - add back support for passing additional cmdline args (Rod Vagg) [#723](https://github.com/nodejs/node-gyp/pull/723) * [[`ff82f2f3b9`](https://github.com/nodejs/node-gyp/commit/ff82f2f3b9)] - fixed broken link in docs to Visual Studio 2013 download (simon-p-r) [#722](https://github.com/nodejs/node-gyp/pull/722) -v3.0.1 2015-09-08 -================= +## v3.0.1 2015-09-08 * [[`846337e36b`](https://github.com/nodejs/node-gyp/commit/846337e36b)] - normalise versions for target == this comparison (Rod Vagg) [#716](https://github.com/nodejs/node-gyp/pull/716) -v3.0.0 2015-09-08 -================= +## v3.0.0 2015-09-08 * [[`9720d0373c`](https://github.com/nodejs/node-gyp/commit/9720d0373c)] - remove node_modules from tree (Rod Vagg) [#711](https://github.com/nodejs/node-gyp/pull/711) * [[`6dcf220db7`](https://github.com/nodejs/node-gyp/commit/6dcf220db7)] - test version major directly, don't use semver.satisfies() (Rod Vagg) [#711](https://github.com/nodejs/node-gyp/pull/711) @@ -496,8 +461,7 @@ v3.0.0 2015-09-08 * [[`85ed107565`](https://github.com/nodejs/node-gyp/commit/85ed107565)] - Merge pull request #664 from othiym23/othiym23/allow-semver-5 (Nathan Rajlich) * [[`0c720d234c`](https://github.com/nodejs/node-gyp/commit/0c720d234c)] - allow semver@5 (Forrest L Norvell) -2.0.2 / 2015-07-14 -================== +## 2.0.2 / 2015-07-14 * Use HTTPS for dist url (#656, @SonicHedgehog) * Merge pull request #648 from nevosegal/master @@ -510,14 +474,12 @@ v3.0.0 2015-09-08 src/win_delay_load_hook.c, and fixes of the long relative path issue on Win32. Fixes #636 (#637, @lygstate). -2.0.1 / 2015-05-28 -================== +## 2.0.1 / 2015-05-28 * configure: try/catch the semver range.test() call * README: update for visual studio 2013 (#510, @samccone) -2.0.0 / 2015-05-24 -================== +## 2.0.0 / 2015-05-24 * configure: check for python2 executable by default, fallback to python * configure: don't clobber existing $PYTHONPATH From fca4795512c67dc8420aaa0d913b5b89a4b147f3 Mon Sep 17 00:00:00 2001 From: DeeDeeG Date: Wed, 19 May 2021 03:28:36 -0400 Subject: [PATCH 200/551] lib: fail gracefully if we can't find the username (#2375) --- lib/find-python.js | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-) diff --git a/lib/find-python.js b/lib/find-python.js index 47a9aa6dd4..e6464d12c6 100644 --- a/lib/find-python.js +++ b/lib/find-python.js @@ -8,20 +8,35 @@ const win = process.platform === 'win32' const logWithPrefix = require('./util').logWithPrefix const systemDrive = process.env.SystemDrive || 'C:' -const username = process.env.USERNAME || process.env.USER || require('os').userInfo().username +const username = process.env.USERNAME || process.env.USER || getOsUserInfo() const localAppData = process.env.LOCALAPPDATA || `${systemDrive}\\${username}\\AppData\\Local` +const foundLocalAppData = process.env.LOCALAPPDATA || username const programFiles = process.env.ProgramW6432 || process.env.ProgramFiles || `${systemDrive}\\Program Files` const programFilesX86 = process.env['ProgramFiles(x86)'] || `${programFiles} (x86)` const winDefaultLocationsArray = [] for (const majorMinor of ['39', '38', '37', '36']) { - winDefaultLocationsArray.push( - `${localAppData}\\Programs\\Python\\Python${majorMinor}\\python.exe`, - `${programFiles}\\Python${majorMinor}\\python.exe`, - `${localAppData}\\Programs\\Python\\Python${majorMinor}-32\\python.exe`, - `${programFiles}\\Python${majorMinor}-32\\python.exe`, - `${programFilesX86}\\Python${majorMinor}-32\\python.exe` - ) + if (foundLocalAppData) { + winDefaultLocationsArray.push( + `${localAppData}\\Programs\\Python\\Python${majorMinor}\\python.exe`, + `${programFiles}\\Python${majorMinor}\\python.exe`, + `${localAppData}\\Programs\\Python\\Python${majorMinor}-32\\python.exe`, + `${programFiles}\\Python${majorMinor}-32\\python.exe`, + `${programFilesX86}\\Python${majorMinor}-32\\python.exe` + ) + } else { + winDefaultLocationsArray.push( + `${programFiles}\\Python${majorMinor}\\python.exe`, + `${programFiles}\\Python${majorMinor}-32\\python.exe`, + `${programFilesX86}\\Python${majorMinor}-32\\python.exe` + ) + } +} + +function getOsUserInfo () { + try { + return require('os').userInfo().username + } catch {} } function PythonFinder (configPython, callback) { From 245dee5b62581309946872ae253226ea3a42c0e3 Mon Sep 17 00:00:00 2001 From: DeeDeeG Date: Wed, 19 May 2021 03:30:17 -0400 Subject: [PATCH 201/551] lib: log as yes/no whether build dir was created (#2370) This bit of logging apparently expected to be given a boolean, but was receiving either a path or undefined based on the result of fs.mkdir. Now it prints either "Yes" or "No", rather than printing either a path or "undefined", respectively. --- lib/configure.js | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/configure.js b/lib/configure.js index 7992b658a4..92cf342561 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -76,7 +76,9 @@ function configure (gyp, argv, callback) { if (err) { return callback(err) } - log.verbose('build dir', '"build" dir needed to be created?', isNew) + log.verbose( + 'build dir', '"build" dir needed to be created?', isNew ? 'Yes' : 'No' + ) if (win) { findVisualStudio(release.semver, gyp.opts.msvs_version, createConfigFile) From 1b4697abf69ef574a48faf832a7098f4c6c224a5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gustavo=20de=20Le=C3=B3n?= Date: Wed, 19 May 2021 02:41:39 -0500 Subject: [PATCH 202/551] doc: Update README.md Visual Studio Community page polski to auto (#2371) changed URL of Visual Studio Community from a default polski URL to the one without the lenguage code --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a192351318..9c646d52a5 100644 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ Install the current version of Python from the [Microsoft Store package](https:/ Install tools and configuration manually: * Install Visual C++ Build Environment: [Visual Studio Build Tools](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) - (using "Visual C++ build tools" workload) or [Visual Studio 2017 Community](https://visualstudio.microsoft.com/pl/thank-you-downloading-visual-studio/?sku=Community) + (using "Visual C++ build tools" workload) or [Visual Studio 2017 Community](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=Community) (using the "Desktop development with C++" workload) * Launch cmd, `npm config set msvs_version 2017` From 14236709de64b100a424396b91a5115639daa0ef Mon Sep 17 00:00:00 2001 From: Jiawen Geng <3759816+gengjiawen@users.noreply.github.com> Date: Wed, 19 May 2021 16:36:27 +0800 Subject: [PATCH 203/551] doc: remove redundant version info (#2403) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 9c646d52a5..9ec3480138 100644 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ Install the current version of Python from the [Microsoft Store package](https:/ Install tools and configuration manually: * Install Visual C++ Build Environment: [Visual Studio Build Tools](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) - (using "Visual C++ build tools" workload) or [Visual Studio 2017 Community](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=Community) + (using "Visual C++ build tools" workload) or [Visual Studio Community](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=Community) (using the "Desktop development with C++" workload) * Launch cmd, `npm config set msvs_version 2017` From 814b1b0eda102afb9fc87e81638a9cf5b650bb10 Mon Sep 17 00:00:00 2001 From: Jiawen Geng <3759816+gengjiawen@users.noreply.github.com> Date: Thu, 27 May 2021 22:05:04 +0800 Subject: [PATCH 204/551] feat(gyp): update gyp to v0.9.1 (#2402) --- gyp/.github/workflows/release-please.yml | 2 +- gyp/CHANGELOG.md | 20 ++++++++++++++++++++ gyp/CODE_OF_CONDUCT.md | 4 ++-- gyp/CONTRIBUTING.md | 2 +- gyp/pylib/gyp/generator/make.py | 2 +- gyp/pylib/gyp/generator/ninja.py | 6 +++++- gyp/pylib/gyp/xcodeproj_file.py | 19 +++++++++++-------- gyp/setup.py | 2 +- gyp/tools/emacs/gyp-tests.el | 2 +- gyp/tools/emacs/gyp.el | 2 +- 10 files changed, 44 insertions(+), 17 deletions(-) diff --git a/gyp/.github/workflows/release-please.yml b/gyp/.github/workflows/release-please.yml index c8e98fbb56..288afdb3b3 100644 --- a/gyp/.github/workflows/release-please.yml +++ b/gyp/.github/workflows/release-please.yml @@ -1,7 +1,7 @@ on: push: branches: - - master + - main name: release-please jobs: diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md index 2f8bbb6249..6d66b3acd2 100644 --- a/gyp/CHANGELOG.md +++ b/gyp/CHANGELOG.md @@ -1,5 +1,25 @@ # Changelog +### [0.9.1](https://www.github.com/nodejs/gyp-next/compare/v0.9.0...v0.9.1) (2021-05-14) + + +### Bug Fixes + +* py lint ([3b6a8ee](https://www.github.com/nodejs/gyp-next/commit/3b6a8ee7a66193a8a6867eba9e1d2b70bdf04402)) + +## [0.9.0](https://www.github.com/nodejs/gyp-next/compare/v0.8.1...v0.9.0) (2021-05-13) + + +### Features + +* use LDFLAGS_host for host toolset ([#98](https://www.github.com/nodejs/gyp-next/issues/98)) ([bea5c7b](https://www.github.com/nodejs/gyp-next/commit/bea5c7bd67d6ad32acbdce79767a5481c70675a2)) + + +### Bug Fixes + +* msvs.py: remove overindentation ([#102](https://www.github.com/nodejs/gyp-next/issues/102)) ([3f83e99](https://www.github.com/nodejs/gyp-next/commit/3f83e99056d004d9579ceb786e06b624ddc36529)) +* update gyp.el to change case to cl-case ([#93](https://www.github.com/nodejs/gyp-next/issues/93)) ([13d5b66](https://www.github.com/nodejs/gyp-next/commit/13d5b66aab35985af9c2fb1174fdc6e1c1407ecc)) + ### [0.8.1](https://www.github.com/nodejs/gyp-next/compare/v0.8.0...v0.8.1) (2021-02-18) diff --git a/gyp/CODE_OF_CONDUCT.md b/gyp/CODE_OF_CONDUCT.md index 4c21140559..d724027fd9 100644 --- a/gyp/CODE_OF_CONDUCT.md +++ b/gyp/CODE_OF_CONDUCT.md @@ -1,4 +1,4 @@ # Code of Conduct -* [Node.js Code of Conduct](https://github.com/nodejs/admin/blob/master/CODE_OF_CONDUCT.md) -* [Node.js Moderation Policy](https://github.com/nodejs/admin/blob/master/Moderation-Policy.md) +* [Node.js Code of Conduct](https://github.com/nodejs/admin/blob/HEAD/CODE_OF_CONDUCT.md) +* [Node.js Moderation Policy](https://github.com/nodejs/admin/blob/HEAD/Moderation-Policy.md) diff --git a/gyp/CONTRIBUTING.md b/gyp/CONTRIBUTING.md index f9dd574a47..1a0bcde2b4 100644 --- a/gyp/CONTRIBUTING.md +++ b/gyp/CONTRIBUTING.md @@ -2,7 +2,7 @@ ## Code of Conduct -This project is bound to the [Node.js Code of Conduct](https://github.com/nodejs/admin/blob/master/CODE_OF_CONDUCT.md). +This project is bound to the [Node.js Code of Conduct](https://github.com/nodejs/admin/blob/HEAD/CODE_OF_CONDUCT.md). ## Developer's Certificate of Origin 1.1 diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index 857875a564..eb9102dd15 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -319,7 +319,7 @@ def CalculateGeneratorInputInfo(params): CXX.host ?= %(CXX.host)s CXXFLAGS.host ?= $(CPPFLAGS_host) $(CXXFLAGS_host) LINK.host ?= %(LINK.host)s -LDFLAGS.host ?= +LDFLAGS.host ?= $(LDFLAGS_host) AR.host ?= %(AR.host)s # Define a dir function that can handle spaces. diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py index c57bec6abb..ca032aef20 100644 --- a/gyp/pylib/gyp/generator/ninja.py +++ b/gyp/pylib/gyp/generator/ninja.py @@ -1417,7 +1417,11 @@ def WriteLinkForArch( is_executable = spec["type"] == "executable" # The ldflags config key is not used on mac or win. On those platforms # linker flags are set via xcode_settings and msvs_settings, respectively. - env_ldflags = os.environ.get("LDFLAGS", "").split() + if self.toolset == "target": + env_ldflags = os.environ.get("LDFLAGS", "").split() + elif self.toolset == "host": + env_ldflags = os.environ.get("LDFLAGS_host", "").split() + if self.flavor == "mac": ldflags = self.xcode_settings.GetLdflags( config_name, diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py index bca4fb79d0..5863ef45df 100644 --- a/gyp/pylib/gyp/xcodeproj_file.py +++ b/gyp/pylib/gyp/xcodeproj_file.py @@ -138,7 +138,9 @@ """ import gyp.common +from functools import cmp_to_key import hashlib +from operator import attrgetter import posixpath import re import struct @@ -423,6 +425,8 @@ def _HashUpdate(hash, data): """ hash.update(struct.pack(">i", len(data))) + if isinstance(data, str): + data = data.encode("utf-8") hash.update(data) if seed_hash is None: @@ -1483,7 +1487,7 @@ def TakeOverOnlyChild(self, recurse=False): def SortGroup(self): self._properties["children"] = sorted( - self._properties["children"], cmp=lambda x, y: x.Compare(y) + self._properties["children"], key=cmp_to_key(lambda x, y: x.Compare(y)) ) # Recurse. @@ -2891,7 +2895,7 @@ def SortGroups(self): # according to their defined order. self._properties["mainGroup"]._properties["children"] = sorted( self._properties["mainGroup"]._properties["children"], - cmp=lambda x, y: x.CompareRootGroup(y), + key=cmp_to_key(lambda x, y: x.CompareRootGroup(y)), ) # Sort everything else by putting group before files, and going @@ -2986,9 +2990,7 @@ def AddOrGetProjectReference(self, other_pbxproject): # Xcode seems to sort this list case-insensitively self._properties["projectReferences"] = sorted( self._properties["projectReferences"], - cmp=lambda x, y: cmp( - x["ProjectRef"].Name().lower(), y["ProjectRef"].Name().lower() - ), + key=lambda x: x["ProjectRef"].Name().lower ) else: # The link already exists. Pull out the relevnt data. @@ -3120,7 +3122,8 @@ def CompareProducts(x, y, remote_products): product_group = ref_dict["ProductGroup"] product_group._properties["children"] = sorted( product_group._properties["children"], - cmp=lambda x, y, rp=remote_products: CompareProducts(x, y, rp), + key=cmp_to_key( + lambda x, y, rp=remote_products: CompareProducts(x, y, rp)), ) @@ -3155,7 +3158,7 @@ def Print(self, file=sys.stdout): else: self._XCPrint(file, 0, "{\n") for property, value in sorted( - self._properties.items(), cmp=lambda x, y: cmp(x, y) + self._properties.items() ): if property == "objects": self._PrintObjects(file) @@ -3183,7 +3186,7 @@ def _PrintObjects(self, file): self._XCPrint(file, 0, "\n") self._XCPrint(file, 0, "/* Begin " + class_name + " section */\n") for object in sorted( - objects_by_class[class_name], cmp=lambda x, y: cmp(x.id, y.id) + objects_by_class[class_name], key=attrgetter("id") ): object.Print(file) self._XCPrint(file, 0, "/* End " + class_name + " section */\n") diff --git a/gyp/setup.py b/gyp/setup.py index de0e3f5f61..f4a9481937 100644 --- a/gyp/setup.py +++ b/gyp/setup.py @@ -15,7 +15,7 @@ setup( name="gyp-next", - version="0.8.1", + version="0.9.1", description="A fork of the GYP build system for use in the Node.js projects", long_description=long_description, long_description_content_type="text/markdown", diff --git a/gyp/tools/emacs/gyp-tests.el b/gyp/tools/emacs/gyp-tests.el index 11b8497886..07afc58a93 100644 --- a/gyp/tools/emacs/gyp-tests.el +++ b/gyp/tools/emacs/gyp-tests.el @@ -30,7 +30,7 @@ "For the purposes of face comparison, we're not interested in the differences between certain faces. For example, the difference between font-lock-comment-delimiter and font-lock-comment-face." - (case face + (cl-case face ((font-lock-comment-delimiter-face) font-lock-comment-face) (t face))) diff --git a/gyp/tools/emacs/gyp.el b/gyp/tools/emacs/gyp.el index b98b155ced..042ff3a925 100644 --- a/gyp/tools/emacs/gyp.el +++ b/gyp/tools/emacs/gyp.el @@ -213,7 +213,7 @@ string-start) (setq string-start (gyp-parse-to limit)) (if string-start - (setq group (case (gyp-section-at-point) + (setq group (cl-case (gyp-section-at-point) ('dependencies 1) ('variables 2) ('conditions 2) From be55870bb3c11467fecbbbf5203d147111d046a7 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri, 28 May 2021 12:01:40 +1000 Subject: [PATCH 205/551] chore: release 8.1.0 (#2418) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- CHANGELOG.md | 25 +++++++++++++++++++++++++ package.json | 2 +- 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 77a71506fa..821ed3f0be 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,30 @@ # Changelog +## [8.1.0](https://www.github.com/nodejs/node-gyp/compare/v8.0.0...v8.1.0) (2021-05-28) + + +### Features + +* **gyp:** update gyp to v0.9.1 ([#2402](https://www.github.com/nodejs/node-gyp/issues/2402)) ([814b1b0](https://www.github.com/nodejs/node-gyp/commit/814b1b0eda102afb9fc87e81638a9cf5b650bb10)) + + +### Miscellaneous + +* add `release-please-action` for automated releases ([#2395](https://www.github.com/nodejs/node-gyp/issues/2395)) ([07e9d7c](https://www.github.com/nodejs/node-gyp/commit/07e9d7c7ee80ba119ea760c635f72fd8e7efe198)) + + +### Core + +* fail gracefully if we can't find the username ([#2375](https://www.github.com/nodejs/node-gyp/issues/2375)) ([fca4795](https://www.github.com/nodejs/node-gyp/commit/fca4795512c67dc8420aaa0d913b5b89a4b147f3)) +* log as yes/no whether build dir was created ([#2370](https://www.github.com/nodejs/node-gyp/issues/2370)) ([245dee5](https://www.github.com/nodejs/node-gyp/commit/245dee5b62581309946872ae253226ea3a42c0e3)) + + +### Doc + +* fix v8.0.0 release date ([4b83c3d](https://www.github.com/nodejs/node-gyp/commit/4b83c3de7300457919d53f26d96ea9ad6f6bedd8)) +* remove redundant version info ([#2403](https://www.github.com/nodejs/node-gyp/issues/2403)) ([1423670](https://www.github.com/nodejs/node-gyp/commit/14236709de64b100a424396b91a5115639daa0ef)) +* Update README.md Visual Studio Community page polski to auto ([#2371](https://www.github.com/nodejs/node-gyp/issues/2371)) ([1b4697a](https://www.github.com/nodejs/node-gyp/commit/1b4697abf69ef574a48faf832a7098f4c6c224a5)) + ## v8.0.0 2021-04-03 * [[`0d8a6f1b19`](https://github.com/nodejs/node-gyp/commit/0d8a6f1b19)] - **ci**: update actions/setup-node to v2 (#2302) (Sora Morimoto) [#2302](https://github.com/nodejs/node-gyp/pull/2302) diff --git a/package.json b/package.json index ee15c98c80..469675b80a 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "8.0.0", + "version": "8.1.0", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From 5f1a06c50f3b0c3d292f64948f85a004cfcc5c87 Mon Sep 17 00:00:00 2001 From: DeeDeeG Date: Fri, 28 May 2021 22:22:21 -0400 Subject: [PATCH 206/551] lib: deep-copy process.config during configure (#2368) --- lib/configure.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/configure.js b/lib/configure.js index 92cf342561..038ccbf20f 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -98,7 +98,7 @@ function configure (gyp, argv, callback) { log.verbose('build/' + configFilename, 'creating config file') - var config = Object.assign({}, process.config) + var config = process.config ? JSON.parse(JSON.stringify(process.config)) : {} var defaults = config.target_defaults var variables = config.variables From cfd12ff3bb0eb4525173413ef6a94b3cd8398cad Mon Sep 17 00:00:00 2001 From: Jiawen Geng <3759816+gengjiawen@users.noreply.github.com> Date: Sat, 29 May 2021 20:48:13 +0800 Subject: [PATCH 207/551] fix: change default gyp update message (#2420) --- update-gyp.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/update-gyp.py b/update-gyp.py index 6998718434..bb84f071a6 100755 --- a/update-gyp.py +++ b/update-gyp.py @@ -43,4 +43,4 @@ ) subprocess.check_output(["git", "add", "gyp"], cwd=CHECKOUT_PATH) -subprocess.check_output(["git", "commit", "-m", "gyp: update gyp to " + args.tag]) +subprocess.check_output(["git", "commit", "-m", "feat(gyp): update gyp to " + args.tag]) From 5cde818aac715477e9e9747966bb6b4c4ed070a8 Mon Sep 17 00:00:00 2001 From: Livia Rett <30511433+liviarett@users.noreply.github.com> Date: Tue, 22 Jun 2021 03:20:50 +0100 Subject: [PATCH 208/551] fix: add error arg back into catch block for older Node.js users --- lib/find-python.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/find-python.js b/lib/find-python.js index e6464d12c6..a445e825b9 100644 --- a/lib/find-python.js +++ b/lib/find-python.js @@ -36,7 +36,7 @@ for (const majorMinor of ['39', '38', '37', '36']) { function getOsUserInfo () { try { return require('os').userInfo().username - } catch {} + } catch (e) {} } function PythonFinder (configPython, callback) { From 2d0ce5595e232a3fc7c562cdf39efb77e2312cc1 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Tue, 22 Jun 2021 07:44:02 +0200 Subject: [PATCH 209/551] chore: fix typos discovered by codespell (#2442) --- CHANGELOG.md | 2 +- lib/find-visualstudio.js | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 821ed3f0be..b002a27ec3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -83,7 +83,7 @@ We consider this a bug-fix rather than semver-major change. * [[`f7bfce96ed`](https://github.com/nodejs/node-gyp/commit/f7bfce96ed)] - **doc**: update acid test and introduce curl|bash test script (Dario Vladovic) [#2105](https://github.com/nodejs/node-gyp/pull/2105) * [[`e529f3309d`](https://github.com/nodejs/node-gyp/commit/e529f3309d)] - **doc**: update README to reflect upgrade to gyp-next (Ujjwal Sharma) [#2092](https://github.com/nodejs/node-gyp/pull/2092) * [[`9aed6286a3`](https://github.com/nodejs/node-gyp/commit/9aed6286a3)] - **doc**: give more attention to Catalina issues doc (Matheus Marchini) [#2134](https://github.com/nodejs/node-gyp/pull/2134) -* [[`963f2a7b48`](https://github.com/nodejs/node-gyp/commit/963f2a7b48)] - **doc**: improve cataline discoverability for search engines (Matheus Marchini) [#2135](https://github.com/nodejs/node-gyp/pull/2135) +* [[`963f2a7b48`](https://github.com/nodejs/node-gyp/commit/963f2a7b48)] - **doc**: improve Catalina discoverability for search engines (Matheus Marchini) [#2135](https://github.com/nodejs/node-gyp/pull/2135) * [[`7b75af349b`](https://github.com/nodejs/node-gyp/commit/7b75af349b)] - **doc**: add macOS Catalina software update info (Karl Horky) [#2078](https://github.com/nodejs/node-gyp/pull/2078) * [[`4f23c7bee2`](https://github.com/nodejs/node-gyp/commit/4f23c7bee2)] - **doc**: update link to the code of conduct (#2073) (Michaël Zasso) [#2073](https://github.com/nodejs/node-gyp/pull/2073) * [[`473cfa283f`](https://github.com/nodejs/node-gyp/commit/473cfa283f)] - **doc**: note in README that Python 3.8 is supported (#2072) (Michaël Zasso) [#2072](https://github.com/nodejs/node-gyp/pull/2072) diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js index 9c6dad90f8..f2cce327e7 100644 --- a/lib/find-visualstudio.js +++ b/lib/find-visualstudio.js @@ -399,7 +399,7 @@ VisualStudioFinder.prototype = { }) }, - // After finding a usable version of Visual Stuido: + // After finding a usable version of Visual Studio: // - add it to validVersions to be displayed at the end if a specific // version was requested and not found; // - check if this is the version that was requested. From 1773c1540d24911199dff43671db5529cdac222c Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Tue, 12 Jun 2012 08:20:17 -0700 Subject: [PATCH 210/551] Initial Commit --- Home.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 Home.md diff --git a/Home.md b/Home.md new file mode 100644 index 0000000000..8e5c290b62 --- /dev/null +++ b/Home.md @@ -0,0 +1 @@ +Welcome to the node-gyp wiki! \ No newline at end of file From d4fd14355bbe57f229f082f47bb2b3670868203f Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Tue, 12 Jun 2012 08:34:29 -0700 Subject: [PATCH 211/551] doc(wiki): Created "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 "\"binding.gyp\"-files-out-in-the-wild.md" diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" new file mode 100644 index 0000000000..d6e8d5153b --- /dev/null +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -0,0 +1,12 @@ +This page contains links to some examples of existing `binding.gyp` files that other node modules are using. Take a look at them for inspiration. + +To add to this page, just add the link to the project's `binding.gyp` file below: + + * [libxmljs](https://github.com/polotek/libxmljs/blob/master/binding.gyp) + * [node-buffertools](https://github.com/bnoordhuis/node-buffertools/blob/master/binding.gyp) + * [node-time](https://github.com/TooTallNate/node-time/blob/master/binding.gyp) + * [node-serialport](https://github.com/voodootikigod/node-serialport/blob/master/binding.gyp) + * [node-serialport2](https://github.com/joeferner/node-serialport2/blob/master/binding.gyp) + * [node-weak](https://github.com/TooTallNate/node-weak/blob/master/binding.gyp) + * [pty.js](https://github.com/chjj/pty.js/blob/master/binding.gyp) + * [ref](https://github.com/TooTallNate/ref/blob/master/binding.gyp) \ No newline at end of file From dc9776648d432bca6775c176641f16da14522d4c Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Tue, 12 Jun 2012 08:36:44 -0700 Subject: [PATCH 212/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 1 + 1 file changed, 1 insertion(+) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index d6e8d5153b..94da2e2d9c 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -4,6 +4,7 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [libxmljs](https://github.com/polotek/libxmljs/blob/master/binding.gyp) * [node-buffertools](https://github.com/bnoordhuis/node-buffertools/blob/master/binding.gyp) + * [node-ffi](https://github.com/rbranson/node-ffi/blob/master/binding.gyp) * [node-time](https://github.com/TooTallNate/node-time/blob/master/binding.gyp) * [node-serialport](https://github.com/voodootikigod/node-serialport/blob/master/binding.gyp) * [node-serialport2](https://github.com/joeferner/node-serialport2/blob/master/binding.gyp) From 27658913e6220cf0371b4b73e25a0e4ab11108a1 Mon Sep 17 00:00:00 2001 From: milani Date: Mon, 18 Jun 2012 07:22:32 -0700 Subject: [PATCH 213/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 94da2e2d9c..17724eca9e 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -10,4 +10,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-serialport2](https://github.com/joeferner/node-serialport2/blob/master/binding.gyp) * [node-weak](https://github.com/TooTallNate/node-weak/blob/master/binding.gyp) * [pty.js](https://github.com/chjj/pty.js/blob/master/binding.gyp) - * [ref](https://github.com/TooTallNate/ref/blob/master/binding.gyp) \ No newline at end of file + * [ref](https://github.com/TooTallNate/ref/blob/master/binding.gyp) + * [appjs](https://github.com/milani/appjs/blob/master/binding.gyp) \ No newline at end of file From 954ee530b3972d1db591fce32368e4e31b5a25d8 Mon Sep 17 00:00:00 2001 From: Psychless Date: Mon, 18 Jun 2012 18:23:24 -0700 Subject: [PATCH 214/551] doc(wiki): fixed node-serialport link --- "\"binding.gyp\"-files-out-in-the-wild.md" | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 17724eca9e..bd2bcdadab 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -6,7 +6,7 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-buffertools](https://github.com/bnoordhuis/node-buffertools/blob/master/binding.gyp) * [node-ffi](https://github.com/rbranson/node-ffi/blob/master/binding.gyp) * [node-time](https://github.com/TooTallNate/node-time/blob/master/binding.gyp) - * [node-serialport](https://github.com/voodootikigod/node-serialport/blob/master/binding.gyp) + * [node-serialport](https://github.com/voodootikigod/node-serialport/blob/master/bindings.gyp) * [node-serialport2](https://github.com/joeferner/node-serialport2/blob/master/binding.gyp) * [node-weak](https://github.com/TooTallNate/node-weak/blob/master/binding.gyp) * [pty.js](https://github.com/chjj/pty.js/blob/master/binding.gyp) From d29fb134f1c4b9dd729ba95f2979e69e0934809f Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Tue, 17 Jul 2012 18:57:05 -0700 Subject: [PATCH 215/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 1 - 1 file changed, 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index bd2bcdadab..248cd79f4e 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -7,7 +7,6 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-ffi](https://github.com/rbranson/node-ffi/blob/master/binding.gyp) * [node-time](https://github.com/TooTallNate/node-time/blob/master/binding.gyp) * [node-serialport](https://github.com/voodootikigod/node-serialport/blob/master/bindings.gyp) - * [node-serialport2](https://github.com/joeferner/node-serialport2/blob/master/binding.gyp) * [node-weak](https://github.com/TooTallNate/node-weak/blob/master/binding.gyp) * [pty.js](https://github.com/chjj/pty.js/blob/master/binding.gyp) * [ref](https://github.com/TooTallNate/ref/blob/master/binding.gyp) From 27b883a350ad0db6b9130d7b996f35855ec34c7a Mon Sep 17 00:00:00 2001 From: mixu Date: Thu, 19 Jul 2012 10:39:10 -0700 Subject: [PATCH 216/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 248cd79f4e..215678acaf 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -10,4 +10,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-weak](https://github.com/TooTallNate/node-weak/blob/master/binding.gyp) * [pty.js](https://github.com/chjj/pty.js/blob/master/binding.gyp) * [ref](https://github.com/TooTallNate/ref/blob/master/binding.gyp) - * [appjs](https://github.com/milani/appjs/blob/master/binding.gyp) \ No newline at end of file + * [appjs](https://github.com/milani/appjs/blob/master/binding.gyp) + * [nwm](https://github.com/mixu/nwm/blob/master/binding.gyp) \ No newline at end of file From e199cfa8fc6161492d2a6ade2190510d0ebf7c0f Mon Sep 17 00:00:00 2001 From: shtylman Date: Thu, 19 Jul 2012 10:43:17 -0700 Subject: [PATCH 217/551] doc(wiki): add bcrypt --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 215678acaf..529a9df4d0 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -11,4 +11,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [pty.js](https://github.com/chjj/pty.js/blob/master/binding.gyp) * [ref](https://github.com/TooTallNate/ref/blob/master/binding.gyp) * [appjs](https://github.com/milani/appjs/blob/master/binding.gyp) - * [nwm](https://github.com/mixu/nwm/blob/master/binding.gyp) \ No newline at end of file + * [nwm](https://github.com/mixu/nwm/blob/master/binding.gyp) + * [bcrypt] (https://github.com/ncb000gt/node.bcrypt.js) \ No newline at end of file From e11bdd84de6144492d3eb327d67cbf2d62da1a76 Mon Sep 17 00:00:00 2001 From: shtylman Date: Thu, 19 Jul 2012 10:44:40 -0700 Subject: [PATCH 218/551] doc(wiki): change bcrypt url to binding.gyp file --- "\"binding.gyp\"-files-out-in-the-wild.md" | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 529a9df4d0..72e8401f7e 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -12,4 +12,4 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [ref](https://github.com/TooTallNate/ref/blob/master/binding.gyp) * [appjs](https://github.com/milani/appjs/blob/master/binding.gyp) * [nwm](https://github.com/mixu/nwm/blob/master/binding.gyp) - * [bcrypt] (https://github.com/ncb000gt/node.bcrypt.js) \ No newline at end of file + * [bcrypt](https://github.com/ncb000gt/node.bcrypt.js/blob/master/binding.gyp) \ No newline at end of file From ced8c968457f285ab8989c291d28173d7730833c Mon Sep 17 00:00:00 2001 From: lloyd Date: Thu, 19 Jul 2012 11:26:08 -0700 Subject: [PATCH 219/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 72e8401f7e..c4b93ebf13 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -12,4 +12,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [ref](https://github.com/TooTallNate/ref/blob/master/binding.gyp) * [appjs](https://github.com/milani/appjs/blob/master/binding.gyp) * [nwm](https://github.com/mixu/nwm/blob/master/binding.gyp) - * [bcrypt](https://github.com/ncb000gt/node.bcrypt.js/blob/master/binding.gyp) \ No newline at end of file + * [bcrypt](https://github.com/ncb000gt/node.bcrypt.js/blob/master/binding.gyp) + * [node-memwatch](https://github.com/lloyd/node-memwatch/blob/master/binding.gyp) \ No newline at end of file From 77f363272930d3d4d24fd3973be22e6237128fcc Mon Sep 17 00:00:00 2001 From: bolgovr Date: Thu, 19 Jul 2012 12:15:03 -0700 Subject: [PATCH 220/551] doc(wiki): add one more example --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index c4b93ebf13..b95009a268 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -13,4 +13,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [appjs](https://github.com/milani/appjs/blob/master/binding.gyp) * [nwm](https://github.com/mixu/nwm/blob/master/binding.gyp) * [bcrypt](https://github.com/ncb000gt/node.bcrypt.js/blob/master/binding.gyp) - * [node-memwatch](https://github.com/lloyd/node-memwatch/blob/master/binding.gyp) \ No newline at end of file + * [node-memwatch](https://github.com/lloyd/node-memwatch/blob/master/binding.gyp) + * [node-ip2location](https://github.com/bolgovr/node-ip2location/blob/master/binding.gyp) \ No newline at end of file From b3547115f6e356358138310e857c7f1ec627a8a7 Mon Sep 17 00:00:00 2001 From: justinlatimer Date: Thu, 19 Jul 2012 14:55:45 -0700 Subject: [PATCH 221/551] doc(wiki): Add a link to the node-midi binding.gyp file. --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index b95009a268..989107bf4b 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -14,4 +14,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [nwm](https://github.com/mixu/nwm/blob/master/binding.gyp) * [bcrypt](https://github.com/ncb000gt/node.bcrypt.js/blob/master/binding.gyp) * [node-memwatch](https://github.com/lloyd/node-memwatch/blob/master/binding.gyp) - * [node-ip2location](https://github.com/bolgovr/node-ip2location/blob/master/binding.gyp) \ No newline at end of file + * [node-ip2location](https://github.com/bolgovr/node-ip2location/blob/master/binding.gyp) + * [node-midi](https://github.com/justinlatimer/node-midi/blob/master/binding.gyp) From 640895d36b7448c646a3b850c1e159106f83c724 Mon Sep 17 00:00:00 2001 From: kkaefer Date: Thu, 19 Jul 2012 16:23:51 -0700 Subject: [PATCH 222/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 4 ++++ 1 file changed, 4 insertions(+) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 989107bf4b..92b31b46ea 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -16,3 +16,7 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-memwatch](https://github.com/lloyd/node-memwatch/blob/master/binding.gyp) * [node-ip2location](https://github.com/bolgovr/node-ip2location/blob/master/binding.gyp) * [node-midi](https://github.com/justinlatimer/node-midi/blob/master/binding.gyp) + * [node-sqlite3](https://github.com/developmentseed/node-sqlite3/blob/master/binding.gyp) + * [node-srs](https://github.com/springmeyer/node-srs/blob/master/build.gyp) + * [node-zipfile](https://github.com/springmeyer/node-zipfile/blob/master/build.gyp) + * [node-mapnik](https://github.com/mapnik/node-mapnik/blob/master/build.gyp) From b6e542f644dbbfe22b88524ec500696e06ee4af7 Mon Sep 17 00:00:00 2001 From: c4milo Date: Thu, 19 Jul 2012 18:05:47 -0700 Subject: [PATCH 223/551] doc(wiki): Adds node-inotify and v8-profiler --- "\"binding.gyp\"-files-out-in-the-wild.md" | 2 ++ 1 file changed, 2 insertions(+) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 92b31b46ea..4534f31b0e 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -20,3 +20,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-srs](https://github.com/springmeyer/node-srs/blob/master/build.gyp) * [node-zipfile](https://github.com/springmeyer/node-zipfile/blob/master/build.gyp) * [node-mapnik](https://github.com/mapnik/node-mapnik/blob/master/build.gyp) + * [node-inotify](https://github.com/c4milo/node-inotify/blob/master/binding.gyp) + * [v8-profiler](https://github.com/c4milo/v8-profiler/blob/master/binding.gyp) \ No newline at end of file From 7ab133752a6c402bb96dcd3d671d73e03e9487ad Mon Sep 17 00:00:00 2001 From: lperrin Date: Fri, 20 Jul 2012 04:05:56 -0700 Subject: [PATCH 224/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 4534f31b0e..2cecf77b34 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -21,4 +21,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-zipfile](https://github.com/springmeyer/node-zipfile/blob/master/build.gyp) * [node-mapnik](https://github.com/mapnik/node-mapnik/blob/master/build.gyp) * [node-inotify](https://github.com/c4milo/node-inotify/blob/master/binding.gyp) - * [v8-profiler](https://github.com/c4milo/v8-profiler/blob/master/binding.gyp) \ No newline at end of file + * [v8-profiler](https://github.com/c4milo/v8-profiler/blob/master/binding.gyp) + * [airtunes](https://github.com/radioline/node_airtunes/blob/master/binding.gyp) \ No newline at end of file From 23e3d485ed894ba7c631e9c062f5e366b50c416c Mon Sep 17 00:00:00 2001 From: c4milo Date: Tue, 24 Jul 2012 18:13:07 -0700 Subject: [PATCH 225/551] doc(wiki): Adds node-fann --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 2cecf77b34..ba3b7b28b0 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -22,4 +22,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-mapnik](https://github.com/mapnik/node-mapnik/blob/master/build.gyp) * [node-inotify](https://github.com/c4milo/node-inotify/blob/master/binding.gyp) * [v8-profiler](https://github.com/c4milo/v8-profiler/blob/master/binding.gyp) - * [airtunes](https://github.com/radioline/node_airtunes/blob/master/binding.gyp) \ No newline at end of file + * [airtunes](https://github.com/radioline/node_airtunes/blob/master/binding.gyp) + * [node-fann](https://github.com/c4milo/node-fann/blob/master/binding.gyp) \ No newline at end of file From 13a955317b39caf98fd1f412d8d3f41599e979fd Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Mon, 30 Jul 2012 16:17:36 -0700 Subject: [PATCH 226/551] doc(wiki): Add node-canvas --- "\"binding.gyp\"-files-out-in-the-wild.md" | 1 + 1 file changed, 1 insertion(+) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index ba3b7b28b0..e321e90d58 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -4,6 +4,7 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [libxmljs](https://github.com/polotek/libxmljs/blob/master/binding.gyp) * [node-buffertools](https://github.com/bnoordhuis/node-buffertools/blob/master/binding.gyp) + * [node-canvas](https://github.com/LearnBoost/node-canvas/blob/master/binding.gyp) * [node-ffi](https://github.com/rbranson/node-ffi/blob/master/binding.gyp) * [node-time](https://github.com/TooTallNate/node-time/blob/master/binding.gyp) * [node-serialport](https://github.com/voodootikigod/node-serialport/blob/master/bindings.gyp) From 14627556966e5d513bdb8e5208f0e1300f68991f Mon Sep 17 00:00:00 2001 From: oransel Date: Wed, 15 Aug 2012 21:45:43 -0700 Subject: [PATCH 227/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index e321e90d58..6585d3dc40 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -24,4 +24,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-inotify](https://github.com/c4milo/node-inotify/blob/master/binding.gyp) * [v8-profiler](https://github.com/c4milo/v8-profiler/blob/master/binding.gyp) * [airtunes](https://github.com/radioline/node_airtunes/blob/master/binding.gyp) - * [node-fann](https://github.com/c4milo/node-fann/blob/master/binding.gyp) \ No newline at end of file + * [node-fann](https://github.com/c4milo/node-fann/blob/master/binding.gyp) + * [node-talib](https://github.com/oransel/node-talib/blob/master/binding.gyp) From c46d00d83bac5173dea8bbbb175a1a7de74fdaca Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Sun, 2 Sep 2012 11:35:12 -0700 Subject: [PATCH 228/551] doc(wiki): Created Linking to OpenSSL (markdown) --- Linking-to-OpenSSL.md | 75 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 Linking-to-OpenSSL.md diff --git a/Linking-to-OpenSSL.md b/Linking-to-OpenSSL.md new file mode 100644 index 0000000000..fb6a2e9143 --- /dev/null +++ b/Linking-to-OpenSSL.md @@ -0,0 +1,75 @@ +A handful of native addons require linking to OpenSSL in one way or another. This introduces a small chalenge since node will sometimes bundle OpenSSL statically (the default for node >= v0.8.x), or sometimes dynamically link to the system OpenSSL (default for node <= v0.6.x). + +Good native addons should account for both scenarios. It's recommended that you use the `binding.gyp` file provided below as a starting-point for any addon that needs to use OpenSSL: + +``` python +{ + 'variables': { + # node v0.6.x doesn't give us its build variables, + # but on Unix it was only possible to use the system OpenSSL library, + # so default the variable to "true", v0.8.x node and up will overwrite it. + 'node_shared_openssl%': 'true' + }, + 'targets': [ + { + 'target_name': 'binding', + 'sources': [ + 'src/binding.cc' + ], + 'conditions': [ + ['node_shared_openssl=="false"', { + # so when "node_shared_openssl" is "false", then OpenSSL has been + # bundled into the node executable. So we need to include the same + # header files that were used when building node. + 'include_dirs': [ + '<(node_root_dir)/deps/openssl/openssl/include' + ] + }] + ] + } + ] +} +``` + +This ensures that when OpenSSL is statically linked into `node` then, the bundled OpenSSL headers are included, but when the system OpenSSL is in use, then only those headers will be used. + +## Windows? + +As you can see this baseline `binding.gyp` file only accounts for the Unix scenario. Currently on Windows the situation is a little less ideal. On Windows, OpenSSL is _always_ statically compiled into the `node` executable, so ideally it would be possible to use that copy of OpenSSL when building native addons. + +Unfortunately it doesn't seem like that is possible at the moment, as there would need to be tweaks made to the generated `node.lib` file to include the openssl glue functions, or a new `openssl.lib` file would need to be created during the node build. I'm not sure which is the easiest/most feasible. + +In the meantime, one possible solution is using another copy of OpenSSL, which is what [`node-bcrypt`](https://github.com/ncb000gt/node.bcrypt.js) currently does. Adding something like this to your `binding.gyp` file's `"conditions"` block would enable this: + +``` python + [ 'OS=="win"', { + 'conditions': [ + # "openssl_root" is the directory on Windows of the OpenSSL files. + # Check the "target_arch" variable to set good default values for + # both 64-bit and 32-bit builds of the module. + ['target_arch=="x64"', { + 'variables': { + 'openssl_root%': 'C:/OpenSSL-Win64' + }, + }, { + 'variables': { + 'openssl_root%': 'C:/OpenSSL-Win32' + }, + }], + ], + 'libraries': [ + '-l<(openssl_root)/lib/libeay32.lib', + ], + 'include_dirs': [ + '<(openssl_root)/include', + ], + }] +``` + +Now you can direct your users to install OpenSSL on Windows from here (be sure to tell them to install the 64-bit version if they're compiling against a 64-bit version of node): http://slproweb.com/products/Win32OpenSSL.html + +Also note that both `node-gyp` and `npm` allow you to overwrite that default `openssl_root` variable on the command line: + +``` bash +$ node-gyp rebuild --openssl-root="C:\Users\Nathan\Desktop\openssl" +``` \ No newline at end of file From b398ef46f660d2b1506508550dadfb4c35639e4b Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Sun, 2 Sep 2012 11:38:25 -0700 Subject: [PATCH 229/551] doc(wiki): Updated Home (markdown) --- Home.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/Home.md b/Home.md index 8e5c290b62..189e315b3e 100644 --- a/Home.md +++ b/Home.md @@ -1 +1,4 @@ -Welcome to the node-gyp wiki! \ No newline at end of file +Welcome to the node-gyp wiki! + + * [["binding.gyp" files out in the wild]] + * [[Linking to OpenSSL]] \ No newline at end of file From 3236069689e7e0eb15b324fce74ab58158956f98 Mon Sep 17 00:00:00 2001 From: voodootikigod Date: Sun, 9 Sep 2012 12:11:04 -0700 Subject: [PATCH 230/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 6585d3dc40..19928493f7 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -7,7 +7,7 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-canvas](https://github.com/LearnBoost/node-canvas/blob/master/binding.gyp) * [node-ffi](https://github.com/rbranson/node-ffi/blob/master/binding.gyp) * [node-time](https://github.com/TooTallNate/node-time/blob/master/binding.gyp) - * [node-serialport](https://github.com/voodootikigod/node-serialport/blob/master/bindings.gyp) + * [node-serialport](https://github.com/voodootikigod/node-serialport/blob/master/binding.gyp) * [node-weak](https://github.com/TooTallNate/node-weak/blob/master/binding.gyp) * [pty.js](https://github.com/chjj/pty.js/blob/master/binding.gyp) * [ref](https://github.com/TooTallNate/ref/blob/master/binding.gyp) @@ -25,4 +25,4 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [v8-profiler](https://github.com/c4milo/v8-profiler/blob/master/binding.gyp) * [airtunes](https://github.com/radioline/node_airtunes/blob/master/binding.gyp) * [node-fann](https://github.com/c4milo/node-fann/blob/master/binding.gyp) - * [node-talib](https://github.com/oransel/node-talib/blob/master/binding.gyp) + * [node-talib](https://github.com/oransel/node-talib/blob/master/binding.gyp) \ No newline at end of file From c00eb778fc7dc27e4dab3a9219035ea20458b33b Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Mon, 24 Sep 2012 14:26:31 -0700 Subject: [PATCH 231/551] doc(wiki): Updated Linking to OpenSSL (markdown) --- Linking-to-OpenSSL.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/Linking-to-OpenSSL.md b/Linking-to-OpenSSL.md index fb6a2e9143..8e271ca61d 100644 --- a/Linking-to-OpenSSL.md +++ b/Linking-to-OpenSSL.md @@ -23,6 +23,17 @@ Good native addons should account for both scenarios. It's recommended that you # header files that were used when building node. 'include_dirs': [ '<(node_root_dir)/deps/openssl/openssl/include' + ], + "conditions" : [ + ["target_arch=='ia32'", { + "include_dirs": [ "<(node_root_dir)/deps/openssl/config/piii" ] + }], + ["target_arch=='x64'", { + "include_dirs": [ "<(node_root_dir)/deps/openssl/config/k8" ] + }], + ["target_arch=='arm'", { + "include_dirs": [ "<(node_root_dir)/deps/openssl/config/arm" ] + }] ] }] ] From 1575bce3a53db628bfb023fd6f3258fdf98c3195 Mon Sep 17 00:00:00 2001 From: rvagg Date: Fri, 2 Nov 2012 19:03:44 -0700 Subject: [PATCH 232/551] doc(wiki): added levelup --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 19928493f7..2ac9098139 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -25,4 +25,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [v8-profiler](https://github.com/c4milo/v8-profiler/blob/master/binding.gyp) * [airtunes](https://github.com/radioline/node_airtunes/blob/master/binding.gyp) * [node-fann](https://github.com/c4milo/node-fann/blob/master/binding.gyp) - * [node-talib](https://github.com/oransel/node-talib/blob/master/binding.gyp) \ No newline at end of file + * [node-talib](https://github.com/oransel/node-talib/blob/master/binding.gyp) + * [node-levelup](https://github.com/rvagg/node-levelup/blob/master/binding.gyp) + [leveldb.gyp](https://github.com/rvagg/node-levelup/blob/master/deps/leveldb/leveldb.gyp) + [snappy.gyp](https://github.com/rvagg/node-levelup/blob/master/deps/snappy/snappy.gyp) \ No newline at end of file From a9b70968fb956eab3b95672048b94350e1565ca3 Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Sun, 4 Nov 2012 22:56:23 -0800 Subject: [PATCH 233/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 2ac9098139..23ae638958 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -5,7 +5,7 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [libxmljs](https://github.com/polotek/libxmljs/blob/master/binding.gyp) * [node-buffertools](https://github.com/bnoordhuis/node-buffertools/blob/master/binding.gyp) * [node-canvas](https://github.com/LearnBoost/node-canvas/blob/master/binding.gyp) - * [node-ffi](https://github.com/rbranson/node-ffi/blob/master/binding.gyp) + * [node-ffi](https://github.com/rbranson/node-ffi/blob/master/binding.gyp) + [libffi](https://github.com/rbranson/node-ffi/blob/master/deps/libffi/libffi.gyp) * [node-time](https://github.com/TooTallNate/node-time/blob/master/binding.gyp) * [node-serialport](https://github.com/voodootikigod/node-serialport/blob/master/binding.gyp) * [node-weak](https://github.com/TooTallNate/node-weak/blob/master/binding.gyp) From 3de9e17e0b8a387eafe7bd18d0ec1e3191d118e8 Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Sun, 4 Nov 2012 22:57:38 -0800 Subject: [PATCH 234/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 23ae638958..5d8552f691 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -17,7 +17,7 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-memwatch](https://github.com/lloyd/node-memwatch/blob/master/binding.gyp) * [node-ip2location](https://github.com/bolgovr/node-ip2location/blob/master/binding.gyp) * [node-midi](https://github.com/justinlatimer/node-midi/blob/master/binding.gyp) - * [node-sqlite3](https://github.com/developmentseed/node-sqlite3/blob/master/binding.gyp) + * [node-sqlite3](https://github.com/developmentseed/node-sqlite3/blob/master/binding.gyp) + [libsqlite3](https://github.com/developmentseed/node-sqlite3/blob/master/deps/sqlite3/binding.gyp) * [node-srs](https://github.com/springmeyer/node-srs/blob/master/build.gyp) * [node-zipfile](https://github.com/springmeyer/node-zipfile/blob/master/build.gyp) * [node-mapnik](https://github.com/mapnik/node-mapnik/blob/master/build.gyp) From d1cd237bad06fa507adb354b9e2181a14dc63d24 Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Sun, 4 Nov 2012 23:12:06 -0800 Subject: [PATCH 235/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 5d8552f691..2fe0d833d5 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -26,4 +26,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [airtunes](https://github.com/radioline/node_airtunes/blob/master/binding.gyp) * [node-fann](https://github.com/c4milo/node-fann/blob/master/binding.gyp) * [node-talib](https://github.com/oransel/node-talib/blob/master/binding.gyp) - * [node-levelup](https://github.com/rvagg/node-levelup/blob/master/binding.gyp) + [leveldb.gyp](https://github.com/rvagg/node-levelup/blob/master/deps/leveldb/leveldb.gyp) + [snappy.gyp](https://github.com/rvagg/node-levelup/blob/master/deps/snappy/snappy.gyp) \ No newline at end of file + * [node-levelup](https://github.com/rvagg/node-levelup/blob/master/binding.gyp) + [leveldb.gyp](https://github.com/rvagg/node-levelup/blob/master/deps/leveldb/leveldb.gyp) + [snappy.gyp](https://github.com/rvagg/node-levelup/blob/master/deps/snappy/snappy.gyp) + * [node-expat](https://github.com/astro/node-expat/blob/master/binding.gyp) + [libexpat](https://github.com/astro/node-expat/blob/master/deps/libexpat/libexpat.gyp) \ No newline at end of file From e0ac8d15af46aadd1c220599e63199b154a514e6 Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Tue, 4 Dec 2012 15:15:44 -0800 Subject: [PATCH 236/551] doc(wiki): Created Updating npm's bundled node gyp (markdown) --- Updating-npm's-bundled-node-gyp.md | 38 ++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 Updating-npm's-bundled-node-gyp.md diff --git a/Updating-npm's-bundled-node-gyp.md b/Updating-npm's-bundled-node-gyp.md new file mode 100644 index 0000000000..e9f9d00744 --- /dev/null +++ b/Updating-npm's-bundled-node-gyp.md @@ -0,0 +1,38 @@ +`npm` bundles its own, internal, copy of `node-gyp`. This internal copy is independent of any globally installed copy of node-gyp that you may have installed via `npm install -g node-gyp`. + +This means that while `node-gyp` doesn't get installed into your `$PATH` by default, npm still keeps its own copy to invoke when you attempt to `npm install` a native addon. + +Sometimes, you may need to update npm's internal node-gyp to a newer version than what is installed. A simple `npm install -g node-gyp` _won't_ do the trick since npm will still continue to use its internal copy over the global one. + +So instead: + +### Linux, Mac OS X, Solaris, etc. + +Unix is easy. Just run the following command. Use `sudo` if necessary. + +``` bash +$ [sudo] npm explore npm -g -- npm install node-gyp +``` + +### Windows + +Windows is a bit tricker, since `npm` gets installed to the "Program Files" directory, which needs admin privileges in order to modify on current Windows. Therefore, run the following commands __inside a `cmd.exe` started with "Run as Administrator"__: + +First we need to find the location of `node`. If you don't already know the location that `node.exe` got installed to, then run: + +``` bash +$ npm install -g which +$ which node +``` + +Now `cd` to the directory that `node.exe` is contained in, and with `node_modules\npm` at the end. i.e.: + +``` bash +$ cd "C:\Program Files\nodejs\node_modules\npm" +``` + +Now you can finally run: + +``` bash +npm install node-gyp +``` \ No newline at end of file From e50e04d7b6a3754ea0aa11fe8cef491b3bc5bdd4 Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Tue, 4 Dec 2012 15:18:13 -0800 Subject: [PATCH 237/551] doc(wiki): Updated Updating npm's bundled node gyp (markdown) --- Updating-npm's-bundled-node-gyp.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Updating-npm's-bundled-node-gyp.md b/Updating-npm's-bundled-node-gyp.md index e9f9d00744..c7ad532ccd 100644 --- a/Updating-npm's-bundled-node-gyp.md +++ b/Updating-npm's-bundled-node-gyp.md @@ -34,5 +34,5 @@ $ cd "C:\Program Files\nodejs\node_modules\npm" Now you can finally run: ``` bash -npm install node-gyp +$ npm install node-gyp ``` \ No newline at end of file From 979a7063b950c088a7f4896fc3a48e1d00dfd231 Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Tue, 4 Dec 2012 15:18:35 -0800 Subject: [PATCH 238/551] doc(wiki): Updated Updating npm's bundled node gyp (markdown) --- Updating-npm's-bundled-node-gyp.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Updating-npm's-bundled-node-gyp.md b/Updating-npm's-bundled-node-gyp.md index c7ad532ccd..c0ef9b7382 100644 --- a/Updating-npm's-bundled-node-gyp.md +++ b/Updating-npm's-bundled-node-gyp.md @@ -6,7 +6,7 @@ Sometimes, you may need to update npm's internal node-gyp to a newer version tha So instead: -### Linux, Mac OS X, Solaris, etc. +## Linux, Mac OS X, Solaris, etc. Unix is easy. Just run the following command. Use `sudo` if necessary. @@ -14,7 +14,7 @@ Unix is easy. Just run the following command. Use `sudo` if necessary. $ [sudo] npm explore npm -g -- npm install node-gyp ``` -### Windows +## Windows Windows is a bit tricker, since `npm` gets installed to the "Program Files" directory, which needs admin privileges in order to modify on current Windows. Therefore, run the following commands __inside a `cmd.exe` started with "Run as Administrator"__: From 4a7f2d0d869a65c99a78504976567017edadf657 Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Tue, 4 Dec 2012 15:58:43 -0800 Subject: [PATCH 239/551] doc(wiki): Updated Updating npm's bundled node gyp (markdown) --- Updating-npm's-bundled-node-gyp.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Updating-npm's-bundled-node-gyp.md b/Updating-npm's-bundled-node-gyp.md index c0ef9b7382..c10232dc20 100644 --- a/Updating-npm's-bundled-node-gyp.md +++ b/Updating-npm's-bundled-node-gyp.md @@ -11,7 +11,7 @@ So instead: Unix is easy. Just run the following command. Use `sudo` if necessary. ``` bash -$ [sudo] npm explore npm -g -- npm install node-gyp +$ [sudo] npm explore npm -g -- npm install node-gyp@latest ``` ## Windows @@ -34,5 +34,5 @@ $ cd "C:\Program Files\nodejs\node_modules\npm" Now you can finally run: ``` bash -$ npm install node-gyp +$ npm install node-gyp@latest ``` \ No newline at end of file From 33561e9cbf5f4eb46111318503c77df2c6eb484a Mon Sep 17 00:00:00 2001 From: ehansin Date: Fri, 1 Mar 2013 22:51:15 -0800 Subject: [PATCH 240/551] doc(wiki): Updated Updating npm's bundled node gyp (markdown) --- Updating-npm's-bundled-node-gyp.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Updating-npm's-bundled-node-gyp.md b/Updating-npm's-bundled-node-gyp.md index c10232dc20..0f512d37b0 100644 --- a/Updating-npm's-bundled-node-gyp.md +++ b/Updating-npm's-bundled-node-gyp.md @@ -25,6 +25,12 @@ $ npm install -g which $ which node ``` +As an alternative to the above, those on Windows Server 2003 and later (this includes Windows 7) can run: + +``` bash +$ where node +``` + Now `cd` to the directory that `node.exe` is contained in, and with `node_modules\npm` at the end. i.e.: ``` bash From 5b80e834c8f79dda9fb2770a876ff3cf649c06f3 Mon Sep 17 00:00:00 2001 From: xverges Date: Thu, 7 Mar 2013 04:16:20 -0800 Subject: [PATCH 241/551] doc(wiki): Created Visual Studio 2010 Setup (markdown) --- Visual-Studio-2010-Setup.md | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 Visual-Studio-2010-Setup.md diff --git a/Visual-Studio-2010-Setup.md b/Visual-Studio-2010-Setup.md new file mode 100644 index 0000000000..c5a104016c --- /dev/null +++ b/Visual-Studio-2010-Setup.md @@ -0,0 +1,10 @@ +On Windows XP/Vista/7, [node-gyp requires Python 2.3 and Visual Studio 2010](https://github.com/TooTallNate/node-gyp#installation) + +According to the readme file in [Microsoft Visual C++ 2010 Service Pack 1 Compiler Update for the Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=4422), _to ensure that your system has a supported configuration, uninstall the following products and then reinstall them in the order listed_: + +1. [Visual Studio 2010](http://www.microsoft.com/visualstudio/eng/downloads#d-2010-express) +1. [Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=8279) +1. [Visual Studio 2010 SP1](http://www.microsoft.com/en-us/download/details.aspx?id=23691) +1. [Visual C++ 2010 SP1 Compiler Update for the Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=4422) + +On x64 environments, the last update in the list fixes errors about missing compilers and `error MSB4019: The imported project "C:\Microsoft.Cpp.Default.props" was not found.` \ No newline at end of file From 0e37ff48b306c12149661b375895741d3d710da7 Mon Sep 17 00:00:00 2001 From: Niggler Date: Sat, 16 Mar 2013 11:14:51 -0700 Subject: [PATCH 242/551] doc(wiki): Updated Home (markdown) --- Home.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Home.md b/Home.md index 189e315b3e..42e0591ab9 100644 --- a/Home.md +++ b/Home.md @@ -1,4 +1,5 @@ Welcome to the node-gyp wiki! * [["binding.gyp" files out in the wild]] - * [[Linking to OpenSSL]] \ No newline at end of file + * [[Linking to OpenSSL]] + * [[Common Issues]] \ No newline at end of file From a38299ea340ceb0e732c6dc6a1b4760257644839 Mon Sep 17 00:00:00 2001 From: Niggler Date: Sat, 16 Mar 2013 11:17:05 -0700 Subject: [PATCH 243/551] doc(wiki): Created Common issues (markdown) --- Common-issues.md | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 Common-issues.md diff --git a/Common-issues.md b/Common-issues.md new file mode 100644 index 0000000000..5648114bf6 --- /dev/null +++ b/Common-issues.md @@ -0,0 +1,7 @@ +## Python Issues OSX + +Make sure you are using the native Python version in OSX. If you use a MacPorts of HomeBrew version, you may run into problems. + +If you have issues with `execvp`, be sure to check your `$PYTHON` environment variable. If it is not set to the native version, unset it and try again. + +Notes: https://gist.github.com/erichocean/5177582 \ No newline at end of file From ea28f0947af91fa638be355143f5df89d2e431c8 Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Mon, 15 Apr 2013 14:48:37 -0700 Subject: [PATCH 244/551] doc(wiki): Updated Home (markdown) --- Home.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Home.md b/Home.md index 42e0591ab9..207745fe8a 100644 --- a/Home.md +++ b/Home.md @@ -2,4 +2,5 @@ Welcome to the node-gyp wiki! * [["binding.gyp" files out in the wild]] * [[Linking to OpenSSL]] - * [[Common Issues]] \ No newline at end of file + * [[Common Issues]] + * [[Updating npm's bundled node-gyp]] \ No newline at end of file From 4eda8275c03dae6d2f5c40f3c1dbe930d84b0f2b Mon Sep 17 00:00:00 2001 From: felquis Date: Thu, 18 Apr 2013 06:46:12 -0700 Subject: [PATCH 245/551] doc(wiki): Add helpful information --- Visual-Studio-2010-Setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Visual-Studio-2010-Setup.md b/Visual-Studio-2010-Setup.md index c5a104016c..f80b295ece 100644 --- a/Visual-Studio-2010-Setup.md +++ b/Visual-Studio-2010-Setup.md @@ -3,7 +3,7 @@ On Windows XP/Vista/7, [node-gyp requires Python 2.3 and Visual Studio 2010](htt According to the readme file in [Microsoft Visual C++ 2010 Service Pack 1 Compiler Update for the Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=4422), _to ensure that your system has a supported configuration, uninstall the following products and then reinstall them in the order listed_: 1. [Visual Studio 2010](http://www.microsoft.com/visualstudio/eng/downloads#d-2010-express) -1. [Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=8279) +1. [Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=8279) Note: If you get error on installation, maybe [this link ](http://stackoverflow.com/questions/1901279/windows-7-sdk-installation-failure) will help you. 1. [Visual Studio 2010 SP1](http://www.microsoft.com/en-us/download/details.aspx?id=23691) 1. [Visual C++ 2010 SP1 Compiler Update for the Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=4422) From 98bc80d7a62ba70c881f3c39d94f804322e57852 Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Tue, 23 Apr 2013 17:00:08 -0700 Subject: [PATCH 246/551] doc(wiki): Created Error: "pre" versions of node cannot be installed (markdown) --- ..."-versions-of-node-cannot-be-installed.md" | 68 +++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 "Error:-\"pre\"-versions-of-node-cannot-be-installed.md" diff --git "a/Error:-\"pre\"-versions-of-node-cannot-be-installed.md" "b/Error:-\"pre\"-versions-of-node-cannot-be-installed.md" new file mode 100644 index 0000000000..decf2e1ea9 --- /dev/null +++ "b/Error:-\"pre\"-versions-of-node-cannot-be-installed.md" @@ -0,0 +1,68 @@ +When using `node-gyp` you might see an error like this when attempting to compile/install a node.js native addon: + +``` +$ npm install bcrypt +npm http GET https://registry.npmjs.org/bcrypt/0.7.5 +npm http 304 https://registry.npmjs.org/bcrypt/0.7.5 +npm http GET https://registry.npmjs.org/bindings/1.0.0 +npm http 304 https://registry.npmjs.org/bindings/1.0.0 + +> bcrypt@0.7.5 install /home/ubuntu/public/song-swap/node_modules/bcrypt +> node-gyp rebuild + +gyp ERR! configure error +gyp ERR! stack Error: "pre" versions of node cannot be installed, use the --nodedir flag instead +gyp ERR! stack at install (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/install.js:69:16) +gyp ERR! stack at Object.self.commands.(anonymous function) [as install] (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/node-gyp.js:56:37) +gyp ERR! stack at getNodeDir (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:219:20) +gyp ERR! stack at /usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:105:9 +gyp ERR! stack at ChildProcess.exithandler (child_process.js:630:7) +gyp ERR! stack at ChildProcess.EventEmitter.emit (events.js:99:17) +gyp ERR! stack at maybeClose (child_process.js:730:16) +gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:797:5) +gyp ERR! System Linux 3.5.0-21-generic +gyp ERR! command "node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild" +gyp ERR! cwd /home/ubuntu/public/song-swap/node_modules/bcrypt +gyp ERR! node -v v0.11.2-pre +gyp ERR! node-gyp -v v0.9.5 +gyp ERR! not ok +npm ERR! bcrypt@0.7.5 install: `node-gyp rebuild` +npm ERR! `sh "-c" "node-gyp rebuild"` failed with 1 +npm ERR! +npm ERR! Failed at the bcrypt@0.7.5 install script. +npm ERR! This is most likely a problem with the bcrypt package, +npm ERR! not with npm itself. +npm ERR! Tell the author that this fails on your system: +npm ERR! node-gyp rebuild +npm ERR! You can get their info via: +npm ERR! npm owner ls bcrypt +npm ERR! There is likely additional logging output above. + +npm ERR! System Linux 3.5.0-21-generic +npm ERR! command "/usr/local/bin/node" "/usr/local/bin/npm" "install" "bcrypt" +npm ERR! cwd /home/ubuntu/public/song-swap +npm ERR! node -v v0.11.2-pre +npm ERR! npm -v 1.2.18 +npm ERR! code ELIFECYCLE +npm ERR! +npm ERR! Additional logging details can be found in: +npm ERR! /home/ubuntu/public/song-swap/npm-debug.log +npm ERR! not ok code 0 +``` + +The main error here is: + +``` +Error: "pre" versions of node cannot be installed, use the --nodedir flag instead +``` + +This error is caused when you attempt to compile a native addon using a version of node.js with `-pre` at the end of the version number. + +## How to avoid (the short answer) + +To avoid this error completely just use a stable release of node.js. i.e. `v0.10.4`, and __not__ `v0.10.4-pre`. + +## How to fix (the long answer) + +This error happens because `node-gyp` does not know what header files were used to compile your "pre" version of node, and therefore it needs you to specify the node source code directory path using the `--nodedir` flag. + From e9f8b33d1f87d04f22cb09a814d7c55d0fa38446 Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Tue, 23 Apr 2013 17:07:03 -0700 Subject: [PATCH 247/551] doc(wiki): Updated Error: "pre" versions of node cannot be installed (markdown) --- ..."-versions-of-node-cannot-be-installed.md" | 28 ++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git "a/Error:-\"pre\"-versions-of-node-cannot-be-installed.md" "b/Error:-\"pre\"-versions-of-node-cannot-be-installed.md" index decf2e1ea9..c1e1158d70 100644 --- "a/Error:-\"pre\"-versions-of-node-cannot-be-installed.md" +++ "b/Error:-\"pre\"-versions-of-node-cannot-be-installed.md" @@ -56,7 +56,12 @@ The main error here is: Error: "pre" versions of node cannot be installed, use the --nodedir flag instead ``` -This error is caused when you attempt to compile a native addon using a version of node.js with `-pre` at the end of the version number. +This error is caused when you attempt to compile a native addon using a version of node.js with `-pre` at the end of the version number: + +``` bash +$ node -v +v0.10.4-pre +``` ## How to avoid (the short answer) @@ -66,3 +71,24 @@ To avoid this error completely just use a stable release of node.js. i.e. `v0.10 This error happens because `node-gyp` does not know what header files were used to compile your "pre" version of node, and therefore it needs you to specify the node source code directory path using the `--nodedir` flag. +For example, if I compiled my development ("pre") version of node.js using the source code in `/Users/nrajlich/node`, then I could invoke `node-gyp` like: + +``` bash +$ node-gyp rebuild --nodedir=/Users/nrajlich/node +``` + +Or install an native addon through `npm` like: + +``` bash +$ npm install bcrypt --nodedir=/Users/nrajlich/node +``` + +### Always use `--nodedir` + +__Note:__ This is for advanced users who use `-pre` versions of node more often than tagged releases. + +If you're invoking `node-gyp` through `npm`, then you can leverage `npm`'s configuration system and not have to specify the `--nodedir` flag all the time: + +``` bash +$ npm config set nodedir /Users/nrajlich/node +``` \ No newline at end of file From 65efe32ccb8d446ce569453364f922dd9d27c945 Mon Sep 17 00:00:00 2001 From: TooTallNate Date: Wed, 24 Apr 2013 20:42:00 -0700 Subject: [PATCH 248/551] doc(wiki): Updated Home (markdown) --- Home.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Home.md b/Home.md index 207745fe8a..68bc828d1e 100644 --- a/Home.md +++ b/Home.md @@ -3,4 +3,5 @@ Welcome to the node-gyp wiki! * [["binding.gyp" files out in the wild]] * [[Linking to OpenSSL]] * [[Common Issues]] - * [[Updating npm's bundled node-gyp]] \ No newline at end of file + * [[Updating npm's bundled node-gyp]] + * [[Error: "pre" versions of node cannot be installed]] \ No newline at end of file From 54db8d7ac33e3f98220960b5d86cfa18a75b53cb Mon Sep 17 00:00:00 2001 From: Dane Springmeyer Date: Mon, 17 Jun 2013 13:13:08 -0700 Subject: [PATCH 249/551] doc(wiki): fix link to gyp file used to build libsqlite3 --- "\"binding.gyp\"-files-out-in-the-wild.md" | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 2fe0d833d5..8231be156e 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -17,7 +17,7 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-memwatch](https://github.com/lloyd/node-memwatch/blob/master/binding.gyp) * [node-ip2location](https://github.com/bolgovr/node-ip2location/blob/master/binding.gyp) * [node-midi](https://github.com/justinlatimer/node-midi/blob/master/binding.gyp) - * [node-sqlite3](https://github.com/developmentseed/node-sqlite3/blob/master/binding.gyp) + [libsqlite3](https://github.com/developmentseed/node-sqlite3/blob/master/deps/sqlite3/binding.gyp) + * [node-sqlite3](https://github.com/developmentseed/node-sqlite3/blob/master/binding.gyp) + [libsqlite3](https://github.com/developmentseed/node-sqlite3/blob/master/deps/sqlite3.gyp) * [node-srs](https://github.com/springmeyer/node-srs/blob/master/build.gyp) * [node-zipfile](https://github.com/springmeyer/node-zipfile/blob/master/build.gyp) * [node-mapnik](https://github.com/mapnik/node-mapnik/blob/master/build.gyp) From 81bfa1f1b63d522a9f8a9ae9ca0c7ae90fe75140 Mon Sep 17 00:00:00 2001 From: Dane Springmeyer Date: Mon, 17 Jun 2013 13:15:00 -0700 Subject: [PATCH 250/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 8231be156e..686838dcae 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -18,9 +18,9 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-ip2location](https://github.com/bolgovr/node-ip2location/blob/master/binding.gyp) * [node-midi](https://github.com/justinlatimer/node-midi/blob/master/binding.gyp) * [node-sqlite3](https://github.com/developmentseed/node-sqlite3/blob/master/binding.gyp) + [libsqlite3](https://github.com/developmentseed/node-sqlite3/blob/master/deps/sqlite3.gyp) - * [node-srs](https://github.com/springmeyer/node-srs/blob/master/build.gyp) + * [node-srs](https://github.com/springmeyer/node-srs/blob/master/binding.gyp) * [node-zipfile](https://github.com/springmeyer/node-zipfile/blob/master/build.gyp) - * [node-mapnik](https://github.com/mapnik/node-mapnik/blob/master/build.gyp) + * [node-mapnik](https://github.com/mapnik/node-mapnik/blob/master/binding.gyp) * [node-inotify](https://github.com/c4milo/node-inotify/blob/master/binding.gyp) * [v8-profiler](https://github.com/c4milo/v8-profiler/blob/master/binding.gyp) * [airtunes](https://github.com/radioline/node_airtunes/blob/master/binding.gyp) From 55ebd6ebacde975bf84f7bf4d8c66e64cc7cd0da Mon Sep 17 00:00:00 2001 From: Alex Treppass Date: Mon, 5 Aug 2013 09:08:02 -0700 Subject: [PATCH 251/551] doc(wiki): Bumping Python version from 2.3 to 2.7 as per the node-gyp readme --- Visual-Studio-2010-Setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Visual-Studio-2010-Setup.md b/Visual-Studio-2010-Setup.md index f80b295ece..65d2e977f5 100644 --- a/Visual-Studio-2010-Setup.md +++ b/Visual-Studio-2010-Setup.md @@ -1,4 +1,4 @@ -On Windows XP/Vista/7, [node-gyp requires Python 2.3 and Visual Studio 2010](https://github.com/TooTallNate/node-gyp#installation) +On Windows XP/Vista/7, [node-gyp requires Python 2.7 and Visual Studio 2010](https://github.com/TooTallNate/node-gyp#installation) According to the readme file in [Microsoft Visual C++ 2010 Service Pack 1 Compiler Update for the Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=4422), _to ensure that your system has a supported configuration, uninstall the following products and then reinstall them in the order listed_: From 61f709ec4d9f256a6467e9ff84430a48eeb629d1 Mon Sep 17 00:00:00 2001 From: Luis Reis Date: Tue, 13 Aug 2013 15:20:42 -0700 Subject: [PATCH 252/551] doc(wiki): Add node-openvg-canvas and node-openvg. --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 686838dcae..fac8320a7e 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -27,4 +27,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-fann](https://github.com/c4milo/node-fann/blob/master/binding.gyp) * [node-talib](https://github.com/oransel/node-talib/blob/master/binding.gyp) * [node-levelup](https://github.com/rvagg/node-levelup/blob/master/binding.gyp) + [leveldb.gyp](https://github.com/rvagg/node-levelup/blob/master/deps/leveldb/leveldb.gyp) + [snappy.gyp](https://github.com/rvagg/node-levelup/blob/master/deps/snappy/snappy.gyp) - * [node-expat](https://github.com/astro/node-expat/blob/master/binding.gyp) + [libexpat](https://github.com/astro/node-expat/blob/master/deps/libexpat/libexpat.gyp) \ No newline at end of file + * [node-expat](https://github.com/astro/node-expat/blob/master/binding.gyp) + [libexpat](https://github.com/astro/node-expat/blob/master/deps/libexpat/libexpat.gyp) + * [node-openvg-canvas](https://github.com/luismreis/node-openvg-canvas/blob/master/binding.gyp) + [node-openvg](https://github.com/luismreis/node-openvg/blob/master/binding.gyp) \ No newline at end of file From 6e392bcdd3dd1691773e6e16e1dffc35931b81e0 Mon Sep 17 00:00:00 2001 From: lilo003 Date: Sun, 18 Aug 2013 09:03:20 -0700 Subject: [PATCH 253/551] doc(wiki): Updated Home (markdown) --- Home.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Home.md b/Home.md index 68bc828d1e..59531ad731 100644 --- a/Home.md +++ b/Home.md @@ -4,4 +4,5 @@ Welcome to the node-gyp wiki! * [[Linking to OpenSSL]] * [[Common Issues]] * [[Updating npm's bundled node-gyp]] - * [[Error: "pre" versions of node cannot be installed]] \ No newline at end of file + * [[Error: "pre" versions of node cannot be installed]] + * [[Visual Studio 2010 Setup]] \ No newline at end of file From 875adbe2a4669fa5f2be0250ffbf98fb55e800fd Mon Sep 17 00:00:00 2001 From: Syrian watermelon Date: Wed, 18 Sep 2013 15:41:10 -0700 Subject: [PATCH 254/551] doc(wiki): Adding link to node-cryptopp's gyp file --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index fac8320a7e..3f3f6765a0 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -28,4 +28,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-talib](https://github.com/oransel/node-talib/blob/master/binding.gyp) * [node-levelup](https://github.com/rvagg/node-levelup/blob/master/binding.gyp) + [leveldb.gyp](https://github.com/rvagg/node-levelup/blob/master/deps/leveldb/leveldb.gyp) + [snappy.gyp](https://github.com/rvagg/node-levelup/blob/master/deps/snappy/snappy.gyp) * [node-expat](https://github.com/astro/node-expat/blob/master/binding.gyp) + [libexpat](https://github.com/astro/node-expat/blob/master/deps/libexpat/libexpat.gyp) - * [node-openvg-canvas](https://github.com/luismreis/node-openvg-canvas/blob/master/binding.gyp) + [node-openvg](https://github.com/luismreis/node-openvg/blob/master/binding.gyp) \ No newline at end of file + * [node-openvg-canvas](https://github.com/luismreis/node-openvg-canvas/blob/master/binding.gyp) + [node-openvg](https://github.com/luismreis/node-openvg/blob/master/binding.gyp) + * [node-cryptopp](https://github.com/BatikhSouri/node-cryptopp/blob/master/binding.gyp) \ No newline at end of file From 8919028921fd304f08044098434f0dc6071fb7cf Mon Sep 17 00:00:00 2001 From: Evan Su Date: Thu, 19 Sep 2013 21:05:19 -0700 Subject: [PATCH 255/551] doc(wiki): Updated Linking to OpenSSL (markdown) --- Linking-to-OpenSSL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Linking-to-OpenSSL.md b/Linking-to-OpenSSL.md index 8e271ca61d..ec80929995 100644 --- a/Linking-to-OpenSSL.md +++ b/Linking-to-OpenSSL.md @@ -1,4 +1,4 @@ -A handful of native addons require linking to OpenSSL in one way or another. This introduces a small chalenge since node will sometimes bundle OpenSSL statically (the default for node >= v0.8.x), or sometimes dynamically link to the system OpenSSL (default for node <= v0.6.x). +A handful of native addons require linking to OpenSSL in one way or another. This introduces a small challenge since node will sometimes bundle OpenSSL statically (the default for node >= v0.8.x), or sometimes dynamically link to the system OpenSSL (default for node <= v0.6.x). Good native addons should account for both scenarios. It's recommended that you use the `binding.gyp` file provided below as a starting-point for any addon that needs to use OpenSSL: From 1a75d2bf2f562ba50846893a516e111cfbb50885 Mon Sep 17 00:00:00 2001 From: Dane Springmeyer Date: Sat, 5 Oct 2013 13:02:53 -0700 Subject: [PATCH 256/551] doc(wiki): add topcube, node-osmium, and node-osrm --- "\"binding.gyp\"-files-out-in-the-wild.md" | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 3f3f6765a0..62803ee604 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -29,4 +29,7 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-levelup](https://github.com/rvagg/node-levelup/blob/master/binding.gyp) + [leveldb.gyp](https://github.com/rvagg/node-levelup/blob/master/deps/leveldb/leveldb.gyp) + [snappy.gyp](https://github.com/rvagg/node-levelup/blob/master/deps/snappy/snappy.gyp) * [node-expat](https://github.com/astro/node-expat/blob/master/binding.gyp) + [libexpat](https://github.com/astro/node-expat/blob/master/deps/libexpat/libexpat.gyp) * [node-openvg-canvas](https://github.com/luismreis/node-openvg-canvas/blob/master/binding.gyp) + [node-openvg](https://github.com/luismreis/node-openvg/blob/master/binding.gyp) - * [node-cryptopp](https://github.com/BatikhSouri/node-cryptopp/blob/master/binding.gyp) \ No newline at end of file + * [node-cryptopp](https://github.com/BatikhSouri/node-cryptopp/blob/master/binding.gyp) + * [topcube](https://github.com/creationix/topcube/blob/master/binding.gyp) + * [node-osmium](https://github.com/springmeyer/node-osmium/blob/master/binding.gyp) + * [node-osrm](https://github.com/DennisOSRM/node-osrm) \ No newline at end of file From 65ba71139e9b7f64ac823e575ee9dbf17d937ce4 Mon Sep 17 00:00:00 2001 From: fov42550564 <42550564@qq.com> Date: Thu, 31 Oct 2013 22:24:37 -0700 Subject: [PATCH 257/551] doc(wiki): Created use of undeclared identifier 'TypedArray' (markdown) --- use-of-undeclared-identifier-'TypedArray'.md | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 use-of-undeclared-identifier-'TypedArray'.md diff --git a/use-of-undeclared-identifier-'TypedArray'.md b/use-of-undeclared-identifier-'TypedArray'.md new file mode 100644 index 0000000000..9034b893d7 --- /dev/null +++ b/use-of-undeclared-identifier-'TypedArray'.md @@ -0,0 +1,2 @@ +Local typedArray = Local::Cast(arg[0]); +why node-gyp not support typedArray \ No newline at end of file From becef316b6c46a33e783667720ee074a0141d1a5 Mon Sep 17 00:00:00 2001 From: tcbeutler Date: Mon, 30 Dec 2013 07:33:39 -0800 Subject: [PATCH 258/551] doc(wiki): Created Visual studio 2012 setup (markdown) --- Visual-studio-2012-setup.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 Visual-studio-2012-setup.md diff --git a/Visual-studio-2012-setup.md b/Visual-studio-2012-setup.md new file mode 100644 index 0000000000..30fcf1b802 --- /dev/null +++ b/Visual-studio-2012-setup.md @@ -0,0 +1 @@ +Explodes \ No newline at end of file From 3601508bb10fa05da0ddc7e70d57e4b4dd679657 Mon Sep 17 00:00:00 2001 From: tcbeutler Date: Mon, 30 Dec 2013 07:33:54 -0800 Subject: [PATCH 259/551] doc(wiki): Destroyed Visual studio 2012 setup (markdown) --- Visual-studio-2012-setup.md | 1 - 1 file changed, 1 deletion(-) delete mode 100644 Visual-studio-2012-setup.md diff --git a/Visual-studio-2012-setup.md b/Visual-studio-2012-setup.md deleted file mode 100644 index 30fcf1b802..0000000000 --- a/Visual-studio-2012-setup.md +++ /dev/null @@ -1 +0,0 @@ -Explodes \ No newline at end of file From fae7516a1d2829b6e234eaded74fb112ebd79a05 Mon Sep 17 00:00:00 2001 From: The Syrian Watermelon Date: Sun, 2 Feb 2014 14:34:45 -0800 Subject: [PATCH 260/551] doc(wiki): Correcting the link to node-osmium --- "\"binding.gyp\"-files-out-in-the-wild.md" | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 62803ee604..a53d3ba815 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -31,5 +31,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-openvg-canvas](https://github.com/luismreis/node-openvg-canvas/blob/master/binding.gyp) + [node-openvg](https://github.com/luismreis/node-openvg/blob/master/binding.gyp) * [node-cryptopp](https://github.com/BatikhSouri/node-cryptopp/blob/master/binding.gyp) * [topcube](https://github.com/creationix/topcube/blob/master/binding.gyp) - * [node-osmium](https://github.com/springmeyer/node-osmium/blob/master/binding.gyp) + * [node-osmium](https://github.com/osmcode/node-osmium/blob/master/binding.gyp) * [node-osrm](https://github.com/DennisOSRM/node-osrm) \ No newline at end of file From 88411588f300e9b7c00fe516ecd977a1feeeb15c Mon Sep 17 00:00:00 2001 From: raztus Date: Fri, 7 Mar 2014 17:02:38 -0800 Subject: [PATCH 261/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index a53d3ba815..3e9b430b5c 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -32,4 +32,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-cryptopp](https://github.com/BatikhSouri/node-cryptopp/blob/master/binding.gyp) * [topcube](https://github.com/creationix/topcube/blob/master/binding.gyp) * [node-osmium](https://github.com/osmcode/node-osmium/blob/master/binding.gyp) - * [node-osrm](https://github.com/DennisOSRM/node-osrm) \ No newline at end of file + * [node-osrm](https://github.com/DennisOSRM/node-osrm) + * [node-oracle](https://github.com/joeferner/node-oracle/blob/master/binding.gyp) \ No newline at end of file From 92e49a858ed69cb4847a26a5676ab56ef5e2de33 Mon Sep 17 00:00:00 2001 From: raztus Date: Fri, 7 Mar 2014 17:14:33 -0800 Subject: [PATCH 262/551] doc(wiki): Fix link to node-zipfile --- "\"binding.gyp\"-files-out-in-the-wild.md" | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 3e9b430b5c..e0c7d825d9 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -19,7 +19,7 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-midi](https://github.com/justinlatimer/node-midi/blob/master/binding.gyp) * [node-sqlite3](https://github.com/developmentseed/node-sqlite3/blob/master/binding.gyp) + [libsqlite3](https://github.com/developmentseed/node-sqlite3/blob/master/deps/sqlite3.gyp) * [node-srs](https://github.com/springmeyer/node-srs/blob/master/binding.gyp) - * [node-zipfile](https://github.com/springmeyer/node-zipfile/blob/master/build.gyp) + * [node-zipfile](https://github.com/mapbox/node-zipfile/blob/master/binding.gyp) * [node-mapnik](https://github.com/mapnik/node-mapnik/blob/master/binding.gyp) * [node-inotify](https://github.com/c4milo/node-inotify/blob/master/binding.gyp) * [v8-profiler](https://github.com/c4milo/v8-profiler/blob/master/binding.gyp) From 378c3632f02c096ed819ec8f2611c65bef0c0554 Mon Sep 17 00:00:00 2001 From: vweevers Date: Sun, 29 Jun 2014 07:58:40 -0700 Subject: [PATCH 263/551] doc(wiki): Explicit link to Visual C++ 2010 Express --- Visual-Studio-2010-Setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Visual-Studio-2010-Setup.md b/Visual-Studio-2010-Setup.md index 65d2e977f5..227becb4ac 100644 --- a/Visual-Studio-2010-Setup.md +++ b/Visual-Studio-2010-Setup.md @@ -2,7 +2,7 @@ On Windows XP/Vista/7, [node-gyp requires Python 2.7 and Visual Studio 2010](htt According to the readme file in [Microsoft Visual C++ 2010 Service Pack 1 Compiler Update for the Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=4422), _to ensure that your system has a supported configuration, uninstall the following products and then reinstall them in the order listed_: -1. [Visual Studio 2010](http://www.microsoft.com/visualstudio/eng/downloads#d-2010-express) +1. [Visual C++ 2010 Express](http://www.microsoft.com/visualstudio/eng/downloads#d-2010-express) or Visual Studio 2010 1. [Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=8279) Note: If you get error on installation, maybe [this link ](http://stackoverflow.com/questions/1901279/windows-7-sdk-installation-failure) will help you. 1. [Visual Studio 2010 SP1](http://www.microsoft.com/en-us/download/details.aspx?id=23691) 1. [Visual C++ 2010 SP1 Compiler Update for the Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=4422) From e64798de8cac6031ad598a86d7599e81b4d20b17 Mon Sep 17 00:00:00 2001 From: Andreas Brekken Date: Mon, 7 Jul 2014 01:40:29 -0700 Subject: [PATCH 264/551] doc(wiki): Added tip about resolving frustrating LNK1181 error --- Visual-Studio-2010-Setup.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Visual-Studio-2010-Setup.md b/Visual-Studio-2010-Setup.md index 227becb4ac..08e69c547b 100644 --- a/Visual-Studio-2010-Setup.md +++ b/Visual-Studio-2010-Setup.md @@ -7,4 +7,6 @@ According to the readme file in [Microsoft Visual C++ 2010 Service Pack 1 Compil 1. [Visual Studio 2010 SP1](http://www.microsoft.com/en-us/download/details.aspx?id=23691) 1. [Visual C++ 2010 SP1 Compiler Update for the Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=4422) -On x64 environments, the last update in the list fixes errors about missing compilers and `error MSB4019: The imported project "C:\Microsoft.Cpp.Default.props" was not found.` \ No newline at end of file +On x64 environments, the last update in the list fixes errors about missing compilers and `error MSB4019: The imported project "C:\Microsoft.Cpp.Default.props" was not found.` + +If you experience the error `LNK1181` file `kernel32.lib` not found, try compiling using the `Windows SDK 7.1 Command Prompt` start menu shortcut. \ No newline at end of file From 59668bb0b904feccf3c09afa2fd37378c77af967 Mon Sep 17 00:00:00 2001 From: ralphtheninja Date: Fri, 15 Aug 2014 02:31:36 -0700 Subject: [PATCH 265/551] doc(wiki): Updated node-levelup to node-leveldown (broken links) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index e0c7d825d9..3ddd2a0956 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -26,7 +26,7 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [airtunes](https://github.com/radioline/node_airtunes/blob/master/binding.gyp) * [node-fann](https://github.com/c4milo/node-fann/blob/master/binding.gyp) * [node-talib](https://github.com/oransel/node-talib/blob/master/binding.gyp) - * [node-levelup](https://github.com/rvagg/node-levelup/blob/master/binding.gyp) + [leveldb.gyp](https://github.com/rvagg/node-levelup/blob/master/deps/leveldb/leveldb.gyp) + [snappy.gyp](https://github.com/rvagg/node-levelup/blob/master/deps/snappy/snappy.gyp) + * [node-leveldown](https://github.com/rvagg/node-leveldown/blob/master/binding.gyp) + [leveldb.gyp](https://github.com/rvagg/node-leveldown/blob/master/deps/leveldb/leveldb.gyp) + [snappy.gyp](https://github.com/rvagg/node-leveldown/blob/master/deps/snappy/snappy.gyp) * [node-expat](https://github.com/astro/node-expat/blob/master/binding.gyp) + [libexpat](https://github.com/astro/node-expat/blob/master/deps/libexpat/libexpat.gyp) * [node-openvg-canvas](https://github.com/luismreis/node-openvg-canvas/blob/master/binding.gyp) + [node-openvg](https://github.com/luismreis/node-openvg/blob/master/binding.gyp) * [node-cryptopp](https://github.com/BatikhSouri/node-cryptopp/blob/master/binding.gyp) From 3d4d9d52d6b5b49de06bb0bb5b68e2686d2b7ebd Mon Sep 17 00:00:00 2001 From: Zeke Sonxx Date: Sun, 24 Aug 2014 15:58:26 -0700 Subject: [PATCH 266/551] doc(wiki): Added details for properly fixing --- Visual-Studio-2010-Setup.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/Visual-Studio-2010-Setup.md b/Visual-Studio-2010-Setup.md index 08e69c547b..4f4333e044 100644 --- a/Visual-Studio-2010-Setup.md +++ b/Visual-Studio-2010-Setup.md @@ -9,4 +9,10 @@ According to the readme file in [Microsoft Visual C++ 2010 Service Pack 1 Compil On x64 environments, the last update in the list fixes errors about missing compilers and `error MSB4019: The imported project "C:\Microsoft.Cpp.Default.props" was not found.` -If you experience the error `LNK1181` file `kernel32.lib` not found, try compiling using the `Windows SDK 7.1 Command Prompt` start menu shortcut. \ No newline at end of file + +#### `LNK1181` file `kernel32.lib` not found +Easy Solution: try compiling using the `Windows SDK 7.1 Command Prompt` start menu shortcut. + +Proper Solution: To properly fix the situation, you'll need to globally set some environment variables. +To get their proper values, launch the `Windows SDK 7.1 Command Prompt`, then within the prompt get the value of the following variables: `PATH` (it adds some things to PATH, you can put these anywhere in the PATH list), `LIBPATH`, `LIB`, `PlatformToolset`, `WindowsSDKDir`, `sdkdir`, `TARGET_PLATFORM`, and `VS120COMNTOOLS`. +Take their values, and put them in either your User or System variables. You can easily get to this menu by doing `Win+Pause/Break`, `Advanced System Settings`, and then `Environment Variables`. \ No newline at end of file From 93392d559ce6f250b9c7fe8177e6c88603809dc1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0=B9=20=D0=A6=D0=B2?= =?UTF-8?q?=D0=B5=D1=82=D1=86=D0=B8=D1=85?= Date: Sun, 24 Aug 2014 22:13:04 -0700 Subject: [PATCH 267/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 3ddd2a0956..3bfca5a08e 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -33,4 +33,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [topcube](https://github.com/creationix/topcube/blob/master/binding.gyp) * [node-osmium](https://github.com/osmcode/node-osmium/blob/master/binding.gyp) * [node-osrm](https://github.com/DennisOSRM/node-osrm) - * [node-oracle](https://github.com/joeferner/node-oracle/blob/master/binding.gyp) \ No newline at end of file + * [node-oracle](https://github.com/joeferner/node-oracle/blob/master/binding.gyp) + * [node-process-list](https://github.com/ReklatsMasters/node-process-list/blob/master/binding.gyp) \ No newline at end of file From 5b4f2d0e1d5d3eadfd03aaf9c1668340f76c4bea Mon Sep 17 00:00:00 2001 From: Richard Winters Date: Fri, 20 Feb 2015 01:48:50 -0500 Subject: [PATCH 268/551] doc(wiki): Added nk-mysql (nodamysql) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 1 + 1 file changed, 1 insertion(+) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 3bfca5a08e..e546ed0f86 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -14,6 +14,7 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [appjs](https://github.com/milani/appjs/blob/master/binding.gyp) * [nwm](https://github.com/mixu/nwm/blob/master/binding.gyp) * [bcrypt](https://github.com/ncb000gt/node.bcrypt.js/blob/master/binding.gyp) + * [nk-mysql](https://github.com/mmod/nodamysql/blob/master/binding.gyp) * [node-memwatch](https://github.com/lloyd/node-memwatch/blob/master/binding.gyp) * [node-ip2location](https://github.com/bolgovr/node-ip2location/blob/master/binding.gyp) * [node-midi](https://github.com/justinlatimer/node-midi/blob/master/binding.gyp) From ceb30885b74f6789374ef52267b84767be93ebe4 Mon Sep 17 00:00:00 2001 From: Richard Winters Date: Fri, 20 Mar 2015 21:58:49 -0400 Subject: [PATCH 269/551] doc(wiki): Added nk-xrm-installer .gyp references, including .py scripts for providing complete reference to examples of fetching source via http, extracting, and moving files (as opposed to copying) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 2 ++ 1 file changed, 2 insertions(+) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index e546ed0f86..8f868b9efd 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -15,6 +15,8 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [nwm](https://github.com/mixu/nwm/blob/master/binding.gyp) * [bcrypt](https://github.com/ncb000gt/node.bcrypt.js/blob/master/binding.gyp) * [nk-mysql](https://github.com/mmod/nodamysql/blob/master/binding.gyp) + * [nk-xrm-installer](https://github.com/mmod/nk-xrm-installer/blob/master/binding.gyp) + [includable.gypi](https://github.com/mmod/nk-xrm-installer/blob/master/includable.gypi) + [unpack.py](https://github.com/mmod/nk-xrm-installer/blob/master/unpack.py) + [disburse.py](https://github.com/mmod/nk-xrm-installer/blob/master/disburse.py) + .py files above provide complete reference for examples of fetching source via http, extracting, and moving files. * [node-memwatch](https://github.com/lloyd/node-memwatch/blob/master/binding.gyp) * [node-ip2location](https://github.com/bolgovr/node-ip2location/blob/master/binding.gyp) * [node-midi](https://github.com/justinlatimer/node-midi/blob/master/binding.gyp) From 7b5dcafafccdceae4b8f2b53ac9081a694b6ade8 Mon Sep 17 00:00:00 2001 From: Mark Jeghers Date: Tue, 7 Apr 2015 17:32:09 -0700 Subject: [PATCH 270/551] doc(wiki): Note: VS2010 seems to be no longer available! VS2013 or nothing! --- Visual-Studio-2010-Setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Visual-Studio-2010-Setup.md b/Visual-Studio-2010-Setup.md index 4f4333e044..a4ef312efa 100644 --- a/Visual-Studio-2010-Setup.md +++ b/Visual-Studio-2010-Setup.md @@ -2,7 +2,7 @@ On Windows XP/Vista/7, [node-gyp requires Python 2.7 and Visual Studio 2010](htt According to the readme file in [Microsoft Visual C++ 2010 Service Pack 1 Compiler Update for the Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=4422), _to ensure that your system has a supported configuration, uninstall the following products and then reinstall them in the order listed_: -1. [Visual C++ 2010 Express](http://www.microsoft.com/visualstudio/eng/downloads#d-2010-express) or Visual Studio 2010 +1. [Visual C++ 2010 Express](http://www.microsoft.com/visualstudio/eng/downloads#d-2010-express) or Visual Studio 2010 (??) 1. [Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=8279) Note: If you get error on installation, maybe [this link ](http://stackoverflow.com/questions/1901279/windows-7-sdk-installation-failure) will help you. 1. [Visual Studio 2010 SP1](http://www.microsoft.com/en-us/download/details.aspx?id=23691) 1. [Visual C++ 2010 SP1 Compiler Update for the Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=4422) From d310a73d64d0065050377baac7047472f7424a1b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marcin=20Cie=C5=9Blak?= Date: Thu, 20 Aug 2015 19:36:49 +0200 Subject: [PATCH 271/551] doc(wiki): node-sass in the wild --- "\"binding.gyp\"-files-out-in-the-wild.md" | 1 + 1 file changed, 1 insertion(+) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 8f868b9efd..a5a94332ff 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -7,6 +7,7 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-canvas](https://github.com/LearnBoost/node-canvas/blob/master/binding.gyp) * [node-ffi](https://github.com/rbranson/node-ffi/blob/master/binding.gyp) + [libffi](https://github.com/rbranson/node-ffi/blob/master/deps/libffi/libffi.gyp) * [node-time](https://github.com/TooTallNate/node-time/blob/master/binding.gyp) + * [node-sass](https://github.com/sass/node-sass/blob/master/binding.gyp) + [libsass](https://github.com/sass/node-sass/blob/master/src/libsass.gyp) * [node-serialport](https://github.com/voodootikigod/node-serialport/blob/master/binding.gyp) * [node-weak](https://github.com/TooTallNate/node-weak/blob/master/binding.gyp) * [pty.js](https://github.com/chjj/pty.js/blob/master/binding.gyp) From 531c724561d947b5d870de8d52dd8c3c51c5ec2d Mon Sep 17 00:00:00 2001 From: Dieter De Paepe Date: Mon, 24 Aug 2015 13:27:14 +0200 Subject: [PATCH 272/551] doc(wiki): Clarification + direct link to VS2010 --- Visual-Studio-2010-Setup.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Visual-Studio-2010-Setup.md b/Visual-Studio-2010-Setup.md index a4ef312efa..7fc7e27732 100644 --- a/Visual-Studio-2010-Setup.md +++ b/Visual-Studio-2010-Setup.md @@ -2,7 +2,7 @@ On Windows XP/Vista/7, [node-gyp requires Python 2.7 and Visual Studio 2010](htt According to the readme file in [Microsoft Visual C++ 2010 Service Pack 1 Compiler Update for the Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=4422), _to ensure that your system has a supported configuration, uninstall the following products and then reinstall them in the order listed_: -1. [Visual C++ 2010 Express](http://www.microsoft.com/visualstudio/eng/downloads#d-2010-express) or Visual Studio 2010 (??) +1. [Visual C++ 2010 Express](http://download.microsoft.com/download/1/E/5/1E5F1C0A-0D5B-426A-A603-1798B951DDAE/VS2010Express1.iso) or Visual Studio 2010 (??) 1. [Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=8279) Note: If you get error on installation, maybe [this link ](http://stackoverflow.com/questions/1901279/windows-7-sdk-installation-failure) will help you. 1. [Visual Studio 2010 SP1](http://www.microsoft.com/en-us/download/details.aspx?id=23691) 1. [Visual C++ 2010 SP1 Compiler Update for the Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=4422) @@ -11,7 +11,7 @@ On x64 environments, the last update in the list fixes errors about missing comp #### `LNK1181` file `kernel32.lib` not found -Easy Solution: try compiling using the `Windows SDK 7.1 Command Prompt` start menu shortcut. +Easy Solution: try compiling (`npm install`) using the `Windows SDK 7.1 Command Prompt` start menu shortcut. Proper Solution: To properly fix the situation, you'll need to globally set some environment variables. To get their proper values, launch the `Windows SDK 7.1 Command Prompt`, then within the prompt get the value of the following variables: `PATH` (it adds some things to PATH, you can put these anywhere in the PATH list), `LIBPATH`, `LIB`, `PlatformToolset`, `WindowsSDKDir`, `sdkdir`, `TARGET_PLATFORM`, and `VS120COMNTOOLS`. From 11858b0655d1eee00c62ad628e719d4378803d14 Mon Sep 17 00:00:00 2001 From: Operations Research Engineering Software+ Date: Mon, 16 Nov 2015 12:53:41 -0800 Subject: [PATCH 273/551] doc(wiki): Updated Updating npm's bundled node gyp (markdown) --- Updating-npm's-bundled-node-gyp.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Updating-npm's-bundled-node-gyp.md b/Updating-npm's-bundled-node-gyp.md index 0f512d37b0..25fd1c3b28 100644 --- a/Updating-npm's-bundled-node-gyp.md +++ b/Updating-npm's-bundled-node-gyp.md @@ -40,5 +40,5 @@ $ cd "C:\Program Files\nodejs\node_modules\npm" Now you can finally run: ``` bash -$ npm install node-gyp@latest +$ npm install -g node-gyp@latest ``` \ No newline at end of file From 3c6692d538f0ce973869aa237118b7d2483feccd Mon Sep 17 00:00:00 2001 From: Flandre Scarlet Date: Mon, 6 Jun 2016 16:56:59 +0800 Subject: [PATCH 274/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 2 ++ 1 file changed, 2 insertions(+) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index a5a94332ff..9c49f1f55c 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -2,6 +2,8 @@ This page contains links to some examples of existing `binding.gyp` files that o To add to this page, just add the link to the project's `binding.gyp` file below: + * [ons](https://github.com/XadillaX/aliyun-ons/blob/master/binding.gyp) + * [thmclrx](https://github.com/XadillaX/thmclrx/blob/master/binding.gyp) * [libxmljs](https://github.com/polotek/libxmljs/blob/master/binding.gyp) * [node-buffertools](https://github.com/bnoordhuis/node-buffertools/blob/master/binding.gyp) * [node-canvas](https://github.com/LearnBoost/node-canvas/blob/master/binding.gyp) From 408b72f561329408daeb17834436e381406efcc8 Mon Sep 17 00:00:00 2001 From: peter--bolier--zero Date: Mon, 6 Jun 2016 16:10:53 +0200 Subject: [PATCH 275/551] doc(wiki): if ouns that the -h did not help. I founs on github that there was support for visual studio 2015, while i couldn't install node-red beacuse it kept telling me the key 2015 was missing. looking in he gyp python code i found the local file was bot up t dat with the github repo. updating took several efforts before i tried to drop the -g option. --- Updating-npm's-bundled-node-gyp.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Updating-npm's-bundled-node-gyp.md b/Updating-npm's-bundled-node-gyp.md index 25fd1c3b28..d593753aae 100644 --- a/Updating-npm's-bundled-node-gyp.md +++ b/Updating-npm's-bundled-node-gyp.md @@ -41,4 +41,6 @@ Now you can finally run: ``` bash $ npm install -g node-gyp@latest -``` \ No newline at end of file +``` + +note: i found that the -g on windows is not correct. It gets installed in C:\Users\\AppData\Roaming\npm\node_modules\gyp which is not the directory where node is installed C:\Program Files (x86)\nodejs\node_modules\npm\node_modules\node-gyp\... \ No newline at end of file From d69dffc16c2b1e3c60dcb5d1c35a49270ba22a35 Mon Sep 17 00:00:00 2001 From: peter--bolier--zero Date: Mon, 6 Jun 2016 16:13:11 +0200 Subject: [PATCH 276/551] doc(wiki): sorry, forgot to mention a specific windows version. --- Updating-npm's-bundled-node-gyp.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Updating-npm's-bundled-node-gyp.md b/Updating-npm's-bundled-node-gyp.md index d593753aae..8dc7697276 100644 --- a/Updating-npm's-bundled-node-gyp.md +++ b/Updating-npm's-bundled-node-gyp.md @@ -43,4 +43,4 @@ Now you can finally run: $ npm install -g node-gyp@latest ``` -note: i found that the -g on windows is not correct. It gets installed in C:\Users\\AppData\Roaming\npm\node_modules\gyp which is not the directory where node is installed C:\Program Files (x86)\nodejs\node_modules\npm\node_modules\node-gyp\... \ No newline at end of file +note: i found that the -g on windows 7 is not correct. It gets installed in C:\Users\\AppData\Roaming\npm\node_modules\gyp which is not the directory where node is installed C:\Program Files (x86)\nodejs\node_modules\npm\node_modules\node-gyp\... \ No newline at end of file From d319b0e98c7085de8e51bc5595eba4264b99a7d5 Mon Sep 17 00:00:00 2001 From: Nick Desaulniers Date: Mon, 27 Jun 2016 18:17:34 -0700 Subject: [PATCH 277/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 9c49f1f55c..dd809070ae 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -40,4 +40,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-osmium](https://github.com/osmcode/node-osmium/blob/master/binding.gyp) * [node-osrm](https://github.com/DennisOSRM/node-osrm) * [node-oracle](https://github.com/joeferner/node-oracle/blob/master/binding.gyp) - * [node-process-list](https://github.com/ReklatsMasters/node-process-list/blob/master/binding.gyp) \ No newline at end of file + * [node-process-list](https://github.com/ReklatsMasters/node-process-list/blob/master/binding.gyp) + * [node-nanomsg](https://github.com/nickdesaulniers/node-nanomsg/blob/master/binding.gyp) \ No newline at end of file From bf4bed1b96a7d22fba6f97f4552ad09f32ac3737 Mon Sep 17 00:00:00 2001 From: Nicola Del Gobbo Date: Sat, 4 Mar 2017 23:16:56 +0100 Subject: [PATCH 278/551] doc(wiki): Added Ghostscript4JS --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index dd809070ae..ef7be12aba 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -41,4 +41,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-osrm](https://github.com/DennisOSRM/node-osrm) * [node-oracle](https://github.com/joeferner/node-oracle/blob/master/binding.gyp) * [node-process-list](https://github.com/ReklatsMasters/node-process-list/blob/master/binding.gyp) - * [node-nanomsg](https://github.com/nickdesaulniers/node-nanomsg/blob/master/binding.gyp) \ No newline at end of file + * [node-nanomsg](https://github.com/nickdesaulniers/node-nanomsg/blob/master/binding.gyp) + * [Ghostscript4JS](https://github.com/NickNaso/ghostscript4js/blob/master/binding.gyp) \ No newline at end of file From d617faee29c40871ca5c8f93efd0ce929a40d541 Mon Sep 17 00:00:00 2001 From: Abdul Hameed Date: Fri, 17 Mar 2017 08:22:35 +0500 Subject: [PATCH 279/551] doc(wiki): I highly missing it in common issue as every windows biggner face that issue --- Common-issues.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/Common-issues.md b/Common-issues.md index 5648114bf6..840c68d52e 100644 --- a/Common-issues.md +++ b/Common-issues.md @@ -4,4 +4,11 @@ Make sure you are using the native Python version in OSX. If you use a MacPorts If you have issues with `execvp`, be sure to check your `$PYTHON` environment variable. If it is not set to the native version, unset it and try again. -Notes: https://gist.github.com/erichocean/5177582 \ No newline at end of file +Notes: https://gist.github.com/erichocean/5177582 + +## npm ERR! `node-gyp rebuild`(Windows) +* just install the build tools from [here](http://landinghub.visualstudio.com/visual-cpp-build-tools) +PLease note the version as is required in below command e.g **2015** or **2017** +* Launch cmd, run `npm config set msvs_version 2015` +* restart and all is well 👍 + From e2dc77730b09d7ee8682d7713a7603a2d7aacabd Mon Sep 17 00:00:00 2001 From: xdf Date: Sun, 30 Apr 2017 21:55:54 +0800 Subject: [PATCH 280/551] doc(wiki): ADDED: Node.js binding to OpenCV --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index ef7be12aba..88c3ddfee0 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -42,4 +42,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-oracle](https://github.com/joeferner/node-oracle/blob/master/binding.gyp) * [node-process-list](https://github.com/ReklatsMasters/node-process-list/blob/master/binding.gyp) * [node-nanomsg](https://github.com/nickdesaulniers/node-nanomsg/blob/master/binding.gyp) - * [Ghostscript4JS](https://github.com/NickNaso/ghostscript4js/blob/master/binding.gyp) \ No newline at end of file + * [Ghostscript4JS](https://github.com/NickNaso/ghostscript4js/blob/master/binding.gyp) + * [nodecv](https://github.com/xudafeng/nodecv/blob/master/binding.gyp) \ No newline at end of file From d766b7427851e6c2edc02e2504a7be9be7e330c0 Mon Sep 17 00:00:00 2001 From: Nicola Del Gobbo Date: Mon, 19 Jun 2017 00:32:39 +0200 Subject: [PATCH 281/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 88c3ddfee0..7847e2623b 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -43,4 +43,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-process-list](https://github.com/ReklatsMasters/node-process-list/blob/master/binding.gyp) * [node-nanomsg](https://github.com/nickdesaulniers/node-nanomsg/blob/master/binding.gyp) * [Ghostscript4JS](https://github.com/NickNaso/ghostscript4js/blob/master/binding.gyp) - * [nodecv](https://github.com/xudafeng/nodecv/blob/master/binding.gyp) \ No newline at end of file + * [nodecv](https://github.com/xudafeng/nodecv/blob/master/binding.gyp) + * [magick-cli](https://github.com/NickNaso/magick-cli/blob/master/binding.gyp) \ No newline at end of file From 9dce0e41650c3fa973e6135a79632d022c662a1d Mon Sep 17 00:00:00 2001 From: Matt Hirsch Date: Wed, 30 Aug 2017 14:44:58 -0400 Subject: [PATCH 282/551] doc(wiki): Adding the sharp library to the list --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 7847e2623b..668d363e52 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -44,4 +44,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-nanomsg](https://github.com/nickdesaulniers/node-nanomsg/blob/master/binding.gyp) * [Ghostscript4JS](https://github.com/NickNaso/ghostscript4js/blob/master/binding.gyp) * [nodecv](https://github.com/xudafeng/nodecv/blob/master/binding.gyp) - * [magick-cli](https://github.com/NickNaso/magick-cli/blob/master/binding.gyp) \ No newline at end of file + * [magick-cli](https://github.com/NickNaso/magick-cli/blob/master/binding.gyp) + * [sharp](https://github.com/lovell/sharp/blob/master/binding.gyp) \ No newline at end of file From bbca21a1e1ede4c473aff365ca71989a5bda7b57 Mon Sep 17 00:00:00 2001 From: Matt Hirsch Date: Wed, 30 Aug 2017 14:47:53 -0400 Subject: [PATCH 283/551] doc(wiki): node-srs was a 404 --- "\"binding.gyp\"-files-out-in-the-wild.md" | 1 - 1 file changed, 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 668d363e52..70ed21b7ea 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -24,7 +24,6 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [node-ip2location](https://github.com/bolgovr/node-ip2location/blob/master/binding.gyp) * [node-midi](https://github.com/justinlatimer/node-midi/blob/master/binding.gyp) * [node-sqlite3](https://github.com/developmentseed/node-sqlite3/blob/master/binding.gyp) + [libsqlite3](https://github.com/developmentseed/node-sqlite3/blob/master/deps/sqlite3.gyp) - * [node-srs](https://github.com/springmeyer/node-srs/blob/master/binding.gyp) * [node-zipfile](https://github.com/mapbox/node-zipfile/blob/master/binding.gyp) * [node-mapnik](https://github.com/mapnik/node-mapnik/blob/master/binding.gyp) * [node-inotify](https://github.com/c4milo/node-inotify/blob/master/binding.gyp) From 5b899b70db729c392ced7c98e8e17590c6499fc3 Mon Sep 17 00:00:00 2001 From: Abdul Hameed Date: Sat, 11 Aug 2018 12:26:36 +0500 Subject: [PATCH 284/551] doc(wiki): C++ build tools version upgraded --- Common-issues.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Common-issues.md b/Common-issues.md index 840c68d52e..e9f1ab483d 100644 --- a/Common-issues.md +++ b/Common-issues.md @@ -7,8 +7,8 @@ If you have issues with `execvp`, be sure to check your `$PYTHON` environment va Notes: https://gist.github.com/erichocean/5177582 ## npm ERR! `node-gyp rebuild`(Windows) -* just install the build tools from [here](http://landinghub.visualstudio.com/visual-cpp-build-tools) -PLease note the version as is required in below command e.g **2015** or **2017** -* Launch cmd, run `npm config set msvs_version 2015` -* restart and all is well 👍 +* just install the build tools from [here](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) +PLease note the version as is required in below command e.g **2017** +* Launch cmd, run `npm config set msvs_version 2017` +* close and open new CMD/terminal and all is well :100: From 93423b43606de9664aeb79635825f5e9941ec9bc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Thu, 11 Oct 2018 03:54:00 +0100 Subject: [PATCH 285/551] doc(wiki): Destroyed Visual Studio 2010 Setup (markdown) --- Visual-Studio-2010-Setup.md | 18 ------------------ 1 file changed, 18 deletions(-) delete mode 100644 Visual-Studio-2010-Setup.md diff --git a/Visual-Studio-2010-Setup.md b/Visual-Studio-2010-Setup.md deleted file mode 100644 index 7fc7e27732..0000000000 --- a/Visual-Studio-2010-Setup.md +++ /dev/null @@ -1,18 +0,0 @@ -On Windows XP/Vista/7, [node-gyp requires Python 2.7 and Visual Studio 2010](https://github.com/TooTallNate/node-gyp#installation) - -According to the readme file in [Microsoft Visual C++ 2010 Service Pack 1 Compiler Update for the Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=4422), _to ensure that your system has a supported configuration, uninstall the following products and then reinstall them in the order listed_: - -1. [Visual C++ 2010 Express](http://download.microsoft.com/download/1/E/5/1E5F1C0A-0D5B-426A-A603-1798B951DDAE/VS2010Express1.iso) or Visual Studio 2010 (??) -1. [Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=8279) Note: If you get error on installation, maybe [this link ](http://stackoverflow.com/questions/1901279/windows-7-sdk-installation-failure) will help you. -1. [Visual Studio 2010 SP1](http://www.microsoft.com/en-us/download/details.aspx?id=23691) -1. [Visual C++ 2010 SP1 Compiler Update for the Windows SDK 7.1](http://www.microsoft.com/en-us/download/details.aspx?id=4422) - -On x64 environments, the last update in the list fixes errors about missing compilers and `error MSB4019: The imported project "C:\Microsoft.Cpp.Default.props" was not found.` - - -#### `LNK1181` file `kernel32.lib` not found -Easy Solution: try compiling (`npm install`) using the `Windows SDK 7.1 Command Prompt` start menu shortcut. - -Proper Solution: To properly fix the situation, you'll need to globally set some environment variables. -To get their proper values, launch the `Windows SDK 7.1 Command Prompt`, then within the prompt get the value of the following variables: `PATH` (it adds some things to PATH, you can put these anywhere in the PATH list), `LIBPATH`, `LIB`, `PlatformToolset`, `WindowsSDKDir`, `sdkdir`, `TARGET_PLATFORM`, and `VS120COMNTOOLS`. -Take their values, and put them in either your User or System variables. You can easily get to this menu by doing `Win+Pause/Break`, `Advanced System Settings`, and then `Environment Variables`. \ No newline at end of file From 3407109325cf7ba1e925656b9eb75feffab0557c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Reis?= Date: Thu, 11 Oct 2018 03:54:21 +0100 Subject: [PATCH 286/551] doc(wiki): Updated Home (markdown) --- Home.md | 1 - 1 file changed, 1 deletion(-) diff --git a/Home.md b/Home.md index 59531ad731..fe099868b2 100644 --- a/Home.md +++ b/Home.md @@ -5,4 +5,3 @@ Welcome to the node-gyp wiki! * [[Common Issues]] * [[Updating npm's bundled node-gyp]] * [[Error: "pre" versions of node cannot be installed]] - * [[Visual Studio 2010 Setup]] \ No newline at end of file From 3aa2c6bdb07971b87505e32e32548d75264bd19f Mon Sep 17 00:00:00 2001 From: Bert Verhelst Date: Fri, 25 Jan 2019 10:14:15 +0100 Subject: [PATCH 287/551] doc(wiki): Lower case L --- Common-issues.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Common-issues.md b/Common-issues.md index e9f1ab483d..ae05fe326a 100644 --- a/Common-issues.md +++ b/Common-issues.md @@ -8,7 +8,7 @@ Notes: https://gist.github.com/erichocean/5177582 ## npm ERR! `node-gyp rebuild`(Windows) * just install the build tools from [here](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) -PLease note the version as is required in below command e.g **2017** +Please note the version as is required in below command e.g **2017** * Launch cmd, run `npm config set msvs_version 2017` * close and open new CMD/terminal and all is well :100: From 7444b47a7caac1e14d1da474a7fcfcf88d328017 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Tue, 3 Dec 2019 15:23:41 +1100 Subject: [PATCH 288/551] doc(wiki): Updated "binding.gyp" files out in the wild (markdown) --- "\"binding.gyp\"-files-out-in-the-wild.md" | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/\"binding.gyp\"-files-out-in-the-wild.md" index 70ed21b7ea..c4603dd3d1 100644 --- "a/\"binding.gyp\"-files-out-in-the-wild.md" +++ "b/\"binding.gyp\"-files-out-in-the-wild.md" @@ -44,4 +44,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [Ghostscript4JS](https://github.com/NickNaso/ghostscript4js/blob/master/binding.gyp) * [nodecv](https://github.com/xudafeng/nodecv/blob/master/binding.gyp) * [magick-cli](https://github.com/NickNaso/magick-cli/blob/master/binding.gyp) - * [sharp](https://github.com/lovell/sharp/blob/master/binding.gyp) \ No newline at end of file + * [sharp](https://github.com/lovell/sharp/blob/master/binding.gyp) + * [krb5](https://github.com/adaltas/node-krb5/blob/master/binding.gyp) \ No newline at end of file From 1dcad873539027511a5f0243baf770ea90f6f4e2 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Mon, 31 May 2021 07:12:38 +0200 Subject: [PATCH 289/551] doc(wiki): Make changes discussed in https://github.com/nodejs/node-gyp/issues/2416 --- Updating-npm's-bundled-node-gyp.md | 41 +++++++++++++++++------------- 1 file changed, 24 insertions(+), 17 deletions(-) diff --git a/Updating-npm's-bundled-node-gyp.md b/Updating-npm's-bundled-node-gyp.md index 8dc7697276..91e5170a62 100644 --- a/Updating-npm's-bundled-node-gyp.md +++ b/Updating-npm's-bundled-node-gyp.md @@ -1,46 +1,53 @@ -`npm` bundles its own, internal, copy of `node-gyp`. This internal copy is independent of any globally installed copy of node-gyp that you may have installed via `npm install -g node-gyp`. +`npm` bundles its own, internal, copy of `node-gyp`. This internal copy is independent of any globally installed copy of node-gyp that +you may have installed via `npm install -g node-gyp`. -This means that while `node-gyp` doesn't get installed into your `$PATH` by default, npm still keeps its own copy to invoke when you attempt to `npm install` a native addon. +This means that while `node-gyp` doesn't get installed into your `$PATH` by default, npm still keeps its own copy to invoke when you +attempt to `npm install` a native add-on. -Sometimes, you may need to update npm's internal node-gyp to a newer version than what is installed. A simple `npm install -g node-gyp` _won't_ do the trick since npm will still continue to use its internal copy over the global one. +Sometimes, you may need to update npm's internal node-gyp to a newer version than what is installed. A simple `npm install -g node-gyp` +_won't_ do the trick since npm will still continue to use its internal copy over the global one. So instead: ## Linux, Mac OS X, Solaris, etc. Unix is easy. Just run the following command. Use `sudo` if necessary. - -``` bash -$ [sudo] npm explore npm -g -- npm install node-gyp@latest +```bash +$ [sudo] npm explore npm/node_modules/npm-lifecycle -g -- npm install node-gyp@latest ``` ## Windows -Windows is a bit tricker, since `npm` gets installed to the "Program Files" directory, which needs admin privileges in order to modify on current Windows. Therefore, run the following commands __inside a `cmd.exe` started with "Run as Administrator"__: +Windows is a bit trickier, since `npm` might be installed to the "Program Files" directory, which needs admin privileges in order to +modify on current Windows. Therefore, run the following commands __inside a `cmd.exe` started with "Run as Administrator"__: First we need to find the location of `node`. If you don't already know the location that `node.exe` got installed to, then run: - -``` bash +```bash $ npm install -g which $ which node ``` As an alternative to the above, those on Windows Server 2003 and later (this includes Windows 7) can run: - -``` bash +```bash $ where node ``` Now `cd` to the directory that `node.exe` is contained in, and with `node_modules\npm` at the end. i.e.: - -``` bash +```bash $ cd "C:\Program Files\nodejs\node_modules\npm" ``` -Now you can finally run: +Now you can run: +```bash +$ npm install node-gyp@latest +``` -``` bash -$ npm install -g node-gyp@latest +Now `cd` to the `npm-lifecycle` directory: +```bash +$ cd node_modules\npm-lifecycle ``` -note: i found that the -g on windows 7 is not correct. It gets installed in C:\Users\\AppData\Roaming\npm\node_modules\gyp which is not the directory where node is installed C:\Program Files (x86)\nodejs\node_modules\npm\node_modules\node-gyp\... \ No newline at end of file +Now you can finally run (again): +```bash +$ npm install node-gyp@latest +``` \ No newline at end of file From 9285ff6e451c52c070a05f05f0a9602621d91d53 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Tue, 1 Jun 2021 07:34:37 +0200 Subject: [PATCH 290/551] doc(wiki): Drop in favor of --- Updating-npm's-bundled-node-gyp.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/Updating-npm's-bundled-node-gyp.md b/Updating-npm's-bundled-node-gyp.md index 91e5170a62..e153ad7a90 100644 --- a/Updating-npm's-bundled-node-gyp.md +++ b/Updating-npm's-bundled-node-gyp.md @@ -23,12 +23,6 @@ modify on current Windows. Therefore, run the following commands __inside a `cmd First we need to find the location of `node`. If you don't already know the location that `node.exe` got installed to, then run: ```bash -$ npm install -g which -$ which node -``` - -As an alternative to the above, those on Windows Server 2003 and later (this includes Windows 7) can run: -```bash $ where node ``` From 0fce46b53340c85e8091cde347d5ed23a443c82f Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Wed, 2 Jun 2021 08:01:29 +0200 Subject: [PATCH 291/551] doc(wiki): Different commands for Windows npm v6 vs. v7 --- Updating-npm's-bundled-node-gyp.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/Updating-npm's-bundled-node-gyp.md b/Updating-npm's-bundled-node-gyp.md index e153ad7a90..1ae618aa9d 100644 --- a/Updating-npm's-bundled-node-gyp.md +++ b/Updating-npm's-bundled-node-gyp.md @@ -26,22 +26,27 @@ First we need to find the location of `node`. If you don't already know the loca $ where node ``` -Now `cd` to the directory that `node.exe` is contained in, and with `node_modules\npm` at the end. i.e.: +Now `cd` to the directory that `node.exe` is contained in i.e.: ```bash -$ cd "C:\Program Files\nodejs\node_modules\npm" +$ cd "C:\Program Files\nodejs" ``` -Now you can run: +Now we need to know your version of `npm`: ```bash -$ npm install node-gyp@latest +npm --version +``` + +If your npm version is ___7 or greater___, do: +```bash +cd node_modules\npm\node_modules\@npmcli\run-script ``` -Now `cd` to the `npm-lifecycle` directory: +If your npm version is ___less than 7___, do: ```bash -$ cd node_modules\npm-lifecycle +cd node_modules\npm\node_modules\npm-lifecycle ``` -Now you can finally run (again): +Finish by running: ```bash $ npm install node-gyp@latest ``` \ No newline at end of file From c3e548736645b535ea5bce613d74ca3e98598243 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Wed, 2 Jun 2021 21:56:06 +0200 Subject: [PATCH 292/551] doc(wiki): Improve Unix instructions --- Updating-npm's-bundled-node-gyp.md | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/Updating-npm's-bundled-node-gyp.md b/Updating-npm's-bundled-node-gyp.md index 1ae618aa9d..7591ffbfd3 100644 --- a/Updating-npm's-bundled-node-gyp.md +++ b/Updating-npm's-bundled-node-gyp.md @@ -9,9 +9,23 @@ _won't_ do the trick since npm will still continue to use its internal copy over So instead: +## Version of npm + +We need to start by knowing your version of `npm`: +```bash +npm --version +``` + ## Linux, Mac OS X, Solaris, etc. Unix is easy. Just run the following command. Use `sudo` if necessary. + +If your npm is version ___7___, do: +```bash +$ [sudo] npm explore npm/node_modules/@npmcli/run-script -g -- npm_config_global=false npm install node-gyp@latest +``` + +Else if your npm is version ___less than 7___, do: ```bash $ [sudo] npm explore npm/node_modules/npm-lifecycle -g -- npm install node-gyp@latest ``` @@ -26,17 +40,12 @@ First we need to find the location of `node`. If you don't already know the loca $ where node ``` -Now `cd` to the directory that `node.exe` is contained in i.e.: +Now `cd` to the directory that `node.exe` is contained in e.g.: ```bash $ cd "C:\Program Files\nodejs" ``` -Now we need to know your version of `npm`: -```bash -npm --version -``` - -If your npm version is ___7 or greater___, do: +If your npm version is ___7___, do: ```bash cd node_modules\npm\node_modules\@npmcli\run-script ``` From d31485415ef69d46effa6090c95698341965de1b Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Wed, 2 Jun 2021 21:58:36 +0200 Subject: [PATCH 293/551] doc(wiki): Updated Updating npm's bundled node gyp (markdown) --- Updating-npm's-bundled-node-gyp.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Updating-npm's-bundled-node-gyp.md b/Updating-npm's-bundled-node-gyp.md index 7591ffbfd3..36d36733f0 100644 --- a/Updating-npm's-bundled-node-gyp.md +++ b/Updating-npm's-bundled-node-gyp.md @@ -50,7 +50,7 @@ If your npm version is ___7___, do: cd node_modules\npm\node_modules\@npmcli\run-script ``` -If your npm version is ___less than 7___, do: +Else if your npm version is ___less than 7___, do: ```bash cd node_modules\npm\node_modules\npm-lifecycle ``` From ee8e1c1e5334096d58e0d6bca6c006f2ee9c88cb Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Fri, 11 Jun 2021 06:32:31 +0200 Subject: [PATCH 294/551] doc(wiki): If permissions error, please try and then the command. --- Updating-npm's-bundled-node-gyp.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/Updating-npm's-bundled-node-gyp.md b/Updating-npm's-bundled-node-gyp.md index 36d36733f0..fcfa59f9eb 100644 --- a/Updating-npm's-bundled-node-gyp.md +++ b/Updating-npm's-bundled-node-gyp.md @@ -16,20 +16,22 @@ We need to start by knowing your version of `npm`: npm --version ``` -## Linux, Mac OS X, Solaris, etc. +## Linux, macOS, Solaris, etc. -Unix is easy. Just run the following command. Use `sudo` if necessary. +Unix is easy. Just run the following command. If your npm is version ___7___, do: ```bash -$ [sudo] npm explore npm/node_modules/@npmcli/run-script -g -- npm_config_global=false npm install node-gyp@latest +$ npm explore npm/node_modules/@npmcli/run-script -g -- npm_config_global=false npm install node-gyp@latest ``` Else if your npm is version ___less than 7___, do: ```bash -$ [sudo] npm explore npm/node_modules/npm-lifecycle -g -- npm install node-gyp@latest +$ npm explore npm/node_modules/npm-lifecycle -g -- npm install node-gyp@latest ``` +If the command fails with a permissions error, please try `sudo` and then the command. + ## Windows Windows is a bit trickier, since `npm` might be installed to the "Program Files" directory, which needs admin privileges in order to From f0a48355d86534ec3bdabcdb3ce3340fa2e17f39 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Mon, 21 Jun 2021 14:36:18 +1000 Subject: [PATCH 295/551] doc(wiki): move wiki docs into doc/ --- .../\"binding.gyp\"-files-out-in-the-wild.md" | 0 Common-issues.md => docs/Common-issues.md | 0 .../Error:-\"pre\"-versions-of-node-cannot-be-installed.md" | 0 Home.md => docs/Home.md | 0 Linking-to-OpenSSL.md => docs/Linking-to-OpenSSL.md | 0 .../Updating-npm's-bundled-node-gyp.md | 0 .../use-of-undeclared-identifier-'TypedArray'.md | 0 7 files changed, 0 insertions(+), 0 deletions(-) rename "\"binding.gyp\"-files-out-in-the-wild.md" => "docs/\"binding.gyp\"-files-out-in-the-wild.md" (100%) rename Common-issues.md => docs/Common-issues.md (100%) rename "Error:-\"pre\"-versions-of-node-cannot-be-installed.md" => "docs/Error:-\"pre\"-versions-of-node-cannot-be-installed.md" (100%) rename Home.md => docs/Home.md (100%) rename Linking-to-OpenSSL.md => docs/Linking-to-OpenSSL.md (100%) rename Updating-npm's-bundled-node-gyp.md => docs/Updating-npm's-bundled-node-gyp.md (100%) rename use-of-undeclared-identifier-'TypedArray'.md => docs/use-of-undeclared-identifier-'TypedArray'.md (100%) diff --git "a/\"binding.gyp\"-files-out-in-the-wild.md" "b/docs/\"binding.gyp\"-files-out-in-the-wild.md" similarity index 100% rename from "\"binding.gyp\"-files-out-in-the-wild.md" rename to "docs/\"binding.gyp\"-files-out-in-the-wild.md" diff --git a/Common-issues.md b/docs/Common-issues.md similarity index 100% rename from Common-issues.md rename to docs/Common-issues.md diff --git "a/Error:-\"pre\"-versions-of-node-cannot-be-installed.md" "b/docs/Error:-\"pre\"-versions-of-node-cannot-be-installed.md" similarity index 100% rename from "Error:-\"pre\"-versions-of-node-cannot-be-installed.md" rename to "docs/Error:-\"pre\"-versions-of-node-cannot-be-installed.md" diff --git a/Home.md b/docs/Home.md similarity index 100% rename from Home.md rename to docs/Home.md diff --git a/Linking-to-OpenSSL.md b/docs/Linking-to-OpenSSL.md similarity index 100% rename from Linking-to-OpenSSL.md rename to docs/Linking-to-OpenSSL.md diff --git a/Updating-npm's-bundled-node-gyp.md b/docs/Updating-npm's-bundled-node-gyp.md similarity index 100% rename from Updating-npm's-bundled-node-gyp.md rename to docs/Updating-npm's-bundled-node-gyp.md diff --git a/use-of-undeclared-identifier-'TypedArray'.md b/docs/use-of-undeclared-identifier-'TypedArray'.md similarity index 100% rename from use-of-undeclared-identifier-'TypedArray'.md rename to docs/use-of-undeclared-identifier-'TypedArray'.md From b52e487eac1eb421573d1e67114a242eeff45a00 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Mon, 21 Jun 2021 21:22:33 +1000 Subject: [PATCH 296/551] doc(wiki): link to docs/ from README --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 9ec3480138..e5a2a026c5 100644 --- a/README.md +++ b/README.md @@ -144,13 +144,15 @@ A barebones `gyp` file appropriate for building a Node.js addon could look like: ## Further reading +The **[docs](./docs/)** directory contains additional documentation on specific node-gyp topics that may be useful if you are experiencing problems installing or building addons using node-gyp. + Some additional resources for Node.js native addons and writing `gyp` configuration files: * ["Going Native" a nodeschool.io tutorial](http://nodeschool.io/#goingnative) * ["Hello World" node addon example](https://github.com/nodejs/node/tree/master/test/addons/hello-world) * [gyp user documentation](https://gyp.gsrc.io/docs/UserDocumentation.md) * [gyp input format reference](https://gyp.gsrc.io/docs/InputFormatReference.md) - * [*"binding.gyp" files out in the wild* wiki page](https://github.com/nodejs/node-gyp/wiki/%22binding.gyp%22-files-out-in-the-wild) + * [*"binding.gyp" files out in the wild* wiki page](./docs/"binding.gyp"-files-out-in-the-wild.md) ## Commands From 161c2353ef5b562f4acfb2fd77608fcbd0800fc0 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Tue, 22 Jun 2021 12:10:09 +1000 Subject: [PATCH 297/551] doc(wiki): safer doc names, remove unnecessary TypedArray doc --- .../Error-pre-versions-of-node-cannot-be-installed.md | 0 ...m's-bundled-node-gyp.md => Updating-npm-bundled-node-gyp.md} | 0 .../binding.gyp-files-in-the-wild.md | 0 docs/use-of-undeclared-identifier-'TypedArray'.md | 2 -- 4 files changed, 2 deletions(-) rename "docs/Error:-\"pre\"-versions-of-node-cannot-be-installed.md" => docs/Error-pre-versions-of-node-cannot-be-installed.md (100%) rename docs/{Updating-npm's-bundled-node-gyp.md => Updating-npm-bundled-node-gyp.md} (100%) rename "docs/\"binding.gyp\"-files-out-in-the-wild.md" => docs/binding.gyp-files-in-the-wild.md (100%) delete mode 100644 docs/use-of-undeclared-identifier-'TypedArray'.md diff --git "a/docs/Error:-\"pre\"-versions-of-node-cannot-be-installed.md" b/docs/Error-pre-versions-of-node-cannot-be-installed.md similarity index 100% rename from "docs/Error:-\"pre\"-versions-of-node-cannot-be-installed.md" rename to docs/Error-pre-versions-of-node-cannot-be-installed.md diff --git a/docs/Updating-npm's-bundled-node-gyp.md b/docs/Updating-npm-bundled-node-gyp.md similarity index 100% rename from docs/Updating-npm's-bundled-node-gyp.md rename to docs/Updating-npm-bundled-node-gyp.md diff --git "a/docs/\"binding.gyp\"-files-out-in-the-wild.md" b/docs/binding.gyp-files-in-the-wild.md similarity index 100% rename from "docs/\"binding.gyp\"-files-out-in-the-wild.md" rename to docs/binding.gyp-files-in-the-wild.md diff --git a/docs/use-of-undeclared-identifier-'TypedArray'.md b/docs/use-of-undeclared-identifier-'TypedArray'.md deleted file mode 100644 index 9034b893d7..0000000000 --- a/docs/use-of-undeclared-identifier-'TypedArray'.md +++ /dev/null @@ -1,2 +0,0 @@ -Local typedArray = Local::Cast(arg[0]); -why node-gyp not support typedArray \ No newline at end of file From b7bccdb527d93b0bb0ce99713f083ce2985fe85c Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Mon, 28 Jun 2021 08:10:59 +0200 Subject: [PATCH 298/551] ci: GitHub Actions Test on node: [12.x, 14.x, 16.x] (#2439) --- .github/workflows/tests.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 3b710b9449..7b12268023 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -8,7 +8,7 @@ jobs: fail-fast: false max-parallel: 15 matrix: - node: [10.x, 12.x, 14.x] + node: [12.x, 14.x, 16.x] python: [3.6, 3.8, 3.9] os: [macos-latest, ubuntu-latest, windows-latest] runs-on: ${{ matrix.os }} From b6e1cc71279092552f9e224be245bf91e6d0c981 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Tue, 6 Jul 2021 08:20:44 +0200 Subject: [PATCH 299/551] Add title to node-gyp version document (#2452) * Add title to node-gyp version document * Update Updating-npm-bundled-node-gyp.md --- docs/Updating-npm-bundled-node-gyp.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/Updating-npm-bundled-node-gyp.md b/docs/Updating-npm-bundled-node-gyp.md index fcfa59f9eb..38795f5fac 100644 --- a/docs/Updating-npm-bundled-node-gyp.md +++ b/docs/Updating-npm-bundled-node-gyp.md @@ -1,5 +1,10 @@ +# Updating the npm-bundled version of node-gyp + +[Many issues](https://github.com/nodejs/node-gyp/labels/ERR%21%20node-gyp%20-v%20%3C%3D%20v5.1.0) are opened by users who are +not running a [current version of node-gyp](https://github.com/nodejs/node-gyp/releases). + `npm` bundles its own, internal, copy of `node-gyp`. This internal copy is independent of any globally installed copy of node-gyp that -you may have installed via `npm install -g node-gyp`. +may have been installed via `npm install -g node-gyp`. This means that while `node-gyp` doesn't get installed into your `$PATH` by default, npm still keeps its own copy to invoke when you attempt to `npm install` a native add-on. @@ -60,4 +65,4 @@ cd node_modules\npm\node_modules\npm-lifecycle Finish by running: ```bash $ npm install node-gyp@latest -``` \ No newline at end of file +``` From c8c0af72e78141a02b5da4cd4d704838333a90bd Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Mon, 28 Jun 2021 13:59:08 +0300 Subject: [PATCH 300/551] fix: doc how to update node-gyp independently from npm --- docs/Updating-npm-bundled-node-gyp.md | 64 +++++++++------------------ 1 file changed, 21 insertions(+), 43 deletions(-) diff --git a/docs/Updating-npm-bundled-node-gyp.md b/docs/Updating-npm-bundled-node-gyp.md index 38795f5fac..bd237b6905 100644 --- a/docs/Updating-npm-bundled-node-gyp.md +++ b/docs/Updating-npm-bundled-node-gyp.md @@ -6,63 +6,41 @@ not running a [current version of node-gyp](https://github.com/nodejs/node-gyp/r `npm` bundles its own, internal, copy of `node-gyp`. This internal copy is independent of any globally installed copy of node-gyp that may have been installed via `npm install -g node-gyp`. -This means that while `node-gyp` doesn't get installed into your `$PATH` by default, npm still keeps its own copy to invoke when you -attempt to `npm install` a native add-on. +Generally, npm's library files are installed inside your global "node_modules",where npm is installed (run `npm prefix` and add `lib/node_modules`, or just `node_modules` for Windows). There are some exceptions to this. Inside this global `node_modules/`there will be an `npm/` directory and inside this you'll find a `node_modules/node-gyp/` directory. So it may look something like `/usr/local/lib/node_modules/npm/node_modules/node-gyp/`. This is the version of node-gyp that ships with npm. -Sometimes, you may need to update npm's internal node-gyp to a newer version than what is installed. A simple `npm install -g node-gyp` -_won't_ do the trick since npm will still continue to use its internal copy over the global one. +When you install a _new_ version of node-gyp with outside of npm, it'll go into your global node_modules, but not under the `npm/node_modules`. So that may look like `/usr/local/lib/node_modules/node-gyp/`. It'll have the `node-gyp` executable linked into your `PATH` so running `node-gyp` will use this version. -So instead: +The catch is that npm won't use this version unless you tell it to, it'll keep on using the one you have installed. You need to instruct it to by setting the `node_gyp` config variable (which goes into your `~/.npmrc`). You do this by running the `npm config set` command as below. Then npm will use the command in the path you supply whenever it needs to build a native addon. -## Version of npm +**Important**: You also need to remember to unset this when you upgrade npm with a newer version of node-gyp, or you have to manually keep your globally installed node-gyp to date. See "Undo" below. -We need to start by knowing your version of `npm`: -```bash -npm --version +## Linux and macOS ``` - -## Linux, macOS, Solaris, etc. - -Unix is easy. Just run the following command. - -If your npm is version ___7___, do: -```bash -$ npm explore npm/node_modules/@npmcli/run-script -g -- npm_config_global=false npm install node-gyp@latest -``` - -Else if your npm is version ___less than 7___, do: -```bash -$ npm explore npm/node_modules/npm-lifecycle -g -- npm install node-gyp@latest +npm install --global node-gyp@latest +npm config set node_gyp $(npm prefix -g)/lib/node_modules/node-gyp/bin/node-gyp.js ``` -If the command fails with a permissions error, please try `sudo` and then the command. +`sudo` may be required for the first command if you get a permission error. ## Windows +@joaocgreis' Windows instructions from [#1753 (comment)](https://github.comnodejs/node-gyp/issues/1753#issuecomment-501827267) -Windows is a bit trickier, since `npm` might be installed to the "Program Files" directory, which needs admin privileges in order to -modify on current Windows. Therefore, run the following commands __inside a `cmd.exe` started with "Run as Administrator"__: - -First we need to find the location of `node`. If you don't already know the location that `node.exe` got installed to, then run: -```bash -$ where node +### Windows Command Prompt ``` - -Now `cd` to the directory that `node.exe` is contained in e.g.: -```bash -$ cd "C:\Program Files\nodejs" +npm install --global node-gyp@latest +for /f "delims=" %P in ('npm prefix -g') do npm config set node_gyp"%P\node_modules\node-gyp\bin\node-gyp.js" ``` -If your npm version is ___7___, do: -```bash -cd node_modules\npm\node_modules\@npmcli\run-script +### Powershell ``` - -Else if your npm version is ___less than 7___, do: -```bash -cd node_modules\npm\node_modules\npm-lifecycle +npm install --global node-gyp@latest +npm prefix -g | % {npm config set node_gyp "$_\node_modules\node-gyp\bin\node-gypjs"} ``` -Finish by running: -```bash -$ npm install node-gyp@latest +## Undo +**Beware** if you don't unset the `node_gyp` config option, npm will continue to use the globally installed version of node-gyp rather than the one it ships with,which may end up being newer. + +``` +npm config delete node_gyp +npm uninstall --global node-gyp ``` From f0882b1264b2fa701adbc81a3be0b3cba80e333d Mon Sep 17 00:00:00 2001 From: rvagg Date: Tue, 6 Jul 2021 12:59:12 +0300 Subject: [PATCH 301/551] fix: missing spaces --- docs/Updating-npm-bundled-node-gyp.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/Updating-npm-bundled-node-gyp.md b/docs/Updating-npm-bundled-node-gyp.md index bd237b6905..01ad5642b2 100644 --- a/docs/Updating-npm-bundled-node-gyp.md +++ b/docs/Updating-npm-bundled-node-gyp.md @@ -6,9 +6,9 @@ not running a [current version of node-gyp](https://github.com/nodejs/node-gyp/r `npm` bundles its own, internal, copy of `node-gyp`. This internal copy is independent of any globally installed copy of node-gyp that may have been installed via `npm install -g node-gyp`. -Generally, npm's library files are installed inside your global "node_modules",where npm is installed (run `npm prefix` and add `lib/node_modules`, or just `node_modules` for Windows). There are some exceptions to this. Inside this global `node_modules/`there will be an `npm/` directory and inside this you'll find a `node_modules/node-gyp/` directory. So it may look something like `/usr/local/lib/node_modules/npm/node_modules/node-gyp/`. This is the version of node-gyp that ships with npm. +Generally, npm's library files are installed inside your global "node_modules", where npm is installed (run `npm prefix` and add `lib/node_modules`, or just `node_modules` for Windows). There are some exceptions to this. Inside this global `node_modules/` there will be an `npm/` directory and inside this you'll find a `node_modules/node-gyp/` directory. So it may look something like `/usr/local/lib/node_modules/npm/node_modules/node-gyp/`. This is the version of node-gyp that ships with npm. -When you install a _new_ version of node-gyp with outside of npm, it'll go into your global node_modules, but not under the `npm/node_modules`. So that may look like `/usr/local/lib/node_modules/node-gyp/`. It'll have the `node-gyp` executable linked into your `PATH` so running `node-gyp` will use this version. +When you install a _new_ version of node-gyp outside of npm, it'll go into your global node_modules, but not under the `npm/node_modules`. So that may look like `/usr/local/lib/node_modules/node-gyp/`. It'll have the `node-gyp` executable linked into your `PATH` so running `node-gyp` will use this version. The catch is that npm won't use this version unless you tell it to, it'll keep on using the one you have installed. You need to instruct it to by setting the `node_gyp` config variable (which goes into your `~/.npmrc`). You do this by running the `npm config set` command as below. Then npm will use the command in the path you supply whenever it needs to build a native addon. @@ -23,7 +23,6 @@ npm config set node_gyp $(npm prefix -g)/lib/node_modules/node-gyp/bin/node-gyp. `sudo` may be required for the first command if you get a permission error. ## Windows -@joaocgreis' Windows instructions from [#1753 (comment)](https://github.comnodejs/node-gyp/issues/1753#issuecomment-501827267) ### Windows Command Prompt ``` @@ -38,7 +37,7 @@ npm prefix -g | % {npm config set node_gyp "$_\node_modules\node-gyp\bin\node-gy ``` ## Undo -**Beware** if you don't unset the `node_gyp` config option, npm will continue to use the globally installed version of node-gyp rather than the one it ships with,which may end up being newer. +**Beware** if you don't unset the `node_gyp` config option, npm will continue to use the globally installed version of node-gyp rather than the one it ships with, which may end up being newer. ``` npm config delete node_gyp From 78361b357bd79d6625c22e354b840e5678da2264 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Fri, 30 Jul 2021 17:45:27 +0200 Subject: [PATCH 302/551] ISSUE_TEMPLATE.md: Instructions for old versions (#2470) * ISSUE_TEMPLATE.md: Instructions for old versions Also, add a caution about `node sass` being deprecated. * Update .github/ISSUE_TEMPLATE.md Co-authored-by: Rod Vagg Co-authored-by: Rod Vagg --- .github/ISSUE_TEMPLATE.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md index 485e26ecae..c6b213d7be 100644 --- a/.github/ISSUE_TEMPLATE.md +++ b/.github/ISSUE_TEMPLATE.md @@ -13,6 +13,10 @@ provide the basic information we require. --> +Please look thru your error log for the string `gyp info using node-gyp@` and if the version number is less than the [current release of node-gyp](https://github.com/nodejs/node-gyp/releases) then __please upgrade__ using the instructions at https://github.com/nodejs/node-gyp/blob/master/docs/Updating-npm-bundled-node-gyp.md and try your command again. + +Requests for help with [`node-sass` are very common](https://github.com/nodejs/node-gyp/issues?q=label%3A%22Node+Sass+--%3E+Dart+Sass%22). Please be aware that this package is deprecated, you should seek alternatives and avoid opening new issues about it here. + * **Node Version**: * **Platform**: * **Compiler**: @@ -46,4 +50,3 @@ Usage: npm - From ec15a3e5012004172713c11eebcc9d852d32d380 Mon Sep 17 00:00:00 2001 From: Mayank <9084735+mayank99@users.noreply.github.com> Date: Wed, 11 Aug 2021 21:58:34 -0400 Subject: [PATCH 303/551] chore(deps): bump tar from 6.1.0 to 6.1.2 (#2474) Addresses https://github.com/npm/node-tar/security/advisories/GHSA-3jfq-g458-7qm9 and https://github.com/npm/node-tar/security/advisories/GHSA-r628-mhmh-qjhw --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index 469675b80a..596e60f914 100644 --- a/package.json +++ b/package.json @@ -30,7 +30,7 @@ "npmlog": "^4.1.2", "rimraf": "^3.0.2", "semver": "^7.3.5", - "tar": "^6.1.0", + "tar": "^6.1.2", "which": "^2.0.2" }, "engines": { From 660dd7b2a822c184be8027b300e68be67b366772 Mon Sep 17 00:00:00 2001 From: nineninesevenfour <75562299+nineninesevenfour@users.noreply.github.com> Date: Fri, 13 Aug 2021 13:59:46 +0200 Subject: [PATCH 304/551] doc: correct link to "binding.gyp files out in the wild" (#2483) correct link to "binding.gyp files out in the wild" --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e5a2a026c5..11ba7ab693 100644 --- a/README.md +++ b/README.md @@ -152,7 +152,7 @@ Some additional resources for Node.js native addons and writing `gyp` configurat * ["Hello World" node addon example](https://github.com/nodejs/node/tree/master/test/addons/hello-world) * [gyp user documentation](https://gyp.gsrc.io/docs/UserDocumentation.md) * [gyp input format reference](https://gyp.gsrc.io/docs/InputFormatReference.md) - * [*"binding.gyp" files out in the wild* wiki page](./docs/"binding.gyp"-files-out-in-the-wild.md) + * [*"binding.gyp" files out in the wild* wiki page](./docs/binding.gyp-files-in-the-wild.md) ## Commands From ed9a9ed653a17c84afa3c327161992d0da7d0cea Mon Sep 17 00:00:00 2001 From: Jiawen Geng <3759816+gengjiawen@users.noreply.github.com> Date: Mon, 23 Aug 2021 14:30:21 +0800 Subject: [PATCH 305/551] feat(gyp): update gyp to v0.9.6 (#2481) --- gyp/.github/workflows/Python_tests.yml | 2 +- gyp/.github/workflows/node-gyp.yml | 4 ++- gyp/CHANGELOG.md | 38 ++++++++++++++++++++ gyp/pylib/gyp/MSVSUtil.py | 2 +- gyp/pylib/gyp/common.py | 2 +- gyp/pylib/gyp/easy_xml.py | 6 ++-- gyp/pylib/gyp/generator/android.py | 6 ++-- gyp/pylib/gyp/generator/make.py | 2 +- gyp/pylib/gyp/generator/msvs.py | 29 ++++++++++----- gyp/pylib/gyp/generator/ninja.py | 32 ++++++++--------- gyp/pylib/gyp/input.py | 2 +- gyp/pylib/gyp/msvs_emulation.py | 49 +++++++++++++++++--------- gyp/pylib/gyp/win_tool.py | 5 +-- gyp/pylib/gyp/xcodeproj_file.py | 8 ++--- gyp/setup.py | 2 +- gyp/test_gyp.py | 21 ++++------- 16 files changed, 137 insertions(+), 73 deletions(-) diff --git a/gyp/.github/workflows/Python_tests.yml b/gyp/.github/workflows/Python_tests.yml index 649251c8dd..92303b635f 100644 --- a/gyp/.github/workflows/Python_tests.yml +++ b/gyp/.github/workflows/Python_tests.yml @@ -23,7 +23,7 @@ jobs: python -m pip install --upgrade pip pip install -r requirements_dev.txt - name: Lint with flake8 - run: flake8 . --count --show-source --statistics + run: flake8 . --ignore=E203,W503 --max-complexity=101 --max-line-length=88 --show-source --statistics - name: Test with pytest run: pytest # - name: Run doctests with pytest diff --git a/gyp/.github/workflows/node-gyp.yml b/gyp/.github/workflows/node-gyp.yml index 59d23fdffc..bd7c85ffda 100644 --- a/gyp/.github/workflows/node-gyp.yml +++ b/gyp/.github/workflows/node-gyp.yml @@ -8,6 +8,8 @@ jobs: fail-fast: false matrix: os: [macos-latest, ubuntu-latest, windows-latest] + python: [3.6, 3.9] + runs-on: ${{ matrix.os }} steps: - name: Clone gyp-next @@ -24,7 +26,7 @@ jobs: node-version: 14.x - uses: actions/setup-python@v2 with: - python-version: 3.9 + python-version: ${{ matrix.python }} - name: Install dependencies run: | cd node-gyp diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md index 6d66b3acd2..f5e9f63476 100644 --- a/gyp/CHANGELOG.md +++ b/gyp/CHANGELOG.md @@ -1,5 +1,43 @@ # Changelog +### [0.9.6](https://www.github.com/nodejs/gyp-next/compare/v0.9.5...v0.9.6) (2021-08-23) + + +### Bug Fixes + +* align flake8 test ([#122](https://www.github.com/nodejs/gyp-next/issues/122)) ([f1faa8d](https://www.github.com/nodejs/gyp-next/commit/f1faa8d3081e1a47e917ff910892f00dff16cf8a)) +* **msvs:** fix paths again in action command arguments ([#121](https://www.github.com/nodejs/gyp-next/issues/121)) ([7159dfb](https://www.github.com/nodejs/gyp-next/commit/7159dfbc5758c9ec717e215f2c36daf482c846a1)) + +### [0.9.5](https://www.github.com/nodejs/gyp-next/compare/v0.9.4...v0.9.5) (2021-08-18) + + +### Bug Fixes + +* add python 3.6 to node-gyp integration test ([3462d4c](https://www.github.com/nodejs/gyp-next/commit/3462d4ce3c31cce747513dc7ca9760c81d57c60e)) +* revert for windows compatibility ([d078e7d](https://www.github.com/nodejs/gyp-next/commit/d078e7d7ae080ddae243188f6415f940376a7368)) +* support msvs_quote_cmd in ninja generator ([#117](https://www.github.com/nodejs/gyp-next/issues/117)) ([46486ac](https://www.github.com/nodejs/gyp-next/commit/46486ac6e9329529d51061e006a5b39631e46729)) + +### [0.9.4](https://www.github.com/nodejs/gyp-next/compare/v0.9.3...v0.9.4) (2021-08-09) + + +### Bug Fixes + +* .S is an extension for asm file on Windows ([#115](https://www.github.com/nodejs/gyp-next/issues/115)) ([d2fad44](https://www.github.com/nodejs/gyp-next/commit/d2fad44ef3a79ca8900f1307060153ded57053fc)) + +### [0.9.3](https://www.github.com/nodejs/gyp-next/compare/v0.9.2...v0.9.3) (2021-07-07) + + +### Bug Fixes + +* build failure with ninja and Python 3 on Windows ([#113](https://www.github.com/nodejs/gyp-next/issues/113)) ([c172d10](https://www.github.com/nodejs/gyp-next/commit/c172d105deff5db4244e583942215918fa80dd3c)) + +### [0.9.2](https://www.github.com/nodejs/gyp-next/compare/v0.9.1...v0.9.2) (2021-05-21) + + +### Bug Fixes + +* add support of utf8 encoding ([#105](https://www.github.com/nodejs/gyp-next/issues/105)) ([4d0f93c](https://www.github.com/nodejs/gyp-next/commit/4d0f93c249286d1f0c0f665f5fe7346119f98cf1)) + ### [0.9.1](https://www.github.com/nodejs/gyp-next/compare/v0.9.0...v0.9.1) (2021-05-14) diff --git a/gyp/pylib/gyp/MSVSUtil.py b/gyp/pylib/gyp/MSVSUtil.py index cb55305eae..36bb782bd3 100644 --- a/gyp/pylib/gyp/MSVSUtil.py +++ b/gyp/pylib/gyp/MSVSUtil.py @@ -55,7 +55,7 @@ def _SuffixName(name, suffix): Target name with suffix added (foo_suffix#target) """ parts = name.rsplit("#", 1) - parts[0] = "{}_{}".format(parts[0], suffix) + parts[0] = f"{parts[0]}_{suffix}" return "#".join(parts) diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py index ba310ce247..9213fcc5e8 100644 --- a/gyp/pylib/gyp/common.py +++ b/gyp/pylib/gyp/common.py @@ -562,7 +562,7 @@ def pop(self, last=True): # pylint: disable=W0221 def __repr__(self): if not self: return f"{self.__class__.__name__}()" - return "{}({!r})".format(self.__class__.__name__, list(self)) + return f"{self.__class__.__name__}({list(self)!r})" def __eq__(self, other): if isinstance(other, OrderedSet): diff --git a/gyp/pylib/gyp/easy_xml.py b/gyp/pylib/gyp/easy_xml.py index e475b5530c..bda1a47468 100644 --- a/gyp/pylib/gyp/easy_xml.py +++ b/gyp/pylib/gyp/easy_xml.py @@ -2,6 +2,7 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. +import sys import re import os import locale @@ -84,7 +85,7 @@ def _ConstructContentList(xml_parts, specification, pretty, level=0): rest = specification[1:] if rest and isinstance(rest[0], dict): for at, val in sorted(rest[0].items()): - xml_parts.append(' {}="{}"'.format(at, _XmlEscape(val, attr=True))) + xml_parts.append(f' {at}="{_XmlEscape(val, attr=True)}"') rest = rest[1:] if rest: xml_parts.append(">") @@ -106,7 +107,8 @@ def _ConstructContentList(xml_parts, specification, pretty, level=0): xml_parts.append("/>%s" % new_line) -def WriteXmlIfChanged(content, path, encoding="utf-8", pretty=False, win32=False): +def WriteXmlIfChanged(content, path, encoding="utf-8", pretty=False, + win32=(sys.platform == "win32")): """ Writes the XML content to disk, touching the file only if it has changed. Args: diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py index 040d8088a2..cdf1a4832c 100644 --- a/gyp/pylib/gyp/generator/android.py +++ b/gyp/pylib/gyp/generator/android.py @@ -349,7 +349,7 @@ def WriteActions(self, actions, extra_sources, extra_outputs): for output in outputs[1:]: # Make each output depend on the main output, with an empty command # to force make to notice that the mtime has changed. - self.WriteLn("{}: {} ;".format(self.LocalPathify(output), main_output)) + self.WriteLn(f"{self.LocalPathify(output)}: {main_output} ;") extra_outputs += outputs self.WriteLn() @@ -616,7 +616,7 @@ def WriteSources(self, spec, configs, extra_sources): if IsCPPExtension(ext) and ext != local_cpp_extension: local_file = root + local_cpp_extension if local_file != source: - self.WriteLn("{}: {}".format(local_file, self.LocalPathify(source))) + self.WriteLn(f"{local_file}: {self.LocalPathify(source)}") self.WriteLn("\tmkdir -p $(@D); cp $< $@") origin_src_dirs.append(os.path.dirname(source)) final_generated_sources.append(local_file) @@ -908,7 +908,7 @@ def WriteTarget( if isinstance(v, list): self.WriteList(v, k) else: - self.WriteLn("{} := {}".format(k, make.QuoteIfNecessary(v))) + self.WriteLn(f"{k} := {make.QuoteIfNecessary(v)}") self.WriteLn("") # Add to the set of targets which represent the gyp 'all' target. We use the diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index eb9102dd15..c595f20fe2 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -2133,7 +2133,7 @@ def WriteSortedXcodeEnv(self, target, env): # export foo := a\ b # it does not -- the backslash is written to the env as literal character. # So don't escape spaces in |env[k]|. - self.WriteLn("{}: export {} := {}".format(QuoteSpaces(target), k, v)) + self.WriteLn(f"{QuoteSpaces(target)}: export {k} := {v}") def Objectify(self, path): """Convert a path to its output directory form.""" diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index 5435eb1e1f..8308fa8433 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -152,7 +152,7 @@ def _NormalizedSource(source): return source -def _FixPath(path): +def _FixPath(path, separator="\\"): """Convert paths to a form that will make sense in a vcproj file. Arguments: @@ -168,9 +168,12 @@ def _FixPath(path): and not _IsWindowsAbsPath(path) ): path = os.path.join(fixpath_prefix, path) - path = path.replace("/", "\\") + if separator == "\\": + path = path.replace("/", "\\") path = _NormalizedSource(path) - if path and path[-1] == "\\": + if separator == "/": + path = path.replace("\\", "/") + if path and path[-1] == separator: path = path[:-1] return path @@ -185,9 +188,9 @@ def _IsWindowsAbsPath(path): return path.startswith("c:") or path.startswith("C:") -def _FixPaths(paths): +def _FixPaths(paths, separator="\\"): """Fix each of the paths of the list.""" - return [_FixPath(i) for i in paths] + return [_FixPath(i, separator) for i in paths] def _ConvertSourcesToFilterHierarchy( @@ -314,7 +317,7 @@ def _ConfigBaseName(config_name, platform_name): def _ConfigFullName(config_name, config_data): platform_name = _ConfigPlatform(config_data) - return "{}|{}".format(_ConfigBaseName(config_name, platform_name), platform_name) + return f"{_ConfigBaseName(config_name, platform_name)}|{platform_name}" def _ConfigWindowsTargetPlatformVersion(config_data, version): @@ -335,7 +338,7 @@ def _ConfigWindowsTargetPlatformVersion(config_data, version): # Find a matching entry in sdk_dir\include. expected_sdk_dir = r"%s\include" % sdk_dir names = sorted( - [ + ( x for x in ( os.listdir(expected_sdk_dir) @@ -343,7 +346,7 @@ def _ConfigWindowsTargetPlatformVersion(config_data, version): else [] ) if x.startswith(version) - ], + ), reverse=True, ) if names: @@ -418,7 +421,15 @@ def _BuildCommandLineForRuleRaw( # file out of the raw command string, and some commands (like python) are # actually batch files themselves. command.insert(0, "call") - arguments = [i.replace("$(InputDir)", "%INPUTDIR%") for i in cmd[1:]] + # Fix the paths + # TODO(quote): This is a really ugly heuristic, and will miss path fixing + # for arguments like "--arg=path" or "/opt:path". + # If the argument starts with a slash or dash, it's probably a command line + # switch + # Return the path with forward slashes because the command using it might + # not support backslashes. + arguments = [i if (i[:1] in "/-") else _FixPath(i, "/") for i in cmd[1:]] + arguments = [i.replace("$(InputDir)", "%INPUTDIR%") for i in arguments] arguments = [MSVSSettings.FixVCMacroSlashes(i) for i in arguments] if quote_cmd: # Support a mode for using cmd directly. diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py index ca032aef20..d173bf2299 100644 --- a/gyp/pylib/gyp/generator/ninja.py +++ b/gyp/pylib/gyp/generator/ninja.py @@ -638,7 +638,7 @@ def GenerateDescription(self, verb, message, fallback): if self.toolset != "target": verb += "(%s)" % self.toolset if message: - return "{} {}".format(verb, self.ExpandSpecial(message)) + return f"{verb} {self.ExpandSpecial(message)}" else: return f"{verb} {self.name}: {fallback}" @@ -654,10 +654,10 @@ def WriteActions( description = self.GenerateDescription( "ACTION", action.get("message", None), name ) - is_cygwin = ( - self.msvs_settings.IsRuleRunUnderCygwin(action) + win_shell_flags = ( + self.msvs_settings.GetRuleShellFlags(action) if self.flavor == "win" - else False + else None ) args = action["action"] depfile = action.get("depfile", None) @@ -665,7 +665,7 @@ def WriteActions( depfile = self.ExpandSpecial(depfile, self.base_to_build) pool = "console" if int(action.get("ninja_use_console", 0)) else None rule_name, _ = self.WriteNewNinjaRule( - name, args, description, is_cygwin, env, pool, depfile=depfile + name, args, description, win_shell_flags, env, pool, depfile=depfile ) inputs = [self.GypPathToNinja(i, env) for i in action["inputs"]] @@ -707,14 +707,14 @@ def WriteRules( rule.get("message", None), ("%s " + generator_default_variables["RULE_INPUT_PATH"]) % name, ) - is_cygwin = ( - self.msvs_settings.IsRuleRunUnderCygwin(rule) + win_shell_flags = ( + self.msvs_settings.GetRuleShellFlags(rule) if self.flavor == "win" - else False + else None ) pool = "console" if int(rule.get("ninja_use_console", 0)) else None rule_name, args = self.WriteNewNinjaRule( - name, args, description, is_cygwin, env, pool + name, args, description, win_shell_flags, env, pool ) # TODO: if the command references the outputs directly, we should @@ -733,7 +733,7 @@ def WriteRules( def cygwin_munge(path): # pylint: disable=cell-var-from-loop - if is_cygwin: + if win_shell_flags and win_shell_flags.cygwin: return path.replace("\\", "/") return path @@ -1221,7 +1221,7 @@ def WriteSourcesForArch( command = "cc_s" elif ( self.flavor == "win" - and ext == "asm" + and ext in ("asm", "S") and not self.msvs_settings.HasExplicitAsmRules(spec) ): command = "asm" @@ -1899,7 +1899,7 @@ def WriteVariableList(self, ninja_file, var, values): ninja_file.variable(var, " ".join(values)) def WriteNewNinjaRule( - self, name, args, description, is_cygwin, env, pool, depfile=None + self, name, args, description, win_shell_flags, env, pool, depfile=None ): """Write out a new ninja "rule" statement for a given command. @@ -1946,13 +1946,14 @@ def WriteNewNinjaRule( if self.flavor == "win": rspfile = rule_name + ".$unique_name.rsp" # The cygwin case handles this inside the bash sub-shell. - run_in = "" if is_cygwin else " " + self.build_to_base - if is_cygwin: + run_in = "" if win_shell_flags.cygwin else " " + self.build_to_base + if win_shell_flags.cygwin: rspfile_content = self.msvs_settings.BuildCygwinBashCommandLine( args, self.build_to_base ) else: - rspfile_content = gyp.msvs_emulation.EncodeRspFileList(args) + rspfile_content = gyp.msvs_emulation.EncodeRspFileList( + args, win_shell_flags.quote) command = ( "%s gyp-win-tool action-wrapper $arch " % sys.executable + rspfile @@ -2389,7 +2390,6 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name ) if flavor == "win": master_ninja.variable("ld_host", ld_host) - master_ninja.variable("ldxx_host", ldxx_host) else: master_ninja.variable( "ld_host", CommandWithWrapper("LINK", wrappers, ld_host) diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index ca7ce44eab..354958bfb2 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -225,7 +225,7 @@ def LoadOneBuildFile(build_file_path, data, aux_data, includes, is_target, check return data[build_file_path] if os.path.exists(build_file_path): - build_file_contents = open(build_file_path).read() + build_file_contents = open(build_file_path, encoding='utf-8').read() else: raise GypError(f"{build_file_path} not found (cwd: {os.getcwd()})") diff --git a/gyp/pylib/gyp/msvs_emulation.py b/gyp/pylib/gyp/msvs_emulation.py index f744e38df1..5b9c2712e0 100644 --- a/gyp/pylib/gyp/msvs_emulation.py +++ b/gyp/pylib/gyp/msvs_emulation.py @@ -7,6 +7,7 @@ build systems, primarily ninja. """ +import collections import os import re import subprocess @@ -19,7 +20,7 @@ windows_quoter_regex = re.compile(r'(\\*)"') -def QuoteForRspFile(arg): +def QuoteForRspFile(arg, quote_cmd=True): """Quote a command line argument so that it appears as one argument when processed via cmd.exe and parsed by CommandLineToArgvW (as is typical for Windows programs).""" @@ -36,7 +37,8 @@ def QuoteForRspFile(arg): # For a literal quote, CommandLineToArgvW requires 2n+1 backslashes # preceding it, and results in n backslashes + the quote. So we substitute # in 2* what we match, +1 more, plus the quote. - arg = windows_quoter_regex.sub(lambda mo: 2 * mo.group(1) + '\\"', arg) + if quote_cmd: + arg = windows_quoter_regex.sub(lambda mo: 2 * mo.group(1) + '\\"', arg) # %'s also need to be doubled otherwise they're interpreted as batch # positional arguments. Also make sure to escape the % so that they're @@ -48,12 +50,17 @@ def QuoteForRspFile(arg): # These commands are used in rsp files, so no escaping for the shell (via ^) # is necessary. - # Finally, wrap the whole thing in quotes so that the above quote rule - # applies and whitespace isn't a word break. - return '"' + arg + '"' + # As a workaround for programs that don't use CommandLineToArgvW, gyp + # supports msvs_quote_cmd=0, which simply disables all quoting. + if quote_cmd: + # Finally, wrap the whole thing in quotes so that the above quote rule + # applies and whitespace isn't a word break. + return f'"{arg}"' + return arg -def EncodeRspFileList(args): + +def EncodeRspFileList(args, quote_cmd): """Process a list of arguments using QuoteCmdExeArgument.""" # Note that the first argument is assumed to be the command. Don't add # quotes around it because then built-ins like 'echo', etc. won't work. @@ -67,7 +74,8 @@ def EncodeRspFileList(args): program = call + " " + os.path.normpath(program) else: program = os.path.normpath(args[0]) - return program + " " + " ".join(QuoteForRspFile(arg) for arg in args[1:]) + return (program + " " + + " ".join(QuoteForRspFile(arg, quote_cmd) for arg in args[1:])) def _GenericRetrieve(root, default, path): @@ -333,7 +341,7 @@ def _TargetConfig(self, config): # first level is globally for the configuration (this is what we consider # "the" config at the gyp level, which will be something like 'Debug' or # 'Release'), VS2015 and later only use this level - if self.vs_version.short_name >= 2015: + if int(self.vs_version.short_name) >= 2015: return config # and a second target-specific configuration, which is an # override for the global one. |config| is remapped here to take into @@ -537,7 +545,7 @@ def GetCflags(self, config): ) ] ) - if self.vs_version.project_version >= 12.0: + if float(self.vs_version.project_version) >= 12.0: # New flag introduced in VS2013 (project version 12.0) Forces writes to # the program database (PDB) to be serialized through MSPDBSRV.EXE. # https://msdn.microsoft.com/en-us/library/dn502518.aspx @@ -933,13 +941,22 @@ def BuildCygwinBashCommandLine(self, args, path_to_base): ) return cmd - def IsRuleRunUnderCygwin(self, rule): - """Determine if an action should be run under cygwin. If the variable is - unset, or set to 1 we use cygwin.""" - return ( - int(rule.get("msvs_cygwin_shell", self.spec.get("msvs_cygwin_shell", 1))) - != 0 - ) + RuleShellFlags = collections.namedtuple("RuleShellFlags", ["cygwin", "quote"]) + + def GetRuleShellFlags(self, rule): + """Return RuleShellFlags about how the given rule should be run. This + includes whether it should run under cygwin (msvs_cygwin_shell), and + whether the commands should be quoted (msvs_quote_cmd).""" + # If the variable is unset, or set to 1 we use cygwin + cygwin = int(rule.get("msvs_cygwin_shell", + self.spec.get("msvs_cygwin_shell", 1))) != 0 + # Default to quoting. There's only a few special instances where the + # target command uses non-standard command line parsing and handle quotes + # and quote escaping differently. + quote_cmd = int(rule.get("msvs_quote_cmd", 1)) + assert quote_cmd != 0 or cygwin != 1, \ + "msvs_quote_cmd=0 only applicable for msvs_cygwin_shell=0" + return MsvsSettings.RuleShellFlags(cygwin, quote_cmd) def _HasExplicitRuleForExtension(self, spec, extension): """Determine if there's an explicit rule for a particular extension.""" diff --git a/gyp/pylib/gyp/win_tool.py b/gyp/pylib/gyp/win_tool.py index 4dbcda50a4..638eee4002 100755 --- a/gyp/pylib/gyp/win_tool.py +++ b/gyp/pylib/gyp/win_tool.py @@ -221,8 +221,9 @@ def ExecLinkWithManifests( # and sometimes doesn't unfortunately. with open(our_manifest) as our_f: with open(assert_manifest) as assert_f: - our_data = our_f.read().translate(None, string.whitespace) - assert_data = assert_f.read().translate(None, string.whitespace) + translator = str.maketrans('', '', string.whitespace) + our_data = our_f.read().translate(translator) + assert_data = assert_f.read().translate(translator) if our_data != assert_data: os.unlink(out) diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py index 5863ef45df..076eea3721 100644 --- a/gyp/pylib/gyp/xcodeproj_file.py +++ b/gyp/pylib/gyp/xcodeproj_file.py @@ -299,8 +299,8 @@ def __repr__(self): try: name = self.Name() except NotImplementedError: - return "<{} at 0x{:x}>".format(self.__class__.__name__, id(self)) - return "<{} {!r} at 0x{:x}>".format(self.__class__.__name__, name, id(self)) + return f"<{self.__class__.__name__} at 0x{id(self):x}>" + return f"<{self.__class__.__name__} {name!r} at 0x{id(self):x}>" def Copy(self): """Make a copy of this object. @@ -2251,7 +2251,7 @@ class PBXContainerItemProxy(XCObject): def __repr__(self): props = self._properties name = "{}.gyp:{}".format(props["containerPortal"].Name(), props["remoteInfo"]) - return "<{} {!r} at 0x{:x}>".format(self.__class__.__name__, name, id(self)) + return f"<{self.__class__.__name__} {name!r} at 0x{id(self):x}>" def Name(self): # Admittedly not the best name, but it's what Xcode uses. @@ -2288,7 +2288,7 @@ class PBXTargetDependency(XCObject): def __repr__(self): name = self._properties.get("name") or self._properties["target"].Name() - return "<{} {!r} at 0x{:x}>".format(self.__class__.__name__, name, id(self)) + return f"<{self.__class__.__name__} {name!r} at 0x{id(self):x}>" def Name(self): # Admittedly not the best name, but it's what Xcode uses. diff --git a/gyp/setup.py b/gyp/setup.py index f4a9481937..0ce46123cc 100644 --- a/gyp/setup.py +++ b/gyp/setup.py @@ -15,7 +15,7 @@ setup( name="gyp-next", - version="0.9.1", + version="0.9.6", description="A fork of the GYP build system for use in the Node.js projects", long_description=long_description, long_description_content_type="text/markdown", diff --git a/gyp/test_gyp.py b/gyp/test_gyp.py index 757d2fc0b0..9ba264170f 100755 --- a/gyp/test_gyp.py +++ b/gyp/test_gyp.py @@ -140,10 +140,7 @@ def main(argv=None): if not args.quiet: runner.print_results() - if runner.failures: - return 1 - else: - return 0 + return 1 if runner.failures else 0 def print_configuration_info(): @@ -152,8 +149,8 @@ def print_configuration_info(): sys.path.append(os.path.abspath("test/lib")) import TestMac - print(" Mac {} {}".format(platform.mac_ver()[0], platform.mac_ver()[2])) - print(" Xcode %s" % TestMac.Xcode.Version()) + print(f" Mac {platform.mac_ver()[0]} {platform.mac_ver()[2]}") + print(f" Xcode {TestMac.Xcode.Version()}") elif sys.platform == "win32": sys.path.append(os.path.abspath("pylib")) import gyp.MSVSVersion @@ -162,8 +159,8 @@ def print_configuration_info(): print(" MSVS %s" % gyp.MSVSVersion.SelectVisualStudioVersion().Description()) elif sys.platform in ("linux", "linux2"): print(" Linux %s" % " ".join(platform.linux_distribution())) - print(" Python %s" % platform.python_version()) - print(" PYTHONPATH=%s" % os.environ["PYTHONPATH"]) + print(f" Python {platform.python_version()}") + print(f" PYTHONPATH={os.environ['PYTHONPATH']}") print() @@ -222,13 +219,9 @@ def run_test(self, test, fmt, i): res_msg = f" {res} {took:.3f}s" self.print_(res_msg) - if ( - stdout - and not stdout.endswith("PASSED\n") - and not (stdout.endswith("NO RESULT\n")) - ): + if stdout and not stdout.endswith(("PASSED\n", "NO RESULT\n")): print() - print("\n".join(" %s" % line for line in stdout.splitlines())) + print("\n".join(f" {line}" for line in stdout.splitlines())) elif not self.isatty: print() From bc47cd60b986eaa55a23050d8f72d1cc117bdba0 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Mon, 23 Aug 2021 06:31:25 +0000 Subject: [PATCH 306/551] chore: release 8.2.0 --- CHANGELOG.md | 119 +++++++++++++++++++++++++++++++++++++++++++++++++++ package.json | 2 +- 2 files changed, 120 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b002a27ec3..7d513542fd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,124 @@ # Changelog +## [8.2.0](https://www.github.com/nodejs/node-gyp/compare/v8.1.0...v8.2.0) (2021-08-23) + + +### Features + +* **gyp:** update gyp to v0.9.6 ([#2481](https://www.github.com/nodejs/node-gyp/issues/2481)) ([ed9a9ed](https://www.github.com/nodejs/node-gyp/commit/ed9a9ed653a17c84afa3c327161992d0da7d0cea)) + + +### Bug Fixes + +* add error arg back into catch block for older Node.js users ([5cde818](https://www.github.com/nodejs/node-gyp/commit/5cde818aac715477e9e9747966bb6b4c4ed070a8)) +* change default gyp update message ([#2420](https://www.github.com/nodejs/node-gyp/issues/2420)) ([cfd12ff](https://www.github.com/nodejs/node-gyp/commit/cfd12ff3bb0eb4525173413ef6a94b3cd8398cad)) +* doc how to update node-gyp independently from npm ([c8c0af7](https://www.github.com/nodejs/node-gyp/commit/c8c0af72e78141a02b5da4cd4d704838333a90bd)) +* missing spaces ([f0882b1](https://www.github.com/nodejs/node-gyp/commit/f0882b1264b2fa701adbc81a3be0b3cba80e333d)) + + +### Core + +* deep-copy process.config during configure ([#2368](https://www.github.com/nodejs/node-gyp/issues/2368)) ([5f1a06c](https://www.github.com/nodejs/node-gyp/commit/5f1a06c50f3b0c3d292f64948f85a004cfcc5c87)) + + +### Miscellaneous + +* **deps:** bump tar from 6.1.0 to 6.1.2 ([#2474](https://www.github.com/nodejs/node-gyp/issues/2474)) ([ec15a3e](https://www.github.com/nodejs/node-gyp/commit/ec15a3e5012004172713c11eebcc9d852d32d380)) +* fix typos discovered by codespell ([#2442](https://www.github.com/nodejs/node-gyp/issues/2442)) ([2d0ce55](https://www.github.com/nodejs/node-gyp/commit/2d0ce5595e232a3fc7c562cdf39efb77e2312cc1)) +* GitHub Actions Test on node: [12.x, 14.x, 16.x] ([#2439](https://www.github.com/nodejs/node-gyp/issues/2439)) ([b7bccdb](https://www.github.com/nodejs/node-gyp/commit/b7bccdb527d93b0bb0ce99713f083ce2985fe85c)) + + +### Doc + +* correct link to "binding.gyp files out in the wild" ([#2483](https://www.github.com/nodejs/node-gyp/issues/2483)) ([660dd7b](https://www.github.com/nodejs/node-gyp/commit/660dd7b2a822c184be8027b300e68be67b366772)) +* **wiki:** Add a link to the node-midi binding.gyp file. ([b354711](https://www.github.com/nodejs/node-gyp/commit/b3547115f6e356358138310e857c7f1ec627a8a7)) +* **wiki:** add bcrypt ([e199cfa](https://www.github.com/nodejs/node-gyp/commit/e199cfa8fc6161492d2a6ade2190510d0ebf7c0f)) +* **wiki:** Add helpful information ([4eda827](https://www.github.com/nodejs/node-gyp/commit/4eda8275c03dae6d2f5c40f3c1dbe930d84b0f2b)) +* **wiki:** Add node-canvas ([13a9553](https://www.github.com/nodejs/node-gyp/commit/13a955317b39caf98fd1f412d8d3f41599e979fd)) +* **wiki:** Add node-openvg-canvas and node-openvg. ([61f709e](https://www.github.com/nodejs/node-gyp/commit/61f709ec4d9f256a6467e9ff84430a48eeb629d1)) +* **wiki:** add one more example ([77f3632](https://www.github.com/nodejs/node-gyp/commit/77f363272930d3d4d24fd3973be22e6237128fcc)) +* **wiki:** add topcube, node-osmium, and node-osrm ([1a75d2b](https://www.github.com/nodejs/node-gyp/commit/1a75d2bf2f562ba50846893a516e111cfbb50885)) +* **wiki:** Added details for properly fixing ([3d4d9d5](https://www.github.com/nodejs/node-gyp/commit/3d4d9d52d6b5b49de06bb0bb5b68e2686d2b7ebd)) +* **wiki:** Added Ghostscript4JS ([bf4bed1](https://www.github.com/nodejs/node-gyp/commit/bf4bed1b96a7d22fba6f97f4552ad09f32ac3737)) +* **wiki:** added levelup ([1575bce](https://www.github.com/nodejs/node-gyp/commit/1575bce3a53db628bfb023fd6f3258fdf98c3195)) +* **wiki:** Added nk-mysql (nodamysql) ([5b4f2d0](https://www.github.com/nodejs/node-gyp/commit/5b4f2d0e1d5d3eadfd03aaf9c1668340f76c4bea)) +* **wiki:** Added nk-xrm-installer .gyp references, including .py scripts for providing complete reference to examples of fetching source via http, extracting, and moving files (as opposed to copying) ([ceb3088](https://www.github.com/nodejs/node-gyp/commit/ceb30885b74f6789374ef52267b84767be93ebe4)) +* **wiki:** Added tip about resolving frustrating LNK1181 error ([e64798d](https://www.github.com/nodejs/node-gyp/commit/e64798de8cac6031ad598a86d7599e81b4d20b17)) +* **wiki:** ADDED: Node.js binding to OpenCV ([e2dc777](https://www.github.com/nodejs/node-gyp/commit/e2dc77730b09d7ee8682d7713a7603a2d7aacabd)) +* **wiki:** Adding link to node-cryptopp's gyp file ([875adbe](https://www.github.com/nodejs/node-gyp/commit/875adbe2a4669fa5f2be0250ffbf98fb55e800fd)) +* **wiki:** Adding the sharp library to the list ([9dce0e4](https://www.github.com/nodejs/node-gyp/commit/9dce0e41650c3fa973e6135a79632d022c662a1d)) +* **wiki:** Adds node-fann ([23e3d48](https://www.github.com/nodejs/node-gyp/commit/23e3d485ed894ba7c631e9c062f5e366b50c416c)) +* **wiki:** Adds node-inotify and v8-profiler ([b6e542f](https://www.github.com/nodejs/node-gyp/commit/b6e542f644dbbfe22b88524ec500696e06ee4af7)) +* **wiki:** Bumping Python version from 2.3 to 2.7 as per the node-gyp readme ([55ebd6e](https://www.github.com/nodejs/node-gyp/commit/55ebd6ebacde975bf84f7bf4d8c66e64cc7cd0da)) +* **wiki:** C++ build tools version upgraded ([5b899b7](https://www.github.com/nodejs/node-gyp/commit/5b899b70db729c392ced7c98e8e17590c6499fc3)) +* **wiki:** change bcrypt url to binding.gyp file ([e11bdd8](https://www.github.com/nodejs/node-gyp/commit/e11bdd84de6144492d3eb327d67cbf2d62da1a76)) +* **wiki:** Clarification + direct link to VS2010 ([531c724](https://www.github.com/nodejs/node-gyp/commit/531c724561d947b5d870de8d52dd8c3c51c5ec2d)) +* **wiki:** Correcting the link to node-osmium ([fae7516](https://www.github.com/nodejs/node-gyp/commit/fae7516a1d2829b6e234eaded74fb112ebd79a05)) +* **wiki:** Created "binding.gyp" files out in the wild (markdown) ([d4fd143](https://www.github.com/nodejs/node-gyp/commit/d4fd14355bbe57f229f082f47bb2b3670868203f)) +* **wiki:** Created Common issues (markdown) ([a38299e](https://www.github.com/nodejs/node-gyp/commit/a38299ea340ceb0e732c6dc6a1b4760257644839)) +* **wiki:** Created Error: "pre" versions of node cannot be installed (markdown) ([98bc80d](https://www.github.com/nodejs/node-gyp/commit/98bc80d7a62ba70c881f3c39d94f804322e57852)) +* **wiki:** Created Linking to OpenSSL (markdown) ([c46d00d](https://www.github.com/nodejs/node-gyp/commit/c46d00d83bac5173dea8bbbb175a1a7de74fdaca)) +* **wiki:** Created Updating npm's bundled node gyp (markdown) ([e0ac8d1](https://www.github.com/nodejs/node-gyp/commit/e0ac8d15af46aadd1c220599e63199b154a514e6)) +* **wiki:** Created use of undeclared identifier 'TypedArray' (markdown) ([65ba711](https://www.github.com/nodejs/node-gyp/commit/65ba71139e9b7f64ac823e575ee9dbf17d937ce4)) +* **wiki:** Created Visual Studio 2010 Setup (markdown) ([5b80e83](https://www.github.com/nodejs/node-gyp/commit/5b80e834c8f79dda9fb2770a876ff3cf649c06f3)) +* **wiki:** Created Visual studio 2012 setup (markdown) ([becef31](https://www.github.com/nodejs/node-gyp/commit/becef316b6c46a33e783667720ee074a0141d1a5)) +* **wiki:** Destroyed Visual Studio 2010 Setup (markdown) ([93423b4](https://www.github.com/nodejs/node-gyp/commit/93423b43606de9664aeb79635825f5e9941ec9bc)) +* **wiki:** Destroyed Visual studio 2012 setup (markdown) ([3601508](https://www.github.com/nodejs/node-gyp/commit/3601508bb10fa05da0ddc7e70d57e4b4dd679657)) +* **wiki:** Different commands for Windows npm v6 vs. v7 ([0fce46b](https://www.github.com/nodejs/node-gyp/commit/0fce46b53340c85e8091cde347d5ed23a443c82f)) +* **wiki:** Drop in favor of ([9285ff6](https://www.github.com/nodejs/node-gyp/commit/9285ff6e451c52c070a05f05f0a9602621d91d53)) +* **wiki:** Explicit link to Visual C++ 2010 Express ([378c363](https://www.github.com/nodejs/node-gyp/commit/378c3632f02c096ed819ec8f2611c65bef0c0554)) +* **wiki:** fix link to gyp file used to build libsqlite3 ([54db8d7](https://www.github.com/nodejs/node-gyp/commit/54db8d7ac33e3f98220960b5d86cfa18a75b53cb)) +* **wiki:** Fix link to node-zipfile ([92e49a8](https://www.github.com/nodejs/node-gyp/commit/92e49a858ed69cb4847a26a5676ab56ef5e2de33)) +* **wiki:** fixed node-serialport link ([954ee53](https://www.github.com/nodejs/node-gyp/commit/954ee530b3972d1db591fce32368e4e31b5a25d8)) +* **wiki:** I highly missing it in common issue as every windows biggner face that issue ([d617fae](https://www.github.com/nodejs/node-gyp/commit/d617faee29c40871ca5c8f93efd0ce929a40d541)) +* **wiki:** if ouns that the -h did not help. I founs on github that there was support for visual studio 2015, while i couldn't install node-red beacuse it kept telling me the key 2015 was missing. looking in he gyp python code i found the local file was bot up t dat with the github repo. updating took several efforts before i tried to drop the -g option. ([408b72f](https://www.github.com/nodejs/node-gyp/commit/408b72f561329408daeb17834436e381406efcc8)) +* **wiki:** If permissions error, please try and then the command. ([ee8e1c1](https://www.github.com/nodejs/node-gyp/commit/ee8e1c1e5334096d58e0d6bca6c006f2ee9c88cb)) +* **wiki:** Improve Unix instructions ([c3e5487](https://www.github.com/nodejs/node-gyp/commit/c3e548736645b535ea5bce613d74ca3e98598243)) +* **wiki:** link to docs/ from README ([b52e487](https://www.github.com/nodejs/node-gyp/commit/b52e487eac1eb421573d1e67114a242eeff45a00)) +* **wiki:** Lower case L ([3aa2c6b](https://www.github.com/nodejs/node-gyp/commit/3aa2c6bdb07971b87505e32e32548d75264bd19f)) +* **wiki:** Make changes discussed in https://github.com/nodejs/node-gyp/issues/2416 ([1dcad87](https://www.github.com/nodejs/node-gyp/commit/1dcad873539027511a5f0243baf770ea90f6f4e2)) +* **wiki:** move wiki docs into doc/ ([f0a4835](https://www.github.com/nodejs/node-gyp/commit/f0a48355d86534ec3bdabcdb3ce3340fa2e17f39)) +* **wiki:** node-sass in the wild ([d310a73](https://www.github.com/nodejs/node-gyp/commit/d310a73d64d0065050377baac7047472f7424a1b)) +* **wiki:** node-srs was a 404 ([bbca21a](https://www.github.com/nodejs/node-gyp/commit/bbca21a1e1ede4c473aff365ca71989a5bda7b57)) +* **wiki:** Note: VS2010 seems to be no longer available! VS2013 or nothing! ([7b5dcaf](https://www.github.com/nodejs/node-gyp/commit/7b5dcafafccdceae4b8f2b53ac9081a694b6ade8)) +* **wiki:** safer doc names, remove unnecessary TypedArray doc ([161c235](https://www.github.com/nodejs/node-gyp/commit/161c2353ef5b562f4acfb2fd77608fcbd0800fc0)) +* **wiki:** sorry, forgot to mention a specific windows version. ([d69dffc](https://www.github.com/nodejs/node-gyp/commit/d69dffc16c2b1e3c60dcb5d1c35a49270ba22a35)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([7444b47](https://www.github.com/nodejs/node-gyp/commit/7444b47a7caac1e14d1da474a7fcfcf88d328017)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([d766b74](https://www.github.com/nodejs/node-gyp/commit/d766b7427851e6c2edc02e2504a7be9be7e330c0)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([d319b0e](https://www.github.com/nodejs/node-gyp/commit/d319b0e98c7085de8e51bc5595eba4264b99a7d5)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([3c6692d](https://www.github.com/nodejs/node-gyp/commit/3c6692d538f0ce973869aa237118b7d2483feccd)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([93392d5](https://www.github.com/nodejs/node-gyp/commit/93392d559ce6f250b9c7fe8177e6c88603809dc1)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([8841158](https://www.github.com/nodejs/node-gyp/commit/88411588f300e9b7c00fe516ecd977a1feeeb15c)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([81bfa1f](https://www.github.com/nodejs/node-gyp/commit/81bfa1f1b63d522a9f8a9ae9ca0c7ae90fe75140)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([d1cd237](https://www.github.com/nodejs/node-gyp/commit/d1cd237bad06fa507adb354b9e2181a14dc63d24)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([3de9e17](https://www.github.com/nodejs/node-gyp/commit/3de9e17e0b8a387eafe7bd18d0ec1e3191d118e8)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([a9b7096](https://www.github.com/nodejs/node-gyp/commit/a9b70968fb956eab3b95672048b94350e1565ca3)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([3236069](https://www.github.com/nodejs/node-gyp/commit/3236069689e7e0eb15b324fce74ab58158956f98)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([1462755](https://www.github.com/nodejs/node-gyp/commit/14627556966e5d513bdb8e5208f0e1300f68991f)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([7ab1337](https://www.github.com/nodejs/node-gyp/commit/7ab133752a6c402bb96dcd3d671d73e03e9487ad)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([640895d](https://www.github.com/nodejs/node-gyp/commit/640895d36b7448c646a3b850c1e159106f83c724)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([ced8c96](https://www.github.com/nodejs/node-gyp/commit/ced8c968457f285ab8989c291d28173d7730833c)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([27b883a](https://www.github.com/nodejs/node-gyp/commit/27b883a350ad0db6b9130d7b996f35855ec34c7a)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([d29fb13](https://www.github.com/nodejs/node-gyp/commit/d29fb134f1c4b9dd729ba95f2979e69e0934809f)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([2765891](https://www.github.com/nodejs/node-gyp/commit/27658913e6220cf0371b4b73e25a0e4ab11108a1)) +* **wiki:** Updated "binding.gyp" files out in the wild (markdown) ([dc97766](https://www.github.com/nodejs/node-gyp/commit/dc9776648d432bca6775c176641f16da14522d4c)) +* **wiki:** Updated Error: "pre" versions of node cannot be installed (markdown) ([e9f8b33](https://www.github.com/nodejs/node-gyp/commit/e9f8b33d1f87d04f22cb09a814d7c55d0fa38446)) +* **wiki:** Updated Home (markdown) ([3407109](https://www.github.com/nodejs/node-gyp/commit/3407109325cf7ba1e925656b9eb75feffab0557c)) +* **wiki:** Updated Home (markdown) ([6e392bc](https://www.github.com/nodejs/node-gyp/commit/6e392bcdd3dd1691773e6e16e1dffc35931b81e0)) +* **wiki:** Updated Home (markdown) ([65efe32](https://www.github.com/nodejs/node-gyp/commit/65efe32ccb8d446ce569453364f922dd9d27c945)) +* **wiki:** Updated Home (markdown) ([ea28f09](https://www.github.com/nodejs/node-gyp/commit/ea28f0947af91fa638be355143f5df89d2e431c8)) +* **wiki:** Updated Home (markdown) ([0e37ff4](https://www.github.com/nodejs/node-gyp/commit/0e37ff48b306c12149661b375895741d3d710da7)) +* **wiki:** Updated Home (markdown) ([b398ef4](https://www.github.com/nodejs/node-gyp/commit/b398ef46f660d2b1506508550dadfb4c35639e4b)) +* **wiki:** Updated Linking to OpenSSL (markdown) ([8919028](https://www.github.com/nodejs/node-gyp/commit/8919028921fd304f08044098434f0dc6071fb7cf)) +* **wiki:** Updated Linking to OpenSSL (markdown) ([c00eb77](https://www.github.com/nodejs/node-gyp/commit/c00eb778fc7dc27e4dab3a9219035ea20458b33b)) +* **wiki:** Updated node-levelup to node-leveldown (broken links) ([59668bb](https://www.github.com/nodejs/node-gyp/commit/59668bb0b904feccf3c09afa2fd37378c77af967)) +* **wiki:** Updated Updating npm's bundled node gyp (markdown) ([d314854](https://www.github.com/nodejs/node-gyp/commit/d31485415ef69d46effa6090c95698341965de1b)) +* **wiki:** Updated Updating npm's bundled node gyp (markdown) ([11858b0](https://www.github.com/nodejs/node-gyp/commit/11858b0655d1eee00c62ad628e719d4378803d14)) +* **wiki:** Updated Updating npm's bundled node gyp (markdown) ([33561e9](https://www.github.com/nodejs/node-gyp/commit/33561e9cbf5f4eb46111318503c77df2c6eb484a)) +* **wiki:** Updated Updating npm's bundled node gyp (markdown) ([4a7f2d0](https://www.github.com/nodejs/node-gyp/commit/4a7f2d0d869a65c99a78504976567017edadf657)) +* **wiki:** Updated Updating npm's bundled node gyp (markdown) ([979a706](https://www.github.com/nodejs/node-gyp/commit/979a7063b950c088a7f4896fc3a48e1d00dfd231)) +* **wiki:** Updated Updating npm's bundled node gyp (markdown) ([e50e04d](https://www.github.com/nodejs/node-gyp/commit/e50e04d7b6a3754ea0aa11fe8cef491b3bc5bdd4)) + ## [8.1.0](https://www.github.com/nodejs/node-gyp/compare/v8.0.0...v8.1.0) (2021-05-28) diff --git a/package.json b/package.json index 596e60f914..ec5c3c5529 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "8.1.0", + "version": "8.2.0", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From f2ad87ff65f98ad66daa7225ad59d99b759a2b07 Mon Sep 17 00:00:00 2001 From: Cheng Zhao Date: Mon, 6 Sep 2021 11:18:24 +0900 Subject: [PATCH 307/551] chore: refactor the creation of config.gypi file --- .gitignore | 1 + lib/configure.js | 102 ++------------------------- lib/create-config-gypi.js | 119 ++++++++++++++++++++++++++++++++ test/test-create-config-gypi.js | 37 ++++++++++ 4 files changed, 162 insertions(+), 97 deletions(-) create mode 100644 lib/create-config-gypi.js create mode 100644 test/test-create-config-gypi.js diff --git a/.gitignore b/.gitignore index 776066e34d..e906ca7280 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ +*.swp gyp/test node_modules test/.node-gyp diff --git a/lib/configure.js b/lib/configure.js index 038ccbf20f..682f1e6f95 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -7,6 +7,7 @@ const os = require('os') const processRelease = require('./process-release') const win = process.platform === 'win32' const findNodeDirectory = require('./find-node-directory') +const createConfigGypi = require('./create-config-gypi') const msgFormat = require('util').format var findPython = require('./find-python') if (win) { @@ -92,107 +93,14 @@ function configure (gyp, argv, callback) { if (err) { return callback(err) } - - var configFilename = 'config.gypi' - var configPath = path.resolve(buildDir, configFilename) - - log.verbose('build/' + configFilename, 'creating config file') - - var config = process.config ? JSON.parse(JSON.stringify(process.config)) : {} - var defaults = config.target_defaults - var variables = config.variables - - // default "config.variables" - if (!variables) { - variables = config.variables = {} - } - - // default "config.defaults" - if (!defaults) { - defaults = config.target_defaults = {} - } - - // don't inherit the "defaults" from node's `process.config` object. - // doing so could cause problems in cases where the `node` executable was - // compiled on a different machine (with different lib/include paths) than - // the machine where the addon is being built to - defaults.cflags = [] - defaults.defines = [] - defaults.include_dirs = [] - defaults.libraries = [] - - // set the default_configuration prop - if ('debug' in gyp.opts) { - defaults.default_configuration = gyp.opts.debug ? 'Debug' : 'Release' - } - - if (!defaults.default_configuration) { - defaults.default_configuration = 'Release' - } - - // set the target_arch variable - variables.target_arch = gyp.opts.arch || process.arch || 'ia32' - if (variables.target_arch === 'arm64') { - defaults.msvs_configuration_platform = 'ARM64' - defaults.xcode_configuration_platform = 'arm64' - } - - // set the node development directory - variables.nodedir = nodeDir - - // disable -T "thin" static archives by default - variables.standalone_static_library = gyp.opts.thin ? 0 : 1 - - if (win) { + if (process.platform === 'win32') { process.env.GYP_MSVS_VERSION = Math.min(vsInfo.versionYear, 2015) process.env.GYP_MSVS_OVERRIDE_PATH = vsInfo.path - defaults.msbuild_toolset = vsInfo.toolset - if (vsInfo.sdk) { - defaults.msvs_windows_target_platform_version = vsInfo.sdk - } - if (variables.target_arch === 'arm64') { - if (vsInfo.versionMajor > 15 || - (vsInfo.versionMajor === 15 && vsInfo.versionMajor >= 9)) { - defaults.msvs_enable_marmasm = 1 - } else { - log.warn('Compiling ARM64 assembly is only available in\n' + - 'Visual Studio 2017 version 15.9 and above') - } - } - variables.msbuild_path = vsInfo.msBuild } - - // loop through the rest of the opts and add the unknown ones as variables. - // this allows for module-specific configure flags like: - // - // $ node-gyp configure --shared-libxml2 - Object.keys(gyp.opts).forEach(function (opt) { - if (opt === 'argv') { - return - } - if (opt in gyp.configDefs) { - return - } - variables[opt.replace(/-/g, '_')] = gyp.opts[opt] + createConfigGypi({ gyp, buildDir, nodeDir, vsInfo }, (err, configPath) => { + configs.push(configPath) + findConfigs(err) }) - - // ensures that any boolean values from `process.config` get stringified - function boolsToString (k, v) { - if (typeof v === 'boolean') { - return String(v) - } - return v - } - - log.silly('build/' + configFilename, config) - - // now write out the config.gypi file to the build/ dir - var prefix = '# Do not edit. File was generated by node-gyp\'s "configure" step' - - var json = JSON.stringify(config, boolsToString, 2) - log.verbose('build/' + configFilename, 'writing out config file: %s', configPath) - configs.push(configPath) - fs.writeFile(configPath, [prefix, json, ''].join('\n'), findConfigs) } function findConfigs (err) { diff --git a/lib/create-config-gypi.js b/lib/create-config-gypi.js new file mode 100644 index 0000000000..4d137e6b59 --- /dev/null +++ b/lib/create-config-gypi.js @@ -0,0 +1,119 @@ +'use strict' + +const fs = require('graceful-fs') +const log = require('npmlog') +const path = require('path') + +function getBaseConfigGypi () { + const config = JSON.parse(JSON.stringify(process.config)) + if (!config.target_defaults) { + config.target_defaults = {} + } + if (!config.variables) { + config.variables = {} + } + return config +} + +function getCurrentConfigGypi ({ gyp, nodeDir, vsInfo }) { + const config = getBaseConfigGypi() + const defaults = config.target_defaults + const variables = config.variables + + // don't inherit the "defaults" from the base config.gypi. + // doing so could cause problems in cases where the `node` executable was + // compiled on a different machine (with different lib/include paths) than + // the machine where the addon is being built to + defaults.cflags = [] + defaults.defines = [] + defaults.include_dirs = [] + defaults.libraries = [] + + // set the default_configuration prop + if ('debug' in gyp.opts) { + defaults.default_configuration = gyp.opts.debug ? 'Debug' : 'Release' + } + + if (!defaults.default_configuration) { + defaults.default_configuration = 'Release' + } + + // set the target_arch variable + variables.target_arch = gyp.opts.arch || process.arch || 'ia32' + if (variables.target_arch === 'arm64') { + defaults.msvs_configuration_platform = 'ARM64' + defaults.xcode_configuration_platform = 'arm64' + } + + // set the node development directory + variables.nodedir = nodeDir + + // disable -T "thin" static archives by default + variables.standalone_static_library = gyp.opts.thin ? 0 : 1 + + if (process.platform === 'win32') { + defaults.msbuild_toolset = vsInfo.toolset + if (vsInfo.sdk) { + defaults.msvs_windows_target_platform_version = vsInfo.sdk + } + if (variables.target_arch === 'arm64') { + if (vsInfo.versionMajor > 15 || + (vsInfo.versionMajor === 15 && vsInfo.versionMajor >= 9)) { + defaults.msvs_enable_marmasm = 1 + } else { + log.warn('Compiling ARM64 assembly is only available in\n' + + 'Visual Studio 2017 version 15.9 and above') + } + } + variables.msbuild_path = vsInfo.msBuild + } + + // loop through the rest of the opts and add the unknown ones as variables. + // this allows for module-specific configure flags like: + // + // $ node-gyp configure --shared-libxml2 + Object.keys(gyp.opts).forEach(function (opt) { + if (opt === 'argv') { + return + } + if (opt in gyp.configDefs) { + return + } + variables[opt.replace(/-/g, '_')] = gyp.opts[opt] + }) + + return config +} + +function createConfigGypi ({ gyp, buildDir, nodeDir, vsInfo }, callback) { + const configFilename = 'config.gypi' + const configPath = path.resolve(buildDir, configFilename) + + log.verbose('build/' + configFilename, 'creating config file') + + const config = getCurrentConfigGypi({ gyp, nodeDir, vsInfo }) + + // ensures that any boolean values in config.gypi get stringified + function boolsToString (k, v) { + if (typeof v === 'boolean') { + return String(v) + } + return v + } + + log.silly('build/' + configFilename, config) + + // now write out the config.gypi file to the build/ dir + const prefix = '# Do not edit. File was generated by node-gyp\'s "configure" step' + + const json = JSON.stringify(config, boolsToString, 2) + log.verbose('build/' + configFilename, 'writing out config file: %s', configPath) + fs.writeFile(configPath, [prefix, json, ''].join('\n'), (err) => { + callback(err, configPath) + }) +} + +module.exports = createConfigGypi +module.exports.test = { + getCurrentConfigGypi: getCurrentConfigGypi +} diff --git a/test/test-create-config-gypi.js b/test/test-create-config-gypi.js new file mode 100644 index 0000000000..933cae1326 --- /dev/null +++ b/test/test-create-config-gypi.js @@ -0,0 +1,37 @@ +'use strict' + +const { test } = require('tap') +const gyp = require('../lib/node-gyp') +const createConfigGypi = require('../lib/create-config-gypi') +const { getCurrentConfigGypi } = createConfigGypi.test + +test('config.gypi with no options', function (t) { + t.plan(2) + + const prog = gyp() + prog.parseArgv([]) + + const config = getCurrentConfigGypi({ gyp: prog, vsInfo: {} }) + t.equal(config.target_defaults.default_configuration, 'Release') + t.equal(config.variables.target_arch, process.arch) +}) + +test('config.gypi with --debug', function (t) { + t.plan(1) + + const prog = gyp() + prog.parseArgv(['_', '_', '--debug']) + + const config = getCurrentConfigGypi({ gyp: prog, vsInfo: {} }) + t.equal(config.target_defaults.default_configuration, 'Debug') +}) + +test('config.gypi with custom options', function (t) { + t.plan(1) + + const prog = gyp() + prog.parseArgv(['_', '_', '--shared-libxml2']) + + const config = getCurrentConfigGypi({ gyp: prog, vsInfo: {} }) + t.equal(config.variables.shared_libxml2, true) +}) From 0a67dcd1307f3560495219253241eafcbf4e2a69 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Tue, 5 Oct 2021 12:20:29 +0200 Subject: [PATCH 308/551] test: Python 3.10 was release on Oct. 4th (#2504) --- .github/workflows/tests.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 7b12268023..f7c9b979e3 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -9,7 +9,7 @@ jobs: max-parallel: 15 matrix: node: [12.x, 14.x, 16.x] - python: [3.6, 3.8, 3.9] + python: ["3.6", "3.8", "3.10"] os: [macos-latest, ubuntu-latest, windows-latest] runs-on: ${{ matrix.os }} steps: From b05b4fe9891f718f40edf547e9b50e982826d48a Mon Sep 17 00:00:00 2001 From: Gar Date: Tue, 5 Oct 2021 13:31:22 -0700 Subject: [PATCH 309/551] chore(deps): bump make-fetch-happen from 8.0.14 to 9.1.0 The breaking change in this module was a cache parameter that `node-gyp` is not using, so this module is not affected. --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index ec5c3c5529..bbaf6419b6 100644 --- a/package.json +++ b/package.json @@ -25,7 +25,7 @@ "env-paths": "^2.2.0", "glob": "^7.1.4", "graceful-fs": "^4.2.6", - "make-fetch-happen": "^8.0.14", + "make-fetch-happen": "^9.1.0", "nopt": "^5.0.0", "npmlog": "^4.1.2", "rimraf": "^3.0.2", From 5585792922a97f0629f143c560efd74470eae87f Mon Sep 17 00:00:00 2001 From: Jiawen Geng Date: Mon, 11 Oct 2021 18:35:15 +0800 Subject: [PATCH 310/551] feat(gyp): update gyp to v0.10.0 (#2521) --- gyp/CHANGELOG.md | 7 +++++++ gyp/pylib/gyp/MSVSVersion.py | 17 ++++++++++++++++- gyp/setup.py | 2 +- 3 files changed, 24 insertions(+), 2 deletions(-) diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md index f5e9f63476..b7d55ed655 100644 --- a/gyp/CHANGELOG.md +++ b/gyp/CHANGELOG.md @@ -1,5 +1,12 @@ # Changelog +## [0.10.0](https://www.github.com/nodejs/gyp-next/compare/v0.9.6...v0.10.0) (2021-08-26) + + +### Features + +* **msvs:** add support for Visual Studio 2022 ([#124](https://www.github.com/nodejs/gyp-next/issues/124)) ([4bd9215](https://www.github.com/nodejs/gyp-next/commit/4bd9215c44d300f06e916aec1d6327c22b78272d)) + ### [0.9.6](https://www.github.com/nodejs/gyp-next/compare/v0.9.5...v0.9.6) (2021-08-23) diff --git a/gyp/pylib/gyp/MSVSVersion.py b/gyp/pylib/gyp/MSVSVersion.py index 134b35557b..8d7f21e82d 100644 --- a/gyp/pylib/gyp/MSVSVersion.py +++ b/gyp/pylib/gyp/MSVSVersion.py @@ -269,6 +269,18 @@ def _CreateVersion(name, path, sdk_based=False): if path: path = os.path.normpath(path) versions = { + "2022": VisualStudioVersion( + "2022", + "Visual Studio 2022", + solution_version="12.00", + project_version="17.0", + flat_sln=False, + uses_vcxproj=True, + path=path, + sdk_based=sdk_based, + default_toolset="v143", + compatible_sdks=["v8.1", "v10.0"], + ), "2019": VisualStudioVersion( "2019", "Visual Studio 2019", @@ -436,6 +448,7 @@ def _DetectVisualStudioVersions(versions_to_check, force_express): 2015 - Visual Studio 2015 (14) 2017 - Visual Studio 2017 (15) 2019 - Visual Studio 2019 (16) + 2022 - Visual Studio 2022 (17) Where (e) is e for express editions of MSVS and blank otherwise. """ version_to_year = { @@ -447,6 +460,7 @@ def _DetectVisualStudioVersions(versions_to_check, force_express): "14.0": "2015", "15.0": "2017", "16.0": "2019", + "17.0": "2022", } versions = [] for version in versions_to_check: @@ -522,7 +536,7 @@ def SelectVisualStudioVersion(version="auto", allow_fallback=True): if version == "auto": version = os.environ.get("GYP_MSVS_VERSION", "auto") version_map = { - "auto": ("16.0", "15.0", "14.0", "12.0", "10.0", "9.0", "8.0", "11.0"), + "auto": ("17.0", "16.0", "15.0", "14.0", "12.0", "10.0", "9.0", "8.0", "11.0"), "2005": ("8.0",), "2005e": ("8.0",), "2008": ("9.0",), @@ -536,6 +550,7 @@ def SelectVisualStudioVersion(version="auto", allow_fallback=True): "2015": ("14.0",), "2017": ("15.0",), "2019": ("16.0",), + "2022": ("17.0",), } override_path = os.environ.get("GYP_MSVS_OVERRIDE_PATH") if override_path: diff --git a/gyp/setup.py b/gyp/setup.py index 0ce46123cc..cf9d7d2e56 100644 --- a/gyp/setup.py +++ b/gyp/setup.py @@ -15,7 +15,7 @@ setup( name="gyp-next", - version="0.9.6", + version="0.10.0", description="A fork of the GYP build system for use in the Node.js projects", long_description=long_description, long_description_content_type="text/markdown", From fb85fb21c4bcba806cca852f6f076108aaf7ef4d Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Mon, 11 Oct 2021 10:36:17 +0000 Subject: [PATCH 311/551] chore: release 8.3.0 --- CHANGELOG.md | 18 ++++++++++++++++++ package.json | 2 +- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d513542fd..ebe8c321ce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,23 @@ # Changelog +## [8.3.0](https://www.github.com/nodejs/node-gyp/compare/v8.2.0...v8.3.0) (2021-10-11) + + +### Features + +* **gyp:** update gyp to v0.10.0 ([#2521](https://www.github.com/nodejs/node-gyp/issues/2521)) ([5585792](https://www.github.com/nodejs/node-gyp/commit/5585792922a97f0629f143c560efd74470eae87f)) + + +### Tests + +* Python 3.10 was release on Oct. 4th ([#2504](https://www.github.com/nodejs/node-gyp/issues/2504)) ([0a67dcd](https://www.github.com/nodejs/node-gyp/commit/0a67dcd1307f3560495219253241eafcbf4e2a69)) + + +### Miscellaneous + +* **deps:** bump make-fetch-happen from 8.0.14 to 9.1.0 ([b05b4fe](https://www.github.com/nodejs/node-gyp/commit/b05b4fe9891f718f40edf547e9b50e982826d48a)) +* refactor the creation of config.gypi file ([f2ad87f](https://www.github.com/nodejs/node-gyp/commit/f2ad87ff65f98ad66daa7225ad59d99b759a2b07)) + ## [8.2.0](https://www.github.com/nodejs/node-gyp/compare/v8.1.0...v8.2.0) (2021-08-23) diff --git a/package.json b/package.json index bbaf6419b6..cf9af6a0ec 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "8.2.0", + "version": "8.3.0", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From 5a00387e5f8018264a1822f6c4d5dbf425f21cf6 Mon Sep 17 00:00:00 2001 From: Jiawen Geng Date: Fri, 29 Oct 2021 10:53:59 +0800 Subject: [PATCH 312/551] feat: support vs2022 (#2533) --- .github/workflows/visual-studio.yml | 25 +++++++++++++++++++++++++ lib/find-visualstudio.js | 14 +++++++++++++- 2 files changed, 38 insertions(+), 1 deletion(-) create mode 100644 .github/workflows/visual-studio.yml diff --git a/.github/workflows/visual-studio.yml b/.github/workflows/visual-studio.yml new file mode 100644 index 0000000000..6bb4574d63 --- /dev/null +++ b/.github/workflows/visual-studio.yml @@ -0,0 +1,25 @@ +name: Tests on Windows +on: [push, pull_request] +jobs: + Tests: + strategy: + fail-fast: false + max-parallel: 15 + matrix: + os: [windows-2022] + runs-on: ${{ matrix.os }} + steps: + - name: Checkout Repository + uses: actions/checkout@v2 + - name: Install Dependencies + run: | + npm install --no-progress + - name: Set Windows environment + if: matrix.os == 'windows-latest' + run: | + echo 'GYP_MSVS_VERSION=2015' >> $Env:GITHUB_ENV + echo 'GYP_MSVS_OVERRIDE_PATH=C:\\Dummy' >> $Env:GITHUB_ENV + - name: Environment Information + run: npx envinfo + - name: Run Node tests + run: npm test diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js index f2cce327e7..64af7be346 100644 --- a/lib/find-visualstudio.js +++ b/lib/find-visualstudio.js @@ -2,6 +2,7 @@ const log = require('npmlog') const execFile = require('child_process').execFile +const fs = require('fs') const path = require('path').win32 const logWithPrefix = require('./util').logWithPrefix const regSearchKeys = require('./util').regSearchKeys @@ -257,6 +258,10 @@ VisualStudioFinder.prototype = { ret.versionYear = 2019 return ret } + if (ret.versionMajor === 17) { + ret.versionYear = 2022 + return ret + } this.log.silly('- unsupported version:', ret.versionMajor) return {} }, @@ -264,15 +269,20 @@ VisualStudioFinder.prototype = { // Helper - process MSBuild information getMSBuild: function getMSBuild (info, versionYear) { const pkg = 'Microsoft.VisualStudio.VC.MSBuild.Base' + const msbuildPath = path.join(info.path, 'MSBuild', 'Current', 'Bin', 'MSBuild.exe') if (info.packages.indexOf(pkg) !== -1) { this.log.silly('- found VC.MSBuild.Base') if (versionYear === 2017) { return path.join(info.path, 'MSBuild', '15.0', 'Bin', 'MSBuild.exe') } if (versionYear === 2019) { - return path.join(info.path, 'MSBuild', 'Current', 'Bin', 'MSBuild.exe') + return msbuildPath } } + // visual studio 2022 don't has msbuild pkg + if (fs.existsSync(msbuildPath)) { + return msbuildPath + } return null }, @@ -293,6 +303,8 @@ VisualStudioFinder.prototype = { return 'v141' } else if (versionYear === 2019) { return 'v142' + } else if (versionYear === 2022) { + return 'v143' } this.log.silly('- invalid versionYear:', versionYear) return null From a27dc08696911c6d81e76cc228697243069103c1 Mon Sep 17 00:00:00 2001 From: Cheng Zhao Date: Tue, 14 Sep 2021 15:00:49 +0900 Subject: [PATCH 313/551] feat: build with config.gypi from node headers --- README.md | 17 +++++++ lib/configure.js | 12 ++--- lib/create-config-gypi.js | 49 ++++++++++++++----- lib/node-gyp.js | 3 +- .../fixtures/nodedir/include/node/config.gypi | 6 +++ test/test-configure-python.js | 5 +- test/test-create-config-gypi.js | 47 +++++++++++++++--- 7 files changed, 112 insertions(+), 27 deletions(-) create mode 100644 test/fixtures/nodedir/include/node/config.gypi diff --git a/README.md b/README.md index 11ba7ab693..71eea0a942 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,22 @@ Python executable, it will be used instead of any of the other configured or builtin Python search paths. If it's not a compatible version, no further searching will be done. +### Build for Third Party Node.js Runtimes + +When building modules for thid party Node.js runtimes like Electron, which have +different build configurations from the official Node.js distribution, you +should use `--dist-url` or `--nodedir` flags to specify the headers of the +runtime to build for. + +Also when `--dist-url` or `--nodedir` flags are passed, node-gyp will use the +`config.gypi` shipped in the headers distribution to generate build +configurations, which is different from the default mode that would use the +`process.config` object of the running Node.js instance. + +Some old versions of Electron shipped malformed `config.gypi` in their headers +distributions, and you might need to pass `--force-process-config` to node-gyp +to work around configuration errors. + ## How to Use To compile your native addon, first go to its root directory: @@ -198,6 +214,7 @@ Some additional resources for Node.js native addons and writing `gyp` configurat | `--python=$path` | Set path to the Python binary | `--msvs_version=$version` | Set Visual Studio version (Windows only) | `--solution=$solution` | Set Visual Studio Solution version (Windows only) +| `--force-process-config` | Force using runtime's `process.config` object to generate `config.gypi` file ## Configuration diff --git a/lib/configure.js b/lib/configure.js index 682f1e6f95..d9b8fe340e 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -97,17 +97,15 @@ function configure (gyp, argv, callback) { process.env.GYP_MSVS_VERSION = Math.min(vsInfo.versionYear, 2015) process.env.GYP_MSVS_OVERRIDE_PATH = vsInfo.path } - createConfigGypi({ gyp, buildDir, nodeDir, vsInfo }, (err, configPath) => { + createConfigGypi({ gyp, buildDir, nodeDir, vsInfo }).then(configPath => { configs.push(configPath) - findConfigs(err) + findConfigs() + }).catch(err => { + callback(err) }) } - function findConfigs (err) { - if (err) { - return callback(err) - } - + function findConfigs () { var name = configNames.shift() if (!name) { return runGyp() diff --git a/lib/create-config-gypi.js b/lib/create-config-gypi.js index 4d137e6b59..dbcb8b485c 100644 --- a/lib/create-config-gypi.js +++ b/lib/create-config-gypi.js @@ -4,19 +4,45 @@ const fs = require('graceful-fs') const log = require('npmlog') const path = require('path') -function getBaseConfigGypi () { - const config = JSON.parse(JSON.stringify(process.config)) +function parseConfigGypi (config) { + // translated from tools/js2c.py of Node.js + // 1. string comments + config = config.replace(/#.*/g, '') + // 2. join multiline strings + config = config.replace(/'$\s+'/mg, '') + // 3. normalize string literals from ' into " + config = config.replace(/'/g, '"') + return JSON.parse(config) +} + +async function getBaseConfigGypi ({ gyp, nodeDir }) { + // try reading $nodeDir/include/node/config.gypi first when: + // 1. --dist-url or --nodedir is specified + // 2. and --force-process-config is not specified + const shouldReadConfigGypi = (gyp.opts.nodedir || gyp.opts['dist-url']) && !gyp.opts['force-process-config'] + if (shouldReadConfigGypi && nodeDir) { + try { + const baseConfigGypiPath = path.resolve(nodeDir, 'include/node/config.gypi') + const baseConfigGypi = await fs.promises.readFile(baseConfigGypiPath) + return parseConfigGypi(baseConfigGypi.toString()) + } catch (err) { + log.warn('read config.gypi', err.message) + } + } + + // fallback to process.config if it is invalid + return JSON.parse(JSON.stringify(process.config)) +} + +async function getCurrentConfigGypi ({ gyp, nodeDir, vsInfo }) { + const config = await getBaseConfigGypi({ gyp, nodeDir }) if (!config.target_defaults) { config.target_defaults = {} } if (!config.variables) { config.variables = {} } - return config -} -function getCurrentConfigGypi ({ gyp, nodeDir, vsInfo }) { - const config = getBaseConfigGypi() const defaults = config.target_defaults const variables = config.variables @@ -85,13 +111,13 @@ function getCurrentConfigGypi ({ gyp, nodeDir, vsInfo }) { return config } -function createConfigGypi ({ gyp, buildDir, nodeDir, vsInfo }, callback) { +async function createConfigGypi ({ gyp, buildDir, nodeDir, vsInfo }) { const configFilename = 'config.gypi' const configPath = path.resolve(buildDir, configFilename) log.verbose('build/' + configFilename, 'creating config file') - const config = getCurrentConfigGypi({ gyp, nodeDir, vsInfo }) + const config = await getCurrentConfigGypi({ gyp, nodeDir, vsInfo }) // ensures that any boolean values in config.gypi get stringified function boolsToString (k, v) { @@ -108,12 +134,13 @@ function createConfigGypi ({ gyp, buildDir, nodeDir, vsInfo }, callback) { const json = JSON.stringify(config, boolsToString, 2) log.verbose('build/' + configFilename, 'writing out config file: %s', configPath) - fs.writeFile(configPath, [prefix, json, ''].join('\n'), (err) => { - callback(err, configPath) - }) + await fs.promises.writeFile(configPath, [prefix, json, ''].join('\n')) + + return configPath } module.exports = createConfigGypi module.exports.test = { + parseConfigGypi: parseConfigGypi, getCurrentConfigGypi: getCurrentConfigGypi } diff --git a/lib/node-gyp.js b/lib/node-gyp.js index 81fc590919..0f11185641 100644 --- a/lib/node-gyp.js +++ b/lib/node-gyp.js @@ -75,7 +75,8 @@ proto.configDefs = { 'dist-url': String, // 'install' tarball: String, // 'install' jobs: String, // 'build' - thin: String // 'configure' + thin: String, // 'configure' + 'force-process-config': Boolean // 'configure' } /** diff --git a/test/fixtures/nodedir/include/node/config.gypi b/test/fixtures/nodedir/include/node/config.gypi new file mode 100644 index 0000000000..e767534082 --- /dev/null +++ b/test/fixtures/nodedir/include/node/config.gypi @@ -0,0 +1,6 @@ +# Test configuration +{ + 'variables': { + 'build_with_electron': true + } +} diff --git a/test/test-configure-python.js b/test/test-configure-python.js index ac25f7972e..4290e7af1b 100644 --- a/test/test-configure-python.js +++ b/test/test-configure-python.js @@ -11,7 +11,10 @@ const configure = requireInject('../lib/configure', { closeSync: function () { }, writeFile: function (file, data, cb) { cb() }, stat: function (file, cb) { cb(null, {}) }, - mkdir: function (dir, options, cb) { cb() } + mkdir: function (dir, options, cb) { cb() }, + promises: { + writeFile: function (file, data) { return Promise.resolve(null) } + } } }) diff --git a/test/test-create-config-gypi.js b/test/test-create-config-gypi.js index 933cae1326..eeac73fab1 100644 --- a/test/test-create-config-gypi.js +++ b/test/test-create-config-gypi.js @@ -1,37 +1,70 @@ 'use strict' +const path = require('path') const { test } = require('tap') const gyp = require('../lib/node-gyp') const createConfigGypi = require('../lib/create-config-gypi') -const { getCurrentConfigGypi } = createConfigGypi.test +const { parseConfigGypi, getCurrentConfigGypi } = createConfigGypi.test -test('config.gypi with no options', function (t) { +test('config.gypi with no options', async function (t) { t.plan(2) const prog = gyp() prog.parseArgv([]) - const config = getCurrentConfigGypi({ gyp: prog, vsInfo: {} }) + const config = await getCurrentConfigGypi({ gyp: prog, vsInfo: {} }) t.equal(config.target_defaults.default_configuration, 'Release') t.equal(config.variables.target_arch, process.arch) }) -test('config.gypi with --debug', function (t) { +test('config.gypi with --debug', async function (t) { t.plan(1) const prog = gyp() prog.parseArgv(['_', '_', '--debug']) - const config = getCurrentConfigGypi({ gyp: prog, vsInfo: {} }) + const config = await getCurrentConfigGypi({ gyp: prog, vsInfo: {} }) t.equal(config.target_defaults.default_configuration, 'Debug') }) -test('config.gypi with custom options', function (t) { +test('config.gypi with custom options', async function (t) { t.plan(1) const prog = gyp() prog.parseArgv(['_', '_', '--shared-libxml2']) - const config = getCurrentConfigGypi({ gyp: prog, vsInfo: {} }) + const config = await getCurrentConfigGypi({ gyp: prog, vsInfo: {} }) t.equal(config.variables.shared_libxml2, true) }) + +test('config.gypi with nodedir', async function (t) { + t.plan(1) + + const nodeDir = path.join(__dirname, 'fixtures', 'nodedir') + + const prog = gyp() + prog.parseArgv(['_', '_', `--nodedir=${nodeDir}`]) + + const config = await getCurrentConfigGypi({ gyp: prog, nodeDir, vsInfo: {} }) + t.equal(config.variables.build_with_electron, true) +}) + +test('config.gypi with --force-process-config', async function (t) { + t.plan(1) + + const nodeDir = path.join(__dirname, 'fixtures', 'nodedir') + + const prog = gyp() + prog.parseArgv(['_', '_', '--force-process-config', `--nodedir=${nodeDir}`]) + + const config = await getCurrentConfigGypi({ gyp: prog, nodeDir, vsInfo: {} }) + t.equal(config.variables.build_with_electron, undefined) +}) + +test('config.gypi parsing', function (t) { + t.plan(1) + + const str = "# Some comments\n{'variables': {'multiline': 'A'\n'B'}}" + const config = parseConfigGypi(str) + t.deepEqual(config, { variables: { multiline: 'AB' } }) +}) From 7073c65f61d2b5b3a4aff3370be430849b9bd0b3 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri, 5 Nov 2021 09:09:29 +0000 Subject: [PATCH 314/551] chore: release 8.4.0 --- CHANGELOG.md | 8 ++++++++ package.json | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ebe8c321ce..e5d1a4d065 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # Changelog +## [8.4.0](https://www.github.com/nodejs/node-gyp/compare/v8.3.0...v8.4.0) (2021-11-05) + + +### Features + +* build with config.gypi from node headers ([a27dc08](https://www.github.com/nodejs/node-gyp/commit/a27dc08696911c6d81e76cc228697243069103c1)) +* support vs2022 ([#2533](https://www.github.com/nodejs/node-gyp/issues/2533)) ([5a00387](https://www.github.com/nodejs/node-gyp/commit/5a00387e5f8018264a1822f6c4d5dbf425f21cf6)) + ## [8.3.0](https://www.github.com/nodejs/node-gyp/compare/v8.2.0...v8.3.0) (2021-10-11) diff --git a/package.json b/package.json index cf9af6a0ec..ea06551817 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "8.3.0", + "version": "8.4.0", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From 787cf7f8e5ddd5039e02b64ace6b7b15e06fe0a4 Mon Sep 17 00:00:00 2001 From: csett86 Date: Fri, 5 Nov 2021 22:24:05 +0100 Subject: [PATCH 315/551] docs: fix typo in powershell node-gyp update --- docs/Updating-npm-bundled-node-gyp.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Updating-npm-bundled-node-gyp.md b/docs/Updating-npm-bundled-node-gyp.md index 01ad5642b2..a7acf44978 100644 --- a/docs/Updating-npm-bundled-node-gyp.md +++ b/docs/Updating-npm-bundled-node-gyp.md @@ -33,7 +33,7 @@ for /f "delims=" %P in ('npm prefix -g') do npm config set node_gyp"%P\node_modu ### Powershell ``` npm install --global node-gyp@latest -npm prefix -g | % {npm config set node_gyp "$_\node_modules\node-gyp\bin\node-gypjs"} +npm prefix -g | % {npm config set node_gyp "$_\node_modules\node-gyp\bin\node-gyp.js"} ``` ## Undo From 8083f6b855bd7f3326af04c5f5269fc28d7f2508 Mon Sep 17 00:00:00 2001 From: Gar Date: Tue, 16 Nov 2021 09:42:21 -0800 Subject: [PATCH 316/551] deps: npmlog@6.0.0 --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index ea06551817..08deaabfd3 100644 --- a/package.json +++ b/package.json @@ -27,7 +27,7 @@ "graceful-fs": "^4.2.6", "make-fetch-happen": "^9.1.0", "nopt": "^5.0.0", - "npmlog": "^4.1.2", + "npmlog": "^6.0.0", "rimraf": "^3.0.2", "semver": "^7.3.5", "tar": "^6.1.2", From cc37b880690706d3c5d04d5a68c76c392a0a23ed Mon Sep 17 00:00:00 2001 From: HeatonZ <905342035@qq.com> Date: Fri, 19 Nov 2021 23:17:12 +0800 Subject: [PATCH 317/551] fix: windows command missing space (#2553) --- docs/Updating-npm-bundled-node-gyp.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Updating-npm-bundled-node-gyp.md b/docs/Updating-npm-bundled-node-gyp.md index a7acf44978..0777687c22 100644 --- a/docs/Updating-npm-bundled-node-gyp.md +++ b/docs/Updating-npm-bundled-node-gyp.md @@ -27,7 +27,7 @@ npm config set node_gyp $(npm prefix -g)/lib/node_modules/node-gyp/bin/node-gyp. ### Windows Command Prompt ``` npm install --global node-gyp@latest -for /f "delims=" %P in ('npm prefix -g') do npm config set node_gyp"%P\node_modules\node-gyp\bin\node-gyp.js" +for /f "delims=" %P in ('npm prefix -g') do npm config set node_gyp "%P\node_modules\node-gyp\bin\node-gyp.js" ``` ### Powershell From f5fa6b86fd2847ca8c1996102f43d44f98780c4a Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri, 19 Nov 2021 15:18:23 +0000 Subject: [PATCH 318/551] chore: release 8.4.1 --- CHANGELOG.md | 17 +++++++++++++++++ package.json | 2 +- 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e5d1a4d065..1e54fd69a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,22 @@ # Changelog +### [8.4.1](https://www.github.com/nodejs/node-gyp/compare/v8.4.0...v8.4.1) (2021-11-19) + + +### Bug Fixes + +* windows command missing space ([#2553](https://www.github.com/nodejs/node-gyp/issues/2553)) ([cc37b88](https://www.github.com/nodejs/node-gyp/commit/cc37b880690706d3c5d04d5a68c76c392a0a23ed)) + + +### Doc + +* fix typo in powershell node-gyp update ([787cf7f](https://www.github.com/nodejs/node-gyp/commit/787cf7f8e5ddd5039e02b64ace6b7b15e06fe0a4)) + + +### Core + +* npmlog@6.0.0 ([8083f6b](https://www.github.com/nodejs/node-gyp/commit/8083f6b855bd7f3326af04c5f5269fc28d7f2508)) + ## [8.4.0](https://www.github.com/nodejs/node-gyp/compare/v8.3.0...v8.4.0) (2021-11-05) diff --git a/package.json b/package.json index 08deaabfd3..fbeae5e20d 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "8.4.0", + "version": "8.4.1", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From c2a185056e2e589b520fbc0bcc59c2935cd07ede Mon Sep 17 00:00:00 2001 From: Rich Trott Date: Sat, 27 Nov 2021 16:00:28 -0800 Subject: [PATCH 319/551] chore: add minimal SECURITY.md (#2560) --- SECURITY.md | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 SECURITY.md diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000000..1e168d76e3 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,2 @@ +If you believe you have found a security issue in the software in this +repository, please consult https://github.com/nodejs/node/blob/HEAD/SECURITY.md. \ No newline at end of file From 2ef5fb86277c4d81baffc0b9f642a8d86be1bfa5 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Mon, 13 Dec 2021 11:05:01 +0100 Subject: [PATCH 320/551] doc: Rename and update Common-issues.md --> docs/README.md (#2567) Update the common problems to track with current issues on this repo and shorten the URL to just https://github.com/nodejs/node-gyp/tree/master/docs --- docs/Common-issues.md | 14 -------------- docs/README.md | 11 +++++++++++ 2 files changed, 11 insertions(+), 14 deletions(-) delete mode 100644 docs/Common-issues.md create mode 100644 docs/README.md diff --git a/docs/Common-issues.md b/docs/Common-issues.md deleted file mode 100644 index ae05fe326a..0000000000 --- a/docs/Common-issues.md +++ /dev/null @@ -1,14 +0,0 @@ -## Python Issues OSX - -Make sure you are using the native Python version in OSX. If you use a MacPorts of HomeBrew version, you may run into problems. - -If you have issues with `execvp`, be sure to check your `$PYTHON` environment variable. If it is not set to the native version, unset it and try again. - -Notes: https://gist.github.com/erichocean/5177582 - -## npm ERR! `node-gyp rebuild`(Windows) -* just install the build tools from [here](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) -Please note the version as is required in below command e.g **2017** -* Launch cmd, run `npm config set msvs_version 2017` -* close and open new CMD/terminal and all is well :100: - diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000000..1bec1f0d57 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,11 @@ +## Versions of `node-gyp` that are earlier than v8.x.x + +Please look thru your error log for the string `gyp info using node-gyp@` and if that version number is less than the [current release of node-gyp](https://github.com/nodejs/node-gyp/releases) then __please upgrade__ using [these instructions](https://github.com/nodejs/node-gyp/blob/master/docs/Updating-npm-bundled-node-gyp.md) and then try your command again. + +## `node-sass` is deprecated + +Please be aware that the package [`node-sass` is deprecated](https://github.com/sass/node-sass#node-sass) so you should actively seek alternatives. Please avoid opening new `node-sass` issues on this repo. You can try `npm install --global node-sass@latest` but we [cannot help much](https://github.com/nodejs/node-gyp/issues?q=is%3Aissue+label%3A%22Node+Sass+--%3E+Dart+Sass%22+) here. + +## Issues finding the installed Visual Studio + +In cmd, [`npm config set msvs_version 20xx`](https://github.com/nodejs/node-gyp#on-windows) with ___xx___ matching your locally installed version of Visual Studio. From 6e8f93be0443f2649d4effa7bc773a9da06a33b4 Mon Sep 17 00:00:00 2001 From: owl from hogvarts <47751812+owl-from-hogvarts@users.noreply.github.com> Date: Tue, 4 Jan 2022 11:50:30 +0300 Subject: [PATCH 321/551] docs: title match content (#2574) --- docs/Force-npm-to-use-global-node-gyp.md | 45 +++++++++++++++++ docs/Updating-npm-bundled-node-gyp.md | 63 ++++++++++++++++-------- 2 files changed, 88 insertions(+), 20 deletions(-) create mode 100644 docs/Force-npm-to-use-global-node-gyp.md diff --git a/docs/Force-npm-to-use-global-node-gyp.md b/docs/Force-npm-to-use-global-node-gyp.md new file mode 100644 index 0000000000..1ced628c65 --- /dev/null +++ b/docs/Force-npm-to-use-global-node-gyp.md @@ -0,0 +1,45 @@ +# Force npm to use global installed node-gyp + +[Many issues](https://github.com/nodejs/node-gyp/labels/ERR%21%20node-gyp%20-v%20%3C%3D%20v5.1.0) are opened by users who are +not running a [current version of node-gyp](https://github.com/nodejs/node-gyp/releases). + +`npm` bundles its own, internal, copy of `node-gyp`. This internal copy is independent of any globally installed copy of node-gyp that +may have been installed via `npm install -g node-gyp`. + +Generally, npm's library files are installed inside your global "node_modules", where npm is installed (run `npm prefix` and add `lib/node_modules`, or just `node_modules` for Windows). There are some exceptions to this. Inside this global `node_modules/` there will be an `npm/` directory and inside this you'll find a `node_modules/node-gyp/` directory. So it may look something like `/usr/local/lib/node_modules/npm/node_modules/node-gyp/`. This is the version of node-gyp that ships with npm. + +When you install a _new_ version of node-gyp outside of npm, it'll go into your global node_modules, but not under the `npm/node_modules`. So that may look like `/usr/local/lib/node_modules/node-gyp/`. It'll have the `node-gyp` executable linked into your `PATH` so running `node-gyp` will use this version. + +The catch is that npm won't use this version unless you tell it to, it'll keep on using the one you have installed. You need to instruct it to by setting the `node_gyp` config variable (which goes into your `~/.npmrc`). You do this by running the `npm config set` command as below. Then npm will use the command in the path you supply whenever it needs to build a native addon. + +**Important**: You also need to remember to unset this when you upgrade npm with a newer version of node-gyp, or you have to manually keep your globally installed node-gyp to date. See "Undo" below. + +## Linux and macOS +``` +npm install --global node-gyp@latest +npm config set node_gyp $(npm prefix -g)/lib/node_modules/node-gyp/bin/node-gyp.js +``` + +`sudo` may be required for the first command if you get a permission error. + +## Windows + +### Windows Command Prompt +``` +npm install --global node-gyp@latest +for /f "delims=" %P in ('npm prefix -g') do npm config set node_gyp "%P\node_modules\node-gyp\bin\node-gyp.js" +``` + +### Powershell +``` +npm install --global node-gyp@latest +npm prefix -g | % {npm config set node_gyp "$_\node_modules\node-gyp\bin\node-gyp.js"} +``` + +## Undo +**Beware** if you don't unset the `node_gyp` config option, npm will continue to use the globally installed version of node-gyp rather than the one it ships with, which may end up being newer. + +``` +npm config delete node_gyp +npm uninstall --global node-gyp +``` diff --git a/docs/Updating-npm-bundled-node-gyp.md b/docs/Updating-npm-bundled-node-gyp.md index 0777687c22..38795f5fac 100644 --- a/docs/Updating-npm-bundled-node-gyp.md +++ b/docs/Updating-npm-bundled-node-gyp.md @@ -6,40 +6,63 @@ not running a [current version of node-gyp](https://github.com/nodejs/node-gyp/r `npm` bundles its own, internal, copy of `node-gyp`. This internal copy is independent of any globally installed copy of node-gyp that may have been installed via `npm install -g node-gyp`. -Generally, npm's library files are installed inside your global "node_modules", where npm is installed (run `npm prefix` and add `lib/node_modules`, or just `node_modules` for Windows). There are some exceptions to this. Inside this global `node_modules/` there will be an `npm/` directory and inside this you'll find a `node_modules/node-gyp/` directory. So it may look something like `/usr/local/lib/node_modules/npm/node_modules/node-gyp/`. This is the version of node-gyp that ships with npm. +This means that while `node-gyp` doesn't get installed into your `$PATH` by default, npm still keeps its own copy to invoke when you +attempt to `npm install` a native add-on. -When you install a _new_ version of node-gyp outside of npm, it'll go into your global node_modules, but not under the `npm/node_modules`. So that may look like `/usr/local/lib/node_modules/node-gyp/`. It'll have the `node-gyp` executable linked into your `PATH` so running `node-gyp` will use this version. +Sometimes, you may need to update npm's internal node-gyp to a newer version than what is installed. A simple `npm install -g node-gyp` +_won't_ do the trick since npm will still continue to use its internal copy over the global one. -The catch is that npm won't use this version unless you tell it to, it'll keep on using the one you have installed. You need to instruct it to by setting the `node_gyp` config variable (which goes into your `~/.npmrc`). You do this by running the `npm config set` command as below. Then npm will use the command in the path you supply whenever it needs to build a native addon. +So instead: -**Important**: You also need to remember to unset this when you upgrade npm with a newer version of node-gyp, or you have to manually keep your globally installed node-gyp to date. See "Undo" below. +## Version of npm -## Linux and macOS -``` -npm install --global node-gyp@latest -npm config set node_gyp $(npm prefix -g)/lib/node_modules/node-gyp/bin/node-gyp.js +We need to start by knowing your version of `npm`: +```bash +npm --version ``` -`sudo` may be required for the first command if you get a permission error. +## Linux, macOS, Solaris, etc. -## Windows +Unix is easy. Just run the following command. -### Windows Command Prompt +If your npm is version ___7___, do: +```bash +$ npm explore npm/node_modules/@npmcli/run-script -g -- npm_config_global=false npm install node-gyp@latest ``` -npm install --global node-gyp@latest -for /f "delims=" %P in ('npm prefix -g') do npm config set node_gyp "%P\node_modules\node-gyp\bin\node-gyp.js" + +Else if your npm is version ___less than 7___, do: +```bash +$ npm explore npm/node_modules/npm-lifecycle -g -- npm install node-gyp@latest ``` -### Powershell +If the command fails with a permissions error, please try `sudo` and then the command. + +## Windows + +Windows is a bit trickier, since `npm` might be installed to the "Program Files" directory, which needs admin privileges in order to +modify on current Windows. Therefore, run the following commands __inside a `cmd.exe` started with "Run as Administrator"__: + +First we need to find the location of `node`. If you don't already know the location that `node.exe` got installed to, then run: +```bash +$ where node ``` -npm install --global node-gyp@latest -npm prefix -g | % {npm config set node_gyp "$_\node_modules\node-gyp\bin\node-gyp.js"} + +Now `cd` to the directory that `node.exe` is contained in e.g.: +```bash +$ cd "C:\Program Files\nodejs" ``` -## Undo -**Beware** if you don't unset the `node_gyp` config option, npm will continue to use the globally installed version of node-gyp rather than the one it ships with, which may end up being newer. +If your npm version is ___7___, do: +```bash +cd node_modules\npm\node_modules\@npmcli\run-script +``` +Else if your npm version is ___less than 7___, do: +```bash +cd node_modules\npm\node_modules\npm-lifecycle ``` -npm config delete node_gyp -npm uninstall --global node-gyp + +Finish by running: +```bash +$ npm install node-gyp@latest ``` From faf6d48f8a77c08a313baf9332358c4b1231c73c Mon Sep 17 00:00:00 2001 From: DeeDeeG Date: Wed, 5 Jan 2022 22:28:41 -0500 Subject: [PATCH 322/551] docs: Add notes/disclaimers for upgrading the copy of node-gyp that npm uses (#2585) --- docs/Force-npm-to-use-global-node-gyp.md | 2 ++ docs/Updating-npm-bundled-node-gyp.md | 8 ++++++-- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/Force-npm-to-use-global-node-gyp.md b/docs/Force-npm-to-use-global-node-gyp.md index 1ced628c65..d12d1526b8 100644 --- a/docs/Force-npm-to-use-global-node-gyp.md +++ b/docs/Force-npm-to-use-global-node-gyp.md @@ -1,5 +1,7 @@ # Force npm to use global installed node-gyp +**Note: These instructions only work with npm 6 or older. For a solution that works with npm 8 (or older), see [Updating-npm-bundled-node-gyp.md](Updating-npm-bundled-node-gyp.md).** + [Many issues](https://github.com/nodejs/node-gyp/labels/ERR%21%20node-gyp%20-v%20%3C%3D%20v5.1.0) are opened by users who are not running a [current version of node-gyp](https://github.com/nodejs/node-gyp/releases). diff --git a/docs/Updating-npm-bundled-node-gyp.md b/docs/Updating-npm-bundled-node-gyp.md index 38795f5fac..1d91af6bb2 100644 --- a/docs/Updating-npm-bundled-node-gyp.md +++ b/docs/Updating-npm-bundled-node-gyp.md @@ -1,5 +1,9 @@ # Updating the npm-bundled version of node-gyp +**Note: These instructions are (only) tested and known to work with npm 8 and older.** + +**Note: These instructions will be undone if you reinstall or upgrade npm or node! For a more permanent (and simpler) solution, see [Force-npm-to-use-global-node-gyp.md](Force-npm-to-use-global-node-gyp.md). (npm 6 or older only!)** + [Many issues](https://github.com/nodejs/node-gyp/labels/ERR%21%20node-gyp%20-v%20%3C%3D%20v5.1.0) are opened by users who are not running a [current version of node-gyp](https://github.com/nodejs/node-gyp/releases). @@ -25,7 +29,7 @@ npm --version Unix is easy. Just run the following command. -If your npm is version ___7___, do: +If your npm is version ___7 or 8___, do: ```bash $ npm explore npm/node_modules/@npmcli/run-script -g -- npm_config_global=false npm install node-gyp@latest ``` @@ -52,7 +56,7 @@ Now `cd` to the directory that `node.exe` is contained in e.g.: $ cd "C:\Program Files\nodejs" ``` -If your npm version is ___7___, do: +If your npm version is ___7 or 8___, do: ```bash cd node_modules\npm\node_modules\@npmcli\run-script ``` From a2f298870692022302fa27a1d42363c4a72df407 Mon Sep 17 00:00:00 2001 From: owl from hogvarts <47751812+owl-from-hogvarts@users.noreply.github.com> Date: Fri, 7 Jan 2022 12:23:52 +0300 Subject: [PATCH 323/551] docs: rephrase explanation of which node-gyp is used by npm (#2587) --- docs/Force-npm-to-use-global-node-gyp.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/Force-npm-to-use-global-node-gyp.md b/docs/Force-npm-to-use-global-node-gyp.md index d12d1526b8..c6304e490a 100644 --- a/docs/Force-npm-to-use-global-node-gyp.md +++ b/docs/Force-npm-to-use-global-node-gyp.md @@ -5,14 +5,14 @@ [Many issues](https://github.com/nodejs/node-gyp/labels/ERR%21%20node-gyp%20-v%20%3C%3D%20v5.1.0) are opened by users who are not running a [current version of node-gyp](https://github.com/nodejs/node-gyp/releases). -`npm` bundles its own, internal, copy of `node-gyp`. This internal copy is independent of any globally installed copy of node-gyp that +npm bundles its own, internal, copy of node-gyp located at `npm/node_modules`, within npm's private dependencies which are separate from *globally* accessible packages. Therefore this internal copy of node-gyp is independent from any globally installed copy of node-gyp that may have been installed via `npm install -g node-gyp`. -Generally, npm's library files are installed inside your global "node_modules", where npm is installed (run `npm prefix` and add `lib/node_modules`, or just `node_modules` for Windows). There are some exceptions to this. Inside this global `node_modules/` there will be an `npm/` directory and inside this you'll find a `node_modules/node-gyp/` directory. So it may look something like `/usr/local/lib/node_modules/npm/node_modules/node-gyp/`. This is the version of node-gyp that ships with npm. +So npm's internal copy of node-gyp **isn't** stored inside *global* `node_modules` and thus isn't available for use as a standalone package. npm uses it's *internal* copy of `node-gyp` to automatically build native addons. -When you install a _new_ version of node-gyp outside of npm, it'll go into your global node_modules, but not under the `npm/node_modules`. So that may look like `/usr/local/lib/node_modules/node-gyp/`. It'll have the `node-gyp` executable linked into your `PATH` so running `node-gyp` will use this version. +When you install a _new_ version of node-gyp outside of npm, it'll go into your *global* `node_modules`, but not under the `npm/node_modules` (where internal copy of node-gyp is stored). So it will get into your `$PATH` and you will be able to use this globally installed version (**but not internal node-gyp of npm**) as any other globally installed package. -The catch is that npm won't use this version unless you tell it to, it'll keep on using the one you have installed. You need to instruct it to by setting the `node_gyp` config variable (which goes into your `~/.npmrc`). You do this by running the `npm config set` command as below. Then npm will use the command in the path you supply whenever it needs to build a native addon. +The catch is that npm **won't** use global version unless you tell it to, it'll keep on using the **internal one**. You need to instruct it to by setting the `node_gyp` config variable (which goes into your `~/.npmrc`). You do this by running the `npm config set` command as below. Then npm will use the command in the path you supply whenever it needs to build a native addon. **Important**: You also need to remember to unset this when you upgrade npm with a newer version of node-gyp, or you have to manually keep your globally installed node-gyp to date. See "Undo" below. From e069f13658a8bfb5fd60f74708cf8be0856d92e3 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Sun, 30 Jan 2022 23:13:36 +0100 Subject: [PATCH 324/551] doc: Update Python versions (#2571) * Add Python 3.10 * Drop Python 3.6 which [EOLs on 23 Dec. 2021](https://devguide.python.org/#status-of-python-branches) * macOS: clarify `Xcode Command Line Tools` standalone vs. from full Xcode * Window: Use the same URL as https://github.com/nodejs/node/blob/master/BUILDING.md#windows --- README.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 71eea0a942..d28a0cc9ef 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ Depending on your operating system, you will need to install: ### On Unix - * Python v3.6, v3.7, v3.8, or v3.9 + * Python v3.7, v3.8, v3.9, or v3.10 * `make` * A proper C/C++ compiler toolchain, like [GCC](https://gcc.gnu.org) @@ -39,13 +39,15 @@ Depending on your operating system, you will need to install: **ATTENTION**: If your Mac has been _upgraded_ to macOS Catalina (10.15), please read [macOS_Catalina.md](macOS_Catalina.md). - * Python v3.6, v3.7, v3.8, or v3.9 - * [Xcode](https://developer.apple.com/xcode/download/) - * You also need to install the `XCode Command Line Tools` by running `xcode-select --install`. Alternatively, if you already have the full Xcode installed, you can find them under the menu `Xcode -> Open Developer Tool -> More Developer Tools...`. This step will install `clang`, `clang++`, and `make`. + * Python v3.7, v3.8, v3.9, or v3.10 + * `XCode Command Line Tools` which will install `clang`, `clang++`, and `make`. + * Install the `XCode Command Line Tools` standalone by running `xcode-select --install`. -- OR -- + * Alternatively, if you already have the [full Xcode installed](https://developer.apple.com/xcode/download/), you can install the Command Line Tools under the menu `Xcode -> Open Developer Tool -> More Developer Tools...`. + ### On Windows -Install the current version of Python from the [Microsoft Store package](https://docs.python.org/3/using/windows.html#the-microsoft-store-package). +Install the current version of Python from the [Microsoft Store package](https://www.microsoft.com/en-us/p/python-310/9pjpw5ldxlz5). Install tools and configuration manually: * Install Visual C++ Build Environment: [Visual Studio Build Tools](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) @@ -59,8 +61,8 @@ Install tools and configuration manually: ### Configuring Python Dependency -`node-gyp` requires that you have installed a compatible version of Python, one of: v3.6, v3.7, -v3.8, or v3.9. If you have multiple Python versions installed, you can identify which Python +`node-gyp` requires that you have installed a compatible version of Python, one of: v3.7, v3.8, +v3.9, or v3.10. If you have multiple Python versions installed, you can identify which Python version `node-gyp` should use in one of the following ways: 1. by setting the `--python` command-line option, e.g.: From 6562f92a6f2e67aeae081ddf5272ff117f1fab07 Mon Sep 17 00:00:00 2001 From: Gar Date: Sun, 30 Jan 2022 18:42:44 -0800 Subject: [PATCH 325/551] deps!: increase "engines" to "node" : "^12.22 || ^14.13 || >=16" (#2601) Makes npm warn users if they are using an unsupported Node version. --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index fbeae5e20d..a92fffadbe 100644 --- a/package.json +++ b/package.json @@ -34,7 +34,7 @@ "which": "^2.0.2" }, "engines": { - "node": ">= 10.12.0" + "node": "^12.22 || ^14.13 || >=16" }, "devDependencies": { "bindings": "^1.5.0", From 78f66604e0df480d4f36a8fa4f3618c046a6fbdc Mon Sep 17 00:00:00 2001 From: Gar Date: Wed, 9 Feb 2022 10:44:52 -0800 Subject: [PATCH 326/551] deps: make-fetch-happen@10.0.1 The breaking change was dropping node10 support, which node-gyp has already done. --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index a92fffadbe..5dc4182e7d 100644 --- a/package.json +++ b/package.json @@ -25,7 +25,7 @@ "env-paths": "^2.2.0", "glob": "^7.1.4", "graceful-fs": "^4.2.6", - "make-fetch-happen": "^9.1.0", + "make-fetch-happen": "^10.0.1", "nopt": "^5.0.0", "npmlog": "^6.0.0", "rimraf": "^3.0.2", From 839e414b63790c815a4a370d0feee8f24a94d40f Mon Sep 17 00:00:00 2001 From: nlf Date: Tue, 15 Feb 2022 09:39:07 -0800 Subject: [PATCH 327/551] fix: update make-fetch-happen to a minimum of 10.0.3 --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index 5dc4182e7d..a1caf1c79e 100644 --- a/package.json +++ b/package.json @@ -25,7 +25,7 @@ "env-paths": "^2.2.0", "glob": "^7.1.4", "graceful-fs": "^4.2.6", - "make-fetch-happen": "^10.0.1", + "make-fetch-happen": "^10.0.3", "nopt": "^5.0.0", "npmlog": "^6.0.0", "rimraf": "^3.0.2", From a32a9aa8e603af319dece46cf0c97691a8ac1ff9 Mon Sep 17 00:00:00 2001 From: Mohamed-Elzohary <48733136+Mohamed-Elzohary@users.noreply.github.com> Date: Fri, 24 Dec 2021 22:06:33 +0200 Subject: [PATCH 328/551] added node-heapdump binding.gyp --- docs/binding.gyp-files-in-the-wild.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/binding.gyp-files-in-the-wild.md b/docs/binding.gyp-files-in-the-wild.md index c4603dd3d1..795d2fd2ec 100644 --- a/docs/binding.gyp-files-in-the-wild.md +++ b/docs/binding.gyp-files-in-the-wild.md @@ -45,4 +45,5 @@ To add to this page, just add the link to the project's `binding.gyp` file below * [nodecv](https://github.com/xudafeng/nodecv/blob/master/binding.gyp) * [magick-cli](https://github.com/NickNaso/magick-cli/blob/master/binding.gyp) * [sharp](https://github.com/lovell/sharp/blob/master/binding.gyp) - * [krb5](https://github.com/adaltas/node-krb5/blob/master/binding.gyp) \ No newline at end of file + * [krb5](https://github.com/adaltas/node-krb5/blob/master/binding.gyp) + * [node-heapdump](https://github.com/bnoordhuis/node-heapdump/blob/master/binding.gyp) From eef4eefccb13ff6a32db862709ee5b2d4edf7e95 Mon Sep 17 00:00:00 2001 From: Cheng Zhao Date: Mon, 8 Nov 2021 11:26:57 +0900 Subject: [PATCH 329/551] fix: _ in npm_config_ env variables --- lib/create-config-gypi.js | 3 ++- lib/node-gyp.js | 4 ++++ test/test-options.js | 11 +++++++++++ 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/lib/create-config-gypi.js b/lib/create-config-gypi.js index dbcb8b485c..ced4911502 100644 --- a/lib/create-config-gypi.js +++ b/lib/create-config-gypi.js @@ -19,7 +19,8 @@ async function getBaseConfigGypi ({ gyp, nodeDir }) { // try reading $nodeDir/include/node/config.gypi first when: // 1. --dist-url or --nodedir is specified // 2. and --force-process-config is not specified - const shouldReadConfigGypi = (gyp.opts.nodedir || gyp.opts['dist-url']) && !gyp.opts['force-process-config'] + const useCustomHeaders = gyp.opts.nodedir || gyp.opts.disturl || gyp.opts['dist-url'] + const shouldReadConfigGypi = useCustomHeaders && !gyp.opts['force-process-config'] if (shouldReadConfigGypi && nodeDir) { try { const baseConfigGypiPath = path.resolve(nodeDir, 'include/node/config.gypi') diff --git a/lib/node-gyp.js b/lib/node-gyp.js index 0f11185641..e492ec1026 100644 --- a/lib/node-gyp.js +++ b/lib/node-gyp.js @@ -149,6 +149,10 @@ proto.parseArgv = function parseOpts (argv) { // gyp@741b7f1 enters an infinite loop when it encounters // zero-length options so ensure those don't get through. if (name) { + // convert names like force_process_config to force-process-config + if (name.includes('_')) { + name = name.replace(/_/g, '-') + } this.opts[name] = val } } diff --git a/test/test-options.js b/test/test-options.js index b2ac62c874..8a634f0e09 100644 --- a/test/test-options.js +++ b/test/test-options.js @@ -29,3 +29,14 @@ test('options in environment', (t) => { t.deepEqual(Object.keys(g.opts).sort(), keys.sort()) }) + +test('options with spaces in environment', (t) => { + t.plan(1) + + process.env.npm_config_force_process_config = 'true' + + const g = gyp() + g.parseArgv(['rebuild']) // Also sets opts.argv. + + t.equal(g.opts['force-process-config'], 'true') +}) From 1d499dd5606f39de2d34fa822fd0fa5ce17fbd06 Mon Sep 17 00:00:00 2001 From: alexcfyung Date: Wed, 23 Feb 2022 21:50:31 -0500 Subject: [PATCH 330/551] lib: add lib.target as path for searching libnode on z/OS --- lib/configure.js | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/configure.js b/lib/configure.js index d9b8fe340e..17a6487fa9 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -169,6 +169,8 @@ function configure (gyp, argv, callback) { }) } else { candidates = [ + 'out/Release/lib.target/libnode', + 'out/Debug/lib.target/libnode', 'out/Release/obj.target/libnode', 'out/Debug/obj.target/libnode', 'lib/libnode' From b1ad49229272492cf9e030083d3cb4ea81afabb1 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 24 Feb 2022 22:46:51 +0000 Subject: [PATCH 331/551] chore: release 9.0.0 --- CHANGELOG.md | 33 +++++++++++++++++++++++++++++++++ package.json | 2 +- 2 files changed, 34 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1e54fd69a6..7a474ed93f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,38 @@ # Changelog +## [9.0.0](https://www.github.com/nodejs/node-gyp/compare/v8.4.1...v9.0.0) (2022-02-24) + + +### ⚠ BREAKING CHANGES + +* increase "engines" to "node" : "^12.22 || ^14.13 || >=16" (#2601) + +### Bug Fixes + +* _ in npm_config_ env variables ([eef4eef](https://www.github.com/nodejs/node-gyp/commit/eef4eefccb13ff6a32db862709ee5b2d4edf7e95)) +* update make-fetch-happen to a minimum of 10.0.3 ([839e414](https://www.github.com/nodejs/node-gyp/commit/839e414b63790c815a4a370d0feee8f24a94d40f)) + + +### Miscellaneous + +* add minimal SECURITY.md ([#2560](https://www.github.com/nodejs/node-gyp/issues/2560)) ([c2a1850](https://www.github.com/nodejs/node-gyp/commit/c2a185056e2e589b520fbc0bcc59c2935cd07ede)) + + +### Doc + +* Add notes/disclaimers for upgrading the copy of node-gyp that npm uses ([#2585](https://www.github.com/nodejs/node-gyp/issues/2585)) ([faf6d48](https://www.github.com/nodejs/node-gyp/commit/faf6d48f8a77c08a313baf9332358c4b1231c73c)) +* Rename and update Common-issues.md --> docs/README.md ([#2567](https://www.github.com/nodejs/node-gyp/issues/2567)) ([2ef5fb8](https://www.github.com/nodejs/node-gyp/commit/2ef5fb86277c4d81baffc0b9f642a8d86be1bfa5)) +* rephrase explanation of which node-gyp is used by npm ([#2587](https://www.github.com/nodejs/node-gyp/issues/2587)) ([a2f2988](https://www.github.com/nodejs/node-gyp/commit/a2f298870692022302fa27a1d42363c4a72df407)) +* title match content ([#2574](https://www.github.com/nodejs/node-gyp/issues/2574)) ([6e8f93b](https://www.github.com/nodejs/node-gyp/commit/6e8f93be0443f2649d4effa7bc773a9da06a33b4)) +* Update Python versions ([#2571](https://www.github.com/nodejs/node-gyp/issues/2571)) ([e069f13](https://www.github.com/nodejs/node-gyp/commit/e069f13658a8bfb5fd60f74708cf8be0856d92e3)) + + +### Core + +* add lib.target as path for searching libnode on z/OS ([1d499dd](https://www.github.com/nodejs/node-gyp/commit/1d499dd5606f39de2d34fa822fd0fa5ce17fbd06)) +* increase "engines" to "node" : "^12.22 || ^14.13 || >=16" ([#2601](https://www.github.com/nodejs/node-gyp/issues/2601)) ([6562f92](https://www.github.com/nodejs/node-gyp/commit/6562f92a6f2e67aeae081ddf5272ff117f1fab07)) +* make-fetch-happen@10.0.1 ([78f6660](https://www.github.com/nodejs/node-gyp/commit/78f66604e0df480d4f36a8fa4f3618c046a6fbdc)) + ### [8.4.1](https://www.github.com/nodejs/node-gyp/compare/v8.4.0...v8.4.1) (2021-11-19) diff --git a/package.json b/package.json index a1caf1c79e..e795db1834 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "8.4.1", + "version": "9.0.0", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From 245cd5bbe4441d4f05e88f2fa20a86425419b6af Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Tue, 1 Mar 2022 17:06:22 +0100 Subject: [PATCH 332/551] test: Upgrade GitHub Actions (#2623) --- .github/workflows/tests.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index f7c9b979e3..83ccf7f547 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -16,11 +16,11 @@ jobs: - name: Checkout Repository uses: actions/checkout@v2 - name: Use Node.js ${{ matrix.node }} - uses: actions/setup-node@v2 + uses: actions/setup-node@v3 with: node-version: ${{ matrix.node }} - name: Use Python ${{ matrix.python }} - uses: actions/setup-python@v2 + uses: actions/setup-python@v3 with: python-version: ${{ matrix.python }} env: From 62d28151bf8266a34e1bcceeb25b4e6e2ae5ca5d Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Mon, 7 Mar 2022 16:31:46 +0100 Subject: [PATCH 333/551] doc: update docs/README.md with latest version number --- docs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/README.md b/docs/README.md index 1bec1f0d57..7027960909 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,4 +1,4 @@ -## Versions of `node-gyp` that are earlier than v8.x.x +## Versions of `node-gyp` that are earlier than v9.x.x Please look thru your error log for the string `gyp info using node-gyp@` and if that version number is less than the [current release of node-gyp](https://github.com/nodejs/node-gyp/releases) then __please upgrade__ using [these instructions](https://github.com/nodejs/node-gyp/blob/master/docs/Updating-npm-bundled-node-gyp.md) and then try your command again. From bf81cd452b931dd4dfa82762c23dd530a075d992 Mon Sep 17 00:00:00 2001 From: Doni Rubiagatra Date: Tue, 3 May 2022 15:52:28 +0700 Subject: [PATCH 334/551] fix: typo on readme --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d28a0cc9ef..48bf763943 100644 --- a/README.md +++ b/README.md @@ -89,7 +89,7 @@ searching will be done. ### Build for Third Party Node.js Runtimes -When building modules for thid party Node.js runtimes like Electron, which have +When building modules for third party Node.js runtimes like Electron, which have different build configurations from the official Node.js distribution, you should use `--dist-url` or `--nodedir` flags to specify the headers of the runtime to build for. From 147e3d34f44a97deb7aa507207680cf0f4e662a2 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Wed, 11 May 2022 14:36:02 +1000 Subject: [PATCH 335/551] fix: new ca & server certs, bundle in .js file and unpack for testing bundling in certs.js rather than including the raw files should avoid some false positives that low-quality security scanners keep on complaining about. --- test/fixtures/ca-bundle.crt | 40 ---------- test/fixtures/ca.crt | 21 ----- test/fixtures/certs.js | 150 ++++++++++++++++++++++++++++++++++++ test/fixtures/server.crt | 21 ----- test/fixtures/server.key | 27 ------- test/test-download.js | 24 ++++-- 6 files changed, 167 insertions(+), 116 deletions(-) delete mode 100644 test/fixtures/ca-bundle.crt delete mode 100644 test/fixtures/ca.crt create mode 100644 test/fixtures/certs.js delete mode 100644 test/fixtures/server.crt delete mode 100644 test/fixtures/server.key diff --git a/test/fixtures/ca-bundle.crt b/test/fixtures/ca-bundle.crt deleted file mode 100644 index fb1dea98a7..0000000000 --- a/test/fixtures/ca-bundle.crt +++ /dev/null @@ -1,40 +0,0 @@ ------BEGIN CERTIFICATE----- -MIIDJjCCAg4CAhnOMA0GCSqGSIb3DQEBBQUAMH0xCzAJBgNVBAYTAlVTMQswCQYD -VQQIDAJDQTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzEZMBcGA1UECgwQU3Ryb25n -TG9vcCwgSW5jLjESMBAGA1UECwwJU3Ryb25nT3BzMRowGAYDVQQDDBFjYS5zdHJv -bmdsb29wLmNvbTAeFw0xNTEyMDgyMzM1MzNaFw00MzA0MjQyMzM1MzNaMBkxFzAV -BgNVBAMMDnN0cm9uZ2xvb3AuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB -CgKCAQEAwOYI7OZ2FX/YjRgLZoDQlbPc5UZXU/j0e1wwiJNPtPEax9Y5Uoza0Pnt -Ikzkc2SfvQ+IJrhXo385tI0W5juuqbHnE7UrjUuPjUX6NHevkxcs/flmjan5wnZM -cPsGhH71WDuUEEflvZihf2Se2x+xgZtMhc5XGmVmRuZFYKvkgUhA2/w8/QrK+jPT -n9QRJxZjWNh2RBdC1B7u4jffSmOSUljYFH1I2eTeY+Rdi6YUIYSU9gEoZxsv3Tia -SomfMF5jt2Mouo6MzA+IhLvvFjcrcph1Qxgi9RkfdCMMd+Ipm9YWELkyG1bDRpQy -0iyHD4gvVsAqz1Y2KdRSdc3Kt+nTqwIDAQABoxkwFzAVBgNVHREEDjAMhwQAAAAA -hwR/AAABMA0GCSqGSIb3DQEBBQUAA4IBAQAhy4J0hML3NgmDRHdL5/iTucBe22Mf -jJjg2aifD1S187dHm+Il4qZNO2plWwAhN0h704f+8wpsaALxUvBIu6nvlvcMP5PH -jGN5JLe2Km3UaPvYOQU2SgacLilu+uBcIo2JSHLV6O7ziqUj5Gior6YxDLCtEZie -Ea8aX5/YjuACtEMJ1JjRqjgkM66XAoUe0E8onOK3FgTIO3tGoTJwRp0zS50pFuP0 -PsZtT04ck6mmXEXXknNoAyBCvPypfms9OHqcUIW9fiQnrGbS/Ri4QSQYj0DtFk/1 -na4fY1gf3zTHxH8259b/TOOaPfTnCEsOQtjUrWNR4xhmVZ+HJy4yytUW ------END CERTIFICATE----- ------BEGIN CERTIFICATE----- -MIIDbzCCAlcCAmm6MA0GCSqGSIb3DQEBCwUAMH0xCzAJBgNVBAYTAlVTMQswCQYD -VQQIDAJDQTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzEZMBcGA1UECgwQU3Ryb25n -TG9vcCwgSW5jLjESMBAGA1UECwwJU3Ryb25nT3BzMRowGAYDVQQDDBFjYS5zdHJv -bmdsb29wLmNvbTAeFw0xNTEyMDgyMzM1MzNaFw00MzA0MjQyMzM1MzNaMH0xCzAJ -BgNVBAYTAlVTMQswCQYDVQQIDAJDQTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzEZ -MBcGA1UECgwQU3Ryb25nTG9vcCwgSW5jLjESMBAGA1UECwwJU3Ryb25nT3BzMRow -GAYDVQQDDBFjYS5zdHJvbmdsb29wLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEP -ADCCAQoCggEBANfj86jkvvYDjHBgiqWhk9Cj+bqiMq3MqnV0CBO4iuK33Fo6XssE -H+yVdXlIBFbFe6t655MdBVOR2Sfj7WqNh96vhu6PyDHiwcQlTaiLU6nhIed1J4Wv -lvnJHFmp8Wbtx5AgLT4UYu03ftvXEl2DLi3vhSL2tRM1ebXHB/KPbRWkb25DPX0P -foOHot3f2dgNe2x6kponf7E/QDmAu3s7Nlkfh+ryDhgGU7wocXEhXbprNqRqOGNo -xbXgUI+/9XDxYT/7Gn5LF/fPjtN+aB0SKMnTsDhprVlZie83mlqJ46fOOrR+vrsQ -mi/1m/TadrARtZoIExC/cQRdVM05EK4tUa8CAwEAATANBgkqhkiG9w0BAQsFAAOC -AQEAQ7k5WhyhDTIGYCNzRnrMHWSzGqa1y4tJMW06wafJNRqTm1cthq1ibc6Hfq5a -K10K0qMcgauRTfQ1MWrVCTW/KnJ1vkhiTOH+RvxapGn84gSaRmV6KZen0+gMsgae -KEGe/3Hn+PmDVV+PTamHgPACfpTww38WHIe/7Ce9gHfG7MZ8cKHNZhDy0IAYPln+ -YRwMLd7JNQffHAbWb2CE1mcea4H/12U8JZW5tHCF6y9V+7IuDzqwIrLKcW3lG17n -VUG6ODF/Ryqn3V5X+TL91YyXi6c34y34IpC7MQDV/67U7+5Bp5CfeDPWW2wVSrW+ -uGZtfEvhbNm6m2i4UNmpCXxUZQ== ------END CERTIFICATE----- diff --git a/test/fixtures/ca.crt b/test/fixtures/ca.crt deleted file mode 100644 index aaf97575b1..0000000000 --- a/test/fixtures/ca.crt +++ /dev/null @@ -1,21 +0,0 @@ ------BEGIN CERTIFICATE----- -MIIDZDCCAkwCCQCAzfCLqrJvuTANBgkqhkiG9w0BAQsFADB0MQswCQYDVQQGEwJV -UzELMAkGA1UECAwCQ0ExEDAOBgNVBAoMB05vZGUuanMxETAPBgNVBAsMCG5vZGUt -Z3lwMRIwEAYDVQQDDAlsb2NhbGhvc3QxHzAdBgkqhkiG9w0BCQEWEGJ1aWxkQG5v -ZGVqcy5vcmcwHhcNMTkwNjIyMDYyMjMzWhcNMjIwNDExMDYyMjMzWjB0MQswCQYD -VQQGEwJVUzELMAkGA1UECAwCQ0ExEDAOBgNVBAoMB05vZGUuanMxETAPBgNVBAsM -CG5vZGUtZ3lwMRIwEAYDVQQDDAlsb2NhbGhvc3QxHzAdBgkqhkiG9w0BCQEWEGJ1 -aWxkQG5vZGVqcy5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDS -CHjvtVW4HdbbUwZ/ZV9s6U4x0KSoyNQrsCZjB8kRpFPe50DS5mfmu2SNBGYKRgzk -4QEEwFB9N2o8YTWsCefSRl6ti4ToPZqulU4hhRKYrEGtMJcRzi3IN7s200JaO3UH -01Su8ruO0NESb5zEU1Ykfh8Lub8TGEAINmgI61d/5d5Aq3kDjUHQJt1Ekw03Ylnu -juQyCGZxLxnngu0mIvwzyL/UeeUgsfQLzvppUk6In7tC1zzMjSPWo0c8qu6KvrW4 -bKYnkZkzdQifzbpO5ERMEsh5HWq0uHa6+dgcVHFvlhdqF4Uat87ygNplVf0txsZB -MNVqbz1k6xkZYMnzDoydAgMBAAEwDQYJKoZIhvcNAQELBQADggEBADspZGtKpWxy -J1W3FA1aeQhMvequQTcMRz4avkm4K4HfTdV1iVD4CbvdezBphouBlyLVLDFJP7RZ -m7dBJVgBwnxufoFLne8cR2MGqDRoySbFT1AtDJdxabE6Fg+QGUpgOQfeBJ6ANlSB -+qJ+HG4QA+Ouh5hxz9mgYwkIsMUABHiwENdZ/kT8Edw4xKgd3uH0YP4iiePMD66c -rzW3uXH5J1jnKgBlpxtog4P6dHCcoq+PZJ17W5bdXNyqC1LPzQqniZ2BNcEZ4ix3 -slAZAOWD1zLLGJhBPMV1fa0sHNBWc6oicr3YK/IDb0cp9kiLvnUu1pHy+LWQGqtC -rceJuGsnJEQ= ------END CERTIFICATE----- diff --git a/test/fixtures/certs.js b/test/fixtures/certs.js new file mode 100644 index 0000000000..766e54b5ed --- /dev/null +++ b/test/fixtures/certs.js @@ -0,0 +1,150 @@ +module.exports['ca.key'] = ` +-----BEGIN RSA PRIVATE KEY----- +MIIEowIBAAKCAQEAtTbG0k2UFUyCdZuip0TTEtXRHh57qosegrpHPBreSNTxt7OT +KfOUZp2rToTHeN9w0ZbV2eKRI5AuFx8Cmlm73/KIHKzSNTBATGMeeHnGaxvL/W/s +KJdTDRNf7/qCXHQ+gsuEWWCFzOZuHmmAQa2IBX2HAQTqXJI8+2iJ9gytFfJLxjqy +6O4u9ugZVHSyQJWs49tGRcWMlNm7EMStADFvJn3S11xe/kwIA2mSI/eddDnzL0Mx +AkR9dQBL66xOABLL5v3QQdhipfHluX6HLbDd/1YsFTuOpgvLRlr72rTAFrQZCokV +hXPiqstn5zJFW5arHakvMR0+OPaICF5feh/4qQIDAQABAoIBAHWg6exnWUF+GY0Y +CrwDS/QFASpI5UNt7M809bqJQlMKjyEMmvF3YJQ/soxUWlsWx1f1TjmR/V6VX6W4 +hmsE5pRXDY13jTfja0lqacQQYAD02TRY63XpzIpHUlYnSWmUN2OVkgKmShQYW9C3 +8P4xE4Nk2TaLJ0oRzy3uzOb/kXcVaJfknBRUnOhuaTSs+w4l4pPXueYA7xuHgVsL +Qq0S4kK+PmdwCMB7gzlAAQhCM3vQ1U4cjC9JIIKSmPy7BcvD0kBfVPIFQ2byGpA1 +VkWBLSyeig0YxA5oIshK5cLiDIfBIiCSEzm4AMhVhGf0tbGEwiPljxKjbarYUUIi +ATMk83UCgYEA7kKeOveuPbMqxmT42swfa9OU5jLUjH+VExU0Kv3BbEjv/OGt0fac +/cs1Ze3vnrtCHudVajocFjydb8B4c62DbA4/T+LcUw/HaMaORbOoICQidi/zZ1Lj +gjg8Ip2WKXEhSAwqUpaFd6w16NZOxiTh+NDaRKywwbe8j57eDH4uR6MCgYEAwrTS +q5ra6+WDGUFMs0y3GMbL8j14PGhxBQBYSTM//NysI+EM6eeKn1cV3BbphEw//jgE +0pVokkjvLAQWWEG2dZyRxRE3YAMgOAIPx5zbJCim3iBVuoqY9ckLg2jF8Fqqubsb +3Rf2/Xzn/rFqsXdhsjGcJpdN66T9aEjwEkAnc0MCgYA5cOYk4UGormFJo147oaqR +nFjxhp+nn7qY9yu0kajoKk1xchct337J0Qv2nv5+DjdKrArzqT7MPaDXKFfhy5s7 +mdO5tr/XZp50rCnws/d8iDmmtLjB2EHxSw10avmg1B1p+UTa1F8pEuOMVt529r1j +9zYoCFo02c8j8PEnoeQWcQKBgQCVBCuQZu5SSM/zTkTTnU0sy0lf1qflI9IMD92B ++JVqg8HDnAR0KF+x38a9MVP7ixgXCuy19t+XxfY269HmLjTlArWV671D4GCSPRGy +plwZ6nr72ieCo3y57+q94jxL3jh3+bozlpnUG/q6tTKBLGs7JDjsWDSsuxOu8tO6 +RBttXQKBgB6LQFOTjDMfsFHKsnQXFUZId3GG/iLg3WCWxEo88T5Rq3JIR0zDpW8H +cKhl/sPY+JVHsxizNCMPtp7Hn7GrB6D/v9LbO0jpG2U0BFiJ6zhiDopbP9B0EAW4 +5JJ+JGKRKoCxs3DmSVyns0gU4j4rVte97UWyVy5TZ8Acr/qrgOA1 +-----END RSA PRIVATE KEY----- +` + +module.exports['ca.crt'] = ` +-----BEGIN CERTIFICATE----- +MIIDmzCCAoOgAwIBAgIUDA0GrvcnG41XT6LYFeNwvq8YV1UwDQYJKoZIhvcNAQEL +BQAwXTELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRAwDgYDVQQKDAdOb2RlLmpz +MREwDwYDVQQLDAhub2RlLWd5cDEcMBoGA1UEAwwTbm9kZS1neXAubm9kZWpzLm9y +ZzAeFw0yMjA1MTEwNDIyMjRaFw00OTA5MjUwNDIyMjRaMF0xCzAJBgNVBAYTAlVT +MQswCQYDVQQIDAJDQTEQMA4GA1UECgwHTm9kZS5qczERMA8GA1UECwwIbm9kZS1n +eXAxHDAaBgNVBAMME25vZGUtZ3lwLm5vZGVqcy5vcmcwggEiMA0GCSqGSIb3DQEB +AQUAA4IBDwAwggEKAoIBAQC1NsbSTZQVTIJ1m6KnRNMS1dEeHnuqix6Cukc8Gt5I +1PG3s5Mp85RmnatOhMd433DRltXZ4pEjkC4XHwKaWbvf8ogcrNI1MEBMYx54ecZr +G8v9b+wol1MNE1/v+oJcdD6Cy4RZYIXM5m4eaYBBrYgFfYcBBOpckjz7aIn2DK0V +8kvGOrLo7i726BlUdLJAlazj20ZFxYyU2bsQxK0AMW8mfdLXXF7+TAgDaZIj9510 +OfMvQzECRH11AEvrrE4AEsvm/dBB2GKl8eW5foctsN3/ViwVO46mC8tGWvvatMAW +tBkKiRWFc+Kqy2fnMkVblqsdqS8xHT449ogIXl96H/ipAgMBAAGjUzBRMB0GA1Ud +DgQWBBT6LcYYABEOAMv4hI/5bC82rGlD/DAfBgNVHSMEGDAWgBT6LcYYABEOAMv4 +hI/5bC82rGlD/DAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQA9 +D+qoKw0njub+NaFRS2DFbSiKb5JKTxVjU5aNusFONFLSXBuRpnYyjjkXpJy8JMWz +g8GFDEPP6kiSb8xaPNrFcUzb4PFzJabNTuaLJpBpd2gNBj5AeYwwpRa2DPv/b4yw +y2mfULuCWS09ZAguI2OcaARlAsFxYN0IuQ6pN1AvGFGee67ve9l2VF/hhwEi4lCk +MM0CWlP6COJ8TX7X0MTtexVOgo9m3hBuTSYEZClYFIdSOk10xkPl8Y3Iz/x6mzfK +Uu2l2ZtYvSdAX1CQMds3ZWt0ChNNEjOKPv4g2QSDhGkiqrmi4wUS81g68wKqOpqn +GbN8uKxIfyMjqZKaujPR +-----END CERTIFICATE----- +` + +module.exports['server.key'] = ` +-----BEGIN RSA PRIVATE KEY----- +MIIEpAIBAAKCAQEAvPM99BkYrBcTM355dhz4XzhSDRGxa9qttUlBSgEsbu2UjsRm +XjDS+60XXd66tWpPwLeUd2uvlC/ltv5ekv+EBu35j1KfA1+K1rtFzb1i40kMCsns +OoXjgpsN2wvkxMdFkT2bkqKCS6X0xzlWea1t4poKh9iG7n3otk4RzPNawfwQ9W5n +o9/8i6AUwDbuK4dhAId/Inw2aKrMyQ+AiSvsDM2wUMq+pV7nP46f7MhR4xiGz14z +ATFdjM3Oo/1NKrr0WgVM6i0eNAtuIDqIs8YE7SfODe/SzpJySxewutfYi62OaLh/ +hmWByj/pF5SoNMLyJHxn4GyKK+Qle9NJAThLiwIDAQABAoIBAQCZs4h/Cvct7etZ +pRUqxnAoDQl5xh28LXvGj1uD1qaNacfBxvO6xR6rSedLHcZlkqBjlTI5XqjJ85h6 +njrSevWsKWMrejsNpGetO1OSA+/wEVixYgY+qPEkKftAZ1Fl3O+zMRlfU8CHxuzy +Lqsweap8fW/5h2JjmJp3ydPjE0aNqpQ+0LtYBBawKDIe2zPNOmTPwz3D8qJNQJKU +Qdj08dO/vPZncllPagGvpqhfv4hMyNChr71eBbEFsi3O5VJxfZyj+fQG0DGocr/y +sV54HtYk5j06wMxZFLQtaJn+1pOXquZMNwodSPnbrR/+CI1SZeB8N6VyqqOdmrDz +5NbfGJiRAoGBAPrCuQxJwgc2MzpEtrXA4+1uuV8QWGy3+wNKxKw4lgyC7peXXrVK +l9FkOOAPr8puPRABgDS9t6vo59BAP3Wrx0oJ9PA/Qn03WYLfJMepWqlK7ni9kS04 +5owRTduK7P190sp0m6iicsnchGSSOchECwB5UmtHysEFiuY0T+0pdNbjAoGBAMDl +57lwZDfNTGGDxLQYVzbrXgKcD60DW9MhvH3uso6cw5NYs3tmENCh9D6YrCNN4PmL +zdp4dKbOFoGJdy22TK31nrezUuNKSK+QKH2gsmNVI+a5QokNO1Cfk+PMLoOR5du2 +nwyVvzdaBwuXU4fhmwvLv/SCFNEJ0EgUILeMETE5AoGBAIwLXf9v3e3bJkb/gy8E +mAbNVLez0D5/ja9r/WTVgW9hXFDLF/iVvS4TE/SGrj2WzYF35RsPbVmUDIrwpsBX +/EfsQaA/JCn8VIBTkR31Bg4QLBjAfijMY22MaHgZIXv83lF1SE2o1ATKpCHqzFx9 +K8vK9e22PZUJPGaOhqjEA13TAoGAEPipSJFw38/6NmInfkjd84EFxmkAoBI5k/vV +36aOoyl7s40MTYEPXavCF3fLPVfuwUXhmKUcbkiXhlIX4De3y15e1n66fjDc8EVY +qqTmzQKCpBwMlI5Ld65yjoo6VW0SsiABIlRSfIY5NHXd7YiV4ZXNj6+aMUIRxyWu +Mzfpk1ECgYBZw8lML+F8XbcmCLGYuicf0V/wgFaJr8nnPeW7AiQrv13Ju1ItEaC8 +Tz8F7OfC+FoUb0MGjXHKquDVBDs4xSuS+Ol+rlZEK68ALpm8sUgp6YjARYiXlprs +6o4kN0P5F+DVU2SP6fo9zKLCxaTH9VAQ9C3VUViGAFLBt4DVDmj77g== +-----END RSA PRIVATE KEY----- +` + +module.exports['server.crt'] = ` +-----BEGIN CERTIFICATE----- +MIIDNzCCAh8CFBg1Ph5t5rBlAbCrEn/PexNy9WunMA0GCSqGSIb3DQEBCwUAMF0x +CzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTEQMA4GA1UECgwHTm9kZS5qczERMA8G +A1UECwwIbm9kZS1neXAxHDAaBgNVBAMME25vZGUtZ3lwLm5vZGVqcy5vcmcwHhcN +MjIwNTExMDQyMjI0WhcNMjMwOTIzMDQyMjI0WjBTMQswCQYDVQQGEwJVUzELMAkG +A1UECAwCQ0ExEDAOBgNVBAoMB05vZGUuanMxETAPBgNVBAsMCG5vZGUtZ3lwMRIw +EAYDVQQDDAlsb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB +AQC88z30GRisFxMzfnl2HPhfOFINEbFr2q21SUFKASxu7ZSOxGZeMNL7rRdd3rq1 +ak/At5R3a6+UL+W2/l6S/4QG7fmPUp8DX4rWu0XNvWLjSQwKyew6heOCmw3bC+TE +x0WRPZuSooJLpfTHOVZ5rW3imgqH2Ibufei2ThHM81rB/BD1bmej3/yLoBTANu4r +h2EAh38ifDZoqszJD4CJK+wMzbBQyr6lXuc/jp/syFHjGIbPXjMBMV2Mzc6j/U0q +uvRaBUzqLR40C24gOoizxgTtJ84N79LOknJLF7C619iLrY5ouH+GZYHKP+kXlKg0 +wvIkfGfgbIor5CV700kBOEuLAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAEhaNEye +JsE4eG1xaGmHq7w9eV0neOaZ58XCuF1sSEMIy9uMnl3aOdctxh/1SYkqJyMct79q +Ra2UZ6mauRlOeqdHb+HZKrFYYUOtd1HOWWJ44Gaya2EQMiTbd/Ns9Th2KTbTOCbL +CHFNska9Ty2ioT7VcrVuIEXFPMua5T4lnCkNJQla800pHHOak2baN/c66Io+8XI2 +xiqaVrLT3qvpzdiiEjo4POeRnWMIgJJshy77Am5JlhaJiAqP1AHfh/tYpliGkjXF +8DSgSoLHSQfalJ4VQvP4XLc/XnmF5Zt6bvwUxCllEBFRneU74bW7/euYX/TpYobr +Y+YM7fGiCqwHdUs= +-----END CERTIFICATE----- +` + +module.exports['ca-bundle.crt'] = ` +-----BEGIN CERTIFICATE----- +MIIDJjCCAg4CAhnOMA0GCSqGSIb3DQEBBQUAMH0xCzAJBgNVBAYTAlVTMQswCQYD +VQQIDAJDQTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzEZMBcGA1UECgwQU3Ryb25n +TG9vcCwgSW5jLjESMBAGA1UECwwJU3Ryb25nT3BzMRowGAYDVQQDDBFjYS5zdHJv +bmdsb29wLmNvbTAeFw0xNTEyMDgyMzM1MzNaFw00MzA0MjQyMzM1MzNaMBkxFzAV +BgNVBAMMDnN0cm9uZ2xvb3AuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB +CgKCAQEAwOYI7OZ2FX/YjRgLZoDQlbPc5UZXU/j0e1wwiJNPtPEax9Y5Uoza0Pnt +Ikzkc2SfvQ+IJrhXo385tI0W5juuqbHnE7UrjUuPjUX6NHevkxcs/flmjan5wnZM +cPsGhH71WDuUEEflvZihf2Se2x+xgZtMhc5XGmVmRuZFYKvkgUhA2/w8/QrK+jPT +n9QRJxZjWNh2RBdC1B7u4jffSmOSUljYFH1I2eTeY+Rdi6YUIYSU9gEoZxsv3Tia +SomfMF5jt2Mouo6MzA+IhLvvFjcrcph1Qxgi9RkfdCMMd+Ipm9YWELkyG1bDRpQy +0iyHD4gvVsAqz1Y2KdRSdc3Kt+nTqwIDAQABoxkwFzAVBgNVHREEDjAMhwQAAAAA +hwR/AAABMA0GCSqGSIb3DQEBBQUAA4IBAQAhy4J0hML3NgmDRHdL5/iTucBe22Mf +jJjg2aifD1S187dHm+Il4qZNO2plWwAhN0h704f+8wpsaALxUvBIu6nvlvcMP5PH +jGN5JLe2Km3UaPvYOQU2SgacLilu+uBcIo2JSHLV6O7ziqUj5Gior6YxDLCtEZie +Ea8aX5/YjuACtEMJ1JjRqjgkM66XAoUe0E8onOK3FgTIO3tGoTJwRp0zS50pFuP0 +PsZtT04ck6mmXEXXknNoAyBCvPypfms9OHqcUIW9fiQnrGbS/Ri4QSQYj0DtFk/1 +na4fY1gf3zTHxH8259b/TOOaPfTnCEsOQtjUrWNR4xhmVZ+HJy4yytUW +-----END CERTIFICATE----- +-----BEGIN CERTIFICATE----- +MIIDbzCCAlcCAmm6MA0GCSqGSIb3DQEBCwUAMH0xCzAJBgNVBAYTAlVTMQswCQYD +VQQIDAJDQTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzEZMBcGA1UECgwQU3Ryb25n +TG9vcCwgSW5jLjESMBAGA1UECwwJU3Ryb25nT3BzMRowGAYDVQQDDBFjYS5zdHJv +bmdsb29wLmNvbTAeFw0xNTEyMDgyMzM1MzNaFw00MzA0MjQyMzM1MzNaMH0xCzAJ +BgNVBAYTAlVTMQswCQYDVQQIDAJDQTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzEZ +MBcGA1UECgwQU3Ryb25nTG9vcCwgSW5jLjESMBAGA1UECwwJU3Ryb25nT3BzMRow +GAYDVQQDDBFjYS5zdHJvbmdsb29wLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEP +ADCCAQoCggEBANfj86jkvvYDjHBgiqWhk9Cj+bqiMq3MqnV0CBO4iuK33Fo6XssE +H+yVdXlIBFbFe6t655MdBVOR2Sfj7WqNh96vhu6PyDHiwcQlTaiLU6nhIed1J4Wv +lvnJHFmp8Wbtx5AgLT4UYu03ftvXEl2DLi3vhSL2tRM1ebXHB/KPbRWkb25DPX0P +foOHot3f2dgNe2x6kponf7E/QDmAu3s7Nlkfh+ryDhgGU7wocXEhXbprNqRqOGNo +xbXgUI+/9XDxYT/7Gn5LF/fPjtN+aB0SKMnTsDhprVlZie83mlqJ46fOOrR+vrsQ +mi/1m/TadrARtZoIExC/cQRdVM05EK4tUa8CAwEAATANBgkqhkiG9w0BAQsFAAOC +AQEAQ7k5WhyhDTIGYCNzRnrMHWSzGqa1y4tJMW06wafJNRqTm1cthq1ibc6Hfq5a +K10K0qMcgauRTfQ1MWrVCTW/KnJ1vkhiTOH+RvxapGn84gSaRmV6KZen0+gMsgae +KEGe/3Hn+PmDVV+PTamHgPACfpTww38WHIe/7Ce9gHfG7MZ8cKHNZhDy0IAYPln+ +YRwMLd7JNQffHAbWb2CE1mcea4H/12U8JZW5tHCF6y9V+7IuDzqwIrLKcW3lG17n +VUG6ODF/Ryqn3V5X+TL91YyXi6c34y34IpC7MQDV/67U7+5Bp5CfeDPWW2wVSrW+ +uGZtfEvhbNm6m2i4UNmpCXxUZQ== +-----END CERTIFICATE----- +` diff --git a/test/fixtures/server.crt b/test/fixtures/server.crt deleted file mode 100644 index 5d0c440e07..0000000000 --- a/test/fixtures/server.crt +++ /dev/null @@ -1,21 +0,0 @@ ------BEGIN CERTIFICATE----- -MIIDYjCCAkoCCQCSlmGR7KzZGTANBgkqhkiG9w0BAQsFADB0MQswCQYDVQQGEwJV -UzELMAkGA1UECAwCQ0ExEDAOBgNVBAoMB05vZGUuanMxETAPBgNVBAsMCG5vZGUt -Z3lwMRIwEAYDVQQDDAlsb2NhbGhvc3QxHzAdBgkqhkiG9w0BCQEWEGJ1aWxkQG5v -ZGVqcy5vcmcwHhcNMTkwNjIyMDYyNTU1WhcNMjkwNjE5MDYyNTU1WjByMQswCQYD -VQQGEwJVUzELMAkGA1UECAwCQ0ExEDAOBgNVBAoMB05vZGUuanMxETAPBgNVBAsM -CG5vZGUtZ3lwMRIwEAYDVQQDDAlsb2NhbGhvc3QxHTAbBgkqhkiG9w0BCQEWDmJ1 -aWxkQGlvanMub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6S1E -2WchgmbJYqCnpN7310ZgHjIOqeJe6MpSue2u6z6mTNd5izgvQNaANmn3xLFCS5zs -uZaTvdPYPkcmSQzb1YcZSUYnAxZifjYARc6kb5GSBl3q+O70ELyFrimXfZ4JI+bd -IG9KiHY17DlvZZZj/csGYVWWg0mkeH3O5LPX6/DXQVh/9+gZ02/cdIBCAtZHQwqx -7tF/qZA/kD4GZNFpU1DYHzf9H6g9htoCqmNHQWrV2T9yFybt6mbZp9kglBmyKYCc -7hmQnb7N/mHn1yIuwhBsirCJTfKH86gN81u8M3+SVHA2VUHDllcNhpDWlmInXA+I -tHdGZHCp95ohqpCPgQIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQCdvYj6CD0ZLwT2 -3t1r+deC3TJuHlNVSeKeT7wIfFnh2FW5riGV0q/w6eXPLTHjuiS6YmpAAbdNUgX/ -sq64FqI2RLpX6pgY5yB0SKopMcJxMLKqmF4zHpIHxtYN5EmN3PR0vehneBR/nZ2T -3ikvWD5JeXlm7Dfw+tjijdxM/sEoDWErGup4mMKMd1s5s830p+ITJUa50d0DLFdH -mqPSbUZF8mMPwGJd+nu1Ht3gTLtK7+gYJgGtXMJmGC0Qg77EJHDB2NbotgDGNmSU -1H9BpAeFHHIcbh2Rr7kkTvnh/c03vFe+CsDZmezcmRpRzW1fKj3YbfqBxU4XwJrL -a5T/N9xU ------END CERTIFICATE----- diff --git a/test/fixtures/server.key b/test/fixtures/server.key deleted file mode 100644 index a8447391e0..0000000000 --- a/test/fixtures/server.key +++ /dev/null @@ -1,27 +0,0 @@ ------BEGIN RSA PRIVATE KEY----- -MIIEowIBAAKCAQEA6S1E2WchgmbJYqCnpN7310ZgHjIOqeJe6MpSue2u6z6mTNd5 -izgvQNaANmn3xLFCS5zsuZaTvdPYPkcmSQzb1YcZSUYnAxZifjYARc6kb5GSBl3q -+O70ELyFrimXfZ4JI+bdIG9KiHY17DlvZZZj/csGYVWWg0mkeH3O5LPX6/DXQVh/ -9+gZ02/cdIBCAtZHQwqx7tF/qZA/kD4GZNFpU1DYHzf9H6g9htoCqmNHQWrV2T9y -Fybt6mbZp9kglBmyKYCc7hmQnb7N/mHn1yIuwhBsirCJTfKH86gN81u8M3+SVHA2 -VUHDllcNhpDWlmInXA+ItHdGZHCp95ohqpCPgQIDAQABAoIBABW8R4ewalo6dJlB -+n6O3jFt+PW3mtBRLqGqgm2cb0q0a1IMX+MPWLBFjmwEErl+AH0F4rcmBx2Ryr17 -amEy1qcf0caXyHksNAApznqzWXag7iizxnxv4cZRnHBwphNqkNWM5p3oYd04j6w2 -amDg1O9KZozaKo6QZclpiMiezwjKG+PVZLT8p7afswjv+yDWPDByhlcGiye9QD1T -VuZ0QCoXp6N/8JxW0gdkLp9NqFvGeGFzJ5h6L+d7A6BWw8akXrBRHHcKkyvVYBfd -myhSzSK4FPFMaxaEY/65FlVSyAO6ezGm3Umx4g7mkFjLdwKWaIOjkBkPeFgl3Pp4 -7Lo5X3UCgYEA/FrrIwmEU5ayulBVScEMKeavu5eNY4r0Sqbpov2oyTdYe8G49Pzy -ryMXfunY43moLKpajGwgTKRGvdqFtK08AAkaCssiAPkP3rZuZvMTF4sLo/vlWrjP -3er+tUqj22BzXi5XV0BAvH8Y3TL8KQ3he/8JxDvkC811/DQ9Y/Da3U8CgYEA7Itw -UM37URma08Bj9VTMoL9ZCyURewX+ZLDb2+O8sXGXJs28i1RkE6PTBlnRmedn+Jjk -byzQ5Cs5wA5uMbhYTA7kgXOs1bvgQqmlLmyL6FfHkucoMhr2Di7VeGf4OxE26JZ8 -JdY4+1MOyI3A2rR8WU+GmHxy0ay4K2xe6W0vsi8CgYBoGLEKIPDe8jkDtgOYivOT -jT9MaLXALB+dc8DIpU4swpHTaxP6qyUIrbcReTEolJSU6Ci16BxiwRkVU8D3yMYJ -VbfSX/zE3fh37FUaToa/nXHN0SjJBZdpeXhcHE//PIgaf48zxKNvnhYJmPB/luQ+ -m/PRaMsnOzfCM2JniYEe7QKBgGwjnxhB4tgDtaWCue/pcZc3gzS2IJS2e8N6mzie -l6Ajhu+FdOHZldrotUuc+la61OxwsVYmDeWR4VftAPGYDj3PPSX1RRl9R5wSRGLB -2wBASQvew6CMdNqtDIh8N56BUzHnwh/mHKzBHuwO6hDSHFsUITtLAY7bwGKRq55Z -fUmfAoGBANOYFyoJoDLcl+Jd750lyqfCifcGtkRdmZMtrPXaYnD8ZGme9vz1vsK/ -4iUkV3mi7Z9s1LXMa/tPPfKdVhCM1PXost3/si0+u1Bz5yKqEPXlyy2ltpIVyGu8 -yiy7y75asp8Iii/1cgtwyp9+VeSif8wJ+MHQoGdGxvAQP80R3EjF ------END RSA PRIVATE KEY----- diff --git a/test/test-download.js b/test/test-download.js index 71a3c0d092..c4caad9e83 100644 --- a/test/test-download.js +++ b/test/test-download.js @@ -12,6 +12,7 @@ const devDir = require('./common').devDir() const rimraf = require('rimraf') const gyp = require('../lib/node-gyp') const log = require('npmlog') +const certs = require('./fixtures/certs') log.level = 'warn' @@ -40,12 +41,12 @@ test('download over http', async (t) => { test('download over https with custom ca', async (t) => { t.plan(3) - const cafile = path.join(__dirname, '/fixtures/ca.crt') - const [cert, key, ca] = await Promise.all([ - fs.promises.readFile(path.join(__dirname, 'fixtures/server.crt'), 'utf8'), - fs.promises.readFile(path.join(__dirname, 'fixtures/server.key'), 'utf8'), - install.test.readCAFile(cafile) - ]) + const cafile = path.join(__dirname, 'fixtures/ca.crt') + const cacontents = certs['ca.crt'] + const cert = certs['server.crt'] + const key = certs['server.key'] + await fs.promises.writeFile(cafile, cacontents, 'utf8') + const ca = await install.test.readCAFile(cafile) t.strictEqual(ca.length, 1) @@ -55,7 +56,10 @@ test('download over https with custom ca', async (t) => { res.end('ok') }) - t.tearDown(() => new Promise((resolve) => server.close(resolve))) + t.tearDown(async () => { + await new Promise((resolve) => server.close(resolve)) + await fs.promises.unlink(cafile) + }) server.on('clientError', (err) => { throw err }) @@ -150,6 +154,12 @@ test('download with missing cafile', async (t) => { }) test('check certificate splitting', async (t) => { + const cafile = path.join(__dirname, 'fixtures/ca-bundle.crt') + const cacontents = certs['ca-bundle.crt'] + await fs.promises.writeFile(cafile, cacontents, 'utf8') + t.tearDown(async () => { + await fs.promises.unlink(cafile) + }) const cas = await install.test.readCAFile(path.join(__dirname, 'fixtures/ca-bundle.crt')) t.plan(2) t.strictEqual(cas.length, 2) From 6f74c762fe3c19bdd20245cb5c02e2dfa65d9451 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Thu, 12 May 2022 13:00:36 +1000 Subject: [PATCH 336/551] fix: extend tap timeout length to allow for slow CI --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index e795db1834..7ff4d6ff67 100644 --- a/package.json +++ b/package.json @@ -45,6 +45,6 @@ }, "scripts": { "lint": "standard */*.js test/**/*.js", - "test": "npm run lint && tap --timeout=120 test/test-*" + "test": "npm run lint && tap --timeout=600 test/test-*" } } From b9ddcd5bbd93b05b03674836b6ebdae2c2e74c8c Mon Sep 17 00:00:00 2001 From: DeeDeeG Date: Fri, 10 Jun 2022 08:11:23 -0400 Subject: [PATCH 337/551] Add Python symlink to path (for non-Windows OSes only) (#2362) * lib: create a Python symlink and add it to PATH Helps to ensure a version of Python validated by lib/find-python.js is used to run various Python scripts generated by gyp. Known to affect gyp-mac-tool, probably affects gyp-flock-tool as well. These Python scripts (such as `gyp-mac-tool`) are invoked directly, via the generated Makefile, so their shebang lines determine which Python binary is used to run them. The shebang lines of these scripts are all `#!/usr/bin/env python3`, so the first `python3` on the user's PATH will be used. By adding a symlink to the Python binary validated by find-python.js, and putting this symlink first on the PATH, we can ensure we use a compatible version of Python to run these scripts. (Only on Unix/Unix-like OSes. Symlinks are tricky on Windows, and Python isn't used at build-time anyhow on Windows, so this intervention isn't useful or necessary on Windows. A similar technique for Windows, no symlinks required, would be to make batch scripts which execute the target binary, much like what Node does for its bundled copy of npm on Windows.) * test: update mocked graceful-fs for configure test Add missing functions "unlink()" and "symlink()" to mocked module. * lib: log any errors when creating Python symlink Warn users about errors, but continue on in case the user does happen to have new enough Python on their PATH. (The symlinks are only meant to fix an issue in a corner case, where the user told `node-gyp` where new enough Python is, but it's not the first `python3` on their PATH. We should not introduce a new potential failure mode to all users when fixing this bug. So no hard errors during the symlink process.) * lib: improve error formatting for Python symlink Logging the entire error object shows the stack twice, and all the other information is contained in the stack. It also messes with the order of what is logged. Rather than logging a bunch of redundant information in a messy way, we can log only the stack. Logging it in a separate log.warn() also gets rid of an extra space character at the beginning of the line. * lib: restore err.errno to logs for symlink errors This info (err.errno) is the only piece of information in the error object that is not redundant to err.stack. * lib: use log.verbose, not log.warn These messages aren't important enough to be `log.warn`s. Log as verbose only; they will also appear in full error output. --- lib/build.js | 7 +++++++ lib/configure.js | 25 ++++++++++++++++++++++++- test/test-configure-python.js | 4 +++- 3 files changed, 34 insertions(+), 2 deletions(-) diff --git a/lib/build.js b/lib/build.js index c2388fb348..3baba4140c 100644 --- a/lib/build.js +++ b/lib/build.js @@ -185,6 +185,13 @@ function build (gyp, argv, callback) { } } + if (!win) { + // Add build-time dependency symlinks (such as Python) to PATH + const buildBinsDir = path.resolve('build', 'node_gyp_bins') + process.env.PATH = `${buildBinsDir}:${process.env.PATH}` + log.verbose('bin symlinks', `adding symlinks (such as Python), at "${buildBinsDir}", to PATH`) + } + var proc = gyp.spawn(command, argv) proc.on('exit', onExit) } diff --git a/lib/configure.js b/lib/configure.js index 17a6487fa9..c7010385b5 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -17,6 +17,7 @@ if (win) { function configure (gyp, argv, callback) { var python var buildDir = path.resolve('build') + var buildBinsDir = path.join(buildDir, 'node_gyp_bins') var configNames = ['config.gypi', 'common.gypi'] var configs = [] var nodeDir @@ -73,7 +74,9 @@ function configure (gyp, argv, callback) { function createBuildDir () { log.verbose('build dir', 'attempting to create "build" dir: %s', buildDir) - fs.mkdir(buildDir, { recursive: true }, function (err, isNew) { + + const deepestBuildDirSubdirectory = win ? buildDir : buildBinsDir + fs.mkdir(deepestBuildDirSubdirectory, { recursive: true }, function (err, isNew) { if (err) { return callback(err) } @@ -84,11 +87,31 @@ function configure (gyp, argv, callback) { findVisualStudio(release.semver, gyp.opts.msvs_version, createConfigFile) } else { + createPythonSymlink() createConfigFile() } }) } + function createPythonSymlink () { + const symlinkDestination = path.join(buildBinsDir, 'python3') + + log.verbose('python symlink', `creating symlink to "${python}" at "${symlinkDestination}"`) + + fs.unlink(symlinkDestination, function (err) { + if (err && err.code !== 'ENOENT') { + log.verbose('python symlink', 'error when attempting to remove existing symlink') + log.verbose('python symlink', err.stack, 'errno: ' + err.errno) + } + fs.symlink(python, symlinkDestination, function (err) { + if (err) { + log.verbose('python symlink', 'error when attempting to create Python symlink') + log.verbose('python symlink', err.stack, 'errno: ' + err.errno) + } + }) + }) + } + function createConfigFile (err, vsInfo) { if (err) { return callback(err) diff --git a/test/test-configure-python.js b/test/test-configure-python.js index 4290e7af1b..aacd75f7c7 100644 --- a/test/test-configure-python.js +++ b/test/test-configure-python.js @@ -14,7 +14,9 @@ const configure = requireInject('../lib/configure', { mkdir: function (dir, options, cb) { cb() }, promises: { writeFile: function (file, data) { return Promise.resolve(null) } - } + }, + unlink: function (path, cb) { cb() }, + symlink: function (target, path, cb) { cb() } } }) From d7687d55666fa77928cce270b8991b8e819c5094 Mon Sep 17 00:00:00 2001 From: Nick Wang Date: Fri, 10 Jun 2022 08:13:05 -0400 Subject: [PATCH 338/551] Clarify wording to redirect to macOS_Catalina.md (#2588) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 48bf763943..7636ad5482 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,7 @@ Depending on your operating system, you will need to install: ### On macOS -**ATTENTION**: If your Mac has been _upgraded_ to macOS Catalina (10.15), please read [macOS_Catalina.md](macOS_Catalina.md). +**ATTENTION**: If your Mac has been _upgraded_ to macOS Catalina (10.15) or higher, please read [macOS_Catalina.md](macOS_Catalina.md). * Python v3.7, v3.8, v3.9, or v3.10 * `XCode Command Line Tools` which will install `clang`, `clang++`, and `make`. From ca1f0681a5567ca8cd51acebccd37a633f19bc6a Mon Sep 17 00:00:00 2001 From: Michael Dawson Date: Wed, 15 Jun 2022 11:57:51 -0400 Subject: [PATCH 339/551] build: update due to rename of primary branch --- .github/workflows/release-please.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml index 7d3cf9dd45..c3057c3a31 100644 --- a/.github/workflows/release-please.yml +++ b/.github/workflows/release-please.yml @@ -3,7 +3,7 @@ name: release-please on: push: branches: - - master + - main jobs: release-please: From 9778dd002466a329a4e348e3dd24c3c11b260336 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Wed, 22 Jun 2022 07:51:10 +0200 Subject: [PATCH 340/551] Migrate macOS acid test from master to main (#2686) Follow-on to #2495 --- macOS_Catalina.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/macOS_Catalina.md b/macOS_Catalina.md index 4fe0f29b21..dde5fe3f7d 100644 --- a/macOS_Catalina.md +++ b/macOS_Catalina.md @@ -37,7 +37,7 @@ If `ProductVersion` is less then `10.15` then this document is not for you. Norm ### The acid test To see if `Xcode Command Line Tools` is installed in a way that will work with `node-gyp`, run: ``` -curl -sL https://github.com/nodejs/node-gyp/raw/master/macOS_Catalina_acid_test.sh | bash +curl -sL https://github.com/nodejs/node-gyp/raw/main/macOS_Catalina_acid_test.sh | bash ``` If test succeeded, _you are done_! You should be ready to [install](https://github.com/nodejs/node-gyp#installation) `node-gyp`. From ea8520e3855374bd15b6d001fe112d58a8d7d737 Mon Sep 17 00:00:00 2001 From: hubbergit Date: Wed, 13 Jul 2022 10:00:11 +0200 Subject: [PATCH 341/551] feat: Update function getSDK() to support Windows 11 SDK (#2565) --- lib/find-visualstudio.js | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js index 64af7be346..8a5cfc1ea9 100644 --- a/lib/find-visualstudio.js +++ b/lib/find-visualstudio.js @@ -314,29 +314,30 @@ VisualStudioFinder.prototype = { getSDK: function getSDK (info) { const win8SDK = 'Microsoft.VisualStudio.Component.Windows81SDK' const win10SDKPrefix = 'Microsoft.VisualStudio.Component.Windows10SDK.' + const win11SDKPrefix = 'Microsoft.VisualStudio.Component.Windows11SDK.' - var Win10SDKVer = 0 + var Win10or11SDKVer = 0 info.packages.forEach((pkg) => { - if (!pkg.startsWith(win10SDKPrefix)) { + if (!pkg.startsWith(win10SDKPrefix) && !pkg.startsWith(win11SDKPrefix)) { return } const parts = pkg.split('.') if (parts.length > 5 && parts[5] !== 'Desktop') { - this.log.silly('- ignoring non-Desktop Win10SDK:', pkg) + this.log.silly('- ignoring non-Desktop Win10/11SDK:', pkg) return } const foundSdkVer = parseInt(parts[4], 10) if (isNaN(foundSdkVer)) { // Microsoft.VisualStudio.Component.Windows10SDK.IpOverUsb - this.log.silly('- failed to parse Win10SDK number:', pkg) + this.log.silly('- failed to parse Win10/11SDK number:', pkg) return } - this.log.silly('- found Win10SDK:', foundSdkVer) - Win10SDKVer = Math.max(Win10SDKVer, foundSdkVer) + this.log.silly('- found Win10/11SDK:', foundSdkVer) + Win10or11SDKVer = Math.max(Win10or11SDKVer, foundSdkVer) }) - if (Win10SDKVer !== 0) { - return `10.0.${Win10SDKVer}.0` + if (Win10or11SDKVer !== 0) { + return `10.0.${Win10or11SDKVer}.0` } else if (info.packages.indexOf(win8SDK) !== -1) { this.log.silly('- found Win8SDK') return '8.1' From 1c64ca7f4702c6eb43ecd16fbd67b5d939041621 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Wed, 13 Jul 2022 10:11:12 +0200 Subject: [PATCH 342/551] test: Upgrade GitHub Actions (#2701) * test: Upgrade GitHub Actions * node: 18x --> 18.x --- .github/workflows/tests.yml | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 83ccf7f547..a3b68bdd5d 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -1,45 +1,52 @@ -# TODO: Line 43, enable pytest --doctest-modules +# https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources +# TODO: Line 48, enable pytest --doctest-modules name: Tests -on: [push, pull_request] +on: + push: + branches: [ main ] + pull_request: + branches: [ main ] jobs: Tests: strategy: fail-fast: false max-parallel: 15 matrix: - node: [12.x, 14.x, 16.x] + node: [14.x, 16.x, 18.x] python: ["3.6", "3.8", "3.10"] os: [macos-latest, ubuntu-latest, windows-latest] runs-on: ${{ matrix.os }} steps: - name: Checkout Repository - uses: actions/checkout@v2 + uses: actions/checkout@v3 - name: Use Node.js ${{ matrix.node }} uses: actions/setup-node@v3 with: node-version: ${{ matrix.node }} - name: Use Python ${{ matrix.python }} - uses: actions/setup-python@v3 + uses: actions/setup-python@v4 with: python-version: ${{ matrix.python }} env: - PYTHON_VERSION: ${{ matrix.python }} + PYTHON_VERSION: ${{ matrix.python }} # Why do this? - name: Install Dependencies run: | npm install --no-progress pip install flake8 pytest - name: Set Windows environment - if: matrix.os == 'windows-latest' + if: startsWith(matrix.os, 'windows') run: | echo 'GYP_MSVS_VERSION=2015' >> $Env:GITHUB_ENV echo 'GYP_MSVS_OVERRIDE_PATH=C:\\Dummy' >> $Env:GITHUB_ENV - name: Lint Python - if: matrix.os == 'ubuntu-latest' + if: startsWith(matrix.os, 'ubuntu') run: flake8 . --ignore=E203,W503 --max-complexity=101 --max-line-length=88 --show-source --statistics - name: Run Python tests run: python -m pytest # - name: Run doctests with pytest # run: python -m pytest --doctest-modules + - name: Environment Information + run: npx envinfo - name: Run Node tests run: npm test From 68b5b5be9c94ac20c55e88654ff6f55234d7130a Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Wed, 13 Jul 2022 10:11:32 +0200 Subject: [PATCH 343/551] test: Try msvs-version: [2016, 2019, 2022] (#2700) * test: Try msvs-version: [2016, 2019, 2022] * main, not master * Don't npm audit fix --force --- .github/workflows/visual-studio.yml | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/.github/workflows/visual-studio.yml b/.github/workflows/visual-studio.yml index 6bb4574d63..12125e5447 100644 --- a/.github/workflows/visual-studio.yml +++ b/.github/workflows/visual-studio.yml @@ -1,23 +1,31 @@ -name: Tests on Windows -on: [push, pull_request] +# https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources + +name: visual-studio +on: + push: + branches: [ main ] + pull_request: + branches: [ main ] jobs: - Tests: + visual-studio: strategy: fail-fast: false - max-parallel: 15 + max-parallel: 8 matrix: - os: [windows-2022] + os: [windows-latest] + msvs-version: [2016, 2019, 2022] # https://github.com/actions/virtual-environments/tree/main/images/win runs-on: ${{ matrix.os }} steps: - name: Checkout Repository - uses: actions/checkout@v2 + uses: actions/checkout@v3 - name: Install Dependencies run: | npm install --no-progress + # npm audit fix --force - name: Set Windows environment - if: matrix.os == 'windows-latest' + if: startsWith(matrix.os, 'windows') run: | - echo 'GYP_MSVS_VERSION=2015' >> $Env:GITHUB_ENV + echo 'GYP_MSVS_VERSION=${{ matrix.msvs-version }}' >> $Env:GITHUB_ENV echo 'GYP_MSVS_OVERRIDE_PATH=C:\\Dummy' >> $Env:GITHUB_ENV - name: Environment Information run: npx envinfo From f0b7863dadfa365afc173025ae95351aec79abd9 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Wed, 13 Jul 2022 10:25:24 +0200 Subject: [PATCH 344/551] fix: re-label (#2689) --- docs/Updating-npm-bundled-node-gyp.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Updating-npm-bundled-node-gyp.md b/docs/Updating-npm-bundled-node-gyp.md index 1d91af6bb2..5759add3fe 100644 --- a/docs/Updating-npm-bundled-node-gyp.md +++ b/docs/Updating-npm-bundled-node-gyp.md @@ -4,7 +4,7 @@ **Note: These instructions will be undone if you reinstall or upgrade npm or node! For a more permanent (and simpler) solution, see [Force-npm-to-use-global-node-gyp.md](Force-npm-to-use-global-node-gyp.md). (npm 6 or older only!)** -[Many issues](https://github.com/nodejs/node-gyp/labels/ERR%21%20node-gyp%20-v%20%3C%3D%20v5.1.0) are opened by users who are +[Many issues](https://github.com/nodejs/node-gyp/issues?q=label%3A"ERR!+node-gyp+-v+<%3D+v9.x.x") are opened by users who are not running a [current version of node-gyp](https://github.com/nodejs/node-gyp/releases). `npm` bundles its own, internal, copy of `node-gyp`. This internal copy is independent of any globally installed copy of node-gyp that From 5f9d86d731af5f2efe1cdadc5461932e182dd9af Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed, 13 Jul 2022 08:25:56 +0000 Subject: [PATCH 345/551] chore: release 9.1.0 --- CHANGELOG.md | 32 ++++++++++++++++++++++++++++++++ package.json | 2 +- 2 files changed, 33 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7a474ed93f..694823fa32 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,37 @@ # Changelog +## [9.1.0](https://www.github.com/nodejs/node-gyp/compare/v9.0.0...v9.1.0) (2022-07-13) + + +### Features + +* Update function getSDK() to support Windows 11 SDK ([#2565](https://www.github.com/nodejs/node-gyp/issues/2565)) ([ea8520e](https://www.github.com/nodejs/node-gyp/commit/ea8520e3855374bd15b6d001fe112d58a8d7d737)) + + +### Bug Fixes + +* extend tap timeout length to allow for slow CI ([6f74c76](https://www.github.com/nodejs/node-gyp/commit/6f74c762fe3c19bdd20245cb5c02e2dfa65d9451)) +* new ca & server certs, bundle in .js file and unpack for testing ([147e3d3](https://www.github.com/nodejs/node-gyp/commit/147e3d34f44a97deb7aa507207680cf0f4e662a2)) +* re-label ([#2689](https://www.github.com/nodejs/node-gyp/issues/2689)) ([f0b7863](https://www.github.com/nodejs/node-gyp/commit/f0b7863dadfa365afc173025ae95351aec79abd9)) +* typo on readme ([bf81cd4](https://www.github.com/nodejs/node-gyp/commit/bf81cd452b931dd4dfa82762c23dd530a075d992)) + + +### Doc + +* update docs/README.md with latest version number ([62d2815](https://www.github.com/nodejs/node-gyp/commit/62d28151bf8266a34e1bcceeb25b4e6e2ae5ca5d)) + + +### Core + +* update due to rename of primary branch ([ca1f068](https://www.github.com/nodejs/node-gyp/commit/ca1f0681a5567ca8cd51acebccd37a633f19bc6a)) + + +### Tests + +* Try msvs-version: [2016, 2019, 2022] ([#2700](https://www.github.com/nodejs/node-gyp/issues/2700)) ([68b5b5b](https://www.github.com/nodejs/node-gyp/commit/68b5b5be9c94ac20c55e88654ff6f55234d7130a)) +* Upgrade GitHub Actions ([#2623](https://www.github.com/nodejs/node-gyp/issues/2623)) ([245cd5b](https://www.github.com/nodejs/node-gyp/commit/245cd5bbe4441d4f05e88f2fa20a86425419b6af)) +* Upgrade GitHub Actions ([#2701](https://www.github.com/nodejs/node-gyp/issues/2701)) ([1c64ca7](https://www.github.com/nodejs/node-gyp/commit/1c64ca7f4702c6eb43ecd16fbd67b5d939041621)) + ## [9.0.0](https://www.github.com/nodejs/node-gyp/compare/v8.4.1...v9.0.0) (2022-02-24) diff --git a/package.json b/package.json index 7ff4d6ff67..ecf4d8ae89 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "9.0.0", + "version": "9.1.0", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From 83c0a12bf23b4cbf3125d41f9e2d4201db76c9ae Mon Sep 17 00:00:00 2001 From: alexcfyung Date: Thu, 14 Jul 2022 22:13:27 -0400 Subject: [PATCH 346/551] lib: enable support for zoslib on z/OS (#2600) Check if zos-base.h is in the directory identified by environment variable ZOSLIB_INCLUDES if set; otherwise search for it from a set of candidates under nodeRootDir. Then pass it as -Dzoslib_include_dir= to gyp_main.py for use in common.gypi to set 'includes_dir' when compiling addons. Co-authored-by: Gaby Baghdadi Co-authored-by: Gaby Baghdadi --- lib/configure.js | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/lib/configure.js b/lib/configure.js index c7010385b5..9a2edb54a8 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -213,6 +213,44 @@ function configure (gyp, argv, callback) { } } + // For z/OS we need to set up the path to zoslib include directory, + // which contains headers included in v8config.h. + var zoslibIncDir + if (process.platform === 'os390') { + logprefix = "find zoslib's zos-base.h:" + let msg + var zoslibIncPath = process.env.ZOSLIB_INCLUDES + if (zoslibIncPath) { + zoslibIncPath = findAccessibleSync(logprefix, zoslibIncPath, ['zos-base.h']) + if (zoslibIncPath === undefined) { + msg = msgFormat('Could not find zos-base.h file in the directory set ' + + 'in ZOSLIB_INCLUDES environment variable: %s; set it ' + + 'to the correct path, or unset it to search %s', process.env.ZOSLIB_INCLUDES, nodeRootDir) + } + } else { + candidates = [ + 'include/node/zoslib/zos-base.h', + 'include/zoslib/zos-base.h', + 'zoslib/include/zos-base.h', + 'install/include/node/zoslib/zos-base.h' + ] + zoslibIncPath = findAccessibleSync(logprefix, nodeRootDir, candidates) + if (zoslibIncPath === undefined) { + msg = msgFormat('Could not find any of %s in directory %s; set ' + + 'environmant variable ZOSLIB_INCLUDES to the path ' + + 'that contains zos-base.h', candidates.toString(), nodeRootDir) + } + } + if (zoslibIncPath !== undefined) { + zoslibIncDir = path.dirname(zoslibIncPath) + log.verbose(logprefix, "Found zoslib's zos-base.h in: %s", zoslibIncDir) + } else if (release.version.split('.')[0] >= 16) { + // zoslib is only shipped in Node v16 and above. + log.error(logprefix, msg) + return callback(new Error(msg)) + } + } + // this logic ported from the old `gyp_addon` python file var gypScript = path.resolve(__dirname, '..', 'gyp', 'gyp_main.py') var addonGypi = path.resolve(__dirname, '..', 'addon.gypi') @@ -240,6 +278,9 @@ function configure (gyp, argv, callback) { argv.push('-Dnode_root_dir=' + nodeDir) if (process.platform === 'aix' || process.platform === 'os390') { argv.push('-Dnode_exp_file=' + nodeExpFile) + if (process.platform === 'os390' && zoslibIncDir) { + argv.push('-Dzoslib_include_dir=' + zoslibIncDir) + } } argv.push('-Dnode_gyp_dir=' + nodeGypDir) From 8958ecf2bb719227bbcbf155891c3186ee219a2e Mon Sep 17 00:00:00 2001 From: Gar Date: Fri, 22 Jul 2022 03:31:10 -0700 Subject: [PATCH 347/551] chore: update dependency - nopt@6.0.0 (#2707) No functional changes, just dropping old node versions from engines, linting, and fixing CI. --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index ecf4d8ae89..9f49299759 100644 --- a/package.json +++ b/package.json @@ -26,7 +26,7 @@ "glob": "^7.1.4", "graceful-fs": "^4.2.6", "make-fetch-happen": "^10.0.3", - "nopt": "^5.0.0", + "nopt": "^6.0.0", "npmlog": "^6.0.0", "rimraf": "^3.0.2", "semver": "^7.3.5", From c379a744c65c7ab07c2c3193d9c7e8f25ae1b05e Mon Sep 17 00:00:00 2001 From: "Mr. Doge" <42662615+FuPeiJiang@users.noreply.github.com> Date: Mon, 22 Aug 2022 10:19:49 -0400 Subject: [PATCH 348/551] fix: node.js debugger adds stderr (but exit code is 0) -> shouldn't throw (#2719) * fix: node.js debugger adds stderr (but exit code is 0) -> shouldn't throw * input.py: subprocess.Popen() -> subprocess.run() --- gyp/pylib/gyp/input.py | 21 +++++++-------------- 1 file changed, 7 insertions(+), 14 deletions(-) diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index 354958bfb2..d9699a0a50 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -961,13 +961,13 @@ def ExpandVariables(input, phase, variables, build_file): # Fix up command with platform specific workarounds. contents = FixupPlatformCommand(contents) try: - p = subprocess.Popen( + # stderr will be printed no matter what + result = subprocess.run( contents, - shell=use_shell, stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - stdin=subprocess.PIPE, + shell=use_shell, cwd=build_file_dir, + check=False ) except Exception as e: raise GypError( @@ -975,19 +975,12 @@ def ExpandVariables(input, phase, variables, build_file): % (e, contents, build_file) ) - p_stdout, p_stderr = p.communicate("") - p_stdout = p_stdout.decode("utf-8") - p_stderr = p_stderr.decode("utf-8") - - if p.wait() != 0 or p_stderr: - sys.stderr.write(p_stderr) - # Simulate check_call behavior, since check_call only exists - # in python 2.5 and later. + if result.returncode > 0: raise GypError( "Call to '%s' returned exit status %d while in %s." - % (contents, p.returncode, build_file) + % (contents, result.returncode, build_file) ) - replacement = p_stdout.rstrip() + replacement = result.stdout.decode("utf-8").rstrip() cached_command_results[cache_key] = replacement else: From 3e2a5324f1c24f3a04bca04cf54fe23d5c4d5e50 Mon Sep 17 00:00:00 2001 From: Kevin Adler Date: Thu, 18 Aug 2022 10:16:04 -0500 Subject: [PATCH 349/551] feat(gyp): update gyp to v0.13.0 --- gyp/.github/workflows/Python_tests.yml | 6 +- gyp/.github/workflows/node-gyp.yml | 10 +- gyp/.github/workflows/nodejs-windows.yml | 4 +- gyp/CHANGELOG.md | 40 +++++ gyp/pylib/gyp/__init__.py | 12 ++ gyp/pylib/gyp/common.py | 12 +- gyp/pylib/gyp/flock_tool.py | 2 +- gyp/pylib/gyp/generator/make.py | 217 ++++++++++++++++++++--- gyp/pylib/gyp/generator/msvs.py | 11 +- gyp/pylib/gyp/generator/ninja.py | 4 +- gyp/setup.py | 2 +- gyp/test_gyp.py | 1 + gyp/tools/pretty_gyp.py | 2 +- 13 files changed, 275 insertions(+), 48 deletions(-) diff --git a/gyp/.github/workflows/Python_tests.yml b/gyp/.github/workflows/Python_tests.yml index 92303b635f..1cfa42f563 100644 --- a/gyp/.github/workflows/Python_tests.yml +++ b/gyp/.github/workflows/Python_tests.yml @@ -11,11 +11,11 @@ jobs: max-parallel: 8 matrix: os: [macos-latest, ubuntu-latest] # , windows-latest] - python-version: [3.6, 3.7, 3.8, 3.9] + python-version: ["3.7", "3.8", "3.9", "3.10"] steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v3 - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v2 + uses: actions/setup-python@v3 with: python-version: ${{ matrix.python-version }} - name: Install dependencies diff --git a/gyp/.github/workflows/node-gyp.yml b/gyp/.github/workflows/node-gyp.yml index bd7c85ffda..fc28a0b512 100644 --- a/gyp/.github/workflows/node-gyp.yml +++ b/gyp/.github/workflows/node-gyp.yml @@ -8,23 +8,23 @@ jobs: fail-fast: false matrix: os: [macos-latest, ubuntu-latest, windows-latest] - python: [3.6, 3.9] + python: ["3.7", "3.10"] runs-on: ${{ matrix.os }} steps: - name: Clone gyp-next - uses: actions/checkout@v2 + uses: actions/checkout@v3 with: path: gyp-next - name: Clone nodejs/node-gyp - uses: actions/checkout@v2 + uses: actions/checkout@v3 with: repository: nodejs/node-gyp path: node-gyp - - uses: actions/setup-node@v2 + - uses: actions/setup-node@v3 with: node-version: 14.x - - uses: actions/setup-python@v2 + - uses: actions/setup-python@v3 with: python-version: ${{ matrix.python }} - name: Install dependencies diff --git a/gyp/.github/workflows/nodejs-windows.yml b/gyp/.github/workflows/nodejs-windows.yml index fffe96e33b..53bd736727 100644 --- a/gyp/.github/workflows/nodejs-windows.yml +++ b/gyp/.github/workflows/nodejs-windows.yml @@ -7,11 +7,11 @@ jobs: runs-on: windows-latest steps: - name: Clone gyp-next - uses: actions/checkout@v2 + uses: actions/checkout@v3 with: path: gyp-next - name: Clone nodejs/node - uses: actions/checkout@v2 + uses: actions/checkout@v3 with: repository: nodejs/node path: node diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md index b7d55ed655..a103250cd5 100644 --- a/gyp/CHANGELOG.md +++ b/gyp/CHANGELOG.md @@ -1,5 +1,45 @@ # Changelog +## [0.13.0](https://www.github.com/nodejs/gyp-next/compare/v0.12.1...v0.13.0) (2022-05-11) + + +### Features + +* add PRODUCT_DIR_ABS variable ([#151](https://www.github.com/nodejs/gyp-next/issues/151)) ([80d2626](https://www.github.com/nodejs/gyp-next/commit/80d26263581db829b61b312a7bdb5cc791df7824)) + + +### Bug Fixes + +* execvp: printf: Argument list too long ([#147](https://www.github.com/nodejs/gyp-next/issues/147)) ([c4e14f3](https://www.github.com/nodejs/gyp-next/commit/c4e14f301673fadbac3ab7882d0b5f4d02530cb9)) + +### [0.12.1](https://www.github.com/nodejs/gyp-next/compare/v0.12.0...v0.12.1) (2022-04-06) + + +### Bug Fixes + +* **msvs:** avoid fixing path for arguments with "=" ([#143](https://www.github.com/nodejs/gyp-next/issues/143)) ([7e8f16e](https://www.github.com/nodejs/gyp-next/commit/7e8f16eb165e042e64bec98fa6c2a0232a42c26b)) + +## [0.12.0](https://www.github.com/nodejs/gyp-next/compare/v0.11.0...v0.12.0) (2022-04-04) + + +### Features + +* support building shared libraries on z/OS ([#137](https://www.github.com/nodejs/gyp-next/issues/137)) ([293bcfa](https://www.github.com/nodejs/gyp-next/commit/293bcfa4c25c6adb743377adafc45a80fee492c6)) + +## [0.11.0](https://www.github.com/nodejs/gyp-next/compare/v0.10.1...v0.11.0) (2022-03-04) + + +### Features + +* Add proper support for IBM i ([#140](https://www.github.com/nodejs/gyp-next/issues/140)) ([fdda4a3](https://www.github.com/nodejs/gyp-next/commit/fdda4a3038b8a7042ad960ce7a223687c24a21b1)) + +### [0.10.1](https://www.github.com/nodejs/gyp-next/compare/v0.10.0...v0.10.1) (2021-11-24) + + +### Bug Fixes + +* **make:** only generate makefile for multiple toolsets if requested ([#133](https://www.github.com/nodejs/gyp-next/issues/133)) ([f463a77](https://www.github.com/nodejs/gyp-next/commit/f463a77705973289ea38fec1b244c922ac438e26)) + ## [0.10.0](https://www.github.com/nodejs/gyp-next/compare/v0.9.6...v0.10.0) (2021-08-26) diff --git a/gyp/pylib/gyp/__init__.py b/gyp/pylib/gyp/__init__.py index 6790ef96a1..976d5b6aa8 100755 --- a/gyp/pylib/gyp/__init__.py +++ b/gyp/pylib/gyp/__init__.py @@ -103,6 +103,18 @@ def Load( for (key, val) in generator.generator_default_variables.items(): default_variables.setdefault(key, val) + output_dir = params["options"].generator_output or params["options"].toplevel_dir + if default_variables["GENERATOR"] == "ninja": + default_variables.setdefault( + "PRODUCT_DIR_ABS", + os.path.join(output_dir, "out", default_variables["build_type"]), + ) + else: + default_variables.setdefault( + "PRODUCT_DIR_ABS", + os.path.join(output_dir, default_variables["CONFIGURATION_NAME"]), + ) + # Give the generator the opportunity to set additional variables based on # the params it will receive in the output phase. if getattr(generator, "CalculateVariables", None): diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py index 9213fcc5e8..0847cdabc7 100644 --- a/gyp/pylib/gyp/common.py +++ b/gyp/pylib/gyp/common.py @@ -454,6 +454,8 @@ def GetFlavor(params): return "aix" if sys.platform.startswith(("os390", "zos")): return "zos" + if sys.platform == "os400": + return "os400" return "linux" @@ -463,9 +465,13 @@ def CopyTool(flavor, out_path, generator_flags={}): to |out_path|.""" # aix and solaris just need flock emulation. mac and win use more complicated # support scripts. - prefix = {"aix": "flock", "solaris": "flock", "mac": "mac", "win": "win"}.get( - flavor, None - ) + prefix = { + "aix": "flock", + "os400": "flock", + "solaris": "flock", + "mac": "mac", + "win": "win", + }.get(flavor, None) if not prefix: return diff --git a/gyp/pylib/gyp/flock_tool.py b/gyp/pylib/gyp/flock_tool.py index 1cb9815263..0754aff26f 100755 --- a/gyp/pylib/gyp/flock_tool.py +++ b/gyp/pylib/gyp/flock_tool.py @@ -41,7 +41,7 @@ def ExecFlock(self, lockfile, *cmd_list): # with EBADF, that's why we use this F_SETLK # hack instead. fd = os.open(lockfile, os.O_WRONLY | os.O_NOCTTY | os.O_CREAT, 0o666) - if sys.platform.startswith("aix"): + if sys.platform.startswith("aix") or sys.platform == "os400": # Python on AIX is compiled with LARGEFILE support, which changes the # struct size. op = struct.pack("hhIllqq", fcntl.F_WRLCK, 0, 0, 0, 0, 0, 0) diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index c595f20fe2..e225326e1d 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -50,7 +50,7 @@ } # Make supports multiple toolsets -generator_supports_multiple_toolsets = True +generator_supports_multiple_toolsets = gyp.common.CrossCompileRequested() # Request sorted dependencies in the order from dependents to dependencies. generator_wants_sorted_dependencies = False @@ -99,6 +99,8 @@ def CalculateVariables(default_variables, params): default_variables.setdefault("OS", operating_system) if flavor == "aix": default_variables.setdefault("SHARED_LIB_SUFFIX", ".a") + elif flavor == "zos": + default_variables.setdefault("SHARED_LIB_SUFFIX", ".x") else: default_variables.setdefault("SHARED_LIB_SUFFIX", ".so") default_variables.setdefault("SHARED_LIB_DIR", "$(builddir)/lib.$(TOOLSET)") @@ -154,6 +156,31 @@ def CalculateGeneratorInputInfo(params): quiet_cmd_link = LINK($(TOOLSET)) $@ cmd_link = $(LINK.$(TOOLSET)) -o $@ $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -Wl,--start-group $(LD_INPUTS) $(LIBS) -Wl,--end-group +# Note: this does not handle spaces in paths +define xargs + $(1) $(word 1,$(2)) +$(if $(word 2,$(2)),$(call xargs,$(1),$(wordlist 2,$(words $(2)),$(2)))) +endef + +define write-to-file + @: >$(1) +$(call xargs,@printf "%s\\n" >>$(1),$(2)) +endef + +OBJ_FILE_LIST := ar-file-list + +define create_archive + rm -f $(1) $(1).$(OBJ_FILE_LIST); mkdir -p `dirname $(1)` + $(call write-to-file,$(1).$(OBJ_FILE_LIST),$(filter %.o,$(2))) + $(AR.$(TOOLSET)) crs $(1) @$(1).$(OBJ_FILE_LIST) +endef + +define create_thin_archive + rm -f $(1) $(OBJ_FILE_LIST); mkdir -p `dirname $(1)` + $(call write-to-file,$(1).$(OBJ_FILE_LIST),$(filter %.o,$(2))) + $(AR.$(TOOLSET)) crsT $(1) @$(1).$(OBJ_FILE_LIST) +endef + # We support two kinds of shared objects (.so): # 1) shared_library, which is just bundling together many dependent libraries # into a link line. @@ -198,6 +225,31 @@ def CalculateGeneratorInputInfo(params): quiet_cmd_alink_thin = AR($(TOOLSET)) $@ cmd_alink_thin = rm -f $@ && $(AR.$(TOOLSET)) crsT $@ $(filter %.o,$^) +# Note: this does not handle spaces in paths +define xargs + $(1) $(word 1,$(2)) +$(if $(word 2,$(2)),$(call xargs,$(1),$(wordlist 2,$(words $(2)),$(2)))) +endef + +define write-to-file + @: >$(1) +$(call xargs,@printf "%s\\n" >>$(1),$(2)) +endef + +OBJ_FILE_LIST := ar-file-list + +define create_archive + rm -f $(1) $(1).$(OBJ_FILE_LIST); mkdir -p `dirname $(1)` + $(call write-to-file,$(1).$(OBJ_FILE_LIST),$(filter %.o,$(2))) + $(AR.$(TOOLSET)) crs $(1) @$(1).$(OBJ_FILE_LIST) +endef + +define create_thin_archive + rm -f $(1) $(OBJ_FILE_LIST); mkdir -p `dirname $(1)` + $(call write-to-file,$(1).$(OBJ_FILE_LIST),$(filter %.o,$(2))) + $(AR.$(TOOLSET)) crsT $(1) @$(1).$(OBJ_FILE_LIST) +endef + # Due to circular dependencies between libraries :(, we wrap the # special "figure out circular dependencies" flags around the entire # input list during linking. @@ -237,6 +289,24 @@ def CalculateGeneratorInputInfo(params): """ # noqa: E501 +LINK_COMMANDS_OS400 = """\ +quiet_cmd_alink = AR($(TOOLSET)) $@ +cmd_alink = rm -f $@ && $(AR.$(TOOLSET)) -X64 crs $@ $(filter %.o,$^) + +quiet_cmd_alink_thin = AR($(TOOLSET)) $@ +cmd_alink_thin = rm -f $@ && $(AR.$(TOOLSET)) -X64 crs $@ $(filter %.o,$^) + +quiet_cmd_link = LINK($(TOOLSET)) $@ +cmd_link = $(LINK.$(TOOLSET)) $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ $(LD_INPUTS) $(LIBS) + +quiet_cmd_solink = SOLINK($(TOOLSET)) $@ +cmd_solink = $(LINK.$(TOOLSET)) -shared $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ $(LD_INPUTS) $(LIBS) + +quiet_cmd_solink_module = SOLINK_MODULE($(TOOLSET)) $@ +cmd_solink_module = $(LINK.$(TOOLSET)) -shared $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ $(filter-out FORCE_DO_CMD, $^) $(LIBS) +""" # noqa: E501 + + LINK_COMMANDS_OS390 = """\ quiet_cmd_alink = AR($(TOOLSET)) $@ cmd_alink = rm -f $@ && $(AR.$(TOOLSET)) crs $@ $(filter %.o,$^) @@ -248,10 +318,10 @@ def CalculateGeneratorInputInfo(params): cmd_link = $(LINK.$(TOOLSET)) $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ $(LD_INPUTS) $(LIBS) quiet_cmd_solink = SOLINK($(TOOLSET)) $@ -cmd_solink = $(LINK.$(TOOLSET)) $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ $(LD_INPUTS) $(LIBS) -Wl,DLL +cmd_solink = $(LINK.$(TOOLSET)) $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -Wl,DLL -o $(patsubst %.x,%.so,$@) $(LD_INPUTS) $(LIBS) && if [ -f $(notdir $@) ]; then /bin/cp $(notdir $@) $@; else true; fi quiet_cmd_solink_module = SOLINK_MODULE($(TOOLSET)) $@ -cmd_solink_module = $(LINK.$(TOOLSET)) $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ $(filter-out FORCE_DO_CMD, $^) $(LIBS) -Wl,DLL +cmd_solink_module = $(LINK.$(TOOLSET)) $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ $(filter-out FORCE_DO_CMD, $^) $(LIBS) """ # noqa: E501 @@ -400,6 +470,9 @@ def CalculateGeneratorInputInfo(params): # send stderr to /dev/null to ignore messages when linking directories. cmd_copy = ln -f "$<" "$@" 2>/dev/null || (rm -rf "$@" && cp %(copy_archive_args)s "$<" "$@") +quiet_cmd_symlink = SYMLINK $@ +cmd_symlink = ln -sf "$<" "$@" + %(link_commands)s """ # noqa: E501 r""" @@ -981,12 +1054,20 @@ def WriteActions( # libraries, but until everything is made cross-compile safe, also use # target libraries. # TODO(piman): when everything is cross-compile safe, remove lib.target - self.WriteLn( - "cmd_%s = LD_LIBRARY_PATH=$(builddir)/lib.host:" - "$(builddir)/lib.target:$$LD_LIBRARY_PATH; " - "export LD_LIBRARY_PATH; " - "%s%s" % (name, cd_action, command) - ) + if self.flavor == "zos" or self.flavor == "aix": + self.WriteLn( + "cmd_%s = LIBPATH=$(builddir)/lib.host:" + "$(builddir)/lib.target:$$LIBPATH; " + "export LIBPATH; " + "%s%s" % (name, cd_action, command) + ) + else: + self.WriteLn( + "cmd_%s = LD_LIBRARY_PATH=$(builddir)/lib.host:" + "$(builddir)/lib.target:$$LD_LIBRARY_PATH; " + "export LD_LIBRARY_PATH; " + "%s%s" % (name, cd_action, command) + ) self.WriteLn() outputs = [self.Absolutify(o) for o in outputs] # The makefile rules are all relative to the top dir, but the gyp actions @@ -1480,6 +1561,8 @@ def ComputeOutputBasename(self, spec): target_prefix = "lib" if self.flavor == "aix": target_ext = ".a" + elif self.flavor == "zos": + target_ext = ".x" else: target_ext = ".so" elif self.type == "none": @@ -1560,6 +1643,14 @@ def ComputeDeps(self, spec): # link_deps.extend(spec.get('libraries', [])) return (gyp.common.uniquer(deps), gyp.common.uniquer(link_deps)) + def GetSharedObjectFromSidedeck(self, sidedeck): + """Return the shared object files based on sidedeck""" + return re.sub(r"\.x$", ".so", sidedeck) + + def GetUnversionedSidedeckFromSidedeck(self, sidedeck): + """Return the shared object files based on sidedeck""" + return re.sub(r"\.\d+\.x$", ".x", sidedeck) + def WriteDependencyOnExtraOutputs(self, target, extra_outputs): self.WriteMakeRule( [self.output_binary], @@ -1768,21 +1859,35 @@ def WriteTarget( self.flavor not in ("mac", "openbsd", "netbsd", "win") and not self.is_standalone_static_library ): - self.WriteDoCmd( - [self.output_binary], - link_deps, - "alink_thin", - part_of_all, - postbuilds=postbuilds, - ) + if self.flavor in ("linux", "android"): + self.WriteMakeRule( + [self.output_binary], + link_deps, + actions=["$(call create_thin_archive,$@,$^)"], + ) + else: + self.WriteDoCmd( + [self.output_binary], + link_deps, + "alink_thin", + part_of_all, + postbuilds=postbuilds, + ) else: - self.WriteDoCmd( - [self.output_binary], - link_deps, - "alink", - part_of_all, - postbuilds=postbuilds, - ) + if self.flavor in ("linux", "android"): + self.WriteMakeRule( + [self.output_binary], + link_deps, + actions=["$(call create_archive,$@,$^)"], + ) + else: + self.WriteDoCmd( + [self.output_binary], + link_deps, + "alink", + part_of_all, + postbuilds=postbuilds, + ) elif self.type == "shared_library": self.WriteLn( "%s: LD_INPUTS := %s" @@ -1798,6 +1903,17 @@ def WriteTarget( part_of_all, postbuilds=postbuilds, ) + # z/OS has a .so target as well as a sidedeck .x target + if self.flavor == "zos": + self.WriteLn( + "%s: %s" + % ( + QuoteSpaces( + self.GetSharedObjectFromSidedeck(self.output_binary) + ), + QuoteSpaces(self.output_binary), + ) + ) elif self.type == "loadable_module": for link_dep in link_deps: assert " " not in link_dep, ( @@ -1855,7 +1971,9 @@ def WriteTarget( else: file_desc = "executable" install_path = self._InstallableTargetInstallPath() - installable_deps = [self.output] + installable_deps = [] + if self.flavor != "zos": + installable_deps.append(self.output) if ( self.flavor == "mac" and "product_dir" not in spec @@ -1880,7 +1998,30 @@ def WriteTarget( comment="Copy this to the %s output path." % file_desc, part_of_all=part_of_all, ) - installable_deps.append(install_path) + if self.flavor != "zos": + installable_deps.append(install_path) + if self.flavor == "zos" and self.type == "shared_library": + # lib.target/libnode.so has a dependency on $(obj).target/libnode.so + self.WriteDoCmd( + [self.GetSharedObjectFromSidedeck(install_path)], + [self.GetSharedObjectFromSidedeck(self.output)], + "copy", + comment="Copy this to the %s output path." % file_desc, + part_of_all=part_of_all, + ) + # Create a symlink of libnode.x to libnode.version.x + self.WriteDoCmd( + [self.GetUnversionedSidedeckFromSidedeck(install_path)], + [install_path], + "symlink", + comment="Symlnk this to the %s output path." % file_desc, + part_of_all=part_of_all, + ) + # Place libnode.version.so and libnode.x symlink in lib.target dir + installable_deps.append(self.GetSharedObjectFromSidedeck(install_path)) + installable_deps.append( + self.GetUnversionedSidedeckFromSidedeck(install_path) + ) if self.output != self.alias and self.alias != self.target: self.WriteMakeRule( [self.alias], @@ -1888,7 +2029,18 @@ def WriteTarget( comment="Short alias for building this %s." % file_desc, phony=True, ) - if part_of_all: + if self.flavor == "zos" and self.type == "shared_library": + # Make sure that .x symlink target is run + self.WriteMakeRule( + ["all"], + [ + self.GetUnversionedSidedeckFromSidedeck(install_path), + self.GetSharedObjectFromSidedeck(install_path), + ], + comment='Add %s to "all" target.' % file_desc, + phony=True, + ) + elif part_of_all: self.WriteMakeRule( ["all"], [install_path], @@ -2184,6 +2336,9 @@ def _InstallableTargetInstallPath(self): # # Install all shared libs into a common directory (per toolset) for # # convenient access with LD_LIBRARY_PATH. # return "$(builddir)/lib.%s/%s" % (self.toolset, self.alias) + if self.flavor == "zos" and self.type == "shared_library": + return "$(builddir)/lib.%s/%s" % (self.toolset, self.alias) + return "$(builddir)/" + self.alias @@ -2351,6 +2506,16 @@ def CalculateMakefilePath(build_file, base_name): "flock_index": 2, } ) + elif flavor == "os400": + copy_archive_arguments = "-pPRf" + header_params.update( + { + "copy_archive_args": copy_archive_arguments, + "link_commands": LINK_COMMANDS_OS400, + "flock": "./gyp-flock-tool flock", + "flock_index": 2, + } + ) build_file, _, _ = gyp.common.ParseQualifiedTarget(target_list[0]) make_global_settings_array = data[build_file].get("make_global_settings", []) diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index 8308fa8433..fd95005784 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -423,12 +423,15 @@ def _BuildCommandLineForRuleRaw( command.insert(0, "call") # Fix the paths # TODO(quote): This is a really ugly heuristic, and will miss path fixing - # for arguments like "--arg=path" or "/opt:path". - # If the argument starts with a slash or dash, it's probably a command line - # switch + # for arguments like "--arg=path", arg=path, or "/opt:path". + # If the argument starts with a slash or dash, or contains an equal sign, + # it's probably a command line switch. # Return the path with forward slashes because the command using it might # not support backslashes. - arguments = [i if (i[:1] in "/-") else _FixPath(i, "/") for i in cmd[1:]] + arguments = [ + i if (i[:1] in "/-" or "=" in i) else _FixPath(i, "/") + for i in cmd[1:] + ] arguments = [i.replace("$(InputDir)", "%INPUTDIR%") for i in arguments] arguments = [MSVSSettings.FixVCMacroSlashes(i) for i in arguments] if quote_cmd: diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py index d173bf2299..3db3771ac9 100644 --- a/gyp/pylib/gyp/generator/ninja.py +++ b/gyp/pylib/gyp/generator/ninja.py @@ -2112,8 +2112,8 @@ class MEMORYSTATUSEX(ctypes.Structure): ctypes.windll.kernel32.GlobalMemoryStatusEx(ctypes.byref(stat)) # VS 2015 uses 20% more working set than VS 2013 and can consume all RAM - # on a 64 GB machine. - mem_limit = max(1, stat.ullTotalPhys // (5 * (2 ** 30))) # total / 5GB + # on a 64 GiB machine. + mem_limit = max(1, stat.ullTotalPhys // (5 * (2 ** 30))) # total / 5GiB hard_cap = max(1, int(os.environ.get("GYP_LINK_CONCURRENCY_MAX", 2 ** 32))) return min(mem_limit, hard_cap) elif sys.platform.startswith("linux"): diff --git a/gyp/setup.py b/gyp/setup.py index cf9d7d2e56..1bb6908dea 100644 --- a/gyp/setup.py +++ b/gyp/setup.py @@ -15,7 +15,7 @@ setup( name="gyp-next", - version="0.10.0", + version="0.13.0", description="A fork of the GYP build system for use in the Node.js projects", long_description=long_description, long_description_content_type="text/markdown", diff --git a/gyp/test_gyp.py b/gyp/test_gyp.py index 9ba264170f..b7bb956b8e 100755 --- a/gyp/test_gyp.py +++ b/gyp/test_gyp.py @@ -116,6 +116,7 @@ def main(argv=None): else: format_list = { "aix5": ["make"], + "os400": ["make"], "freebsd7": ["make"], "freebsd8": ["make"], "openbsd5": ["make"], diff --git a/gyp/tools/pretty_gyp.py b/gyp/tools/pretty_gyp.py index 4ffa444551..6eef3a1bbf 100755 --- a/gyp/tools/pretty_gyp.py +++ b/gyp/tools/pretty_gyp.py @@ -90,7 +90,7 @@ def count_braces(line): """ open_braces = ["[", "(", "{"] close_braces = ["]", ")", "}"] - closing_prefix_re = re.compile(r"(.*?[^\s\]\}\)]+.*?)([\]\}\)],?)\s*$") + closing_prefix_re = re.compile(r"[^\s\]\}\)]\s*[\]\}\)]+,?\s*$") cnt = 0 stripline = COMMENT_RE.sub(r"", line) stripline = QUOTE_RE.sub(r"''", stripline) From a26494fbb8883d9ef784503979e115dec3e2791e Mon Sep 17 00:00:00 2001 From: Kevin Adler Date: Thu, 3 Mar 2022 13:22:14 -0600 Subject: [PATCH 350/551] feat: Add proper support for IBM i Python 3.9 on IBM i now properly returns "os400" for sys.platform instead of claiming to be AIX as it did previously. While the IBM i PASE environment is compatible with AIX, it is a subset and has numerous differences which makes it beneficial to distinguish, however this means that it now needs explicit support here. --- addon.gypi | 5 +++++ lib/build.js | 2 ++ lib/configure.js | 8 ++++---- test/test-find-node-directory.js | 2 +- 4 files changed, 12 insertions(+), 5 deletions(-) diff --git a/addon.gypi b/addon.gypi index 9327b0d722..b3489e39bd 100644 --- a/addon.gypi +++ b/addon.gypi @@ -103,6 +103,11 @@ '-Wl,-bimport:<(node_exp_file)' ], }], + [ 'OS=="os400"', { + 'ldflags': [ + '-Wl,-bimport:<(node_exp_file)' + ], + }], [ 'OS=="zos"', { 'cflags': [ '-q64', diff --git a/lib/build.js b/lib/build.js index 3baba4140c..ea1f90652a 100644 --- a/lib/build.js +++ b/lib/build.js @@ -11,6 +11,8 @@ function build (gyp, argv, callback) { var platformMake = 'make' if (process.platform === 'aix') { platformMake = 'gmake' + } else if (process.platform === 'os400') { + platformMake = 'gmake' } else if (process.platform.indexOf('bsd') !== -1) { platformMake = 'gmake' } else if (win && argv.length > 0) { diff --git a/lib/configure.js b/lib/configure.js index 9a2edb54a8..1ca3ade709 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -176,12 +176,12 @@ function configure (gyp, argv, callback) { // For AIX and z/OS we need to set up the path to the exports file // which contains the symbols needed for linking. var nodeExpFile - if (process.platform === 'aix' || process.platform === 'os390') { - var ext = process.platform === 'aix' ? 'exp' : 'x' + if (process.platform === 'aix' || process.platform === 'os390' || process.platform === 'os400') { + var ext = process.platform === 'os390' ? 'x' : 'exp' var nodeRootDir = findNodeDirectory() var candidates - if (process.platform === 'aix') { + if (process.platform === 'aix' || process.platform === 'os400') { candidates = [ 'include/node/node', 'out/Release/node', @@ -276,7 +276,7 @@ function configure (gyp, argv, callback) { argv.push('-Dlibrary=shared_library') argv.push('-Dvisibility=default') argv.push('-Dnode_root_dir=' + nodeDir) - if (process.platform === 'aix' || process.platform === 'os390') { + if (process.platform === 'aix' || process.platform === 'os390' || process.platform === 'os400') { argv.push('-Dnode_exp_file=' + nodeExpFile) if (process.platform === 'os390' && zoslibIncDir) { argv.push('-Dzoslib_include_dir=' + zoslibIncDir) diff --git a/test/test-find-node-directory.js b/test/test-find-node-directory.js index f1380d162a..fa6223c65d 100644 --- a/test/test-find-node-directory.js +++ b/test/test-find-node-directory.js @@ -4,7 +4,7 @@ const test = require('tap').test const path = require('path') const findNodeDirectory = require('../lib/find-node-directory') -const platforms = ['darwin', 'freebsd', 'linux', 'sunos', 'win32', 'aix'] +const platforms = ['darwin', 'freebsd', 'linux', 'sunos', 'win32', 'aix', 'os400'] // we should find the directory based on the directory // the script is running in and it should match the layout From 33deab4ca807e615bd042ec74576637889118573 Mon Sep 17 00:00:00 2001 From: Rich Trott Date: Sat, 1 Oct 2022 19:05:01 -0700 Subject: [PATCH 351/551] Adding tarfile member sanitization to extractall() (#2741) Co-authored-by: TrellixVulnTeam --- update-gyp.py | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/update-gyp.py b/update-gyp.py index bb84f071a6..19524bd6a7 100755 --- a/update-gyp.py +++ b/update-gyp.py @@ -33,7 +33,25 @@ print("Unzipping...") with tarfile.open(tar_file, "r:gz") as tar_ref: - tar_ref.extractall(unzip_target) + def is_within_directory(directory, target): + + abs_directory = os.path.abspath(directory) + abs_target = os.path.abspath(target) + + prefix = os.path.commonprefix([abs_directory, abs_target]) + + return prefix == abs_directory + + def safe_extract(tar, path=".", members=None, *, numeric_owner=False): + + for member in tar.getmembers(): + member_path = os.path.join(path, member.name) + if not is_within_directory(path, member_path): + raise Exception("Attempted Path Traversal in Tar File") + + tar.extractall(path, members, numeric_owner) + + safe_extract(tar_ref, unzip_target) print("Moving to current checkout (" + CHECKOUT_PATH + ")...") if os.path.exists(CHECKOUT_GYP_PATH): From 4bc4747f2785356a2b666f6371dadca90a530b5b Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue, 4 Oct 2022 21:39:25 +1100 Subject: [PATCH 352/551] chore: release 9.2.0 (#2735) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- CHANGELOG.md | 23 +++++++++++++++++++++++ package.json | 2 +- 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 694823fa32..07bc923ec8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,28 @@ # Changelog +## [9.2.0](https://www.github.com/nodejs/node-gyp/compare/v9.1.0...v9.2.0) (2022-10-02) + + +### Features + +* Add proper support for IBM i ([a26494f](https://www.github.com/nodejs/node-gyp/commit/a26494fbb8883d9ef784503979e115dec3e2791e)) +* **gyp:** update gyp to v0.13.0 ([3e2a532](https://www.github.com/nodejs/node-gyp/commit/3e2a5324f1c24f3a04bca04cf54fe23d5c4d5e50)) + + +### Bug Fixes + +* node.js debugger adds stderr (but exit code is 0) -> shouldn't throw ([#2719](https://www.github.com/nodejs/node-gyp/issues/2719)) ([c379a74](https://www.github.com/nodejs/node-gyp/commit/c379a744c65c7ab07c2c3193d9c7e8f25ae1b05e)) + + +### Core + +* enable support for zoslib on z/OS ([#2600](https://www.github.com/nodejs/node-gyp/issues/2600)) ([83c0a12](https://www.github.com/nodejs/node-gyp/commit/83c0a12bf23b4cbf3125d41f9e2d4201db76c9ae)) + + +### Miscellaneous + +* update dependency - nopt@6.0.0 ([#2707](https://www.github.com/nodejs/node-gyp/issues/2707)) ([8958ecf](https://www.github.com/nodejs/node-gyp/commit/8958ecf2bb719227bbcbf155891c3186ee219a2e)) + ## [9.1.0](https://www.github.com/nodejs/node-gyp/compare/v9.0.0...v9.1.0) (2022-07-13) diff --git a/package.json b/package.json index 9f49299759..804f2d2dd4 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "9.1.0", + "version": "9.2.0", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From 7d0c83d2a95aca743dff972826d0da26203acfc4 Mon Sep 17 00:00:00 2001 From: Gaby Baghdadi Date: Fri, 7 Oct 2022 22:50:05 -0400 Subject: [PATCH 353/551] feat: support IBM Open XL C/C++ on z/OS (#2743) --- addon.gypi | 38 ++++++++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 12 deletions(-) diff --git a/addon.gypi b/addon.gypi index b3489e39bd..b4ac369acb 100644 --- a/addon.gypi +++ b/addon.gypi @@ -109,21 +109,35 @@ ], }], [ 'OS=="zos"', { - 'cflags': [ - '-q64', - '-Wc,DLL', - '-qlonglong', - '-qenum=int', - '-qxclang=-fexec-charset=ISO8859-1' + 'conditions': [ + [ '" Date: Sat, 8 Oct 2022 17:17:06 +0200 Subject: [PATCH 354/551] feat: remove support for VS2015 in Node.js >=19 (#2746) --- lib/find-visualstudio.js | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js index 8a5cfc1ea9..d3815112e6 100644 --- a/lib/find-visualstudio.js +++ b/lib/find-visualstudio.js @@ -347,6 +347,11 @@ VisualStudioFinder.prototype = { // Find an installation of Visual Studio 2015 to use findVisualStudio2015: function findVisualStudio2015 (cb) { + if (this.nodeSemver.major >= 19) { + this.addLog( + 'not looking for VS2015 as it is only supported up to Node.js 18') + return cb(null) + } return this.findOldVS({ version: '14.0', versionMajor: 14, From 713b8dcdbf44532ca9453a127da266386cc737f8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C3=ABl=20Zasso?= Date: Mon, 10 Oct 2022 13:51:12 +0200 Subject: [PATCH 355/551] feat(gyp): update gyp to v0.14.0 (#2749) --- gyp/.github/workflows/Python_tests.yml | 16 ++++++--- gyp/.github/workflows/node-gyp.yml | 13 ++++--- gyp/.github/workflows/nodejs-windows.yml | 9 +++-- gyp/.github/workflows/release-please.yml | 4 +-- gyp/CHANGELOG.md | 16 +++++++++ gyp/pylib/gyp/__init__.py | 12 +++++++ gyp/pylib/gyp/common.py | 1 + gyp/pylib/gyp/generator/make.py | 46 ++++++++++++++++++++---- gyp/pylib/gyp/generator/ninja.py | 4 +-- gyp/pylib/gyp/xcodeproj_file.py | 2 +- gyp/pyproject.toml | 41 +++++++++++++++++++++ gyp/requirements_dev.txt | 2 -- gyp/setup.py | 42 ---------------------- 13 files changed, 141 insertions(+), 67 deletions(-) create mode 100644 gyp/pyproject.toml delete mode 100644 gyp/requirements_dev.txt delete mode 100644 gyp/setup.py diff --git a/gyp/.github/workflows/Python_tests.yml b/gyp/.github/workflows/Python_tests.yml index 1cfa42f563..aad135027c 100644 --- a/gyp/.github/workflows/Python_tests.yml +++ b/gyp/.github/workflows/Python_tests.yml @@ -2,7 +2,12 @@ # TODO: Enable pytest --doctest-modules name: Python_tests -on: [push, pull_request] +on: + push: + branches: [ main ] + pull_request: + branches: [ main ] + workflow_dispatch: jobs: Python_tests: runs-on: ${{ matrix.os }} @@ -11,17 +16,18 @@ jobs: max-parallel: 8 matrix: os: [macos-latest, ubuntu-latest] # , windows-latest] - python-version: ["3.7", "3.8", "3.9", "3.10"] + python-version: ["3.7", "3.8", "3.9", "3.10", "3.11-dev"] steps: - uses: actions/checkout@v3 - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v3 + uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} - name: Install dependencies run: | - python -m pip install --upgrade pip - pip install -r requirements_dev.txt + python -m pip install --upgrade pip setuptools + pip install --editable ".[dev]" + - run: ./gyp -V && ./gyp --version && gyp -V && gyp --version - name: Lint with flake8 run: flake8 . --ignore=E203,W503 --max-complexity=101 --max-line-length=88 --show-source --statistics - name: Test with pytest diff --git a/gyp/.github/workflows/node-gyp.yml b/gyp/.github/workflows/node-gyp.yml index fc28a0b512..7cc1f9e075 100644 --- a/gyp/.github/workflows/node-gyp.yml +++ b/gyp/.github/workflows/node-gyp.yml @@ -1,9 +1,12 @@ name: node-gyp integration - -on: [push, pull_request] - +on: + push: + branches: [ main ] + pull_request: + branches: [ main ] + workflow_dispatch: jobs: - test: + integration: strategy: fail-fast: false matrix: @@ -24,7 +27,7 @@ jobs: - uses: actions/setup-node@v3 with: node-version: 14.x - - uses: actions/setup-python@v3 + - uses: actions/setup-python@v4 with: python-version: ${{ matrix.python }} - name: Install dependencies diff --git a/gyp/.github/workflows/nodejs-windows.yml b/gyp/.github/workflows/nodejs-windows.yml index 53bd736727..4e6c9548ff 100644 --- a/gyp/.github/workflows/nodejs-windows.yml +++ b/gyp/.github/workflows/nodejs-windows.yml @@ -1,10 +1,15 @@ name: Node.js Windows integration -on: [push, pull_request] +on: + push: + branches: [ main ] + pull_request: + branches: [ main ] + workflow_dispatch: jobs: build-windows: - runs-on: windows-latest + runs-on: windows-2019 steps: - name: Clone gyp-next uses: actions/checkout@v3 diff --git a/gyp/.github/workflows/release-please.yml b/gyp/.github/workflows/release-please.yml index 288afdb3b3..665c4c48fe 100644 --- a/gyp/.github/workflows/release-please.yml +++ b/gyp/.github/workflows/release-please.yml @@ -8,9 +8,9 @@ jobs: release-please: runs-on: ubuntu-latest steps: - - uses: GoogleCloudPlatform/release-please-action@v2 + - uses: google-github-actions/release-please-action@v3 with: token: ${{ secrets.GITHUB_TOKEN }} release-type: python package-name: gyp-next - bump-minor-pre-major: Yes + bump-minor-pre-major: true diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md index a103250cd5..4b4968f6a4 100644 --- a/gyp/CHANGELOG.md +++ b/gyp/CHANGELOG.md @@ -1,5 +1,21 @@ # Changelog +## [0.14.0](https://github.com/nodejs/gyp-next/compare/v0.13.0...v0.14.0) (2022-10-08) + + +### Features + +* Add command line argument for `gyp --version` ([#164](https://github.com/nodejs/gyp-next/issues/164)) ([5c9f4d0](https://github.com/nodejs/gyp-next/commit/5c9f4d05678dd855e18ed2327219e5d18e5374db)) +* ninja build for iOS ([#174](https://github.com/nodejs/gyp-next/issues/174)) ([b6f2714](https://github.com/nodejs/gyp-next/commit/b6f271424e0033d7ed54d437706695af2ba7a1bf)) +* **zos:** support IBM Open XL C/C++ & PL/I compilers on z/OS ([#178](https://github.com/nodejs/gyp-next/issues/178)) ([43a7211](https://github.com/nodejs/gyp-next/commit/43a72110ae3fafb13c9625cc7a969624b27cda47)) + + +### Bug Fixes + +* lock windows env ([#163](https://github.com/nodejs/gyp-next/issues/163)) ([44bd0dd](https://github.com/nodejs/gyp-next/commit/44bd0ddc93ea0b5770a44dd326a2e4ae62c21442)) +* move configuration information into pyproject.toml ([#176](https://github.com/nodejs/gyp-next/issues/176)) ([d69d8ec](https://github.com/nodejs/gyp-next/commit/d69d8ece6dbff7af4f2ea073c9fd170baf8cb7f7)) +* node.js debugger adds stderr (but exit code is 0) -> shouldn't throw ([#179](https://github.com/nodejs/gyp-next/issues/179)) ([1a457d9](https://github.com/nodejs/gyp-next/commit/1a457d9ed08cfd30c9fa551bc5cf0d90fb583787)) + ## [0.13.0](https://www.github.com/nodejs/gyp-next/compare/v0.12.1...v0.13.0) (2022-05-11) diff --git a/gyp/pylib/gyp/__init__.py b/gyp/pylib/gyp/__init__.py index 976d5b6aa8..2aa39d0318 100755 --- a/gyp/pylib/gyp/__init__.py +++ b/gyp/pylib/gyp/__init__.py @@ -15,6 +15,7 @@ import traceback from gyp.common import GypError + # Default debug modes for GYP debug = {} @@ -463,8 +464,19 @@ def gyp_main(args): metavar="TARGET", help="include only TARGET and its deep dependencies", ) + parser.add_argument( + "-V", + "--version", + dest="version", + action="store_true", + help="Show the version and exit.", + ) options, build_files_arg = parser.parse_args(args) + if options.version: + import pkg_resources + print(f"v{pkg_resources.get_distribution('gyp-next').version}") + return 0 build_files = build_files_arg # Set up the configuration directory (defaults to ~/.gyp) diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py index 0847cdabc7..d77adee8af 100644 --- a/gyp/pylib/gyp/common.py +++ b/gyp/pylib/gyp/common.py @@ -470,6 +470,7 @@ def CopyTool(flavor, out_path, generator_flags={}): "os400": "flock", "solaris": "flock", "mac": "mac", + "ios": "mac", "win": "win", }.get(flavor, None) if not prefix: diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index e225326e1d..f1d01a629d 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -101,6 +101,7 @@ def CalculateVariables(default_variables, params): default_variables.setdefault("SHARED_LIB_SUFFIX", ".a") elif flavor == "zos": default_variables.setdefault("SHARED_LIB_SUFFIX", ".x") + COMPILABLE_EXTENSIONS.update({".pli": "pli"}) else: default_variables.setdefault("SHARED_LIB_SUFFIX", ".so") default_variables.setdefault("SHARED_LIB_DIR", "$(builddir)/lib.$(TOOLSET)") @@ -318,7 +319,7 @@ def CalculateGeneratorInputInfo(params): cmd_link = $(LINK.$(TOOLSET)) $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ $(LD_INPUTS) $(LIBS) quiet_cmd_solink = SOLINK($(TOOLSET)) $@ -cmd_solink = $(LINK.$(TOOLSET)) $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -Wl,DLL -o $(patsubst %.x,%.so,$@) $(LD_INPUTS) $(LIBS) && if [ -f $(notdir $@) ]; then /bin/cp $(notdir $@) $@; else true; fi +cmd_solink = $(LINK.$(TOOLSET)) $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ $(LD_INPUTS) $(LIBS) quiet_cmd_solink_module = SOLINK_MODULE($(TOOLSET)) $@ cmd_solink_module = $(LINK.$(TOOLSET)) $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ $(filter-out FORCE_DO_CMD, $^) $(LIBS) @@ -378,6 +379,7 @@ def CalculateGeneratorInputInfo(params): LINK.target ?= %(LINK.target)s LDFLAGS.target ?= $(LDFLAGS) AR.target ?= $(AR) +PLI.target ?= %(PLI.target)s # C++ apps need to be linked with g++. LINK ?= $(CXX.target) @@ -391,6 +393,7 @@ def CalculateGeneratorInputInfo(params): LINK.host ?= %(LINK.host)s LDFLAGS.host ?= $(LDFLAGS_host) AR.host ?= %(AR.host)s +PLI.host ?= %(PLI.host)s # Define a dir function that can handle spaces. # http://www.gnu.org/software/make/manual/make.html#Syntax-of-Functions @@ -628,6 +631,15 @@ def WriteRootHeaderSuffixRules(writer): writer.write("\n") +SHARED_HEADER_OS390_COMMANDS = """ +PLIFLAGS.target ?= -qlp=64 -qlimits=extname=31 $(PLIFLAGS) +PLIFLAGS.host ?= -qlp=64 -qlimits=extname=31 $(PLIFLAGS) + +quiet_cmd_pli = PLI($(TOOLSET)) $@ +cmd_pli = $(PLI.$(TOOLSET)) $(GYP_PLIFLAGS) $(PLIFLAGS.$(TOOLSET)) -c $< && \ + if [ -f $(notdir $@) ]; then /bin/cp $(notdir $@) $@; else true; fi +""" + SHARED_HEADER_SUFFIX_RULES_COMMENT1 = """\ # Suffix rules, putting all outputs into $(obj). """ @@ -2450,10 +2462,12 @@ def CalculateMakefilePath(build_file, base_name): "AR.target": GetEnvironFallback(("AR_target", "AR"), "$(AR)"), "CXX.target": GetEnvironFallback(("CXX_target", "CXX"), "$(CXX)"), "LINK.target": GetEnvironFallback(("LINK_target", "LINK"), "$(LINK)"), + "PLI.target": GetEnvironFallback(("PLI_target", "PLI"), "pli"), "CC.host": GetEnvironFallback(("CC_host", "CC"), "gcc"), "AR.host": GetEnvironFallback(("AR_host", "AR"), "ar"), "CXX.host": GetEnvironFallback(("CXX_host", "CXX"), "g++"), "LINK.host": GetEnvironFallback(("LINK_host", "LINK"), "$(CXX.host)"), + "PLI.host": GetEnvironFallback(("PLI_host", "PLI"), "pli"), } if flavor == "mac": flock_command = "./gyp-mac-tool flock" @@ -2469,16 +2483,36 @@ def CalculateMakefilePath(build_file, base_name): header_params.update({"link_commands": LINK_COMMANDS_ANDROID}) elif flavor == "zos": copy_archive_arguments = "-fPR" - makedep_arguments = "-qmakedep=gcc" + CC_target = GetEnvironFallback(("CC_target", "CC"), "njsc") + makedep_arguments = "-MMD" + if CC_target == "clang": + CC_host = GetEnvironFallback(("CC_host", "CC"), "clang") + CXX_target = GetEnvironFallback(("CXX_target", "CXX"), "clang++") + CXX_host = GetEnvironFallback(("CXX_host", "CXX"), "clang++") + elif CC_target == "ibm-clang64": + CC_host = GetEnvironFallback(("CC_host", "CC"), "ibm-clang64") + CXX_target = GetEnvironFallback(("CXX_target", "CXX"), "ibm-clang++64") + CXX_host = GetEnvironFallback(("CXX_host", "CXX"), "ibm-clang++64") + elif CC_target == "ibm-clang": + CC_host = GetEnvironFallback(("CC_host", "CC"), "ibm-clang") + CXX_target = GetEnvironFallback(("CXX_target", "CXX"), "ibm-clang++") + CXX_host = GetEnvironFallback(("CXX_host", "CXX"), "ibm-clang++") + else: + # Node.js versions prior to v18: + makedep_arguments = "-qmakedep=gcc" + CC_host = GetEnvironFallback(("CC_host", "CC"), "njsc") + CXX_target = GetEnvironFallback(("CXX_target", "CXX"), "njsc++") + CXX_host = GetEnvironFallback(("CXX_host", "CXX"), "njsc++") header_params.update( { "copy_archive_args": copy_archive_arguments, "makedep_args": makedep_arguments, "link_commands": LINK_COMMANDS_OS390, - "CC.target": GetEnvironFallback(("CC_target", "CC"), "njsc"), - "CXX.target": GetEnvironFallback(("CXX_target", "CXX"), "njsc++"), - "CC.host": GetEnvironFallback(("CC_host", "CC"), "njsc"), - "CXX.host": GetEnvironFallback(("CXX_host", "CXX"), "njsc++"), + "extra_commands": SHARED_HEADER_OS390_COMMANDS, + "CC.target": CC_target, + "CXX.target": CXX_target, + "CC.host": CC_host, + "CXX.host": CXX_host, } ) elif flavor == "solaris": diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py index 3db3771ac9..ca04ee13a1 100644 --- a/gyp/pylib/gyp/generator/ninja.py +++ b/gyp/pylib/gyp/generator/ninja.py @@ -1583,7 +1583,7 @@ def WriteTarget(self, spec, config_name, config, link_deps, compile_deps): elif spec["type"] == "static_library": self.target.binary = self.ComputeOutput(spec) if ( - self.flavor not in ("mac", "openbsd", "netbsd", "win") + self.flavor not in ("ios", "mac", "netbsd", "openbsd", "win") and not self.is_standalone_static_library ): self.ninja.build( @@ -2496,7 +2496,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name ), ) - if flavor != "mac" and flavor != "win": + if flavor not in ("ios", "mac", "win"): master_ninja.rule( "alink", description="AR $out", diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py index 076eea3721..0e941eb471 100644 --- a/gyp/pylib/gyp/xcodeproj_file.py +++ b/gyp/pylib/gyp/xcodeproj_file.py @@ -2990,7 +2990,7 @@ def AddOrGetProjectReference(self, other_pbxproject): # Xcode seems to sort this list case-insensitively self._properties["projectReferences"] = sorted( self._properties["projectReferences"], - key=lambda x: x["ProjectRef"].Name().lower + key=lambda x: x["ProjectRef"].Name().lower() ) else: # The link already exists. Pull out the relevnt data. diff --git a/gyp/pyproject.toml b/gyp/pyproject.toml new file mode 100644 index 0000000000..d8a5451520 --- /dev/null +++ b/gyp/pyproject.toml @@ -0,0 +1,41 @@ +[build-system] +requires = ["setuptools>=61.0"] +build-backend = "setuptools.build_meta" + +[project] +name = "gyp-next" +version = "0.14.0" +authors = [ + { name="Node.js contributors", email="ryzokuken@disroot.org" }, +] +description = "A fork of the GYP build system for use in the Node.js projects" +readme = "README.md" +license = { file="LICENSE" } +requires-python = ">=3.6" +classifiers = [ + "Development Status :: 3 - Alpha", + "Environment :: Console", + "Intended Audience :: Developers", + "License :: OSI Approved :: BSD License", + "Natural Language :: English", + "Programming Language :: Python", + "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3.6", + "Programming Language :: Python :: 3.7", + "Programming Language :: Python :: 3.8", + "Programming Language :: Python :: 3.9", + "Programming Language :: Python :: 3.10", +] + +[project.optional-dependencies] +dev = ["flake8", "pytest"] + +[project.scripts] +gyp = "gyp:script_main" + +[project.urls] +"Homepage" = "https://github.com/nodejs/gyp-next" + +[tool.setuptools] +package-dir = {"" = "pylib"} +packages = ["gyp", "gyp.generator"] diff --git a/gyp/requirements_dev.txt b/gyp/requirements_dev.txt deleted file mode 100644 index 28ecacab60..0000000000 --- a/gyp/requirements_dev.txt +++ /dev/null @@ -1,2 +0,0 @@ -flake8 -pytest diff --git a/gyp/setup.py b/gyp/setup.py deleted file mode 100644 index 1bb6908dea..0000000000 --- a/gyp/setup.py +++ /dev/null @@ -1,42 +0,0 @@ -#!/usr/bin/env python3 - -# Copyright (c) 2009 Google Inc. All rights reserved. -# Use of this source code is governed by a BSD-style license that can be -# found in the LICENSE file. - -from os import path - -from setuptools import setup - -here = path.abspath(path.dirname(__file__)) -# Get the long description from the README file -with open(path.join(here, "README.md")) as in_file: - long_description = in_file.read() - -setup( - name="gyp-next", - version="0.13.0", - description="A fork of the GYP build system for use in the Node.js projects", - long_description=long_description, - long_description_content_type="text/markdown", - author="Node.js contributors", - author_email="ryzokuken@disroot.org", - url="https://github.com/nodejs/gyp-next", - package_dir={"": "pylib"}, - packages=["gyp", "gyp.generator"], - entry_points={"console_scripts": ["gyp=gyp:script_main"]}, - python_requires=">=3.6", - classifiers=[ - "Development Status :: 3 - Alpha", - "Environment :: Console", - "Intended Audience :: Developers", - "License :: OSI Approved :: BSD License", - "Natural Language :: English", - "Programming Language :: Python", - "Programming Language :: Python :: 3", - "Programming Language :: Python :: 3.6", - "Programming Language :: Python :: 3.7", - "Programming Language :: Python :: 3.8", - "Programming Language :: Python :: 3.9", - ], -) From 2cc72be3b307d302afdd042cd920076dfe7380e6 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Mon, 10 Oct 2022 11:51:52 +0000 Subject: [PATCH 356/551] chore: release 9.3.0 --- CHANGELOG.md | 9 +++++++++ package.json | 2 +- 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 07bc923ec8..54f0b4f427 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,14 @@ # Changelog +## [9.3.0](https://www.github.com/nodejs/node-gyp/compare/v9.2.0...v9.3.0) (2022-10-10) + + +### Features + +* **gyp:** update gyp to v0.14.0 ([#2749](https://www.github.com/nodejs/node-gyp/issues/2749)) ([713b8dc](https://www.github.com/nodejs/node-gyp/commit/713b8dcdbf44532ca9453a127da266386cc737f8)) +* remove support for VS2015 in Node.js >=19 ([#2746](https://www.github.com/nodejs/node-gyp/issues/2746)) ([131d1a4](https://www.github.com/nodejs/node-gyp/commit/131d1a463baf034a04154bcda753a8295f112a34)) +* support IBM Open XL C/C++ on z/OS ([#2743](https://www.github.com/nodejs/node-gyp/issues/2743)) ([7d0c83d](https://www.github.com/nodejs/node-gyp/commit/7d0c83d2a95aca743dff972826d0da26203acfc4)) + ## [9.2.0](https://www.github.com/nodejs/node-gyp/compare/v9.1.0...v9.2.0) (2022-10-02) diff --git a/package.json b/package.json index 804f2d2dd4..932e8cb3b3 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "9.2.0", + "version": "9.3.0", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From ee46f9d2b56eb238ce5d8199077ce5c98bdbd64c Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Mon, 14 Nov 2022 08:43:25 +0100 Subject: [PATCH 357/551] Add Python 3.11 to the testing https://docs.python.org/3/whatsnew/3.11.html --- .github/workflows/tests.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index a3b68bdd5d..8737344da1 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -14,7 +14,7 @@ jobs: max-parallel: 15 matrix: node: [14.x, 16.x, 18.x] - python: ["3.6", "3.8", "3.10"] + python: ["3.6", "3.8", "3.10", "3.11"] os: [macos-latest, ubuntu-latest, windows-latest] runs-on: ${{ matrix.os }} steps: From 38f01fa57d10fdb3db7697121d957bc2e0e96508 Mon Sep 17 00:00:00 2001 From: Luke Karrys Date: Sat, 10 Dec 2022 13:29:32 -0700 Subject: [PATCH 358/551] ci: update python test matrix (#2774) * ci: drop python 3.6 from test matrix * Update .github/workflows/tests.yml Co-authored-by: Christian Clauss Co-authored-by: Christian Clauss --- .github/workflows/tests.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 8737344da1..8f34d4e11f 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -14,7 +14,7 @@ jobs: max-parallel: 15 matrix: node: [14.x, 16.x, 18.x] - python: ["3.6", "3.8", "3.10", "3.11"] + python: ["3.7", "3.9", "3.11"] os: [macos-latest, ubuntu-latest, windows-latest] runs-on: ${{ matrix.os }} steps: From 888efb9055857afee6a6b54550722cf9ae3ee323 Mon Sep 17 00:00:00 2001 From: Luke Karrys Date: Fri, 16 Dec 2022 14:11:15 -0700 Subject: [PATCH 359/551] fix: increase node 12 support to ^12.13 (#2771) --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index 932e8cb3b3..016f022e8a 100644 --- a/package.json +++ b/package.json @@ -34,7 +34,7 @@ "which": "^2.0.2" }, "engines": { - "node": "^12.22 || ^14.13 || >=16" + "node": "^12.13 || ^14.13 || >=16" }, "devDependencies": { "bindings": "^1.5.0", From 39ac2c135db8a9e62bf22f0c7a4469ae6c381325 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri, 16 Dec 2022 21:11:53 +0000 Subject: [PATCH 360/551] chore: release 9.3.1 --- CHANGELOG.md | 12 ++++++++++++ package.json | 2 +- 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 54f0b4f427..4131521515 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,17 @@ # Changelog +### [9.3.1](https://www.github.com/nodejs/node-gyp/compare/v9.3.0...v9.3.1) (2022-12-16) + + +### Bug Fixes + +* increase node 12 support to ^12.13 ([#2771](https://www.github.com/nodejs/node-gyp/issues/2771)) ([888efb9](https://www.github.com/nodejs/node-gyp/commit/888efb9055857afee6a6b54550722cf9ae3ee323)) + + +### Miscellaneous + +* update python test matrix ([#2774](https://www.github.com/nodejs/node-gyp/issues/2774)) ([38f01fa](https://www.github.com/nodejs/node-gyp/commit/38f01fa57d10fdb3db7697121d957bc2e0e96508)) + ## [9.3.0](https://www.github.com/nodejs/node-gyp/compare/v9.2.0...v9.3.0) (2022-10-10) diff --git a/package.json b/package.json index 016f022e8a..f95ebeadec 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "9.3.0", + "version": "9.3.1", "installVersion": 9, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From fc0ddc6523c62b10e5ca1257500b3ceac01450a7 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Mon, 13 Mar 2023 11:43:22 +0100 Subject: [PATCH 361/551] feat: Upgrade Python linting from flake8 to ruff (#2815) [Ruff](https://beta.ruff.rs/) supports [over 500 lint rules](https://beta.ruff.rs/docs/rules) including bandit, isort, pylint, pyupgrade, and flake8 plus its plugins and is written in Rust for speed. This GitHub Action will provide contributors with intuitive GitHub Annotations. ![image](https://user-images.githubusercontent.com/3709715/223758136-afc386d2-70aa-4eff-953a-2c2d82ceea23.png) The `Required` in the checks below should be: 1. Removed from `flake8-annotation` and added to `ruff-annotation` which replaces it. 2. Removed from `isort` and added to `ruff` which replaces it. --- .github/workflows/tests.yml | 11 +++++++---- gyp/pylib/gyp/generator/eclipse.py | 2 +- gyp/pylib/gyp/xcodeproj_file.py | 2 +- 3 files changed, 9 insertions(+), 6 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 8f34d4e11f..73f6fcec45 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -8,6 +8,12 @@ on: pull_request: branches: [ main ] jobs: + Lint_Python: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - run: pip install --user ruff + - run: ruff --format=github --select="E,F,PLC,PLE,UP,W,YTT" --ignore="S101,UP031" --target-version=py37 . Tests: strategy: fail-fast: false @@ -33,15 +39,12 @@ jobs: - name: Install Dependencies run: | npm install --no-progress - pip install flake8 pytest + pip install pytest - name: Set Windows environment if: startsWith(matrix.os, 'windows') run: | echo 'GYP_MSVS_VERSION=2015' >> $Env:GITHUB_ENV echo 'GYP_MSVS_OVERRIDE_PATH=C:\\Dummy' >> $Env:GITHUB_ENV - - name: Lint Python - if: startsWith(matrix.os, 'ubuntu') - run: flake8 . --ignore=E203,W503 --max-complexity=101 --max-line-length=88 --show-source --statistics - name: Run Python tests run: python -m pytest # - name: Run doctests with pytest diff --git a/gyp/pylib/gyp/generator/eclipse.py b/gyp/pylib/gyp/generator/eclipse.py index 1ff0dc83ae..a851b4db75 100644 --- a/gyp/pylib/gyp/generator/eclipse.py +++ b/gyp/pylib/gyp/generator/eclipse.py @@ -24,7 +24,7 @@ import gyp.common import gyp.msvs_emulation import shlex -import xml.etree.cElementTree as ET +import xml.etree.ElementTree as ET generator_wants_static_library_dependencies_adjusted = False diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py index 0e941eb471..4e0ec5e882 100644 --- a/gyp/pylib/gyp/xcodeproj_file.py +++ b/gyp/pylib/gyp/xcodeproj_file.py @@ -2770,7 +2770,7 @@ def __init__(self, properties=None, id=None, parent=None, path=None): self.path = path self._other_pbxprojects = {} # super - return XCContainerPortal.__init__(self, properties, id, parent) + XCContainerPortal.__init__(self, properties, id, parent) def Name(self): name = self.path From 41882a975bbdccddf451415ff445108703eca8a3 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Mon, 13 Mar 2023 12:14:24 +0100 Subject: [PATCH 362/551] Improved advise on repacing node-sass with sass (#2758) * Improved advise on repacing node-sass with sass * Update README.md --- docs/README.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/README.md b/docs/README.md index 7027960909..2c3133190f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,7 +4,14 @@ Please look thru your error log for the string `gyp info using node-gyp@` and if ## `node-sass` is deprecated -Please be aware that the package [`node-sass` is deprecated](https://github.com/sass/node-sass#node-sass) so you should actively seek alternatives. Please avoid opening new `node-sass` issues on this repo. You can try `npm install --global node-sass@latest` but we [cannot help much](https://github.com/nodejs/node-gyp/issues?q=is%3Aissue+label%3A%22Node+Sass+--%3E+Dart+Sass%22+) here. +Please be aware that the package [`node-sass` is deprecated](https://github.com/sass/node-sass#node-sass) so you should actively seek alternatives. You can try: +``` +npm uninstall node-sass +npm install sass --save +# or ... +npm install --global node-sass@latest +``` +But in any case, please avoid opening new `node-sass` issues on this repo because we [cannot help much](https://github.com/nodejs/node-gyp/issues?q=is%3Aissue+label%3A%22Node+Sass+--%3E+Dart+Sass%22+). ## Issues finding the installed Visual Studio From 337e8e68209bd2481cbb11dacce61234dc5c9419 Mon Sep 17 00:00:00 2001 From: Raymond Zhao <7199958+rzhao271@users.noreply.github.com> Date: Thu, 6 Apr 2023 11:07:02 -0700 Subject: [PATCH 363/551] chore: get update-gyp.py to work with Python >= v3.5 (#2826) * chore: get update-gyp.py to work with Python v3.9 * Ruff ignore rule PLC1901 --------- Co-authored-by: Christian Clauss --- .github/workflows/tests.yml | 2 +- update-gyp.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 73f6fcec45..0da6cdbf87 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -13,7 +13,7 @@ jobs: steps: - uses: actions/checkout@v3 - run: pip install --user ruff - - run: ruff --format=github --select="E,F,PLC,PLE,UP,W,YTT" --ignore="S101,UP031" --target-version=py37 . + - run: ruff --format=github --select="E,F,PLC,PLE,UP,W,YTT" --ignore="PLC1901,S101,UP031" --target-version=py37 . Tests: strategy: fail-fast: false diff --git a/update-gyp.py b/update-gyp.py index 19524bd6a7..70e2d10028 100755 --- a/update-gyp.py +++ b/update-gyp.py @@ -49,7 +49,7 @@ def safe_extract(tar, path=".", members=None, *, numeric_owner=False): if not is_within_directory(path, member_path): raise Exception("Attempted Path Traversal in Tar File") - tar.extractall(path, members, numeric_owner) + tar.extractall(path, members, numeric_owner=numeric_owner) safe_extract(tar_ref, unzip_target) From c7927e228dfde059c93e08c26b54dd8026144583 Mon Sep 17 00:00:00 2001 From: Maksim Beliaev Date: Mon, 10 Apr 2023 11:34:26 +0200 Subject: [PATCH 364/551] doc: Update README.md (#2822) Co-authored-by: Christian Clauss --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index 7636ad5482..1f0c9b1584 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,6 @@ Install tools and configuration manually: * Install Visual C++ Build Environment: [Visual Studio Build Tools](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) (using "Visual C++ build tools" workload) or [Visual Studio Community](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=Community) (using the "Desktop development with C++" workload) - * Launch cmd, `npm config set msvs_version 2017` If the above steps didn't work for you, please visit [Microsoft's Node.js Guidelines for Windows](https://github.com/Microsoft/nodejs-guidelines/blob/master/windows-environment.md#compiling-native-addon-modules) for additional tips. From 02480f6b6894a6bb81dd3e1365bc4df6d5336f7c Mon Sep 17 00:00:00 2001 From: ravindraP20 <72969399+ravindraP20@users.noreply.github.com> Date: Sat, 22 Apr 2023 10:25:19 +0530 Subject: [PATCH 365/551] update make-fetch-happen to 11.0.3 (#2796) http-cache-semantics 4.1.0 is vulnerable https://www.cve.org/CVERecord?id=CVE-2022-25881 --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index f95ebeadec..11b10b69f0 100644 --- a/package.json +++ b/package.json @@ -25,7 +25,7 @@ "env-paths": "^2.2.0", "glob": "^7.1.4", "graceful-fs": "^4.2.6", - "make-fetch-happen": "^10.0.3", + "make-fetch-happen": "^11.0.3", "nopt": "^6.0.0", "npmlog": "^6.0.0", "rimraf": "^3.0.2", From 6f3c2d3c6c0de0dbf8c7245f34c2e0b3eea53812 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Wed, 26 Apr 2023 18:41:38 +0200 Subject: [PATCH 366/551] docs: docs/README.md add advise about deprecated node-sass (#2828) --- docs/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/README.md b/docs/README.md index 2c3133190f..487fb0a57e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,6 +11,7 @@ npm install sass --save # or ... npm install --global node-sass@latest ``` +`node-sass` projects _may_ work by downgrading to Node.js v14 but [that release is end-of-life](https://github.com/nodejs/release#release-schedule). But in any case, please avoid opening new `node-sass` issues on this repo because we [cannot help much](https://github.com/nodejs/node-gyp/issues?q=is%3Aissue+label%3A%22Node+Sass+--%3E+Dart+Sass%22+). ## Issues finding the installed Visual Studio From bb76021d35964d2bb125bc6214286f35ae4e6cad Mon Sep 17 00:00:00 2001 From: Dennis Ameling Date: Mon, 25 Apr 2022 22:28:41 +0200 Subject: [PATCH 367/551] feat: add support for native windows arm64 build tools Visual Studio 2022 17.4 adds a native C++ compiler for Windows on ARM. This allows arm64 devices to leverage native build tools, leading to a 35% (or more) speed increase. https://devblogs.microsoft.com/visualstudio/arm64-visual-studio-is-officially-here/ Signed-off-by: Dennis Ameling --- README.md | 4 +- lib/find-visualstudio.js | 15 +- test/fixtures/VS_2022_Community_workload.txt | 569 +++++++++++++++++++ test/test-find-visualstudio.js | 63 +- 4 files changed, 646 insertions(+), 5 deletions(-) create mode 100644 test/fixtures/VS_2022_Community_workload.txt diff --git a/README.md b/README.md index 1f0c9b1584..99494a38d0 100644 --- a/README.md +++ b/README.md @@ -56,7 +56,9 @@ Install tools and configuration manually: If the above steps didn't work for you, please visit [Microsoft's Node.js Guidelines for Windows](https://github.com/Microsoft/nodejs-guidelines/blob/master/windows-environment.md#compiling-native-addon-modules) for additional tips. - To target native ARM64 Node.js on Windows 10 on ARM, add the components "Visual C++ compilers and libraries for ARM64" and "Visual C++ ATL for ARM64". + To target native ARM64 Node.js on Windows on ARM, add the components "Visual C++ compilers and libraries for ARM64" and "Visual C++ ATL for ARM64". + + To use the native ARM64 C++ compiler on Windows on ARM, ensure that you have Visual Studio 2022 [17.4 or later](https://devblogs.microsoft.com/visualstudio/arm64-visual-studio-is-officially-here/) installed. ### Configuring Python Dependency diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js index d3815112e6..16f6e79559 100644 --- a/lib/find-visualstudio.js +++ b/lib/find-visualstudio.js @@ -266,10 +266,15 @@ VisualStudioFinder.prototype = { return {} }, + msBuildPathExists: function msBuildPathExists (path) { + return fs.existsSync(path) + }, + // Helper - process MSBuild information getMSBuild: function getMSBuild (info, versionYear) { const pkg = 'Microsoft.VisualStudio.VC.MSBuild.Base' const msbuildPath = path.join(info.path, 'MSBuild', 'Current', 'Bin', 'MSBuild.exe') + const msbuildPathArm64 = path.join(info.path, 'MSBuild', 'Current', 'Bin', 'arm64', 'MSBuild.exe') if (info.packages.indexOf(pkg) !== -1) { this.log.silly('- found VC.MSBuild.Base') if (versionYear === 2017) { @@ -279,8 +284,14 @@ VisualStudioFinder.prototype = { return msbuildPath } } - // visual studio 2022 don't has msbuild pkg - if (fs.existsSync(msbuildPath)) { + /** + * Visual Studio 2022 doesn't have the MSBuild package. + * Support for compiling _on_ ARM64 was added in MSVC 14.32.31326, + * so let's leverage it if the user has an ARM64 device. + */ + if (process.arch === 'arm64' && this.msBuildPathExists(msbuildPathArm64)) { + return msbuildPathArm64 + } else if (this.msBuildPathExists(msbuildPath)) { return msbuildPath } return null diff --git a/test/fixtures/VS_2022_Community_workload.txt b/test/fixtures/VS_2022_Community_workload.txt new file mode 100644 index 0000000000..7cd20f8598 --- /dev/null +++ b/test/fixtures/VS_2022_Community_workload.txt @@ -0,0 +1,569 @@ +[ + { + "path": "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community", + "version": "17.4.33213.308", + "packages": [ + "Microsoft.VisualStudio.Product.Community", + "Microsoft.VisualStudio.PackageGroup.LiveShare.VSCore", + "Microsoft.VisualStudio.LiveShare.VSCore", + "Microsoft.VisualStudio.Workload.NativeDesktop", + "Microsoft.VisualStudio.Component.VC.ASAN", + "Microsoft.VisualCpp.ASAN.X86", + "Microsoft.VC.14.34.17.4.ASAN.X86.base", + "Microsoft.VC.14.34.17.4.ASAN.X64.base", + "Microsoft.VC.14.34.17.4.ASAN.Headers.base", + "Microsoft.VisualStudio.VC.IDE.Project.Factories", + "Microsoft.VisualStudio.Component.VC.TestAdapterForGoogleTest", + "Microsoft.VisualStudio.VC.Ide.TestAdapterForGoogleTest", + "Microsoft.VisualStudio.Component.VC.TestAdapterForBoostTest", + "Microsoft.VisualStudio.VC.Ide.TestAdapterForBoostTest", + "Microsoft.VisualStudio.Component.VC.CMake.Project", + "Microsoft.VisualStudio.ComponentGroup.WebToolsExtensions.CMake", + "Microsoft.VisualStudio.VC.CMake", + "Microsoft.VisualStudio.VC.CMake.Project", + "Microsoft.VisualStudio.VC.CMake.Client", + "Microsoft.VisualStudio.VC.ExternalBuildFramework", + "Microsoft.VisualStudio.Component.VC.DiagnosticTools", + "Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core", + "Microsoft.VisualStudio.PackageGroup.TestTools.Native", + "Microsoft.VisualStudio.Component.VC.Redist.14.Latest", + "Microsoft.VisualStudio.VC.Templates.UnitTest", + "Microsoft.VisualStudio.VC.UnitTest.Desktop.Build.Core", + "Microsoft.VisualStudio.TestTools.TestPlatform.V1.CPP", + "Microsoft.VisualStudio.VC.Templates.UnitTest.Resources", + "Microsoft.VisualStudio.VC.Templates.Desktop", + "Microsoft.VisualStudio.Component.Graphics", + "Microsoft.VisualStudio.Graphics.Viewers", + "Microsoft.VisualStudio.Graphics.Viewers.Resources", + "Microsoft.VisualStudio.Component.VC.ATL.ARM64", + "Microsoft.VisualCpp.ATL.ARM64", + "Microsoft.VC.14.34.17.4.ATL.ARM64.base", + "Microsoft.VisualStudio.Component.VC.ATL", + "Microsoft.VisualStudio.VC.Ide.ATL", + "Microsoft.VisualStudio.VC.Ide.ATL.Resources", + "Microsoft.VisualCpp.ATL.X86", + "Microsoft.VC.14.34.17.4.ATL.X86.base", + "Microsoft.VisualCpp.ATL.X64", + "Microsoft.VC.14.34.17.4.ATL.X64.base", + "Microsoft.VC.14.34.17.4.Props.ATLMFC", + "Microsoft.VisualCpp.ATL.Source", + "Microsoft.VC.14.34.17.4.ATL.Source.base", + "Microsoft.VisualCpp.ATL.Headers", + "Microsoft.VC.14.34.17.4.ATL.Headers.base", + "Microsoft.VC.14.34.17.4.Servicing.ATL", + "Microsoft.VisualStudio.Component.VC.Tools.ARM64", + "Microsoft.VisualStudio.VC.MSBuild.v170.ARM64.v143", + "Microsoft.VisualStudio.VC.MSBuild.v170.ARM64", + "Microsoft.VS.VC.vcvars.arm64.Shortcuts", + "Microsoft.VisualCpp.CA.Ext.Hostx64.TargetARM64", + "Microsoft.VC.14.34.17.4.CA.Ext.Hostx64.TargetARM64.base", + "Microsoft.VC.14.34.17.4.CA.Ext.Hostx64.TargetARM64.Res.base", + "Microsoft.VisualCpp.CA.Ext.Hostx86.TargetARM64", + "Microsoft.VC.14.34.17.4.CA.Ext.Hostx86.TargetARM64.base", + "Microsoft.VC.14.34.17.4.CA.Ext.Hostx86.TargetARM64.Res.base", + "Microsoft.VisualCpp.CA.Ext.HostARM64.TargetARM64", + "Microsoft.VC.14.34.17.4.CA.Ext.HostARM64.TargetARM64.base", + "Microsoft.VC.14.34.17.4.CA.Ext.HostARM64.TargetARM64.Res.base", + "Microsoft.VisualCpp.Tools.Hostx86.Targetarm64", + "Microsoft.VC.14.34.17.4.Tools.Hostx86.Targetarm64.base", + "Microsoft.VC.14.34.17.4.Tools.HostX86.TargetARM64.Res.base", + "Microsoft.VisualCpp.Tools.HostARM64.TargetARM64", + "Microsoft.VC.14.34.17.4.Tools.HostARM64.TargetARM64.base", + "Microsoft.VC.14.34.17.4.Tools.HostARM64.TargetARM64.Res.base", + "Microsoft.VisualCpp.CRT.Redist.ARM64.OneCore.Desktop", + "Microsoft.VC.14.34.17.4.CRT.Redist.ARM64.OneCore.Desktop.base", + "Microsoft.VisualCpp.CRT.Redist.ARM64", + "Microsoft.VC.14.34.17.4.CRT.Redist.ARM64.base", + "Microsoft.VisualCpp.CRT.ARM64.OneCore.Desktop", + "Microsoft.VC.14.34.17.4.CRT.ARM64.OneCore.Desktop.base", + "Microsoft.VC.14.34.17.4.CRT.ARM64.OneCore.Desktop.debug.base", + "Microsoft.VisualCpp.CRT.ARM64.Store", + "Microsoft.VC.14.34.17.4.CRT.ARM64.Store.base", + "Microsoft.VisualCpp.CRT.ARM64.Desktop", + "Microsoft.VC.14.34.17.4.CRT.ARM64.Desktop.base", + "Microsoft.VC.14.34.17.4.CRT.ARM64.Desktop.debug.base", + "Microsoft.VisualStudio.PackageGroup.VC.Tools.x64.ARM64", + "Microsoft.VisualCpp.Tools.Core", + "Microsoft.VisualCpp.PGO.ARM64", + "Microsoft.VC.14.34.17.4.PGO.ARM64.base", + "Microsoft.VisualCpp.Premium.Tools.Hostx86.Targetarm64", + "Microsoft.VC.14.34.17.4.Premium.Tools.Hostx86.Targetarm64.base", + "Microsoft.VC.14.34.17.4.Prem.HostX86.TargetARM64.Res.base", + "Microsoft.VisualCpp.Premium.Tools.HostX64.TargetARM64", + "Microsoft.VC.14.34.17.4.Premium.Tools.HostX64.TargetARM64.base", + "Microsoft.VC.14.34.17.4.Prem.HostX64.TargetARM64.Res.base", + "Microsoft.VisualCpp.Premium.Tools.ARM64.Base", + "Microsoft.VC.14.34.17.4.Premium.Tools.ARM64.Base.base", + "Microsoft.VisualCpp.Tools.HostX64.TargetARM64", + "Microsoft.VC.14.34.17.4.Tools.HostX64.TargetARM64.base", + "Microsoft.VC.14.34.17.4.Props.ARM64", + "Microsoft.VC.14.34.17.4.Tools.HostX64.TargetARM64.Res.base", + "Microsoft.VisualStudio.Component.VC.Tools.ARM64EC", + "Microsoft.VisualStudio.Component.Windows11SDK.22621", + "Win11SDK_10.0.22621", + "Microsoft.VisualStudio.VC.MSBuild.v170.ARM64EC.v143", + "Microsoft.VisualStudio.VC.MSBuild.v170.ARM64EC", + "Microsoft.VisualCpp.CRT.ARM64EC.Store", + "Microsoft.VC.14.34.17.4.CRT.ARM64EC.Store.base", + "Microsoft.VisualStudio.Component.VC.Tools.x86.x64", + "Microsoft.VisualCpp.CodeAnalysis.Extensions", + "Microsoft.VisualCpp.CA.Ext.HostARM64.Targetx64", + "Microsoft.VC.14.34.17.4.CA.Ext.HostARM64.Targetx64.base", + "Microsoft.VC.14.34.17.4.CA.Ext.HostARM64.Targetx64.Res.base", + "Microsoft.VisualCpp.CA.Ext.HostARM64.Targetx86", + "Microsoft.VC.14.34.17.4.CA.Ext.HostARM64.Targetx86.base", + "Microsoft.VC.14.34.17.4.CA.Ext.HostARM64.Targetx86.Res.base", + "Microsoft.VisualCpp.CA.Ext.Hostx86.Targetx64", + "Microsoft.VC.14.34.17.4.CA.Ext.Hostx86.Targetx64.base", + "Microsoft.VC.14.34.17.4.CA.Ext.Hostx86.Targetx64.Res.base", + "Microsoft.VisualCpp.CA.Ext.Hostx86.Targetx86", + "Microsoft.VC.14.34.17.4.CA.Ext.Hostx86.Targetx86.base", + "Microsoft.VC.14.34.17.4.CA.Ext.Hostx86.Targetx86.Res.base", + "Microsoft.VisualCpp.CA.Ext.Hostx64.Targetx64", + "Microsoft.VC.14.34.17.4.CA.Ext.Hostx64.Targetx64.base", + "Microsoft.VC.14.34.17.4.CA.Ext.Hostx64.Targetx64.Res.base", + "Microsoft.VisualCpp.CA.Ext.Hostx64.Targetx86", + "Microsoft.VC.14.34.17.4.CA.Ext.Hostx64.Targetx86.base", + "Microsoft.VC.14.34.17.4.Servicing.CAExtensions", + "Microsoft.VC.14.34.17.4.CA.Ext.Hostx64.Targetx86.Res.base", + "Microsoft.VisualCpp.Tools.HostX64.TargetX86", + "Microsoft.VC.14.34.17.4.Tools.HostX64.TargetX86.base", + "Microsoft.VC.14.34.17.4.Tools.HostX64.TargetX86.Res.base", + "Microsoft.VisualCpp.Tools.HostX64.TargetX64", + "Microsoft.VC.14.34.17.4.Tools.HostX64.TargetX64.base", + "Microsoft.VC.14.34.17.4.Tools.HostX64.TargetX64.Res.base", + "Microsoft.VisualCpp.Tools.HostARM64.TargetX86", + "Microsoft.VC.14.34.17.4.Tools.HostARM64.TargetX86.base", + "Microsoft.VisualCpp.RuntimeDebug.14", + "Microsoft.VC.14.34.17.4.Tools.HostARM64.TargetX86.Res.base", + "Microsoft.VisualCpp.Tools.HostARM64.TargetX64", + "Microsoft.VC.14.34.17.4.Tools.HostARM64.TargetX64.base", + "Microsoft.VisualCpp.RuntimeDebug.14.ARM64", + "Microsoft.VisualCpp.Redist.14.Latest", + "Microsoft.VisualCpp.Redist.14.Latest", + "Microsoft.VC.14.34.17.4.Tools.HostARM64.Targetx64.Res.base", + "Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX64", + "Microsoft.VC.14.34.17.4.Premium.Tools.HostX86.TargetX64.base", + "Microsoft.VC.14.34.17.4.Prem.Hostx86.Targetx64.Res.base", + "Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86", + "Microsoft.VC.14.34.17.4.Premium.Tools.HostX86.TargetX86.base", + "Microsoft.VC.14.34.17.4.Prem.HostX86.TargetX86.Res.base", + "Microsoft.VisualCpp.Premium.Tools.HostARM64.TargetX86", + "Microsoft.VC.14.34.17.4.Premium.Tools.HostARM64.TargetX86.base", + "Microsoft.VC.14.34.17.4.Prem.HostARM64.TargetX86.Res.base", + "Microsoft.VisualCpp.Premium.Tools.HostARM64.TargetX64", + "Microsoft.VC.14.34.17.4.Premium.Tools.HostARM64.TargetX64.base", + "Microsoft.VC.14.34.17.4.Prem.HostARM64.Targetx64.Res.base", + "Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86", + "Microsoft.VC.14.34.17.4.Premium.Tools.HostX64.TargetX86.base", + "Microsoft.VC.14.34.17.4.Prem.HostX64.TargetX86.Res.base", + "Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64", + "Microsoft.VC.14.34.17.4.Premium.Tools.HostX64.TargetX64.base", + "Microsoft.VC.14.34.17.4.Prem.HostX64.TargetX64.Res.base", + "Microsoft.VisualCpp.PGO.X86", + "Microsoft.VC.14.34.17.4.PGO.X86.base", + "Microsoft.VisualCpp.PGO.X64", + "Microsoft.VC.14.34.17.4.PGO.X64.base", + "Microsoft.VisualCpp.PGO.Headers", + "Microsoft.VC.14.34.17.4.PGO.Headers.base", + "Microsoft.VisualCpp.CRT.x86.Store", + "Microsoft.VC.14.34.17.4.CRT.x86.Store.base", + "Microsoft.VisualCpp.CRT.x86.OneCore.Desktop", + "Microsoft.VC.14.34.17.4.CRT.x86.OneCore.Desktop.base", + "Microsoft.VisualCpp.CRT.x64.Store", + "Microsoft.VC.14.34.17.4.CRT.x64.Store.base", + "Microsoft.VisualCpp.CRT.x64.OneCore.Desktop", + "Microsoft.VC.14.34.17.4.CRT.x64.OneCore.Desktop.base", + "Microsoft.VisualCpp.CRT.Redist.x86.OneCore.Desktop", + "Microsoft.VC.14.34.17.4.CRT.Redist.x86.OneCore.Desktop.base", + "Microsoft.VisualCpp.CRT.Redist.x64.OneCore.Desktop", + "Microsoft.VC.14.34.17.4.CRT.Redist.x64.OneCore.Desktop.base", + "Microsoft.VisualStudio.PackageGroup.VC.Tools.x86", + "Microsoft.VisualCpp.Tools.Hostx86.Targetx64.Res", + "Microsoft.VisualCpp.Tools.HostX86.TargetX64", + "Microsoft.VC.14.34.17.4.Tools.HostX86.TargetX64.base", + "Microsoft.VC.14.34.17.4.Props.x64", + "Microsoft.VC.14.34.17.4.Tools.Hostx86.Targetx64.Res.base", + "Microsoft.VisualCpp.Tools.HostX86.TargetX86.Res", + "Microsoft.VisualCpp.Tools.HostX86.TargetX86", + "Microsoft.VC.14.34.17.4.Tools.HostX86.TargetX86.base", + "Microsoft.VC.14.34.17.4.Servicing.Compilers", + "Microsoft.VC.14.34.17.4.Props.x86", + "Microsoft.VC.14.34.17.4.Props", + "Microsoft.VC.14.34.17.4.Tools.HostX86.TargetX86.Res.base", + "Microsoft.VisualCpp.Tools.Core.Resources", + "Microsoft.VisualCpp.Tools.Core.x86", + "Microsoft.VC.14.34.17.4.Tools.Core.Props", + "Microsoft.VisualCpp.DIA.SDK", + "Microsoft.VisualCpp.Servicing.DIASDK", + "Microsoft.VisualCpp.CRT.x86.Desktop", + "Microsoft.VC.14.34.17.4.CRT.x86.Desktop.base", + "Microsoft.VisualCpp.CRT.x64.Desktop", + "Microsoft.VC.14.34.17.4.CRT.x64.Desktop.base", + "Microsoft.VisualCpp.CRT.Source", + "Microsoft.VC.14.34.17.4.CRT.Source.base", + "Microsoft.VisualCpp.CRT.Redist.X86", + "Microsoft.VC.14.34.17.4.CRT.Redist.X86.base", + "Microsoft.VisualCpp.CRT.Redist.X64", + "Microsoft.VisualCpp.CRT.Redist.Resources", + "Microsoft.VC.14.34.17.4.CRT.Redist.X64.base", + "Microsoft.VisualCpp.CRT.Headers", + "Microsoft.VC.14.34.17.4.CRT.Headers.base", + "Microsoft.VC.14.34.17.4.Servicing.CrtHeaders", + "Microsoft.VC.14.34.17.4.Servicing", + "Microsoft.VisualStudio.Component.VC.CoreIde", + "Microsoft.VisualStudio.VC.Ide.Pro", + "Microsoft.VisualStudio.VC.Ide.Pro.Resources", + "Microsoft.VisualStudio.VC.Templates.General", + "Microsoft.VisualStudio.VC.Templates.General.Resources", + "Microsoft.VisualStudio.VC.Items.Pro", + "Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Reduced", + "Microsoft.VisualStudio.VC.Ide.x64", + "Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Express", + "Microsoft.VisualStudio.VC.vcvars", + "Microsoft.VS.VC.vcvars.x86.Shortcuts", + "Microsoft.VS.VC.vcvars.x64.Shortcuts", + "Microsoft.VS.VC.vcvars.arm64_x64.Shortcuts", + "Microsoft.VisualStudio.VC.MSBuild.v170.X64.v143", + "Microsoft.VisualStudio.VC.MSBuild.v170.X64", + "Microsoft.VisualStudio.VC.MSBuild.v170.ARM.v143", + "Microsoft.VisualStudio.VC.MSBuild.v170.ARM", + "Microsoft.VisualStudio.VC.MSBuild.v170.x86.v143", + "Microsoft.VisualStudio.VC.MSBuild.v170.X86", + "Microsoft.VisualStudio.VC.MSBuild.v170.Base", + "Microsoft.VisualStudio.VC.MSBuild.v170.Base.Resources", + "Microsoft.VisualStudio.VC.Ide.WinXPlus", + "Microsoft.VisualStudio.VC.Ide.Dskx", + "Microsoft.VisualStudio.VC.Ide.Dskx.Resources", + "Microsoft.VisualStudio.VC.Ide.Base", + "Microsoft.VisualStudio.VC.Ide.LanguageService", + "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.Scripts", + "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.PythonDistro", + "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.10", + "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.9", + "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.8", + "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.7", + "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.6", + "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.5", + "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.4", + "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.3", + "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.2", + "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.1", + "Microsoft.VisualStudio.VC.Ide.VCPkgDatabase", + "Microsoft.VisualStudio.VC.Ide.Core", + "Microsoft.VisualStudio.VC.Ide.ProjectSystem", + "Microsoft.VisualStudio.VC.Ide.ProjectSystem.Resources", + "Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine", + "Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine.Resources", + "Microsoft.VisualStudio.VC.Ide.LanguageService.Resources", + "Microsoft.VisualStudio.VC.Llvm.Base", + "Microsoft.VisualStudio.VC.Ide.Base.Resources", + "Microsoft.Net.PackageGroup.4.8.1.Redist", + "Microsoft.VisualStudio.Component.IntelliCode", + "Microsoft.VisualStudio.IntelliCode.CSharp", + "Microsoft.VisualStudio.IntelliCode", + "Component.Microsoft.VisualStudio.LiveShare.2022", + "Microsoft.VisualStudio.Component.Debugger.JustInTime", + "Microsoft.VisualStudio.Debugger.ImmersiveActivateHelper.Msi", + "Microsoft.VisualStudio.Debugger.JustInTime", + "Microsoft.VisualStudio.Debugger.JustInTime.Msi", + "Microsoft.VisualStudio.LiveShare.2022", + "Microsoft.Icecap.Analysis", + "Microsoft.Icecap.Analysis.Resources", + "Microsoft.Icecap.Analysis.Resources.Targeted", + "Microsoft.Icecap.Collection.Msi", + "Microsoft.Icecap.Collection.Msi.Targeted", + "Microsoft.Icecap.Collection.Msi.Resources", + "Microsoft.Icecap.Collection.Msi.Resources.Targeted", + "Microsoft.DiagnosticsHub.Instrumentation", + "Microsoft.DiagnosticsHub.Instrumentation.Targeted", + "Microsoft.DiagnosticsHub.CpuSampling", + "Microsoft.DiagnosticsHub.CpuSampling.Targeted", + "Microsoft.PackageGroup.DiagnosticsHub.Platform", + "Microsoft.VisualStudio.InstrumentationEngine.ARM64", + "Microsoft.VisualStudio.InstrumentationEngine", + "Microsoft.DiagnosticsHub.Runtime.ExternalDependencies", + "SQLiteCore", + "SQLiteCore.Targeted", + "Microsoft.DiagnosticsHub.Runtime.ExternalDependencies.Targeted", + "Microsoft.DiagnosticsHub.Runtime", + "Microsoft.DiagnosticsHub.Runtime.Targeted", + "Microsoft.DiagnosticsHub.Collection.ExternalDependencies.arm64", + "Microsoft.DiagnosticsHub.Collection", + "Microsoft.DiagnosticsHub.Collection.Service", + "Microsoft.VisualStudio.VC.Ide.MDD", + "Microsoft.VisualStudio.VC.Ide.Linux.ConnectionManager", + "Microsoft.VisualStudio.VisualC.Utilities", + "Microsoft.VisualStudio.VisualC.Utilities.Resources", + "Microsoft.VisualStudio.VC.Ide.Linux.ConnectionManager.Resources", + "Microsoft.VisualStudio.VC.Ide.ResourceEditor", + "Microsoft.VisualStudio.VC.Ide.ResourceEditor.Resources", + "Microsoft.VisualStudio.PackageGroup.TestTools.Core", + "Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.V2.CLI", + "Microsoft.VisualStudio.TestTools.TestPlatform.V2.CLI", + "Microsoft.VisualStudio.TestTools.Pex.Common", + "Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.V1.CLI", + "Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.Legacy", + "Microsoft.VisualStudio.PackageGroup.MinShell.Interop", + "Microsoft.VisualStudio.TestTools.TP.Legacy.Tips.Msi", + "Microsoft.VisualStudio.TestTools.TP.Legacy.Tips.Common", + "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Tips", + "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Tips.Resources", + "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.TestSettings", + "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Professional", + "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Common", + "Microsoft.VisualStudio.TestTools.TP.Legacy.Common.Res", + "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Core", + "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Core.Resources", + "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Agent", + "Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.IDE", + "Microsoft.VisualStudio.Cache.Service", + "Microsoft.VisualStudio.TestTools.TestWIExtension", + "Microsoft.VisualStudio.TestTools.TestPlatform.V1.CLI", + "Microsoft.VisualStudio.TestTools.TestPlatform.IDE", + "Microsoft.VisualStudio.PackageGroup.TestTools.CodeCoverage", + "Microsoft.VisualStudio.PackageGroup.TestTools.DataCollectors", + "Microsoft.VisualStudio.Component.NuGet", + "Microsoft.CredentialProvider", + "Microsoft.VisualStudio.NuGet.Licenses", + "Microsoft.VisualStudio.Component.TextTemplating", + "Microsoft.VisualStudio.TextTemplating.MSBuild", + "Microsoft.VisualStudio.TextTemplating.Integration", + "Microsoft.VisualStudio.TextTemplating.Core", + "Microsoft.VisualStudio.TextTemplating.Integration.Resources", + "Microsoft.VisualCpp.CRT.ClickOnce.Msi", + "Microsoft.VisualStudio.Component.Roslyn.LanguageServices", + "Microsoft.VisualStudio.InteractiveWindow", + "Microsoft.DiaSymReader.Native", + "Microsoft.VisualCpp.Redist.14", + "Microsoft.VisualCpp.Redist.14", + "Microsoft.VisualCpp.Servicing.Redist", + "Microsoft.VisualStudio.PackageGroup.StaticAnalysis", + "Microsoft.VisualStudio.StaticAnalysis.IDE", + "Microsoft.VisualStudio.StaticAnalysis.IDE.Resources", + "Microsoft.VisualStudio.StaticAnalysis.FxCop.Resources", + "Microsoft.VisualStudio.StaticAnalysis.auxil", + "Microsoft.VisualStudio.StaticAnalysis.auxil.Resources", + "Roslyn.VisualStudio.Setup.ServiceHub", + "Microsoft.Component.MSBuild", + "Microsoft.NuGet.Build.Tasks.Setup", + "Microsoft.VisualStudio.Component.Roslyn.Compiler", + "Microsoft.CodeAnalysis.Compilers", + "Microsoft.VisualStudio.Component.JavaScript.TypeScript", + "Microsoft.VisualStudio.JavaScript.ProjectSystem", + "Microsoft.VisualStudio.ComponentGroup.WebToolsExtensions", + "Microsoft.VisualStudio.ProTools", + "sqlsysclrtypes", + "SQLCommon", + "Microsoft.VisualStudio.ProTools.Resources", + "Microsoft.VisualStudio.Web.Scaffolding", + "Microsoft.VisualStudio.WebToolsExtensions", + "Microsoft.VisualStudio.ConnectedServices.Core", + "Microsoft.VisualStudio.WebTools", + "Microsoft.VisualStudio.WebToolsExtensions.MSBuild", + "Microsoft.VisualStudio.WebTools.Resources", + "Microsoft.VisualStudio.WebTools.WSP.FSA", + "Microsoft.VisualStudio.WebTools.WSP.FSA.Resources", + "Microsoft.VisualStudio.PackageGroup.Debugger.Script", + "Microsoft.VisualStudio.Component.TypeScript.TSServer", + "Microsoft.VisualStudio.Package.TypeScript.TSServer", + "Microsoft.VisualStudio.PackageGroup.JavaScript.Language", + "Microsoft.VisualStudio.Package.NodeJs", + "TypeScript.Build", + "TypeScript.LanguageService", + "TypeScript.Tools", + "Microsoft.VisualStudio.PackageGroup.Community", + "Microsoft.VisualStudio.Community.VB.x86", + "Microsoft.VisualStudio.Community.VB.x64", + "Microsoft.VisualStudio.PackageGroup.Core", + "Microsoft.VisualStudio.CodeSense.Community", + "Microsoft.VisualStudio.TestTools.TeamFoundationClient", + "Microsoft.VisualStudio.PackageGroup.Debugger.Core", + "Microsoft.VisualStudio.Debugger.BrokeredServices", + "Microsoft.VisualStudio.Debugger.VSCodeDebuggerHost", + "Microsoft.VisualStudio.Debugger.AzureAttach", + "Microsoft.VisualStudio.Web.Azure.Common", + "Microsoft.WebTools.Shared", + "Microsoft.WebTools.DotNet.Core.ItemTemplates", + "Microsoft.VisualStudio.PackageGroup.Debugger.TimeTravel.Replay", + "Microsoft.VisualStudio.VC.Ide.Debugger", + "Microsoft.VisualStudio.VC.Ide.Debugger.Concord", + "Microsoft.VisualStudio.VC.Ide.Debugger.Concord.Resources", + "Microsoft.VisualStudio.VC.Ide.Debugger.Resources", + "Microsoft.VisualStudio.VC.Ide.Common", + "Microsoft.VisualStudio.VC.Ide.Common.Resources", + "Microsoft.VisualStudio.Debugger.CollectionAgents", + "Microsoft.VisualStudio.Debugger.Parallel", + "Microsoft.VisualStudio.Debugger.Parallel.Resources", + "Microsoft.VisualStudio.Debugger.Managed", + "Microsoft.CodeAnalysis.ExpressionEvaluator", + "Microsoft.CodeAnalysis.VisualStudio.Setup", + "Microsoft.VisualStudio.Debugger.Concord.Managed", + "Microsoft.VisualStudio.Debugger.Concord.Managed.Resources", + "Microsoft.VisualStudio.Debugger.Managed.Resources", + "Microsoft.VisualStudio.Debugger.TargetComposition", + "Microsoft.VisualStudio.Debugger.TargetComposition.Remote.arm64", + "Microsoft.VisualStudio.Debugger.TargetComposition.Remote", + "Microsoft.VisualStudio.Debugger.TargetComposition.Remote", + "Microsoft.VisualStudio.Debugger.Remote", + "Microsoft.VisualStudio.Debugger.Concord.Remote", + "Microsoft.VisualStudio.Debugger.Concord.Remote.Resources", + "Microsoft.VisualStudio.Debugger.Remote", + "Microsoft.VisualStudio.Debugger.Remote.ARM64", + "Microsoft.VisualStudio.Debugger.Concord.Remote.ARM64", + "Microsoft.VisualStudio.Debugger.Concord.Remote.Resources.ARM64", + "Microsoft.VisualStudio.Debugger.Remote.ARM", + "Microsoft.VisualStudio.Debugger.Concord.Remote.ARM", + "Microsoft.VisualStudio.Debugger.Concord.Remote.Resources.ARM", + "Microsoft.VisualStudio.Debugger.Remote.Resources.ARM", + "Microsoft.VisualStudio.Debugger.Remote.Resources.ARM64", + "Microsoft.VisualStudio.Debugger.Concord.Remote", + "Microsoft.VisualStudio.Debugger.Concord.Remote.Resources", + "Microsoft.VisualStudio.Debugger.Remote.Resources", + "Microsoft.VisualStudio.Debugger.Remote.Resources", + "Microsoft.VisualStudio.Debugger", + "Microsoft.VisualStudio.VC.MSVCDis", + "Microsoft.IntelliTrace.DiagnosticsHub", + "Microsoft.VisualStudio.Debugger.Concord", + "Microsoft.VisualStudio.Debugger.Concord.Resources", + "Microsoft.VisualStudio.Debugger.Resources", + "Microsoft.VisualStudio.Debugger.Package.DiagHub.Client", + "Microsoft.VisualStudio.Debugger.Remote.DiagnosticsHub.Client", + "Microsoft.VisualStudio.Debugger.Remote.DiagnosticsHub.Client", + "Microsoft.VisualStudio.Debugger.Remote.DiagnosticsHub.Client", + "Microsoft.PackageGroup.ClientDiagnostics", + "Microsoft.VisualStudio.AppResponsiveness", + "Microsoft.VisualStudio.AppResponsiveness.Targeted", + "Microsoft.VisualStudio.AppResponsiveness.Resources", + "Microsoft.VisualStudio.ClientDiagnostics", + "Microsoft.VisualStudio.ClientDiagnostics.Targeted", + "Microsoft.VisualStudio.ClientDiagnostics.Resources", + "Microsoft.VisualStudio.PackageGroup.CommunityCore", + "Microsoft.VisualStudio.ProjectSystem.Full", + "Microsoft.VisualStudio.LiveShareApi", + "Microsoft.VisualStudio.ProjectSystem.Query", + "Microsoft.VisualStudio.ProjectSystem", + "Microsoft.VisualStudio.Community.x86", + "Microsoft.VisualStudio.Community.x64", + "Microsoft.VisualStudio.Community.Msi.Resources", + "Microsoft.VisualStudio.Community.Msi", + "Microsoft.VisualStudio.Community.Shared.Msi", + "Microsoft.VisualStudio.Devenv.Msi", + "Microsoft.VisualStudio.Devenv.Shared.Msi", + "Microsoft.VisualStudio.MinShell.Interop.Msi", + "Microsoft.VisualStudio.MinShell.Interop.Shared.Msi", + "Microsoft.VisualStudio.Editors", + "Microsoft.VisualStudio.Workload.CoreEditor", + "Microsoft.VisualStudio.Component.CoreEditor", + "Microsoft.VisualStudio.PackageGroup.CoreEditor", + "Microsoft.WebView2", + "Microsoft.VisualStudio.ScriptedHost", + "Microsoft.VisualStudio.ScriptedHost.Targeted", + "Microsoft.VisualCpp.Tools.Common.UtilsPrereq", + "Microsoft.VisualCpp.Tools.Common.Utils", + "Microsoft.VisualCpp.Tools.Common.Utils.Resources", + "Microsoft.VisualStudio.PackageGroup.VsDevCmd", + "Microsoft.VisualStudio.VsDevCmd.Ext.NetFxSdk", + "Microsoft.VisualStudio.VsDevCmd.Core.WinSdk", + "Microsoft.VisualStudio.VsDevCmd.Core.DotNet", + "Microsoft.VisualStudio.VC.DevCmd", + "Microsoft.VisualStudio.VC.DevCmd.Resources", + "Microsoft.VisualStudio.VirtualTree", + "Microsoft.DiaSymReader", + "Microsoft.Build.Dependencies", + "Microsoft.Build.FileTracker.Msi", + "Microsoft.Build", + "Microsoft.VisualStudio.PackageGroup.NuGet", + "Microsoft.DataAI.NuGetRecommender", + "Microsoft.VisualStudio.NuGet.Core", + "Microsoft.Build.Arm64", + "Microsoft.Build.UnGAC", + "Microsoft.VisualStudio.TextMateGrammars", + "Microsoft.VisualStudio.Platform.Markdown", + "Microsoft.VisualStudio.Platform.CrossRepositorySearch", + "Microsoft.VisualStudio.PackageGroup.TeamExplorer.Common", + "Microsoft.VisualStudio.TeamExplorer", + "Microsoft.VisualStudio.PackageGroup.ServiceHub", + "Microsoft.ServiceHub.Node", + "Microsoft.ServiceHub.Managed", + "Microsoft.ServiceHub.arm64", + "Microsoft.VisualStudio.ProjectServices", + "Microsoft.VisualStudio.OpenFolder.VSIX", + "Microsoft.VisualStudio.FileHandler.Msi", + "Microsoft.VisualStudio.FileHandler.Msi", + "Microsoft.VisualStudio.PackageGroup.MinShell", + "Microsoft.VisualStudio.MinShell.Msi", + "Microsoft.VisualStudio.MinShell.Shared.Msi", + "Microsoft.VisualStudio.MinShell.Msi.Resources", + "Microsoft.VisualStudio.MinShell.Interop", + "CoreEditorFonts", + "Microsoft.VisualStudio.Log", + "Microsoft.VisualStudio.Log.Targeted", + "Microsoft.VisualStudio.Log.Resources", + "Microsoft.VisualStudio.Finalizer", + "Microsoft.VisualStudio.Devenv", + "Microsoft.VisualStudio.Devenv.Resources", + "Microsoft.VisualStudio.CoreEditor", + "Microsoft.VisualStudio.Navigation.RichCodeNav", + "Microsoft.VisualStudio.Platform.NavigateTo", + "Microsoft.VisualStudio.Connected", + "SQLitePCLRaw", + "SQLitePCLRaw.Targeted", + "Microsoft.VisualStudio.Connected.Auto", + "Microsoft.VisualStudio.Connected.Auto.Resources", + "Microsoft.VisualStudio.AzureSDK", + "Microsoft.VisualStudio.PerfLib", + "Microsoft.VisualStudio.Connected.Resources", + "Microsoft.Net.PackageGroup.4.8.Redist", + "Microsoft.VisualStudio.PackageGroup.Progression", + "Microsoft.VisualStudio.PerformanceProvider", + "Microsoft.VisualStudio.GraphModel", + "Microsoft.VisualStudio.GraphProvider", + "Microsoft.VisualStudio.Community.VB.Targeted", + "Microsoft.VisualStudio.Community.VB.Neutral", + "Microsoft.VisualStudio.Community.CSharp.Targeted", + "Microsoft.VisualStudio.Community.CSharp.Neutral", + "Microsoft.VisualStudio.Community.ProductArch.TargetedExtra", + "Microsoft.VisualStudio.Community.ProductArch.Targeted", + "Microsoft.VisualStudio.Community.ProductArch.NeutralExtra", + "Microsoft.DiaSymReader.PortablePdb", + "Microsoft.IntelliTrace.CollectorCab", + "Microsoft.VisualStudio.Community.VB.Resources.Targeted", + "Microsoft.VisualStudio.Community.VB.Resources.Neutral", + "Microsoft.VisualStudio.Community.CSharp.Resources.Targeted", + "Microsoft.VisualStudio.Community.CSharp.Resources.Neutral", + "Microsoft.VisualStudio.Community.ProductArch.Resources.Targeted", + "Microsoft.VisualStudio.Community.ProductArch.Resources.NeutralExtra", + "Microsoft.VisualStudio.Net.Eula.Resources", + "Microsoft.VisualStudio.Community.ProductArch.Resources.Neutral", + "Microsoft.VisualStudio.WebSiteProject.DTE", + "Microsoft.VisualStudio.Diagnostics.AspNetHelper", + "Microsoft.VisualStudio.Diagnostics.AspNetHelper.Standard", + "Microsoft.MSHtml", + "Microsoft.VisualStudio.Platform.CallHierarchy", + "Microsoft.VisualStudio.Community.ProductArch.Neutral", + "Microsoft.VisualStudio.MinShell", + "Microsoft.VisualStudio.VsWebProtocolSelector.Msi", + "Microsoft.Net.6.WindowsDesktop.Runtime", + "Microsoft.Net.6.Runtime", + "Microsoft.VisualStudio.PackageGroup.Setup.Common", + "Microsoft.VisualStudio.Setup.WMIProvider", + "Microsoft.VisualStudio.Setup.Configuration.Interop", + "Microsoft.VisualStudio.Setup.Configuration", + "Microsoft.VisualStudio.Extensibility.Container", + "Microsoft.VisualStudio.LanguageServer", + "Microsoft.VisualStudio.Platform.Terminal", + "Microsoft.VisualStudio.MefHosting", + "Microsoft.VisualStudio.Initializer", + "Microsoft.VisualStudio.ExtensionManager", + "Microsoft.VisualStudio.Platform.Editor", + "Microsoft.VisualStudio.MinShell.Targeted", + "Microsoft.VisualStudio.NativeImageSupport", + "Microsoft.VisualStudio.Devenv.Config", + "Microsoft.VisualStudio.MinShell.Resources.arm64", + "Microsoft.VisualStudio.MinShell.Auto", + "Microsoft.VisualStudio.MinShell.Auto.Resources", + "Microsoft.VisualStudio.Branding.Community" + ] + } +] diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js index 1327cf8841..6cb4f10a52 100644 --- a/test/test-find-visualstudio.js +++ b/test/test-find-visualstudio.js @@ -411,6 +411,43 @@ test('VS2019 Community with C++ workload', function (t) { finder.findVisualStudio() }) +test('VS2022 Preview with C++ workload', function (t) { + t.plan(2) + + const msBuildPath = process.arch === 'arm64' + ? 'C:\\Program Files\\Microsoft Visual Studio\\2022\\' + + 'Community\\MSBuild\\Current\\Bin\\arm64\\MSBuild.exe' + : 'C:\\Program Files\\Microsoft Visual Studio\\2022\\' + + 'Community\\MSBuild\\Current\\Bin\\MSBuild.exe' + + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info, { + msBuild: msBuildPath, + path: + 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community', + sdk: '10.0.22621.0', + toolset: 'v143', + version: '17.4.33213.308', + versionMajor: 17, + versionMinor: 4, + versionYear: 2022 + }) + }) + + poison(finder, 'regSearchKeys') + finder.msBuildPathExists = (path) => { + return true + } + finder.findVisualStudio2017OrNewer = (cb) => { + const file = path.join(__dirname, 'fixtures', + 'VS_2022_Community_workload.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '', cb) + } + finder.findVisualStudio() +}) + function allVsVersions (t, finder) { finder.findVisualStudio2017OrNewer = (cb) => { const data0 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', @@ -427,8 +464,10 @@ function allVsVersions (t, finder) { 'VS_2019_BuildTools_minimal.txt'))) const data6 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', 'VS_2019_Community_workload.txt'))) + const data7 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', + 'VS_2022_Community_workload.txt'))) const data = JSON.stringify(data0.concat(data1, data2, data3, data4, - data5, data6)) + data5, data6, data7)) finder.parseData(null, data, '', cb) } finder.regSearchKeys = (keys, value, addOpts, cb) => { @@ -570,6 +609,22 @@ test('look for VS2019 by installation path', function (t) { finder.findVisualStudio() }) +test('look for VS2022 by version number', function (t) { + t.plan(2) + + const finder = new TestVisualStudioFinder(semverV1, '2022', (err, info) => { + t.strictEqual(err, null) + t.deepEqual(info.versionYear, 2022) + }) + + finder.msBuildPathExists = (path) => { + return true + } + + allVsVersions(t, finder) + finder.findVisualStudio() +}) + test('msvs_version match should be case insensitive', function (t) { t.plan(2) @@ -590,9 +645,13 @@ test('latest version should be found by default', function (t) { const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { t.strictEqual(err, null) - t.deepEqual(info.versionYear, 2019) + t.deepEqual(info.versionYear, 2022) }) + finder.msBuildPathExists = (path) => { + return true + } + allVsVersions(t, finder) finder.findVisualStudio() }) From aaa117c514430aa2c1e568b95df1b6ed1c1fd3b6 Mon Sep 17 00:00:00 2001 From: David Sanders Date: Thu, 25 May 2023 08:24:07 -0700 Subject: [PATCH 368/551] fix: extract tarball to temp directory on Windows (#2846) * fix: check for errors while extracting downloaded tarball Signed-off-by: David Sanders * test: parallel installs Signed-off-by: David Sanders * fix: extract tarball to temp directory on Windows Signed-off-by: David Sanders --------- Signed-off-by: David Sanders --- lib/install.js | 181 ++++++++++++++++++++++++++++-------------- package.json | 5 +- test/test-download.js | 2 +- test/test-install.js | 86 +++++++++++++++++++- 4 files changed, 212 insertions(+), 62 deletions(-) diff --git a/lib/install.js b/lib/install.js index 99f6d8592a..f6a44c48d3 100644 --- a/lib/install.js +++ b/lib/install.js @@ -2,6 +2,8 @@ const fs = require('graceful-fs') const os = require('os') +const { backOff } = require('exponential-backoff') +const rm = require('rimraf') const tar = require('tar') const path = require('path') const util = require('util') @@ -98,6 +100,40 @@ async function install (fs, gyp, argv) { } } + async function copyDirectory (src, dest) { + try { + await fs.promises.stat(src) + } catch { + throw new Error(`Missing source directory for copy: ${src}`) + } + await fs.promises.mkdir(dest, { recursive: true }) + const entries = await fs.promises.readdir(src, { withFileTypes: true }) + for (const entry of entries) { + if (entry.isDirectory()) { + await copyDirectory(path.join(src, entry.name), path.join(dest, entry.name)) + } else if (entry.isFile()) { + // with parallel installs, copying files may cause file errors on + // Windows so use an exponential backoff to resolve collisions + await backOff(async () => { + try { + await fs.promises.copyFile(path.join(src, entry.name), path.join(dest, entry.name)) + } catch (err) { + // if ensure, check if file already exists and that's good enough + if (gyp.opts.ensure && err.code === 'EBUSY') { + try { + await fs.promises.stat(path.join(dest, entry.name)) + return + } catch {} + } + throw err + } + }) + } else { + throw new Error('Unexpected file directory entry type') + } + } + } + async function go () { log.verbose('ensuring nodedir is created', devDir) @@ -118,6 +154,7 @@ async function install (fs, gyp, argv) { // now download the node tarball const tarPath = gyp.opts.tarball + let extractErrors = false let extractCount = 0 const contentShasums = {} const expectShasums = {} @@ -136,71 +173,99 @@ async function install (fs, gyp, argv) { return isValid } + function onwarn (code, message) { + extractErrors = true + log.error('error while extracting tarball', code, message) + } + // download the tarball and extract! - if (tarPath) { - await tar.extract({ - file: tarPath, - strip: 1, - filter: isValid, - cwd: devDir - }) - } else { - try { - const res = await download(gyp, release.tarballUrl) + // on Windows there can be file errors from tar if parallel installs + // are happening (not uncommon with multiple native modules) so + // extract the tarball to a temp directory first and then copy over + const tarExtractDir = win ? await fs.promises.mkdtemp(path.join(os.tmpdir(), 'node-gyp-tmp-')) : devDir - if (res.status !== 200) { - throw new Error(`${res.status} response downloading ${release.tarballUrl}`) - } + try { + if (tarPath) { + await tar.extract({ + file: tarPath, + strip: 1, + filter: isValid, + onwarn, + cwd: tarExtractDir + }) + } else { + try { + const res = await download(gyp, release.tarballUrl) - await streamPipeline( - res.body, - // content checksum - new ShaSum((_, checksum) => { - const filename = path.basename(release.tarballUrl).trim() - contentShasums[filename] = checksum - log.verbose('content checksum', filename, checksum) - }), - tar.extract({ - strip: 1, - cwd: devDir, - filter: isValid - }) - ) - } catch (err) { - // something went wrong downloading the tarball? - if (err.code === 'ENOTFOUND') { - throw new Error('This is most likely not a problem with node-gyp or the package itself and\n' + - 'is related to network connectivity. In most cases you are behind a proxy or have bad \n' + - 'network settings.') + if (res.status !== 200) { + throw new Error(`${res.status} response downloading ${release.tarballUrl}`) + } + + await streamPipeline( + res.body, + // content checksum + new ShaSum((_, checksum) => { + const filename = path.basename(release.tarballUrl).trim() + contentShasums[filename] = checksum + log.verbose('content checksum', filename, checksum) + }), + tar.extract({ + strip: 1, + cwd: tarExtractDir, + filter: isValid, + onwarn + }) + ) + } catch (err) { + // something went wrong downloading the tarball? + if (err.code === 'ENOTFOUND') { + throw new Error('This is most likely not a problem with node-gyp or the package itself and\n' + + 'is related to network connectivity. In most cases you are behind a proxy or have bad \n' + + 'network settings.') + } + throw err } - throw err } - } - // invoked after the tarball has finished being extracted - if (extractCount === 0) { - throw new Error('There was a fatal problem while downloading/extracting the tarball') - } + // invoked after the tarball has finished being extracted + if (extractErrors || extractCount === 0) { + throw new Error('There was a fatal problem while downloading/extracting the tarball') + } + + log.verbose('tarball', 'done parsing tarball') + + const installVersionPath = path.resolve(tarExtractDir, 'installVersion') + await Promise.all([ + // need to download node.lib + ...(win ? downloadNodeLib() : []), + // write the "installVersion" file + fs.promises.writeFile(installVersionPath, gyp.package.installVersion + '\n'), + // Only download SHASUMS.txt if we downloaded something in need of SHA verification + ...(!tarPath || win ? [downloadShasums()] : []) + ]) + + log.verbose('download contents checksum', JSON.stringify(contentShasums)) + // check content shasums + for (const k in contentShasums) { + log.verbose('validating download checksum for ' + k, '(%s == %s)', contentShasums[k], expectShasums[k]) + if (contentShasums[k] !== expectShasums[k]) { + throw new Error(k + ' local checksum ' + contentShasums[k] + ' not match remote ' + expectShasums[k]) + } + } - log.verbose('tarball', 'done parsing tarball') - - const installVersionPath = path.resolve(devDir, 'installVersion') - await Promise.all([ - // need to download node.lib - ...(win ? downloadNodeLib() : []), - // write the "installVersion" file - fs.promises.writeFile(installVersionPath, gyp.package.installVersion + '\n'), - // Only download SHASUMS.txt if we downloaded something in need of SHA verification - ...(!tarPath || win ? [downloadShasums()] : []) - ]) - - log.verbose('download contents checksum', JSON.stringify(contentShasums)) - // check content shasums - for (const k in contentShasums) { - log.verbose('validating download checksum for ' + k, '(%s == %s)', contentShasums[k], expectShasums[k]) - if (contentShasums[k] !== expectShasums[k]) { - throw new Error(k + ' local checksum ' + contentShasums[k] + ' not match remote ' + expectShasums[k]) + // copy over the files from the temp tarball extract directory to devDir + if (tarExtractDir !== devDir) { + await copyDirectory(tarExtractDir, devDir) + } + } finally { + if (tarExtractDir !== devDir) { + try { + // try to cleanup temp dir + await util.promisify(rm)(tarExtractDir) + } catch { + log.warn('failed to clean up temp tarball extract directory') + } } } @@ -232,7 +297,7 @@ async function install (fs, gyp, argv) { log.verbose('on Windows; need to download `' + release.name + '.lib`...') const archs = ['ia32', 'x64', 'arm64'] return archs.map(async (arch) => { - const dir = path.resolve(devDir, arch) + const dir = path.resolve(tarExtractDir, arch) const targetLibPath = path.resolve(dir, release.name + '.lib') const { libUrl, libPath } = release[arch] const name = `${arch} ${release.name}.lib` diff --git a/package.json b/package.json index 11b10b69f0..8964222bfd 100644 --- a/package.json +++ b/package.json @@ -12,7 +12,7 @@ "gyp" ], "version": "9.3.1", - "installVersion": 9, + "installVersion": 10, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { "type": "git", @@ -23,6 +23,7 @@ "main": "./lib/node-gyp.js", "dependencies": { "env-paths": "^2.2.0", + "exponential-backoff": "^3.1.1", "glob": "^7.1.4", "graceful-fs": "^4.2.6", "make-fetch-happen": "^11.0.3", @@ -45,6 +46,6 @@ }, "scripts": { "lint": "standard */*.js test/**/*.js", - "test": "npm run lint && tap --timeout=600 test/test-*" + "test": "npm run lint && tap --timeout=1200 test/test-*" } } diff --git a/test/test-download.js b/test/test-download.js index c4caad9e83..582358dc4f 100644 --- a/test/test-download.js +++ b/test/test-download.js @@ -188,7 +188,7 @@ test('download headers (actual)', async (t) => { await util.promisify(install)(prog, []) const data = await fs.promises.readFile(path.join(expectedDir, 'installVersion'), 'utf8') - t.strictEqual(data, '9\n', 'correct installVersion') + t.strictEqual(data, '10\n', 'correct installVersion') const list = await fs.promises.readdir(path.join(expectedDir, 'include/node')) t.ok(list.includes('common.gypi')) diff --git a/test/test-install.js b/test/test-install.js index 5039dc992e..e1612cfadf 100644 --- a/test/test-install.js +++ b/test/test-install.js @@ -1,8 +1,16 @@ 'use strict' const { test } = require('tap') -const { test: { install } } = require('../lib/install') +const path = require('path') +const os = require('os') +const util = require('util') +const { test: { download, install } } = require('../lib/install') +const rimraf = require('rimraf') +const gyp = require('../lib/node-gyp') const log = require('npmlog') +const semver = require('semver') +const stream = require('stream') +const streamPipeline = util.promisify(stream.pipeline) log.level = 'error' // we expect a warning @@ -44,3 +52,79 @@ test('EACCES retry once', async (t) => { } } }) + +// only run these tests if we are running a version of Node with predictable version path behavior +const skipParallelInstallTests = process.env.FAST_TEST || + process.release.name !== 'node' || + semver.prerelease(process.version) !== null || + semver.satisfies(process.version, '<10') + +async function parallelInstallsTest (t, fs, devDir, prog) { + if (skipParallelInstallTests) { + return t.skip('Skipping parallel installs test due to test environment configuration') + } + + t.tearDown(async () => { + await util.promisify(rimraf)(devDir) + }) + + const expectedDir = path.join(devDir, process.version.replace(/^v/, '')) + await util.promisify(rimraf)(expectedDir) + + await Promise.all([ + install(fs, prog, []), + install(fs, prog, []), + install(fs, prog, []), + install(fs, prog, []), + install(fs, prog, []), + install(fs, prog, []), + install(fs, prog, []), + install(fs, prog, []), + install(fs, prog, []), + install(fs, prog, []) + ]) +} + +test('parallel installs (ensure=true)', async (t) => { + const fs = require('graceful-fs') + const devDir = await util.promisify(fs.mkdtemp)(path.join(os.tmpdir(), 'node-gyp-test-')) + + const prog = gyp() + prog.parseArgv([]) + prog.devDir = devDir + prog.opts.ensure = true + log.level = 'warn' + + await parallelInstallsTest(t, fs, devDir, prog) +}) + +test('parallel installs (ensure=false)', async (t) => { + const fs = require('graceful-fs') + const devDir = await util.promisify(fs.mkdtemp)(path.join(os.tmpdir(), 'node-gyp-test-')) + + const prog = gyp() + prog.parseArgv([]) + prog.devDir = devDir + prog.opts.ensure = false + log.level = 'warn' + + await parallelInstallsTest(t, fs, devDir, prog) +}) + +test('parallel installs (tarball)', async (t) => { + const fs = require('graceful-fs') + const devDir = await util.promisify(fs.mkdtemp)(path.join(os.tmpdir(), 'node-gyp-test-')) + + const prog = gyp() + prog.parseArgv([]) + prog.devDir = devDir + prog.opts.tarball = path.join(devDir, 'node-headers.tar.gz') + log.level = 'warn' + + await streamPipeline( + (await download(prog, 'https://nodejs.org/dist/v16.0.0/node-v16.0.0.tar.gz')).body, + fs.createWriteStream(prog.opts.tarball) + ) + + await parallelInstallsTest(t, fs, devDir, prog) +}) From 5df2b72a8fc124eb7c785912c8142dc6997c0819 Mon Sep 17 00:00:00 2001 From: Stefan Stojanovic Date: Mon, 5 Jun 2023 17:05:29 +0200 Subject: [PATCH 369/551] Migration from tap to mocha (#2851) * migrate from tap to mocha After make-fetch-happen update GitHub Actions started failing. Migrating from tap to mocha testing framework for GitHub Action stability. * write custom test reporter for more verbose output Implemented a simple custom mocha test reporter to replace the default one. Made test report more developer friendly. --- package.json | 6 +- test/reporter.js | 75 ++ test/test-addon.js | 200 +++--- test/test-configure-python.js | 111 ++- test/test-create-config-gypi.js | 91 ++- test/test-download.js | 365 +++++----- test/test-find-accessible-sync.js | 99 ++- test/test-find-node-directory.js | 196 +++-- test/test-find-python.js | 353 +++++---- test/test-find-visualstudio.js | 1109 ++++++++++++++--------------- test/test-install.js | 193 ++--- test/test-options.js | 59 +- test/test-process-release.js | 751 ++++++++++--------- 13 files changed, 1773 insertions(+), 1835 deletions(-) create mode 100644 test/reporter.js diff --git a/package.json b/package.json index 8964222bfd..15cb6a72ea 100644 --- a/package.json +++ b/package.json @@ -39,13 +39,13 @@ }, "devDependencies": { "bindings": "^1.5.0", + "mocha": "^10.2.0", "nan": "^2.14.2", "require-inject": "^1.4.4", - "standard": "^14.3.4", - "tap": "^12.7.0" + "standard": "^14.3.4" }, "scripts": { "lint": "standard */*.js test/**/*.js", - "test": "npm run lint && tap --timeout=1200 test/test-*" + "test": "npm run lint && mocha --reporter=test/reporter.js test/test-download.js test/test-*" } } diff --git a/test/reporter.js b/test/reporter.js new file mode 100644 index 0000000000..9964b1b5d5 --- /dev/null +++ b/test/reporter.js @@ -0,0 +1,75 @@ +const Mocha = require('mocha') + +class Reporter { + constructor (runner) { + this.failedTests = [] + + runner.on(Mocha.Runner.constants.EVENT_RUN_BEGIN, () => { + console.log('Starting tests') + }) + + runner.on(Mocha.Runner.constants.EVENT_RUN_END, () => { + console.log('Tests finished') + console.log() + console.log('****************') + console.log('* TESTS REPORT *') + console.log('****************') + console.log() + console.log(`Executed ${runner.stats.suites} suites with ${runner.stats.tests} tests in ${runner.stats.duration} ms`) + console.log(` Passed: ${runner.stats.passes}`) + console.log(` Skipped: ${runner.stats.pending}`) + console.log(` Failed: ${runner.stats.failures}`) + if (this.failedTests.length > 0) { + console.log() + console.log(' Failed test details') + this.failedTests.forEach((failedTest, index) => { + console.log() + console.log(` ${index + 1}.'${failedTest.test.fullTitle()}'`) + console.log(` Name: ${failedTest.error.name}`) + console.log(` Message: ${failedTest.error.message}`) + console.log(` Code: ${failedTest.error.code}`) + console.log(` Stack: ${failedTest.error.stack}`) + }) + } + console.log() + }) + + runner.on(Mocha.Runner.constants.EVENT_SUITE_BEGIN, (suite) => { + if (suite.root) { + return + } + console.log(`Starting suite '${suite.title}'`) + }) + + runner.on(Mocha.Runner.constants.EVENT_SUITE_END, (suite) => { + if (suite.root) { + return + } + console.log(`Suite '${suite.title}' finished`) + console.log() + }) + + runner.on(Mocha.Runner.constants.EVENT_TEST_BEGIN, (test) => { + console.log(`Starting test '${test.title}'`) + }) + + runner.on(Mocha.Runner.constants.EVENT_TEST_PASS, (test) => { + console.log(`Test '${test.title}' passed in ${test.duration} ms`) + }) + + runner.on(Mocha.Runner.constants.EVENT_TEST_PENDING, (test) => { + console.log(`Test '${test.title}' skipped in ${test.duration} ms`) + }) + + runner.on(Mocha.Runner.constants.EVENT_TEST_FAIL, (test, error) => { + this.failedTests.push({ test, error }) + console.log(`Test '${test.title}' failed in ${test.duration} ms with ${error}`) + }) + + runner.on(Mocha.Runner.constants.EVENT_TEST_END, (test) => { + console.log() + }) + } +} + +module.exports = Reporter diff --git a/test/test-addon.js b/test/test-addon.js index f79eff73c1..43556620a8 100644 --- a/test/test-addon.js +++ b/test/test-addon.js @@ -1,6 +1,7 @@ 'use strict' -const test = require('tap').test +const { describe, it } = require('mocha') +const assert = require('assert') const path = require('path') const fs = require('graceful-fs') const childProcess = require('child_process') @@ -35,116 +36,117 @@ function checkCharmapValid () { return lines.pop() === 'True' } -test('build simple addon', function (t) { - t.plan(3) - - // Set the loglevel otherwise the output disappears when run via 'npm test' - var cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose'] - var proc = execFile(process.execPath, cmd, function (err, stdout, stderr) { - var logLines = stderr.toString().trim().split(/\r?\n/) - var lastLine = logLines[logLines.length - 1] - t.strictEqual(err, null) - t.strictEqual(lastLine, 'gyp info ok', 'should end in ok') - t.strictEqual(runHello().trim(), 'world') +describe('addon', function () { + this.timeout(300000) + + it('build simple addon', function (done) { + // Set the loglevel otherwise the output disappears when run via 'npm test' + var cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose'] + var proc = execFile(process.execPath, cmd, function (err, stdout, stderr) { + var logLines = stderr.toString().trim().split(/\r?\n/) + var lastLine = logLines[logLines.length - 1] + assert.strictEqual(err, null) + assert.strictEqual(lastLine, 'gyp info ok', 'should end in ok') + assert.strictEqual(runHello().trim(), 'world') + done() + }) + proc.stdout.setEncoding('utf-8') + proc.stderr.setEncoding('utf-8') }) - proc.stdout.setEncoding('utf-8') - proc.stderr.setEncoding('utf-8') -}) - -test('build simple addon in path with non-ascii characters', function (t) { - t.plan(1) - if (!checkCharmapValid()) { - return t.skip('python console app can\'t encode non-ascii character.') - } + it('build simple addon in path with non-ascii characters', function (done) { + if (!checkCharmapValid()) { + return this.skip('python console app can\'t encode non-ascii character.') + } - var testDirNames = { - cp936: '文件夹', - cp1252: 'Latīna', - cp932: 'フォルダ' - } - // Select non-ascii characters by current encoding - var testDirName = testDirNames[getEncoding()] - // If encoding is UTF-8 or other then no need to test - if (!testDirName) { - return t.skip('no need to test') - } + var testDirNames = { + cp936: '文件夹', + cp1252: 'Latīna', + cp932: 'フォルダ' + } + // Select non-ascii characters by current encoding + var testDirName = testDirNames[getEncoding()] + // If encoding is UTF-8 or other then no need to test + if (!testDirName) { + return this.skip('no need to test') + } - t.plan(3) + this.timeout(300000) - var data - var configPath = path.join(addonPath, 'build', 'config.gypi') - try { - data = fs.readFileSync(configPath, 'utf8') - } catch (err) { - t.error(err) - return - } - var config = JSON.parse(data.replace(/#.+\n/, '')) - var nodeDir = config.variables.nodedir - var testNodeDir = path.join(addonPath, testDirName) - // Create symbol link to path with non-ascii characters - try { - fs.symlinkSync(nodeDir, testNodeDir, 'dir') - } catch (err) { - switch (err.code) { - case 'EEXIST': break - case 'EPERM': - t.error(err, 'Please try to running console as an administrator') - return - default: - t.error(err) - return + var data + var configPath = path.join(addonPath, 'build', 'config.gypi') + try { + data = fs.readFileSync(configPath, 'utf8') + } catch (err) { + assert.fail(err) + return } - } - - var cmd = [ - nodeGyp, - 'rebuild', - '-C', - addonPath, - '--loglevel=verbose', - '-nodedir=' + testNodeDir - ] - var proc = execFile(process.execPath, cmd, function (err, stdout, stderr) { + var config = JSON.parse(data.replace(/#.+\n/, '')) + var nodeDir = config.variables.nodedir + var testNodeDir = path.join(addonPath, testDirName) + // Create symbol link to path with non-ascii characters try { - fs.unlink(testNodeDir) + fs.symlinkSync(nodeDir, testNodeDir, 'dir') } catch (err) { - t.error(err) + switch (err.code) { + case 'EEXIST': break + case 'EPERM': + assert.fail(err, null, 'Please try to running console as an administrator') + return + default: + assert.fail(err) + return + } } - var logLines = stderr.toString().trim().split(/\r?\n/) - var lastLine = logLines[logLines.length - 1] - t.strictEqual(err, null) - t.strictEqual(lastLine, 'gyp info ok', 'should end in ok') - t.strictEqual(runHello().trim(), 'world') + var cmd = [ + nodeGyp, + 'rebuild', + '-C', + addonPath, + '--loglevel=verbose', + '-nodedir=' + testNodeDir + ] + var proc = execFile(process.execPath, cmd, function (err, stdout, stderr) { + try { + fs.unlink(testNodeDir) + } catch (err) { + assert.fail(err) + } + + var logLines = stderr.toString().trim().split(/\r?\n/) + var lastLine = logLines[logLines.length - 1] + assert.strictEqual(err, null) + assert.strictEqual(lastLine, 'gyp info ok', 'should end in ok') + assert.strictEqual(runHello().trim(), 'world') + done() + }) + proc.stdout.setEncoding('utf-8') + proc.stderr.setEncoding('utf-8') }) - proc.stdout.setEncoding('utf-8') - proc.stderr.setEncoding('utf-8') -}) - -test('addon works with renamed host executable', function (t) { - // No `fs.copyFileSync` before node8. - if (process.version.substr(1).split('.')[0] < 8) { - t.skip('skipping test for old node version') - t.end() - return - } - t.plan(3) - - var notNodePath = path.join(os.tmpdir(), 'notnode' + path.extname(process.execPath)) - fs.copyFileSync(process.execPath, notNodePath) + it('addon works with renamed host executable', function (done) { + // No `fs.copyFileSync` before node8. + if (process.version.substr(1).split('.')[0] < 8) { + return this.skip('skipping test for old node version') + } - var cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose'] - var proc = execFile(process.execPath, cmd, function (err, stdout, stderr) { - var logLines = stderr.toString().trim().split(/\r?\n/) - var lastLine = logLines[logLines.length - 1] - t.strictEqual(err, null) - t.strictEqual(lastLine, 'gyp info ok', 'should end in ok') - t.strictEqual(runHello(notNodePath).trim(), 'world') - fs.unlinkSync(notNodePath) + this.timeout(300000) + + var notNodePath = path.join(os.tmpdir(), 'notnode' + path.extname(process.execPath)) + fs.copyFileSync(process.execPath, notNodePath) + + var cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose'] + var proc = execFile(process.execPath, cmd, function (err, stdout, stderr) { + var logLines = stderr.toString().trim().split(/\r?\n/) + var lastLine = logLines[logLines.length - 1] + assert.strictEqual(err, null) + assert.strictEqual(lastLine, 'gyp info ok', 'should end in ok') + assert.strictEqual(runHello(notNodePath).trim(), 'world') + fs.unlinkSync(notNodePath) + done() + }) + proc.stdout.setEncoding('utf-8') + proc.stderr.setEncoding('utf-8') }) - proc.stdout.setEncoding('utf-8') - proc.stderr.setEncoding('utf-8') }) diff --git a/test/test-configure-python.js b/test/test-configure-python.js index aacd75f7c7..ab1e5511fa 100644 --- a/test/test-configure-python.js +++ b/test/test-configure-python.js @@ -1,6 +1,7 @@ 'use strict' -const test = require('tap').test +const { describe, it } = require('mocha') +const assert = require('assert') const path = require('path') const devDir = require('./common').devDir() const gyp = require('../lib/node-gyp') @@ -22,63 +23,59 @@ const configure = requireInject('../lib/configure', { const EXPECTED_PYPATH = path.join(__dirname, '..', 'gyp', 'pylib') const SEPARATOR = process.platform === 'win32' ? ';' : ':' -const SPAWN_RESULT = { on: function () { } } +const SPAWN_RESULT = cb => ({ on: function () { cb() } }) require('npmlog').level = 'warn' -test('configure PYTHONPATH with no existing env', function (t) { - t.plan(1) - - delete process.env.PYTHONPATH - - var prog = gyp() - prog.parseArgv([]) - prog.spawn = function () { - t.equal(process.env.PYTHONPATH, EXPECTED_PYPATH) - return SPAWN_RESULT - } - prog.devDir = devDir - configure(prog, [], t.fail) -}) - -test('configure PYTHONPATH with existing env of one dir', function (t) { - t.plan(2) - - var existingPath = path.join('a', 'b') - process.env.PYTHONPATH = existingPath - - var prog = gyp() - prog.parseArgv([]) - prog.spawn = function () { - t.equal(process.env.PYTHONPATH, [EXPECTED_PYPATH, existingPath].join(SEPARATOR)) - - var dirs = process.env.PYTHONPATH.split(SEPARATOR) - t.deepEqual(dirs, [EXPECTED_PYPATH, existingPath]) - - return SPAWN_RESULT - } - prog.devDir = devDir - configure(prog, [], t.fail) -}) - -test('configure PYTHONPATH with existing env of multiple dirs', function (t) { - t.plan(2) - - var pythonDir1 = path.join('a', 'b') - var pythonDir2 = path.join('b', 'c') - var existingPath = [pythonDir1, pythonDir2].join(SEPARATOR) - process.env.PYTHONPATH = existingPath - - var prog = gyp() - prog.parseArgv([]) - prog.spawn = function () { - t.equal(process.env.PYTHONPATH, [EXPECTED_PYPATH, existingPath].join(SEPARATOR)) - - var dirs = process.env.PYTHONPATH.split(SEPARATOR) - t.deepEqual(dirs, [EXPECTED_PYPATH, pythonDir1, pythonDir2]) - - return SPAWN_RESULT - } - prog.devDir = devDir - configure(prog, [], t.fail) +describe('configure-python', function () { + it('configure PYTHONPATH with no existing env', function (done) { + delete process.env.PYTHONPATH + + var prog = gyp() + prog.parseArgv([]) + prog.spawn = function () { + assert.strictEqual(process.env.PYTHONPATH, EXPECTED_PYPATH) + return SPAWN_RESULT(done) + } + prog.devDir = devDir + configure(prog, [], assert.fail) + }) + + it('configure PYTHONPATH with existing env of one dir', function (done) { + var existingPath = path.join('a', 'b') + process.env.PYTHONPATH = existingPath + + var prog = gyp() + prog.parseArgv([]) + prog.spawn = function () { + assert.strictEqual(process.env.PYTHONPATH, [EXPECTED_PYPATH, existingPath].join(SEPARATOR)) + + var dirs = process.env.PYTHONPATH.split(SEPARATOR) + assert.deepStrictEqual(dirs, [EXPECTED_PYPATH, existingPath]) + + return SPAWN_RESULT(done) + } + prog.devDir = devDir + configure(prog, [], assert.fail) + }) + + it('configure PYTHONPATH with existing env of multiple dirs', function (done) { + var pythonDir1 = path.join('a', 'b') + var pythonDir2 = path.join('b', 'c') + var existingPath = [pythonDir1, pythonDir2].join(SEPARATOR) + process.env.PYTHONPATH = existingPath + + var prog = gyp() + prog.parseArgv([]) + prog.spawn = function () { + assert.strictEqual(process.env.PYTHONPATH, [EXPECTED_PYPATH, existingPath].join(SEPARATOR)) + + var dirs = process.env.PYTHONPATH.split(SEPARATOR) + assert.deepStrictEqual(dirs, [EXPECTED_PYPATH, pythonDir1, pythonDir2]) + + return SPAWN_RESULT(done) + } + prog.devDir = devDir + configure(prog, [], assert.fail) + }) }) diff --git a/test/test-create-config-gypi.js b/test/test-create-config-gypi.js index eeac73fab1..725819b6aa 100644 --- a/test/test-create-config-gypi.js +++ b/test/test-create-config-gypi.js @@ -1,70 +1,61 @@ 'use strict' const path = require('path') -const { test } = require('tap') +const { describe, it } = require('mocha') +const assert = require('assert') const gyp = require('../lib/node-gyp') const createConfigGypi = require('../lib/create-config-gypi') const { parseConfigGypi, getCurrentConfigGypi } = createConfigGypi.test -test('config.gypi with no options', async function (t) { - t.plan(2) +describe('create-config-gypi', function () { + it('config.gypi with no options', async function () { + const prog = gyp() + prog.parseArgv([]) - const prog = gyp() - prog.parseArgv([]) + const config = await getCurrentConfigGypi({ gyp: prog, vsInfo: {} }) + assert.strictEqual(config.target_defaults.default_configuration, 'Release') + assert.strictEqual(config.variables.target_arch, process.arch) + }) - const config = await getCurrentConfigGypi({ gyp: prog, vsInfo: {} }) - t.equal(config.target_defaults.default_configuration, 'Release') - t.equal(config.variables.target_arch, process.arch) -}) - -test('config.gypi with --debug', async function (t) { - t.plan(1) - - const prog = gyp() - prog.parseArgv(['_', '_', '--debug']) + it('config.gypi with --debug', async function () { + const prog = gyp() + prog.parseArgv(['_', '_', '--debug']) - const config = await getCurrentConfigGypi({ gyp: prog, vsInfo: {} }) - t.equal(config.target_defaults.default_configuration, 'Debug') -}) + const config = await getCurrentConfigGypi({ gyp: prog, vsInfo: {} }) + assert.strictEqual(config.target_defaults.default_configuration, 'Debug') + }) -test('config.gypi with custom options', async function (t) { - t.plan(1) + it('config.gypi with custom options', async function () { + const prog = gyp() + prog.parseArgv(['_', '_', '--shared-libxml2']) - const prog = gyp() - prog.parseArgv(['_', '_', '--shared-libxml2']) + const config = await getCurrentConfigGypi({ gyp: prog, vsInfo: {} }) + assert.strictEqual(config.variables.shared_libxml2, true) + }) - const config = await getCurrentConfigGypi({ gyp: prog, vsInfo: {} }) - t.equal(config.variables.shared_libxml2, true) -}) + it('config.gypi with nodedir', async function () { + const nodeDir = path.join(__dirname, 'fixtures', 'nodedir') -test('config.gypi with nodedir', async function (t) { - t.plan(1) + const prog = gyp() + prog.parseArgv(['_', '_', `--nodedir=${nodeDir}`]) - const nodeDir = path.join(__dirname, 'fixtures', 'nodedir') + const config = await getCurrentConfigGypi({ gyp: prog, nodeDir, vsInfo: {} }) + assert.strictEqual(config.variables.build_with_electron, true) + }) - const prog = gyp() - prog.parseArgv(['_', '_', `--nodedir=${nodeDir}`]) + it('config.gypi with --force-process-config', async function () { + const nodeDir = path.join(__dirname, 'fixtures', 'nodedir') - const config = await getCurrentConfigGypi({ gyp: prog, nodeDir, vsInfo: {} }) - t.equal(config.variables.build_with_electron, true) -}) - -test('config.gypi with --force-process-config', async function (t) { - t.plan(1) - - const nodeDir = path.join(__dirname, 'fixtures', 'nodedir') - - const prog = gyp() - prog.parseArgv(['_', '_', '--force-process-config', `--nodedir=${nodeDir}`]) - - const config = await getCurrentConfigGypi({ gyp: prog, nodeDir, vsInfo: {} }) - t.equal(config.variables.build_with_electron, undefined) -}) + const prog = gyp() + prog.parseArgv(['_', '_', '--force-process-config', `--nodedir=${nodeDir}`]) -test('config.gypi parsing', function (t) { - t.plan(1) + const config = await getCurrentConfigGypi({ gyp: prog, nodeDir, vsInfo: {} }) + assert.strictEqual(config.variables.build_with_electron, undefined) + }) - const str = "# Some comments\n{'variables': {'multiline': 'A'\n'B'}}" - const config = parseConfigGypi(str) - t.deepEqual(config, { variables: { multiline: 'AB' } }) + it('config.gypi parsing', function () { + const str = "# Some comments\n{'variables': {'multiline': 'A'\n'B'}}" + const config = parseConfigGypi(str) + assert.deepStrictEqual(config, { variables: { multiline: 'AB' } }) + }) }) diff --git a/test/test-download.js b/test/test-download.js index 582358dc4f..6eeba8a1dd 100644 --- a/test/test-download.js +++ b/test/test-download.js @@ -1,6 +1,7 @@ 'use strict' -const { test } = require('tap') +const { describe, it, after } = require('mocha') +const assert = require('assert') const fs = require('fs') const path = require('path') const util = require('util') @@ -16,202 +17,194 @@ const certs = require('./fixtures/certs') log.level = 'warn' -test('download over http', async (t) => { - t.plan(2) - - const server = http.createServer((req, res) => { - t.strictEqual(req.headers['user-agent'], `node-gyp v42 (node ${process.version})`) - res.end('ok') - }) - - t.tearDown(() => new Promise((resolve) => server.close(resolve))) - - const host = 'localhost' - await new Promise((resolve) => server.listen(0, host, resolve)) - const { port } = server.address() - const gyp = { - opts: {}, - version: '42' - } - const url = `http://${host}:${port}` - const res = await install.test.download(gyp, url) - t.strictEqual(await res.text(), 'ok') -}) - -test('download over https with custom ca', async (t) => { - t.plan(3) - - const cafile = path.join(__dirname, 'fixtures/ca.crt') - const cacontents = certs['ca.crt'] - const cert = certs['server.crt'] - const key = certs['server.key'] - await fs.promises.writeFile(cafile, cacontents, 'utf8') - const ca = await install.test.readCAFile(cafile) - - t.strictEqual(ca.length, 1) - - const options = { ca: ca, cert: cert, key: key } - const server = https.createServer(options, (req, res) => { - t.strictEqual(req.headers['user-agent'], `node-gyp v42 (node ${process.version})`) - res.end('ok') +describe('download', function () { + it('download over http', async function () { + const server = http.createServer((req, res) => { + assert.strictEqual(req.headers['user-agent'], `node-gyp v42 (node ${process.version})`) + res.end('ok') + }) + + after(() => new Promise((resolve) => server.close(resolve))) + + const host = 'localhost' + await new Promise((resolve) => server.listen(0, host, resolve)) + const { port } = server.address() + const gyp = { + opts: {}, + version: '42' + } + const url = `http://${host}:${port}` + const res = await install.test.download(gyp, url) + assert.strictEqual(await res.text(), 'ok') }) - t.tearDown(async () => { - await new Promise((resolve) => server.close(resolve)) - await fs.promises.unlink(cafile) + it('download over https with custom ca', async function () { + const cafile = path.join(__dirname, 'fixtures/ca.crt') + const cacontents = certs['ca.crt'] + const cert = certs['server.crt'] + const key = certs['server.key'] + await fs.promises.writeFile(cafile, cacontents, 'utf8') + const ca = await install.test.readCAFile(cafile) + + assert.strictEqual(ca.length, 1) + + const options = { ca: ca, cert: cert, key: key } + const server = https.createServer(options, (req, res) => { + assert.strictEqual(req.headers['user-agent'], `node-gyp v42 (node ${process.version})`) + res.end('ok') + }) + + after(async () => { + await new Promise((resolve) => server.close(resolve)) + await fs.promises.unlink(cafile) + }) + + server.on('clientError', (err) => { throw err }) + + const host = 'localhost' + await new Promise((resolve) => server.listen(0, host, resolve)) + const { port } = server.address() + const gyp = { + opts: { cafile }, + version: '42' + } + const url = `https://${host}:${port}` + const res = await install.test.download(gyp, url) + assert.strictEqual(await res.text(), 'ok') }) - server.on('clientError', (err) => { throw err }) - - const host = 'localhost' - await new Promise((resolve) => server.listen(0, host, resolve)) - const { port } = server.address() - const gyp = { - opts: { cafile }, - version: '42' - } - const url = `https://${host}:${port}` - const res = await install.test.download(gyp, url) - t.strictEqual(await res.text(), 'ok') -}) - -test('download over http with proxy', async (t) => { - t.plan(2) - - const server = http.createServer((_, res) => { - res.end('ok') + it('download over http with proxy', async function () { + const server = http.createServer((_, res) => { + res.end('ok') + }) + + const pserver = http.createServer((req, res) => { + assert.strictEqual(req.headers['user-agent'], `node-gyp v42 (node ${process.version})`) + res.end('proxy ok') + }) + + after(() => Promise.all([ + new Promise((resolve) => server.close(resolve)), + new Promise((resolve) => pserver.close(resolve)) + ])) + + const host = 'localhost' + await new Promise((resolve) => server.listen(0, host, resolve)) + const { port } = server.address() + await new Promise((resolve) => pserver.listen(port + 1, host, resolve)) + const gyp = { + opts: { + proxy: `http://${host}:${port + 1}`, + noproxy: 'bad' + }, + version: '42' + } + const url = `http://${host}:${port}` + const res = await install.test.download(gyp, url) + assert.strictEqual(await res.text(), 'proxy ok') }) - const pserver = http.createServer((req, res) => { - t.strictEqual(req.headers['user-agent'], `node-gyp v42 (node ${process.version})`) - res.end('proxy ok') + it('download over http with noproxy', async function () { + const server = http.createServer((req, res) => { + assert.strictEqual(req.headers['user-agent'], `node-gyp v42 (node ${process.version})`) + res.end('ok') + }) + + const pserver = http.createServer((_, res) => { + res.end('proxy ok') + }) + + after(() => Promise.all([ + new Promise((resolve) => server.close(resolve)), + new Promise((resolve) => pserver.close(resolve)) + ])) + + const host = 'localhost' + await new Promise((resolve) => server.listen(0, host, resolve)) + const { port } = server.address() + await new Promise((resolve) => pserver.listen(port + 1, host, resolve)) + const gyp = { + opts: { + proxy: `http://${host}:${port + 1}`, + noproxy: host + }, + version: '42' + } + const url = `http://${host}:${port}` + const res = await install.test.download(gyp, url) + assert.strictEqual(await res.text(), 'ok') }) - t.tearDown(() => Promise.all([ - new Promise((resolve) => server.close(resolve)), - new Promise((resolve) => pserver.close(resolve)) - ])) - - const host = 'localhost' - await new Promise((resolve) => server.listen(0, host, resolve)) - const { port } = server.address() - await new Promise((resolve) => pserver.listen(port + 1, host, resolve)) - const gyp = { - opts: { - proxy: `http://${host}:${port + 1}`, - noproxy: 'bad' - }, - version: '42' - } - const url = `http://${host}:${port}` - const res = await install.test.download(gyp, url) - t.strictEqual(await res.text(), 'proxy ok') -}) - -test('download over http with noproxy', async (t) => { - t.plan(2) - - const server = http.createServer((req, res) => { - t.strictEqual(req.headers['user-agent'], `node-gyp v42 (node ${process.version})`) - res.end('ok') + it('download with missing cafile', async function () { + const gyp = { + opts: { cafile: 'no.such.file' } + } + try { + await install.test.download(gyp, {}, 'http://bad/') + } catch (e) { + assert.ok(/no.such.file/.test(e.message)) + } }) - const pserver = http.createServer((_, res) => { - res.end('proxy ok') + it('check certificate splitting', async function () { + const cafile = path.join(__dirname, 'fixtures/ca-bundle.crt') + const cacontents = certs['ca-bundle.crt'] + await fs.promises.writeFile(cafile, cacontents, 'utf8') + after(async () => { + await fs.promises.unlink(cafile) + }) + const cas = await install.test.readCAFile(path.join(__dirname, 'fixtures/ca-bundle.crt')) + assert.strictEqual(cas.length, 2) + assert.notStrictEqual(cas[0], cas[1]) }) - t.tearDown(() => Promise.all([ - new Promise((resolve) => server.close(resolve)), - new Promise((resolve) => pserver.close(resolve)) - ])) - - const host = 'localhost' - await new Promise((resolve) => server.listen(0, host, resolve)) - const { port } = server.address() - await new Promise((resolve) => pserver.listen(port + 1, host, resolve)) - const gyp = { - opts: { - proxy: `http://${host}:${port + 1}`, - noproxy: host - }, - version: '42' - } - const url = `http://${host}:${port}` - const res = await install.test.download(gyp, url) - t.strictEqual(await res.text(), 'ok') -}) - -test('download with missing cafile', async (t) => { - t.plan(1) - const gyp = { - opts: { cafile: 'no.such.file' } - } - try { - await install.test.download(gyp, {}, 'http://bad/') - } catch (e) { - t.ok(/no.such.file/.test(e.message)) - } -}) - -test('check certificate splitting', async (t) => { - const cafile = path.join(__dirname, 'fixtures/ca-bundle.crt') - const cacontents = certs['ca-bundle.crt'] - await fs.promises.writeFile(cafile, cacontents, 'utf8') - t.tearDown(async () => { - await fs.promises.unlink(cafile) + // only run this test if we are running a version of Node with predictable version path behavior + + it('download headers (actual)', async function () { + if (process.env.FAST_TEST || + process.release.name !== 'node' || + semver.prerelease(process.version) !== null || + semver.satisfies(process.version, '<10')) { + return this.skip('Skipping actual download of headers due to test environment configuration') + } + + this.timeout(300000) + + const expectedDir = path.join(devDir, process.version.replace(/^v/, '')) + await util.promisify(rimraf)(expectedDir) + + const prog = gyp() + prog.parseArgv([]) + prog.devDir = devDir + log.level = 'warn' + await util.promisify(install)(prog, []) + + const data = await fs.promises.readFile(path.join(expectedDir, 'installVersion'), 'utf8') + assert.strictEqual(data, '10\n', 'correct installVersion') + + const list = await fs.promises.readdir(path.join(expectedDir, 'include/node')) + assert.ok(list.includes('common.gypi')) + assert.ok(list.includes('config.gypi')) + assert.ok(list.includes('node.h')) + assert.ok(list.includes('node_version.h')) + assert.ok(list.includes('openssl')) + assert.ok(list.includes('uv')) + assert.ok(list.includes('uv.h')) + assert.ok(list.includes('v8-platform.h')) + assert.ok(list.includes('v8.h')) + assert.ok(list.includes('zlib.h')) + + const lines = (await fs.promises.readFile(path.join(expectedDir, 'include/node/node_version.h'), 'utf8')).split('\n') + + // extract the 3 version parts from the defines to build a valid version string and + // and check them against our current env version + const version = ['major', 'minor', 'patch'].reduce((version, type) => { + const re = new RegExp(`^#define\\sNODE_${type.toUpperCase()}_VERSION`) + const line = lines.find((l) => re.test(l)) + const i = line ? parseInt(line.replace(/^[^0-9]+([0-9]+).*$/, '$1'), 10) : 'ERROR' + return `${version}${type !== 'major' ? '.' : 'v'}${i}` + }, '') + + assert.strictEqual(version, process.version) }) - const cas = await install.test.readCAFile(path.join(__dirname, 'fixtures/ca-bundle.crt')) - t.plan(2) - t.strictEqual(cas.length, 2) - t.notStrictEqual(cas[0], cas[1]) -}) - -// only run this test if we are running a version of Node with predictable version path behavior - -test('download headers (actual)', async (t) => { - if (process.env.FAST_TEST || - process.release.name !== 'node' || - semver.prerelease(process.version) !== null || - semver.satisfies(process.version, '<10')) { - return t.skip('Skipping actual download of headers due to test environment configuration') - } - - t.plan(12) - - const expectedDir = path.join(devDir, process.version.replace(/^v/, '')) - await util.promisify(rimraf)(expectedDir) - - const prog = gyp() - prog.parseArgv([]) - prog.devDir = devDir - log.level = 'warn' - await util.promisify(install)(prog, []) - - const data = await fs.promises.readFile(path.join(expectedDir, 'installVersion'), 'utf8') - t.strictEqual(data, '10\n', 'correct installVersion') - - const list = await fs.promises.readdir(path.join(expectedDir, 'include/node')) - t.ok(list.includes('common.gypi')) - t.ok(list.includes('config.gypi')) - t.ok(list.includes('node.h')) - t.ok(list.includes('node_version.h')) - t.ok(list.includes('openssl')) - t.ok(list.includes('uv')) - t.ok(list.includes('uv.h')) - t.ok(list.includes('v8-platform.h')) - t.ok(list.includes('v8.h')) - t.ok(list.includes('zlib.h')) - - const lines = (await fs.promises.readFile(path.join(expectedDir, 'include/node/node_version.h'), 'utf8')).split('\n') - - // extract the 3 version parts from the defines to build a valid version string and - // and check them against our current env version - const version = ['major', 'minor', 'patch'].reduce((version, type) => { - const re = new RegExp(`^#define\\sNODE_${type.toUpperCase()}_VERSION`) - const line = lines.find((l) => re.test(l)) - const i = line ? parseInt(line.replace(/^[^0-9]+([0-9]+).*$/, '$1'), 10) : 'ERROR' - return `${version}${type !== 'major' ? '.' : 'v'}${i}` - }, '') - - t.strictEqual(version, process.version) }) diff --git a/test/test-find-accessible-sync.js b/test/test-find-accessible-sync.js index 0a2e584c4f..7edbc0c764 100644 --- a/test/test-find-accessible-sync.js +++ b/test/test-find-accessible-sync.js @@ -1,6 +1,7 @@ 'use strict' -const test = require('tap').test +const { describe, it } = require('mocha') +const assert = require('assert') const path = require('path') const requireInject = require('require-inject') const configure = requireInject('../lib/configure', { @@ -27,58 +28,46 @@ const readableFiles = [ path.resolve(dir, readableFileInDir) ] -test('find accessible - empty array', function (t) { - t.plan(1) - - var candidates = [] - var found = configure.test.findAccessibleSync('test', dir, candidates) - t.strictEqual(found, undefined) -}) - -test('find accessible - single item array, readable', function (t) { - t.plan(1) - - var candidates = [readableFile] - var found = configure.test.findAccessibleSync('test', dir, candidates) - t.strictEqual(found, path.resolve(dir, readableFile)) -}) - -test('find accessible - single item array, readable in subdir', function (t) { - t.plan(1) - - var candidates = [readableFileInDir] - var found = configure.test.findAccessibleSync('test', dir, candidates) - t.strictEqual(found, path.resolve(dir, readableFileInDir)) -}) - -test('find accessible - single item array, unreadable', function (t) { - t.plan(1) - - var candidates = ['unreadable_file'] - var found = configure.test.findAccessibleSync('test', dir, candidates) - t.strictEqual(found, undefined) -}) - -test('find accessible - multi item array, no matches', function (t) { - t.plan(1) - - var candidates = ['non_existent_file', 'unreadable_file'] - var found = configure.test.findAccessibleSync('test', dir, candidates) - t.strictEqual(found, undefined) -}) - -test('find accessible - multi item array, single match', function (t) { - t.plan(1) - - var candidates = ['non_existent_file', readableFile] - var found = configure.test.findAccessibleSync('test', dir, candidates) - t.strictEqual(found, path.resolve(dir, readableFile)) -}) - -test('find accessible - multi item array, return first match', function (t) { - t.plan(1) - - var candidates = ['non_existent_file', anotherReadableFile, readableFile] - var found = configure.test.findAccessibleSync('test', dir, candidates) - t.strictEqual(found, path.resolve(dir, anotherReadableFile)) +describe('find-accessible-sync', function () { + it('find accessible - empty array', function () { + var candidates = [] + var found = configure.test.findAccessibleSync('test', dir, candidates) + assert.strictEqual(found, undefined) + }) + + it('find accessible - single item array, readable', function () { + var candidates = [readableFile] + var found = configure.test.findAccessibleSync('test', dir, candidates) + assert.strictEqual(found, path.resolve(dir, readableFile)) + }) + + it('find accessible - single item array, readable in subdir', function () { + var candidates = [readableFileInDir] + var found = configure.test.findAccessibleSync('test', dir, candidates) + assert.strictEqual(found, path.resolve(dir, readableFileInDir)) + }) + + it('find accessible - single item array, unreadable', function () { + var candidates = ['unreadable_file'] + var found = configure.test.findAccessibleSync('test', dir, candidates) + assert.strictEqual(found, undefined) + }) + + it('find accessible - multi item array, no matches', function () { + var candidates = ['non_existent_file', 'unreadable_file'] + var found = configure.test.findAccessibleSync('test', dir, candidates) + assert.strictEqual(found, undefined) + }) + + it('find accessible - multi item array, single match', function () { + var candidates = ['non_existent_file', readableFile] + var found = configure.test.findAccessibleSync('test', dir, candidates) + assert.strictEqual(found, path.resolve(dir, readableFile)) + }) + + it('find accessible - multi item array, return first match', function () { + var candidates = ['non_existent_file', anotherReadableFile, readableFile] + var found = configure.test.findAccessibleSync('test', dir, candidates) + assert.strictEqual(found, path.resolve(dir, anotherReadableFile)) + }) }) diff --git a/test/test-find-node-directory.js b/test/test-find-node-directory.js index fa6223c65d..ca299f6330 100644 --- a/test/test-find-node-directory.js +++ b/test/test-find-node-directory.js @@ -1,119 +1,115 @@ 'use strict' -const test = require('tap').test +const { describe, it } = require('mocha') +const assert = require('assert') const path = require('path') const findNodeDirectory = require('../lib/find-node-directory') const platforms = ['darwin', 'freebsd', 'linux', 'sunos', 'win32', 'aix', 'os400'] -// we should find the directory based on the directory -// the script is running in and it should match the layout -// in a build tree where npm is installed in -// .... /deps/npm -test('test find-node-directory - node install', function (t) { - t.plan(platforms.length) - for (var next = 0; next < platforms.length; next++) { - var processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } - t.equal( - findNodeDirectory('/x/deps/npm/node_modules/node-gyp/lib', processObj), - path.join('/x')) - } -}) +describe('find-node-directory', function () { + // we should find the directory based on the directory + // the script is running in and it should match the layout + // in a build tree where npm is installed in + // .... /deps/npm + it('test find-node-directory - node install', function () { + for (var next = 0; next < platforms.length; next++) { + var processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } + assert.strictEqual( + findNodeDirectory('/x/deps/npm/node_modules/node-gyp/lib', processObj), + path.join('/x')) + } + }) -// we should find the directory based on the directory -// the script is running in and it should match the layout -// in an installed tree where npm is installed in -// .... /lib/node_modules/npm or .../node_modules/npm -// depending on the patform -test('test find-node-directory - node build', function (t) { - t.plan(platforms.length) - for (var next = 0; next < platforms.length; next++) { - var processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } - if (platforms[next] === 'win32') { - t.equal( - findNodeDirectory('/y/node_modules/npm/node_modules/node-gyp/lib', - processObj), path.join('/y')) - } else { - t.equal( - findNodeDirectory('/y/lib/node_modules/npm/node_modules/node-gyp/lib', - processObj), path.join('/y')) + // we should find the directory based on the directory + // the script is running in and it should match the layout + // in an installed tree where npm is installed in + // .... /lib/node_modules/npm or .../node_modules/npm + // depending on the patform + it('test find-node-directory - node build', function () { + for (var next = 0; next < platforms.length; next++) { + var processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } + if (platforms[next] === 'win32') { + assert.strictEqual( + findNodeDirectory('/y/node_modules/npm/node_modules/node-gyp/lib', + processObj), path.join('/y')) + } else { + assert.strictEqual( + findNodeDirectory('/y/lib/node_modules/npm/node_modules/node-gyp/lib', + processObj), path.join('/y')) + } } - } -}) + }) -// we should find the directory based on the execPath -// for node and match because it was in the bin directory -test('test find-node-directory - node in bin directory', function (t) { - t.plan(platforms.length) - for (var next = 0; next < platforms.length; next++) { - var processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } - t.equal( - findNodeDirectory('/nothere/npm/node_modules/node-gyp/lib', processObj), - path.join('/x/y')) - } -}) + // we should find the directory based on the execPath + // for node and match because it was in the bin directory + it('test find-node-directory - node in bin directory', function () { + for (var next = 0; next < platforms.length; next++) { + var processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } + assert.strictEqual( + findNodeDirectory('/nothere/npm/node_modules/node-gyp/lib', processObj), + path.join('/x/y')) + } + }) -// we should find the directory based on the execPath -// for node and match because it was in the Release directory -test('test find-node-directory - node in build release dir', function (t) { - t.plan(platforms.length) - for (var next = 0; next < platforms.length; next++) { - var processObj - if (platforms[next] === 'win32') { - processObj = { execPath: '/x/y/Release/node', platform: platforms[next] } - } else { - processObj = { - execPath: '/x/y/out/Release/node', - platform: platforms[next] + // we should find the directory based on the execPath + // for node and match because it was in the Release directory + it('test find-node-directory - node in build release dir', function () { + for (var next = 0; next < platforms.length; next++) { + var processObj + if (platforms[next] === 'win32') { + processObj = { execPath: '/x/y/Release/node', platform: platforms[next] } + } else { + processObj = { + execPath: '/x/y/out/Release/node', + platform: platforms[next] + } } + + assert.strictEqual( + findNodeDirectory('/nothere/npm/node_modules/node-gyp/lib', processObj), + path.join('/x/y')) } + }) - t.equal( - findNodeDirectory('/nothere/npm/node_modules/node-gyp/lib', processObj), - path.join('/x/y')) - } -}) + // we should find the directory based on the execPath + // for node and match because it was in the Debug directory + it('test find-node-directory - node in Debug release dir', function () { + for (var next = 0; next < platforms.length; next++) { + var processObj + if (platforms[next] === 'win32') { + processObj = { execPath: '/a/b/Debug/node', platform: platforms[next] } + } else { + processObj = { execPath: '/a/b/out/Debug/node', platform: platforms[next] } + } -// we should find the directory based on the execPath -// for node and match because it was in the Debug directory -test('test find-node-directory - node in Debug release dir', function (t) { - t.plan(platforms.length) - for (var next = 0; next < platforms.length; next++) { - var processObj - if (platforms[next] === 'win32') { - processObj = { execPath: '/a/b/Debug/node', platform: platforms[next] } - } else { - processObj = { execPath: '/a/b/out/Debug/node', platform: platforms[next] } + assert.strictEqual( + findNodeDirectory('/nothere/npm/node_modules/node-gyp/lib', processObj), + path.join('/a/b')) } + }) - t.equal( - findNodeDirectory('/nothere/npm/node_modules/node-gyp/lib', processObj), - path.join('/a/b')) - } -}) - -// we should not find it as it will not match based on the execPath nor -// the directory from which the script is running -test('test find-node-directory - not found', function (t) { - t.plan(platforms.length) - for (var next = 0; next < platforms.length; next++) { - var processObj = { execPath: '/x/y/z/y', platform: next } - t.equal(findNodeDirectory('/a/b/c/d', processObj), '') - } -}) + // we should not find it as it will not match based on the execPath nor + // the directory from which the script is running + it('test find-node-directory - not found', function () { + for (var next = 0; next < platforms.length; next++) { + var processObj = { execPath: '/x/y/z/y', platform: next } + assert.strictEqual(findNodeDirectory('/a/b/c/d', processObj), '') + } + }) -// we should find the directory based on the directory -// the script is running in and it should match the layout -// in a build tree where npm is installed in -// .... /deps/npm -// same test as above but make sure additional directory entries -// don't cause an issue -test('test find-node-directory - node install', function (t) { - t.plan(platforms.length) - for (var next = 0; next < platforms.length; next++) { - var processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } - t.equal( - findNodeDirectory('/x/y/z/a/b/c/deps/npm/node_modules/node-gyp/lib', - processObj), path.join('/x/y/z/a/b/c')) - } + // we should find the directory based on the directory + // the script is running in and it should match the layout + // in a build tree where npm is installed in + // .... /deps/npm + // same test as above but make sure additional directory entries + // don't cause an issue + it('test find-node-directory - node install', function () { + for (var next = 0; next < platforms.length; next++) { + var processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } + assert.strictEqual( + findNodeDirectory('/x/y/z/a/b/c/deps/npm/node_modules/node-gyp/lib', + processObj), path.join('/x/y/z/a/b/c')) + } + }) }) diff --git a/test/test-find-python.js b/test/test-find-python.js index 67d0b2664f..592c480f24 100644 --- a/test/test-find-python.js +++ b/test/test-find-python.js @@ -2,225 +2,212 @@ delete process.env.PYTHON -const test = require('tap').test +const { describe, it } = require('mocha') +const assert = require('assert') const findPython = require('../lib/find-python') const execFile = require('child_process').execFile const PythonFinder = findPython.test.PythonFinder require('npmlog').level = 'warn' -test('find python', function (t) { - t.plan(4) - - findPython.test.findPython(null, function (err, found) { - t.strictEqual(err, null) - var proc = execFile(found, ['-V'], function (err, stdout, stderr) { - t.strictEqual(err, null) - t.ok(/Python 3/.test(stdout)) - t.strictEqual(stderr, '') +describe('find-python', function () { + it('find python', function () { + findPython.test.findPython(null, function (err, found) { + assert.strictEqual(err, null) + var proc = execFile(found, ['-V'], function (err, stdout, stderr) { + assert.strictEqual(err, null) + assert.ok(/Python 3/.test(stdout)) + assert.strictEqual(stderr, '') + }) + proc.stdout.setEncoding('utf-8') + proc.stderr.setEncoding('utf-8') }) - proc.stdout.setEncoding('utf-8') - proc.stderr.setEncoding('utf-8') }) -}) -function poison (object, property) { - function fail () { - console.error(Error(`Property ${property} should not have been accessed.`)) - process.abort() - } - var descriptor = { - configurable: false, - enumerable: false, - get: fail, - set: fail - } - Object.defineProperty(object, property, descriptor) -} - -function TestPythonFinder () { - PythonFinder.apply(this, arguments) -} -TestPythonFinder.prototype = Object.create(PythonFinder.prototype) -// Silence npmlog - remove for debugging -TestPythonFinder.prototype.log = { - silly: () => {}, - verbose: () => {}, - info: () => {}, - warn: () => {}, - error: () => {} -} -delete TestPythonFinder.prototype.env.NODE_GYP_FORCE_PYTHON - -test('find python - python', function (t) { - t.plan(6) - - var f = new TestPythonFinder('python', done) - f.execFile = function (program, args, opts, cb) { - f.execFile = function (program, args, opts, cb) { - poison(f, 'execFile') - t.strictEqual(program, '/path/python') - t.ok(/sys\.version_info/.test(args[1])) - cb(null, '3.9.1') - } - t.strictEqual(program, - process.platform === 'win32' ? '"python"' : 'python') - t.ok(/sys\.executable/.test(args[1])) - cb(null, '/path/python') + function poison (object, property) { + function fail () { + console.error(Error(`Property ${property} should not have been accessed.`)) + process.abort() + } + var descriptor = { + configurable: false, + enumerable: false, + get: fail, + set: fail + } + Object.defineProperty(object, property, descriptor) } - f.findPython() - function done (err, python) { - t.strictEqual(err, null) - t.strictEqual(python, '/path/python') + function TestPythonFinder () { + PythonFinder.apply(this, arguments) } -}) - -test('find python - python too old', function (t) { - t.plan(2) + TestPythonFinder.prototype = Object.create(PythonFinder.prototype) + // Silence npmlog - remove for debugging + TestPythonFinder.prototype.log = { + silly: () => {}, + verbose: () => {}, + info: () => {}, + warn: () => {}, + error: () => {} + } + delete TestPythonFinder.prototype.env.NODE_GYP_FORCE_PYTHON - var f = new TestPythonFinder(null, done) - f.execFile = function (program, args, opts, cb) { - if (/sys\.executable/.test(args[args.length - 1])) { + it('find python - python', function () { + var f = new TestPythonFinder('python', done) + f.execFile = function (program, args, opts, cb) { + f.execFile = function (program, args, opts, cb) { + poison(f, 'execFile') + assert.strictEqual(program, '/path/python') + assert.ok(/sys\.version_info/.test(args[1])) + cb(null, '3.9.1') + } + assert.strictEqual(program, + process.platform === 'win32' ? '"python"' : 'python') + assert.ok(/sys\.executable/.test(args[1])) cb(null, '/path/python') - } else if (/sys\.version_info/.test(args[args.length - 1])) { - cb(null, '2.3.4') - } else { - t.fail() } - } - f.findPython() - - function done (err) { - t.ok(/Could not find any Python/.test(err)) - t.ok(/not supported/i.test(f.errorLog)) - } -}) - -test('find python - no python', function (t) { - t.plan(2) + f.findPython() - var f = new TestPythonFinder(null, done) - f.execFile = function (program, args, opts, cb) { - if (/sys\.executable/.test(args[args.length - 1])) { - cb(new Error('not found')) - } else if (/sys\.version_info/.test(args[args.length - 1])) { - cb(new Error('not a Python executable')) - } else { - t.fail() + function done (err, python) { + assert.strictEqual(err, null) + assert.strictEqual(python, '/path/python') } - } - f.findPython() + }) - function done (err) { - t.ok(/Could not find any Python/.test(err)) - t.ok(/not in PATH/.test(f.errorLog)) - } -}) + it('find python - python too old', function () { + var f = new TestPythonFinder(null, done) + f.execFile = function (program, args, opts, cb) { + if (/sys\.executable/.test(args[args.length - 1])) { + cb(null, '/path/python') + } else if (/sys\.version_info/.test(args[args.length - 1])) { + cb(null, '2.3.4') + } else { + assert.fail() + } + } + f.findPython() -test('find python - no python2, no python, unix', function (t) { - t.plan(2) + function done (err) { + assert.ok(/Could not find any Python/.test(err)) + assert.ok(/not supported/i.test(f.errorLog)) + } + }) - var f = new TestPythonFinder(null, done) - f.checkPyLauncher = t.fail - f.win = false + it('find python - no python', function () { + var f = new TestPythonFinder(null, done) + f.execFile = function (program, args, opts, cb) { + if (/sys\.executable/.test(args[args.length - 1])) { + cb(new Error('not found')) + } else if (/sys\.version_info/.test(args[args.length - 1])) { + cb(new Error('not a Python executable')) + } else { + assert.fail() + } + } + f.findPython() - f.execFile = function (program, args, opts, cb) { - if (/sys\.executable/.test(args[args.length - 1])) { - cb(new Error('not found')) - } else { - t.fail() + function done (err) { + assert.ok(/Could not find any Python/.test(err)) + assert.ok(/not in PATH/.test(f.errorLog)) } - } - f.findPython() + }) - function done (err) { - t.ok(/Could not find any Python/.test(err)) - t.ok(/not in PATH/.test(f.errorLog)) - } -}) + it('find python - no python2, no python, unix', function () { + var f = new TestPythonFinder(null, done) + f.checkPyLauncher = assert.fail + f.win = false -test('find python - no python, use python launcher', function (t) { - t.plan(4) - - var f = new TestPythonFinder(null, done) - f.win = true - - f.execFile = function (program, args, opts, cb) { - if (program === 'py.exe') { - t.notEqual(args.indexOf('-3'), -1) - t.notEqual(args.indexOf('-c'), -1) - return cb(null, 'Z:\\snake.exe') - } - if (/sys\.executable/.test(args[args.length - 1])) { - cb(new Error('not found')) - } else if (f.winDefaultLocations.includes(program)) { - cb(new Error('not found')) - } else if (/sys\.version_info/.test(args[args.length - 1])) { - if (program === 'Z:\\snake.exe') { - cb(null, '3.9.0') + f.execFile = function (program, args, opts, cb) { + if (/sys\.executable/.test(args[args.length - 1])) { + cb(new Error('not found')) } else { - t.fail() + assert.fail() } - } else { - t.fail() } - } - f.findPython() + f.findPython() - function done (err, python) { - t.strictEqual(err, null) - t.strictEqual(python, 'Z:\\snake.exe') - } -}) - -test('find python - no python, no python launcher, good guess', function (t) { - t.plan(2) + function done (err) { + assert.ok(/Could not find any Python/.test(err)) + assert.ok(/not in PATH/.test(f.errorLog)) + } + }) - var f = new TestPythonFinder(null, done) - f.win = true - const expectedProgram = f.winDefaultLocations[0] + it('find python - no python, use python launcher', function () { + var f = new TestPythonFinder(null, done) + f.win = true - f.execFile = function (program, args, opts, cb) { - if (program === 'py.exe') { - return cb(new Error('not found')) + f.execFile = function (program, args, opts, cb) { + if (program === 'py.exe') { + assert.notStrictEqual(args.indexOf('-3'), -1) + assert.notStrictEqual(args.indexOf('-c'), -1) + return cb(null, 'Z:\\snake.exe') + } + if (/sys\.executable/.test(args[args.length - 1])) { + cb(new Error('not found')) + } else if (f.winDefaultLocations.includes(program)) { + cb(new Error('not found')) + } else if (/sys\.version_info/.test(args[args.length - 1])) { + if (program === 'Z:\\snake.exe') { + cb(null, '3.9.0') + } else { + assert.fail() + } + } else { + assert.fail() + } } - if (/sys\.executable/.test(args[args.length - 1])) { - cb(new Error('not found')) - } else if (program === expectedProgram && - /sys\.version_info/.test(args[args.length - 1])) { - cb(null, '3.7.3') - } else { - t.fail() + f.findPython() + + function done (err, python) { + assert.strictEqual(err, null) + assert.strictEqual(python, 'Z:\\snake.exe') } - } - f.findPython() + }) - function done (err, python) { - t.strictEqual(err, null) - t.ok(python === expectedProgram) - } -}) + it('find python - no python, no python launcher, good guess', function () { + var f = new TestPythonFinder(null, done) + f.win = true + const expectedProgram = f.winDefaultLocations[0] + + f.execFile = function (program, args, opts, cb) { + if (program === 'py.exe') { + return cb(new Error('not found')) + } + if (/sys\.executable/.test(args[args.length - 1])) { + cb(new Error('not found')) + } else if (program === expectedProgram && + /sys\.version_info/.test(args[args.length - 1])) { + cb(null, '3.7.3') + } else { + assert.fail() + } + } + f.findPython() -test('find python - no python, no python launcher, bad guess', function (t) { - t.plan(2) + function done (err, python) { + assert.strictEqual(err, null) + assert.ok(python === expectedProgram) + } + }) - var f = new TestPythonFinder(null, done) - f.win = true + it('find python - no python, no python launcher, bad guess', function () { + var f = new TestPythonFinder(null, done) + f.win = true - f.execFile = function (program, args, opts, cb) { - if (/sys\.executable/.test(args[args.length - 1])) { - cb(new Error('not found')) - } else if (/sys\.version_info/.test(args[args.length - 1])) { - cb(new Error('not a Python executable')) - } else { - t.fail() + f.execFile = function (program, args, opts, cb) { + if (/sys\.executable/.test(args[args.length - 1])) { + cb(new Error('not found')) + } else if (/sys\.version_info/.test(args[args.length - 1])) { + cb(new Error('not a Python executable')) + } else { + assert.fail() + } } - } - f.findPython() + f.findPython() - function done (err) { - t.ok(/Could not find any Python/.test(err)) - t.ok(/not in PATH/.test(f.errorLog)) - } + function done (err) { + assert.ok(/Could not find any Python/.test(err)) + assert.ok(/not in PATH/.test(f.errorLog)) + } + }) }) diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js index 6cb4f10a52..29d9a7dba5 100644 --- a/test/test-find-visualstudio.js +++ b/test/test-find-visualstudio.js @@ -1,6 +1,7 @@ 'use strict' -const test = require('tap').test +const { describe, it } = require('mocha') +const assert = require('assert') const fs = require('fs') const path = require('path') const findVisualStudio = require('../lib/find-visualstudio') @@ -35,701 +36,635 @@ TestVisualStudioFinder.prototype.log = { error: () => {} } -test('VS2013', function (t) { - t.plan(4) - - const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info, { - msBuild: 'C:\\MSBuild12\\MSBuild.exe', - path: 'C:\\VS2013', - sdk: null, - toolset: 'v120', - version: '12.0', - versionMajor: 12, - versionMinor: 0, - versionYear: 2013 +describe('find-visualstudio', function () { + it('VS2013', function () { + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info, { + msBuild: 'C:\\MSBuild12\\MSBuild.exe', + path: 'C:\\VS2013', + sdk: null, + toolset: 'v120', + version: '12.0', + versionMajor: 12, + versionMinor: 0, + versionYear: 2013 + }) }) - }) - finder.findVisualStudio2017OrNewer = (cb) => { - finder.parseData(new Error(), '', '', cb) - } - finder.regSearchKeys = (keys, value, addOpts, cb) => { - for (var i = 0; i < keys.length; ++i) { - const fullName = `${keys[i]}\\${value}` - switch (fullName) { - case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': - case 'HKLM\\Software\\Wow6432Node\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': - continue - case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\12.0': - t.pass(`expected search for registry value ${fullName}`) - return cb(null, 'C:\\VS2013\\VC\\') - case 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions\\12.0\\MSBuildToolsPath': - t.pass(`expected search for registry value ${fullName}`) - return cb(null, 'C:\\MSBuild12\\') - default: - t.fail(`unexpected search for registry value ${fullName}`) + finder.findVisualStudio2017OrNewer = (cb) => { + finder.parseData(new Error(), '', '', cb) + } + finder.regSearchKeys = (keys, value, addOpts, cb) => { + for (var i = 0; i < keys.length; ++i) { + const fullName = `${keys[i]}\\${value}` + switch (fullName) { + case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': + case 'HKLM\\Software\\Wow6432Node\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': + continue + case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\12.0': + assert.ok(true, `expected search for registry value ${fullName}`) + return cb(null, 'C:\\VS2013\\VC\\') + case 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions\\12.0\\MSBuildToolsPath': + assert.ok(true, `expected search for registry value ${fullName}`) + return cb(null, 'C:\\MSBuild12\\') + default: + assert.fail(`unexpected search for registry value ${fullName}`) + } } + return cb(new Error()) } - return cb(new Error()) - } - finder.findVisualStudio() -}) - -test('VS2013 should not be found on new node versions', function (t) { - t.plan(2) - - const finder = new TestVisualStudioFinder({ - major: 10, - minor: 0, - patch: 0 - }, null, (err, info) => { - t.ok(/find .* Visual Studio/i.test(err), 'expect error') - t.false(info, 'no data') + finder.findVisualStudio() }) - finder.findVisualStudio2017OrNewer = (cb) => { - const file = path.join(__dirname, 'fixtures', 'VS_2017_Unusable.txt') - const data = fs.readFileSync(file) - finder.parseData(null, data, '', cb) - } - finder.regSearchKeys = (keys, value, addOpts, cb) => { - for (var i = 0; i < keys.length; ++i) { - const fullName = `${keys[i]}\\${value}` - switch (fullName) { - case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': - case 'HKLM\\Software\\Wow6432Node\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': - continue - default: - t.fail(`unexpected search for registry value ${fullName}`) + it('VS2013 should not be found on new node versions', function () { + const finder = new TestVisualStudioFinder({ + major: 10, + minor: 0, + patch: 0 + }, null, (err, info) => { + assert.ok(/find .* Visual Studio/i.test(err), 'expect error') + assert.ok(!info, 'no data') + }) + + finder.findVisualStudio2017OrNewer = (cb) => { + const file = path.join(__dirname, 'fixtures', 'VS_2017_Unusable.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '', cb) + } + finder.regSearchKeys = (keys, value, addOpts, cb) => { + for (var i = 0; i < keys.length; ++i) { + const fullName = `${keys[i]}\\${value}` + switch (fullName) { + case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': + case 'HKLM\\Software\\Wow6432Node\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': + continue + default: + assert.fail(`unexpected search for registry value ${fullName}`) + } } + return cb(new Error()) } - return cb(new Error()) - } - finder.findVisualStudio() -}) + finder.findVisualStudio() + }) -test('VS2015', function (t) { - t.plan(4) - - const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info, { - msBuild: 'C:\\MSBuild14\\MSBuild.exe', - path: 'C:\\VS2015', - sdk: null, - toolset: 'v140', - version: '14.0', - versionMajor: 14, - versionMinor: 0, - versionYear: 2015 + it('VS2015', function () { + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info, { + msBuild: 'C:\\MSBuild14\\MSBuild.exe', + path: 'C:\\VS2015', + sdk: null, + toolset: 'v140', + version: '14.0', + versionMajor: 14, + versionMinor: 0, + versionYear: 2015 + }) }) - }) - finder.findVisualStudio2017OrNewer = (cb) => { - finder.parseData(new Error(), '', '', cb) - } - finder.regSearchKeys = (keys, value, addOpts, cb) => { - for (var i = 0; i < keys.length; ++i) { - const fullName = `${keys[i]}\\${value}` - switch (fullName) { - case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': - t.pass(`expected search for registry value ${fullName}`) - return cb(null, 'C:\\VS2015\\VC\\') - case 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions\\14.0\\MSBuildToolsPath': - t.pass(`expected search for registry value ${fullName}`) - return cb(null, 'C:\\MSBuild14\\') - default: - t.fail(`unexpected search for registry value ${fullName}`) + finder.findVisualStudio2017OrNewer = (cb) => { + finder.parseData(new Error(), '', '', cb) + } + finder.regSearchKeys = (keys, value, addOpts, cb) => { + for (var i = 0; i < keys.length; ++i) { + const fullName = `${keys[i]}\\${value}` + switch (fullName) { + case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': + assert.ok(true, `expected search for registry value ${fullName}`) + return cb(null, 'C:\\VS2015\\VC\\') + case 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions\\14.0\\MSBuildToolsPath': + assert.ok(true, `expected search for registry value ${fullName}`) + return cb(null, 'C:\\MSBuild14\\') + default: + assert.fail(`unexpected search for registry value ${fullName}`) + } } + return cb(new Error()) } - return cb(new Error()) - } - finder.findVisualStudio() -}) - -test('error from PowerShell', function (t) { - t.plan(2) - - const finder = new TestVisualStudioFinder(semverV1, null, null) - - finder.parseData(new Error(), '', '', (info) => { - t.ok(/use PowerShell/i.test(finder.errorLog[0]), 'expect error') - t.false(info, 'no data') + finder.findVisualStudio() }) -}) - -test('empty output from PowerShell', function (t) { - t.plan(2) - const finder = new TestVisualStudioFinder(semverV1, null, null) + it('error from PowerShell', function () { + const finder = new TestVisualStudioFinder(semverV1, null, null) - finder.parseData(null, '', '', (info) => { - t.ok(/use PowerShell/i.test(finder.errorLog[0]), 'expect error') - t.false(info, 'no data') + finder.parseData(new Error(), '', '', (info) => { + assert.ok(/use PowerShell/i.test(finder.errorLog[0]), 'expect error') + assert.ok(!info, 'no data') + }) }) -}) - -test('output from PowerShell not JSON', function (t) { - t.plan(2) - const finder = new TestVisualStudioFinder(semverV1, null, null) + it('empty output from PowerShell', function () { + const finder = new TestVisualStudioFinder(semverV1, null, null) - finder.parseData(null, 'AAAABBBB', '', (info) => { - t.ok(/use PowerShell/i.test(finder.errorLog[0]), 'expect error') - t.false(info, 'no data') + finder.parseData(null, '', '', (info) => { + assert.ok(/use PowerShell/i.test(finder.errorLog[0]), 'expect error') + assert.ok(!info, 'no data') + }) }) -}) - -test('wrong JSON from PowerShell', function (t) { - t.plan(2) - const finder = new TestVisualStudioFinder(semverV1, null, null) + it('output from PowerShell not JSON', function () { + const finder = new TestVisualStudioFinder(semverV1, null, null) - finder.parseData(null, '{}', '', (info) => { - t.ok(/use PowerShell/i.test(finder.errorLog[0]), 'expect error') - t.false(info, 'no data') + finder.parseData(null, 'AAAABBBB', '', (info) => { + assert.ok(/use PowerShell/i.test(finder.errorLog[0]), 'expect error') + assert.ok(!info, 'no data') + }) }) -}) -test('empty JSON from PowerShell', function (t) { - t.plan(2) + it('wrong JSON from PowerShell', function () { + const finder = new TestVisualStudioFinder(semverV1, null, null) - const finder = new TestVisualStudioFinder(semverV1, null, null) - - finder.parseData(null, '[]', '', (info) => { - t.ok(/find .* Visual Studio/i.test(finder.errorLog[0]), 'expect error') - t.false(info, 'no data') + finder.parseData(null, '{}', '', (info) => { + assert.ok(/use PowerShell/i.test(finder.errorLog[0]), 'expect error') + assert.ok(!info, 'no data') + }) }) -}) - -test('future version', function (t) { - t.plan(3) - const finder = new TestVisualStudioFinder(semverV1, null, null) + it('empty JSON from PowerShell', function () { + const finder = new TestVisualStudioFinder(semverV1, null, null) - finder.parseData(null, JSON.stringify([{ - packages: [ - 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64', - 'Microsoft.VisualStudio.Component.Windows10SDK.17763', - 'Microsoft.VisualStudio.VC.MSBuild.Base' - ], - path: 'C:\\VS', - version: '9999.9999.9999.9999' - }]), '', (info) => { - t.ok(/unknown version/i.test(finder.errorLog[0]), 'expect error') - t.ok(/find .* Visual Studio/i.test(finder.errorLog[1]), 'expect error') - t.false(info, 'no data') + finder.parseData(null, '[]', '', (info) => { + assert.ok(/find .* Visual Studio/i.test(finder.errorLog[0]), 'expect error') + assert.ok(!info, 'no data') + }) }) -}) -test('single unusable VS2017', function (t) { - t.plan(3) + it('future version', function () { + const finder = new TestVisualStudioFinder(semverV1, null, null) + + finder.parseData(null, JSON.stringify([{ + packages: [ + 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64', + 'Microsoft.VisualStudio.Component.Windows10SDK.17763', + 'Microsoft.VisualStudio.VC.MSBuild.Base' + ], + path: 'C:\\VS', + version: '9999.9999.9999.9999' + }]), '', (info) => { + assert.ok(/unknown version/i.test(finder.errorLog[0]), 'expect error') + assert.ok(/find .* Visual Studio/i.test(finder.errorLog[1]), 'expect error') + assert.ok(!info, 'no data') + }) + }) - const finder = new TestVisualStudioFinder(semverV1, null, null) + it('single unusable VS2017', function () { + const finder = new TestVisualStudioFinder(semverV1, null, null) - const file = path.join(__dirname, 'fixtures', 'VS_2017_Unusable.txt') - const data = fs.readFileSync(file) - finder.parseData(null, data, '', (info) => { - t.ok(/checking/i.test(finder.errorLog[0]), 'expect error') - t.ok(/find .* Visual Studio/i.test(finder.errorLog[2]), 'expect error') - t.false(info, 'no data') + const file = path.join(__dirname, 'fixtures', 'VS_2017_Unusable.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '', (info) => { + assert.ok(/checking/i.test(finder.errorLog[0]), 'expect error') + assert.ok(/find .* Visual Studio/i.test(finder.errorLog[2]), 'expect error') + assert.ok(!info, 'no data') + }) }) -}) -test('minimal VS2017 Build Tools', function (t) { - t.plan(2) - - const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info, { - msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\' + - 'BuildTools\\MSBuild\\15.0\\Bin\\MSBuild.exe', - path: - 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTools', - sdk: '10.0.17134.0', - toolset: 'v141', - version: '15.9.28307.665', - versionMajor: 15, - versionMinor: 9, - versionYear: 2017 + it('minimal VS2017 Build Tools', function () { + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info, { + msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\' + + 'BuildTools\\MSBuild\\15.0\\Bin\\MSBuild.exe', + path: + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTools', + sdk: '10.0.17134.0', + toolset: 'v141', + version: '15.9.28307.665', + versionMajor: 15, + versionMinor: 9, + versionYear: 2017 + }) }) - }) - poison(finder, 'regSearchKeys') - finder.findVisualStudio2017OrNewer = (cb) => { - const file = path.join(__dirname, 'fixtures', - 'VS_2017_BuildTools_minimal.txt') - const data = fs.readFileSync(file) - finder.parseData(null, data, '', cb) - } - finder.findVisualStudio() -}) + poison(finder, 'regSearchKeys') + finder.findVisualStudio2017OrNewer = (cb) => { + const file = path.join(__dirname, 'fixtures', + 'VS_2017_BuildTools_minimal.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '', cb) + } + finder.findVisualStudio() + }) -test('VS2017 Community with C++ workload', function (t) { - t.plan(2) - - const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info, { - msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\' + - 'Community\\MSBuild\\15.0\\Bin\\MSBuild.exe', - path: - 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community', - sdk: '10.0.17763.0', - toolset: 'v141', - version: '15.9.28307.665', - versionMajor: 15, - versionMinor: 9, - versionYear: 2017 + it('VS2017 Community with C++ workload', function () { + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info, { + msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\' + + 'Community\\MSBuild\\15.0\\Bin\\MSBuild.exe', + path: + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community', + sdk: '10.0.17763.0', + toolset: 'v141', + version: '15.9.28307.665', + versionMajor: 15, + versionMinor: 9, + versionYear: 2017 + }) }) - }) - poison(finder, 'regSearchKeys') - finder.findVisualStudio2017OrNewer = (cb) => { - const file = path.join(__dirname, 'fixtures', - 'VS_2017_Community_workload.txt') - const data = fs.readFileSync(file) - finder.parseData(null, data, '', cb) - } - finder.findVisualStudio() -}) + poison(finder, 'regSearchKeys') + finder.findVisualStudio2017OrNewer = (cb) => { + const file = path.join(__dirname, 'fixtures', + 'VS_2017_Community_workload.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '', cb) + } + finder.findVisualStudio() + }) -test('VS2017 Express', function (t) { - t.plan(2) - - const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info, { - msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\' + - 'WDExpress\\MSBuild\\15.0\\Bin\\MSBuild.exe', - path: - 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\WDExpress', - sdk: '10.0.17763.0', - toolset: 'v141', - version: '15.9.28307.858', - versionMajor: 15, - versionMinor: 9, - versionYear: 2017 + it('VS2017 Express', function () { + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info, { + msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\' + + 'WDExpress\\MSBuild\\15.0\\Bin\\MSBuild.exe', + path: + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\WDExpress', + sdk: '10.0.17763.0', + toolset: 'v141', + version: '15.9.28307.858', + versionMajor: 15, + versionMinor: 9, + versionYear: 2017 + }) }) - }) - poison(finder, 'regSearchKeys') - finder.findVisualStudio2017OrNewer = (cb) => { - const file = path.join(__dirname, 'fixtures', 'VS_2017_Express.txt') - const data = fs.readFileSync(file) - finder.parseData(null, data, '', cb) - } - finder.findVisualStudio() -}) + poison(finder, 'regSearchKeys') + finder.findVisualStudio2017OrNewer = (cb) => { + const file = path.join(__dirname, 'fixtures', 'VS_2017_Express.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '', cb) + } + finder.findVisualStudio() + }) -test('VS2019 Preview with C++ workload', function (t) { - t.plan(2) - - const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info, { - msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' + - 'Preview\\MSBuild\\Current\\Bin\\MSBuild.exe', - path: - 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Preview', - sdk: '10.0.17763.0', - toolset: 'v142', - version: '16.0.28608.199', - versionMajor: 16, - versionMinor: 0, - versionYear: 2019 + it('VS2019 Preview with C++ workload', function () { + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info, { + msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' + + 'Preview\\MSBuild\\Current\\Bin\\MSBuild.exe', + path: + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Preview', + sdk: '10.0.17763.0', + toolset: 'v142', + version: '16.0.28608.199', + versionMajor: 16, + versionMinor: 0, + versionYear: 2019 + }) }) - }) - poison(finder, 'regSearchKeys') - finder.findVisualStudio2017OrNewer = (cb) => { - const file = path.join(__dirname, 'fixtures', - 'VS_2019_Preview.txt') - const data = fs.readFileSync(file) - finder.parseData(null, data, '', cb) - } - finder.findVisualStudio() -}) + poison(finder, 'regSearchKeys') + finder.findVisualStudio2017OrNewer = (cb) => { + const file = path.join(__dirname, 'fixtures', + 'VS_2019_Preview.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '', cb) + } + finder.findVisualStudio() + }) -test('minimal VS2019 Build Tools', function (t) { - t.plan(2) - - const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info, { - msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' + - 'BuildTools\\MSBuild\\Current\\Bin\\MSBuild.exe', - path: - 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools', - sdk: '10.0.17134.0', - toolset: 'v142', - version: '16.1.28922.388', - versionMajor: 16, - versionMinor: 1, - versionYear: 2019 + it('minimal VS2019 Build Tools', function () { + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info, { + msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' + + 'BuildTools\\MSBuild\\Current\\Bin\\MSBuild.exe', + path: + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools', + sdk: '10.0.17134.0', + toolset: 'v142', + version: '16.1.28922.388', + versionMajor: 16, + versionMinor: 1, + versionYear: 2019 + }) }) - }) - poison(finder, 'regSearchKeys') - finder.findVisualStudio2017OrNewer = (cb) => { - const file = path.join(__dirname, 'fixtures', - 'VS_2019_BuildTools_minimal.txt') - const data = fs.readFileSync(file) - finder.parseData(null, data, '', cb) - } - finder.findVisualStudio() -}) + poison(finder, 'regSearchKeys') + finder.findVisualStudio2017OrNewer = (cb) => { + const file = path.join(__dirname, 'fixtures', + 'VS_2019_BuildTools_minimal.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '', cb) + } + finder.findVisualStudio() + }) -test('VS2019 Community with C++ workload', function (t) { - t.plan(2) - - const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info, { - msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' + - 'Community\\MSBuild\\Current\\Bin\\MSBuild.exe', - path: - 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community', - sdk: '10.0.17763.0', - toolset: 'v142', - version: '16.1.28922.388', - versionMajor: 16, - versionMinor: 1, - versionYear: 2019 + it('VS2019 Community with C++ workload', function () { + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info, { + msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' + + 'Community\\MSBuild\\Current\\Bin\\MSBuild.exe', + path: + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community', + sdk: '10.0.17763.0', + toolset: 'v142', + version: '16.1.28922.388', + versionMajor: 16, + versionMinor: 1, + versionYear: 2019 + }) }) - }) - poison(finder, 'regSearchKeys') - finder.findVisualStudio2017OrNewer = (cb) => { - const file = path.join(__dirname, 'fixtures', - 'VS_2019_Community_workload.txt') - const data = fs.readFileSync(file) - finder.parseData(null, data, '', cb) - } - finder.findVisualStudio() -}) + poison(finder, 'regSearchKeys') + finder.findVisualStudio2017OrNewer = (cb) => { + const file = path.join(__dirname, 'fixtures', + 'VS_2019_Community_workload.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '', cb) + } + finder.findVisualStudio() + }) -test('VS2022 Preview with C++ workload', function (t) { - t.plan(2) - - const msBuildPath = process.arch === 'arm64' - ? 'C:\\Program Files\\Microsoft Visual Studio\\2022\\' + - 'Community\\MSBuild\\Current\\Bin\\arm64\\MSBuild.exe' - : 'C:\\Program Files\\Microsoft Visual Studio\\2022\\' + - 'Community\\MSBuild\\Current\\Bin\\MSBuild.exe' - - const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info, { - msBuild: msBuildPath, - path: - 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community', - sdk: '10.0.22621.0', - toolset: 'v143', - version: '17.4.33213.308', - versionMajor: 17, - versionMinor: 4, - versionYear: 2022 + it('VS2022 Preview with C++ workload', function () { + const msBuildPath = process.arch === 'arm64' + ? 'C:\\Program Files\\Microsoft Visual Studio\\2022\\' + + 'Community\\MSBuild\\Current\\Bin\\arm64\\MSBuild.exe' + : 'C:\\Program Files\\Microsoft Visual Studio\\2022\\' + + 'Community\\MSBuild\\Current\\Bin\\MSBuild.exe' + + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info, { + msBuild: msBuildPath, + path: + 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community', + sdk: '10.0.22621.0', + toolset: 'v143', + version: '17.4.33213.308', + versionMajor: 17, + versionMinor: 4, + versionYear: 2022 + }) }) - }) - poison(finder, 'regSearchKeys') - finder.msBuildPathExists = (path) => { - return true - } - finder.findVisualStudio2017OrNewer = (cb) => { - const file = path.join(__dirname, 'fixtures', - 'VS_2022_Community_workload.txt') - const data = fs.readFileSync(file) - finder.parseData(null, data, '', cb) - } - finder.findVisualStudio() -}) + poison(finder, 'regSearchKeys') + finder.msBuildPathExists = (path) => { + return true + } + finder.findVisualStudio2017OrNewer = (cb) => { + const file = path.join(__dirname, 'fixtures', + 'VS_2022_Community_workload.txt') + const data = fs.readFileSync(file) + finder.parseData(null, data, '', cb) + } + finder.findVisualStudio() + }) -function allVsVersions (t, finder) { - finder.findVisualStudio2017OrNewer = (cb) => { - const data0 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', - 'VS_2017_Unusable.txt'))) - const data1 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', - 'VS_2017_BuildTools_minimal.txt'))) - const data2 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', - 'VS_2017_Community_workload.txt'))) - const data3 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', - 'VS_2017_Express.txt'))) - const data4 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', - 'VS_2019_Preview.txt'))) - const data5 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', - 'VS_2019_BuildTools_minimal.txt'))) - const data6 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', - 'VS_2019_Community_workload.txt'))) - const data7 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', - 'VS_2022_Community_workload.txt'))) - const data = JSON.stringify(data0.concat(data1, data2, data3, data4, - data5, data6, data7)) - finder.parseData(null, data, '', cb) - } - finder.regSearchKeys = (keys, value, addOpts, cb) => { - for (var i = 0; i < keys.length; ++i) { - const fullName = `${keys[i]}\\${value}` - switch (fullName) { - case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': - case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\12.0': - continue - case 'HKLM\\Software\\Wow6432Node\\Microsoft\\VisualStudio\\SxS\\VC7\\12.0': - return cb(null, 'C:\\VS2013\\VC\\') - case 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions\\12.0\\MSBuildToolsPath': - return cb(null, 'C:\\MSBuild12\\') - case 'HKLM\\Software\\Wow6432Node\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': - return cb(null, 'C:\\VS2015\\VC\\') - case 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions\\14.0\\MSBuildToolsPath': - return cb(null, 'C:\\MSBuild14\\') - default: - t.fail(`unexpected search for registry value ${fullName}`) + function allVsVersions (finder) { + finder.findVisualStudio2017OrNewer = (cb) => { + const data0 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', + 'VS_2017_Unusable.txt'))) + const data1 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', + 'VS_2017_BuildTools_minimal.txt'))) + const data2 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', + 'VS_2017_Community_workload.txt'))) + const data3 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', + 'VS_2017_Express.txt'))) + const data4 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', + 'VS_2019_Preview.txt'))) + const data5 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', + 'VS_2019_BuildTools_minimal.txt'))) + const data6 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', + 'VS_2019_Community_workload.txt'))) + const data7 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', + 'VS_2022_Community_workload.txt'))) + const data = JSON.stringify(data0.concat(data1, data2, data3, data4, + data5, data6, data7)) + finder.parseData(null, data, '', cb) + } + finder.regSearchKeys = (keys, value, addOpts, cb) => { + for (var i = 0; i < keys.length; ++i) { + const fullName = `${keys[i]}\\${value}` + switch (fullName) { + case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': + case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\12.0': + continue + case 'HKLM\\Software\\Wow6432Node\\Microsoft\\VisualStudio\\SxS\\VC7\\12.0': + return cb(null, 'C:\\VS2013\\VC\\') + case 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions\\12.0\\MSBuildToolsPath': + return cb(null, 'C:\\MSBuild12\\') + case 'HKLM\\Software\\Wow6432Node\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': + return cb(null, 'C:\\VS2015\\VC\\') + case 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions\\14.0\\MSBuildToolsPath': + return cb(null, 'C:\\MSBuild14\\') + default: + assert.fail(`unexpected search for registry value ${fullName}`) + } } + return cb(new Error()) } - return cb(new Error()) } -} -test('fail when looking for invalid path', function (t) { - t.plan(2) + it('fail when looking for invalid path', function () { + const finder = new TestVisualStudioFinder(semverV1, 'AABB', (err, info) => { + assert.ok(/find .* Visual Studio/i.test(err), 'expect error') + assert.ok(!info, 'no data') + }) - const finder = new TestVisualStudioFinder(semverV1, 'AABB', (err, info) => { - t.ok(/find .* Visual Studio/i.test(err), 'expect error') - t.false(info, 'no data') + allVsVersions(finder) + finder.findVisualStudio() }) - allVsVersions(t, finder) - finder.findVisualStudio() -}) - -test('look for VS2013 by version number', function (t) { - t.plan(2) + it('look for VS2013 by version number', function () { + const finder = new TestVisualStudioFinder(semverV1, '2013', (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info.versionYear, 2013) + }) - const finder = new TestVisualStudioFinder(semverV1, '2013', (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info.versionYear, 2013) + allVsVersions(finder) + finder.findVisualStudio() }) - allVsVersions(t, finder) - finder.findVisualStudio() -}) + it('look for VS2013 by installation path', function () { + const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2013', + (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info.path, 'C:\\VS2013') + }) -test('look for VS2013 by installation path', function (t) { - t.plan(2) + allVsVersions(finder) + finder.findVisualStudio() + }) - const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2013', - (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info.path, 'C:\\VS2013') + it('look for VS2015 by version number', function () { + const finder = new TestVisualStudioFinder(semverV1, '2015', (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info.versionYear, 2015) }) - allVsVersions(t, finder) - finder.findVisualStudio() -}) - -test('look for VS2015 by version number', function (t) { - t.plan(2) - - const finder = new TestVisualStudioFinder(semverV1, '2015', (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info.versionYear, 2015) + allVsVersions(finder) + finder.findVisualStudio() }) - allVsVersions(t, finder) - finder.findVisualStudio() -}) + it('look for VS2015 by installation path', function () { + const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2015', + (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info.path, 'C:\\VS2015') + }) -test('look for VS2015 by installation path', function (t) { - t.plan(2) + allVsVersions(finder) + finder.findVisualStudio() + }) - const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2015', - (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info.path, 'C:\\VS2015') + it('look for VS2017 by version number', function () { + const finder = new TestVisualStudioFinder(semverV1, '2017', (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info.versionYear, 2017) }) - allVsVersions(t, finder) - finder.findVisualStudio() -}) - -test('look for VS2017 by version number', function (t) { - t.plan(2) - - const finder = new TestVisualStudioFinder(semverV1, '2017', (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info.versionYear, 2017) + allVsVersions(finder) + finder.findVisualStudio() }) - allVsVersions(t, finder) - finder.findVisualStudio() -}) - -test('look for VS2017 by installation path', function (t) { - t.plan(2) + it('look for VS2017 by installation path', function () { + const finder = new TestVisualStudioFinder(semverV1, + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community', + (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info.path, + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community') + }) + + allVsVersions(finder) + finder.findVisualStudio() + }) - const finder = new TestVisualStudioFinder(semverV1, - 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community', - (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info.path, - 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community') + it('look for VS2019 by version number', function () { + const finder = new TestVisualStudioFinder(semverV1, '2019', (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info.versionYear, 2019) }) - allVsVersions(t, finder) - finder.findVisualStudio() -}) - -test('look for VS2019 by version number', function (t) { - t.plan(2) - - const finder = new TestVisualStudioFinder(semverV1, '2019', (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info.versionYear, 2019) + allVsVersions(finder) + finder.findVisualStudio() }) - allVsVersions(t, finder) - finder.findVisualStudio() -}) - -test('look for VS2019 by installation path', function (t) { - t.plan(2) + it('look for VS2019 by installation path', function () { + const finder = new TestVisualStudioFinder(semverV1, + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools', + (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info.path, + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools') + }) + + allVsVersions(finder) + finder.findVisualStudio() + }) - const finder = new TestVisualStudioFinder(semverV1, - 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools', - (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info.path, - 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools') + it('look for VS2022 by version number', function () { + const finder = new TestVisualStudioFinder(semverV1, '2022', (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info.versionYear, 2022) }) - allVsVersions(t, finder) - finder.findVisualStudio() -}) - -test('look for VS2022 by version number', function (t) { - t.plan(2) + finder.msBuildPathExists = (path) => { + return true + } - const finder = new TestVisualStudioFinder(semverV1, '2022', (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info.versionYear, 2022) + allVsVersions(finder) + finder.findVisualStudio() }) - finder.msBuildPathExists = (path) => { - return true - } - - allVsVersions(t, finder) - finder.findVisualStudio() -}) - -test('msvs_version match should be case insensitive', function (t) { - t.plan(2) + it('msvs_version match should be case insensitive', function () { + const finder = new TestVisualStudioFinder(semverV1, + 'c:\\program files (x86)\\microsoft visual studio\\2019\\BUILDTOOLS', + (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info.path, + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools') + }) + + allVsVersions(finder) + finder.findVisualStudio() + }) - const finder = new TestVisualStudioFinder(semverV1, - 'c:\\program files (x86)\\microsoft visual studio\\2019\\BUILDTOOLS', - (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info.path, - 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools') + it('latest version should be found by default', function () { + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info.versionYear, 2022) }) - allVsVersions(t, finder) - finder.findVisualStudio() -}) - -test('latest version should be found by default', function (t) { - t.plan(2) + finder.msBuildPathExists = (path) => { + return true + } - const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info.versionYear, 2022) + allVsVersions(finder) + finder.findVisualStudio() }) - finder.msBuildPathExists = (path) => { - return true - } - - allVsVersions(t, finder) - finder.findVisualStudio() -}) + it('run on a usable VS Command Prompt', function () { + process.env.VCINSTALLDIR = 'C:\\VS2015\\VC' + // VSINSTALLDIR is not defined on Visual C++ Build Tools 2015 + delete process.env.VSINSTALLDIR -test('run on a usable VS Command Prompt', function (t) { - t.plan(2) - - process.env.VCINSTALLDIR = 'C:\\VS2015\\VC' - // VSINSTALLDIR is not defined on Visual C++ Build Tools 2015 - delete process.env.VSINSTALLDIR + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info.path, 'C:\\VS2015') + }) - const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info.path, 'C:\\VS2015') + allVsVersions(finder) + finder.findVisualStudio() }) - allVsVersions(t, finder) - finder.findVisualStudio() -}) + it('VCINSTALLDIR match should be case insensitive', function () { + process.env.VCINSTALLDIR = + 'c:\\program files (x86)\\microsoft visual studio\\2019\\BUILDTOOLS\\VC' -test('VCINSTALLDIR match should be case insensitive', function (t) { - t.plan(2) - - process.env.VCINSTALLDIR = - 'c:\\program files (x86)\\microsoft visual studio\\2019\\BUILDTOOLS\\VC' + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info.path, + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools') + }) - const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info.path, - 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools') + allVsVersions(finder) + finder.findVisualStudio() }) - allVsVersions(t, finder) - finder.findVisualStudio() -}) - -test('run on a unusable VS Command Prompt', function (t) { - t.plan(2) + it('run on a unusable VS Command Prompt', function () { + process.env.VCINSTALLDIR = + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildToolsUnusable\\VC' - process.env.VCINSTALLDIR = - 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildToolsUnusable\\VC' + const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { + assert.ok(/find .* Visual Studio/i.test(err), 'expect error') + assert.ok(!info, 'no data') + }) - const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => { - t.ok(/find .* Visual Studio/i.test(err), 'expect error') - t.false(info, 'no data') + allVsVersions(finder) + finder.findVisualStudio() }) - allVsVersions(t, finder) - finder.findVisualStudio() -}) - -test('run on a VS Command Prompt with matching msvs_version', function (t) { - t.plan(2) + it('run on a VS Command Prompt with matching msvs_version', function () { + process.env.VCINSTALLDIR = 'C:\\VS2015\\VC' - process.env.VCINSTALLDIR = 'C:\\VS2015\\VC' + const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2015', + (err, info) => { + assert.strictEqual(err, null) + assert.deepStrictEqual(info.path, 'C:\\VS2015') + }) - const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2015', - (err, info) => { - t.strictEqual(err, null) - t.deepEqual(info.path, 'C:\\VS2015') - }) - - allVsVersions(t, finder) - finder.findVisualStudio() -}) - -test('run on a VS Command Prompt with mismatched msvs_version', function (t) { - t.plan(2) + allVsVersions(finder) + finder.findVisualStudio() + }) - process.env.VCINSTALLDIR = - 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC' + it('run on a VS Command Prompt with mismatched msvs_version', function () { + process.env.VCINSTALLDIR = + 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC' - const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2015', - (err, info) => { - t.ok(/find .* Visual Studio/i.test(err), 'expect error') - t.false(info, 'no data') - }) + const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2015', + (err, info) => { + assert.ok(/find .* Visual Studio/i.test(err), 'expect error') + assert.ok(!info, 'no data') + }) - allVsVersions(t, finder) - finder.findVisualStudio() + allVsVersions(finder) + finder.findVisualStudio() + }) }) diff --git a/test/test-install.js b/test/test-install.js index e1612cfadf..235acf5231 100644 --- a/test/test-install.js +++ b/test/test-install.js @@ -1,6 +1,7 @@ 'use strict' -const { test } = require('tap') +const { describe, it, after } = require('mocha') +const assert = require('assert') const path = require('path') const os = require('os') const util = require('util') @@ -14,117 +15,123 @@ const streamPipeline = util.promisify(stream.pipeline) log.level = 'error' // we expect a warning -test('EACCES retry once', async (t) => { - t.plan(3) - - const fs = { - promises: { - stat (_) { - const err = new Error() - err.code = 'EACCES' - t.ok(true) - throw err +describe('install', function () { + it('EACCES retry once', async () => { + const fs = { + promises: { + stat (_) { + const err = new Error() + err.code = 'EACCES' + assert.ok(true) + throw err + } } } - } - const Gyp = { - devDir: __dirname, - opts: { - ensure: true - }, - commands: { - install (argv, cb) { - install(fs, Gyp, argv).then(cb, cb) + const Gyp = { + devDir: __dirname, + opts: { + ensure: true }, - remove (_, cb) { - cb() + commands: { + install (argv, cb) { + install(fs, Gyp, argv).then(cb, cb) + }, + remove (_, cb) { + cb() + } } } - } - try { - await install(fs, Gyp, []) - } catch (err) { - t.ok(true) - if (/"pre" versions of node cannot be installed/.test(err.message)) { - t.ok(true) + try { + await install(fs, Gyp, []) + } catch (err) { + assert.ok(true) + if (/"pre" versions of node cannot be installed/.test(err.message)) { + assert.ok(true) + } } - } -}) + }) -// only run these tests if we are running a version of Node with predictable version path behavior -const skipParallelInstallTests = process.env.FAST_TEST || - process.release.name !== 'node' || - semver.prerelease(process.version) !== null || - semver.satisfies(process.version, '<10') + // only run these tests if we are running a version of Node with predictable version path behavior + const skipParallelInstallTests = process.env.FAST_TEST || + process.release.name !== 'node' || + semver.prerelease(process.version) !== null || + semver.satisfies(process.version, '<10') + + async function parallelInstallsTest (test, fs, devDir, prog) { + if (skipParallelInstallTests) { + return test.skip('Skipping parallel installs test due to test environment configuration') + } -async function parallelInstallsTest (t, fs, devDir, prog) { - if (skipParallelInstallTests) { - return t.skip('Skipping parallel installs test due to test environment configuration') + after(async () => { + await util.promisify(rimraf)(devDir) + }) + + const expectedDir = path.join(devDir, process.version.replace(/^v/, '')) + await util.promisify(rimraf)(expectedDir) + + await Promise.all([ + install(fs, prog, []), + install(fs, prog, []), + install(fs, prog, []), + install(fs, prog, []), + install(fs, prog, []), + install(fs, prog, []), + install(fs, prog, []), + install(fs, prog, []), + install(fs, prog, []), + install(fs, prog, []) + ]) } - t.tearDown(async () => { - await util.promisify(rimraf)(devDir) + it('parallel installs (ensure=true)', async function () { + this.timeout(600000) + + const fs = require('graceful-fs') + const devDir = await util.promisify(fs.mkdtemp)(path.join(os.tmpdir(), 'node-gyp-test-')) + + const prog = gyp() + prog.parseArgv([]) + prog.devDir = devDir + prog.opts.ensure = true + log.level = 'warn' + + await parallelInstallsTest(this, fs, devDir, prog) }) - const expectedDir = path.join(devDir, process.version.replace(/^v/, '')) - await util.promisify(rimraf)(expectedDir) - - await Promise.all([ - install(fs, prog, []), - install(fs, prog, []), - install(fs, prog, []), - install(fs, prog, []), - install(fs, prog, []), - install(fs, prog, []), - install(fs, prog, []), - install(fs, prog, []), - install(fs, prog, []), - install(fs, prog, []) - ]) -} - -test('parallel installs (ensure=true)', async (t) => { - const fs = require('graceful-fs') - const devDir = await util.promisify(fs.mkdtemp)(path.join(os.tmpdir(), 'node-gyp-test-')) - - const prog = gyp() - prog.parseArgv([]) - prog.devDir = devDir - prog.opts.ensure = true - log.level = 'warn' - - await parallelInstallsTest(t, fs, devDir, prog) -}) + it('parallel installs (ensure=false)', async function () { + this.timeout(600000) -test('parallel installs (ensure=false)', async (t) => { - const fs = require('graceful-fs') - const devDir = await util.promisify(fs.mkdtemp)(path.join(os.tmpdir(), 'node-gyp-test-')) + const fs = require('graceful-fs') + const devDir = await util.promisify(fs.mkdtemp)(path.join(os.tmpdir(), 'node-gyp-test-')) - const prog = gyp() - prog.parseArgv([]) - prog.devDir = devDir - prog.opts.ensure = false - log.level = 'warn' + const prog = gyp() + prog.parseArgv([]) + prog.devDir = devDir + prog.opts.ensure = false + log.level = 'warn' - await parallelInstallsTest(t, fs, devDir, prog) -}) + await parallelInstallsTest(this, fs, devDir, prog) + }) -test('parallel installs (tarball)', async (t) => { - const fs = require('graceful-fs') - const devDir = await util.promisify(fs.mkdtemp)(path.join(os.tmpdir(), 'node-gyp-test-')) + it('parallel installs (tarball)', async function () { + this.timeout(600000) - const prog = gyp() - prog.parseArgv([]) - prog.devDir = devDir - prog.opts.tarball = path.join(devDir, 'node-headers.tar.gz') - log.level = 'warn' + const fs = require('graceful-fs') + const devDir = await util.promisify(fs.mkdtemp)(path.join(os.tmpdir(), 'node-gyp-test-')) - await streamPipeline( - (await download(prog, 'https://nodejs.org/dist/v16.0.0/node-v16.0.0.tar.gz')).body, - fs.createWriteStream(prog.opts.tarball) - ) + const prog = gyp() + prog.parseArgv([]) + prog.devDir = devDir + prog.opts.tarball = path.join(devDir, 'node-headers.tar.gz') + log.level = 'warn' - await parallelInstallsTest(t, fs, devDir, prog) + await streamPipeline( + (await download(prog, `https://nodejs.org/dist/${process.version}/node-${process.version}.tar.gz`)).body, + fs.createWriteStream(prog.opts.tarball) + ) + + await parallelInstallsTest(this, fs, devDir, prog) + }) }) diff --git a/test/test-options.js b/test/test-options.js index 8a634f0e09..24e79c80a1 100644 --- a/test/test-options.js +++ b/test/test-options.js @@ -1,42 +1,41 @@ 'use strict' -const test = require('tap').test +const { describe, it } = require('mocha') +const assert = require('assert') const gyp = require('../lib/node-gyp') -test('options in environment', (t) => { - t.plan(1) +describe('options', function () { + it('options in environment', () => { + // `npm test` dumps a ton of npm_config_* variables in the environment. + Object.keys(process.env) + .filter((key) => /^npm_config_/.test(key)) + .forEach((key) => { delete process.env[key] }) - // `npm test` dumps a ton of npm_config_* variables in the environment. - Object.keys(process.env) - .filter((key) => /^npm_config_/.test(key)) - .forEach((key) => { delete process.env[key] }) + // in some platforms, certain keys are stubborn and cannot be removed + const keys = Object.keys(process.env) + .filter((key) => /^npm_config_/.test(key)) + .map((key) => key.substring('npm_config_'.length)) + .concat('argv', 'x') - // in some platforms, certain keys are stubborn and cannot be removed - const keys = Object.keys(process.env) - .filter((key) => /^npm_config_/.test(key)) - .map((key) => key.substring('npm_config_'.length)) - .concat('argv', 'x') + // Zero-length keys should get filtered out. + process.env.npm_config_ = '42' + // Other keys should get added. + process.env.npm_config_x = '42' + // Except loglevel. + process.env.npm_config_loglevel = 'debug' - // Zero-length keys should get filtered out. - process.env.npm_config_ = '42' - // Other keys should get added. - process.env.npm_config_x = '42' - // Except loglevel. - process.env.npm_config_loglevel = 'debug' + const g = gyp() + g.parseArgv(['rebuild']) // Also sets opts.argv. - const g = gyp() - g.parseArgv(['rebuild']) // Also sets opts.argv. + assert.deepStrictEqual(Object.keys(g.opts).sort(), keys.sort()) + }) - t.deepEqual(Object.keys(g.opts).sort(), keys.sort()) -}) - -test('options with spaces in environment', (t) => { - t.plan(1) - - process.env.npm_config_force_process_config = 'true' + it('options with spaces in environment', () => { + process.env.npm_config_force_process_config = 'true' - const g = gyp() - g.parseArgv(['rebuild']) // Also sets opts.argv. + const g = gyp() + g.parseArgv(['rebuild']) // Also sets opts.argv. - t.equal(g.opts['force-process-config'], 'true') + assert.strictEqual(g.opts['force-process-config'], 'true') + }) }) diff --git a/test/test-process-release.js b/test/test-process-release.js index c3ee0703c5..0f40666473 100644 --- a/test/test-process-release.js +++ b/test/test-process-release.js @@ -1,434 +1,401 @@ 'use strict' -const test = require('tap').test +const { describe, it } = require('mocha') +const assert = require('assert') const processRelease = require('../lib/process-release') -test('test process release - process.version = 0.8.20', function (t) { - t.plan(2) - - var release = processRelease([], { opts: {} }, 'v0.8.20', null) - - t.equal(release.semver.version, '0.8.20') - delete release.semver - - t.deepEqual(release, { - version: '0.8.20', - name: 'node', - baseUrl: 'https://nodejs.org/dist/v0.8.20/', - tarballUrl: 'https://nodejs.org/dist/v0.8.20/node-v0.8.20.tar.gz', - shasumsUrl: 'https://nodejs.org/dist/v0.8.20/SHASUMS256.txt', - versionDir: '0.8.20', - ia32: { libUrl: 'https://nodejs.org/dist/v0.8.20/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.8.20/x64/node.lib', libPath: 'x64/node.lib' }, - arm64: { libUrl: 'https://nodejs.org/dist/v0.8.20/arm64/node.lib', libPath: 'arm64/node.lib' } +describe('process-release', function () { + it('test process release - process.version = 0.8.20', function () { + var release = processRelease([], { opts: {} }, 'v0.8.20', null) + + assert.strictEqual(release.semver.version, '0.8.20') + delete release.semver + + assert.deepStrictEqual(release, { + version: '0.8.20', + name: 'node', + baseUrl: 'https://nodejs.org/dist/v0.8.20/', + tarballUrl: 'https://nodejs.org/dist/v0.8.20/node-v0.8.20.tar.gz', + shasumsUrl: 'https://nodejs.org/dist/v0.8.20/SHASUMS256.txt', + versionDir: '0.8.20', + ia32: { libUrl: 'https://nodejs.org/dist/v0.8.20/node.lib', libPath: 'node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.8.20/x64/node.lib', libPath: 'x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/dist/v0.8.20/arm64/node.lib', libPath: 'arm64/node.lib' } + }) }) -}) -test('test process release - process.version = 0.10.21', function (t) { - t.plan(2) - - var release = processRelease([], { opts: {} }, 'v0.10.21', null) - - t.equal(release.semver.version, '0.10.21') - delete release.semver - - t.deepEqual(release, { - version: '0.10.21', - name: 'node', - baseUrl: 'https://nodejs.org/dist/v0.10.21/', - tarballUrl: 'https://nodejs.org/dist/v0.10.21/node-v0.10.21.tar.gz', - shasumsUrl: 'https://nodejs.org/dist/v0.10.21/SHASUMS256.txt', - versionDir: '0.10.21', - ia32: { libUrl: 'https://nodejs.org/dist/v0.10.21/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.10.21/x64/node.lib', libPath: 'x64/node.lib' }, - arm64: { libUrl: 'https://nodejs.org/dist/v0.10.21/arm64/node.lib', libPath: 'arm64/node.lib' } + it('test process release - process.version = 0.10.21', function () { + var release = processRelease([], { opts: {} }, 'v0.10.21', null) + + assert.strictEqual(release.semver.version, '0.10.21') + delete release.semver + + assert.deepStrictEqual(release, { + version: '0.10.21', + name: 'node', + baseUrl: 'https://nodejs.org/dist/v0.10.21/', + tarballUrl: 'https://nodejs.org/dist/v0.10.21/node-v0.10.21.tar.gz', + shasumsUrl: 'https://nodejs.org/dist/v0.10.21/SHASUMS256.txt', + versionDir: '0.10.21', + ia32: { libUrl: 'https://nodejs.org/dist/v0.10.21/node.lib', libPath: 'node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.10.21/x64/node.lib', libPath: 'x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/dist/v0.10.21/arm64/node.lib', libPath: 'arm64/node.lib' } + }) }) -}) -// prior to -headers.tar.gz -test('test process release - process.version = 0.12.9', function (t) { - t.plan(2) - - var release = processRelease([], { opts: {} }, 'v0.12.9', null) - - t.equal(release.semver.version, '0.12.9') - delete release.semver - - t.deepEqual(release, { - version: '0.12.9', - name: 'node', - baseUrl: 'https://nodejs.org/dist/v0.12.9/', - tarballUrl: 'https://nodejs.org/dist/v0.12.9/node-v0.12.9.tar.gz', - shasumsUrl: 'https://nodejs.org/dist/v0.12.9/SHASUMS256.txt', - versionDir: '0.12.9', - ia32: { libUrl: 'https://nodejs.org/dist/v0.12.9/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.12.9/x64/node.lib', libPath: 'x64/node.lib' }, - arm64: { libUrl: 'https://nodejs.org/dist/v0.12.9/arm64/node.lib', libPath: 'arm64/node.lib' } + // prior to -headers.tar.gz + it('test process release - process.version = 0.12.9', function () { + var release = processRelease([], { opts: {} }, 'v0.12.9', null) + + assert.strictEqual(release.semver.version, '0.12.9') + delete release.semver + + assert.deepStrictEqual(release, { + version: '0.12.9', + name: 'node', + baseUrl: 'https://nodejs.org/dist/v0.12.9/', + tarballUrl: 'https://nodejs.org/dist/v0.12.9/node-v0.12.9.tar.gz', + shasumsUrl: 'https://nodejs.org/dist/v0.12.9/SHASUMS256.txt', + versionDir: '0.12.9', + ia32: { libUrl: 'https://nodejs.org/dist/v0.12.9/node.lib', libPath: 'node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.12.9/x64/node.lib', libPath: 'x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/dist/v0.12.9/arm64/node.lib', libPath: 'arm64/node.lib' } + }) }) -}) -// prior to -headers.tar.gz -test('test process release - process.version = 0.10.41', function (t) { - t.plan(2) - - var release = processRelease([], { opts: {} }, 'v0.10.41', null) - - t.equal(release.semver.version, '0.10.41') - delete release.semver - - t.deepEqual(release, { - version: '0.10.41', - name: 'node', - baseUrl: 'https://nodejs.org/dist/v0.10.41/', - tarballUrl: 'https://nodejs.org/dist/v0.10.41/node-v0.10.41.tar.gz', - shasumsUrl: 'https://nodejs.org/dist/v0.10.41/SHASUMS256.txt', - versionDir: '0.10.41', - ia32: { libUrl: 'https://nodejs.org/dist/v0.10.41/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.10.41/x64/node.lib', libPath: 'x64/node.lib' }, - arm64: { libUrl: 'https://nodejs.org/dist/v0.10.41/arm64/node.lib', libPath: 'arm64/node.lib' } + // prior to -headers.tar.gz + it('test process release - process.version = 0.10.41', function () { + var release = processRelease([], { opts: {} }, 'v0.10.41', null) + + assert.strictEqual(release.semver.version, '0.10.41') + delete release.semver + + assert.deepStrictEqual(release, { + version: '0.10.41', + name: 'node', + baseUrl: 'https://nodejs.org/dist/v0.10.41/', + tarballUrl: 'https://nodejs.org/dist/v0.10.41/node-v0.10.41.tar.gz', + shasumsUrl: 'https://nodejs.org/dist/v0.10.41/SHASUMS256.txt', + versionDir: '0.10.41', + ia32: { libUrl: 'https://nodejs.org/dist/v0.10.41/node.lib', libPath: 'node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.10.41/x64/node.lib', libPath: 'x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/dist/v0.10.41/arm64/node.lib', libPath: 'arm64/node.lib' } + }) }) -}) -// has -headers.tar.gz -test('test process release - process.release ~ node@0.10.42', function (t) { - t.plan(2) - - var release = processRelease([], { opts: {} }, 'v0.10.42', null) - - t.equal(release.semver.version, '0.10.42') - delete release.semver - - t.deepEqual(release, { - version: '0.10.42', - name: 'node', - baseUrl: 'https://nodejs.org/dist/v0.10.42/', - tarballUrl: 'https://nodejs.org/dist/v0.10.42/node-v0.10.42-headers.tar.gz', - shasumsUrl: 'https://nodejs.org/dist/v0.10.42/SHASUMS256.txt', - versionDir: '0.10.42', - ia32: { libUrl: 'https://nodejs.org/dist/v0.10.42/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.10.42/x64/node.lib', libPath: 'x64/node.lib' }, - arm64: { libUrl: 'https://nodejs.org/dist/v0.10.42/arm64/node.lib', libPath: 'arm64/node.lib' } + // has -headers.tar.gz + it('test process release - process.release ~ node@0.10.42', function () { + var release = processRelease([], { opts: {} }, 'v0.10.42', null) + + assert.strictEqual(release.semver.version, '0.10.42') + delete release.semver + + assert.deepStrictEqual(release, { + version: '0.10.42', + name: 'node', + baseUrl: 'https://nodejs.org/dist/v0.10.42/', + tarballUrl: 'https://nodejs.org/dist/v0.10.42/node-v0.10.42-headers.tar.gz', + shasumsUrl: 'https://nodejs.org/dist/v0.10.42/SHASUMS256.txt', + versionDir: '0.10.42', + ia32: { libUrl: 'https://nodejs.org/dist/v0.10.42/node.lib', libPath: 'node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.10.42/x64/node.lib', libPath: 'x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/dist/v0.10.42/arm64/node.lib', libPath: 'arm64/node.lib' } + }) }) -}) -// has -headers.tar.gz -test('test process release - process.release ~ node@0.12.10', function (t) { - t.plan(2) - - var release = processRelease([], { opts: {} }, 'v0.12.10', null) - - t.equal(release.semver.version, '0.12.10') - delete release.semver - - t.deepEqual(release, { - version: '0.12.10', - name: 'node', - baseUrl: 'https://nodejs.org/dist/v0.12.10/', - tarballUrl: 'https://nodejs.org/dist/v0.12.10/node-v0.12.10-headers.tar.gz', - shasumsUrl: 'https://nodejs.org/dist/v0.12.10/SHASUMS256.txt', - versionDir: '0.12.10', - ia32: { libUrl: 'https://nodejs.org/dist/v0.12.10/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.12.10/x64/node.lib', libPath: 'x64/node.lib' }, - arm64: { libUrl: 'https://nodejs.org/dist/v0.12.10/arm64/node.lib', libPath: 'arm64/node.lib' } + // has -headers.tar.gz + it('test process release - process.release ~ node@0.12.10', function () { + var release = processRelease([], { opts: {} }, 'v0.12.10', null) + + assert.strictEqual(release.semver.version, '0.12.10') + delete release.semver + + assert.deepStrictEqual(release, { + version: '0.12.10', + name: 'node', + baseUrl: 'https://nodejs.org/dist/v0.12.10/', + tarballUrl: 'https://nodejs.org/dist/v0.12.10/node-v0.12.10-headers.tar.gz', + shasumsUrl: 'https://nodejs.org/dist/v0.12.10/SHASUMS256.txt', + versionDir: '0.12.10', + ia32: { libUrl: 'https://nodejs.org/dist/v0.12.10/node.lib', libPath: 'node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.12.10/x64/node.lib', libPath: 'x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/dist/v0.12.10/arm64/node.lib', libPath: 'arm64/node.lib' } + }) }) -}) -test('test process release - process.release ~ node@4.1.23', function (t) { - t.plan(2) - - var release = processRelease([], { opts: {} }, 'v4.1.23', { - name: 'node', - headersUrl: 'https://nodejs.org/dist/v4.1.23/node-v4.1.23-headers.tar.gz' + it('test process release - process.release ~ node@4.1.23', function () { + var release = processRelease([], { opts: {} }, 'v4.1.23', { + name: 'node', + headersUrl: 'https://nodejs.org/dist/v4.1.23/node-v4.1.23-headers.tar.gz' + }) + + assert.strictEqual(release.semver.version, '4.1.23') + delete release.semver + + assert.deepStrictEqual(release, { + version: '4.1.23', + name: 'node', + baseUrl: 'https://nodejs.org/dist/v4.1.23/', + tarballUrl: 'https://nodejs.org/dist/v4.1.23/node-v4.1.23-headers.tar.gz', + shasumsUrl: 'https://nodejs.org/dist/v4.1.23/SHASUMS256.txt', + versionDir: '4.1.23', + ia32: { libUrl: 'https://nodejs.org/dist/v4.1.23/win-x86/node.lib', libPath: 'win-x86/node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/dist/v4.1.23/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } + }) }) - t.equal(release.semver.version, '4.1.23') - delete release.semver - - t.deepEqual(release, { - version: '4.1.23', - name: 'node', - baseUrl: 'https://nodejs.org/dist/v4.1.23/', - tarballUrl: 'https://nodejs.org/dist/v4.1.23/node-v4.1.23-headers.tar.gz', - shasumsUrl: 'https://nodejs.org/dist/v4.1.23/SHASUMS256.txt', - versionDir: '4.1.23', - ia32: { libUrl: 'https://nodejs.org/dist/v4.1.23/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' }, - arm64: { libUrl: 'https://nodejs.org/dist/v4.1.23/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } + it('test process release - process.release ~ node@4.1.23 / corp build', function () { + var release = processRelease([], { opts: {} }, 'v4.1.23', { + name: 'node', + headersUrl: 'https://some.custom.location/node-v4.1.23-headers.tar.gz' + }) + + assert.strictEqual(release.semver.version, '4.1.23') + delete release.semver + + assert.deepStrictEqual(release, { + version: '4.1.23', + name: 'node', + baseUrl: 'https://some.custom.location/', + tarballUrl: 'https://some.custom.location/node-v4.1.23-headers.tar.gz', + shasumsUrl: 'https://some.custom.location/SHASUMS256.txt', + versionDir: '4.1.23', + ia32: { libUrl: 'https://some.custom.location/win-x86/node.lib', libPath: 'win-x86/node.lib' }, + x64: { libUrl: 'https://some.custom.location/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + arm64: { libUrl: 'https://some.custom.location/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } + }) }) -}) -test('test process release - process.release ~ node@4.1.23 / corp build', function (t) { - t.plan(2) - - var release = processRelease([], { opts: {} }, 'v4.1.23', { - name: 'node', - headersUrl: 'https://some.custom.location/node-v4.1.23-headers.tar.gz' + it('test process release - process.release ~ node@12.8.0 Windows', function () { + var release = processRelease([], { opts: {} }, 'v12.8.0', { + name: 'node', + sourceUrl: 'https://nodejs.org/download/release/v12.8.0/node-v12.8.0.tar.gz', + headersUrl: 'https://nodejs.org/download/release/v12.8.0/node-v12.8.0-headers.tar.gz', + libUrl: 'https://nodejs.org/download/release/v12.8.0/win-x64/node.lib' + }) + + assert.strictEqual(release.semver.version, '12.8.0') + delete release.semver + + assert.deepStrictEqual(release, { + version: '12.8.0', + name: 'node', + baseUrl: 'https://nodejs.org/download/release/v12.8.0/', + tarballUrl: 'https://nodejs.org/download/release/v12.8.0/node-v12.8.0-headers.tar.gz', + shasumsUrl: 'https://nodejs.org/download/release/v12.8.0/SHASUMS256.txt', + versionDir: '12.8.0', + ia32: { libUrl: 'https://nodejs.org/download/release/v12.8.0/win-x86/node.lib', libPath: 'win-x86/node.lib' }, + x64: { libUrl: 'https://nodejs.org/download/release/v12.8.0/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/download/release/v12.8.0/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } + }) }) - t.equal(release.semver.version, '4.1.23') - delete release.semver - - t.deepEqual(release, { - version: '4.1.23', - name: 'node', - baseUrl: 'https://some.custom.location/', - tarballUrl: 'https://some.custom.location/node-v4.1.23-headers.tar.gz', - shasumsUrl: 'https://some.custom.location/SHASUMS256.txt', - versionDir: '4.1.23', - ia32: { libUrl: 'https://some.custom.location/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://some.custom.location/win-x64/node.lib', libPath: 'win-x64/node.lib' }, - arm64: { libUrl: 'https://some.custom.location/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } + it('test process release - process.release ~ node@12.8.0 Windows ARM64', function () { + var release = processRelease([], { opts: {} }, 'v12.8.0', { + name: 'node', + sourceUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/node-v12.8.0.tar.gz', + headersUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/node-v12.8.0-headers.tar.gz', + libUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/win-arm64/node.lib' + }) + + assert.strictEqual(release.semver.version, '12.8.0') + delete release.semver + + assert.deepStrictEqual(release, { + version: '12.8.0', + name: 'node', + baseUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/', + tarballUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/node-v12.8.0-headers.tar.gz', + shasumsUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/SHASUMS256.txt', + versionDir: '12.8.0', + ia32: { libUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/win-x86/node.lib', libPath: 'win-x86/node.lib' }, + x64: { libUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + arm64: { libUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } + }) }) -}) -test('test process release - process.release ~ node@12.8.0 Windows', function (t) { - t.plan(2) - - var release = processRelease([], { opts: {} }, 'v12.8.0', { - name: 'node', - sourceUrl: 'https://nodejs.org/download/release/v12.8.0/node-v12.8.0.tar.gz', - headersUrl: 'https://nodejs.org/download/release/v12.8.0/node-v12.8.0-headers.tar.gz', - libUrl: 'https://nodejs.org/download/release/v12.8.0/win-x64/node.lib' - }) - - t.equal(release.semver.version, '12.8.0') - delete release.semver - - t.deepEqual(release, { - version: '12.8.0', - name: 'node', - baseUrl: 'https://nodejs.org/download/release/v12.8.0/', - tarballUrl: 'https://nodejs.org/download/release/v12.8.0/node-v12.8.0-headers.tar.gz', - shasumsUrl: 'https://nodejs.org/download/release/v12.8.0/SHASUMS256.txt', - versionDir: '12.8.0', - ia32: { libUrl: 'https://nodejs.org/download/release/v12.8.0/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://nodejs.org/download/release/v12.8.0/win-x64/node.lib', libPath: 'win-x64/node.lib' }, - arm64: { libUrl: 'https://nodejs.org/download/release/v12.8.0/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } + it('test process release - process.release ~ node@4.1.23 --target=0.10.40', function () { + var release = processRelease([], { opts: { target: '0.10.40' } }, 'v4.1.23', { + name: 'node', + headersUrl: 'https://nodejs.org/dist/v4.1.23/node-v4.1.23-headers.tar.gz' + }) + + assert.strictEqual(release.semver.version, '0.10.40') + delete release.semver + + assert.deepStrictEqual(release, { + version: '0.10.40', + name: 'node', + baseUrl: 'https://nodejs.org/dist/v0.10.40/', + tarballUrl: 'https://nodejs.org/dist/v0.10.40/node-v0.10.40.tar.gz', + shasumsUrl: 'https://nodejs.org/dist/v0.10.40/SHASUMS256.txt', + versionDir: '0.10.40', + ia32: { libUrl: 'https://nodejs.org/dist/v0.10.40/node.lib', libPath: 'node.lib' }, + x64: { libUrl: 'https://nodejs.org/dist/v0.10.40/x64/node.lib', libPath: 'x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/dist/v0.10.40/arm64/node.lib', libPath: 'arm64/node.lib' } + }) }) -}) -test('test process release - process.release ~ node@12.8.0 Windows ARM64', function (t) { - t.plan(2) - - var release = processRelease([], { opts: {} }, 'v12.8.0', { - name: 'node', - sourceUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/node-v12.8.0.tar.gz', - headersUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/node-v12.8.0-headers.tar.gz', - libUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/win-arm64/node.lib' + it('test process release - process.release ~ node@4.1.23 --dist-url=https://foo.bar/baz', function () { + var release = processRelease([], { opts: { 'dist-url': 'https://foo.bar/baz' } }, 'v4.1.23', { + name: 'node', + headersUrl: 'https://nodejs.org/dist/v4.1.23/node-v4.1.23-headers.tar.gz' + }) + + assert.strictEqual(release.semver.version, '4.1.23') + delete release.semver + + assert.deepStrictEqual(release, { + version: '4.1.23', + name: 'node', + baseUrl: 'https://foo.bar/baz/v4.1.23/', + tarballUrl: 'https://foo.bar/baz/v4.1.23/node-v4.1.23-headers.tar.gz', + shasumsUrl: 'https://foo.bar/baz/v4.1.23/SHASUMS256.txt', + versionDir: '4.1.23', + ia32: { libUrl: 'https://foo.bar/baz/v4.1.23/win-x86/node.lib', libPath: 'win-x86/node.lib' }, + x64: { libUrl: 'https://foo.bar/baz/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + arm64: { libUrl: 'https://foo.bar/baz/v4.1.23/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } + }) }) - t.equal(release.semver.version, '12.8.0') - delete release.semver - - t.deepEqual(release, { - version: '12.8.0', - name: 'node', - baseUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/', - tarballUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/node-v12.8.0-headers.tar.gz', - shasumsUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/SHASUMS256.txt', - versionDir: '12.8.0', - ia32: { libUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/win-x64/node.lib', libPath: 'win-x64/node.lib' }, - arm64: { libUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } + it('test process release - process.release ~ frankenstein@4.1.23', function () { + var release = processRelease([], { opts: {} }, 'v4.1.23', { + name: 'frankenstein', + headersUrl: 'https://frankensteinjs.org/dist/v4.1.23/frankenstein-v4.1.23-headers.tar.gz' + }) + + assert.strictEqual(release.semver.version, '4.1.23') + delete release.semver + + assert.deepStrictEqual(release, { + version: '4.1.23', + name: 'frankenstein', + baseUrl: 'https://frankensteinjs.org/dist/v4.1.23/', + tarballUrl: 'https://frankensteinjs.org/dist/v4.1.23/frankenstein-v4.1.23-headers.tar.gz', + shasumsUrl: 'https://frankensteinjs.org/dist/v4.1.23/SHASUMS256.txt', + versionDir: 'frankenstein-4.1.23', + ia32: { libUrl: 'https://frankensteinjs.org/dist/v4.1.23/win-x86/frankenstein.lib', libPath: 'win-x86/frankenstein.lib' }, + x64: { libUrl: 'https://frankensteinjs.org/dist/v4.1.23/win-x64/frankenstein.lib', libPath: 'win-x64/frankenstein.lib' }, + arm64: { libUrl: 'https://frankensteinjs.org/dist/v4.1.23/win-arm64/frankenstein.lib', libPath: 'win-arm64/frankenstein.lib' } + }) }) -}) -test('test process release - process.release ~ node@4.1.23 --target=0.10.40', function (t) { - t.plan(2) - - var release = processRelease([], { opts: { target: '0.10.40' } }, 'v4.1.23', { - name: 'node', - headersUrl: 'https://nodejs.org/dist/v4.1.23/node-v4.1.23-headers.tar.gz' + it('test process release - process.release ~ frankenstein@4.1.23 --dist-url=http://foo.bar/baz/', function () { + var release = processRelease([], { opts: { 'dist-url': 'http://foo.bar/baz/' } }, 'v4.1.23', { + name: 'frankenstein', + headersUrl: 'https://frankensteinjs.org/dist/v4.1.23/frankenstein-v4.1.23.tar.gz' + }) + + assert.strictEqual(release.semver.version, '4.1.23') + delete release.semver + + assert.deepStrictEqual(release, { + version: '4.1.23', + name: 'frankenstein', + baseUrl: 'http://foo.bar/baz/v4.1.23/', + tarballUrl: 'http://foo.bar/baz/v4.1.23/frankenstein-v4.1.23-headers.tar.gz', + shasumsUrl: 'http://foo.bar/baz/v4.1.23/SHASUMS256.txt', + versionDir: 'frankenstein-4.1.23', + ia32: { libUrl: 'http://foo.bar/baz/v4.1.23/win-x86/frankenstein.lib', libPath: 'win-x86/frankenstein.lib' }, + x64: { libUrl: 'http://foo.bar/baz/v4.1.23/win-x64/frankenstein.lib', libPath: 'win-x64/frankenstein.lib' }, + arm64: { libUrl: 'http://foo.bar/baz/v4.1.23/win-arm64/frankenstein.lib', libPath: 'win-arm64/frankenstein.lib' } + }) }) - t.equal(release.semver.version, '0.10.40') - delete release.semver - - t.deepEqual(release, { - version: '0.10.40', - name: 'node', - baseUrl: 'https://nodejs.org/dist/v0.10.40/', - tarballUrl: 'https://nodejs.org/dist/v0.10.40/node-v0.10.40.tar.gz', - shasumsUrl: 'https://nodejs.org/dist/v0.10.40/SHASUMS256.txt', - versionDir: '0.10.40', - ia32: { libUrl: 'https://nodejs.org/dist/v0.10.40/node.lib', libPath: 'node.lib' }, - x64: { libUrl: 'https://nodejs.org/dist/v0.10.40/x64/node.lib', libPath: 'x64/node.lib' }, - arm64: { libUrl: 'https://nodejs.org/dist/v0.10.40/arm64/node.lib', libPath: 'arm64/node.lib' } + it('test process release - process.release ~ node@4.0.0-rc.4', function () { + var release = processRelease([], { opts: {} }, 'v4.0.0-rc.4', { + name: 'node', + headersUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz' + }) + + assert.strictEqual(release.semver.version, '4.0.0-rc.4') + delete release.semver + + assert.deepStrictEqual(release, { + version: '4.0.0-rc.4', + name: 'node', + baseUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/', + tarballUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz', + shasumsUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/SHASUMS256.txt', + versionDir: '4.0.0-rc.4', + ia32: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x86/node.lib', libPath: 'win-x86/node.lib' }, + x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } + }) }) -}) - -test('test process release - process.release ~ node@4.1.23 --dist-url=https://foo.bar/baz', function (t) { - t.plan(2) - - var release = processRelease([], { opts: { 'dist-url': 'https://foo.bar/baz' } }, 'v4.1.23', { - name: 'node', - headersUrl: 'https://nodejs.org/dist/v4.1.23/node-v4.1.23-headers.tar.gz' - }) - - t.equal(release.semver.version, '4.1.23') - delete release.semver - - t.deepEqual(release, { - version: '4.1.23', - name: 'node', - baseUrl: 'https://foo.bar/baz/v4.1.23/', - tarballUrl: 'https://foo.bar/baz/v4.1.23/node-v4.1.23-headers.tar.gz', - shasumsUrl: 'https://foo.bar/baz/v4.1.23/SHASUMS256.txt', - versionDir: '4.1.23', - ia32: { libUrl: 'https://foo.bar/baz/v4.1.23/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://foo.bar/baz/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' }, - arm64: { libUrl: 'https://foo.bar/baz/v4.1.23/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } - }) -}) - -test('test process release - process.release ~ frankenstein@4.1.23', function (t) { - t.plan(2) - - var release = processRelease([], { opts: {} }, 'v4.1.23', { - name: 'frankenstein', - headersUrl: 'https://frankensteinjs.org/dist/v4.1.23/frankenstein-v4.1.23-headers.tar.gz' - }) - - t.equal(release.semver.version, '4.1.23') - delete release.semver - - t.deepEqual(release, { - version: '4.1.23', - name: 'frankenstein', - baseUrl: 'https://frankensteinjs.org/dist/v4.1.23/', - tarballUrl: 'https://frankensteinjs.org/dist/v4.1.23/frankenstein-v4.1.23-headers.tar.gz', - shasumsUrl: 'https://frankensteinjs.org/dist/v4.1.23/SHASUMS256.txt', - versionDir: 'frankenstein-4.1.23', - ia32: { libUrl: 'https://frankensteinjs.org/dist/v4.1.23/win-x86/frankenstein.lib', libPath: 'win-x86/frankenstein.lib' }, - x64: { libUrl: 'https://frankensteinjs.org/dist/v4.1.23/win-x64/frankenstein.lib', libPath: 'win-x64/frankenstein.lib' }, - arm64: { libUrl: 'https://frankensteinjs.org/dist/v4.1.23/win-arm64/frankenstein.lib', libPath: 'win-arm64/frankenstein.lib' } - }) -}) - -test('test process release - process.release ~ frankenstein@4.1.23 --dist-url=http://foo.bar/baz/', function (t) { - t.plan(2) - - var release = processRelease([], { opts: { 'dist-url': 'http://foo.bar/baz/' } }, 'v4.1.23', { - name: 'frankenstein', - headersUrl: 'https://frankensteinjs.org/dist/v4.1.23/frankenstein-v4.1.23.tar.gz' - }) - - t.equal(release.semver.version, '4.1.23') - delete release.semver - - t.deepEqual(release, { - version: '4.1.23', - name: 'frankenstein', - baseUrl: 'http://foo.bar/baz/v4.1.23/', - tarballUrl: 'http://foo.bar/baz/v4.1.23/frankenstein-v4.1.23-headers.tar.gz', - shasumsUrl: 'http://foo.bar/baz/v4.1.23/SHASUMS256.txt', - versionDir: 'frankenstein-4.1.23', - ia32: { libUrl: 'http://foo.bar/baz/v4.1.23/win-x86/frankenstein.lib', libPath: 'win-x86/frankenstein.lib' }, - x64: { libUrl: 'http://foo.bar/baz/v4.1.23/win-x64/frankenstein.lib', libPath: 'win-x64/frankenstein.lib' }, - arm64: { libUrl: 'http://foo.bar/baz/v4.1.23/win-arm64/frankenstein.lib', libPath: 'win-arm64/frankenstein.lib' } - }) -}) - -test('test process release - process.release ~ node@4.0.0-rc.4', function (t) { - t.plan(2) - - var release = processRelease([], { opts: {} }, 'v4.0.0-rc.4', { - name: 'node', - headersUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz' - }) - - t.equal(release.semver.version, '4.0.0-rc.4') - delete release.semver - - t.deepEqual(release, { - version: '4.0.0-rc.4', - name: 'node', - baseUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/', - tarballUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz', - shasumsUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/SHASUMS256.txt', - versionDir: '4.0.0-rc.4', - ia32: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' }, - arm64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } - }) -}) - -test('test process release - process.release ~ node@4.0.0-rc.4 passed as argv[0]', function (t) { - t.plan(2) + it('test process release - process.release ~ node@4.0.0-rc.4 passed as argv[0]', function () { // note the missing 'v' on the arg, it should normalise when checking - // whether we're on the default or not - var release = processRelease(['4.0.0-rc.4'], { opts: {} }, 'v4.0.0-rc.4', { - name: 'node', - headersUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz' - }) - - t.equal(release.semver.version, '4.0.0-rc.4') - delete release.semver - - t.deepEqual(release, { - version: '4.0.0-rc.4', - name: 'node', - baseUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/', - tarballUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz', - shasumsUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/SHASUMS256.txt', - versionDir: '4.0.0-rc.4', - ia32: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' }, - arm64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } + // whether we're on the default or not + var release = processRelease(['4.0.0-rc.4'], { opts: {} }, 'v4.0.0-rc.4', { + name: 'node', + headersUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz' + }) + + assert.strictEqual(release.semver.version, '4.0.0-rc.4') + delete release.semver + + assert.deepStrictEqual(release, { + version: '4.0.0-rc.4', + name: 'node', + baseUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/', + tarballUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz', + shasumsUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/SHASUMS256.txt', + versionDir: '4.0.0-rc.4', + ia32: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x86/node.lib', libPath: 'win-x86/node.lib' }, + x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } + }) }) -}) - -test('test process release - process.release ~ node@4.0.0-rc.4 - bogus string passed as argv[0]', function (t) { - t.plan(2) + it('test process release - process.release ~ node@4.0.0-rc.4 - bogus string passed as argv[0]', function () { // additional arguments can be passed in on the commandline that should be ignored if they - // are not specifying a valid version @ position 0 - var release = processRelease(['this is no version!'], { opts: {} }, 'v4.0.0-rc.4', { - name: 'node', - headersUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz' + // are not specifying a valid version @ position 0 + var release = processRelease(['this is no version!'], { opts: {} }, 'v4.0.0-rc.4', { + name: 'node', + headersUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz' + }) + + assert.strictEqual(release.semver.version, '4.0.0-rc.4') + delete release.semver + + assert.deepStrictEqual(release, { + version: '4.0.0-rc.4', + name: 'node', + baseUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/', + tarballUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz', + shasumsUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/SHASUMS256.txt', + versionDir: '4.0.0-rc.4', + ia32: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x86/node.lib', libPath: 'win-x86/node.lib' }, + x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + arm64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } + }) }) - t.equal(release.semver.version, '4.0.0-rc.4') - delete release.semver - - t.deepEqual(release, { - version: '4.0.0-rc.4', - name: 'node', - baseUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/', - tarballUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz', - shasumsUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/SHASUMS256.txt', - versionDir: '4.0.0-rc.4', - ia32: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-x64/node.lib', libPath: 'win-x64/node.lib' }, - arm64: { libUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } + it('test process release - NODEJS_ORG_MIRROR', function () { + process.env.NODEJS_ORG_MIRROR = 'http://foo.bar' + + var release = processRelease([], { opts: {} }, 'v4.1.23', { + name: 'node', + headersUrl: 'https://nodejs.org/dist/v4.1.23/node-v4.1.23-headers.tar.gz' + }) + + assert.strictEqual(release.semver.version, '4.1.23') + delete release.semver + + assert.deepStrictEqual(release, { + version: '4.1.23', + name: 'node', + baseUrl: 'http://foo.bar/v4.1.23/', + tarballUrl: 'http://foo.bar/v4.1.23/node-v4.1.23-headers.tar.gz', + shasumsUrl: 'http://foo.bar/v4.1.23/SHASUMS256.txt', + versionDir: '4.1.23', + ia32: { libUrl: 'http://foo.bar/v4.1.23/win-x86/node.lib', libPath: 'win-x86/node.lib' }, + x64: { libUrl: 'http://foo.bar/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' }, + arm64: { libUrl: 'http://foo.bar/v4.1.23/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } + }) + + delete process.env.NODEJS_ORG_MIRROR }) }) - -test('test process release - NODEJS_ORG_MIRROR', function (t) { - t.plan(2) - - process.env.NODEJS_ORG_MIRROR = 'http://foo.bar' - - var release = processRelease([], { opts: {} }, 'v4.1.23', { - name: 'node', - headersUrl: 'https://nodejs.org/dist/v4.1.23/node-v4.1.23-headers.tar.gz' - }) - - t.equal(release.semver.version, '4.1.23') - delete release.semver - - t.deepEqual(release, { - version: '4.1.23', - name: 'node', - baseUrl: 'http://foo.bar/v4.1.23/', - tarballUrl: 'http://foo.bar/v4.1.23/node-v4.1.23-headers.tar.gz', - shasumsUrl: 'http://foo.bar/v4.1.23/SHASUMS256.txt', - versionDir: '4.1.23', - ia32: { libUrl: 'http://foo.bar/v4.1.23/win-x86/node.lib', libPath: 'win-x86/node.lib' }, - x64: { libUrl: 'http://foo.bar/v4.1.23/win-x64/node.lib', libPath: 'win-x64/node.lib' }, - arm64: { libUrl: 'http://foo.bar/v4.1.23/win-arm64/node.lib', libPath: 'win-arm64/node.lib' } - }) - - delete process.env.NODEJS_ORG_MIRROR -}) From 55048f8be5707c295fb0876306aded75638a8b63 Mon Sep 17 00:00:00 2001 From: David Sanders Date: Tue, 6 Jun 2023 13:36:15 +0800 Subject: [PATCH 370/551] fix: log statement is for devDir not nodedir (#2840) Signed-off-by: David Sanders --- lib/install.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/install.js b/lib/install.js index f6a44c48d3..2d5db4e97d 100644 --- a/lib/install.js +++ b/lib/install.js @@ -135,14 +135,14 @@ async function install (fs, gyp, argv) { } async function go () { - log.verbose('ensuring nodedir is created', devDir) + log.verbose('ensuring devDir is created', devDir) // first create the dir for the node dev files try { const created = await fs.promises.mkdir(devDir, { recursive: true }) if (created) { - log.verbose('created nodedir', created) + log.verbose('created devDir', created) } } catch (err) { if (err.code === 'EACCES') { From 7a3fe1c4373fca5ab2b2807bb059260a950c5c84 Mon Sep 17 00:00:00 2001 From: Stefan Stojanovic Date: Fri, 9 Jun 2023 17:38:42 +0200 Subject: [PATCH 371/551] win,install: only download target_arch node.lib (#2857) Instead of downloading node.lib for all architectures, just download the one that will be needed. Install.js changed to enable downloading just node.lib for node versions that already have tarball downloaded and extracted. Not fetching lib now fails the installation. Increased installVersion because of the changes. Refs: https://github.com/nodejs/node-gyp/issues/2847 --- lib/install.js | 169 +++++++++++++++++++++++------------------- package.json | 2 +- test/test-download.js | 2 +- 3 files changed, 95 insertions(+), 78 deletions(-) diff --git a/lib/install.js b/lib/install.js index 2d5db4e97d..1eb9f14c67 100644 --- a/lib/install.js +++ b/lib/install.js @@ -22,6 +22,10 @@ const streamPipeline = util.promisify(stream.pipeline) async function install (fs, gyp, argv) { const release = processRelease(argv, gyp, process.version, process.release) + // Detecting target_arch based on logic from create-cnfig-gyp.js. Used on Windows only. + const arch = win ? (gyp.opts.target_arch || gyp.opts.arch || process.arch || 'ia32') : '' + // Used to prevent downloading tarball if only new node.lib is required on Windows. + let shouldDownloadTarball = true // Determine which node dev files version we are installing log.verbose('install', 'input version string %j', release.version) @@ -92,6 +96,26 @@ async function install (fs, gyp, argv) { } } log.verbose('install', 'version is good') + if (win) { + log.verbose('on Windows; need to check node.lib') + const nodeLibPath = path.resolve(devDir, arch, 'node.lib') + try { + await fs.promises.stat(nodeLibPath) + } catch (err) { + if (err.code === 'ENOENT') { + log.verbose('install', `version not already installed for ${arch}, continuing with install`, release.version) + try { + shouldDownloadTarball = false + return await go() + } catch (err) { + return rollback(err) + } + } else if (err.code === 'EACCES') { + return eaccesFallback(err) + } + throw err + } + } } else { try { return await go() @@ -179,6 +203,7 @@ async function install (fs, gyp, argv) { } // download the tarball and extract! + // Ommited on Windows if only new node.lib is required // on Windows there can be file errors from tar if parallel installs // are happening (not uncommon with multiple native modules) so @@ -186,59 +211,61 @@ async function install (fs, gyp, argv) { const tarExtractDir = win ? await fs.promises.mkdtemp(path.join(os.tmpdir(), 'node-gyp-tmp-')) : devDir try { - if (tarPath) { - await tar.extract({ - file: tarPath, - strip: 1, - filter: isValid, - onwarn, - cwd: tarExtractDir - }) - } else { - try { - const res = await download(gyp, release.tarballUrl) + if (shouldDownloadTarball) { + if (tarPath) { + await tar.extract({ + file: tarPath, + strip: 1, + filter: isValid, + onwarn, + cwd: tarExtractDir + }) + } else { + try { + const res = await download(gyp, release.tarballUrl) - if (res.status !== 200) { - throw new Error(`${res.status} response downloading ${release.tarballUrl}`) - } + if (res.status !== 200) { + throw new Error(`${res.status} response downloading ${release.tarballUrl}`) + } - await streamPipeline( - res.body, - // content checksum - new ShaSum((_, checksum) => { - const filename = path.basename(release.tarballUrl).trim() - contentShasums[filename] = checksum - log.verbose('content checksum', filename, checksum) - }), - tar.extract({ - strip: 1, - cwd: tarExtractDir, - filter: isValid, - onwarn - }) - ) - } catch (err) { + await streamPipeline( + res.body, + // content checksum + new ShaSum((_, checksum) => { + const filename = path.basename(release.tarballUrl).trim() + contentShasums[filename] = checksum + log.verbose('content checksum', filename, checksum) + }), + tar.extract({ + strip: 1, + cwd: tarExtractDir, + filter: isValid, + onwarn + }) + ) + } catch (err) { // something went wrong downloading the tarball? - if (err.code === 'ENOTFOUND') { - throw new Error('This is most likely not a problem with node-gyp or the package itself and\n' + + if (err.code === 'ENOTFOUND') { + throw new Error('This is most likely not a problem with node-gyp or the package itself and\n' + 'is related to network connectivity. In most cases you are behind a proxy or have bad \n' + 'network settings.') + } + throw err } - throw err } - } - // invoked after the tarball has finished being extracted - if (extractErrors || extractCount === 0) { - throw new Error('There was a fatal problem while downloading/extracting the tarball') - } + // invoked after the tarball has finished being extracted + if (extractErrors || extractCount === 0) { + throw new Error('There was a fatal problem while downloading/extracting the tarball') + } - log.verbose('tarball', 'done parsing tarball') + log.verbose('tarball', 'done parsing tarball') + } const installVersionPath = path.resolve(tarExtractDir, 'installVersion') await Promise.all([ - // need to download node.lib - ...(win ? downloadNodeLib() : []), + // need to download node.lib + ...(win ? [downloadNodeLib()] : []), // write the "installVersion" file fs.promises.writeFile(installVersionPath, gyp.package.installVersion + '\n'), // Only download SHASUMS.txt if we downloaded something in need of SHA verification @@ -293,43 +320,33 @@ async function install (fs, gyp, argv) { log.verbose('checksum data', JSON.stringify(expectShasums)) } - function downloadNodeLib () { + async function downloadNodeLib () { log.verbose('on Windows; need to download `' + release.name + '.lib`...') - const archs = ['ia32', 'x64', 'arm64'] - return archs.map(async (arch) => { - const dir = path.resolve(tarExtractDir, arch) - const targetLibPath = path.resolve(dir, release.name + '.lib') - const { libUrl, libPath } = release[arch] - const name = `${arch} ${release.name}.lib` - log.verbose(name, 'dir', dir) - log.verbose(name, 'url', libUrl) - - await fs.promises.mkdir(dir, { recursive: true }) - log.verbose('streaming', name, 'to:', targetLibPath) - - const res = await download(gyp, libUrl) - - if (res.status === 403 || res.status === 404) { - if (arch === 'arm64') { - // Arm64 is a newer platform on Windows and not all node distributions provide it. - log.verbose(`${name} was not found in ${libUrl}`) - } else { - log.warn(`${name} was not found in ${libUrl}`) - } - return - } else if (res.status !== 200) { - throw new Error(`${res.status} status code downloading ${name}`) - } + const dir = path.resolve(tarExtractDir, arch) + const targetLibPath = path.resolve(dir, release.name + '.lib') + const { libUrl, libPath } = release[arch] + const name = `${arch} ${release.name}.lib` + log.verbose(name, 'dir', dir) + log.verbose(name, 'url', libUrl) + + await fs.promises.mkdir(dir, { recursive: true }) + log.verbose('streaming', name, 'to:', targetLibPath) + + const res = await download(gyp, libUrl) + + // Since only required node.lib is downloaded throw error if it is not fetched + if (res.status !== 200) { + throw new Error(`${res.status} status code downloading ${name}`) + } - return streamPipeline( - res.body, - new ShaSum((_, checksum) => { - contentShasums[libPath] = checksum - log.verbose('content checksum', libPath, checksum) - }), - fs.createWriteStream(targetLibPath) - ) - }) + return streamPipeline( + res.body, + new ShaSum((_, checksum) => { + contentShasums[libPath] = checksum + log.verbose('content checksum', libPath, checksum) + }), + fs.createWriteStream(targetLibPath) + ) } // downloadNodeLib() } // go() diff --git a/package.json b/package.json index 15cb6a72ea..331599447f 100644 --- a/package.json +++ b/package.json @@ -12,7 +12,7 @@ "gyp" ], "version": "9.3.1", - "installVersion": 10, + "installVersion": 11, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { "type": "git", diff --git a/test/test-download.js b/test/test-download.js index 6eeba8a1dd..1dd5a51b06 100644 --- a/test/test-download.js +++ b/test/test-download.js @@ -180,7 +180,7 @@ describe('download', function () { await util.promisify(install)(prog, []) const data = await fs.promises.readFile(path.join(expectedDir, 'installVersion'), 'utf8') - assert.strictEqual(data, '10\n', 'correct installVersion') + assert.strictEqual(data, '11\n', 'correct installVersion') const list = await fs.promises.readdir(path.join(expectedDir, 'include/node')) assert.ok(list.includes('common.gypi')) From a0b3d1c3afed71a74501476fcbc6ee3fface4d13 Mon Sep 17 00:00:00 2001 From: Stefan Stojanovic Date: Mon, 12 Jun 2023 21:41:39 +0200 Subject: [PATCH 372/551] test: remove deprecated Node.js and Python (#2868) * test: remove deprecated node.js and python Removed Node.js v14.x and Python v3.7. Also added Node.js v20.x. * Update .github/workflows/tests.yml Co-authored-by: Christian Clauss --------- Co-authored-by: Christian Clauss --- .github/workflows/tests.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 0da6cdbf87..517b2d95a4 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -19,8 +19,8 @@ jobs: fail-fast: false max-parallel: 15 matrix: - node: [14.x, 16.x, 18.x] - python: ["3.7", "3.9", "3.11"] + node: [16.x, 18.x, 20.x] + python: ["3.8", "3.11"] os: [macos-latest, ubuntu-latest, windows-latest] runs-on: ${{ matrix.os }} steps: From 33391db3a0008eff8408890da6ab232f2f90fcab Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Mon, 12 Jun 2023 19:42:16 +0000 Subject: [PATCH 373/551] chore: release 9.4.0 --- CHANGELOG.md | 30 ++++++++++++++++++++++++++++++ package.json | 2 +- 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4131521515..9fb5f11847 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,35 @@ # Changelog +## [9.4.0](https://www.github.com/nodejs/node-gyp/compare/v9.3.1...v9.4.0) (2023-06-12) + + +### Features + +* add support for native windows arm64 build tools ([bb76021](https://www.github.com/nodejs/node-gyp/commit/bb76021d35964d2bb125bc6214286f35ae4e6cad)) +* Upgrade Python linting from flake8 to ruff ([#2815](https://www.github.com/nodejs/node-gyp/issues/2815)) ([fc0ddc6](https://www.github.com/nodejs/node-gyp/commit/fc0ddc6523c62b10e5ca1257500b3ceac01450a7)) + + +### Bug Fixes + +* extract tarball to temp directory on Windows ([#2846](https://www.github.com/nodejs/node-gyp/issues/2846)) ([aaa117c](https://www.github.com/nodejs/node-gyp/commit/aaa117c514430aa2c1e568b95df1b6ed1c1fd3b6)) +* log statement is for devDir not nodedir ([#2840](https://www.github.com/nodejs/node-gyp/issues/2840)) ([55048f8](https://www.github.com/nodejs/node-gyp/commit/55048f8be5707c295fb0876306aded75638a8b63)) + + +### Miscellaneous + +* get update-gyp.py to work with Python >= v3.5 ([#2826](https://www.github.com/nodejs/node-gyp/issues/2826)) ([337e8e6](https://www.github.com/nodejs/node-gyp/commit/337e8e68209bd2481cbb11dacce61234dc5c9419)) + + +### Doc + +* docs/README.md add advise about deprecated node-sass ([#2828](https://www.github.com/nodejs/node-gyp/issues/2828)) ([6f3c2d3](https://www.github.com/nodejs/node-gyp/commit/6f3c2d3c6c0de0dbf8c7245f34c2e0b3eea53812)) +* Update README.md ([#2822](https://www.github.com/nodejs/node-gyp/issues/2822)) ([c7927e2](https://www.github.com/nodejs/node-gyp/commit/c7927e228dfde059c93e08c26b54dd8026144583)) + + +### Tests + +* remove deprecated Node.js and Python ([#2868](https://www.github.com/nodejs/node-gyp/issues/2868)) ([a0b3d1c](https://www.github.com/nodejs/node-gyp/commit/a0b3d1c3afed71a74501476fcbc6ee3fface4d13)) + ### [9.3.1](https://www.github.com/nodejs/node-gyp/compare/v9.3.0...v9.3.1) (2022-12-16) diff --git a/package.json b/package.json index 331599447f..7e9fb648ab 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "bindings", "gyp" ], - "version": "9.3.1", + "version": "9.4.0", "installVersion": 11, "author": "Nathan Rajlich (http://tootallnate.net)", "repository": { From 192eec2aca15c41b77427163f9c8bb0ef1a38edd Mon Sep 17 00:00:00 2001 From: Luke Karrys Date: Tue, 20 Jun 2023 06:44:18 -0700 Subject: [PATCH 374/551] Sync deps and engines with npm (#2770) * feat!: update `engines.node` to `^14.17.0 || ^16.13.0 || >=18.0.0` * deps: nopt@^7.0.0 * feat: replace npmlog with proc-log * deps: standard@17.0.0 and fix linting errors * deps: which@3.0.0 - this also promiisifies the build command * deps: glob@8.0.3 * feat: drop rimraf dependency * fix: use fs/promises in favor of fs.promises --- bin/node-gyp.js | 18 +-- lib/build.js | 175 ++++++++++++++---------------- lib/clean.js | 14 ++- lib/configure.js | 67 ++++++------ lib/create-config-gypi.js | 6 +- lib/find-node-directory.js | 10 +- lib/find-python.js | 34 +++--- lib/find-visualstudio.js | 29 +++-- lib/install.js | 2 +- lib/list.js | 4 +- lib/log.js | 165 ++++++++++++++++++++++++++++ lib/node-gyp.js | 25 ++--- lib/process-release.js | 45 ++++---- lib/remove.js | 41 ++++--- lib/util.js | 22 +--- package.json | 13 +-- test/process-exec-sync.js | 26 ++--- test/simple-proxy.js | 4 +- test/test-addon.js | 48 ++++---- test/test-configure-python.js | 23 ++-- test/test-download.js | 25 ++--- test/test-find-accessible-sync.js | 30 ++--- test/test-find-node-directory.js | 28 ++--- test/test-find-python.js | 28 ++--- test/test-find-visualstudio.js | 18 +-- test/test-install.js | 12 +- test/test-process-release.js | 36 +++--- 27 files changed, 533 insertions(+), 415 deletions(-) create mode 100644 lib/log.js diff --git a/bin/node-gyp.js b/bin/node-gyp.js index 8652ea21ec..3441973da4 100755 --- a/bin/node-gyp.js +++ b/bin/node-gyp.js @@ -6,7 +6,7 @@ process.title = 'node-gyp' const envPaths = require('env-paths') const gyp = require('../') -const log = require('npmlog') +const log = require('../lib/log') const os = require('os') /** @@ -14,11 +14,11 @@ const os = require('os') */ const prog = gyp() -var completed = false +let completed = false prog.parseArgv(process.argv) prog.devDir = prog.opts.devdir -var homeDir = os.homedir() +const homeDir = os.homedir() if (prog.devDir) { prog.devDir = prog.devDir.replace(/^~/, homeDir) } else if (homeDir) { @@ -48,11 +48,11 @@ log.info('using', 'node@%s | %s | %s', process.versions.node, process.platform, * Change dir if -C/--directory was passed. */ -var dir = prog.opts.directory +const dir = prog.opts.directory if (dir) { - var fs = require('fs') + const fs = require('fs') try { - var stat = fs.statSync(dir) + const stat = fs.statSync(dir) if (stat.isDirectory()) { log.info('chdir', dir) process.chdir(dir) @@ -69,7 +69,7 @@ if (dir) { } function run () { - var command = prog.todo.shift() + const command = prog.todo.shift() if (!command) { // done! completed = true @@ -86,7 +86,7 @@ function run () { return process.exit(1) } if (command.name === 'list') { - var versions = arguments[1] + const versions = arguments[1] if (versions.length > 0) { versions.forEach(function (version) { console.log(version) @@ -120,7 +120,7 @@ process.on('uncaughtException', function (err) { function errorMessage () { // copied from npm's lib/utils/error-handler.js - var os = require('os') + const os = require('os') log.error('System', os.type() + ' ' + os.release()) log.error('command', process.argv .map(JSON.stringify).join(' ')) diff --git a/lib/build.js b/lib/build.js index ea1f90652a..3b95b76353 100644 --- a/lib/build.js +++ b/lib/build.js @@ -1,14 +1,15 @@ 'use strict' -const fs = require('graceful-fs') +const fs = require('graceful-fs').promises +const { promisify } = require('util') const path = require('path') -const glob = require('glob') -const log = require('npmlog') +const glob = promisify(require('glob')) +const log = require('./log') const which = require('which') const win = process.platform === 'win32' -function build (gyp, argv, callback) { - var platformMake = 'make' +async function build (gyp, argv) { + let platformMake = 'make' if (process.platform === 'aix') { platformMake = 'gmake' } else if (process.platform === 'os400') { @@ -21,113 +22,103 @@ function build (gyp, argv, callback) { }) } - var makeCommand = gyp.opts.make || process.env.MAKE || platformMake - var command = win ? 'msbuild' : makeCommand - var jobs = gyp.opts.jobs || process.env.JOBS - var buildType - var config - var arch - var nodeDir - var guessedSolution + const makeCommand = gyp.opts.make || process.env.MAKE || platformMake + let command = win ? 'msbuild' : makeCommand + const jobs = gyp.opts.jobs || process.env.JOBS + let buildType + let config + let arch + let nodeDir + let guessedSolution - loadConfigGypi() + await loadConfigGypi() /** * Load the "config.gypi" file that was generated during "configure". */ - function loadConfigGypi () { - var configPath = path.resolve('build', 'config.gypi') - - fs.readFile(configPath, 'utf8', function (err, data) { - if (err) { - if (err.code === 'ENOENT') { - callback(new Error('You must run `node-gyp configure` first!')) - } else { - callback(err) - } - return + async function loadConfigGypi () { + let data + try { + const configPath = path.resolve('build', 'config.gypi') + data = await fs.readFile(configPath, 'utf8') + } catch (err) { + if (err.code === 'ENOENT') { + throw new Error('You must run `node-gyp configure` first!') + } else { + throw err } - config = JSON.parse(data.replace(/#.+\n/, '')) + } - // get the 'arch', 'buildType', and 'nodeDir' vars from the config - buildType = config.target_defaults.default_configuration - arch = config.variables.target_arch - nodeDir = config.variables.nodedir + config = JSON.parse(data.replace(/#.+\n/, '')) - if ('debug' in gyp.opts) { - buildType = gyp.opts.debug ? 'Debug' : 'Release' - } - if (!buildType) { - buildType = 'Release' - } + // get the 'arch', 'buildType', and 'nodeDir' vars from the config + buildType = config.target_defaults.default_configuration + arch = config.variables.target_arch + nodeDir = config.variables.nodedir - log.verbose('build type', buildType) - log.verbose('architecture', arch) - log.verbose('node dev dir', nodeDir) + if ('debug' in gyp.opts) { + buildType = gyp.opts.debug ? 'Debug' : 'Release' + } + if (!buildType) { + buildType = 'Release' + } - if (win) { - findSolutionFile() - } else { - doWhich() - } - }) + log.verbose('build type', buildType) + log.verbose('architecture', arch) + log.verbose('node dev dir', nodeDir) + + if (win) { + await findSolutionFile() + } else { + await doWhich() + } } /** * On Windows, find the first build/*.sln file. */ - function findSolutionFile () { - glob('build/*.sln', function (err, files) { - if (err) { - return callback(err) - } - if (files.length === 0) { - return callback(new Error('Could not find *.sln file. Did you run "configure"?')) - } - guessedSolution = files[0] - log.verbose('found first Solution file', guessedSolution) - doWhich() - }) + async function findSolutionFile () { + const files = await glob('build/*.sln') + if (files.length === 0) { + throw new Error('Could not find *.sln file. Did you run "configure"?') + } + guessedSolution = files[0] + log.verbose('found first Solution file', guessedSolution) + await doWhich() } /** * Uses node-which to locate the msbuild / make executable. */ - function doWhich () { + async function doWhich () { // On Windows use msbuild provided by node-gyp configure if (win) { if (!config.variables.msbuild_path) { - return callback(new Error( - 'MSBuild is not set, please run `node-gyp configure`.')) + throw new Error('MSBuild is not set, please run `node-gyp configure`.') } command = config.variables.msbuild_path log.verbose('using MSBuild:', command) - doBuild() + await doBuild() return } + // First make sure we have the build command in the PATH - which(command, function (err, execPath) { - if (err) { - // Some other error or 'make' not found on Unix, report that to the user - callback(err) - return - } - log.verbose('`which` succeeded for `' + command + '`', execPath) - doBuild() - }) + const execPath = await which(command) + log.verbose('`which` succeeded for `' + command + '`', execPath) + await doBuild() } /** * Actually spawn the process and compile the module. */ - function doBuild () { + async function doBuild () { // Enable Verbose build - var verbose = log.levels[log.level] <= log.levels.verbose - var j + const verbose = log.logger.isVisible('verbose') + let j if (!win && verbose) { argv.push('V=1') @@ -147,10 +138,12 @@ function build (gyp, argv, callback) { // Convert .gypi config target_arch to MSBuild /Platform // Since there are many ways to state '32-bit Intel', default to it. // N.B. msbuild's Condition string equality tests are case-insensitive. - var archLower = arch.toLowerCase() - var p = archLower === 'x64' ? 'x64' - : (archLower === 'arm' ? 'ARM' - : (archLower === 'arm64' ? 'ARM64' : 'Win32')) + const archLower = arch.toLowerCase() + const p = archLower === 'x64' + ? 'x64' + : (archLower === 'arm' + ? 'ARM' + : (archLower === 'arm64' ? 'ARM64' : 'Win32')) argv.push('/p:Configuration=' + buildType + ';Platform=' + p) if (jobs) { j = parseInt(jobs, 10) @@ -179,7 +172,7 @@ function build (gyp, argv, callback) { if (win) { // did the user specify their own .sln file? - var hasSln = argv.some(function (arg) { + const hasSln = argv.some(function (arg) { return path.extname(arg) === '.sln' }) if (!hasSln) { @@ -194,20 +187,20 @@ function build (gyp, argv, callback) { log.verbose('bin symlinks', `adding symlinks (such as Python), at "${buildBinsDir}", to PATH`) } - var proc = gyp.spawn(command, argv) - proc.on('exit', onExit) - } - - function onExit (code, signal) { - if (code !== 0) { - return callback(new Error('`' + command + '` failed with exit code: ' + code)) - } - if (signal) { - return callback(new Error('`' + command + '` got signal: ' + signal)) - } - callback() + const proc = gyp.spawn(command, argv) + await new Promise((resolve, reject) => proc.on('exit', (code, signal) => { + if (code !== 0) { + return reject(new Error('`' + command + '` failed with exit code: ' + code)) + } + if (signal) { + return reject(new Error('`' + command + '` got signal: ' + signal)) + } + resolve() + })) } } -module.exports = build +module.exports = function (gyp, argv, callback) { + build(gyp, argv).then(callback.bind(undefined, null), callback) +} module.exports.usage = 'Invokes `' + (win ? 'msbuild' : 'make') + '` and builds the module' diff --git a/lib/clean.js b/lib/clean.js index dbfa4dbb99..844b47aed1 100644 --- a/lib/clean.js +++ b/lib/clean.js @@ -1,15 +1,17 @@ 'use strict' -const rm = require('rimraf') -const log = require('npmlog') +const fs = require('fs/promises') +const log = require('./log') -function clean (gyp, argv, callback) { +async function clean (gyp, argv) { // Remove the 'build' dir - var buildDir = 'build' + const buildDir = 'build' log.verbose('clean', 'removing "%s" directory', buildDir) - rm(buildDir, callback) + await fs.rm(buildDir, { recursive: true, force: true }) } -module.exports = clean +module.exports = function (gyp, argv, callback) { + clean(gyp, argv).then(callback.bind(undefined, null), callback) +} module.exports.usage = 'Removes any generated build files and the "out" dir' diff --git a/lib/configure.js b/lib/configure.js index 1ca3ade709..97dccc6db8 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -2,26 +2,27 @@ const fs = require('graceful-fs') const path = require('path') -const log = require('npmlog') +const log = require('./log') const os = require('os') const processRelease = require('./process-release') const win = process.platform === 'win32' const findNodeDirectory = require('./find-node-directory') const createConfigGypi = require('./create-config-gypi') const msgFormat = require('util').format -var findPython = require('./find-python') +const findPython = require('./find-python') +let findVisualStudio if (win) { - var findVisualStudio = require('./find-visualstudio') + findVisualStudio = require('./find-visualstudio') } function configure (gyp, argv, callback) { - var python - var buildDir = path.resolve('build') - var buildBinsDir = path.join(buildDir, 'node_gyp_bins') - var configNames = ['config.gypi', 'common.gypi'] - var configs = [] - var nodeDir - var release = processRelease(argv, gyp, process.version, process.release) + let python + const buildDir = path.resolve('build') + const buildBinsDir = path.join(buildDir, 'node_gyp_bins') + const configNames = ['config.gypi', 'common.gypi'] + const configs = [] + let nodeDir + const release = processRelease(argv, gyp, process.version, process.release) findPython(gyp.opts.python, function (err, found) { if (err) { @@ -129,11 +130,11 @@ function configure (gyp, argv, callback) { } function findConfigs () { - var name = configNames.shift() + const name = configNames.shift() if (!name) { return runGyp() } - var fullPath = path.resolve(name) + const fullPath = path.resolve(name) log.verbose(name, 'checking for gypi file: %s', fullPath) fs.stat(fullPath, function (err) { @@ -175,11 +176,13 @@ function configure (gyp, argv, callback) { // For AIX and z/OS we need to set up the path to the exports file // which contains the symbols needed for linking. - var nodeExpFile + let nodeExpFile + let nodeRootDir + let candidates + let logprefix = 'find exports file' if (process.platform === 'aix' || process.platform === 'os390' || process.platform === 'os400') { - var ext = process.platform === 'os390' ? 'x' : 'exp' - var nodeRootDir = findNodeDirectory() - var candidates + const ext = process.platform === 'os390' ? 'x' : 'exp' + nodeRootDir = findNodeDirectory() if (process.platform === 'aix' || process.platform === 'os400') { candidates = [ @@ -202,12 +205,11 @@ function configure (gyp, argv, callback) { }) } - var logprefix = 'find exports file' nodeExpFile = findAccessibleSync(logprefix, nodeRootDir, candidates) if (nodeExpFile !== undefined) { log.verbose(logprefix, 'Found exports file: %s', nodeExpFile) } else { - var msg = msgFormat('Could not find node.%s file in %s', ext, nodeRootDir) + const msg = msgFormat('Could not find node.%s file in %s', ext, nodeRootDir) log.error(logprefix, 'Could not find exports file') return callback(new Error(msg)) } @@ -215,11 +217,11 @@ function configure (gyp, argv, callback) { // For z/OS we need to set up the path to zoslib include directory, // which contains headers included in v8config.h. - var zoslibIncDir + let zoslibIncDir if (process.platform === 'os390') { logprefix = "find zoslib's zos-base.h:" let msg - var zoslibIncPath = process.env.ZOSLIB_INCLUDES + let zoslibIncPath = process.env.ZOSLIB_INCLUDES if (zoslibIncPath) { zoslibIncPath = findAccessibleSync(logprefix, zoslibIncPath, ['zos-base.h']) if (zoslibIncPath === undefined) { @@ -252,22 +254,22 @@ function configure (gyp, argv, callback) { } // this logic ported from the old `gyp_addon` python file - var gypScript = path.resolve(__dirname, '..', 'gyp', 'gyp_main.py') - var addonGypi = path.resolve(__dirname, '..', 'addon.gypi') - var commonGypi = path.resolve(nodeDir, 'include/node/common.gypi') + const gypScript = path.resolve(__dirname, '..', 'gyp', 'gyp_main.py') + const addonGypi = path.resolve(__dirname, '..', 'addon.gypi') + let commonGypi = path.resolve(nodeDir, 'include/node/common.gypi') fs.stat(commonGypi, function (err) { if (err) { commonGypi = path.resolve(nodeDir, 'common.gypi') } - var outputDir = 'build' + let outputDir = 'build' if (win) { // Windows expects an absolute path outputDir = buildDir } - var nodeGypDir = path.resolve(__dirname, '..') + const nodeGypDir = path.resolve(__dirname, '..') - var nodeLibFile = path.join(nodeDir, + let nodeLibFile = path.join(nodeDir, !gyp.opts.nodedir ? '<(target_arch)' : '$(Configuration)', release.name + '.lib') @@ -309,13 +311,13 @@ function configure (gyp, argv, callback) { argv.unshift(gypScript) // make sure python uses files that came with this particular node package - var pypath = [path.join(__dirname, '..', 'gyp', 'pylib')] + const pypath = [path.join(__dirname, '..', 'gyp', 'pylib')] if (process.env.PYTHONPATH) { pypath.push(process.env.PYTHONPATH) } process.env.PYTHONPATH = pypath.join(win ? ';' : ':') - var cp = gyp.spawn(python, argv) + const cp = gyp.spawn(python, argv) cp.on('exit', onCpExit) }) } @@ -336,10 +338,11 @@ function configure (gyp, argv, callback) { * readable. */ function findAccessibleSync (logprefix, dir, candidates) { - for (var next = 0; next < candidates.length; next++) { - var candidate = path.resolve(dir, candidates[next]) + for (let next = 0; next < candidates.length; next++) { + const candidate = path.resolve(dir, candidates[next]) + let fd try { - var fd = fs.openSync(candidate, 'r') + fd = fs.openSync(candidate, 'r') } catch (e) { // this candidate was not found or not readable, do nothing log.silly(logprefix, 'Could not open %s: %s', candidate, e.message) @@ -355,6 +358,6 @@ function findAccessibleSync (logprefix, dir, candidates) { module.exports = configure module.exports.test = { - findAccessibleSync: findAccessibleSync + findAccessibleSync } module.exports.usage = 'Generates ' + (win ? 'MSVC project files' : 'a Makefile') + ' for the current module' diff --git a/lib/create-config-gypi.js b/lib/create-config-gypi.js index ced4911502..2eab6f3035 100644 --- a/lib/create-config-gypi.js +++ b/lib/create-config-gypi.js @@ -1,7 +1,7 @@ 'use strict' const fs = require('graceful-fs') -const log = require('npmlog') +const log = require('./log') const path = require('path') function parseConfigGypi (config) { @@ -142,6 +142,6 @@ async function createConfigGypi ({ gyp, buildDir, nodeDir, vsInfo }) { module.exports = createConfigGypi module.exports.test = { - parseConfigGypi: parseConfigGypi, - getCurrentConfigGypi: getCurrentConfigGypi + parseConfigGypi, + getCurrentConfigGypi } diff --git a/lib/find-node-directory.js b/lib/find-node-directory.js index 0dd781a6cf..8838b81d33 100644 --- a/lib/find-node-directory.js +++ b/lib/find-node-directory.js @@ -1,7 +1,7 @@ 'use strict' const path = require('path') -const log = require('npmlog') +const log = require('./log') function findNodeDirectory (scriptLocation, processObj) { // set dirname and process if not passed in @@ -14,10 +14,10 @@ function findNodeDirectory (scriptLocation, processObj) { } // Have a look to see what is above us, to try and work out where we are - var npmParentDirectory = path.join(scriptLocation, '../../../..') + const npmParentDirectory = path.join(scriptLocation, '../../../..') log.verbose('node-gyp root', 'npm_parent_directory is ' + path.basename(npmParentDirectory)) - var nodeRootDir = '' + let nodeRootDir = '' log.verbose('node-gyp root', 'Finding node root directory') if (path.basename(npmParentDirectory) === 'deps') { @@ -41,8 +41,8 @@ function findNodeDirectory (scriptLocation, processObj) { } else { // We don't know where we are, try working it out from the location // of the node binary - var nodeDir = path.dirname(processObj.execPath) - var directoryUp = path.basename(nodeDir) + const nodeDir = path.dirname(processObj.execPath) + const directoryUp = path.basename(nodeDir) if (directoryUp === 'bin') { nodeRootDir = path.join(nodeDir, '..') } else if (directoryUp === 'Release' || directoryUp === 'Debug') { diff --git a/lib/find-python.js b/lib/find-python.js index a445e825b9..c8129c0f65 100644 --- a/lib/find-python.js +++ b/lib/find-python.js @@ -1,11 +1,10 @@ 'use strict' -const log = require('npmlog') +const log = require('./log') const semver = require('semver') const cp = require('child_process') const extend = require('util')._extend // eslint-disable-line const win = process.platform === 'win32' -const logWithPrefix = require('./util').logWithPrefix const systemDrive = process.env.SystemDrive || 'C:' const username = process.env.USERNAME || process.env.USER || getOsUserInfo() @@ -46,7 +45,7 @@ function PythonFinder (configPython, callback) { } PythonFinder.prototype = { - log: logWithPrefix(log, 'find Python'), + log: log.withPrefix('find Python'), argsExecutable: ['-c', 'import sys; print(sys.executable);'], argsVersion: ['-c', 'import sys; print("%s.%s.%s" % sys.version_info[:3]);'], semverRange: '>=3.6.0', @@ -54,7 +53,7 @@ PythonFinder.prototype = { // These can be overridden for testing: execFile: cp.execFile, env: process.env, - win: win, + win, pyLauncher: 'py.exe', winDefaultLocations: winDefaultLocationsArray, @@ -69,7 +68,7 @@ PythonFinder.prototype = { // Ignore errors, keep trying until Python is found. findPython: function findPython () { const SKIP = 0; const FAIL = 1 - var toCheck = getChecks.apply(this) + const toCheck = getChecks.apply(this) function getChecks () { if (this.env.NODE_GYP_FORCE_PYTHON) { @@ -85,7 +84,7 @@ PythonFinder.prototype = { }] } - var checks = [ + const checks = [ { before: () => { if (!this.configPython) { @@ -128,7 +127,7 @@ PythonFinder.prototype = { ] if (this.win) { - for (var i = 0; i < this.winDefaultLocations.length; ++i) { + for (let i = 0; i < this.winDefaultLocations.length; ++i) { const location = this.winDefaultLocations[i] checks.push({ before: () => { @@ -181,9 +180,9 @@ PythonFinder.prototype = { // Will exit the Python finder on success. // If on Windows, run in a CMD shell to support BAT/CMD launchers. checkCommand: function checkCommand (command, errorCallback) { - var exec = command - var args = this.argsExecutable - var shell = false + let exec = command + let args = this.argsExecutable + let shell = false if (this.win) { // Arguments have to be manually quoted exec = `"${exec}"` @@ -250,7 +249,7 @@ PythonFinder.prototype = { this.addLog(`- version is "${version}"`) const range = new semver.Range(this.semverRange) - var valid = false + let valid = false try { valid = range.test(version) } catch (err) { @@ -272,9 +271,9 @@ PythonFinder.prototype = { // Run an executable or shell command, trimming the output. run: function run (exec, args, shell, callback) { - var env = extend({}, this.env) + const env = extend({}, this.env) env.TERM = 'dumb' - const opts = { env: env, shell: shell } + const opts = { env, shell } this.log.silly('execFile: exec = %j', exec) this.log.silly('execFile: args = %j', args) @@ -306,7 +305,8 @@ PythonFinder.prototype = { fail: function fail () { const errorLog = this.errorLog.join('\n') - const pathExample = this.win ? 'C:\\Path\\To\\python.exe' + const pathExample = this.win + ? 'C:\\Path\\To\\python.exe' : '/path/to/pythonexecutable' // For Windows 80 col console, use up to the column before the one marked // with X (total 79 chars including logger prefix, 58 chars usable here): @@ -333,12 +333,12 @@ PythonFinder.prototype = { } function findPython (configPython, callback) { - var finder = new PythonFinder(configPython, callback) + const finder = new PythonFinder(configPython, callback) finder.findPython() } module.exports = findPython module.exports.test = { - PythonFinder: PythonFinder, - findPython: findPython + PythonFinder, + findPython } diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js index 16f6e79559..45ef7d3fb1 100644 --- a/lib/find-visualstudio.js +++ b/lib/find-visualstudio.js @@ -1,10 +1,9 @@ 'use strict' -const log = require('npmlog') +const log = require('./log') const execFile = require('child_process').execFile const fs = require('fs') const path = require('path').win32 -const logWithPrefix = require('./util').logWithPrefix const regSearchKeys = require('./util').regSearchKeys function findVisualStudio (nodeSemver, configMsvsVersion, callback) { @@ -22,9 +21,9 @@ function VisualStudioFinder (nodeSemver, configMsvsVersion, callback) { } VisualStudioFinder.prototype = { - log: logWithPrefix(log, 'find VS'), + log: log.withPrefix('find VS'), - regSearchKeys: regSearchKeys, + regSearchKeys, // Logs a message at verbose level, but also saves it to be displayed later // at error level if an error occurs. This should help diagnose the problem. @@ -126,10 +125,10 @@ VisualStudioFinder.prototype = { // Invoke the PowerShell script to get information about Visual Studio 2017 // or newer installations findVisualStudio2017OrNewer: function findVisualStudio2017OrNewer (cb) { - var ps = path.join(process.env.SystemRoot, 'System32', + const ps = path.join(process.env.SystemRoot, 'System32', 'WindowsPowerShell', 'v1.0', 'powershell.exe') - var csFile = path.join(__dirname, 'Find-VisualStudio.cs') - var psArgs = [ + const csFile = path.join(__dirname, 'Find-VisualStudio.cs') + const psArgs = [ '-ExecutionPolicy', 'Unrestricted', '-NoProfile', @@ -138,7 +137,7 @@ VisualStudioFinder.prototype = { ] this.log.silly('Running', ps, psArgs) - var child = execFile(ps, psArgs, { encoding: 'utf8' }, + const child = execFile(ps, psArgs, { encoding: 'utf8' }, (err, stdout, stderr) => { this.parseData(err, stdout, stderr, cb) }) @@ -161,7 +160,7 @@ VisualStudioFinder.prototype = { return failPowershell() } - var vsInfo + let vsInfo try { vsInfo = JSON.parse(stdout) } catch (e) { @@ -178,7 +177,7 @@ VisualStudioFinder.prototype = { vsInfo = vsInfo.map((info) => { this.log.silly(`processing installation: "${info.path}"`) info.path = path.resolve(info.path) - var ret = this.getVersionInfo(info) + const ret = this.getVersionInfo(info) ret.path = info.path ret.msBuild = this.getMSBuild(info, ret.versionYear) ret.toolset = this.getToolset(info, ret.versionYear) @@ -199,7 +198,7 @@ VisualStudioFinder.prototype = { // Sort to place newer versions first vsInfo.sort((a, b) => b.versionYear - a.versionYear) - for (var i = 0; i < vsInfo.length; ++i) { + for (let i = 0; i < vsInfo.length; ++i) { const info = vsInfo[i] this.addLog(`checking VS${info.versionYear} (${info.version}) found ` + `at:\n"${info.path}"`) @@ -245,7 +244,7 @@ VisualStudioFinder.prototype = { return {} } this.log.silly('- version match = %j', match) - var ret = { + const ret = { version: info.version, versionMajor: parseInt(match[1], 10), versionMinor: parseInt(match[2], 10) @@ -327,7 +326,7 @@ VisualStudioFinder.prototype = { const win10SDKPrefix = 'Microsoft.VisualStudio.Component.Windows10SDK.' const win11SDKPrefix = 'Microsoft.VisualStudio.Component.Windows11SDK.' - var Win10or11SDKVer = 0 + let Win10or11SDKVer = 0 info.packages.forEach((pkg) => { if (!pkg.startsWith(win10SDKPrefix) && !pkg.startsWith(win11SDKPrefix)) { return @@ -458,6 +457,6 @@ VisualStudioFinder.prototype = { module.exports = findVisualStudio module.exports.test = { - VisualStudioFinder: VisualStudioFinder, - findVisualStudio: findVisualStudio + VisualStudioFinder, + findVisualStudio } diff --git a/lib/install.js b/lib/install.js index 1eb9f14c67..7ac34d2b83 100644 --- a/lib/install.js +++ b/lib/install.js @@ -9,7 +9,7 @@ const path = require('path') const util = require('util') const stream = require('stream') const crypto = require('crypto') -const log = require('npmlog') +const log = require('./log') const semver = require('semver') const fetch = require('make-fetch-happen') const processRelease = require('./process-release') diff --git a/lib/list.js b/lib/list.js index 405ebc0d88..d93370018f 100644 --- a/lib/list.js +++ b/lib/list.js @@ -1,10 +1,10 @@ 'use strict' const fs = require('graceful-fs') -const log = require('npmlog') +const log = require('./log') function list (gyp, args, callback) { - var devDir = gyp.devDir + const devDir = gyp.devDir log.verbose('list', 'using node-gyp dir:', devDir) fs.readdir(devDir, onreaddir) diff --git a/lib/log.js b/lib/log.js new file mode 100644 index 0000000000..47fa8a3372 --- /dev/null +++ b/lib/log.js @@ -0,0 +1,165 @@ +'use strict' + +const procLog = require('proc-log') +const { format } = require('util') + +// helper to emit log messages with a predefined prefix +const logLevels = Object.keys(procLog).filter((k) => typeof procLog[k] === 'function') +const withPrefix = (prefix) => logLevels.reduce((acc, level) => { + acc[level] = (...args) => procLog[level](prefix, ...args) + return acc +}, {}) + +// very basic ansi color generator +const COLORS = { + wrap: (str, colors) => { + const codes = colors.filter(c => typeof c === 'number') + return `\x1b[${codes.join(';')}m${str}\x1b[0m` + }, + inverse: 7, + fg: { + black: 30, + red: 31, + green: 32, + yellow: 33, + blue: 34, + magenta: 35, + cyan: 36, + white: 37 + }, + bg: { + black: 40, + red: 41, + green: 42, + yellow: 43, + blue: 44, + magenta: 45, + cyan: 46, + white: 47 + } +} + +class Logger { + #buffer = [] + #paused = null + #level = null + #stream = null + + // ordered from loudest to quietest + #levels = [{ + id: 'silly', + display: 'sill', + style: { inverse: true } + }, { + id: 'verbose', + display: 'verb', + style: { fg: 'cyan', bg: 'black' } + }, { + id: 'info', + style: { fg: 'green' } + }, { + id: 'http', + style: { fg: 'green', bg: 'black' } + }, { + id: 'notice', + style: { fg: 'cyan', bg: 'black' } + }, { + id: 'warn', + display: 'WARN', + style: { fg: 'black', bg: 'yellow' } + }, { + id: 'error', + display: 'ERR!', + style: { fg: 'red', bg: 'black' } + }] + + constructor () { + process.on('log', (...args) => this.#onLog(...args)) + this.#levels = new Map(this.#levels.map((level, index) => [level.id, { ...level, index }])) + this.level = 'info' + this.stream = process.stderr + procLog.pause() + } + + get stream () { + return this.#stream + } + + set stream (stream) { + this.#stream = stream + } + + get level () { + return this.#levels.get(this.#level) ?? null + } + + set level (level) { + this.#level = this.#levels.get(level)?.id ?? null + } + + isVisible (level) { + return this.level?.index <= this.#levels.get(level)?.index ?? -1 + } + + #onLog (...args) { + const [level] = args + + if (level === 'pause') { + this.#paused = true + return + } + + if (level === 'resume') { + this.#paused = false + this.#buffer.forEach((b) => this.#log(...b)) + this.#buffer.length = 0 + return + } + + if (this.#paused) { + this.#buffer.push(args) + return + } + + this.#log(...args) + } + + #color (str, { fg, bg, inverse }) { + if (!this.#stream?.isTTY) { + return str + } + + return COLORS.wrap(str, [ + COLORS.fg[fg], + COLORS.bg[bg], + inverse && COLORS.inverse + ]) + } + + #log (levelId, msgPrefix, ...args) { + if (!this.isVisible(levelId) || typeof this.#stream?.write !== 'function') { + return + } + + const level = this.#levels.get(levelId) + + const prefixParts = [ + this.#color('gyp', { fg: 'white', bg: 'black' }), + this.#color(level.display ?? level.id, level.style) + ] + if (msgPrefix) { + prefixParts.push(this.#color(msgPrefix, { fg: 'magenta' })) + } + + const prefix = prefixParts.join(' ').trim() + ' ' + const lines = format(...args).split(/\r?\n/).map(l => prefix + l.trim()) + + this.#stream.write(lines.join('\n') + '\n') + } +} + +module.exports = { + logger: new Logger(), + withPrefix, + ...procLog +} diff --git a/lib/node-gyp.js b/lib/node-gyp.js index e492ec1026..21d12460c8 100644 --- a/lib/node-gyp.js +++ b/lib/node-gyp.js @@ -2,7 +2,7 @@ const path = require('path') const nopt = require('nopt') -const log = require('npmlog') +const log = require('./log') const childProcess = require('child_process') const EE = require('events').EventEmitter const inherits = require('util').inherits @@ -22,15 +22,12 @@ const aliases = { rm: 'remove' } -// differentiate node-gyp's logs from npm's -log.heading = 'gyp' - function gyp () { return new Gyp() } function Gyp () { - var self = this + const self = this this.devDir = '' this.commands = {} @@ -44,7 +41,7 @@ function Gyp () { } inherits(Gyp, EE) exports.Gyp = Gyp -var proto = Gyp.prototype +const proto = Gyp.prototype /** * Export the contents of the package.json. @@ -108,7 +105,7 @@ proto.parseArgv = function parseOpts (argv) { this.opts = nopt(this.configDefs, this.shorthands, argv) this.argv = this.opts.argv.remain.slice() - var commands = this.todo = [] + const commands = this.todo = [] // create a copy of the argv array with aliases mapped argv = this.argv.map(function (arg) { @@ -122,7 +119,7 @@ proto.parseArgv = function parseOpts (argv) { // process the mapped args into "command" objects ("name" and "args" props) argv.slice().forEach(function (arg) { if (arg in this.commands) { - var args = argv.splice(0, argv.indexOf(arg)) + const args = argv.splice(0, argv.indexOf(arg)) argv.shift() if (commands.length > 0) { commands[commands.length - 1].args = args @@ -135,14 +132,14 @@ proto.parseArgv = function parseOpts (argv) { } // support for inheriting config env variables from npm - var npmConfigPrefix = 'npm_config_' + const npmConfigPrefix = 'npm_config_' Object.keys(process.env).forEach(function (name) { if (name.indexOf(npmConfigPrefix) !== 0) { return } - var val = process.env[name] + const val = process.env[name] if (name === npmConfigPrefix + 'loglevel') { - log.level = val + log.logger.level = val } else { // add the user-defined options to the config name = name.substring(npmConfigPrefix.length) @@ -159,7 +156,7 @@ proto.parseArgv = function parseOpts (argv) { }, this) if (this.opts.loglevel) { - log.level = this.opts.loglevel + log.logger.level = this.opts.loglevel } log.resume() } @@ -175,7 +172,7 @@ proto.spawn = function spawn (command, args, opts) { if (!opts.silent && !opts.stdio) { opts.stdio = [0, 1, 2] } - var cp = childProcess.spawn(command, args, opts) + const cp = childProcess.spawn(command, args, opts) log.info('spawn', command) log.info('spawn args', args) return cp @@ -186,7 +183,7 @@ proto.spawn = function spawn (command, args, opts) { */ proto.usage = function usage () { - var str = [ + const str = [ '', ' Usage: node-gyp [options]', '', diff --git a/lib/process-release.js b/lib/process-release.js index 95b55e4426..c9a319dfad 100644 --- a/lib/process-release.js +++ b/lib/process-release.js @@ -1,11 +1,11 @@ -/* eslint-disable node/no-deprecated-api */ +/* eslint-disable n/no-deprecated-api */ 'use strict' const semver = require('semver') const url = require('url') const path = require('path') -const log = require('npmlog') +const log = require('./log') // versions where -headers.tar.gz started shipping const headersTarballRange = '>= 3.0.0 || ~0.12.10 || ~0.10.42' @@ -17,29 +17,28 @@ const bitsreV3 = /\/win-(x86|ia32|x64)\// // io.js v3.x.x shipped with "ia32" bu // file names. Inputs come from command-line switches (--target, --dist-url), // `process.version` and `process.release` where it exists. function processRelease (argv, gyp, defaultVersion, defaultRelease) { - var version = (semver.valid(argv[0]) && argv[0]) || gyp.opts.target || defaultVersion - var versionSemver = semver.parse(version) - var overrideDistUrl = gyp.opts['dist-url'] || gyp.opts.disturl - var isDefaultVersion - var isNamedForLegacyIojs - var name - var distBaseUrl - var baseUrl - var libUrl32 - var libUrl64 - var libUrlArm64 - var tarballUrl - var canGetHeaders + let version = (semver.valid(argv[0]) && argv[0]) || gyp.opts.target || defaultVersion + const versionSemver = semver.parse(version) + let overrideDistUrl = gyp.opts['dist-url'] || gyp.opts.disturl + let isNamedForLegacyIojs + let name + let distBaseUrl + let baseUrl + let libUrl32 + let libUrl64 + let libUrlArm64 + let tarballUrl + let canGetHeaders if (!versionSemver) { // not a valid semver string, nothing we can do - return { version: version } + return { version } } // flatten version into String version = versionSemver.version // defaultVersion should come from process.version so ought to be valid semver - isDefaultVersion = version === semver.parse(defaultVersion).version + const isDefaultVersion = version === semver.parse(defaultVersion).version // can't use process.release if we're using --target=x.y.z if (!isDefaultVersion) { @@ -101,11 +100,11 @@ function processRelease (argv, gyp, defaultVersion, defaultRelease) { } return { - version: version, + version, semver: versionSemver, - name: name, - baseUrl: baseUrl, - tarballUrl: tarballUrl, + name, + baseUrl, + tarballUrl, shasumsUrl: url.resolve(baseUrl, 'SHASUMS256.txt'), versionDir: (name !== 'node' ? name + '-' : '') + version, ia32: { @@ -128,8 +127,8 @@ function normalizePath (p) { } function resolveLibUrl (name, defaultUrl, arch, versionMajor) { - var base = url.resolve(defaultUrl, './') - var hasLibUrl = bitsre.test(defaultUrl) || (versionMajor === 3 && bitsreV3.test(defaultUrl)) + const base = url.resolve(defaultUrl, './') + const hasLibUrl = bitsre.test(defaultUrl) || (versionMajor === 3 && bitsreV3.test(defaultUrl)) if (!hasLibUrl) { // let's assume it's a baseUrl then diff --git a/lib/remove.js b/lib/remove.js index 8c945e5659..9561cb95e5 100644 --- a/lib/remove.js +++ b/lib/remove.js @@ -1,46 +1,45 @@ 'use strict' -const fs = require('fs') -const rm = require('rimraf') +const fs = require('fs/promises') const path = require('path') -const log = require('npmlog') +const log = require('./log') const semver = require('semver') -function remove (gyp, argv, callback) { - var devDir = gyp.devDir +async function remove (gyp, argv) { + const devDir = gyp.devDir log.verbose('remove', 'using node-gyp dir:', devDir) // get the user-specified version to remove - var version = argv[0] || gyp.opts.target + let version = argv[0] || gyp.opts.target log.verbose('remove', 'removing target version:', version) if (!version) { - return callback(new Error('You must specify a version number to remove. Ex: "' + process.version + '"')) + throw new Error('You must specify a version number to remove. Ex: "' + process.version + '"') } - var versionSemver = semver.parse(version) + const versionSemver = semver.parse(version) if (versionSemver) { // flatten the version Array into a String version = versionSemver.version } - var versionPath = path.resolve(gyp.devDir, version) + const versionPath = path.resolve(gyp.devDir, version) log.verbose('remove', 'removing development files for version:', version) // first check if its even installed - fs.stat(versionPath, function (err) { - if (err) { - if (err.code === 'ENOENT') { - callback(null, 'version was already uninstalled: ' + version) - } else { - callback(err) - } - return + try { + await fs.stat(versionPath) + } catch (err) { + if (err.code === 'ENOENT') { + return 'version was already uninstalled: ' + version } - // Go ahead and delete the dir - rm(versionPath, callback) - }) + throw err + } + + await fs.rm(versionPath, { recursive: true, force: true }) } -module.exports = exports = remove +module.exports = function (gyp, argv, callback) { + remove(gyp, argv).then(callback.bind(undefined, null), callback) +} module.exports.usage = 'Removes the node development files for the specified version' diff --git a/lib/util.js b/lib/util.js index 3e23c628e6..5950aa25b4 100644 --- a/lib/util.js +++ b/lib/util.js @@ -1,22 +1,9 @@ 'use strict' -const log = require('npmlog') +const log = require('./log') const execFile = require('child_process').execFile const path = require('path') -function logWithPrefix (log, prefix) { - function setPrefix (logFunction) { - return (...args) => logFunction.apply(null, [ prefix, ...args ]) // eslint-disable-line - } - return { - silly: setPrefix(log.silly), - verbose: setPrefix(log.verbose), - info: setPrefix(log.info), - warn: setPrefix(log.warn), - error: setPrefix(log.error) - } -} - function regGetValue (key, value, addOpts, cb) { const outReValue = value.replace(/\W/g, '.') const outRe = new RegExp(`^\\s+${outReValue}\\s+REG_\\w+\\s+(\\S.*)$`, 'im') @@ -45,7 +32,7 @@ function regGetValue (key, value, addOpts, cb) { } function regSearchKeys (keys, value, addOpts, cb) { - var i = 0 + let i = 0 const search = () => { log.silly('reg-search', 'looking for %j in %j', value, keys[i]) regGetValue(keys[i], value, addOpts, (err, res) => { @@ -58,7 +45,6 @@ function regSearchKeys (keys, value, addOpts, cb) { } module.exports = { - logWithPrefix: logWithPrefix, - regGetValue: regGetValue, - regSearchKeys: regSearchKeys + regGetValue, + regSearchKeys } diff --git a/package.json b/package.json index 7e9fb648ab..31c8a25d79 100644 --- a/package.json +++ b/package.json @@ -24,25 +24,24 @@ "dependencies": { "env-paths": "^2.2.0", "exponential-backoff": "^3.1.1", - "glob": "^7.1.4", + "glob": "^8.0.3", "graceful-fs": "^4.2.6", "make-fetch-happen": "^11.0.3", - "nopt": "^6.0.0", - "npmlog": "^6.0.0", - "rimraf": "^3.0.2", + "nopt": "^7.0.0", + "proc-log": "^3.0.0", "semver": "^7.3.5", "tar": "^6.1.2", - "which": "^2.0.2" + "which": "^3.0.0" }, "engines": { - "node": "^12.13 || ^14.13 || >=16" + "node": "^14.17.0 || ^16.13.0 || >=18.0.0" }, "devDependencies": { "bindings": "^1.5.0", "mocha": "^10.2.0", "nan": "^2.14.2", "require-inject": "^1.4.4", - "standard": "^14.3.4" + "standard": "^17.0.0" }, "scripts": { "lint": "standard */*.js test/**/*.js", diff --git a/test/process-exec-sync.js b/test/process-exec-sync.js index 21763bc26d..0a9002cc80 100644 --- a/test/process-exec-sync.js +++ b/test/process-exec-sync.js @@ -12,7 +12,7 @@ function startsWith (str, search, pos) { } function processExecSync (file, args, options) { - var child, error, timeout, tmpdir, command + let error, command command = makeCommand(file, args) /* @@ -22,10 +22,10 @@ function processExecSync (file, args, options) { options = options || {} // init timeout - timeout = Date.now() + options.timeout + const timeout = Date.now() + options.timeout // init tmpdir - var osTempBase = '/tmp' - var os = determineOS() + let osTempBase = '/tmp' + const os = determineOS() osTempBase = '/tmp' if (process.env.TMP) { @@ -36,7 +36,7 @@ function processExecSync (file, args, options) { osTempBase += '/' } - tmpdir = osTempBase + 'processExecSync.' + Date.now() + Math.random() + const tmpdir = osTempBase + 'processExecSync.' + Date.now() + Math.random() fs.mkdirSync(tmpdir) // init command @@ -49,13 +49,13 @@ function processExecSync (file, args, options) { } // init child - child = childProcess.exec(command, options) + const child = childProcess.exec(command, options) - var maxTry = 100000 // increases the test time by 6 seconds on win-2016-node-0.10 - var tryCount = 0 + const maxTry = 100000 // increases the test time by 6 seconds on win-2016-node-0.10 + let tryCount = 0 while (tryCount < maxTry) { try { - var x = fs.readFileSync(tmpdir + '/status') + const x = fs.readFileSync(tmpdir + '/status') if (x.toString() === '0') { break } @@ -87,10 +87,10 @@ function processExecSync (file, args, options) { } function makeCommand (file, args) { - var command, quote + let command, quote command = file if (args.length > 0) { - for (var i in args) { + for (const i in args) { command = command + ' ' if (args[i][0] === '-') { command = command + args[i] @@ -112,8 +112,8 @@ function makeCommand (file, args) { } function determineOS () { - var os = '' - var tmpVar = '' + let os = '' + let tmpVar = '' if (process.env.OSTYPE) { tmpVar = process.env.OSTYPE } else if (process.env.OS) { diff --git a/test/simple-proxy.js b/test/simple-proxy.js index cb0dfcfec7..4f8c4137ef 100644 --- a/test/simple-proxy.js +++ b/test/simple-proxy.js @@ -6,7 +6,7 @@ const server = http.createServer(handler) const port = +process.argv[2] const prefix = process.argv[3] const upstream = process.argv[4] -var calls = 0 +let calls = 0 server.listen(port) @@ -15,7 +15,7 @@ function handler (req, res) { throw new Error('request url [' + req.url + '] does not start with [' + prefix + ']') } - var upstreamUrl = upstream + req.url.substring(prefix.length) + const upstreamUrl = upstream + req.url.substring(prefix.length) https.get(upstreamUrl, function (ures) { ures.on('end', function () { if (++calls === 2) { diff --git a/test/test-addon.js b/test/test-addon.js index 43556620a8..373647dc4c 100644 --- a/test/test-addon.js +++ b/test/test-addon.js @@ -15,24 +15,24 @@ function runHello (hostProcess) { if (!hostProcess) { hostProcess = process.execPath } - var testCode = "console.log(require('hello_world').hello())" + const testCode = "console.log(require('hello_world').hello())" return execFileSync(hostProcess, ['-e', testCode], { cwd: __dirname }).toString() } function getEncoding () { - var code = 'import locale;print(locale.getdefaultlocale()[1])' + const code = 'import locale;print(locale.getdefaultlocale()[1])' return execFileSync('python', ['-c', code]).toString().trim() } function checkCharmapValid () { - var data + let data try { data = execFileSync('python', ['fixtures/test-charmap.py'], { cwd: __dirname }) } catch (err) { return false } - var lines = data.toString().trim().split('\n') + const lines = data.toString().trim().split('\n') return lines.pop() === 'True' } @@ -41,10 +41,10 @@ describe('addon', function () { it('build simple addon', function (done) { // Set the loglevel otherwise the output disappears when run via 'npm test' - var cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose'] - var proc = execFile(process.execPath, cmd, function (err, stdout, stderr) { - var logLines = stderr.toString().trim().split(/\r?\n/) - var lastLine = logLines[logLines.length - 1] + const cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose'] + const proc = execFile(process.execPath, cmd, function (err, stdout, stderr) { + const logLines = stderr.toString().trim().split(/\r?\n/) + const lastLine = logLines[logLines.length - 1] assert.strictEqual(err, null) assert.strictEqual(lastLine, 'gyp info ok', 'should end in ok') assert.strictEqual(runHello().trim(), 'world') @@ -59,13 +59,13 @@ describe('addon', function () { return this.skip('python console app can\'t encode non-ascii character.') } - var testDirNames = { + const testDirNames = { cp936: '文件夹', cp1252: 'Latīna', cp932: 'フォルダ' } // Select non-ascii characters by current encoding - var testDirName = testDirNames[getEncoding()] + const testDirName = testDirNames[getEncoding()] // If encoding is UTF-8 or other then no need to test if (!testDirName) { return this.skip('no need to test') @@ -73,17 +73,17 @@ describe('addon', function () { this.timeout(300000) - var data - var configPath = path.join(addonPath, 'build', 'config.gypi') + let data + const configPath = path.join(addonPath, 'build', 'config.gypi') try { data = fs.readFileSync(configPath, 'utf8') } catch (err) { assert.fail(err) return } - var config = JSON.parse(data.replace(/#.+\n/, '')) - var nodeDir = config.variables.nodedir - var testNodeDir = path.join(addonPath, testDirName) + const config = JSON.parse(data.replace(/#.+\n/, '')) + const nodeDir = config.variables.nodedir + const testNodeDir = path.join(addonPath, testDirName) // Create symbol link to path with non-ascii characters try { fs.symlinkSync(nodeDir, testNodeDir, 'dir') @@ -99,7 +99,7 @@ describe('addon', function () { } } - var cmd = [ + const cmd = [ nodeGyp, 'rebuild', '-C', @@ -107,15 +107,15 @@ describe('addon', function () { '--loglevel=verbose', '-nodedir=' + testNodeDir ] - var proc = execFile(process.execPath, cmd, function (err, stdout, stderr) { + const proc = execFile(process.execPath, cmd, function (err, stdout, stderr) { try { fs.unlink(testNodeDir) } catch (err) { assert.fail(err) } - var logLines = stderr.toString().trim().split(/\r?\n/) - var lastLine = logLines[logLines.length - 1] + const logLines = stderr.toString().trim().split(/\r?\n/) + const lastLine = logLines[logLines.length - 1] assert.strictEqual(err, null) assert.strictEqual(lastLine, 'gyp info ok', 'should end in ok') assert.strictEqual(runHello().trim(), 'world') @@ -133,13 +133,13 @@ describe('addon', function () { this.timeout(300000) - var notNodePath = path.join(os.tmpdir(), 'notnode' + path.extname(process.execPath)) + const notNodePath = path.join(os.tmpdir(), 'notnode' + path.extname(process.execPath)) fs.copyFileSync(process.execPath, notNodePath) - var cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose'] - var proc = execFile(process.execPath, cmd, function (err, stdout, stderr) { - var logLines = stderr.toString().trim().split(/\r?\n/) - var lastLine = logLines[logLines.length - 1] + const cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose'] + const proc = execFile(process.execPath, cmd, function (err, stdout, stderr) { + const logLines = stderr.toString().trim().split(/\r?\n/) + const lastLine = logLines[logLines.length - 1] assert.strictEqual(err, null) assert.strictEqual(lastLine, 'gyp info ok', 'should end in ok') assert.strictEqual(runHello(notNodePath).trim(), 'world') diff --git a/test/test-configure-python.js b/test/test-configure-python.js index ab1e5511fa..9c042b18d5 100644 --- a/test/test-configure-python.js +++ b/test/test-configure-python.js @@ -5,6 +5,7 @@ const assert = require('assert') const path = require('path') const devDir = require('./common').devDir() const gyp = require('../lib/node-gyp') +const log = require('../lib/log') const requireInject = require('require-inject') const configure = requireInject('../lib/configure', { 'graceful-fs': { @@ -21,17 +22,17 @@ const configure = requireInject('../lib/configure', { } }) +log.logger.stream = null + const EXPECTED_PYPATH = path.join(__dirname, '..', 'gyp', 'pylib') const SEPARATOR = process.platform === 'win32' ? ';' : ':' const SPAWN_RESULT = cb => ({ on: function () { cb() } }) -require('npmlog').level = 'warn' - describe('configure-python', function () { it('configure PYTHONPATH with no existing env', function (done) { delete process.env.PYTHONPATH - var prog = gyp() + const prog = gyp() prog.parseArgv([]) prog.spawn = function () { assert.strictEqual(process.env.PYTHONPATH, EXPECTED_PYPATH) @@ -42,15 +43,15 @@ describe('configure-python', function () { }) it('configure PYTHONPATH with existing env of one dir', function (done) { - var existingPath = path.join('a', 'b') + const existingPath = path.join('a', 'b') process.env.PYTHONPATH = existingPath - var prog = gyp() + const prog = gyp() prog.parseArgv([]) prog.spawn = function () { assert.strictEqual(process.env.PYTHONPATH, [EXPECTED_PYPATH, existingPath].join(SEPARATOR)) - var dirs = process.env.PYTHONPATH.split(SEPARATOR) + const dirs = process.env.PYTHONPATH.split(SEPARATOR) assert.deepStrictEqual(dirs, [EXPECTED_PYPATH, existingPath]) return SPAWN_RESULT(done) @@ -60,17 +61,17 @@ describe('configure-python', function () { }) it('configure PYTHONPATH with existing env of multiple dirs', function (done) { - var pythonDir1 = path.join('a', 'b') - var pythonDir2 = path.join('b', 'c') - var existingPath = [pythonDir1, pythonDir2].join(SEPARATOR) + const pythonDir1 = path.join('a', 'b') + const pythonDir2 = path.join('b', 'c') + const existingPath = [pythonDir1, pythonDir2].join(SEPARATOR) process.env.PYTHONPATH = existingPath - var prog = gyp() + const prog = gyp() prog.parseArgv([]) prog.spawn = function () { assert.strictEqual(process.env.PYTHONPATH, [EXPECTED_PYPATH, existingPath].join(SEPARATOR)) - var dirs = process.env.PYTHONPATH.split(SEPARATOR) + const dirs = process.env.PYTHONPATH.split(SEPARATOR) assert.deepStrictEqual(dirs, [EXPECTED_PYPATH, pythonDir1, pythonDir2]) return SPAWN_RESULT(done) diff --git a/test/test-download.js b/test/test-download.js index 1dd5a51b06..0a13df520b 100644 --- a/test/test-download.js +++ b/test/test-download.js @@ -2,7 +2,7 @@ const { describe, it, after } = require('mocha') const assert = require('assert') -const fs = require('fs') +const fs = require('fs/promises') const path = require('path') const util = require('util') const http = require('http') @@ -10,12 +10,11 @@ const https = require('https') const install = require('../lib/install') const semver = require('semver') const devDir = require('./common').devDir() -const rimraf = require('rimraf') const gyp = require('../lib/node-gyp') -const log = require('npmlog') +const log = require('../lib/log') const certs = require('./fixtures/certs') -log.level = 'warn' +log.logger.stream = null describe('download', function () { it('download over http', async function () { @@ -43,12 +42,12 @@ describe('download', function () { const cacontents = certs['ca.crt'] const cert = certs['server.crt'] const key = certs['server.key'] - await fs.promises.writeFile(cafile, cacontents, 'utf8') + await fs.writeFile(cafile, cacontents, 'utf8') const ca = await install.test.readCAFile(cafile) assert.strictEqual(ca.length, 1) - const options = { ca: ca, cert: cert, key: key } + const options = { ca, cert, key } const server = https.createServer(options, (req, res) => { assert.strictEqual(req.headers['user-agent'], `node-gyp v42 (node ${process.version})`) res.end('ok') @@ -56,7 +55,7 @@ describe('download', function () { after(async () => { await new Promise((resolve) => server.close(resolve)) - await fs.promises.unlink(cafile) + await fs.unlink(cafile) }) server.on('clientError', (err) => { throw err }) @@ -149,9 +148,9 @@ describe('download', function () { it('check certificate splitting', async function () { const cafile = path.join(__dirname, 'fixtures/ca-bundle.crt') const cacontents = certs['ca-bundle.crt'] - await fs.promises.writeFile(cafile, cacontents, 'utf8') + await fs.writeFile(cafile, cacontents, 'utf8') after(async () => { - await fs.promises.unlink(cafile) + await fs.unlink(cafile) }) const cas = await install.test.readCAFile(path.join(__dirname, 'fixtures/ca-bundle.crt')) assert.strictEqual(cas.length, 2) @@ -171,7 +170,7 @@ describe('download', function () { this.timeout(300000) const expectedDir = path.join(devDir, process.version.replace(/^v/, '')) - await util.promisify(rimraf)(expectedDir) + await fs.rm(expectedDir, { recursive: true, force: true }) const prog = gyp() prog.parseArgv([]) @@ -179,10 +178,10 @@ describe('download', function () { log.level = 'warn' await util.promisify(install)(prog, []) - const data = await fs.promises.readFile(path.join(expectedDir, 'installVersion'), 'utf8') + const data = await fs.readFile(path.join(expectedDir, 'installVersion'), 'utf8') assert.strictEqual(data, '11\n', 'correct installVersion') - const list = await fs.promises.readdir(path.join(expectedDir, 'include/node')) + const list = await fs.readdir(path.join(expectedDir, 'include/node')) assert.ok(list.includes('common.gypi')) assert.ok(list.includes('config.gypi')) assert.ok(list.includes('node.h')) @@ -194,7 +193,7 @@ describe('download', function () { assert.ok(list.includes('v8.h')) assert.ok(list.includes('zlib.h')) - const lines = (await fs.promises.readFile(path.join(expectedDir, 'include/node/node_version.h'), 'utf8')).split('\n') + const lines = (await fs.readFile(path.join(expectedDir, 'include/node/node_version.h'), 'utf8')).split('\n') // extract the 3 version parts from the defines to build a valid version string and // and check them against our current env version diff --git a/test/test-find-accessible-sync.js b/test/test-find-accessible-sync.js index 7edbc0c764..9ec12dbb33 100644 --- a/test/test-find-accessible-sync.js +++ b/test/test-find-accessible-sync.js @@ -11,7 +11,7 @@ const configure = requireInject('../lib/configure', { if (readableFiles.some(function (f) { return f === path })) { return 0 } else { - var error = new Error('ENOENT - not found') + const error = new Error('ENOENT - not found') throw error } } @@ -30,44 +30,44 @@ const readableFiles = [ describe('find-accessible-sync', function () { it('find accessible - empty array', function () { - var candidates = [] - var found = configure.test.findAccessibleSync('test', dir, candidates) + const candidates = [] + const found = configure.test.findAccessibleSync('test', dir, candidates) assert.strictEqual(found, undefined) }) it('find accessible - single item array, readable', function () { - var candidates = [readableFile] - var found = configure.test.findAccessibleSync('test', dir, candidates) + const candidates = [readableFile] + const found = configure.test.findAccessibleSync('test', dir, candidates) assert.strictEqual(found, path.resolve(dir, readableFile)) }) it('find accessible - single item array, readable in subdir', function () { - var candidates = [readableFileInDir] - var found = configure.test.findAccessibleSync('test', dir, candidates) + const candidates = [readableFileInDir] + const found = configure.test.findAccessibleSync('test', dir, candidates) assert.strictEqual(found, path.resolve(dir, readableFileInDir)) }) it('find accessible - single item array, unreadable', function () { - var candidates = ['unreadable_file'] - var found = configure.test.findAccessibleSync('test', dir, candidates) + const candidates = ['unreadable_file'] + const found = configure.test.findAccessibleSync('test', dir, candidates) assert.strictEqual(found, undefined) }) it('find accessible - multi item array, no matches', function () { - var candidates = ['non_existent_file', 'unreadable_file'] - var found = configure.test.findAccessibleSync('test', dir, candidates) + const candidates = ['non_existent_file', 'unreadable_file'] + const found = configure.test.findAccessibleSync('test', dir, candidates) assert.strictEqual(found, undefined) }) it('find accessible - multi item array, single match', function () { - var candidates = ['non_existent_file', readableFile] - var found = configure.test.findAccessibleSync('test', dir, candidates) + const candidates = ['non_existent_file', readableFile] + const found = configure.test.findAccessibleSync('test', dir, candidates) assert.strictEqual(found, path.resolve(dir, readableFile)) }) it('find accessible - multi item array, return first match', function () { - var candidates = ['non_existent_file', anotherReadableFile, readableFile] - var found = configure.test.findAccessibleSync('test', dir, candidates) + const candidates = ['non_existent_file', anotherReadableFile, readableFile] + const found = configure.test.findAccessibleSync('test', dir, candidates) assert.strictEqual(found, path.resolve(dir, anotherReadableFile)) }) }) diff --git a/test/test-find-node-directory.js b/test/test-find-node-directory.js index ca299f6330..e36a5c04ea 100644 --- a/test/test-find-node-directory.js +++ b/test/test-find-node-directory.js @@ -13,8 +13,8 @@ describe('find-node-directory', function () { // in a build tree where npm is installed in // .... /deps/npm it('test find-node-directory - node install', function () { - for (var next = 0; next < platforms.length; next++) { - var processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } + for (let next = 0; next < platforms.length; next++) { + const processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } assert.strictEqual( findNodeDirectory('/x/deps/npm/node_modules/node-gyp/lib', processObj), path.join('/x')) @@ -27,8 +27,8 @@ describe('find-node-directory', function () { // .... /lib/node_modules/npm or .../node_modules/npm // depending on the patform it('test find-node-directory - node build', function () { - for (var next = 0; next < platforms.length; next++) { - var processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } + for (let next = 0; next < platforms.length; next++) { + const processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } if (platforms[next] === 'win32') { assert.strictEqual( findNodeDirectory('/y/node_modules/npm/node_modules/node-gyp/lib', @@ -44,8 +44,8 @@ describe('find-node-directory', function () { // we should find the directory based on the execPath // for node and match because it was in the bin directory it('test find-node-directory - node in bin directory', function () { - for (var next = 0; next < platforms.length; next++) { - var processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } + for (let next = 0; next < platforms.length; next++) { + const processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } assert.strictEqual( findNodeDirectory('/nothere/npm/node_modules/node-gyp/lib', processObj), path.join('/x/y')) @@ -55,8 +55,8 @@ describe('find-node-directory', function () { // we should find the directory based on the execPath // for node and match because it was in the Release directory it('test find-node-directory - node in build release dir', function () { - for (var next = 0; next < platforms.length; next++) { - var processObj + for (let next = 0; next < platforms.length; next++) { + let processObj if (platforms[next] === 'win32') { processObj = { execPath: '/x/y/Release/node', platform: platforms[next] } } else { @@ -75,8 +75,8 @@ describe('find-node-directory', function () { // we should find the directory based on the execPath // for node and match because it was in the Debug directory it('test find-node-directory - node in Debug release dir', function () { - for (var next = 0; next < platforms.length; next++) { - var processObj + for (let next = 0; next < platforms.length; next++) { + let processObj if (platforms[next] === 'win32') { processObj = { execPath: '/a/b/Debug/node', platform: platforms[next] } } else { @@ -92,8 +92,8 @@ describe('find-node-directory', function () { // we should not find it as it will not match based on the execPath nor // the directory from which the script is running it('test find-node-directory - not found', function () { - for (var next = 0; next < platforms.length; next++) { - var processObj = { execPath: '/x/y/z/y', platform: next } + for (let next = 0; next < platforms.length; next++) { + const processObj = { execPath: '/x/y/z/y', platform: next } assert.strictEqual(findNodeDirectory('/a/b/c/d', processObj), '') } }) @@ -105,8 +105,8 @@ describe('find-node-directory', function () { // same test as above but make sure additional directory entries // don't cause an issue it('test find-node-directory - node install', function () { - for (var next = 0; next < platforms.length; next++) { - var processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } + for (let next = 0; next < platforms.length; next++) { + const processObj = { execPath: '/x/y/bin/node', platform: platforms[next] } assert.strictEqual( findNodeDirectory('/x/y/z/a/b/c/deps/npm/node_modules/node-gyp/lib', processObj), path.join('/x/y/z/a/b/c')) diff --git a/test/test-find-python.js b/test/test-find-python.js index 592c480f24..e1880bea7a 100644 --- a/test/test-find-python.js +++ b/test/test-find-python.js @@ -8,13 +8,11 @@ const findPython = require('../lib/find-python') const execFile = require('child_process').execFile const PythonFinder = findPython.test.PythonFinder -require('npmlog').level = 'warn' - describe('find-python', function () { it('find python', function () { findPython.test.findPython(null, function (err, found) { assert.strictEqual(err, null) - var proc = execFile(found, ['-V'], function (err, stdout, stderr) { + const proc = execFile(found, ['-V'], function (err, stdout, stderr) { assert.strictEqual(err, null) assert.ok(/Python 3/.test(stdout)) assert.strictEqual(stderr, '') @@ -29,7 +27,7 @@ describe('find-python', function () { console.error(Error(`Property ${property} should not have been accessed.`)) process.abort() } - var descriptor = { + const descriptor = { configurable: false, enumerable: false, get: fail, @@ -42,18 +40,10 @@ describe('find-python', function () { PythonFinder.apply(this, arguments) } TestPythonFinder.prototype = Object.create(PythonFinder.prototype) - // Silence npmlog - remove for debugging - TestPythonFinder.prototype.log = { - silly: () => {}, - verbose: () => {}, - info: () => {}, - warn: () => {}, - error: () => {} - } delete TestPythonFinder.prototype.env.NODE_GYP_FORCE_PYTHON it('find python - python', function () { - var f = new TestPythonFinder('python', done) + const f = new TestPythonFinder('python', done) f.execFile = function (program, args, opts, cb) { f.execFile = function (program, args, opts, cb) { poison(f, 'execFile') @@ -75,7 +65,7 @@ describe('find-python', function () { }) it('find python - python too old', function () { - var f = new TestPythonFinder(null, done) + const f = new TestPythonFinder(null, done) f.execFile = function (program, args, opts, cb) { if (/sys\.executable/.test(args[args.length - 1])) { cb(null, '/path/python') @@ -94,7 +84,7 @@ describe('find-python', function () { }) it('find python - no python', function () { - var f = new TestPythonFinder(null, done) + const f = new TestPythonFinder(null, done) f.execFile = function (program, args, opts, cb) { if (/sys\.executable/.test(args[args.length - 1])) { cb(new Error('not found')) @@ -113,7 +103,7 @@ describe('find-python', function () { }) it('find python - no python2, no python, unix', function () { - var f = new TestPythonFinder(null, done) + const f = new TestPythonFinder(null, done) f.checkPyLauncher = assert.fail f.win = false @@ -133,7 +123,7 @@ describe('find-python', function () { }) it('find python - no python, use python launcher', function () { - var f = new TestPythonFinder(null, done) + const f = new TestPythonFinder(null, done) f.win = true f.execFile = function (program, args, opts, cb) { @@ -165,7 +155,7 @@ describe('find-python', function () { }) it('find python - no python, no python launcher, good guess', function () { - var f = new TestPythonFinder(null, done) + const f = new TestPythonFinder(null, done) f.win = true const expectedProgram = f.winDefaultLocations[0] @@ -191,7 +181,7 @@ describe('find-python', function () { }) it('find python - no python, no python launcher, bad guess', function () { - var f = new TestPythonFinder(null, done) + const f = new TestPythonFinder(null, done) f.win = true f.execFile = function (program, args, opts, cb) { diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js index 29d9a7dba5..def1fc81f7 100644 --- a/test/test-find-visualstudio.js +++ b/test/test-find-visualstudio.js @@ -16,7 +16,7 @@ function poison (object, property) { console.error(Error(`Property ${property} should not have been accessed.`)) process.abort() } - var descriptor = { + const descriptor = { configurable: false, enumerable: false, get: fail, @@ -27,14 +27,6 @@ function poison (object, property) { function TestVisualStudioFinder () { VisualStudioFinder.apply(this, arguments) } TestVisualStudioFinder.prototype = Object.create(VisualStudioFinder.prototype) -// Silence npmlog - remove for debugging -TestVisualStudioFinder.prototype.log = { - silly: () => {}, - verbose: () => {}, - info: () => {}, - warn: () => {}, - error: () => {} -} describe('find-visualstudio', function () { it('VS2013', function () { @@ -56,7 +48,7 @@ describe('find-visualstudio', function () { finder.parseData(new Error(), '', '', cb) } finder.regSearchKeys = (keys, value, addOpts, cb) => { - for (var i = 0; i < keys.length; ++i) { + for (let i = 0; i < keys.length; ++i) { const fullName = `${keys[i]}\\${value}` switch (fullName) { case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': @@ -93,7 +85,7 @@ describe('find-visualstudio', function () { finder.parseData(null, data, '', cb) } finder.regSearchKeys = (keys, value, addOpts, cb) => { - for (var i = 0; i < keys.length; ++i) { + for (let i = 0; i < keys.length; ++i) { const fullName = `${keys[i]}\\${value}` switch (fullName) { case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': @@ -127,7 +119,7 @@ describe('find-visualstudio', function () { finder.parseData(new Error(), '', '', cb) } finder.regSearchKeys = (keys, value, addOpts, cb) => { - for (var i = 0; i < keys.length; ++i) { + for (let i = 0; i < keys.length; ++i) { const fullName = `${keys[i]}\\${value}` switch (fullName) { case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': @@ -439,7 +431,7 @@ describe('find-visualstudio', function () { finder.parseData(null, data, '', cb) } finder.regSearchKeys = (keys, value, addOpts, cb) => { - for (var i = 0; i < keys.length; ++i) { + for (let i = 0; i < keys.length; ++i) { const fullName = `${keys[i]}\\${value}` switch (fullName) { case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0': diff --git a/test/test-install.js b/test/test-install.js index 235acf5231..af83ad9442 100644 --- a/test/test-install.js +++ b/test/test-install.js @@ -1,20 +1,17 @@ 'use strict' const { describe, it, after } = require('mocha') +const { rm } = require('fs/promises') const assert = require('assert') const path = require('path') const os = require('os') const util = require('util') const { test: { download, install } } = require('../lib/install') -const rimraf = require('rimraf') const gyp = require('../lib/node-gyp') -const log = require('npmlog') const semver = require('semver') const stream = require('stream') const streamPipeline = util.promisify(stream.pipeline) -log.level = 'error' // we expect a warning - describe('install', function () { it('EACCES retry once', async () => { const fs = { @@ -65,11 +62,11 @@ describe('install', function () { } after(async () => { - await util.promisify(rimraf)(devDir) + await rm(devDir, { recursive: true, force: true }) }) const expectedDir = path.join(devDir, process.version.replace(/^v/, '')) - await util.promisify(rimraf)(expectedDir) + await rm(expectedDir, { recursive: true, force: true }) await Promise.all([ install(fs, prog, []), @@ -95,7 +92,6 @@ describe('install', function () { prog.parseArgv([]) prog.devDir = devDir prog.opts.ensure = true - log.level = 'warn' await parallelInstallsTest(this, fs, devDir, prog) }) @@ -110,7 +106,6 @@ describe('install', function () { prog.parseArgv([]) prog.devDir = devDir prog.opts.ensure = false - log.level = 'warn' await parallelInstallsTest(this, fs, devDir, prog) }) @@ -125,7 +120,6 @@ describe('install', function () { prog.parseArgv([]) prog.devDir = devDir prog.opts.tarball = path.join(devDir, 'node-headers.tar.gz') - log.level = 'warn' await streamPipeline( (await download(prog, `https://nodejs.org/dist/${process.version}/node-${process.version}.tar.gz`)).body, diff --git a/test/test-process-release.js b/test/test-process-release.js index 0f40666473..31a87be9e9 100644 --- a/test/test-process-release.js +++ b/test/test-process-release.js @@ -6,7 +6,7 @@ const processRelease = require('../lib/process-release') describe('process-release', function () { it('test process release - process.version = 0.8.20', function () { - var release = processRelease([], { opts: {} }, 'v0.8.20', null) + const release = processRelease([], { opts: {} }, 'v0.8.20', null) assert.strictEqual(release.semver.version, '0.8.20') delete release.semver @@ -25,7 +25,7 @@ describe('process-release', function () { }) it('test process release - process.version = 0.10.21', function () { - var release = processRelease([], { opts: {} }, 'v0.10.21', null) + const release = processRelease([], { opts: {} }, 'v0.10.21', null) assert.strictEqual(release.semver.version, '0.10.21') delete release.semver @@ -45,7 +45,7 @@ describe('process-release', function () { // prior to -headers.tar.gz it('test process release - process.version = 0.12.9', function () { - var release = processRelease([], { opts: {} }, 'v0.12.9', null) + const release = processRelease([], { opts: {} }, 'v0.12.9', null) assert.strictEqual(release.semver.version, '0.12.9') delete release.semver @@ -65,7 +65,7 @@ describe('process-release', function () { // prior to -headers.tar.gz it('test process release - process.version = 0.10.41', function () { - var release = processRelease([], { opts: {} }, 'v0.10.41', null) + const release = processRelease([], { opts: {} }, 'v0.10.41', null) assert.strictEqual(release.semver.version, '0.10.41') delete release.semver @@ -85,7 +85,7 @@ describe('process-release', function () { // has -headers.tar.gz it('test process release - process.release ~ node@0.10.42', function () { - var release = processRelease([], { opts: {} }, 'v0.10.42', null) + const release = processRelease([], { opts: {} }, 'v0.10.42', null) assert.strictEqual(release.semver.version, '0.10.42') delete release.semver @@ -105,7 +105,7 @@ describe('process-release', function () { // has -headers.tar.gz it('test process release - process.release ~ node@0.12.10', function () { - var release = processRelease([], { opts: {} }, 'v0.12.10', null) + const release = processRelease([], { opts: {} }, 'v0.12.10', null) assert.strictEqual(release.semver.version, '0.12.10') delete release.semver @@ -124,7 +124,7 @@ describe('process-release', function () { }) it('test process release - process.release ~ node@4.1.23', function () { - var release = processRelease([], { opts: {} }, 'v4.1.23', { + const release = processRelease([], { opts: {} }, 'v4.1.23', { name: 'node', headersUrl: 'https://nodejs.org/dist/v4.1.23/node-v4.1.23-headers.tar.gz' }) @@ -146,7 +146,7 @@ describe('process-release', function () { }) it('test process release - process.release ~ node@4.1.23 / corp build', function () { - var release = processRelease([], { opts: {} }, 'v4.1.23', { + const release = processRelease([], { opts: {} }, 'v4.1.23', { name: 'node', headersUrl: 'https://some.custom.location/node-v4.1.23-headers.tar.gz' }) @@ -168,7 +168,7 @@ describe('process-release', function () { }) it('test process release - process.release ~ node@12.8.0 Windows', function () { - var release = processRelease([], { opts: {} }, 'v12.8.0', { + const release = processRelease([], { opts: {} }, 'v12.8.0', { name: 'node', sourceUrl: 'https://nodejs.org/download/release/v12.8.0/node-v12.8.0.tar.gz', headersUrl: 'https://nodejs.org/download/release/v12.8.0/node-v12.8.0-headers.tar.gz', @@ -192,7 +192,7 @@ describe('process-release', function () { }) it('test process release - process.release ~ node@12.8.0 Windows ARM64', function () { - var release = processRelease([], { opts: {} }, 'v12.8.0', { + const release = processRelease([], { opts: {} }, 'v12.8.0', { name: 'node', sourceUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/node-v12.8.0.tar.gz', headersUrl: 'https://unofficial-builds.nodejs.org/download/release/v12.8.0/node-v12.8.0-headers.tar.gz', @@ -216,7 +216,7 @@ describe('process-release', function () { }) it('test process release - process.release ~ node@4.1.23 --target=0.10.40', function () { - var release = processRelease([], { opts: { target: '0.10.40' } }, 'v4.1.23', { + const release = processRelease([], { opts: { target: '0.10.40' } }, 'v4.1.23', { name: 'node', headersUrl: 'https://nodejs.org/dist/v4.1.23/node-v4.1.23-headers.tar.gz' }) @@ -238,7 +238,7 @@ describe('process-release', function () { }) it('test process release - process.release ~ node@4.1.23 --dist-url=https://foo.bar/baz', function () { - var release = processRelease([], { opts: { 'dist-url': 'https://foo.bar/baz' } }, 'v4.1.23', { + const release = processRelease([], { opts: { 'dist-url': 'https://foo.bar/baz' } }, 'v4.1.23', { name: 'node', headersUrl: 'https://nodejs.org/dist/v4.1.23/node-v4.1.23-headers.tar.gz' }) @@ -260,7 +260,7 @@ describe('process-release', function () { }) it('test process release - process.release ~ frankenstein@4.1.23', function () { - var release = processRelease([], { opts: {} }, 'v4.1.23', { + const release = processRelease([], { opts: {} }, 'v4.1.23', { name: 'frankenstein', headersUrl: 'https://frankensteinjs.org/dist/v4.1.23/frankenstein-v4.1.23-headers.tar.gz' }) @@ -282,7 +282,7 @@ describe('process-release', function () { }) it('test process release - process.release ~ frankenstein@4.1.23 --dist-url=http://foo.bar/baz/', function () { - var release = processRelease([], { opts: { 'dist-url': 'http://foo.bar/baz/' } }, 'v4.1.23', { + const release = processRelease([], { opts: { 'dist-url': 'http://foo.bar/baz/' } }, 'v4.1.23', { name: 'frankenstein', headersUrl: 'https://frankensteinjs.org/dist/v4.1.23/frankenstein-v4.1.23.tar.gz' }) @@ -304,7 +304,7 @@ describe('process-release', function () { }) it('test process release - process.release ~ node@4.0.0-rc.4', function () { - var release = processRelease([], { opts: {} }, 'v4.0.0-rc.4', { + const release = processRelease([], { opts: {} }, 'v4.0.0-rc.4', { name: 'node', headersUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz' }) @@ -328,7 +328,7 @@ describe('process-release', function () { it('test process release - process.release ~ node@4.0.0-rc.4 passed as argv[0]', function () { // note the missing 'v' on the arg, it should normalise when checking // whether we're on the default or not - var release = processRelease(['4.0.0-rc.4'], { opts: {} }, 'v4.0.0-rc.4', { + const release = processRelease(['4.0.0-rc.4'], { opts: {} }, 'v4.0.0-rc.4', { name: 'node', headersUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz' }) @@ -352,7 +352,7 @@ describe('process-release', function () { it('test process release - process.release ~ node@4.0.0-rc.4 - bogus string passed as argv[0]', function () { // additional arguments can be passed in on the commandline that should be ignored if they // are not specifying a valid version @ position 0 - var release = processRelease(['this is no version!'], { opts: {} }, 'v4.0.0-rc.4', { + const release = processRelease(['this is no version!'], { opts: {} }, 'v4.0.0-rc.4', { name: 'node', headersUrl: 'https://nodejs.org/download/rc/v4.0.0-rc.4/node-v4.0.0-rc.4-headers.tar.gz' }) @@ -376,7 +376,7 @@ describe('process-release', function () { it('test process release - NODEJS_ORG_MIRROR', function () { process.env.NODEJS_ORG_MIRROR = 'http://foo.bar' - var release = processRelease([], { opts: {} }, 'v4.1.23', { + const release = processRelease([], { opts: {} }, 'v4.1.23', { name: 'node', headersUrl: 'https://nodejs.org/dist/v4.1.23/node-v4.1.23-headers.tar.gz' }) From b030555cdb754d9c23906e7e707115cd077bbf76 Mon Sep 17 00:00:00 2001 From: Stefan Stojanovic Date: Thu, 22 Jun 2023 12:13:50 +0200 Subject: [PATCH 375/551] lib: find python checks order changed on windows (#2872) These changes favor py launcher over other checks excluding command line or npm configuration and environment variable checks. Also, updated supported python versions list. Fixes: https://github.com/nodejs/node-gyp/issues/2871 --- lib/find-python.js | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/lib/find-python.js b/lib/find-python.js index c8129c0f65..3288839b50 100644 --- a/lib/find-python.js +++ b/lib/find-python.js @@ -14,7 +14,7 @@ const programFiles = process.env.ProgramW6432 || process.env.ProgramFiles || `${ const programFilesX86 = process.env['ProgramFiles(x86)'] || `${programFiles} (x86)` const winDefaultLocationsArray = [] -for (const majorMinor of ['39', '38', '37', '36']) { +for (const majorMinor of ['311', '310', '39', '38']) { if (foundLocalAppData) { winDefaultLocationsArray.push( `${localAppData}\\Programs\\Python\\Python${majorMinor}\\python.exe`, @@ -113,7 +113,20 @@ PythonFinder.prototype = { }, check: this.checkCommand, arg: this.env.PYTHON - }, + } + ] + + if (this.win) { + checks.push({ + before: () => { + this.addLog( + 'checking if the py launcher can be used to find Python 3') + }, + check: this.checkPyLauncher + }) + } + + checks.push(...[ { before: () => { this.addLog('checking if "python3" can be used') }, check: this.checkCommand, @@ -124,7 +137,7 @@ PythonFinder.prototype = { check: this.checkCommand, arg: 'python' } - ] + ]) if (this.win) { for (let i = 0; i < this.winDefaultLocations.length; ++i) { @@ -138,13 +151,6 @@ PythonFinder.prototype = { arg: location }) } - checks.push({ - before: () => { - this.addLog( - 'checking if the py launcher can be used to find Python 3') - }, - check: this.checkPyLauncher - }) } return checks From 53c99ae573bd5a5435e843b7de6b2e684f4de4d3 Mon Sep 17 00:00:00 2001 From: James Cook Date: Sun, 25 Jun 2023 08:35:41 -0700 Subject: [PATCH 376/551] Fix reading msvs version on Windows (#2644) * fix: fix reading msvs version on windows --- lib/configure.js | 2 +- lib/node-gyp.js | 2 +- test/test-options.js | 9 +++++++++ 3 files changed, 11 insertions(+), 2 deletions(-) diff --git a/lib/configure.js b/lib/configure.js index 97dccc6db8..bcd8bb23ea 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -85,7 +85,7 @@ function configure (gyp, argv, callback) { 'build dir', '"build" dir needed to be created?', isNew ? 'Yes' : 'No' ) if (win) { - findVisualStudio(release.semver, gyp.opts.msvs_version, + findVisualStudio(release.semver, gyp.opts['msvs-version'], createConfigFile) } else { createPythonSymlink() diff --git a/lib/node-gyp.js b/lib/node-gyp.js index 21d12460c8..392a0ecfa6 100644 --- a/lib/node-gyp.js +++ b/lib/node-gyp.js @@ -60,7 +60,7 @@ proto.configDefs = { debug: Boolean, // 'build' directory: String, // bin make: String, // 'build' - msvs_version: String, // 'configure' + 'msvs-version': String, // 'configure' ensure: Boolean, // 'install' solution: String, // 'build' (windows only) proxy: String, // 'install' diff --git a/test/test-options.js b/test/test-options.js index 24e79c80a1..8d281db8e8 100644 --- a/test/test-options.js +++ b/test/test-options.js @@ -38,4 +38,13 @@ describe('options', function () { assert.strictEqual(g.opts['force-process-config'], 'true') }) + + it('options with msvs_version', () => { + process.env.npm_config_msvs_version = '2017' + + const g = gyp() + g.parseArgv(['rebuild']) // Also sets opts.argv. + + assert.strictEqual(g.opts['msvs-version'], '2017') + }) }) From c9caa2ecf3c7deae68444ce8fabb32d2dca651cd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rare=C8=99?= <6453351+raress96@users.noreply.github.com> Date: Thu, 13 Jul 2023 19:08:41 +0300 Subject: [PATCH 377/551] docs: Update windows installation instructions in README.md (#2882) * Update windows installation instructions in README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 99494a38d0..65bb5ea1e0 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ Install the current version of Python from the [Microsoft Store package](https:/ Install tools and configuration manually: * Install Visual C++ Build Environment: [Visual Studio Build Tools](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) - (using "Visual C++ build tools" workload) or [Visual Studio Community](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=Community) + (using "Visual C++ build tools" if using a version older than VS2019, otherwise use "Desktop development with C++" workload) or [Visual Studio Community](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=Community) (using the "Desktop development with C++" workload) If the above steps didn't work for you, please visit [Microsoft's Node.js Guidelines for Windows](https://github.com/Microsoft/nodejs-guidelines/blob/master/windows-environment.md#compiling-native-addon-modules) for additional tips. From 1bfb083f8b0145c3f49507e64ddf7c4dcd75f83a Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Thu, 20 Jul 2023 15:04:39 +0200 Subject: [PATCH 378/551] Fix Python lint error by using an f-string (#2886) --- gyp/pylib/gyp/generator/make.py | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index f1d01a629d..05bb609d16 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -1991,11 +1991,8 @@ def WriteTarget( and "product_dir" not in spec and self.toolset == "target" ): - # On mac, products are created in install_path immediately. - assert install_path == self.output, "{} != {}".format( - install_path, - self.output, - ) + # On macOS, products are created in install_path immediately. + assert install_path == self.output, f"{install_path} != {self.output}" # Point the target alias to the final binary output. self.WriteMakeRule( From 445c28fabc5fbdf9c3bb3341fb70660a3530f6ad Mon Sep 17 00:00:00 2001 From: Stefan Stojanovic Date: Mon, 24 Jul 2023 12:38:47 +0200 Subject: [PATCH 379/551] test: increase mocha timeout (#2887) --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index 31c8a25d79..d861261002 100644 --- a/package.json +++ b/package.json @@ -45,6 +45,6 @@ }, "scripts": { "lint": "standard */*.js test/**/*.js", - "test": "npm run lint && mocha --reporter=test/reporter.js test/test-download.js test/test-*" + "test": "npm run lint && mocha --timeout 15000 --reporter=test/reporter.js test/test-download.js test/test-*" } } From 0f1f667b737d21905e283df100a2cb639993562a Mon Sep 17 00:00:00 2001 From: Tim Perry <1526883+pimterry@users.noreply.github.com> Date: Tue, 1 Aug 2023 10:02:59 +0200 Subject: [PATCH 380/551] fix: create Python symlink only during builds, and clean it up after (#2721) * fix: create Python symlink only during builds, and clean it up after Previously in b9ddcd5bbd93b05b03674836b6ebdae2c2e74c8c this was created during configuration, and the symlink persisted indefinitely. This causes problems with many tools that do not expect a codebase to include symlinks to external absolute paths. This PR largely reverts that commit, and instead writes the path to link to into the config, and then creates the symlink only temporarily during the build process, always deleting it afterwards. * assert install_path == self.output, f"{install_path} != {self.output}" --------- Co-authored-by: Christian Clauss --- lib/build.js | 29 ++++++++++++++++++++++++++--- lib/configure.js | 26 ++------------------------ lib/create-config-gypi.js | 9 ++++++--- test/test-configure-python.js | 4 +--- 4 files changed, 35 insertions(+), 33 deletions(-) diff --git a/lib/build.js b/lib/build.js index 3b95b76353..0bcc9bea19 100644 --- a/lib/build.js +++ b/lib/build.js @@ -30,6 +30,8 @@ async function build (gyp, argv) { let arch let nodeDir let guessedSolution + let python + let buildBinsDir await loadConfigGypi() @@ -56,6 +58,7 @@ async function build (gyp, argv) { buildType = config.target_defaults.default_configuration arch = config.variables.target_arch nodeDir = config.variables.nodedir + python = config.variables.python if ('debug' in gyp.opts) { buildType = gyp.opts.debug ? 'Debug' : 'Release' @@ -67,6 +70,7 @@ async function build (gyp, argv) { log.verbose('build type', buildType) log.verbose('architecture', arch) log.verbose('node dev dir', nodeDir) + log.verbose('python', python) if (win) { await findSolutionFile() @@ -182,13 +186,32 @@ async function build (gyp, argv) { if (!win) { // Add build-time dependency symlinks (such as Python) to PATH - const buildBinsDir = path.resolve('build', 'node_gyp_bins') + buildBinsDir = path.resolve('build', 'node_gyp_bins') process.env.PATH = `${buildBinsDir}:${process.env.PATH}` - log.verbose('bin symlinks', `adding symlinks (such as Python), at "${buildBinsDir}", to PATH`) + await fs.mkdir(buildBinsDir, { recursive: true }) + const symlinkDestination = path.join(buildBinsDir, 'python3') + try { + await fs.unlink(symlinkDestination) + } catch (err) { + if (err.code !== 'ENOENT') throw err + } + await fs.symlink(python, symlinkDestination) + log.verbose('bin symlinks', `created symlink to "${python}" in "${buildBinsDir}" and added to PATH`) } const proc = gyp.spawn(command, argv) - await new Promise((resolve, reject) => proc.on('exit', (code, signal) => { + await new Promise((resolve, reject) => proc.on('exit', async (code, signal) => { + if (buildBinsDir) { + // Clean up the build-time dependency symlinks: + if (fs.rm) { + // fs.rm is only available in Node 14+ + await fs.rm(buildBinsDir, { recursive: true }) + } else { + // Only used for old node, as recursive rmdir is deprecated in Node 14+ + await fs.rmdir(buildBinsDir, { recursive: true }) + } + } + if (code !== 0) { return reject(new Error('`' + command + '` failed with exit code: ' + code)) } diff --git a/lib/configure.js b/lib/configure.js index bcd8bb23ea..01c3f497ad 100644 --- a/lib/configure.js +++ b/lib/configure.js @@ -18,7 +18,6 @@ if (win) { function configure (gyp, argv, callback) { let python const buildDir = path.resolve('build') - const buildBinsDir = path.join(buildDir, 'node_gyp_bins') const configNames = ['config.gypi', 'common.gypi'] const configs = [] let nodeDir @@ -76,8 +75,7 @@ function configure (gyp, argv, callback) { function createBuildDir () { log.verbose('build dir', 'attempting to create "build" dir: %s', buildDir) - const deepestBuildDirSubdirectory = win ? buildDir : buildBinsDir - fs.mkdir(deepestBuildDirSubdirectory, { recursive: true }, function (err, isNew) { + fs.mkdir(buildDir, { recursive: true }, function (err, isNew) { if (err) { return callback(err) } @@ -88,31 +86,11 @@ function configure (gyp, argv, callback) { findVisualStudio(release.semver, gyp.opts['msvs-version'], createConfigFile) } else { - createPythonSymlink() createConfigFile() } }) } - function createPythonSymlink () { - const symlinkDestination = path.join(buildBinsDir, 'python3') - - log.verbose('python symlink', `creating symlink to "${python}" at "${symlinkDestination}"`) - - fs.unlink(symlinkDestination, function (err) { - if (err && err.code !== 'ENOENT') { - log.verbose('python symlink', 'error when attempting to remove existing symlink') - log.verbose('python symlink', err.stack, 'errno: ' + err.errno) - } - fs.symlink(python, symlinkDestination, function (err) { - if (err) { - log.verbose('python symlink', 'error when attempting to create Python symlink') - log.verbose('python symlink', err.stack, 'errno: ' + err.errno) - } - }) - }) - } - function createConfigFile (err, vsInfo) { if (err) { return callback(err) @@ -121,7 +99,7 @@ function configure (gyp, argv, callback) { process.env.GYP_MSVS_VERSION = Math.min(vsInfo.versionYear, 2015) process.env.GYP_MSVS_OVERRIDE_PATH = vsInfo.path } - createConfigGypi({ gyp, buildDir, nodeDir, vsInfo }).then(configPath => { + createConfigGypi({ gyp, buildDir, nodeDir, vsInfo, python }).then(configPath => { configs.push(configPath) findConfigs() }).catch(err => { diff --git a/lib/create-config-gypi.js b/lib/create-config-gypi.js index 2eab6f3035..8687379385 100644 --- a/lib/create-config-gypi.js +++ b/lib/create-config-gypi.js @@ -35,7 +35,7 @@ async function getBaseConfigGypi ({ gyp, nodeDir }) { return JSON.parse(JSON.stringify(process.config)) } -async function getCurrentConfigGypi ({ gyp, nodeDir, vsInfo }) { +async function getCurrentConfigGypi ({ gyp, nodeDir, vsInfo, python }) { const config = await getBaseConfigGypi({ gyp, nodeDir }) if (!config.target_defaults) { config.target_defaults = {} @@ -75,6 +75,9 @@ async function getCurrentConfigGypi ({ gyp, nodeDir, vsInfo }) { // set the node development directory variables.nodedir = nodeDir + // set the configured Python path + variables.python = python + // disable -T "thin" static archives by default variables.standalone_static_library = gyp.opts.thin ? 0 : 1 @@ -112,13 +115,13 @@ async function getCurrentConfigGypi ({ gyp, nodeDir, vsInfo }) { return config } -async function createConfigGypi ({ gyp, buildDir, nodeDir, vsInfo }) { +async function createConfigGypi ({ gyp, buildDir, nodeDir, vsInfo, python }) { const configFilename = 'config.gypi' const configPath = path.resolve(buildDir, configFilename) log.verbose('build/' + configFilename, 'creating config file') - const config = await getCurrentConfigGypi({ gyp, nodeDir, vsInfo }) + const config = await getCurrentConfigGypi({ gyp, nodeDir, vsInfo, python }) // ensures that any boolean values in config.gypi get stringified function boolsToString (k, v) { diff --git a/test/test-configure-python.js b/test/test-configure-python.js index 9c042b18d5..dcce3e5de2 100644 --- a/test/test-configure-python.js +++ b/test/test-configure-python.js @@ -16,9 +16,7 @@ const configure = requireInject('../lib/configure', { mkdir: function (dir, options, cb) { cb() }, promises: { writeFile: function (file, data) { return Promise.resolve(null) } - }, - unlink: function (path, cb) { cb() }, - symlink: function (target, path, cb) { cb() } + } } }) From bb93b946a9c74934b59164deb52128cf913c97d5 Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Fri, 25 Aug 2023 15:04:27 +0200 Subject: [PATCH 381/551] docs: README.md Do not hardcode the supported versions of Python (#2880) --- .github/workflows/tests.yml | 3 +- README.md | 40 +++++++------- macOS_Catalina.md | 104 ------------------------------------ 3 files changed, 24 insertions(+), 123 deletions(-) delete mode 100644 macOS_Catalina.md diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 517b2d95a4..17fde83da5 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -13,8 +13,9 @@ jobs: steps: - uses: actions/checkout@v3 - run: pip install --user ruff - - run: ruff --format=github --select="E,F,PLC,PLE,UP,W,YTT" --ignore="PLC1901,S101,UP031" --target-version=py37 . + - run: ruff --format=github --select="E,F,PLC,PLE,UP,W,YTT" --ignore="E721,PLC1901,S101,UP031" --target-version=py38 . Tests: + needs: Lint_Python # Lint_Python takes ~5 seconds, so wait for it to pass before running the full matrix of tests. strategy: fail-fast: false max-parallel: 15 diff --git a/README.md b/README.md index 65bb5ea1e0..fc0e8ee0a2 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,8 @@ `node-gyp` is a cross-platform command-line tool written in Node.js for compiling native addon modules for Node.js. It contains a vendored copy of the [gyp-next](https://github.com/nodejs/gyp-next) project that was previously used -by the Chromium team, extended to support the development of Node.js native addons. +by the Chromium team and extended to support the development of Node.js native +addons. Note that `node-gyp` is _not_ used to build Node.js itself. @@ -31,15 +32,13 @@ Depending on your operating system, you will need to install: ### On Unix - * Python v3.7, v3.8, v3.9, or v3.10 + * [A supported version of Python](https://devguide.python.org/versions/) * `make` * A proper C/C++ compiler toolchain, like [GCC](https://gcc.gnu.org) ### On macOS -**ATTENTION**: If your Mac has been _upgraded_ to macOS Catalina (10.15) or higher, please read [macOS_Catalina.md](macOS_Catalina.md). - - * Python v3.7, v3.8, v3.9, or v3.10 + * [A supported version of Python](https://devguide.python.org/versions/) * `XCode Command Line Tools` which will install `clang`, `clang++`, and `make`. * Install the `XCode Command Line Tools` standalone by running `xcode-select --install`. -- OR -- * Alternatively, if you already have the [full Xcode installed](https://developer.apple.com/xcode/download/), you can install the Command Line Tools under the menu `Xcode -> Open Developer Tool -> More Developer Tools...`. @@ -47,7 +46,8 @@ Depending on your operating system, you will need to install: ### On Windows -Install the current version of Python from the [Microsoft Store package](https://www.microsoft.com/en-us/p/python-310/9pjpw5ldxlz5). +Install the current [version of Python](https://devguide.python.org/versions/) from the +[Microsoft Store](https://apps.microsoft.com/store/search?publisher=Python+Software+Foundation). Install tools and configuration manually: * Install Visual C++ Build Environment: [Visual Studio Build Tools](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) @@ -62,9 +62,9 @@ Install tools and configuration manually: ### Configuring Python Dependency -`node-gyp` requires that you have installed a compatible version of Python, one of: v3.7, v3.8, -v3.9, or v3.10. If you have multiple Python versions installed, you can identify which Python -version `node-gyp` should use in one of the following ways: +`node-gyp` requires that you have installed a [supported version of Python](https://devguide.python.org/versions/). +If you have multiple versions of Python installed, you can identify which version +`node-gyp` should use in one of the following ways: 1. by setting the `--python` command-line option, e.g.: @@ -73,24 +73,28 @@ node-gyp --python /path/to/executable/python ``` 2. If `node-gyp` is called by way of `npm`, *and* you have multiple versions of -Python installed, then you can set `npm`'s 'python' config key to the appropriate -value: - +Python installed, then you can set the `npm_config_python` environment variable +to the appropriate path: ``` bash -npm config set python /path/to/executable/python +export npm_config_python=/path/to/executable/python +``` +    Or on Windows: +```console +py --list-paths # To see the installed Python versions +set npm_config_python=C:\path\to\python.exe ``` 3. If the `PYTHON` environment variable is set to the path of a Python executable, -then that version will be used, if it is a compatible version. +then that version will be used if it is a supported version. 4. If the `NODE_GYP_FORCE_PYTHON` environment variable is set to the path of a Python executable, it will be used instead of any of the other configured or -builtin Python search paths. If it's not a compatible version, no further +built-in Python search paths. If it's not a compatible version, no further searching will be done. ### Build for Third Party Node.js Runtimes -When building modules for third party Node.js runtimes like Electron, which have +When building modules for third-party Node.js runtimes like Electron, which have different build configurations from the official Node.js distribution, you should use `--dist-url` or `--nodedir` flags to specify the headers of the runtime to build for. @@ -106,7 +110,7 @@ to work around configuration errors. ## How to Use -To compile your native addon, first go to its root directory: +To compile your native addon first go to its root directory: ``` bash cd my_node_addon @@ -240,7 +244,7 @@ Or this on Windows: set npm_config_devdir=c:\temp\.gyp ``` -### `npm` configuration +### `npm` configuration for npm versions before v9 Use the form `OPTION_NAME` for any of the command options listed above. diff --git a/macOS_Catalina.md b/macOS_Catalina.md deleted file mode 100644 index dde5fe3f7d..0000000000 --- a/macOS_Catalina.md +++ /dev/null @@ -1,104 +0,0 @@ -# Installation notes for macOS Catalina (v10.15) - -_This document specifically refers to upgrades from previous versions of macOS to Catalina (10.15). It should be removed from the source repository when Catalina ceases to be the latest macOS version or when future Catalina versions no longer raise these issues._ - -**Both upgrading to macOS Catalina and running a Software Update in Catalina may cause normal `node-gyp` installations to fail. This might manifest as the following error during `npm install`:** - -```console -gyp: No Xcode or CLT version detected! -``` - -## node-gyp v7 - -The newest release of `node-gyp` should solve this problem. If you are using `node-gyp` directly then you should be able to install v7 and use it as-is. - -If you need to use `node-gyp` from within `npm` (e.g. through `npm install`), you will have to install `node-gyp` (either globally with `-g` or to a predictable location) and tell `npm` where the new version is. Either use: - -* `npm config set node_gyp `; or -* run `npm` with an environment variable prefix: `npm_config_node_gyp= npm install` - -Where "path to node-gyp" is to the `node-gyp` executable which may be a symlink in your global bin directory (e.g. `/usr/local/bin/node-gyp`), or a path to the `node-gyp` installation directory and the `bin/node-gyp.js` file within it (e.g. `/usr/local/lib/node_modules/node-gyp/bin/node-gyp.js`). - -**If you use `npm config set` to change your global `node_gyp` you are responsible for keeping it up to date and can't rely on `npm` to give you a newer version when available.** Use `npm config delete node_gyp` to unset this configuration option. - -## Fixing Catalina for older versions of `node-gyp` - -### Is my Mac running macOS Catalina? -Let's first make sure that your Mac is running Catalina: -``` -% sw_vers - ProductName: Mac OS X - ProductVersion: 10.15 - BuildVersion: 19A602 -``` -If `ProductVersion` is less then `10.15` then this document is not for you. Normal install docs for `node-gyp` on macOS can be found at https://github.com/nodejs/node-gyp#on-macos - - -### The acid test -To see if `Xcode Command Line Tools` is installed in a way that will work with `node-gyp`, run: -``` -curl -sL https://github.com/nodejs/node-gyp/raw/main/macOS_Catalina_acid_test.sh | bash -``` - -If test succeeded, _you are done_! You should be ready to [install](https://github.com/nodejs/node-gyp#installation) `node-gyp`. - -If test failed, there is a problem with your Xcode Command Line Tools installation. [Continue to Solutions](#Solutions). - -### Solutions -There are three ways to install the Xcode libraries `node-gyp` needs on macOS. People running Catalina have had success with some but not others in a way that has been unpredictable. - -1. With the full Xcode (~7.6 GB download) from the `App Store` app. -2. With the _much_ smaller Xcode Command Line Tools via `xcode-select --install` -3. With the _much_ smaller Xcode Command Line Tools via manual download. **For people running the latest version of Catalina (10.15.2 at the time of this writing), this has worked when the other two solutions haven't.** - -### Installing `node-gyp` using the full Xcode -1. `xcodebuild -version` should show `Xcode 11.1` or later. - * If not, then install/upgrade Xcode from the App Store app. -2. Open the Xcode app and... - * Under __Preferences > Locations__ select the tools if their location is empty. - * Allow Xcode app to do an essential install of the most recent compiler tools. -3. Once all installations are _complete_, quit out of Xcode. -4. `sudo xcodebuild -license accept` # If you agree with the licensing terms. -5. `softwareupdate -l` # No listing is a good sign. - * If Xcode or Tools upgrades are listed, use "Software Upgrade" to install them. -6. `xcode-select -version` # Should return `xcode-select version 2370` or later. -7. `xcode-select -print-path` # Should return `/Applications/Xcode.app/Contents/Developer` -8. Try the [_acid test_ steps above](#The-acid-test) to see if your Mac is ready. -9. If the _acid test_ does _not_ pass then... -10. `sudo xcode-select --reset` # Enter root password. No output is normal. -11. Repeat step 7 above. Is the path different this time? Repeat the _acid test_. - -### Installing `node-gyp` using the Xcode Command Line Tools via `xcode-select --install` -1. If the _acid test_ has not succeeded, then try `xcode-select --install` -2. If the installation command returns `xcode-select: error: command line tools are already installed, use "Software Update" to install updates`, continue to [remove and reinstall](#i-did-all-that-and-the-acid-test-still-does-not-pass--) -3. Wait until the install process is _complete_. -4. `softwareupdate -l` # No listing is a good sign. - * If Xcode or Tools upgrades are listed, use "Software Update" to install them. -5. `xcode-select -version` # Should return `xcode-select version 2370` or later. -6. `xcode-select -print-path` # Should return `/Library/Developer/CommandLineTools` -7. Try the [_acid test_ steps above](#The-acid-test) to see if your Mac is ready. -8. If the _acid test_ does _not_ pass then... -9. `sudo xcode-select --reset` # Enter root password. No output is normal. -10. Repeat step 5 above. Is the path different this time? Repeat the _acid test_. - -### Installing `node-gyp` using the Xcode Command Line Tools via manual download -1. Download the appropriate version of the "Command Line Tools for Xcode" for your version of Catalina from . As of MacOS 10.15.5, that's [Command_Line_Tools_for_Xcode_11.5.dmg](https://download.developer.apple.com/Developer_Tools/Command_Line_Tools_for_Xcode_11.5/Command_Line_Tools_for_Xcode_11.5.dmg) -2. Install the package. -3. Run the [_acid test_ steps above](#The-acid-test). - -### I did all that and the acid test still does not pass :-( -1. `sudo rm -rf $(xcode-select -print-path)` # Enter root password. No output is normal. -2. `sudo rm -rf /Library/Developer/CommandLineTools` # Enter root password. -3. `sudo xcode-select --reset` -4. `xcode-select --install` -5. If the [_acid test_ steps above](#The-acid-test) still does _not_ pass then... -6. `npm explore npm -g -- npm install node-gyp@latest` -7. `npm explore npm -g -- npm explore npm-lifecycle -- npm install node-gyp@latest` -8. If the _acid test_ still does _not_ pass then... -9. Add a comment to https://github.com/nodejs/node-gyp/issues/1927 so we can improve. - -Lessons learned from: -* https://github.com/nodejs/node-gyp/issues/1779 -* https://github.com/nodejs/node-gyp/issues/1861 -* https://github.com/nodejs/node-gyp/issues/1927 and elsewhere -* Thanks to @rrrix for discovering Solution 3 From d3615c66f7e7a65de48ce9860b1fe13710d20988 Mon Sep 17 00:00:00 2001 From: Iulian Onofrei <5748627+revolter@users.noreply.github.com> Date: Sat, 26 Aug 2023 11:48:03 +0300 Subject: [PATCH 382/551] Fix incorrect Xcode casing in README (#2896) --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index fc0e8ee0a2..8a7ad87a01 100644 --- a/README.md +++ b/README.md @@ -39,8 +39,8 @@ Depending on your operating system, you will need to install: ### On macOS * [A supported version of Python](https://devguide.python.org/versions/) - * `XCode Command Line Tools` which will install `clang`, `clang++`, and `make`. - * Install the `XCode Command Line Tools` standalone by running `xcode-select --install`. -- OR -- + * `Xcode Command Line Tools` which will install `clang`, `clang++`, and `make`. + * Install the `Xcode Command Line Tools` standalone by running `xcode-select --install`. -- OR -- * Alternatively, if you already have the [full Xcode installed](https://developer.apple.com/xcode/download/), you can install the Command Line Tools under the menu `Xcode -> Open Developer Tool -> More Developer Tools...`. From 5746691a36f7b37019d4b8d4e9616aec43d20410 Mon Sep 17 00:00:00 2001 From: Rich Trott Date: Wed, 27 Sep 2023 22:56:36 -0700 Subject: [PATCH 383/551] test: update expired certs (#2908) --- test/fixtures/certs.js | 91 +++++++++++++++++++++--------------------- 1 file changed, 46 insertions(+), 45 deletions(-) diff --git a/test/fixtures/certs.js b/test/fixtures/certs.js index 766e54b5ed..f03f1de0d7 100644 --- a/test/fixtures/certs.js +++ b/test/fixtures/certs.js @@ -54,55 +54,56 @@ GbN8uKxIfyMjqZKaujPR ` module.exports['server.key'] = ` ------BEGIN RSA PRIVATE KEY----- -MIIEpAIBAAKCAQEAvPM99BkYrBcTM355dhz4XzhSDRGxa9qttUlBSgEsbu2UjsRm -XjDS+60XXd66tWpPwLeUd2uvlC/ltv5ekv+EBu35j1KfA1+K1rtFzb1i40kMCsns -OoXjgpsN2wvkxMdFkT2bkqKCS6X0xzlWea1t4poKh9iG7n3otk4RzPNawfwQ9W5n -o9/8i6AUwDbuK4dhAId/Inw2aKrMyQ+AiSvsDM2wUMq+pV7nP46f7MhR4xiGz14z -ATFdjM3Oo/1NKrr0WgVM6i0eNAtuIDqIs8YE7SfODe/SzpJySxewutfYi62OaLh/ -hmWByj/pF5SoNMLyJHxn4GyKK+Qle9NJAThLiwIDAQABAoIBAQCZs4h/Cvct7etZ -pRUqxnAoDQl5xh28LXvGj1uD1qaNacfBxvO6xR6rSedLHcZlkqBjlTI5XqjJ85h6 -njrSevWsKWMrejsNpGetO1OSA+/wEVixYgY+qPEkKftAZ1Fl3O+zMRlfU8CHxuzy -Lqsweap8fW/5h2JjmJp3ydPjE0aNqpQ+0LtYBBawKDIe2zPNOmTPwz3D8qJNQJKU -Qdj08dO/vPZncllPagGvpqhfv4hMyNChr71eBbEFsi3O5VJxfZyj+fQG0DGocr/y -sV54HtYk5j06wMxZFLQtaJn+1pOXquZMNwodSPnbrR/+CI1SZeB8N6VyqqOdmrDz -5NbfGJiRAoGBAPrCuQxJwgc2MzpEtrXA4+1uuV8QWGy3+wNKxKw4lgyC7peXXrVK -l9FkOOAPr8puPRABgDS9t6vo59BAP3Wrx0oJ9PA/Qn03WYLfJMepWqlK7ni9kS04 -5owRTduK7P190sp0m6iicsnchGSSOchECwB5UmtHysEFiuY0T+0pdNbjAoGBAMDl -57lwZDfNTGGDxLQYVzbrXgKcD60DW9MhvH3uso6cw5NYs3tmENCh9D6YrCNN4PmL -zdp4dKbOFoGJdy22TK31nrezUuNKSK+QKH2gsmNVI+a5QokNO1Cfk+PMLoOR5du2 -nwyVvzdaBwuXU4fhmwvLv/SCFNEJ0EgUILeMETE5AoGBAIwLXf9v3e3bJkb/gy8E -mAbNVLez0D5/ja9r/WTVgW9hXFDLF/iVvS4TE/SGrj2WzYF35RsPbVmUDIrwpsBX -/EfsQaA/JCn8VIBTkR31Bg4QLBjAfijMY22MaHgZIXv83lF1SE2o1ATKpCHqzFx9 -K8vK9e22PZUJPGaOhqjEA13TAoGAEPipSJFw38/6NmInfkjd84EFxmkAoBI5k/vV -36aOoyl7s40MTYEPXavCF3fLPVfuwUXhmKUcbkiXhlIX4De3y15e1n66fjDc8EVY -qqTmzQKCpBwMlI5Ld65yjoo6VW0SsiABIlRSfIY5NHXd7YiV4ZXNj6+aMUIRxyWu -Mzfpk1ECgYBZw8lML+F8XbcmCLGYuicf0V/wgFaJr8nnPeW7AiQrv13Ju1ItEaC8 -Tz8F7OfC+FoUb0MGjXHKquDVBDs4xSuS+Ol+rlZEK68ALpm8sUgp6YjARYiXlprs -6o4kN0P5F+DVU2SP6fo9zKLCxaTH9VAQ9C3VUViGAFLBt4DVDmj77g== ------END RSA PRIVATE KEY----- +-----BEGIN PRIVATE KEY----- +MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDpxqKqzq8PuOuo +M09X1gdh6V1nMmJ6Oqk/d8ZSAHhGBTjB5BBILtrxJPAdsNCRc8gPCtSuAzu9r0ct +D2znJjto8HO3a0s/4mlqMQ8iSvfH0c8nGLPxRXg0C9tR0VOyqJrqAPn9X6utxVQj +H2jLjZngZllhqlxIUxYuXO7MZfC2MQ4DRsp6GaJTftcOxTxevMdT6lK0yl3mDn0W +DyV9aWR5oCVXKIRK7jmF9bi9ETHwJHffFrduaaKYLQ/UOEtNek/pJsz0DNO0ePkm +DAt0GQkX3hmKd0VRgGEtaO+MAGbrDucPuDylMkC1GJm1lq6+6rkyQRNoAd8/yOJD +GihBJmLPAgMBAAECggEBAL3+WM/3IGHnyWavJMnfQaq6rdWkJlLugAT8BCs7BITr +04AJKY5wvjID8j4/KJM+BRbsl4NBT3lPDcq6YajO8rPL0E/+nG60RTYv3vvg79Xv +V6uPsRbifdnW1Q1+0cY+r4CFAKeC7JVS7ZmJ+nKMh8XPiM8OVOfW1w0hLFbkdqiq +UetA5H0oiJr0rFWh9Gfzc0avpmawXmZJqK8lKWB1rt2teu3bQWqn20dGr2IyCipz +KlGh3QImYogjqWEqPyowPGxnq8KWQ40Pjx61dy6cqYrw77yxLHGR8Az+TlRQZf/K +BeT3UkXCU5BTyC3PKZJpcYB/GMB8GBKRdNK+8H3OXskCgYEA963tUnLoiCwJjxvv +gIiMY5mIUAMnTDsCxSYekfkDvt4qdttn/+Y8/5/lXMSlCt+WZcXH0V8gmVOMqfHB +QRXWpi0uJnl9sUllb80Mw5PnS317UlJIbkro3crVUhIbrn9tKVFP1GO9xszNda8E +MkHV8FGmSaihXGnA8KEkTKg1CisCgYEA8aEjATIpDXyxQM4M9B1S0jI/nHOrg2lr +dMKQWJVruuvwrrZl6CFNIm2/PBXyHjCXqROXs7aXkE3vfRSypYNKBZl7BX08c0nE +koMVXwhGbSbUXjFx5bcUKaG/2eOK+ZV4ivNQokq1iIK+WKINAfP69ewzD7EAgbBg +KY7owSxka+0CgYB4Urh+W3B35tzl9y5NBQkewdGk/UM0F17rI++p/o1BRnDeuQw3 +F0T+8lDc1nNPavuHiaPfJRWTJzGoxdeapN9Yb46CBnd3jy6GN9lBkjLFS7qDbZHe +cunaBdXIPx/Pj/waHHRpu+LQF2KhD1s8hxtF2oSsOA3b9UxUGhSmYPkTbQKBgEP8 +muTTQEnTM+yQDYUCWzNZgBx9T10CZIHN3N+P62gEywvdtn7CH/n39z7ozd9AvOuN +37lpPuwTgbcoA7weXM2Gid7ZhhDKSM0QpQrAQVClBEwcjXedM8cjA+BC7e+b5vbx +z1ZavwlSAEzgC9jo1Uws0ZEwtHvJLMWEuGjiHL9hAoGAWEbj2cxhpCKTK3C8Qf+C +BscBwAZKmUasdWQBUzRFpR1UAO3fFBatjMPwrFYnt+ZgiCXTGFcoEa7mjoP/r5Mh +j1nRCoPQVcbmB1B9AtiaEa1AA+BqYF1r2aErbcsGrCV5OHU7TYFk1dep1SAQP/0W +9MQ+1Af5ttYSFYlJLnWJo4M= +-----END PRIVATE KEY----- ` module.exports['server.crt'] = ` -----BEGIN CERTIFICATE----- -MIIDNzCCAh8CFBg1Ph5t5rBlAbCrEn/PexNy9WunMA0GCSqGSIb3DQEBCwUAMF0x -CzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTEQMA4GA1UECgwHTm9kZS5qczERMA8G -A1UECwwIbm9kZS1neXAxHDAaBgNVBAMME25vZGUtZ3lwLm5vZGVqcy5vcmcwHhcN -MjIwNTExMDQyMjI0WhcNMjMwOTIzMDQyMjI0WjBTMQswCQYDVQQGEwJVUzELMAkG -A1UECAwCQ0ExEDAOBgNVBAoMB05vZGUuanMxETAPBgNVBAsMCG5vZGUtZ3lwMRIw -EAYDVQQDDAlsb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB -AQC88z30GRisFxMzfnl2HPhfOFINEbFr2q21SUFKASxu7ZSOxGZeMNL7rRdd3rq1 -ak/At5R3a6+UL+W2/l6S/4QG7fmPUp8DX4rWu0XNvWLjSQwKyew6heOCmw3bC+TE -x0WRPZuSooJLpfTHOVZ5rW3imgqH2Ibufei2ThHM81rB/BD1bmej3/yLoBTANu4r -h2EAh38ifDZoqszJD4CJK+wMzbBQyr6lXuc/jp/syFHjGIbPXjMBMV2Mzc6j/U0q -uvRaBUzqLR40C24gOoizxgTtJ84N79LOknJLF7C619iLrY5ouH+GZYHKP+kXlKg0 -wvIkfGfgbIor5CV700kBOEuLAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAEhaNEye -JsE4eG1xaGmHq7w9eV0neOaZ58XCuF1sSEMIy9uMnl3aOdctxh/1SYkqJyMct79q -Ra2UZ6mauRlOeqdHb+HZKrFYYUOtd1HOWWJ44Gaya2EQMiTbd/Ns9Th2KTbTOCbL -CHFNska9Ty2ioT7VcrVuIEXFPMua5T4lnCkNJQla800pHHOak2baN/c66Io+8XI2 -xiqaVrLT3qvpzdiiEjo4POeRnWMIgJJshy77Am5JlhaJiAqP1AHfh/tYpliGkjXF -8DSgSoLHSQfalJ4VQvP4XLc/XnmF5Zt6bvwUxCllEBFRneU74bW7/euYX/TpYobr -Y+YM7fGiCqwHdUs= +MIIDPjCCAiagAwIBAgIJALrth9K8D7HIMA0GCSqGSIb3DQEBCwUAMF0xCzAJBgNV +BAYTAlVTMQswCQYDVQQIDAJDQTEQMA4GA1UECgwHTm9kZS5qczERMA8GA1UECwwI +bm9kZS1neXAxHDAaBgNVBAMME25vZGUtZ3lwLm5vZGVqcy5vcmcwHhcNMjMwOTI4 +MDUxODM2WhcNMzMwOTI1MDUxODM2WjBgMQswCQYDVQQGEwJVUzETMBEGA1UECAwK +Q2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzEQMA4GA1UECgwHTm9k +ZS5qczESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A +MIIBCgKCAQEA6caiqs6vD7jrqDNPV9YHYeldZzJiejqpP3fGUgB4RgU4weQQSC7a +8STwHbDQkXPIDwrUrgM7va9HLQ9s5yY7aPBzt2tLP+JpajEPIkr3x9HPJxiz8UV4 +NAvbUdFTsqia6gD5/V+rrcVUIx9oy42Z4GZZYapcSFMWLlzuzGXwtjEOA0bKehmi +U37XDsU8XrzHU+pStMpd5g59Fg8lfWlkeaAlVyiESu45hfW4vREx8CR33xa3bmmi +mC0P1DhLTXpP6SbM9AzTtHj5JgwLdBkJF94ZindFUYBhLWjvjABm6w7nD7g8pTJA +tRiZtZauvuq5MkETaAHfP8jiQxooQSZizwIDAQABMA0GCSqGSIb3DQEBCwUAA4IB +AQBwgEyrqJOV8SC7PVTtEOqfSyrM7lJjVcmwXEIFPVCPxXnDtLS9+OaQe9ybjOR/ +Bi/AvZK4gwsV9G5Bvbl0/sphYEKYLEpP76jhdETcBwhaEgK3itumoREeriut4bZI +OM6b1O45CoD67Lm87CUwLOdcNzPu4k7mat+xog5aFwaQuRjLBmmZcjl41QjVr9ti +La4PCMh7NwVMtHRqbYvgq785PsKAh+j4FSX1sj9NRzRPoJJ2qsre1Qn5tL/i6ovj +6s+3GxOQ5I1UzJX22PZFu003a582ha1CEFM0VaeDzzwbGNcV5SP+g2nw55zx9YRR +Rg8nGmjRuRtbs+/XAre2eQ5p -----END CERTIFICATE----- ` From b3d41aeb737ddd54cc292f363abc561dcc0a614e Mon Sep 17 00:00:00 2001 From: DeeDeeG Date: Tue, 3 Oct 2023 01:32:06 -0400 Subject: [PATCH 384/551] doc: Add note about Python symlinks (PR 2362) to CHANGELOG.md for 9.1.0 (#2783) The PR for this change was merged without a prefixed name, such as "lib:" or "fix:". That means release-please didn't include it in the changelog for v9.1.0. This change did end up affecting users, though. (See issue 2713 and PR 2721). Therefore, I believe it should be noted in the CHANGELOG.md, so users can better understand the behavior they are seeing. --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9fb5f11847..edf79f7025 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -98,6 +98,7 @@ ### Core * update due to rename of primary branch ([ca1f068](https://www.github.com/nodejs/node-gyp/commit/ca1f0681a5567ca8cd51acebccd37a633f19bc6a)) +* Add Python symlink to path (for non-Windows OSes only) ([#2362](https://github.com/nodejs/node-gyp/pull/2362)) ([b9ddcd5](https://github.com/nodejs/node-gyp/commit/b9ddcd5bbd93b05b03674836b6ebdae2c2e74c8c)) ### Tests From 91fd8ffe6a955ca7546f0b74462a466d8e11a5ef Mon Sep 17 00:00:00 2001 From: Christian Clauss Date: Fri, 27 Oct 2023 10:52:08 +0200 Subject: [PATCH 385/551] Python lint: ruff --format is now --output-format Fixes the failing `ruff` linting in GitHub Actions. --- .github/workflows/tests.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 17fde83da5..7070df4e25 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -11,9 +11,9 @@ jobs: Lint_Python: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - run: pip install --user ruff - - run: ruff --format=github --select="E,F,PLC,PLE,UP,W,YTT" --ignore="E721,PLC1901,S101,UP031" --target-version=py38 . + - run: ruff --output-format=github --select="E,F,PLC,PLE,UP,W,YTT" --ignore="E721,PLC1901,S101,UP031" --target-version=py38 . Tests: needs: Lint_Python # Lint_Python takes ~5 seconds, so wait for it to pass before running the full matrix of tests. strategy: @@ -26,7 +26,7 @@ jobs: runs-on: ${{ matrix.os }} steps: - name: Checkout Repository - uses: actions/checkout@v3 + uses: actions/checkout@v4 - name: Use Node.js ${{ matrix.node }} uses: actions/setup-node@v3 with: From 26683e993df038fb94d89f2276f3535e4522d79a Mon Sep 17 00:00:00 2001 From: Alex Date: Fri, 27 Oct 2023 19:45:09 +0200 Subject: [PATCH 386/551] chore: GitHub Workflows security hardening (#2740) * build: harden tests.yml permissions Signed-off-by: Alex * build: harden release-please.yml permissions Signed-off-by: Alex * build: harden visual-studio.yml permissions Signed-off-by: Alex * Update release-please.yml --------- Signed-off-by: Alex --- .github/workflows/release-please.yml | 4 ++++ .github/workflows/tests.yml | 4 ++++ .github/workflows/visual-studio.yml | 4 ++++ 3 files changed, 12 insertions(+) diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml index c3057c3a31..cef38bcc06 100644 --- a/.github/workflows/release-please.yml +++ b/.github/workflows/release-please.yml @@ -7,6 +7,10 @@ on: jobs: release-please: + permissions: + contents: write # to create release commit (google-github-actions/release-please-action) + pull-requests: write # to create release PR (google-github-actions/release-please-action) + runs-on: ubuntu-latest steps: - uses: google-github-actions/release-please-action@v2 diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 7070df4e25..1cca2a81d2 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -7,6 +7,10 @@ on: branches: [ main ] pull_request: branches: [ main ] + +permissions: + contents: read # to fetch code (actions/checkout) + jobs: Lint_Python: runs-on: ubuntu-latest diff --git a/.github/workflows/visual-studio.yml b/.github/workflows/visual-studio.yml index 12125e5447..8abe36ecc2 100644 --- a/.github/workflows/visual-studio.yml +++ b/.github/workflows/visual-studio.yml @@ -6,6 +6,10 @@ on: branches: [ main ] pull_request: branches: [ main ] + +permissions: + contents: read # to fetch code (actions/checkout) + jobs: visual-studio: strategy: From 4a50fe31574217c4b2a798fc72b19947a64ceea1 Mon Sep 17 00:00:00 2001 From: Luke Karrys Date: Fri, 27 Oct 2023 10:49:33 -0700 Subject: [PATCH 387/551] chore: empty commit to add changelog entries from #2770 feat!: update engines.node to ^14.17.0 || ^16.13.0 || >=18.0.0 deps: nopt@^7.0.0 feat: replace npmlog with proc-log deps: standard@17.0.0 and fix linting errors deps: which@3.0.0 fix: promisify build command deps: glob@8.0.3 feat: drop rimraf dependency fix: use fs/promises in favor of fs.promises From d644ce48311edf090d0e920ad449e5766c757933 Mon Sep 17 00:00:00 2001 From: David Sanders Date: Fri, 27 Oct 2023 19:53:27 -0400 Subject: [PATCH 388/551] docs: update applicable GitHub links from master to main (#2843) Signed-off-by: David Sanders --- .github/ISSUE_TEMPLATE.md | 2 +- .github/PULL_REQUEST_TEMPLATE.md | 2 +- CONTRIBUTING.md | 2 +- README.md | 4 ++-- docs/README.md | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md index c6b213d7be..59aad48337 100644 --- a/.github/ISSUE_TEMPLATE.md +++ b/.github/ISSUE_TEMPLATE.md @@ -13,7 +13,7 @@ provide the basic information we require. --> -Please look thru your error log for the string `gyp info using node-gyp@` and if the version number is less than the [current release of node-gyp](https://github.com/nodejs/node-gyp/releases) then __please upgrade__ using the instructions at https://github.com/nodejs/node-gyp/blob/master/docs/Updating-npm-bundled-node-gyp.md and try your command again. +Please look thru your error log for the string `gyp info using node-gyp@` and if the version number is less than the [current release of node-gyp](https://github.com/nodejs/node-gyp/releases) then __please upgrade__ using the instructions at https://github.com/nodejs/node-gyp/blob/main/docs/Updating-npm-bundled-node-gyp.md and try your command again. Requests for help with [`node-sass` are very common](https://github.com/nodejs/node-gyp/issues?q=label%3A%22Node+Sass+--%3E+Dart+Sass%22). Please be aware that this package is deprecated, you should seek alternatives and avoid opening new issues about it here. diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index bcc4bb1a04..779897573a 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,7 +1,7 @@ ##### Checklist diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c1c50eab4e..5b977898f1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -3,7 +3,7 @@ ## Code of Conduct Please read the -[Code of Conduct](https://github.com/nodejs/admin/blob/master/CODE_OF_CONDUCT.md) +[Code of Conduct](https://github.com/nodejs/admin/blob/main/CODE_OF_CONDUCT.md) which explains the minimum behavior expectations for node-gyp contributors. diff --git a/README.md b/README.md index 8a7ad87a01..f46ee06308 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # `node-gyp` - Node.js native addon build tool -[![Build Status](https://github.com/nodejs/node-gyp/workflows/Tests/badge.svg?branch=master)](https://github.com/nodejs/node-gyp/actions?query=workflow%3ATests+branch%3Amaster) +[![Build Status](https://github.com/nodejs/node-gyp/workflows/Tests/badge.svg?branch=main)](https://github.com/nodejs/node-gyp/actions?query=workflow%3ATests+branch%3Amain) ![npm](https://img.shields.io/npm/dm/node-gyp) `node-gyp` is a cross-platform command-line tool written in Node.js for @@ -172,7 +172,7 @@ The **[docs](./docs/)** directory contains additional documentation on specific Some additional resources for Node.js native addons and writing `gyp` configuration files: * ["Going Native" a nodeschool.io tutorial](http://nodeschool.io/#goingnative) - * ["Hello World" node addon example](https://github.com/nodejs/node/tree/master/test/addons/hello-world) + * ["Hello World" node addon example](https://github.com/nodejs/node/tree/main/test/addons/hello-world) * [gyp user documentation](https://gyp.gsrc.io/docs/UserDocumentation.md) * [gyp input format reference](https://gyp.gsrc.io/docs/InputFormatReference.md) * [*"binding.gyp" files out in the wild* wiki page](./docs/binding.gyp-files-in-the-wild.md) diff --git a/docs/README.md b/docs/README.md index 487fb0a57e..3b8c86179d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,6 +1,6 @@ ## Versions of `node-gyp` that are earlier than v9.x.x -Please look thru your error log for the string `gyp info using node-gyp@` and if that version number is less than the [current release of node-gyp](https://github.com/nodejs/node-gyp/releases) then __please upgrade__ using [these instructions](https://github.com/nodejs/node-gyp/blob/master/docs/Updating-npm-bundled-node-gyp.md) and then try your command again. +Please look thru your error log for the string `gyp info using node-gyp@` and if that version number is less than the [current release of node-gyp](https://github.com/nodejs/node-gyp/releases) then __please upgrade__ using [these instructions](https://github.com/nodejs/node-gyp/blob/main/docs/Updating-npm-bundled-node-gyp.md) and then try your command again. ## `node-sass` is deprecated From 707927cd579205ef2b4b17e61c1cce24c056b452 Mon Sep 17 00:00:00 2001 From: DeeDeeG Date: Fri, 27 Oct 2023 20:12:14 -0400 Subject: [PATCH 389/551] feat(gyp): update gyp to v0.16.1 (#2923) * feat(gyp): update gyp to v0.15.1 * Add Python 3.12 to tests * Try to fix CI * Try specifying msvs-version * Modify the visual-studio matrix * Fix pythonLocation var * Fix Python tests * Get path * polish * feat(gyp): update gyp to v0.16.0 * feat(gyp): update gyp to v0.16.1 * CI: Don't install Python 'packaging' module (vendored in 'gyp-next' now) * Apply suggestions from code review * Upgrade to actions/checkout@v4 --------- Co-authored-by: Raymond Zhao <7199958+rzhao271@users.noreply.github.com> Co-authored-by: Christian Clauss --- .github/workflows/tests.yml | 12 +- .github/workflows/visual-studio.yml | 20 +- gyp/.flake8 | 4 - gyp/.github/workflows/Python_tests.yml | 13 +- gyp/.github/workflows/node-gyp.yml | 29 +- gyp/.github/workflows/nodejs-windows.yml | 6 +- gyp/AUTHORS | 1 + gyp/CHANGELOG.md | 42 + gyp/README.md | 23 + gyp/pylib/gyp/MSVSNew.py | 26 +- gyp/pylib/gyp/MSVSProject.py | 2 +- gyp/pylib/gyp/MSVSSettings.py | 2 +- gyp/pylib/gyp/__init__.py | 6 +- gyp/pylib/gyp/common.py | 21 +- gyp/pylib/gyp/easy_xml.py | 6 +- gyp/pylib/gyp/easy_xml_test.py | 4 + gyp/pylib/gyp/generator/analyzer.py | 26 +- gyp/pylib/gyp/generator/android.py | 2 +- gyp/pylib/gyp/generator/cmake.py | 9 +- .../gyp/generator/compile_commands_json.py | 9 +- gyp/pylib/gyp/generator/eclipse.py | 5 +- gyp/pylib/gyp/generator/make.py | 13 +- gyp/pylib/gyp/generator/msvs.py | 78 +- gyp/pylib/gyp/generator/ninja.py | 7 +- gyp/pylib/gyp/generator/xcode.py | 7 +- gyp/pylib/gyp/input.py | 57 +- gyp/pylib/gyp/msvs_emulation.py | 25 +- gyp/pylib/gyp/win_tool.py | 9 +- gyp/pylib/gyp/xcode_emulation.py | 12 +- gyp/pylib/gyp/xcodeproj_file.py | 41 +- gyp/pylib/packaging/LICENSE | 3 + gyp/pylib/packaging/LICENSE.APACHE | 177 +++ gyp/pylib/packaging/LICENSE.BSD | 23 + gyp/pylib/packaging/__init__.py | 15 + gyp/pylib/packaging/_elffile.py | 108 ++ gyp/pylib/packaging/_manylinux.py | 252 ++++ gyp/pylib/packaging/_musllinux.py | 83 ++ gyp/pylib/packaging/_parser.py | 359 ++++++ gyp/pylib/packaging/_structures.py | 61 + gyp/pylib/packaging/_tokenizer.py | 192 +++ gyp/pylib/packaging/markers.py | 252 ++++ gyp/pylib/packaging/metadata.py | 825 +++++++++++++ gyp/pylib/packaging/py.typed | 0 gyp/pylib/packaging/requirements.py | 90 ++ gyp/pylib/packaging/specifiers.py | 1030 +++++++++++++++++ gyp/pylib/packaging/tags.py | 553 +++++++++ gyp/pylib/packaging/utils.py | 172 +++ gyp/pylib/packaging/version.py | 563 +++++++++ gyp/pyproject.toml | 88 +- gyp/tools/pretty_sln.py | 4 +- gyp/tools/pretty_vcproj.py | 2 +- 51 files changed, 5122 insertions(+), 247 deletions(-) delete mode 100644 gyp/.flake8 create mode 100644 gyp/pylib/packaging/LICENSE create mode 100644 gyp/pylib/packaging/LICENSE.APACHE create mode 100644 gyp/pylib/packaging/LICENSE.BSD create mode 100644 gyp/pylib/packaging/__init__.py create mode 100644 gyp/pylib/packaging/_elffile.py create mode 100644 gyp/pylib/packaging/_manylinux.py create mode 100644 gyp/pylib/packaging/_musllinux.py create mode 100644 gyp/pylib/packaging/_parser.py create mode 100644 gyp/pylib/packaging/_structures.py create mode 100644 gyp/pylib/packaging/_tokenizer.py create mode 100644 gyp/pylib/packaging/markers.py create mode 100644 gyp/pylib/packaging/metadata.py create mode 100644 gyp/pylib/packaging/py.typed create mode 100644 gyp/pylib/packaging/requirements.py create mode 100644 gyp/pylib/packaging/specifiers.py create mode 100644 gyp/pylib/packaging/tags.py create mode 100644 gyp/pylib/packaging/utils.py create mode 100644 gyp/pylib/packaging/version.py diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 1cca2a81d2..310b4f06ea 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -25,7 +25,7 @@ jobs: max-parallel: 15 matrix: node: [16.x, 18.x, 20.x] - python: ["3.8", "3.11"] + python: ["3.8", "3.10", "3.12"] os: [macos-latest, ubuntu-latest, windows-latest] runs-on: ${{ matrix.os }} steps: @@ -56,5 +56,11 @@ jobs: # run: python -m pytest --doctest-modules - name: Environment Information run: npx envinfo - - name: Run Node tests - run: npm test + - name: Run Node tests (macOS or Linux) + if: runner.os != 'Windows' + shell: bash + run: npm test --python="${pythonLocation}/python" + - name: Run tests (Windows) + if: runner.os == 'Windows' + shell: pwsh + run: npm run test --python="${env:pythonLocation}\\python.exe" diff --git a/.github/workflows/visual-studio.yml b/.github/workflows/visual-studio.yml index 8abe36ecc2..8b796256da 100644 --- a/.github/workflows/visual-studio.yml +++ b/.github/workflows/visual-studio.yml @@ -16,22 +16,22 @@ jobs: fail-fast: false max-parallel: 8 matrix: - os: [windows-latest] - msvs-version: [2016, 2019, 2022] # https://github.com/actions/virtual-environments/tree/main/images/win + include: + - os: windows-2019 + msvs-verison: 2019 + - os: windows-2022 + msvs-version: 2022 runs-on: ${{ matrix.os }} steps: - name: Checkout Repository - uses: actions/checkout@v3 + uses: actions/checkout@v4 - name: Install Dependencies run: | npm install --no-progress - # npm audit fix --force - - name: Set Windows environment - if: startsWith(matrix.os, 'windows') - run: | - echo 'GYP_MSVS_VERSION=${{ matrix.msvs-version }}' >> $Env:GITHUB_ENV - echo 'GYP_MSVS_OVERRIDE_PATH=C:\\Dummy' >> $Env:GITHUB_ENV - name: Environment Information run: npx envinfo - name: Run Node tests - run: npm test + shell: pwsh + run: | + $pythonLocation = (Get-Command python).Source + npm run test --python="${pythonLocation}" --msvs-version="${{ matrix.msvs-version }}" diff --git a/gyp/.flake8 b/gyp/.flake8 deleted file mode 100644 index ea0c7680ef..0000000000 --- a/gyp/.flake8 +++ /dev/null @@ -1,4 +0,0 @@ -[flake8] -max-complexity = 101 -max-line-length = 88 -extend-ignore = E203 # whitespace before ':' to agree with psf/black diff --git a/gyp/.github/workflows/Python_tests.yml b/gyp/.github/workflows/Python_tests.yml index aad135027c..049d5fe50c 100644 --- a/gyp/.github/workflows/Python_tests.yml +++ b/gyp/.github/workflows/Python_tests.yml @@ -13,24 +13,25 @@ jobs: runs-on: ${{ matrix.os }} strategy: fail-fast: false - max-parallel: 8 + max-parallel: 5 matrix: os: [macos-latest, ubuntu-latest] # , windows-latest] - python-version: ["3.7", "3.8", "3.9", "3.10", "3.11-dev"] + python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} + allow-prereleases: true - name: Install dependencies run: | python -m pip install --upgrade pip setuptools pip install --editable ".[dev]" - run: ./gyp -V && ./gyp --version && gyp -V && gyp --version - - name: Lint with flake8 - run: flake8 . --ignore=E203,W503 --max-complexity=101 --max-line-length=88 --show-source --statistics - - name: Test with pytest + - name: Lint with ruff # See pyproject.toml for settings + run: ruff --output-format=github . + - name: Test with pytest # See pyproject.toml for settings run: pytest # - name: Run doctests with pytest # run: pytest --doctest-modules diff --git a/gyp/.github/workflows/node-gyp.yml b/gyp/.github/workflows/node-gyp.yml index 7cc1f9e075..ebe7497521 100644 --- a/gyp/.github/workflows/node-gyp.yml +++ b/gyp/.github/workflows/node-gyp.yml @@ -11,26 +11,33 @@ jobs: fail-fast: false matrix: os: [macos-latest, ubuntu-latest, windows-latest] - python: ["3.7", "3.10"] + python: ["3.8", "3.10", "3.12"] runs-on: ${{ matrix.os }} steps: - name: Clone gyp-next - uses: actions/checkout@v3 + uses: actions/checkout@v4 with: path: gyp-next - name: Clone nodejs/node-gyp - uses: actions/checkout@v3 + uses: actions/checkout@v4 with: repository: nodejs/node-gyp path: node-gyp - uses: actions/setup-node@v3 with: - node-version: 14.x + node-version: 18.x - uses: actions/setup-python@v4 with: python-version: ${{ matrix.python }} - - name: Install dependencies + allow-prereleases: true + - name: Install Python dependencies + run: | + cd gyp-next + python -m pip install --upgrade pip setuptools + pip install --editable . + pip uninstall -y gyp-next + - name: Install Node.js dependencies run: | cd node-gyp npm install --no-progress @@ -39,7 +46,15 @@ jobs: run: | rm -rf node-gyp/gyp cp -r gyp-next node-gyp/gyp - - name: Run tests + - name: Run tests (macOS or Linux) + if: runner.os != 'Windows' + shell: bash + run: | + cd node-gyp + npm test --python="${pythonLocation}/python" + - name: Run tests (Windows) + if: runner.os == 'Windows' + shell: pwsh run: | cd node-gyp - npm test + npm run test --python="${env:pythonLocation}\\python.exe" diff --git a/gyp/.github/workflows/nodejs-windows.yml b/gyp/.github/workflows/nodejs-windows.yml index 4e6c9548ff..3f52ff9ce7 100644 --- a/gyp/.github/workflows/nodejs-windows.yml +++ b/gyp/.github/workflows/nodejs-windows.yml @@ -9,14 +9,14 @@ on: jobs: build-windows: - runs-on: windows-2019 + runs-on: windows-latest steps: - name: Clone gyp-next - uses: actions/checkout@v3 + uses: actions/checkout@v4 with: path: gyp-next - name: Clone nodejs/node - uses: actions/checkout@v3 + uses: actions/checkout@v4 with: repository: nodejs/node path: node diff --git a/gyp/AUTHORS b/gyp/AUTHORS index f49a357b9e..c05d25b2cb 100644 --- a/gyp/AUTHORS +++ b/gyp/AUTHORS @@ -14,3 +14,4 @@ Tom Freudenberg Julien Brianceau Refael Ackermann Ujjwal Sharma +Christian Clauss diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md index 4b4968f6a4..483943e013 100644 --- a/gyp/CHANGELOG.md +++ b/gyp/CHANGELOG.md @@ -1,5 +1,47 @@ # Changelog +## [0.16.1](https://github.com/nodejs/gyp-next/compare/v0.16.0...v0.16.1) (2023-10-25) + + +### Bug Fixes + +* add quotes for command in msvs generator ([#217](https://github.com/nodejs/gyp-next/issues/217)) ([d3b7bcd](https://github.com/nodejs/gyp-next/commit/d3b7bcdec90d6c1b1affc15ece706e63007b7264)) + +## [0.16.0](https://github.com/nodejs/gyp-next/compare/v0.15.1...v0.16.0) (2023-10-23) + + +### Features + +* add VCToolsVersion for msvs ([#209](https://github.com/nodejs/gyp-next/issues/209)) ([0e35ab8](https://github.com/nodejs/gyp-next/commit/0e35ab812d890fb75cf89a19ea72bc93dd6ba186)) + +## [0.15.1](https://github.com/nodejs/gyp-next/compare/v0.15.0...v0.15.1) (2023-09-08) + + +### Bug Fixes + +* some Python lint issues ([#200](https://github.com/nodejs/gyp-next/issues/200)) ([d2dfe4e](https://github.com/nodejs/gyp-next/commit/d2dfe4e66b64c16b38bef984782db93d12674f05)) +* use generator_output as output_dir ([#191](https://github.com/nodejs/gyp-next/issues/191)) ([35ffeb1](https://github.com/nodejs/gyp-next/commit/35ffeb1da8ef3fc8311e2e812cff550568f7e8a2)) + +## [0.15.0](https://github.com/nodejs/gyp-next/compare/v0.14.1...v0.15.0) (2023-03-30) + + +### Features + +* **msvs:** add SpectreMitigation attribute ([#190](https://github.com/nodejs/gyp-next/issues/190)) ([853e464](https://github.com/nodejs/gyp-next/commit/853e4643b6737224a5aa0720a4108461a0230991)) + +## [0.14.1](https://github.com/nodejs/gyp-next/compare/v0.14.0...v0.14.1) (2023-02-19) + + +### Bug Fixes + +* flake8 extended-ignore ([#186](https://github.com/nodejs/gyp-next/issues/186)) ([c38493c](https://github.com/nodejs/gyp-next/commit/c38493c2556aa63b6dc40ab585c18aef5ca270d3)) +* No build_type in default_variables ([#183](https://github.com/nodejs/gyp-next/issues/183)) ([ac262fe](https://github.com/nodejs/gyp-next/commit/ac262fe82453c4e8dc47529338d157eb0b5ec0fb)) + + +### Documentation + +* README.md: Add pipx installation and run instructions ([#165](https://github.com/nodejs/gyp-next/issues/165)) ([4d28b15](https://github.com/nodejs/gyp-next/commit/4d28b155568dc35f11c7f86124d1dd42ba428bed)) + ## [0.14.0](https://github.com/nodejs/gyp-next/compare/v0.13.0...v0.14.0) (2022-10-08) diff --git a/gyp/README.md b/gyp/README.md index 9ffc2b21e8..be1d7b9ebf 100644 --- a/gyp/README.md +++ b/gyp/README.md @@ -5,3 +5,26 @@ Documents are available at [gyp.gsrc.io](https://gyp.gsrc.io), or you can check __gyp-next__ is [released](https://github.com/nodejs/gyp-next/releases) to the [__Python Packaging Index__](https://pypi.org/project/gyp-next) (PyPI) and can be installed with the command: * `python3 -m pip install gyp-next` + +When used as a command line utility, __gyp-next__ can also be installed with [pipx](https://pypa.github.io/pipx): +* `pipx install gyp-next` +``` +Installing to a new venv 'gyp-next' + installed package gyp-next 0.13.0, installed using Python 3.10.6 + These apps are now globally available + - gyp +done! ✨ 🌟 ✨ +``` + +Or to run __gyp-next__ directly without installing it: +* `pipx run gyp-next --help` +``` +NOTE: running app 'gyp' from 'gyp-next' +usage: usage: gyp [options ...] [build_file ...] + +options: + -h, --help show this help message and exit + --build CONFIGS configuration for build after project generation + --check check format of gyp files + [ ... ] +``` diff --git a/gyp/pylib/gyp/MSVSNew.py b/gyp/pylib/gyp/MSVSNew.py index d6b189760c..bc0e93d07f 100644 --- a/gyp/pylib/gyp/MSVSNew.py +++ b/gyp/pylib/gyp/MSVSNew.py @@ -285,19 +285,17 @@ def Write(self, writer=gyp.common.WriteOnDiff): "\tEndProjectSection\r\n" ) - if isinstance(e, MSVSFolder): - if e.items: - f.write("\tProjectSection(SolutionItems) = preProject\r\n") - for i in e.items: - f.write(f"\t\t{i} = {i}\r\n") - f.write("\tEndProjectSection\r\n") - - if isinstance(e, MSVSProject): - if e.dependencies: - f.write("\tProjectSection(ProjectDependencies) = postProject\r\n") - for d in e.dependencies: - f.write(f"\t\t{d.get_guid()} = {d.get_guid()}\r\n") - f.write("\tEndProjectSection\r\n") + if isinstance(e, MSVSFolder) and e.items: + f.write("\tProjectSection(SolutionItems) = preProject\r\n") + for i in e.items: + f.write(f"\t\t{i} = {i}\r\n") + f.write("\tEndProjectSection\r\n") + + if isinstance(e, MSVSProject) and e.dependencies: + f.write("\tProjectSection(ProjectDependencies) = postProject\r\n") + for d in e.dependencies: + f.write(f"\t\t{d.get_guid()} = {d.get_guid()}\r\n") + f.write("\tEndProjectSection\r\n") f.write("EndProject\r\n") @@ -353,7 +351,7 @@ def Write(self, writer=gyp.common.WriteOnDiff): # Folder mappings # Omit this section if there are no folders - if any([e.entries for e in all_entries if isinstance(e, MSVSFolder)]): + if any(e.entries for e in all_entries if isinstance(e, MSVSFolder)): f.write("\tGlobalSection(NestedProjects) = preSolution\r\n") for e in all_entries: if not isinstance(e, MSVSFolder): diff --git a/gyp/pylib/gyp/MSVSProject.py b/gyp/pylib/gyp/MSVSProject.py index f0cfabe834..629f3f61b4 100644 --- a/gyp/pylib/gyp/MSVSProject.py +++ b/gyp/pylib/gyp/MSVSProject.py @@ -79,7 +79,7 @@ def __init__(self, project_path, version, name, guid=None, platforms=None): self.files_section = ["Files"] # Keep a dict keyed on filename to speed up access. - self.files_dict = dict() + self.files_dict = {} def AddToolFile(self, path): """Adds a tool file to the project. diff --git a/gyp/pylib/gyp/MSVSSettings.py b/gyp/pylib/gyp/MSVSSettings.py index e89a971a3b..93633dbca1 100644 --- a/gyp/pylib/gyp/MSVSSettings.py +++ b/gyp/pylib/gyp/MSVSSettings.py @@ -141,7 +141,7 @@ class _Boolean(_Type): """Boolean settings, can have the values 'false' or 'true'.""" def _Validate(self, value): - if value != "true" and value != "false": + if value not in {"true", "false"}: raise ValueError("expected bool; got %r" % value) def ValidateMSVS(self, value): diff --git a/gyp/pylib/gyp/__init__.py b/gyp/pylib/gyp/__init__.py index 2aa39d0318..d6cc01307d 100755 --- a/gyp/pylib/gyp/__init__.py +++ b/gyp/pylib/gyp/__init__.py @@ -108,7 +108,9 @@ def Load( if default_variables["GENERATOR"] == "ninja": default_variables.setdefault( "PRODUCT_DIR_ABS", - os.path.join(output_dir, "out", default_variables["build_type"]), + os.path.join( + output_dir, "out", default_variables.get("build_type", "default") + ), ) else: default_variables.setdefault( @@ -622,7 +624,7 @@ def gyp_main(args): if options.generator_flags: gen_flags += options.generator_flags generator_flags = NameValueListToDict(gen_flags) - if DEBUG_GENERAL in gyp.debug.keys(): + if DEBUG_GENERAL in gyp.debug: DebugOutput(DEBUG_GENERAL, "generator_flags: %s", generator_flags) # Generate all requested formats (use a set in case we got one format request diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py index d77adee8af..b73a0c55b1 100644 --- a/gyp/pylib/gyp/common.py +++ b/gyp/pylib/gyp/common.py @@ -144,20 +144,16 @@ def RelativePath(path, relative_to, follow_path_symlink=True): # symlink, this option has no effect. # Convert to normalized (and therefore absolute paths). - if follow_path_symlink: - path = os.path.realpath(path) - else: - path = os.path.abspath(path) + path = os.path.realpath(path) if follow_path_symlink else os.path.abspath(path) relative_to = os.path.realpath(relative_to) # On Windows, we can't create a relative path to a different drive, so just # use the absolute path. - if sys.platform == "win32": - if ( - os.path.splitdrive(path)[0].lower() - != os.path.splitdrive(relative_to)[0].lower() - ): - return path + if sys.platform == "win32" and ( + os.path.splitdrive(path)[0].lower() + != os.path.splitdrive(relative_to)[0].lower() + ): + return path # Split the paths into components. path_split = path.split(os.path.sep) @@ -277,10 +273,7 @@ def EncodePOSIXShellArgument(argument): if not isinstance(argument, str): argument = str(argument) - if _quote.search(argument): - quote = '"' - else: - quote = "" + quote = '"' if _quote.search(argument) else "" encoded = quote + re.sub(_escape, r"\\\1", argument) + quote diff --git a/gyp/pylib/gyp/easy_xml.py b/gyp/pylib/gyp/easy_xml.py index bda1a47468..02567b2514 100644 --- a/gyp/pylib/gyp/easy_xml.py +++ b/gyp/pylib/gyp/easy_xml.py @@ -121,7 +121,11 @@ def WriteXmlIfChanged(content, path, encoding="utf-8", pretty=False, if win32 and os.linesep != "\r\n": xml_string = xml_string.replace("\n", "\r\n") - default_encoding = locale.getdefaultlocale()[1] + try: # getdefaultlocale() was removed in Python 3.11 + default_encoding = locale.getdefaultlocale()[1] + except AttributeError: + default_encoding = locale.getencoding() + if default_encoding and default_encoding.upper() != encoding.upper(): xml_string = xml_string.encode(encoding) diff --git a/gyp/pylib/gyp/easy_xml_test.py b/gyp/pylib/gyp/easy_xml_test.py index 342f693a32..2d9b15210d 100755 --- a/gyp/pylib/gyp/easy_xml_test.py +++ b/gyp/pylib/gyp/easy_xml_test.py @@ -76,6 +76,8 @@ def test_EasyXml_complex(self): '\'Debug|Win32\'" Label="Configuration">' "Application" "Unicode" + "SpectreLoadCF" + "14.36.32532" "" "" ) @@ -99,6 +101,8 @@ def test_EasyXml_complex(self): }, ["ConfigurationType", "Application"], ["CharacterSet", "Unicode"], + ["SpectreMitigation", "SpectreLoadCF"], + ["VCToolsVersion", "14.36.32532"], ], ] ) diff --git a/gyp/pylib/gyp/generator/analyzer.py b/gyp/pylib/gyp/generator/analyzer.py index f15df00c36..1334f2fca9 100644 --- a/gyp/pylib/gyp/generator/analyzer.py +++ b/gyp/pylib/gyp/generator/analyzer.py @@ -379,7 +379,7 @@ def _GenerateTargets(data, target_list, target_dicts, toplevel_dir, files, build target.is_executable = target_type == "executable" target.is_static_library = target_type == "static_library" target.is_or_has_linked_ancestor = ( - target_type == "executable" or target_type == "shared_library" + target_type in {"executable", "shared_library"} ) build_file = gyp.common.ParseQualifiedTarget(target_name)[0] @@ -433,14 +433,14 @@ def _GetUnqualifiedToTargetMapping(all_targets, to_find): if not to_find: return {}, [] to_find = set(to_find) - for target_name in all_targets.keys(): + for target_name in all_targets: extracted = gyp.common.ParseQualifiedTarget(target_name) if len(extracted) > 1 and extracted[1] in to_find: to_find.remove(extracted[1]) result[extracted[1]] = all_targets[target_name] if not to_find: return result, [] - return result, [x for x in to_find] + return result, list(to_find) def _DoesTargetDependOnMatchingTargets(target): @@ -451,8 +451,8 @@ def _DoesTargetDependOnMatchingTargets(target): if target.match_status == MATCH_STATUS_DOESNT_MATCH: return False if ( - target.match_status == MATCH_STATUS_MATCHES - or target.match_status == MATCH_STATUS_MATCHES_BY_DEPENDENCY + target.match_status in {MATCH_STATUS_MATCHES, + MATCH_STATUS_MATCHES_BY_DEPENDENCY} ): return True for dep in target.deps: @@ -683,11 +683,9 @@ def find_matching_test_target_names(self): ) test_target_names_contains_all = "all" in self._test_target_names if test_target_names_contains_all: - test_targets = [ - x for x in (set(test_targets_no_all) | set(self._root_targets)) - ] + test_targets = list(set(test_targets_no_all) | set(self._root_targets)) else: - test_targets = [x for x in test_targets_no_all] + test_targets = list(test_targets_no_all) print("supplied test_targets") for target_name in self._test_target_names: print("\t", target_name) @@ -702,9 +700,9 @@ def find_matching_test_target_names(self): if matching_test_targets_contains_all: # Remove any of the targets for all that were not explicitly supplied, # 'all' is subsequentely added to the matching names below. - matching_test_targets = [ - x for x in (set(matching_test_targets) & set(test_targets_no_all)) - ] + matching_test_targets = list( + set(matching_test_targets) & set(test_targets_no_all) + ) print("matched test_targets") for target in matching_test_targets: print("\t", target.name) @@ -729,9 +727,7 @@ def find_matching_compile_target_names(self): self._supplied_target_names_no_all(), self._unqualified_mapping ) if "all" in self._supplied_target_names(): - supplied_targets = [ - x for x in (set(supplied_targets) | set(self._root_targets)) - ] + supplied_targets = list(set(supplied_targets) | set(self._root_targets)) print("Supplied test_targets & compile_targets") for target in supplied_targets: print("\t", target.name) diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py index cdf1a4832c..d3c97c666d 100644 --- a/gyp/pylib/gyp/generator/android.py +++ b/gyp/pylib/gyp/generator/android.py @@ -697,7 +697,7 @@ def ComputeOutputParts(self, spec): target, ) - if self.type != "static_library" and self.type != "shared_library": + if self.type not in {"static_library", "shared_library"}: target_prefix = spec.get("product_prefix", target_prefix) target = spec.get("product_name", target) product_ext = spec.get("product_extension") diff --git a/gyp/pylib/gyp/generator/cmake.py b/gyp/pylib/gyp/generator/cmake.py index c95d18415c..320a891aa8 100644 --- a/gyp/pylib/gyp/generator/cmake.py +++ b/gyp/pylib/gyp/generator/cmake.py @@ -103,7 +103,7 @@ def NormjoinPathForceCMakeSource(base_path, rel_path): """ if os.path.isabs(rel_path): return rel_path - if any([rel_path.startswith(var) for var in FULL_PATH_VARS]): + if any(rel_path.startswith(var) for var in FULL_PATH_VARS): return rel_path # TODO: do we need to check base_path for absolute variables as well? return os.path.join( @@ -328,7 +328,7 @@ def WriteActions(target_name, actions, extra_sources, extra_deps, path_to_gyp, o def NormjoinRulePathForceCMakeSource(base_path, rel_path, rule_source): if rel_path.startswith(("${RULE_INPUT_PATH}", "${RULE_INPUT_DIRNAME}")): - if any([rule_source.startswith(var) for var in FULL_PATH_VARS]): + if any(rule_source.startswith(var) for var in FULL_PATH_VARS): return rel_path return NormjoinPathForceCMakeSource(base_path, rel_path) @@ -929,10 +929,7 @@ def WriteTarget( product_prefix = spec.get("product_prefix", default_product_prefix) product_name = spec.get("product_name", default_product_name) product_ext = spec.get("product_extension") - if product_ext: - product_ext = "." + product_ext - else: - product_ext = default_product_ext + product_ext = "." + product_ext if product_ext else default_product_ext SetTargetProperty(output, cmake_target_name, "PREFIX", product_prefix) SetTargetProperty( diff --git a/gyp/pylib/gyp/generator/compile_commands_json.py b/gyp/pylib/gyp/generator/compile_commands_json.py index f330a04dea..0ffa3bb598 100644 --- a/gyp/pylib/gyp/generator/compile_commands_json.py +++ b/gyp/pylib/gyp/generator/compile_commands_json.py @@ -34,7 +34,7 @@ def IsMac(params): - return "mac" == gyp.common.GetFlavor(params) + return gyp.common.GetFlavor(params) == "mac" def CalculateVariables(default_variables, params): @@ -93,7 +93,7 @@ def resolve(filename): gyp.common.EncodePOSIXShellArgument(file), ) ) - commands.append(dict(command=command, directory=output_dir, file=file)) + commands.append({"command": command, "directory": output_dir, "file": file}) def GenerateOutput(target_list, target_dicts, data, params): @@ -108,7 +108,10 @@ def GenerateOutput(target_list, target_dicts, data, params): cwd = os.path.dirname(build_file) AddCommandsForTarget(cwd, target, params, per_config_commands) - output_dir = params["generator_flags"].get("output_dir", "out") + try: + output_dir = params["options"].generator_output + except (AttributeError, KeyError): + output_dir = params["generator_flags"].get("output_dir", "out") for configuration_name, commands in per_config_commands.items(): filename = os.path.join(output_dir, configuration_name, "compile_commands.json") gyp.common.EnsureDirExists(filename) diff --git a/gyp/pylib/gyp/generator/eclipse.py b/gyp/pylib/gyp/generator/eclipse.py index a851b4db75..52aeae6050 100644 --- a/gyp/pylib/gyp/generator/eclipse.py +++ b/gyp/pylib/gyp/generator/eclipse.py @@ -248,10 +248,7 @@ def GetAllDefines(target_list, target_dicts, data, config_name, params, compiler continue cpp_line_parts = cpp_line.split(" ", 2) key = cpp_line_parts[1] - if len(cpp_line_parts) >= 3: - val = cpp_line_parts[2] - else: - val = "1" + val = cpp_line_parts[2] if len(cpp_line_parts) >= 3 else "1" all_defines[key] = val return all_defines diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py index 05bb609d16..1b9974948e 100644 --- a/gyp/pylib/gyp/generator/make.py +++ b/gyp/pylib/gyp/generator/make.py @@ -681,10 +681,7 @@ def WriteRootHeaderSuffixRules(writer): def Compilable(filename): """Return true if the file is compilable (should be in OBJS).""" - for res in (filename.endswith(e) for e in COMPILABLE_EXTENSIONS): - if res: - return True - return False + return any(res for res in (filename.endswith(e) for e in COMPILABLE_EXTENSIONS)) def Linkable(filename): @@ -778,7 +775,7 @@ def __init__(self, generator_flags, flavor): self.suffix_rules_objdir2 = {} # Generate suffix rules for all compilable extensions. - for ext in COMPILABLE_EXTENSIONS.keys(): + for ext in COMPILABLE_EXTENSIONS: # Suffix rules for source folder. self.suffix_rules_srcdir.update( { @@ -1066,7 +1063,7 @@ def WriteActions( # libraries, but until everything is made cross-compile safe, also use # target libraries. # TODO(piman): when everything is cross-compile safe, remove lib.target - if self.flavor == "zos" or self.flavor == "aix": + if self.flavor in {"zos", "aix"}: self.WriteLn( "cmd_%s = LIBPATH=$(builddir)/lib.host:" "$(builddir)/lib.target:$$LIBPATH; " @@ -1991,7 +1988,7 @@ def WriteTarget( and "product_dir" not in spec and self.toolset == "target" ): - # On macOS, products are created in install_path immediately. + # On mac, products are created in install_path immediately. assert install_path == self.output, f"{install_path} != {self.output}" # Point the target alias to the final binary output. @@ -2031,7 +2028,7 @@ def WriteTarget( installable_deps.append( self.GetUnversionedSidedeckFromSidedeck(install_path) ) - if self.output != self.alias and self.alias != self.target: + if self.alias not in (self.output, self.target): self.WriteMakeRule( [self.alias], installable_deps, diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py index fd95005784..13b0794b4d 100644 --- a/gyp/pylib/gyp/generator/msvs.py +++ b/gyp/pylib/gyp/generator/msvs.py @@ -164,7 +164,7 @@ def _FixPath(path, separator="\\"): fixpath_prefix and path and not os.path.isabs(path) - and not path[0] == "$" + and path[0] != "$" and not _IsWindowsAbsPath(path) ): path = os.path.join(fixpath_prefix, path) @@ -281,9 +281,9 @@ def _ToolSetOrAppend(tools, tool_name, setting, value, only_if_unset=False): else: value = [i.replace("/", "\\") for i in value] if not tools.get(tool_name): - tools[tool_name] = dict() + tools[tool_name] = {} tool = tools[tool_name] - if "CompileAsWinRT" == setting: + if setting == "CompileAsWinRT": return if tool.get(setting): if only_if_unset: @@ -412,10 +412,7 @@ def _BuildCommandLineForRuleRaw( return input_dir_preamble + cmd else: # Convert cat --> type to mimic unix. - if cmd[0] == "cat": - command = ["type"] - else: - command = [cmd[0].replace("/", "\\")] + command = ["type"] if cmd[0] == "cat" else [cmd[0].replace("/", "\\")] # Add call before command to ensure that commands can be tied together one # after the other without aborting in Incredibuild, since IB makes a bat # file out of the raw command string, and some commands (like python) are @@ -438,6 +435,7 @@ def _BuildCommandLineForRuleRaw( # Support a mode for using cmd directly. # Convert any paths to native form (first element is used directly). # TODO(quote): regularize quoting path names throughout the module + command[1] = '"%s"' % command[1] arguments = ['"%s"' % i for i in arguments] # Collapse into a single command. return input_dir_preamble + " ".join(command + arguments) @@ -687,7 +685,7 @@ def _GenerateExternalRules(rules, output_dir, spec, sources, options, actions_to all_outputs.update(OrderedSet(outputs)) # Only use one target from each rule as the dependency for # 'all' so we don't try to build each rule multiple times. - first_outputs.append(list(outputs)[0]) + first_outputs.append(next(iter(outputs))) # Get the unique output directories for this rule. output_dirs = [os.path.split(i)[0] for i in outputs] for od in output_dirs: @@ -756,7 +754,7 @@ def _EscapeEnvironmentVariableExpansion(s): Returns: The escaped string. - """ # noqa: E731,E123,E501 + """ s = s.replace("%", "%%") return s @@ -1189,7 +1187,7 @@ def _AddConfigurationToMSVSProject(p, spec, config_type, config_name, config): precompiled_header = config.get("msvs_precompiled_header") # Prepare the list of tools as a dictionary. - tools = dict() + tools = {} # Add in user specified msvs_settings. msvs_settings = config.get("msvs_settings", {}) MSVSSettings.ValidateMSVSSettings(msvs_settings) @@ -1384,10 +1382,7 @@ def _GetDefines(config): """ defines = [] for d in config.get("defines", []): - if type(d) == list: - fd = "=".join([str(dpart) for dpart in d]) - else: - fd = str(d) + fd = "=".join([str(dpart) for dpart in d]) if isinstance(d, list) else str(d) defines.append(fd) return defines @@ -1578,10 +1573,10 @@ def _AdjustSourcesAndConvertToFilterHierarchy( # such as ../../src/modules/module1 etc. if version.UsesVcxproj(): while ( - all([isinstance(s, MSVSProject.Filter) for s in sources]) + all(isinstance(s, MSVSProject.Filter) for s in sources) and len({s.name for s in sources}) == 1 ): - assert all([len(s.contents) == 1 for s in sources]) + assert all(len(s.contents) == 1 for s in sources) sources = [s.contents[0] for s in sources] else: while len(sources) == 1 and isinstance(sources[0], MSVSProject.Filter): @@ -1598,10 +1593,7 @@ def _IdlFilesHandledNonNatively(spec, sources): if rule["extension"] == "idl" and int(rule.get("msvs_external_rule", 0)): using_idl = True break - if using_idl: - excluded_idl = [i for i in sources if i.endswith(".idl")] - else: - excluded_idl = [] + excluded_idl = [i for i in sources if i.endswith(".idl")] if using_idl else [] return excluded_idl @@ -1819,7 +1811,7 @@ def _GetPathDict(root, path): parent, folder = os.path.split(path) parent_dict = _GetPathDict(root, parent) if folder not in parent_dict: - parent_dict[folder] = dict() + parent_dict[folder] = {} return parent_dict[folder] @@ -3013,18 +3005,26 @@ def _GetMSBuildConfigurationDetails(spec, build_file): msbuild_attributes = _GetMSBuildAttributes(spec, settings, build_file) condition = _GetConfigurationCondition(name, settings, spec) character_set = msbuild_attributes.get("CharacterSet") + vctools_version = msbuild_attributes.get("VCToolsVersion") config_type = msbuild_attributes.get("ConfigurationType") _AddConditionalProperty(properties, condition, "ConfigurationType", config_type) + spectre_mitigation = msbuild_attributes.get('SpectreMitigation') + if spectre_mitigation: + _AddConditionalProperty(properties, condition, "SpectreMitigation", + spectre_mitigation) if config_type == "Driver": _AddConditionalProperty(properties, condition, "DriverType", "WDM") _AddConditionalProperty( properties, condition, "TargetVersion", _ConfigTargetVersion(settings) ) - if character_set: - if "msvs_enable_winrt" not in spec: - _AddConditionalProperty( - properties, condition, "CharacterSet", character_set - ) + if character_set and "msvs_enable_winrt" not in spec: + _AddConditionalProperty( + properties, condition, "CharacterSet", character_set + ) + if vctools_version and "msvs_enable_winrt" not in spec: + _AddConditionalProperty( + properties, condition, "VCToolsVersion", vctools_version + ) return _GetMSBuildPropertyGroup(spec, "Configuration", properties) @@ -3104,6 +3104,10 @@ def _ConvertMSVSBuildAttributes(spec, config, build_file): msbuild_attributes[a] = _ConvertMSVSCharacterSet(msvs_attributes[a]) elif a == "ConfigurationType": msbuild_attributes[a] = _ConvertMSVSConfigurationType(msvs_attributes[a]) + elif a == "SpectreMitigation": + msbuild_attributes[a] = msvs_attributes[a] + elif a == "VCToolsVersion": + msbuild_attributes[a] = msvs_attributes[a] else: print("Warning: Do not know how to convert MSVS attribute " + a) return msbuild_attributes @@ -3326,15 +3330,14 @@ def _GetMSBuildToolSettingsSections(spec, configurations): for tool_name, tool_settings in sorted(msbuild_settings.items()): # Skip the tool named '' which is a holder of global settings handled # by _GetMSBuildConfigurationGlobalProperties. - if tool_name: - if tool_settings: - tool = [tool_name] - for name, value in sorted(tool_settings.items()): - formatted_value = _GetValueFormattedForMSBuild( - tool_name, name, value - ) - tool.append([name, formatted_value]) - group.append(tool) + if tool_name and tool_settings: + tool = [tool_name] + for name, value in sorted(tool_settings.items()): + formatted_value = _GetValueFormattedForMSBuild( + tool_name, name, value + ) + tool.append([name, formatted_value]) + group.append(tool) groups.append(group) return groups @@ -3462,10 +3465,7 @@ def _GetValueFormattedForMSBuild(tool_name, name, value): "Link": ["AdditionalOptions"], "Lib": ["AdditionalOptions"], } - if tool_name in exceptions and name in exceptions[tool_name]: - char = " " - else: - char = ";" + char = " " if name in exceptions.get(tool_name, []) else ";" formatted_value = char.join( [MSVSSettings.ConvertVCMacrosToMSBuild(i) for i in value] ) diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py index ca04ee13a1..8ba341e96d 100644 --- a/gyp/pylib/gyp/generator/ninja.py +++ b/gyp/pylib/gyp/generator/ninja.py @@ -1815,10 +1815,7 @@ def ComputeOutputFileName(self, spec, type=None): "executable": default_variables["EXECUTABLE_SUFFIX"], } extension = spec.get("product_extension") - if extension: - extension = "." + extension - else: - extension = DEFAULT_EXTENSION.get(type, "") + extension = "." + extension if extension else DEFAULT_EXTENSION.get(type, "") if "product_name" in spec: # If we were given an explicit name, use that. @@ -2533,7 +2530,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name description="SOLINK $lib", restat=True, command=mtime_preserving_solink_base - % {"suffix": "@$link_file_list"}, # noqa: E501 + % {"suffix": "@$link_file_list"}, rspfile="$link_file_list", rspfile_content=( "-Wl,--whole-archive $in $solibs -Wl," "--no-whole-archive $libs" diff --git a/gyp/pylib/gyp/generator/xcode.py b/gyp/pylib/gyp/generator/xcode.py index 2f4d17e514..1ac672c387 100644 --- a/gyp/pylib/gyp/generator/xcode.py +++ b/gyp/pylib/gyp/generator/xcode.py @@ -439,7 +439,7 @@ def Finalize2(self, xcode_targets, xcode_target_to_target_dict): # it opens the project file, which will result in unnecessary diffs. # TODO(mark): This is evil because it relies on internal knowledge of # PBXProject._other_pbxprojects. - for other_pbxproject in self.project._other_pbxprojects.keys(): + for other_pbxproject in self.project._other_pbxprojects: self.project.AddOrGetProjectReference(other_pbxproject) self.project.SortRemoteProductReferences() @@ -1118,10 +1118,7 @@ def GenerateOutput(target_list, target_dicts, data, params): for concrete_output_index, concrete_output in enumerate( concrete_outputs ): - if concrete_output_index == 0: - bol = "" - else: - bol = " " + bol = "" if concrete_output_index == 0 else " " makefile.write(f"{bol}{concrete_output} \\\n") concrete_output_dir = posixpath.dirname(concrete_output) diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py index d9699a0a50..8f39519dee 100644 --- a/gyp/pylib/gyp/input.py +++ b/gyp/pylib/gyp/input.py @@ -16,9 +16,9 @@ import sys import threading import traceback -from distutils.version import StrictVersion from gyp.common import GypError from gyp.common import OrderedSet +from packaging.version import Version # A list of types that are treated as linkable. linkable_types = [ @@ -225,7 +225,7 @@ def LoadOneBuildFile(build_file_path, data, aux_data, includes, is_target, check return data[build_file_path] if os.path.exists(build_file_path): - build_file_contents = open(build_file_path, encoding='utf-8').read() + build_file_contents = open(build_file_path, encoding="utf-8").read() else: raise GypError(f"{build_file_path} not found (cwd: {os.getcwd()})") @@ -870,10 +870,7 @@ def ExpandVariables(input, phase, variables, build_file): # This works around actions/rules which have more inputs than will # fit on the command line. if file_list: - if type(contents) is list: - contents_list = contents - else: - contents_list = contents.split(" ") + contents_list = contents if type(contents) is list else contents.split(" ") replacement = contents_list[0] if os.path.isabs(replacement): raise GypError('| cannot handle absolute paths, got "%s"' % replacement) @@ -1183,7 +1180,7 @@ def EvalSingleCondition(cond_expr, true_dict, false_dict, phase, variables, buil else: ast_code = compile(cond_expr_expanded, "", "eval") cached_conditions_asts[cond_expr_expanded] = ast_code - env = {"__builtins__": {}, "v": StrictVersion} + env = {"__builtins__": {}, "v": Version} if eval(ast_code, env, variables): return true_dict return false_dict @@ -1579,14 +1576,12 @@ def ExpandWildcardDependencies(targets, data): continue dependency_target_name = dependency_target_dict["target_name"] if ( - dependency_target != "*" - and dependency_target != dependency_target_name + dependency_target not in {"*", dependency_target_name} ): continue dependency_target_toolset = dependency_target_dict["toolset"] if ( - dependency_toolset != "*" - and dependency_toolset != dependency_target_toolset + dependency_toolset not in {"*", dependency_target_toolset} ): continue dependency = gyp.common.QualifiedTarget( @@ -1630,15 +1625,14 @@ def RemoveSelfDependencies(targets): dependencies = target_dict.get(dependency_key, []) if dependencies: for t in dependencies: - if t == target_name: - if ( - targets[t] - .get("variables", {}) - .get("prune_self_dependency", 0) - ): - target_dict[dependency_key] = Filter( - dependencies, target_name - ) + if t == target_name and ( + targets[t] + .get("variables", {}) + .get("prune_self_dependency", 0) + ): + target_dict[dependency_key] = Filter( + dependencies, target_name + ) def RemoveLinkDependenciesFromNoneTargets(targets): @@ -2238,10 +2232,7 @@ def is_in_set_or_list(x, s, items): singleton = False if type(item) in (str, int): # The cheap and easy case. - if is_paths: - to_item = MakePathRelative(to_file, fro_file, item) - else: - to_item = item + to_item = MakePathRelative(to_file, fro_file, item) if is_paths else item if not (type(item) is str and item.startswith("-")): # Any string that doesn't begin with a "-" is a singleton - it can @@ -2467,10 +2458,7 @@ def SetUpConfigurations(target, target_dict): new_configuration_dict = {} for (key, target_val) in target_dict.items(): key_ext = key[-1:] - if key_ext in key_suffixes: - key_base = key[:-1] - else: - key_base = key + key_base = key[:-1] if key_ext in key_suffixes else key if key_base not in non_configuration_keys: new_configuration_dict[key] = gyp.simple_copy.deepcopy(target_val) @@ -2482,7 +2470,7 @@ def SetUpConfigurations(target, target_dict): merged_configurations[configuration] = new_configuration_dict # Put the new configurations back into the target dict as a configuration. - for configuration in merged_configurations.keys(): + for configuration in merged_configurations: target_dict["configurations"][configuration] = merged_configurations[ configuration ] @@ -2499,19 +2487,16 @@ def SetUpConfigurations(target, target_dict): delete_keys = [] for key in target_dict: key_ext = key[-1:] - if key_ext in key_suffixes: - key_base = key[:-1] - else: - key_base = key + key_base = key[:-1] if key_ext in key_suffixes else key if key_base not in non_configuration_keys: delete_keys.append(key) for key in delete_keys: del target_dict[key] # Check the configurations to see if they contain invalid keys. - for configuration in target_dict["configurations"].keys(): + for configuration in target_dict["configurations"]: configuration_dict = target_dict["configurations"][configuration] - for key in configuration_dict.keys(): + for key in configuration_dict: if key in invalid_configuration_keys: raise GypError( "%s not allowed in the %s configuration, found in " @@ -2554,7 +2539,7 @@ def ProcessListFiltersInDict(name, the_dict): del_lists = [] for key, value in the_dict.items(): operation = key[-1] - if operation != "!" and operation != "/": + if operation not in {"!", "/"}: continue if type(value) is not list: diff --git a/gyp/pylib/gyp/msvs_emulation.py b/gyp/pylib/gyp/msvs_emulation.py index 5b9c2712e0..38fa21dd66 100644 --- a/gyp/pylib/gyp/msvs_emulation.py +++ b/gyp/pylib/gyp/msvs_emulation.py @@ -93,7 +93,7 @@ def _AddPrefix(element, prefix): if element is None: return element # Note, not Iterable because we don't want to handle strings like that. - if isinstance(element, list) or isinstance(element, tuple): + if isinstance(element, (list, tuple)): return [prefix + e for e in element] else: return prefix + element @@ -105,7 +105,7 @@ def _DoRemapping(element, map): if map is not None and element is not None: if not callable(map): map = map.get # Assume it's a dict, otherwise a callable to do the remap. - if isinstance(element, list) or isinstance(element, tuple): + if isinstance(element, (list, tuple)): element = filter(None, [map(elem) for elem in element]) else: element = map(element) @@ -117,7 +117,7 @@ def _AppendOrReturn(append, element): then add |element| to it, adding each item in |element| if it's a list or tuple.""" if append is not None and element is not None: - if isinstance(element, list) or isinstance(element, tuple): + if isinstance(element, (list, tuple)): append.extend(element) else: append.append(element) @@ -183,7 +183,7 @@ def ExtractSharedMSVSSystemIncludes(configs, generator_flags): expanded_system_includes = OrderedSet( [ExpandMacros(include, env) for include in all_system_includes] ) - if any(["$" in include for include in expanded_system_includes]): + if any("$" in include for include in expanded_system_includes): # Some path relies on target-specific variables, bail. return None @@ -255,10 +255,7 @@ def GetVSMacroEnv(self, base_to_build=None, config=None): """Get a dict of variables mapping internal VS macro names to their gyp equivalents.""" target_arch = self.GetArch(config) - if target_arch == "x86": - target_platform = "Win32" - else: - target_platform = target_arch + target_platform = "Win32" if target_arch == "x86" else target_arch target_name = self.spec.get("product_prefix", "") + self.spec.get( "product_name", self.spec["target_name"] ) @@ -738,10 +735,7 @@ def GetLdflags( # TODO(scottmg): This should sort of be somewhere else (not really a flag). ld("AdditionalDependencies", prefix="") - if self.GetArch(config) == "x86": - safeseh_default = "true" - else: - safeseh_default = None + safeseh_default = "true" if self.GetArch(config) == "x86" else None ld( "ImageHasSafeExceptionHandlers", map={"false": ":NO", "true": ""}, @@ -960,15 +954,12 @@ def GetRuleShellFlags(self, rule): def _HasExplicitRuleForExtension(self, spec, extension): """Determine if there's an explicit rule for a particular extension.""" - for rule in spec.get("rules", []): - if rule["extension"] == extension: - return True - return False + return any(rule["extension"] == extension for rule in spec.get("rules", [])) def _HasExplicitIdlActions(self, spec): """Determine if an action should not run midl for .idl files.""" return any( - [action.get("explicit_idl_action", 0) for action in spec.get("actions", [])] + action.get("explicit_idl_action", 0) for action in spec.get("actions", []) ) def HasExplicitIdlRulesOrActions(self, spec): diff --git a/gyp/pylib/gyp/win_tool.py b/gyp/pylib/gyp/win_tool.py index 638eee4002..171d729574 100755 --- a/gyp/pylib/gyp/win_tool.py +++ b/gyp/pylib/gyp/win_tool.py @@ -219,11 +219,10 @@ def ExecLinkWithManifests( our_manifest = "%(out)s.manifest" % variables # Load and normalize the manifests. mt.exe sometimes removes whitespace, # and sometimes doesn't unfortunately. - with open(our_manifest) as our_f: - with open(assert_manifest) as assert_f: - translator = str.maketrans('', '', string.whitespace) - our_data = our_f.read().translate(translator) - assert_data = assert_f.read().translate(translator) + with open(our_manifest) as our_f, open(assert_manifest) as assert_f: + translator = str.maketrans("", "", string.whitespace) + our_data = our_f.read().translate(translator) + assert_data = assert_f.read().translate(translator) if our_data != assert_data: os.unlink(out) diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py index a75d8eeab7..29caf1ce7f 100644 --- a/gyp/pylib/gyp/xcode_emulation.py +++ b/gyp/pylib/gyp/xcode_emulation.py @@ -685,10 +685,7 @@ def GetCflags(self, configname, arch=None): if platform_root: cflags.append("-F" + platform_root + "/Developer/Library/Frameworks/") - if sdk_root: - framework_root = sdk_root - else: - framework_root = "" + framework_root = sdk_root if sdk_root else "" config = self.spec["configurations"][self.configname] framework_dirs = config.get("mac_framework_dirs", []) for directory in framework_dirs: @@ -1248,10 +1245,7 @@ def _AdjustLibrary(self, library, config_name=None): l_flag = "-framework " + os.path.splitext(os.path.basename(library))[0] else: m = self.library_re.match(library) - if m: - l_flag = "-l" + m.group(1) - else: - l_flag = library + l_flag = "-l" + m.group(1) if m else library sdk_root = self._SdkPath(config_name) if not sdk_root: @@ -1545,7 +1539,7 @@ def CLTVersion(): except GypError: continue - regex = re.compile(r'Command Line Tools for Xcode\s+(?P\S+)') + regex = re.compile(r"Command Line Tools for Xcode\s+(?P\S+)") try: output = GetStdout(["/usr/sbin/softwareupdate", "--history"]) return re.search(regex, output).groupdict()["version"] diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py index 4e0ec5e882..33c667c266 100644 --- a/gyp/pylib/gyp/xcodeproj_file.py +++ b/gyp/pylib/gyp/xcodeproj_file.py @@ -971,7 +971,7 @@ def __init__(self, properties=None, id=None, parent=None): if "path" in self._properties and "name" not in self._properties: path = self._properties["path"] name = posixpath.basename(path) - if name != "" and path != name: + if name not in ("", path): self.SetProperty("name", name) if "path" in self._properties and ( @@ -2355,9 +2355,8 @@ def __init__( # property was supplied, set "productName" if it is not present. Also set # the "PRODUCT_NAME" build setting in each configuration, but only if # the setting is not present in any build configuration. - if "name" in self._properties: - if "productName" not in self._properties: - self.SetProperty("productName", self._properties["name"]) + if "name" in self._properties and "productName" not in self._properties: + self.SetProperty("productName", self._properties["name"]) if "productName" in self._properties: if "buildConfigurationList" in self._properties: @@ -2547,13 +2546,12 @@ def __init__( force_extension = suffix[1:] if ( - self._properties["productType"] - == "com.apple.product-type-bundle.unit.test" - or self._properties["productType"] - == "com.apple.product-type-bundle.ui-testing" - ): - if force_extension is None: - force_extension = suffix[1:] + self._properties["productType"] in { + "com.apple.product-type-bundle.unit.test", + "com.apple.product-type-bundle.ui-testing" + } + ) and force_extension is None: + force_extension = suffix[1:] if force_extension is not None: # If it's a wrapper (bundle), set WRAPPER_EXTENSION. @@ -2636,10 +2634,13 @@ def HeadersPhase(self): # frameworks phases, if any. insert_at = len(self._properties["buildPhases"]) for index, phase in enumerate(self._properties["buildPhases"]): - if ( - isinstance(phase, PBXResourcesBuildPhase) - or isinstance(phase, PBXSourcesBuildPhase) - or isinstance(phase, PBXFrameworksBuildPhase) + if isinstance( + phase, + ( + PBXResourcesBuildPhase, + PBXSourcesBuildPhase, + PBXFrameworksBuildPhase, + ), ): insert_at = index break @@ -2658,9 +2659,7 @@ def ResourcesPhase(self): # phases, if any. insert_at = len(self._properties["buildPhases"]) for index, phase in enumerate(self._properties["buildPhases"]): - if isinstance(phase, PBXSourcesBuildPhase) or isinstance( - phase, PBXFrameworksBuildPhase - ): + if isinstance(phase, (PBXSourcesBuildPhase, PBXFrameworksBuildPhase)): insert_at = index break @@ -2701,8 +2700,10 @@ def AddDependency(self, other): other._properties["productType"] == static_library_type or ( ( - other._properties["productType"] == shared_library_type - or other._properties["productType"] == framework_type + other._properties["productType"] in { + shared_library_type, + framework_type + } ) and ( (not other.HasBuildSetting("MACH_O_TYPE")) diff --git a/gyp/pylib/packaging/LICENSE b/gyp/pylib/packaging/LICENSE new file mode 100644 index 0000000000..6f62d44e4e --- /dev/null +++ b/gyp/pylib/packaging/LICENSE @@ -0,0 +1,3 @@ +This software is made available under the terms of *either* of the licenses +found in LICENSE.APACHE or LICENSE.BSD. Contributions to this software is made +under the terms of *both* these licenses. diff --git a/gyp/pylib/packaging/LICENSE.APACHE b/gyp/pylib/packaging/LICENSE.APACHE new file mode 100644 index 0000000000..f433b1a53f --- /dev/null +++ b/gyp/pylib/packaging/LICENSE.APACHE @@ -0,0 +1,177 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS diff --git a/gyp/pylib/packaging/LICENSE.BSD b/gyp/pylib/packaging/LICENSE.BSD new file mode 100644 index 0000000000..42ce7b75c9 --- /dev/null +++ b/gyp/pylib/packaging/LICENSE.BSD @@ -0,0 +1,23 @@ +Copyright (c) Donald Stufft and individual contributors. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + 1. Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/gyp/pylib/packaging/__init__.py b/gyp/pylib/packaging/__init__.py new file mode 100644 index 0000000000..5fd9183831 --- /dev/null +++ b/gyp/pylib/packaging/__init__.py @@ -0,0 +1,15 @@ +# This file is dual licensed under the terms of the Apache License, Version +# 2.0, and the BSD License. See the LICENSE file in the root of this repository +# for complete details. + +__title__ = "packaging" +__summary__ = "Core utilities for Python packages" +__uri__ = "https://github.com/pypa/packaging" + +__version__ = "23.3.dev0" + +__author__ = "Donald Stufft and individual contributors" +__email__ = "donald@stufft.io" + +__license__ = "BSD-2-Clause or Apache-2.0" +__copyright__ = "2014 %s" % __author__ diff --git a/gyp/pylib/packaging/_elffile.py b/gyp/pylib/packaging/_elffile.py new file mode 100644 index 0000000000..6fb19b30bb --- /dev/null +++ b/gyp/pylib/packaging/_elffile.py @@ -0,0 +1,108 @@ +""" +ELF file parser. + +This provides a class ``ELFFile`` that parses an ELF executable in a similar +interface to ``ZipFile``. Only the read interface is implemented. + +Based on: https://gist.github.com/lyssdod/f51579ae8d93c8657a5564aefc2ffbca +ELF header: https://refspecs.linuxfoundation.org/elf/gabi4+/ch4.eheader.html +""" + +import enum +import os +import struct +from typing import IO, Optional, Tuple + + +class ELFInvalid(ValueError): + pass + + +class EIClass(enum.IntEnum): + C32 = 1 + C64 = 2 + + +class EIData(enum.IntEnum): + Lsb = 1 + Msb = 2 + + +class EMachine(enum.IntEnum): + I386 = 3 + S390 = 22 + Arm = 40 + X8664 = 62 + AArc64 = 183 + + +class ELFFile: + """ + Representation of an ELF executable. + """ + + def __init__(self, f: IO[bytes]) -> None: + self._f = f + + try: + ident = self._read("16B") + except struct.error: + raise ELFInvalid("unable to parse identification") + magic = bytes(ident[:4]) + if magic != b"\x7fELF": + raise ELFInvalid(f"invalid magic: {magic!r}") + + self.capacity = ident[4] # Format for program header (bitness). + self.encoding = ident[5] # Data structure encoding (endianness). + + try: + # e_fmt: Format for program header. + # p_fmt: Format for section header. + # p_idx: Indexes to find p_type, p_offset, and p_filesz. + e_fmt, self._p_fmt, self._p_idx = { + (1, 1): ("HHIIIIIHHH", ">IIIIIIII", (0, 1, 4)), # 32-bit MSB. + (2, 1): ("HHIQQQIHHH", ">IIQQQQQQ", (0, 2, 5)), # 64-bit MSB. + }[(self.capacity, self.encoding)] + except KeyError: + raise ELFInvalid( + f"unrecognized capacity ({self.capacity}) or " + f"encoding ({self.encoding})" + ) + + try: + ( + _, + self.machine, # Architecture type. + _, + _, + self._e_phoff, # Offset of program header. + _, + self.flags, # Processor-specific flags. + _, + self._e_phentsize, # Size of section. + self._e_phnum, # Number of sections. + ) = self._read(e_fmt) + except struct.error as e: + raise ELFInvalid("unable to parse machine and section information") from e + + def _read(self, fmt: str) -> Tuple[int, ...]: + return struct.unpack(fmt, self._f.read(struct.calcsize(fmt))) + + @property + def interpreter(self) -> Optional[str]: + """ + The path recorded in the ``PT_INTERP`` section header. + """ + for index in range(self._e_phnum): + self._f.seek(self._e_phoff + self._e_phentsize * index) + try: + data = self._read(self._p_fmt) + except struct.error: + continue + if data[self._p_idx[0]] != 3: # Not PT_INTERP. + continue + self._f.seek(data[self._p_idx[1]]) + return os.fsdecode(self._f.read(data[self._p_idx[2]])).strip("\0") + return None diff --git a/gyp/pylib/packaging/_manylinux.py b/gyp/pylib/packaging/_manylinux.py new file mode 100644 index 0000000000..3705d50db9 --- /dev/null +++ b/gyp/pylib/packaging/_manylinux.py @@ -0,0 +1,252 @@ +import collections +import contextlib +import functools +import os +import re +import sys +import warnings +from typing import Dict, Generator, Iterator, NamedTuple, Optional, Sequence, Tuple + +from ._elffile import EIClass, EIData, ELFFile, EMachine + +EF_ARM_ABIMASK = 0xFF000000 +EF_ARM_ABI_VER5 = 0x05000000 +EF_ARM_ABI_FLOAT_HARD = 0x00000400 + + +# `os.PathLike` not a generic type until Python 3.9, so sticking with `str` +# as the type for `path` until then. +@contextlib.contextmanager +def _parse_elf(path: str) -> Generator[Optional[ELFFile], None, None]: + try: + with open(path, "rb") as f: + yield ELFFile(f) + except (OSError, TypeError, ValueError): + yield None + + +def _is_linux_armhf(executable: str) -> bool: + # hard-float ABI can be detected from the ELF header of the running + # process + # https://static.docs.arm.com/ihi0044/g/aaelf32.pdf + with _parse_elf(executable) as f: + return ( + f is not None + and f.capacity == EIClass.C32 + and f.encoding == EIData.Lsb + and f.machine == EMachine.Arm + and f.flags & EF_ARM_ABIMASK == EF_ARM_ABI_VER5 + and f.flags & EF_ARM_ABI_FLOAT_HARD == EF_ARM_ABI_FLOAT_HARD + ) + + +def _is_linux_i686(executable: str) -> bool: + with _parse_elf(executable) as f: + return ( + f is not None + and f.capacity == EIClass.C32 + and f.encoding == EIData.Lsb + and f.machine == EMachine.I386 + ) + + +def _have_compatible_abi(executable: str, archs: Sequence[str]) -> bool: + if "armv7l" in archs: + return _is_linux_armhf(executable) + if "i686" in archs: + return _is_linux_i686(executable) + allowed_archs = {"x86_64", "aarch64", "ppc64", "ppc64le", "s390x", "loongarch64"} + return any(arch in allowed_archs for arch in archs) + + +# If glibc ever changes its major version, we need to know what the last +# minor version was, so we can build the complete list of all versions. +# For now, guess what the highest minor version might be, assume it will +# be 50 for testing. Once this actually happens, update the dictionary +# with the actual value. +_LAST_GLIBC_MINOR: Dict[int, int] = collections.defaultdict(lambda: 50) + + +class _GLibCVersion(NamedTuple): + major: int + minor: int + + +def _glibc_version_string_confstr() -> Optional[str]: + """ + Primary implementation of glibc_version_string using os.confstr. + """ + # os.confstr is quite a bit faster than ctypes.DLL. It's also less likely + # to be broken or missing. This strategy is used in the standard library + # platform module. + # https://github.com/python/cpython/blob/fcf1d003bf4f0100c/Lib/platform.py#L175-L183 + try: + # Should be a string like "glibc 2.17". + version_string: str = getattr(os, "confstr")("CS_GNU_LIBC_VERSION") + assert version_string is not None + _, version = version_string.rsplit() + except (AssertionError, AttributeError, OSError, ValueError): + # os.confstr() or CS_GNU_LIBC_VERSION not available (or a bad value)... + return None + return version + + +def _glibc_version_string_ctypes() -> Optional[str]: + """ + Fallback implementation of glibc_version_string using ctypes. + """ + try: + import ctypes + except ImportError: + return None + + # ctypes.CDLL(None) internally calls dlopen(NULL), and as the dlopen + # manpage says, "If filename is NULL, then the returned handle is for the + # main program". This way we can let the linker do the work to figure out + # which libc our process is actually using. + # + # We must also handle the special case where the executable is not a + # dynamically linked executable. This can occur when using musl libc, + # for example. In this situation, dlopen() will error, leading to an + # OSError. Interestingly, at least in the case of musl, there is no + # errno set on the OSError. The single string argument used to construct + # OSError comes from libc itself and is therefore not portable to + # hard code here. In any case, failure to call dlopen() means we + # can proceed, so we bail on our attempt. + try: + process_namespace = ctypes.CDLL(None) + except OSError: + return None + + try: + gnu_get_libc_version = process_namespace.gnu_get_libc_version + except AttributeError: + # Symbol doesn't exist -> therefore, we are not linked to + # glibc. + return None + + # Call gnu_get_libc_version, which returns a string like "2.5" + gnu_get_libc_version.restype = ctypes.c_char_p + version_str: str = gnu_get_libc_version() + # py2 / py3 compatibility: + if not isinstance(version_str, str): + version_str = version_str.decode("ascii") + + return version_str + + +def _glibc_version_string() -> Optional[str]: + """Returns glibc version string, or None if not using glibc.""" + return _glibc_version_string_confstr() or _glibc_version_string_ctypes() + + +def _parse_glibc_version(version_str: str) -> Tuple[int, int]: + """Parse glibc version. + + We use a regexp instead of str.split because we want to discard any + random junk that might come after the minor version -- this might happen + in patched/forked versions of glibc (e.g. Linaro's version of glibc + uses version strings like "2.20-2014.11"). See gh-3588. + """ + m = re.match(r"(?P[0-9]+)\.(?P[0-9]+)", version_str) + if not m: + warnings.warn( + f"Expected glibc version with 2 components major.minor," + f" got: {version_str}", + RuntimeWarning, + ) + return -1, -1 + return int(m.group("major")), int(m.group("minor")) + + +@functools.lru_cache() +def _get_glibc_version() -> Tuple[int, int]: + version_str = _glibc_version_string() + if version_str is None: + return (-1, -1) + return _parse_glibc_version(version_str) + + +# From PEP 513, PEP 600 +def _is_compatible(arch: str, version: _GLibCVersion) -> bool: + sys_glibc = _get_glibc_version() + if sys_glibc < version: + return False + # Check for presence of _manylinux module. + try: + import _manylinux # noqa + except ImportError: + return True + if hasattr(_manylinux, "manylinux_compatible"): + result = _manylinux.manylinux_compatible(version[0], version[1], arch) + if result is not None: + return bool(result) + return True + if version == _GLibCVersion(2, 5): + if hasattr(_manylinux, "manylinux1_compatible"): + return bool(_manylinux.manylinux1_compatible) + if version == _GLibCVersion(2, 12): + if hasattr(_manylinux, "manylinux2010_compatible"): + return bool(_manylinux.manylinux2010_compatible) + if version == _GLibCVersion(2, 17): + if hasattr(_manylinux, "manylinux2014_compatible"): + return bool(_manylinux.manylinux2014_compatible) + return True + + +_LEGACY_MANYLINUX_MAP = { + # CentOS 7 w/ glibc 2.17 (PEP 599) + (2, 17): "manylinux2014", + # CentOS 6 w/ glibc 2.12 (PEP 571) + (2, 12): "manylinux2010", + # CentOS 5 w/ glibc 2.5 (PEP 513) + (2, 5): "manylinux1", +} + + +def platform_tags(archs: Sequence[str]) -> Iterator[str]: + """Generate manylinux tags compatible to the current platform. + + :param archs: Sequence of compatible architectures. + The first one shall be the closest to the actual architecture and be the part of + platform tag after the ``linux_`` prefix, e.g. ``x86_64``. + The ``linux_`` prefix is assumed as a prerequisite for the current platform to + be manylinux-compatible. + + :returns: An iterator of compatible manylinux tags. + """ + if not _have_compatible_abi(sys.executable, archs): + return + # Oldest glibc to be supported regardless of architecture is (2, 17). + too_old_glibc2 = _GLibCVersion(2, 16) + if set(archs) & {"x86_64", "i686"}: + # On x86/i686 also oldest glibc to be supported is (2, 5). + too_old_glibc2 = _GLibCVersion(2, 4) + current_glibc = _GLibCVersion(*_get_glibc_version()) + glibc_max_list = [current_glibc] + # We can assume compatibility across glibc major versions. + # https://sourceware.org/bugzilla/show_bug.cgi?id=24636 + # + # Build a list of maximum glibc versions so that we can + # output the canonical list of all glibc from current_glibc + # down to too_old_glibc2, including all intermediary versions. + for glibc_major in range(current_glibc.major - 1, 1, -1): + glibc_minor = _LAST_GLIBC_MINOR[glibc_major] + glibc_max_list.append(_GLibCVersion(glibc_major, glibc_minor)) + for arch in archs: + for glibc_max in glibc_max_list: + if glibc_max.major == too_old_glibc2.major: + min_minor = too_old_glibc2.minor + else: + # For other glibc major versions oldest supported is (x, 0). + min_minor = -1 + for glibc_minor in range(glibc_max.minor, min_minor, -1): + glibc_version = _GLibCVersion(glibc_max.major, glibc_minor) + tag = "manylinux_{}_{}".format(*glibc_version) + if _is_compatible(arch, glibc_version): + yield f"{tag}_{arch}" + # Handle the legacy manylinux1, manylinux2010, manylinux2014 tags. + if glibc_version in _LEGACY_MANYLINUX_MAP: + legacy_tag = _LEGACY_MANYLINUX_MAP[glibc_version] + if _is_compatible(arch, glibc_version): + yield f"{legacy_tag}_{arch}" diff --git a/gyp/pylib/packaging/_musllinux.py b/gyp/pylib/packaging/_musllinux.py new file mode 100644 index 0000000000..86419df9d7 --- /dev/null +++ b/gyp/pylib/packaging/_musllinux.py @@ -0,0 +1,83 @@ +"""PEP 656 support. + +This module implements logic to detect if the currently running Python is +linked against musl, and what musl version is used. +""" + +import functools +import re +import subprocess +import sys +from typing import Iterator, NamedTuple, Optional, Sequence + +from ._elffile import ELFFile + + +class _MuslVersion(NamedTuple): + major: int + minor: int + + +def _parse_musl_version(output: str) -> Optional[_MuslVersion]: + lines = [n for n in (n.strip() for n in output.splitlines()) if n] + if len(lines) < 2 or lines[0][:4] != "musl": + return None + m = re.match(r"Version (\d+)\.(\d+)", lines[1]) + if not m: + return None + return _MuslVersion(major=int(m.group(1)), minor=int(m.group(2))) + + +@functools.lru_cache() +def _get_musl_version(executable: str) -> Optional[_MuslVersion]: + """Detect currently-running musl runtime version. + + This is done by checking the specified executable's dynamic linking + information, and invoking the loader to parse its output for a version + string. If the loader is musl, the output would be something like:: + + musl libc (x86_64) + Version 1.2.2 + Dynamic Program Loader + """ + try: + with open(executable, "rb") as f: + ld = ELFFile(f).interpreter + except (OSError, TypeError, ValueError): + return None + if ld is None or "musl" not in ld: + return None + proc = subprocess.run([ld], stderr=subprocess.PIPE, text=True) + return _parse_musl_version(proc.stderr) + + +def platform_tags(archs: Sequence[str]) -> Iterator[str]: + """Generate musllinux tags compatible to the current platform. + + :param archs: Sequence of compatible architectures. + The first one shall be the closest to the actual architecture and be the part of + platform tag after the ``linux_`` prefix, e.g. ``x86_64``. + The ``linux_`` prefix is assumed as a prerequisite for the current platform to + be musllinux-compatible. + + :returns: An iterator of compatible musllinux tags. + """ + sys_musl = _get_musl_version(sys.executable) + if sys_musl is None: # Python not dynamically linked against musl. + return + for arch in archs: + for minor in range(sys_musl.minor, -1, -1): + yield f"musllinux_{sys_musl.major}_{minor}_{arch}" + + +if __name__ == "__main__": # pragma: no cover + import sysconfig + + plat = sysconfig.get_platform() + assert plat.startswith("linux-"), "not linux" + + print("plat:", plat) + print("musl:", _get_musl_version(sys.executable)) + print("tags:", end=" ") + for t in platform_tags(re.sub(r"[.-]", "_", plat.split("-", 1)[-1])): + print(t, end="\n ") diff --git a/gyp/pylib/packaging/_parser.py b/gyp/pylib/packaging/_parser.py new file mode 100644 index 0000000000..4576981c2d --- /dev/null +++ b/gyp/pylib/packaging/_parser.py @@ -0,0 +1,359 @@ +"""Handwritten parser of dependency specifiers. + +The docstring for each __parse_* function contains ENBF-inspired grammar representing +the implementation. +""" + +import ast +from typing import Any, List, NamedTuple, Optional, Tuple, Union + +from ._tokenizer import DEFAULT_RULES, Tokenizer + + +class Node: + def __init__(self, value: str) -> None: + self.value = value + + def __str__(self) -> str: + return self.value + + def __repr__(self) -> str: + return f"<{self.__class__.__name__}('{self}')>" + + def serialize(self) -> str: + raise NotImplementedError + + +class Variable(Node): + def serialize(self) -> str: + return str(self) + + +class Value(Node): + def serialize(self) -> str: + return f'"{self}"' + + +class Op(Node): + def serialize(self) -> str: + return str(self) + + +MarkerVar = Union[Variable, Value] +MarkerItem = Tuple[MarkerVar, Op, MarkerVar] +# MarkerAtom = Union[MarkerItem, List["MarkerAtom"]] +# MarkerList = List[Union["MarkerList", MarkerAtom, str]] +# mypy does not support recursive type definition +# https://github.com/python/mypy/issues/731 +MarkerAtom = Any +MarkerList = List[Any] + + +class ParsedRequirement(NamedTuple): + name: str + url: str + extras: List[str] + specifier: str + marker: Optional[MarkerList] + + +# -------------------------------------------------------------------------------------- +# Recursive descent parser for dependency specifier +# -------------------------------------------------------------------------------------- +def parse_requirement(source: str) -> ParsedRequirement: + return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES)) + + +def _parse_requirement(tokenizer: Tokenizer) -> ParsedRequirement: + """ + requirement = WS? IDENTIFIER WS? extras WS? requirement_details + """ + tokenizer.consume("WS") + + name_token = tokenizer.expect( + "IDENTIFIER", expected="package name at the start of dependency specifier" + ) + name = name_token.text + tokenizer.consume("WS") + + extras = _parse_extras(tokenizer) + tokenizer.consume("WS") + + url, specifier, marker = _parse_requirement_details(tokenizer) + tokenizer.expect("END", expected="end of dependency specifier") + + return ParsedRequirement(name, url, extras, specifier, marker) + + +def _parse_requirement_details( + tokenizer: Tokenizer, +) -> Tuple[str, str, Optional[MarkerList]]: + """ + requirement_details = AT URL (WS requirement_marker?)? + | specifier WS? (requirement_marker)? + """ + + specifier = "" + url = "" + marker = None + + if tokenizer.check("AT"): + tokenizer.read() + tokenizer.consume("WS") + + url_start = tokenizer.position + url = tokenizer.expect("URL", expected="URL after @").text + if tokenizer.check("END", peek=True): + return (url, specifier, marker) + + tokenizer.expect("WS", expected="whitespace after URL") + + # The input might end after whitespace. + if tokenizer.check("END", peek=True): + return (url, specifier, marker) + + marker = _parse_requirement_marker( + tokenizer, span_start=url_start, after="URL and whitespace" + ) + else: + specifier_start = tokenizer.position + specifier = _parse_specifier(tokenizer) + tokenizer.consume("WS") + + if tokenizer.check("END", peek=True): + return (url, specifier, marker) + + marker = _parse_requirement_marker( + tokenizer, + span_start=specifier_start, + after=( + "version specifier" + if specifier + else "name and no valid version specifier" + ), + ) + + return (url, specifier, marker) + + +def _parse_requirement_marker( + tokenizer: Tokenizer, *, span_start: int, after: str +) -> MarkerList: + """ + requirement_marker = SEMICOLON marker WS? + """ + + if not tokenizer.check("SEMICOLON"): + tokenizer.raise_syntax_error( + f"Expected end or semicolon (after {after})", + span_start=span_start, + ) + tokenizer.read() + + marker = _parse_marker(tokenizer) + tokenizer.consume("WS") + + return marker + + +def _parse_extras(tokenizer: Tokenizer) -> List[str]: + """ + extras = (LEFT_BRACKET wsp* extras_list? wsp* RIGHT_BRACKET)? + """ + if not tokenizer.check("LEFT_BRACKET", peek=True): + return [] + + with tokenizer.enclosing_tokens( + "LEFT_BRACKET", + "RIGHT_BRACKET", + around="extras", + ): + tokenizer.consume("WS") + extras = _parse_extras_list(tokenizer) + tokenizer.consume("WS") + + return extras + + +def _parse_extras_list(tokenizer: Tokenizer) -> List[str]: + """ + extras_list = identifier (wsp* ',' wsp* identifier)* + """ + extras: List[str] = [] + + if not tokenizer.check("IDENTIFIER"): + return extras + + extras.append(tokenizer.read().text) + + while True: + tokenizer.consume("WS") + if tokenizer.check("IDENTIFIER", peek=True): + tokenizer.raise_syntax_error("Expected comma between extra names") + elif not tokenizer.check("COMMA"): + break + + tokenizer.read() + tokenizer.consume("WS") + + extra_token = tokenizer.expect("IDENTIFIER", expected="extra name after comma") + extras.append(extra_token.text) + + return extras + + +def _parse_specifier(tokenizer: Tokenizer) -> str: + """ + specifier = LEFT_PARENTHESIS WS? version_many WS? RIGHT_PARENTHESIS + | WS? version_many WS? + """ + with tokenizer.enclosing_tokens( + "LEFT_PARENTHESIS", + "RIGHT_PARENTHESIS", + around="version specifier", + ): + tokenizer.consume("WS") + parsed_specifiers = _parse_version_many(tokenizer) + tokenizer.consume("WS") + + return parsed_specifiers + + +def _parse_version_many(tokenizer: Tokenizer) -> str: + """ + version_many = (SPECIFIER (WS? COMMA WS? SPECIFIER)*)? + """ + parsed_specifiers = "" + while tokenizer.check("SPECIFIER"): + span_start = tokenizer.position + parsed_specifiers += tokenizer.read().text + if tokenizer.check("VERSION_PREFIX_TRAIL", peek=True): + tokenizer.raise_syntax_error( + ".* suffix can only be used with `==` or `!=` operators", + span_start=span_start, + span_end=tokenizer.position + 1, + ) + if tokenizer.check("VERSION_LOCAL_LABEL_TRAIL", peek=True): + tokenizer.raise_syntax_error( + "Local version label can only be used with `==` or `!=` operators", + span_start=span_start, + span_end=tokenizer.position, + ) + tokenizer.consume("WS") + if not tokenizer.check("COMMA"): + break + parsed_specifiers += tokenizer.read().text + tokenizer.consume("WS") + + return parsed_specifiers + + +# -------------------------------------------------------------------------------------- +# Recursive descent parser for marker expression +# -------------------------------------------------------------------------------------- +def parse_marker(source: str) -> MarkerList: + return _parse_full_marker(Tokenizer(source, rules=DEFAULT_RULES)) + + +def _parse_full_marker(tokenizer: Tokenizer) -> MarkerList: + retval = _parse_marker(tokenizer) + tokenizer.expect("END", expected="end of marker expression") + return retval + + +def _parse_marker(tokenizer: Tokenizer) -> MarkerList: + """ + marker = marker_atom (BOOLOP marker_atom)+ + """ + expression = [_parse_marker_atom(tokenizer)] + while tokenizer.check("BOOLOP"): + token = tokenizer.read() + expr_right = _parse_marker_atom(tokenizer) + expression.extend((token.text, expr_right)) + return expression + + +def _parse_marker_atom(tokenizer: Tokenizer) -> MarkerAtom: + """ + marker_atom = WS? LEFT_PARENTHESIS WS? marker WS? RIGHT_PARENTHESIS WS? + | WS? marker_item WS? + """ + + tokenizer.consume("WS") + if tokenizer.check("LEFT_PARENTHESIS", peek=True): + with tokenizer.enclosing_tokens( + "LEFT_PARENTHESIS", + "RIGHT_PARENTHESIS", + around="marker expression", + ): + tokenizer.consume("WS") + marker: MarkerAtom = _parse_marker(tokenizer) + tokenizer.consume("WS") + else: + marker = _parse_marker_item(tokenizer) + tokenizer.consume("WS") + return marker + + +def _parse_marker_item(tokenizer: Tokenizer) -> MarkerItem: + """ + marker_item = WS? marker_var WS? marker_op WS? marker_var WS? + """ + tokenizer.consume("WS") + marker_var_left = _parse_marker_var(tokenizer) + tokenizer.consume("WS") + marker_op = _parse_marker_op(tokenizer) + tokenizer.consume("WS") + marker_var_right = _parse_marker_var(tokenizer) + tokenizer.consume("WS") + return (marker_var_left, marker_op, marker_var_right) + + +def _parse_marker_var(tokenizer: Tokenizer) -> MarkerVar: + """ + marker_var = VARIABLE | QUOTED_STRING + """ + if tokenizer.check("VARIABLE"): + return process_env_var(tokenizer.read().text.replace(".", "_")) + elif tokenizer.check("QUOTED_STRING"): + return process_python_str(tokenizer.read().text) + else: + tokenizer.raise_syntax_error( + message="Expected a marker variable or quoted string" + ) + + +def process_env_var(env_var: str) -> Variable: + if ( + env_var == "platform_python_implementation" + or env_var == "python_implementation" + ): + return Variable("platform_python_implementation") + else: + return Variable(env_var) + + +def process_python_str(python_str: str) -> Value: + value = ast.literal_eval(python_str) + return Value(str(value)) + + +def _parse_marker_op(tokenizer: Tokenizer) -> Op: + """ + marker_op = IN | NOT IN | OP + """ + if tokenizer.check("IN"): + tokenizer.read() + return Op("in") + elif tokenizer.check("NOT"): + tokenizer.read() + tokenizer.expect("WS", expected="whitespace after 'not'") + tokenizer.expect("IN", expected="'in' after 'not'") + return Op("not in") + elif tokenizer.check("OP"): + return Op(tokenizer.read().text) + else: + return tokenizer.raise_syntax_error( + "Expected marker operator, one of " + "<=, <, !=, ==, >=, >, ~=, ===, in, not in" + ) diff --git a/gyp/pylib/packaging/_structures.py b/gyp/pylib/packaging/_structures.py new file mode 100644 index 0000000000..90a6465f96 --- /dev/null +++ b/gyp/pylib/packaging/_structures.py @@ -0,0 +1,61 @@ +# This file is dual licensed under the terms of the Apache License, Version +# 2.0, and the BSD License. See the LICENSE file in the root of this repository +# for complete details. + + +class InfinityType: + def __repr__(self) -> str: + return "Infinity" + + def __hash__(self) -> int: + return hash(repr(self)) + + def __lt__(self, other: object) -> bool: + return False + + def __le__(self, other: object) -> bool: + return False + + def __eq__(self, other: object) -> bool: + return isinstance(other, self.__class__) + + def __gt__(self, other: object) -> bool: + return True + + def __ge__(self, other: object) -> bool: + return True + + def __neg__(self: object) -> "NegativeInfinityType": + return NegativeInfinity + + +Infinity = InfinityType() + + +class NegativeInfinityType: + def __repr__(self) -> str: + return "-Infinity" + + def __hash__(self) -> int: + return hash(repr(self)) + + def __lt__(self, other: object) -> bool: + return True + + def __le__(self, other: object) -> bool: + return True + + def __eq__(self, other: object) -> bool: + return isinstance(other, self.__class__) + + def __gt__(self, other: object) -> bool: + return False + + def __ge__(self, other: object) -> bool: + return False + + def __neg__(self: object) -> InfinityType: + return Infinity + + +NegativeInfinity = NegativeInfinityType() diff --git a/gyp/pylib/packaging/_tokenizer.py b/gyp/pylib/packaging/_tokenizer.py new file mode 100644 index 0000000000..dd0d648d49 --- /dev/null +++ b/gyp/pylib/packaging/_tokenizer.py @@ -0,0 +1,192 @@ +import contextlib +import re +from dataclasses import dataclass +from typing import Dict, Iterator, NoReturn, Optional, Tuple, Union + +from .specifiers import Specifier + + +@dataclass +class Token: + name: str + text: str + position: int + + +class ParserSyntaxError(Exception): + """The provided source text could not be parsed correctly.""" + + def __init__( + self, + message: str, + *, + source: str, + span: Tuple[int, int], + ) -> None: + self.span = span + self.message = message + self.source = source + + super().__init__() + + def __str__(self) -> str: + marker = " " * self.span[0] + "~" * (self.span[1] - self.span[0]) + "^" + return "\n ".join([self.message, self.source, marker]) + + +DEFAULT_RULES: "Dict[str, Union[str, re.Pattern[str]]]" = { + "LEFT_PARENTHESIS": r"\(", + "RIGHT_PARENTHESIS": r"\)", + "LEFT_BRACKET": r"\[", + "RIGHT_BRACKET": r"\]", + "SEMICOLON": r";", + "COMMA": r",", + "QUOTED_STRING": re.compile( + r""" + ( + ('[^']*') + | + ("[^"]*") + ) + """, + re.VERBOSE, + ), + "OP": r"(===|==|~=|!=|<=|>=|<|>)", + "BOOLOP": r"\b(or|and)\b", + "IN": r"\bin\b", + "NOT": r"\bnot\b", + "VARIABLE": re.compile( + r""" + \b( + python_version + |python_full_version + |os[._]name + |sys[._]platform + |platform_(release|system) + |platform[._](version|machine|python_implementation) + |python_implementation + |implementation_(name|version) + |extra + )\b + """, + re.VERBOSE, + ), + "SPECIFIER": re.compile( + Specifier._operator_regex_str + Specifier._version_regex_str, + re.VERBOSE | re.IGNORECASE, + ), + "AT": r"\@", + "URL": r"[^ \t]+", + "IDENTIFIER": r"\b[a-zA-Z0-9][a-zA-Z0-9._-]*\b", + "VERSION_PREFIX_TRAIL": r"\.\*", + "VERSION_LOCAL_LABEL_TRAIL": r"\+[a-z0-9]+(?:[-_\.][a-z0-9]+)*", + "WS": r"[ \t]+", + "END": r"$", +} + + +class Tokenizer: + """Context-sensitive token parsing. + + Provides methods to examine the input stream to check whether the next token + matches. + """ + + def __init__( + self, + source: str, + *, + rules: "Dict[str, Union[str, re.Pattern[str]]]", + ) -> None: + self.source = source + self.rules: Dict[str, re.Pattern[str]] = { + name: re.compile(pattern) for name, pattern in rules.items() + } + self.next_token: Optional[Token] = None + self.position = 0 + + def consume(self, name: str) -> None: + """Move beyond provided token name, if at current position.""" + if self.check(name): + self.read() + + def check(self, name: str, *, peek: bool = False) -> bool: + """Check whether the next token has the provided name. + + By default, if the check succeeds, the token *must* be read before + another check. If `peek` is set to `True`, the token is not loaded and + would need to be checked again. + """ + assert ( + self.next_token is None + ), f"Cannot check for {name!r}, already have {self.next_token!r}" + assert name in self.rules, f"Unknown token name: {name!r}" + + expression = self.rules[name] + + match = expression.match(self.source, self.position) + if match is None: + return False + if not peek: + self.next_token = Token(name, match[0], self.position) + return True + + def expect(self, name: str, *, expected: str) -> Token: + """Expect a certain token name next, failing with a syntax error otherwise. + + The token is *not* read. + """ + if not self.check(name): + raise self.raise_syntax_error(f"Expected {expected}") + return self.read() + + def read(self) -> Token: + """Consume the next token and return it.""" + token = self.next_token + assert token is not None + + self.position += len(token.text) + self.next_token = None + + return token + + def raise_syntax_error( + self, + message: str, + *, + span_start: Optional[int] = None, + span_end: Optional[int] = None, + ) -> NoReturn: + """Raise ParserSyntaxError at the given position.""" + span = ( + self.position if span_start is None else span_start, + self.position if span_end is None else span_end, + ) + raise ParserSyntaxError( + message, + source=self.source, + span=span, + ) + + @contextlib.contextmanager + def enclosing_tokens( + self, open_token: str, close_token: str, *, around: str + ) -> Iterator[None]: + if self.check(open_token): + open_position = self.position + self.read() + else: + open_position = None + + yield + + if open_position is None: + return + + if not self.check(close_token): + self.raise_syntax_error( + f"Expected matching {close_token} for {open_token}, after {around}", + span_start=open_position, + ) + + self.read() diff --git a/gyp/pylib/packaging/markers.py b/gyp/pylib/packaging/markers.py new file mode 100644 index 0000000000..8b98fca723 --- /dev/null +++ b/gyp/pylib/packaging/markers.py @@ -0,0 +1,252 @@ +# This file is dual licensed under the terms of the Apache License, Version +# 2.0, and the BSD License. See the LICENSE file in the root of this repository +# for complete details. + +import operator +import os +import platform +import sys +from typing import Any, Callable, Dict, List, Optional, Tuple, Union + +from ._parser import ( + MarkerAtom, + MarkerList, + Op, + Value, + Variable, + parse_marker as _parse_marker, +) +from ._tokenizer import ParserSyntaxError +from .specifiers import InvalidSpecifier, Specifier +from .utils import canonicalize_name + +__all__ = [ + "InvalidMarker", + "UndefinedComparison", + "UndefinedEnvironmentName", + "Marker", + "default_environment", +] + +Operator = Callable[[str, str], bool] + + +class InvalidMarker(ValueError): + """ + An invalid marker was found, users should refer to PEP 508. + """ + + +class UndefinedComparison(ValueError): + """ + An invalid operation was attempted on a value that doesn't support it. + """ + + +class UndefinedEnvironmentName(ValueError): + """ + A name was attempted to be used that does not exist inside of the + environment. + """ + + +def _normalize_extra_values(results: Any) -> Any: + """ + Normalize extra values. + """ + if isinstance(results[0], tuple): + lhs, op, rhs = results[0] + if isinstance(lhs, Variable) and lhs.value == "extra": + normalized_extra = canonicalize_name(rhs.value) + rhs = Value(normalized_extra) + elif isinstance(rhs, Variable) and rhs.value == "extra": + normalized_extra = canonicalize_name(lhs.value) + lhs = Value(normalized_extra) + results[0] = lhs, op, rhs + return results + + +def _format_marker( + marker: Union[List[str], MarkerAtom, str], first: Optional[bool] = True +) -> str: + + assert isinstance(marker, (list, tuple, str)) + + # Sometimes we have a structure like [[...]] which is a single item list + # where the single item is itself it's own list. In that case we want skip + # the rest of this function so that we don't get extraneous () on the + # outside. + if ( + isinstance(marker, list) + and len(marker) == 1 + and isinstance(marker[0], (list, tuple)) + ): + return _format_marker(marker[0]) + + if isinstance(marker, list): + inner = (_format_marker(m, first=False) for m in marker) + if first: + return " ".join(inner) + else: + return "(" + " ".join(inner) + ")" + elif isinstance(marker, tuple): + return " ".join([m.serialize() for m in marker]) + else: + return marker + + +_operators: Dict[str, Operator] = { + "in": lambda lhs, rhs: lhs in rhs, + "not in": lambda lhs, rhs: lhs not in rhs, + "<": operator.lt, + "<=": operator.le, + "==": operator.eq, + "!=": operator.ne, + ">=": operator.ge, + ">": operator.gt, +} + + +def _eval_op(lhs: str, op: Op, rhs: str) -> bool: + try: + spec = Specifier("".join([op.serialize(), rhs])) + except InvalidSpecifier: + pass + else: + return spec.contains(lhs, prereleases=True) + + oper: Optional[Operator] = _operators.get(op.serialize()) + if oper is None: + raise UndefinedComparison(f"Undefined {op!r} on {lhs!r} and {rhs!r}.") + + return oper(lhs, rhs) + + +def _normalize(*values: str, key: str) -> Tuple[str, ...]: + # PEP 685 – Comparison of extra names for optional distribution dependencies + # https://peps.python.org/pep-0685/ + # > When comparing extra names, tools MUST normalize the names being + # > compared using the semantics outlined in PEP 503 for names + if key == "extra": + return tuple(canonicalize_name(v) for v in values) + + # other environment markers don't have such standards + return values + + +def _evaluate_markers(markers: MarkerList, environment: Dict[str, str]) -> bool: + groups: List[List[bool]] = [[]] + + for marker in markers: + assert isinstance(marker, (list, tuple, str)) + + if isinstance(marker, list): + groups[-1].append(_evaluate_markers(marker, environment)) + elif isinstance(marker, tuple): + lhs, op, rhs = marker + + if isinstance(lhs, Variable): + environment_key = lhs.value + lhs_value = environment[environment_key] + rhs_value = rhs.value + else: + lhs_value = lhs.value + environment_key = rhs.value + rhs_value = environment[environment_key] + + lhs_value, rhs_value = _normalize(lhs_value, rhs_value, key=environment_key) + groups[-1].append(_eval_op(lhs_value, op, rhs_value)) + else: + assert marker in ["and", "or"] + if marker == "or": + groups.append([]) + + return any(all(item) for item in groups) + + +def format_full_version(info: "sys._version_info") -> str: + version = "{0.major}.{0.minor}.{0.micro}".format(info) + kind = info.releaselevel + if kind != "final": + version += kind[0] + str(info.serial) + return version + + +def default_environment() -> Dict[str, str]: + iver = format_full_version(sys.implementation.version) + implementation_name = sys.implementation.name + return { + "implementation_name": implementation_name, + "implementation_version": iver, + "os_name": os.name, + "platform_machine": platform.machine(), + "platform_release": platform.release(), + "platform_system": platform.system(), + "platform_version": platform.version(), + "python_full_version": platform.python_version(), + "platform_python_implementation": platform.python_implementation(), + "python_version": ".".join(platform.python_version_tuple()[:2]), + "sys_platform": sys.platform, + } + + +class Marker: + def __init__(self, marker: str) -> None: + # Note: We create a Marker object without calling this constructor in + # packaging.requirements.Requirement. If any additional logic is + # added here, make sure to mirror/adapt Requirement. + try: + self._markers = _normalize_extra_values(_parse_marker(marker)) + # The attribute `_markers` can be described in terms of a recursive type: + # MarkerList = List[Union[Tuple[Node, ...], str, MarkerList]] + # + # For example, the following expression: + # python_version > "3.6" or (python_version == "3.6" and os_name == "unix") + # + # is parsed into: + # [ + # (, ')>, ), + # 'and', + # [ + # (, , ), + # 'or', + # (, , ) + # ] + # ] + except ParserSyntaxError as e: + raise InvalidMarker(str(e)) from e + + def __str__(self) -> str: + return _format_marker(self._markers) + + def __repr__(self) -> str: + return f"" + + def __hash__(self) -> int: + return hash((self.__class__.__name__, str(self))) + + def __eq__(self, other: Any) -> bool: + if not isinstance(other, Marker): + return NotImplemented + + return str(self) == str(other) + + def evaluate(self, environment: Optional[Dict[str, str]] = None) -> bool: + """Evaluate a marker. + + Return the boolean from evaluating the given marker against the + environment. environment is an optional argument to override all or + part of the determined environment. + + The environment is determined from the current Python process. + """ + current_environment = default_environment() + current_environment["extra"] = "" + if environment is not None: + current_environment.update(environment) + # The API used to allow setting extra to None. We need to handle this + # case for backwards compatibility. + if current_environment["extra"] is None: + current_environment["extra"] = "" + + return _evaluate_markers(self._markers, current_environment) diff --git a/gyp/pylib/packaging/metadata.py b/gyp/pylib/packaging/metadata.py new file mode 100644 index 0000000000..fb27493079 --- /dev/null +++ b/gyp/pylib/packaging/metadata.py @@ -0,0 +1,825 @@ +import email.feedparser +import email.header +import email.message +import email.parser +import email.policy +import sys +import typing +from typing import ( + Any, + Callable, + Dict, + Generic, + List, + Optional, + Tuple, + Type, + Union, + cast, +) + +from . import requirements, specifiers, utils, version as version_module + +T = typing.TypeVar("T") +if sys.version_info[:2] >= (3, 8): # pragma: no cover + from typing import Literal, TypedDict +else: # pragma: no cover + if typing.TYPE_CHECKING: + from typing_extensions import Literal, TypedDict + else: + try: + from typing_extensions import Literal, TypedDict + except ImportError: + + class Literal: + def __init_subclass__(*_args, **_kwargs): + pass + + class TypedDict: + def __init_subclass__(*_args, **_kwargs): + pass + + +try: + ExceptionGroup +except NameError: # pragma: no cover + + class ExceptionGroup(Exception): # noqa: N818 + """A minimal implementation of :external:exc:`ExceptionGroup` from Python 3.11. + + If :external:exc:`ExceptionGroup` is already defined by Python itself, + that version is used instead. + """ + + message: str + exceptions: List[Exception] + + def __init__(self, message: str, exceptions: List[Exception]) -> None: + self.message = message + self.exceptions = exceptions + + def __repr__(self) -> str: + return f"{self.__class__.__name__}({self.message!r}, {self.exceptions!r})" + +else: # pragma: no cover + ExceptionGroup = ExceptionGroup + + +class InvalidMetadata(ValueError): + """A metadata field contains invalid data.""" + + field: str + """The name of the field that contains invalid data.""" + + def __init__(self, field: str, message: str) -> None: + self.field = field + super().__init__(message) + + +# The RawMetadata class attempts to make as few assumptions about the underlying +# serialization formats as possible. The idea is that as long as a serialization +# formats offer some very basic primitives in *some* way then we can support +# serializing to and from that format. +class RawMetadata(TypedDict, total=False): + """A dictionary of raw core metadata. + + Each field in core metadata maps to a key of this dictionary (when data is + provided). The key is lower-case and underscores are used instead of dashes + compared to the equivalent core metadata field. Any core metadata field that + can be specified multiple times or can hold multiple values in a single + field have a key with a plural name. See :class:`Metadata` whose attributes + match the keys of this dictionary. + + Core metadata fields that can be specified multiple times are stored as a + list or dict depending on which is appropriate for the field. Any fields + which hold multiple values in a single field are stored as a list. + + """ + + # Metadata 1.0 - PEP 241 + metadata_version: str + name: str + version: str + platforms: List[str] + summary: str + description: str + keywords: List[str] + home_page: str + author: str + author_email: str + license: str + + # Metadata 1.1 - PEP 314 + supported_platforms: List[str] + download_url: str + classifiers: List[str] + requires: List[str] + provides: List[str] + obsoletes: List[str] + + # Metadata 1.2 - PEP 345 + maintainer: str + maintainer_email: str + requires_dist: List[str] + provides_dist: List[str] + obsoletes_dist: List[str] + requires_python: str + requires_external: List[str] + project_urls: Dict[str, str] + + # Metadata 2.0 + # PEP 426 attempted to completely revamp the metadata format + # but got stuck without ever being able to build consensus on + # it and ultimately ended up withdrawn. + # + # However, a number of tools had started emitting METADATA with + # `2.0` Metadata-Version, so for historical reasons, this version + # was skipped. + + # Metadata 2.1 - PEP 566 + description_content_type: str + provides_extra: List[str] + + # Metadata 2.2 - PEP 643 + dynamic: List[str] + + # Metadata 2.3 - PEP 685 + # No new fields were added in PEP 685, just some edge case were + # tightened up to provide better interoptability. + + +_STRING_FIELDS = { + "author", + "author_email", + "description", + "description_content_type", + "download_url", + "home_page", + "license", + "maintainer", + "maintainer_email", + "metadata_version", + "name", + "requires_python", + "summary", + "version", +} + +_LIST_FIELDS = { + "classifiers", + "dynamic", + "obsoletes", + "obsoletes_dist", + "platforms", + "provides", + "provides_dist", + "provides_extra", + "requires", + "requires_dist", + "requires_external", + "supported_platforms", +} + +_DICT_FIELDS = { + "project_urls", +} + + +def _parse_keywords(data: str) -> List[str]: + """Split a string of comma-separate keyboards into a list of keywords.""" + return [k.strip() for k in data.split(",")] + + +def _parse_project_urls(data: List[str]) -> Dict[str, str]: + """Parse a list of label/URL string pairings separated by a comma.""" + urls = {} + for pair in data: + # Our logic is slightly tricky here as we want to try and do + # *something* reasonable with malformed data. + # + # The main thing that we have to worry about, is data that does + # not have a ',' at all to split the label from the Value. There + # isn't a singular right answer here, and we will fail validation + # later on (if the caller is validating) so it doesn't *really* + # matter, but since the missing value has to be an empty str + # and our return value is dict[str, str], if we let the key + # be the missing value, then they'd have multiple '' values that + # overwrite each other in a accumulating dict. + # + # The other potentional issue is that it's possible to have the + # same label multiple times in the metadata, with no solid "right" + # answer with what to do in that case. As such, we'll do the only + # thing we can, which is treat the field as unparseable and add it + # to our list of unparsed fields. + parts = [p.strip() for p in pair.split(",", 1)] + parts.extend([""] * (max(0, 2 - len(parts)))) # Ensure 2 items + + # TODO: The spec doesn't say anything about if the keys should be + # considered case sensitive or not... logically they should + # be case-preserving and case-insensitive, but doing that + # would open up more cases where we might have duplicate + # entries. + label, url = parts + if label in urls: + # The label already exists in our set of urls, so this field + # is unparseable, and we can just add the whole thing to our + # unparseable data and stop processing it. + raise KeyError("duplicate labels in project urls") + urls[label] = url + + return urls + + +def _get_payload(msg: email.message.Message, source: Union[bytes, str]) -> str: + """Get the body of the message.""" + # If our source is a str, then our caller has managed encodings for us, + # and we don't need to deal with it. + if isinstance(source, str): + payload: str = msg.get_payload() + return payload + # If our source is a bytes, then we're managing the encoding and we need + # to deal with it. + else: + bpayload: bytes = msg.get_payload(decode=True) + try: + return bpayload.decode("utf8", "strict") + except UnicodeDecodeError: + raise ValueError("payload in an invalid encoding") + + +# The various parse_FORMAT functions here are intended to be as lenient as +# possible in their parsing, while still returning a correctly typed +# RawMetadata. +# +# To aid in this, we also generally want to do as little touching of the +# data as possible, except where there are possibly some historic holdovers +# that make valid data awkward to work with. +# +# While this is a lower level, intermediate format than our ``Metadata`` +# class, some light touch ups can make a massive difference in usability. + +# Map METADATA fields to RawMetadata. +_EMAIL_TO_RAW_MAPPING = { + "author": "author", + "author-email": "author_email", + "classifier": "classifiers", + "description": "description", + "description-content-type": "description_content_type", + "download-url": "download_url", + "dynamic": "dynamic", + "home-page": "home_page", + "keywords": "keywords", + "license": "license", + "maintainer": "maintainer", + "maintainer-email": "maintainer_email", + "metadata-version": "metadata_version", + "name": "name", + "obsoletes": "obsoletes", + "obsoletes-dist": "obsoletes_dist", + "platform": "platforms", + "project-url": "project_urls", + "provides": "provides", + "provides-dist": "provides_dist", + "provides-extra": "provides_extra", + "requires": "requires", + "requires-dist": "requires_dist", + "requires-external": "requires_external", + "requires-python": "requires_python", + "summary": "summary", + "supported-platform": "supported_platforms", + "version": "version", +} +_RAW_TO_EMAIL_MAPPING = {raw: email for email, raw in _EMAIL_TO_RAW_MAPPING.items()} + + +def parse_email(data: Union[bytes, str]) -> Tuple[RawMetadata, Dict[str, List[str]]]: + """Parse a distribution's metadata stored as email headers (e.g. from ``METADATA``). + + This function returns a two-item tuple of dicts. The first dict is of + recognized fields from the core metadata specification. Fields that can be + parsed and translated into Python's built-in types are converted + appropriately. All other fields are left as-is. Fields that are allowed to + appear multiple times are stored as lists. + + The second dict contains all other fields from the metadata. This includes + any unrecognized fields. It also includes any fields which are expected to + be parsed into a built-in type but were not formatted appropriately. Finally, + any fields that are expected to appear only once but are repeated are + included in this dict. + + """ + raw: Dict[str, Union[str, List[str], Dict[str, str]]] = {} + unparsed: Dict[str, List[str]] = {} + + if isinstance(data, str): + parsed = email.parser.Parser(policy=email.policy.compat32).parsestr(data) + else: + parsed = email.parser.BytesParser(policy=email.policy.compat32).parsebytes(data) + + # We have to wrap parsed.keys() in a set, because in the case of multiple + # values for a key (a list), the key will appear multiple times in the + # list of keys, but we're avoiding that by using get_all(). + for name in frozenset(parsed.keys()): + # Header names in RFC are case insensitive, so we'll normalize to all + # lower case to make comparisons easier. + name = name.lower() + + # We use get_all() here, even for fields that aren't multiple use, + # because otherwise someone could have e.g. two Name fields, and we + # would just silently ignore it rather than doing something about it. + headers = parsed.get_all(name) or [] + + # The way the email module works when parsing bytes is that it + # unconditionally decodes the bytes as ascii using the surrogateescape + # handler. When you pull that data back out (such as with get_all() ), + # it looks to see if the str has any surrogate escapes, and if it does + # it wraps it in a Header object instead of returning the string. + # + # As such, we'll look for those Header objects, and fix up the encoding. + value = [] + # Flag if we have run into any issues processing the headers, thus + # signalling that the data belongs in 'unparsed'. + valid_encoding = True + for h in headers: + # It's unclear if this can return more types than just a Header or + # a str, so we'll just assert here to make sure. + assert isinstance(h, (email.header.Header, str)) + + # If it's a header object, we need to do our little dance to get + # the real data out of it. In cases where there is invalid data + # we're going to end up with mojibake, but there's no obvious, good + # way around that without reimplementing parts of the Header object + # ourselves. + # + # That should be fine since, if mojibacked happens, this key is + # going into the unparsed dict anyways. + if isinstance(h, email.header.Header): + # The Header object stores it's data as chunks, and each chunk + # can be independently encoded, so we'll need to check each + # of them. + chunks: List[Tuple[bytes, Optional[str]]] = [] + for bin, encoding in email.header.decode_header(h): + try: + bin.decode("utf8", "strict") + except UnicodeDecodeError: + # Enable mojibake. + encoding = "latin1" + valid_encoding = False + else: + encoding = "utf8" + chunks.append((bin, encoding)) + + # Turn our chunks back into a Header object, then let that + # Header object do the right thing to turn them into a + # string for us. + value.append(str(email.header.make_header(chunks))) + # This is already a string, so just add it. + else: + value.append(h) + + # We've processed all of our values to get them into a list of str, + # but we may have mojibake data, in which case this is an unparsed + # field. + if not valid_encoding: + unparsed[name] = value + continue + + raw_name = _EMAIL_TO_RAW_MAPPING.get(name) + if raw_name is None: + # This is a bit of a weird situation, we've encountered a key that + # we don't know what it means, so we don't know whether it's meant + # to be a list or not. + # + # Since we can't really tell one way or another, we'll just leave it + # as a list, even though it may be a single item list, because that's + # what makes the most sense for email headers. + unparsed[name] = value + continue + + # If this is one of our string fields, then we'll check to see if our + # value is a list of a single item. If it is then we'll assume that + # it was emitted as a single string, and unwrap the str from inside + # the list. + # + # If it's any other kind of data, then we haven't the faintest clue + # what we should parse it as, and we have to just add it to our list + # of unparsed stuff. + if raw_name in _STRING_FIELDS and len(value) == 1: + raw[raw_name] = value[0] + # If this is one of our list of string fields, then we can just assign + # the value, since email *only* has strings, and our get_all() call + # above ensures that this is a list. + elif raw_name in _LIST_FIELDS: + raw[raw_name] = value + # Special Case: Keywords + # The keywords field is implemented in the metadata spec as a str, + # but it conceptually is a list of strings, and is serialized using + # ", ".join(keywords), so we'll do some light data massaging to turn + # this into what it logically is. + elif raw_name == "keywords" and len(value) == 1: + raw[raw_name] = _parse_keywords(value[0]) + # Special Case: Project-URL + # The project urls is implemented in the metadata spec as a list of + # specially-formatted strings that represent a key and a value, which + # is fundamentally a mapping, however the email format doesn't support + # mappings in a sane way, so it was crammed into a list of strings + # instead. + # + # We will do a little light data massaging to turn this into a map as + # it logically should be. + elif raw_name == "project_urls": + try: + raw[raw_name] = _parse_project_urls(value) + except KeyError: + unparsed[name] = value + # Nothing that we've done has managed to parse this, so it'll just + # throw it in our unparseable data and move on. + else: + unparsed[name] = value + + # We need to support getting the Description from the message payload in + # addition to getting it from the the headers. This does mean, though, there + # is the possibility of it being set both ways, in which case we put both + # in 'unparsed' since we don't know which is right. + try: + payload = _get_payload(parsed, data) + except ValueError: + unparsed.setdefault("description", []).append( + parsed.get_payload(decode=isinstance(data, bytes)) + ) + else: + if payload: + # Check to see if we've already got a description, if so then both + # it, and this body move to unparseable. + if "description" in raw: + description_header = cast(str, raw.pop("description")) + unparsed.setdefault("description", []).extend( + [description_header, payload] + ) + elif "description" in unparsed: + unparsed["description"].append(payload) + else: + raw["description"] = payload + + # We need to cast our `raw` to a metadata, because a TypedDict only support + # literal key names, but we're computing our key names on purpose, but the + # way this function is implemented, our `TypedDict` can only have valid key + # names. + return cast(RawMetadata, raw), unparsed + + +_NOT_FOUND = object() + + +# Keep the two values in sync. +_VALID_METADATA_VERSIONS = ["1.0", "1.1", "1.2", "2.1", "2.2", "2.3"] +_MetadataVersion = Literal["1.0", "1.1", "1.2", "2.1", "2.2", "2.3"] + +_REQUIRED_ATTRS = frozenset(["metadata_version", "name", "version"]) + + +class _Validator(Generic[T]): + """Validate a metadata field. + + All _process_*() methods correspond to a core metadata field. The method is + called with the field's raw value. If the raw value is valid it is returned + in its "enriched" form (e.g. ``version.Version`` for the ``Version`` field). + If the raw value is invalid, :exc:`InvalidMetadata` is raised (with a cause + as appropriate). + """ + + name: str + raw_name: str + added: _MetadataVersion + + def __init__( + self, + *, + added: _MetadataVersion = "1.0", + ) -> None: + self.added = added + + def __set_name__(self, _owner: "Metadata", name: str) -> None: + self.name = name + self.raw_name = _RAW_TO_EMAIL_MAPPING[name] + + def __get__(self, instance: "Metadata", _owner: Type["Metadata"]) -> T: + # With Python 3.8, the caching can be replaced with functools.cached_property(). + # No need to check the cache as attribute lookup will resolve into the + # instance's __dict__ before __get__ is called. + cache = instance.__dict__ + value = instance._raw.get(self.name) + + # To make the _process_* methods easier, we'll check if the value is None + # and if this field is NOT a required attribute, and if both of those + # things are true, we'll skip the the converter. This will mean that the + # converters never have to deal with the None union. + if self.name in _REQUIRED_ATTRS or value is not None: + try: + converter: Callable[[Any], T] = getattr(self, f"_process_{self.name}") + except AttributeError: + pass + else: + value = converter(value) + + cache[self.name] = value + try: + del instance._raw[self.name] # type: ignore[misc] + except KeyError: + pass + + return cast(T, value) + + def _invalid_metadata( + self, msg: str, cause: Optional[Exception] = None + ) -> InvalidMetadata: + exc = InvalidMetadata( + self.raw_name, msg.format_map({"field": repr(self.raw_name)}) + ) + exc.__cause__ = cause + return exc + + def _process_metadata_version(self, value: str) -> _MetadataVersion: + # Implicitly makes Metadata-Version required. + if value not in _VALID_METADATA_VERSIONS: + raise self._invalid_metadata(f"{value!r} is not a valid metadata version") + return cast(_MetadataVersion, value) + + def _process_name(self, value: str) -> str: + if not value: + raise self._invalid_metadata("{field} is a required field") + # Validate the name as a side-effect. + try: + utils.canonicalize_name(value, validate=True) + except utils.InvalidName as exc: + raise self._invalid_metadata( + f"{value!r} is invalid for {{field}}", cause=exc + ) + else: + return value + + def _process_version(self, value: str) -> version_module.Version: + if not value: + raise self._invalid_metadata("{field} is a required field") + try: + return version_module.parse(value) + except version_module.InvalidVersion as exc: + raise self._invalid_metadata( + f"{value!r} is invalid for {{field}}", cause=exc + ) + + def _process_summary(self, value: str) -> str: + """Check the field contains no newlines.""" + if "\n" in value: + raise self._invalid_metadata("{field} must be a single line") + return value + + def _process_description_content_type(self, value: str) -> str: + content_types = {"text/plain", "text/x-rst", "text/markdown"} + message = email.message.EmailMessage() + message["content-type"] = value + + content_type, parameters = ( + # Defaults to `text/plain` if parsing failed. + message.get_content_type().lower(), + message["content-type"].params, + ) + # Check if content-type is valid or defaulted to `text/plain` and thus was + # not parseable. + if content_type not in content_types or content_type not in value.lower(): + raise self._invalid_metadata( + f"{{field}} must be one of {list(content_types)}, not {value!r}" + ) + + charset = parameters.get("charset", "UTF-8") + if charset != "UTF-8": + raise self._invalid_metadata( + f"{{field}} can only specify the UTF-8 charset, not {list(charset)}" + ) + + markdown_variants = {"GFM", "CommonMark"} + variant = parameters.get("variant", "GFM") # Use an acceptable default. + if content_type == "text/markdown" and variant not in markdown_variants: + raise self._invalid_metadata( + f"valid Markdown variants for {{field}} are {list(markdown_variants)}, " + f"not {variant!r}", + ) + return value + + def _process_dynamic(self, value: List[str]) -> List[str]: + for dynamic_field in map(str.lower, value): + if dynamic_field in {"name", "version", "metadata-version"}: + raise self._invalid_metadata( + f"{value!r} is not allowed as a dynamic field" + ) + elif dynamic_field not in _EMAIL_TO_RAW_MAPPING: + raise self._invalid_metadata(f"{value!r} is not a valid dynamic field") + return list(map(str.lower, value)) + + def _process_provides_extra( + self, + value: List[str], + ) -> List[utils.NormalizedName]: + normalized_names = [] + try: + for name in value: + normalized_names.append(utils.canonicalize_name(name, validate=True)) + except utils.InvalidName as exc: + raise self._invalid_metadata( + f"{name!r} is invalid for {{field}}", cause=exc + ) + else: + return normalized_names + + def _process_requires_python(self, value: str) -> specifiers.SpecifierSet: + try: + return specifiers.SpecifierSet(value) + except specifiers.InvalidSpecifier as exc: + raise self._invalid_metadata( + f"{value!r} is invalid for {{field}}", cause=exc + ) + + def _process_requires_dist( + self, + value: List[str], + ) -> List[requirements.Requirement]: + reqs = [] + try: + for req in value: + reqs.append(requirements.Requirement(req)) + except requirements.InvalidRequirement as exc: + raise self._invalid_metadata(f"{req!r} is invalid for {{field}}", cause=exc) + else: + return reqs + + +class Metadata: + """Representation of distribution metadata. + + Compared to :class:`RawMetadata`, this class provides objects representing + metadata fields instead of only using built-in types. Any invalid metadata + will cause :exc:`InvalidMetadata` to be raised (with a + :py:attr:`~BaseException.__cause__` attribute as appropriate). + """ + + _raw: RawMetadata + + @classmethod + def from_raw(cls, data: RawMetadata, *, validate: bool = True) -> "Metadata": + """Create an instance from :class:`RawMetadata`. + + If *validate* is true, all metadata will be validated. All exceptions + related to validation will be gathered and raised as an :class:`ExceptionGroup`. + """ + ins = cls() + ins._raw = data.copy() # Mutations occur due to caching enriched values. + + if validate: + exceptions: List[Exception] = [] + try: + metadata_version = ins.metadata_version + metadata_age = _VALID_METADATA_VERSIONS.index(metadata_version) + except InvalidMetadata as metadata_version_exc: + exceptions.append(metadata_version_exc) + metadata_version = None + + # Make sure to check for the fields that are present, the required + # fields (so their absence can be reported). + fields_to_check = frozenset(ins._raw) | _REQUIRED_ATTRS + # Remove fields that have already been checked. + fields_to_check -= {"metadata_version"} + + for key in fields_to_check: + try: + if metadata_version: + # Can't use getattr() as that triggers descriptor protocol which + # will fail due to no value for the instance argument. + try: + field_metadata_version = cls.__dict__[key].added + except KeyError: + exc = InvalidMetadata(key, f"unrecognized field: {key!r}") + exceptions.append(exc) + continue + field_age = _VALID_METADATA_VERSIONS.index( + field_metadata_version + ) + if field_age > metadata_age: + field = _RAW_TO_EMAIL_MAPPING[key] + exc = InvalidMetadata( + field, + "{field} introduced in metadata version " + "{field_metadata_version}, not {metadata_version}", + ) + exceptions.append(exc) + continue + getattr(ins, key) + except InvalidMetadata as exc: + exceptions.append(exc) + + if exceptions: + raise ExceptionGroup("invalid metadata", exceptions) + + return ins + + @classmethod + def from_email( + cls, data: Union[bytes, str], *, validate: bool = True + ) -> "Metadata": + """Parse metadata from email headers. + + If *validate* is true, the metadata will be validated. All exceptions + related to validation will be gathered and raised as an :class:`ExceptionGroup`. + """ + raw, unparsed = parse_email(data) + + if validate: + exceptions: list[Exception] = [] + for unparsed_key in unparsed: + if unparsed_key in _EMAIL_TO_RAW_MAPPING: + message = f"{unparsed_key!r} has invalid data" + else: + message = f"unrecognized field: {unparsed_key!r}" + exceptions.append(InvalidMetadata(unparsed_key, message)) + + if exceptions: + raise ExceptionGroup("unparsed", exceptions) + + try: + return cls.from_raw(raw, validate=validate) + except ExceptionGroup as exc_group: + raise ExceptionGroup( + "invalid or unparsed metadata", exc_group.exceptions + ) from None + + metadata_version: _Validator[_MetadataVersion] = _Validator() + """:external:ref:`core-metadata-metadata-version` + (required; validated to be a valid metadata version)""" + name: _Validator[str] = _Validator() + """:external:ref:`core-metadata-name` + (required; validated using :func:`~packaging.utils.canonicalize_name` and its + *validate* parameter)""" + version: _Validator[version_module.Version] = _Validator() + """:external:ref:`core-metadata-version` (required)""" + dynamic: _Validator[Optional[List[str]]] = _Validator( + added="2.2", + ) + """:external:ref:`core-metadata-dynamic` + (validated against core metadata field names and lowercased)""" + platforms: _Validator[Optional[List[str]]] = _Validator() + """:external:ref:`core-metadata-platform`""" + supported_platforms: _Validator[Optional[List[str]]] = _Validator(added="1.1") + """:external:ref:`core-metadata-supported-platform`""" + summary: _Validator[Optional[str]] = _Validator() + """:external:ref:`core-metadata-summary` (validated to contain no newlines)""" + description: _Validator[Optional[str]] = _Validator() # TODO 2.1: can be in body + """:external:ref:`core-metadata-description`""" + description_content_type: _Validator[Optional[str]] = _Validator(added="2.1") + """:external:ref:`core-metadata-description-content-type` (validated)""" + keywords: _Validator[Optional[List[str]]] = _Validator() + """:external:ref:`core-metadata-keywords`""" + home_page: _Validator[Optional[str]] = _Validator() + """:external:ref:`core-metadata-home-page`""" + download_url: _Validator[Optional[str]] = _Validator(added="1.1") + """:external:ref:`core-metadata-download-url`""" + author: _Validator[Optional[str]] = _Validator() + """:external:ref:`core-metadata-author`""" + author_email: _Validator[Optional[str]] = _Validator() + """:external:ref:`core-metadata-author-email`""" + maintainer: _Validator[Optional[str]] = _Validator(added="1.2") + """:external:ref:`core-metadata-maintainer`""" + maintainer_email: _Validator[Optional[str]] = _Validator(added="1.2") + """:external:ref:`core-metadata-maintainer-email`""" + license: _Validator[Optional[str]] = _Validator() + """:external:ref:`core-metadata-license`""" + classifiers: _Validator[Optional[List[str]]] = _Validator(added="1.1") + """:external:ref:`core-metadata-classifier`""" + requires_dist: _Validator[Optional[List[requirements.Requirement]]] = _Validator( + added="1.2" + ) + """:external:ref:`core-metadata-requires-dist`""" + requires_python: _Validator[Optional[specifiers.SpecifierSet]] = _Validator( + added="1.2" + ) + """:external:ref:`core-metadata-requires-python`""" + # Because `Requires-External` allows for non-PEP 440 version specifiers, we + # don't do any processing on the values. + requires_external: _Validator[Optional[List[str]]] = _Validator(added="1.2") + """:external:ref:`core-metadata-requires-external`""" + project_urls: _Validator[Optional[Dict[str, str]]] = _Validator(added="1.2") + """:external:ref:`core-metadata-project-url`""" + # PEP 685 lets us raise an error if an extra doesn't pass `Name` validation + # regardless of metadata version. + provides_extra: _Validator[Optional[List[utils.NormalizedName]]] = _Validator( + added="2.1", + ) + """:external:ref:`core-metadata-provides-extra`""" + provides_dist: _Validator[Optional[List[str]]] = _Validator(added="1.2") + """:external:ref:`core-metadata-provides-dist`""" + obsoletes_dist: _Validator[Optional[List[str]]] = _Validator(added="1.2") + """:external:ref:`core-metadata-obsoletes-dist`""" + requires: _Validator[Optional[List[str]]] = _Validator(added="1.1") + """``Requires`` (deprecated)""" + provides: _Validator[Optional[List[str]]] = _Validator(added="1.1") + """``Provides`` (deprecated)""" + obsoletes: _Validator[Optional[List[str]]] = _Validator(added="1.1") + """``Obsoletes`` (deprecated)""" diff --git a/gyp/pylib/packaging/py.typed b/gyp/pylib/packaging/py.typed new file mode 100644 index 0000000000..e69de29bb2 diff --git a/gyp/pylib/packaging/requirements.py b/gyp/pylib/packaging/requirements.py new file mode 100644 index 0000000000..0c00eba331 --- /dev/null +++ b/gyp/pylib/packaging/requirements.py @@ -0,0 +1,90 @@ +# This file is dual licensed under the terms of the Apache License, Version +# 2.0, and the BSD License. See the LICENSE file in the root of this repository +# for complete details. + +from typing import Any, Iterator, Optional, Set + +from ._parser import parse_requirement as _parse_requirement +from ._tokenizer import ParserSyntaxError +from .markers import Marker, _normalize_extra_values +from .specifiers import SpecifierSet +from .utils import canonicalize_name + + +class InvalidRequirement(ValueError): + """ + An invalid requirement was found, users should refer to PEP 508. + """ + + +class Requirement: + """Parse a requirement. + + Parse a given requirement string into its parts, such as name, specifier, + URL, and extras. Raises InvalidRequirement on a badly-formed requirement + string. + """ + + # TODO: Can we test whether something is contained within a requirement? + # If so how do we do that? Do we need to test against the _name_ of + # the thing as well as the version? What about the markers? + # TODO: Can we normalize the name and extra name? + + def __init__(self, requirement_string: str) -> None: + try: + parsed = _parse_requirement(requirement_string) + except ParserSyntaxError as e: + raise InvalidRequirement(str(e)) from e + + self.name: str = parsed.name + self.url: Optional[str] = parsed.url or None + self.extras: Set[str] = set(parsed.extras if parsed.extras else []) + self.specifier: SpecifierSet = SpecifierSet(parsed.specifier) + self.marker: Optional[Marker] = None + if parsed.marker is not None: + self.marker = Marker.__new__(Marker) + self.marker._markers = _normalize_extra_values(parsed.marker) + + def _iter_parts(self, name: str) -> Iterator[str]: + yield name + + if self.extras: + formatted_extras = ",".join(sorted(self.extras)) + yield f"[{formatted_extras}]" + + if self.specifier: + yield str(self.specifier) + + if self.url: + yield f"@ {self.url}" + if self.marker: + yield " " + + if self.marker: + yield f"; {self.marker}" + + def __str__(self) -> str: + return "".join(self._iter_parts(self.name)) + + def __repr__(self) -> str: + return f"" + + def __hash__(self) -> int: + return hash( + ( + self.__class__.__name__, + *self._iter_parts(canonicalize_name(self.name)), + ) + ) + + def __eq__(self, other: Any) -> bool: + if not isinstance(other, Requirement): + return NotImplemented + + return ( + canonicalize_name(self.name) == canonicalize_name(other.name) + and self.extras == other.extras + and self.specifier == other.specifier + and self.url == other.url + and self.marker == other.marker + ) diff --git a/gyp/pylib/packaging/specifiers.py b/gyp/pylib/packaging/specifiers.py new file mode 100644 index 0000000000..94448327ae --- /dev/null +++ b/gyp/pylib/packaging/specifiers.py @@ -0,0 +1,1030 @@ +# This file is dual licensed under the terms of the Apache License, Version +# 2.0, and the BSD License. See the LICENSE file in the root of this repository +# for complete details. +""" +.. testsetup:: + + from packaging.specifiers import Specifier, SpecifierSet, InvalidSpecifier + from packaging.version import Version +""" + +import abc +import itertools +import re +from typing import ( + Callable, + Iterable, + Iterator, + List, + Optional, + Set, + Tuple, + TypeVar, + Union, +) + +from .utils import canonicalize_version +from .version import Version + +UnparsedVersion = Union[Version, str] +UnparsedVersionVar = TypeVar("UnparsedVersionVar", bound=UnparsedVersion) +CallableOperator = Callable[[Version, str], bool] + + +def _coerce_version(version: UnparsedVersion) -> Version: + if not isinstance(version, Version): + version = Version(version) + return version + + +class InvalidSpecifier(ValueError): + """ + Raised when attempting to create a :class:`Specifier` with a specifier + string that is invalid. + + >>> Specifier("lolwat") + Traceback (most recent call last): + ... + packaging.specifiers.InvalidSpecifier: Invalid specifier: 'lolwat' + """ + + +class BaseSpecifier(metaclass=abc.ABCMeta): + @abc.abstractmethod + def __str__(self) -> str: + """ + Returns the str representation of this Specifier-like object. This + should be representative of the Specifier itself. + """ + + @abc.abstractmethod + def __hash__(self) -> int: + """ + Returns a hash value for this Specifier-like object. + """ + + @abc.abstractmethod + def __eq__(self, other: object) -> bool: + """ + Returns a boolean representing whether or not the two Specifier-like + objects are equal. + + :param other: The other object to check against. + """ + + @property + @abc.abstractmethod + def prereleases(self) -> Optional[bool]: + """Whether or not pre-releases as a whole are allowed. + + This can be set to either ``True`` or ``False`` to explicitly enable or disable + prereleases or it can be set to ``None`` (the default) to use default semantics. + """ + + @prereleases.setter + def prereleases(self, value: bool) -> None: + """Setter for :attr:`prereleases`. + + :param value: The value to set. + """ + + @abc.abstractmethod + def contains(self, item: str, prereleases: Optional[bool] = None) -> bool: + """ + Determines if the given item is contained within this specifier. + """ + + @abc.abstractmethod + def filter( + self, iterable: Iterable[UnparsedVersionVar], prereleases: Optional[bool] = None + ) -> Iterator[UnparsedVersionVar]: + """ + Takes an iterable of items and filters them so that only items which + are contained within this specifier are allowed in it. + """ + + +class Specifier(BaseSpecifier): + """This class abstracts handling of version specifiers. + + .. tip:: + + It is generally not required to instantiate this manually. You should instead + prefer to work with :class:`SpecifierSet` instead, which can parse + comma-separated version specifiers (which is what package metadata contains). + """ + + _operator_regex_str = r""" + (?P(~=|==|!=|<=|>=|<|>|===)) + """ + _version_regex_str = r""" + (?P + (?: + # The identity operators allow for an escape hatch that will + # do an exact string match of the version you wish to install. + # This will not be parsed by PEP 440 and we cannot determine + # any semantic meaning from it. This operator is discouraged + # but included entirely as an escape hatch. + (?<====) # Only match for the identity operator + \s* + [^\s;)]* # The arbitrary version can be just about anything, + # we match everything except for whitespace, a + # semi-colon for marker support, and a closing paren + # since versions can be enclosed in them. + ) + | + (?: + # The (non)equality operators allow for wild card and local + # versions to be specified so we have to define these two + # operators separately to enable that. + (?<===|!=) # Only match for equals and not equals + + \s* + v? + (?:[0-9]+!)? # epoch + [0-9]+(?:\.[0-9]+)* # release + + # You cannot use a wild card and a pre-release, post-release, a dev or + # local version together so group them with a | and make them optional. + (?: + \.\* # Wild card syntax of .* + | + (?: # pre release + [-_\.]? + (alpha|beta|preview|pre|a|b|c|rc) + [-_\.]? + [0-9]* + )? + (?: # post release + (?:-[0-9]+)|(?:[-_\.]?(post|rev|r)[-_\.]?[0-9]*) + )? + (?:[-_\.]?dev[-_\.]?[0-9]*)? # dev release + (?:\+[a-z0-9]+(?:[-_\.][a-z0-9]+)*)? # local + )? + ) + | + (?: + # The compatible operator requires at least two digits in the + # release segment. + (?<=~=) # Only match for the compatible operator + + \s* + v? + (?:[0-9]+!)? # epoch + [0-9]+(?:\.[0-9]+)+ # release (We have a + instead of a *) + (?: # pre release + [-_\.]? + (alpha|beta|preview|pre|a|b|c|rc) + [-_\.]? + [0-9]* + )? + (?: # post release + (?:-[0-9]+)|(?:[-_\.]?(post|rev|r)[-_\.]?[0-9]*) + )? + (?:[-_\.]?dev[-_\.]?[0-9]*)? # dev release + ) + | + (?: + # All other operators only allow a sub set of what the + # (non)equality operators do. Specifically they do not allow + # local versions to be specified nor do they allow the prefix + # matching wild cards. + (?=": "greater_than_equal", + "<": "less_than", + ">": "greater_than", + "===": "arbitrary", + } + + def __init__(self, spec: str = "", prereleases: Optional[bool] = None) -> None: + """Initialize a Specifier instance. + + :param spec: + The string representation of a specifier which will be parsed and + normalized before use. + :param prereleases: + This tells the specifier if it should accept prerelease versions if + applicable or not. The default of ``None`` will autodetect it from the + given specifiers. + :raises InvalidSpecifier: + If the given specifier is invalid (i.e. bad syntax). + """ + match = self._regex.search(spec) + if not match: + raise InvalidSpecifier(f"Invalid specifier: '{spec}'") + + self._spec: Tuple[str, str] = ( + match.group("operator").strip(), + match.group("version").strip(), + ) + + # Store whether or not this Specifier should accept prereleases + self._prereleases = prereleases + + # https://github.com/python/mypy/pull/13475#pullrequestreview-1079784515 + @property # type: ignore[override] + def prereleases(self) -> bool: + # If there is an explicit prereleases set for this, then we'll just + # blindly use that. + if self._prereleases is not None: + return self._prereleases + + # Look at all of our specifiers and determine if they are inclusive + # operators, and if they are if they are including an explicit + # prerelease. + operator, version = self._spec + if operator in ["==", ">=", "<=", "~=", "==="]: + # The == specifier can include a trailing .*, if it does we + # want to remove before parsing. + if operator == "==" and version.endswith(".*"): + version = version[:-2] + + # Parse the version, and if it is a pre-release than this + # specifier allows pre-releases. + if Version(version).is_prerelease: + return True + + return False + + @prereleases.setter + def prereleases(self, value: bool) -> None: + self._prereleases = value + + @property + def operator(self) -> str: + """The operator of this specifier. + + >>> Specifier("==1.2.3").operator + '==' + """ + return self._spec[0] + + @property + def version(self) -> str: + """The version of this specifier. + + >>> Specifier("==1.2.3").version + '1.2.3' + """ + return self._spec[1] + + def __repr__(self) -> str: + """A representation of the Specifier that shows all internal state. + + >>> Specifier('>=1.0.0') + =1.0.0')> + >>> Specifier('>=1.0.0', prereleases=False) + =1.0.0', prereleases=False)> + >>> Specifier('>=1.0.0', prereleases=True) + =1.0.0', prereleases=True)> + """ + pre = ( + f", prereleases={self.prereleases!r}" + if self._prereleases is not None + else "" + ) + + return f"<{self.__class__.__name__}({str(self)!r}{pre})>" + + def __str__(self) -> str: + """A string representation of the Specifier that can be round-tripped. + + >>> str(Specifier('>=1.0.0')) + '>=1.0.0' + >>> str(Specifier('>=1.0.0', prereleases=False)) + '>=1.0.0' + """ + return "{}{}".format(*self._spec) + + @property + def _canonical_spec(self) -> Tuple[str, str]: + canonical_version = canonicalize_version( + self._spec[1], + strip_trailing_zero=(self._spec[0] != "~="), + ) + return self._spec[0], canonical_version + + def __hash__(self) -> int: + return hash(self._canonical_spec) + + def __eq__(self, other: object) -> bool: + """Whether or not the two Specifier-like objects are equal. + + :param other: The other object to check against. + + The value of :attr:`prereleases` is ignored. + + >>> Specifier("==1.2.3") == Specifier("== 1.2.3.0") + True + >>> (Specifier("==1.2.3", prereleases=False) == + ... Specifier("==1.2.3", prereleases=True)) + True + >>> Specifier("==1.2.3") == "==1.2.3" + True + >>> Specifier("==1.2.3") == Specifier("==1.2.4") + False + >>> Specifier("==1.2.3") == Specifier("~=1.2.3") + False + """ + if isinstance(other, str): + try: + other = self.__class__(str(other)) + except InvalidSpecifier: + return NotImplemented + elif not isinstance(other, self.__class__): + return NotImplemented + + return self._canonical_spec == other._canonical_spec + + def _get_operator(self, op: str) -> CallableOperator: + operator_callable: CallableOperator = getattr( + self, f"_compare_{self._operators[op]}" + ) + return operator_callable + + def _compare_compatible(self, prospective: Version, spec: str) -> bool: + + # Compatible releases have an equivalent combination of >= and ==. That + # is that ~=2.2 is equivalent to >=2.2,==2.*. This allows us to + # implement this in terms of the other specifiers instead of + # implementing it ourselves. The only thing we need to do is construct + # the other specifiers. + + # We want everything but the last item in the version, but we want to + # ignore suffix segments. + prefix = _version_join( + list(itertools.takewhile(_is_not_suffix, _version_split(spec)))[:-1] + ) + + # Add the prefix notation to the end of our string + prefix += ".*" + + return self._get_operator(">=")(prospective, spec) and self._get_operator("==")( + prospective, prefix + ) + + def _compare_equal(self, prospective: Version, spec: str) -> bool: + + # We need special logic to handle prefix matching + if spec.endswith(".*"): + # In the case of prefix matching we want to ignore local segment. + normalized_prospective = canonicalize_version( + prospective.public, strip_trailing_zero=False + ) + # Get the normalized version string ignoring the trailing .* + normalized_spec = canonicalize_version(spec[:-2], strip_trailing_zero=False) + # Split the spec out by bangs and dots, and pretend that there is + # an implicit dot in between a release segment and a pre-release segment. + split_spec = _version_split(normalized_spec) + + # Split the prospective version out by bangs and dots, and pretend + # that there is an implicit dot in between a release segment and + # a pre-release segment. + split_prospective = _version_split(normalized_prospective) + + # 0-pad the prospective version before shortening it to get the correct + # shortened version. + padded_prospective, _ = _pad_version(split_prospective, split_spec) + + # Shorten the prospective version to be the same length as the spec + # so that we can determine if the specifier is a prefix of the + # prospective version or not. + shortened_prospective = padded_prospective[: len(split_spec)] + + return shortened_prospective == split_spec + else: + # Convert our spec string into a Version + spec_version = Version(spec) + + # If the specifier does not have a local segment, then we want to + # act as if the prospective version also does not have a local + # segment. + if not spec_version.local: + prospective = Version(prospective.public) + + return prospective == spec_version + + def _compare_not_equal(self, prospective: Version, spec: str) -> bool: + return not self._compare_equal(prospective, spec) + + def _compare_less_than_equal(self, prospective: Version, spec: str) -> bool: + + # NB: Local version identifiers are NOT permitted in the version + # specifier, so local version labels can be universally removed from + # the prospective version. + return Version(prospective.public) <= Version(spec) + + def _compare_greater_than_equal(self, prospective: Version, spec: str) -> bool: + + # NB: Local version identifiers are NOT permitted in the version + # specifier, so local version labels can be universally removed from + # the prospective version. + return Version(prospective.public) >= Version(spec) + + def _compare_less_than(self, prospective: Version, spec_str: str) -> bool: + + # Convert our spec to a Version instance, since we'll want to work with + # it as a version. + spec = Version(spec_str) + + # Check to see if the prospective version is less than the spec + # version. If it's not we can short circuit and just return False now + # instead of doing extra unneeded work. + if not prospective < spec: + return False + + # This special case is here so that, unless the specifier itself + # includes is a pre-release version, that we do not accept pre-release + # versions for the version mentioned in the specifier (e.g. <3.1 should + # not match 3.1.dev0, but should match 3.0.dev0). + if not spec.is_prerelease and prospective.is_prerelease: + if Version(prospective.base_version) == Version(spec.base_version): + return False + + # If we've gotten to here, it means that prospective version is both + # less than the spec version *and* it's not a pre-release of the same + # version in the spec. + return True + + def _compare_greater_than(self, prospective: Version, spec_str: str) -> bool: + + # Convert our spec to a Version instance, since we'll want to work with + # it as a version. + spec = Version(spec_str) + + # Check to see if the prospective version is greater than the spec + # version. If it's not we can short circuit and just return False now + # instead of doing extra unneeded work. + if not prospective > spec: + return False + + # This special case is here so that, unless the specifier itself + # includes is a post-release version, that we do not accept + # post-release versions for the version mentioned in the specifier + # (e.g. >3.1 should not match 3.0.post0, but should match 3.2.post0). + if not spec.is_postrelease and prospective.is_postrelease: + if Version(prospective.base_version) == Version(spec.base_version): + return False + + # Ensure that we do not allow a local version of the version mentioned + # in the specifier, which is technically greater than, to match. + if prospective.local is not None: + if Version(prospective.base_version) == Version(spec.base_version): + return False + + # If we've gotten to here, it means that prospective version is both + # greater than the spec version *and* it's not a pre-release of the + # same version in the spec. + return True + + def _compare_arbitrary(self, prospective: Version, spec: str) -> bool: + return str(prospective).lower() == str(spec).lower() + + def __contains__(self, item: Union[str, Version]) -> bool: + """Return whether or not the item is contained in this specifier. + + :param item: The item to check for. + + This is used for the ``in`` operator and behaves the same as + :meth:`contains` with no ``prereleases`` argument passed. + + >>> "1.2.3" in Specifier(">=1.2.3") + True + >>> Version("1.2.3") in Specifier(">=1.2.3") + True + >>> "1.0.0" in Specifier(">=1.2.3") + False + >>> "1.3.0a1" in Specifier(">=1.2.3") + False + >>> "1.3.0a1" in Specifier(">=1.2.3", prereleases=True) + True + """ + return self.contains(item) + + def contains( + self, item: UnparsedVersion, prereleases: Optional[bool] = None + ) -> bool: + """Return whether or not the item is contained in this specifier. + + :param item: + The item to check for, which can be a version string or a + :class:`Version` instance. + :param prereleases: + Whether or not to match prereleases with this Specifier. If set to + ``None`` (the default), it uses :attr:`prereleases` to determine + whether or not prereleases are allowed. + + >>> Specifier(">=1.2.3").contains("1.2.3") + True + >>> Specifier(">=1.2.3").contains(Version("1.2.3")) + True + >>> Specifier(">=1.2.3").contains("1.0.0") + False + >>> Specifier(">=1.2.3").contains("1.3.0a1") + False + >>> Specifier(">=1.2.3", prereleases=True).contains("1.3.0a1") + True + >>> Specifier(">=1.2.3").contains("1.3.0a1", prereleases=True) + True + """ + + # Determine if prereleases are to be allowed or not. + if prereleases is None: + prereleases = self.prereleases + + # Normalize item to a Version, this allows us to have a shortcut for + # "2.0" in Specifier(">=2") + normalized_item = _coerce_version(item) + + # Determine if we should be supporting prereleases in this specifier + # or not, if we do not support prereleases than we can short circuit + # logic if this version is a prereleases. + if normalized_item.is_prerelease and not prereleases: + return False + + # Actually do the comparison to determine if this item is contained + # within this Specifier or not. + operator_callable: CallableOperator = self._get_operator(self.operator) + return operator_callable(normalized_item, self.version) + + def filter( + self, iterable: Iterable[UnparsedVersionVar], prereleases: Optional[bool] = None + ) -> Iterator[UnparsedVersionVar]: + """Filter items in the given iterable, that match the specifier. + + :param iterable: + An iterable that can contain version strings and :class:`Version` instances. + The items in the iterable will be filtered according to the specifier. + :param prereleases: + Whether or not to allow prereleases in the returned iterator. If set to + ``None`` (the default), it will be intelligently decide whether to allow + prereleases or not (based on the :attr:`prereleases` attribute, and + whether the only versions matching are prereleases). + + This method is smarter than just ``filter(Specifier().contains, [...])`` + because it implements the rule from :pep:`440` that a prerelease item + SHOULD be accepted if no other versions match the given specifier. + + >>> list(Specifier(">=1.2.3").filter(["1.2", "1.3", "1.5a1"])) + ['1.3'] + >>> list(Specifier(">=1.2.3").filter(["1.2", "1.2.3", "1.3", Version("1.4")])) + ['1.2.3', '1.3', ] + >>> list(Specifier(">=1.2.3").filter(["1.2", "1.5a1"])) + ['1.5a1'] + >>> list(Specifier(">=1.2.3").filter(["1.3", "1.5a1"], prereleases=True)) + ['1.3', '1.5a1'] + >>> list(Specifier(">=1.2.3", prereleases=True).filter(["1.3", "1.5a1"])) + ['1.3', '1.5a1'] + """ + + yielded = False + found_prereleases = [] + + kw = {"prereleases": prereleases if prereleases is not None else True} + + # Attempt to iterate over all the values in the iterable and if any of + # them match, yield them. + for version in iterable: + parsed_version = _coerce_version(version) + + if self.contains(parsed_version, **kw): + # If our version is a prerelease, and we were not set to allow + # prereleases, then we'll store it for later in case nothing + # else matches this specifier. + if parsed_version.is_prerelease and not ( + prereleases or self.prereleases + ): + found_prereleases.append(version) + # Either this is not a prerelease, or we should have been + # accepting prereleases from the beginning. + else: + yielded = True + yield version + + # Now that we've iterated over everything, determine if we've yielded + # any values, and if we have not and we have any prereleases stored up + # then we will go ahead and yield the prereleases. + if not yielded and found_prereleases: + for version in found_prereleases: + yield version + + +_prefix_regex = re.compile(r"^([0-9]+)((?:a|b|c|rc)[0-9]+)$") + + +def _version_split(version: str) -> List[str]: + """Split version into components. + + The split components are intended for version comparison. The logic does + not attempt to retain the original version string, so joining the + components back with :func:`_version_join` may not produce the original + version string. + """ + result: List[str] = [] + + epoch, _, rest = version.rpartition("!") + result.append(epoch or "0") + + for item in rest.split("."): + match = _prefix_regex.search(item) + if match: + result.extend(match.groups()) + else: + result.append(item) + return result + + +def _version_join(components: List[str]) -> str: + """Join split version components into a version string. + + This function assumes the input came from :func:`_version_split`, where the + first component must be the epoch (either empty or numeric), and all other + components numeric. + """ + epoch, *rest = components + return f"{epoch}!{'.'.join(rest)}" + + +def _is_not_suffix(segment: str) -> bool: + return not any( + segment.startswith(prefix) for prefix in ("dev", "a", "b", "rc", "post") + ) + + +def _pad_version(left: List[str], right: List[str]) -> Tuple[List[str], List[str]]: + left_split, right_split = [], [] + + # Get the release segment of our versions + left_split.append(list(itertools.takewhile(lambda x: x.isdigit(), left))) + right_split.append(list(itertools.takewhile(lambda x: x.isdigit(), right))) + + # Get the rest of our versions + left_split.append(left[len(left_split[0]) :]) + right_split.append(right[len(right_split[0]) :]) + + # Insert our padding + left_split.insert(1, ["0"] * max(0, len(right_split[0]) - len(left_split[0]))) + right_split.insert(1, ["0"] * max(0, len(left_split[0]) - len(right_split[0]))) + + return (list(itertools.chain(*left_split)), list(itertools.chain(*right_split))) + + +class SpecifierSet(BaseSpecifier): + """This class abstracts handling of a set of version specifiers. + + It can be passed a single specifier (``>=3.0``), a comma-separated list of + specifiers (``>=3.0,!=3.1``), or no specifier at all. + """ + + def __init__( + self, specifiers: str = "", prereleases: Optional[bool] = None + ) -> None: + """Initialize a SpecifierSet instance. + + :param specifiers: + The string representation of a specifier or a comma-separated list of + specifiers which will be parsed and normalized before use. + :param prereleases: + This tells the SpecifierSet if it should accept prerelease versions if + applicable or not. The default of ``None`` will autodetect it from the + given specifiers. + + :raises InvalidSpecifier: + If the given ``specifiers`` are not parseable than this exception will be + raised. + """ + + # Split on `,` to break each individual specifier into it's own item, and + # strip each item to remove leading/trailing whitespace. + split_specifiers = [s.strip() for s in specifiers.split(",") if s.strip()] + + # Parsed each individual specifier, attempting first to make it a + # Specifier. + parsed: Set[Specifier] = set() + for specifier in split_specifiers: + parsed.add(Specifier(specifier)) + + # Turn our parsed specifiers into a frozen set and save them for later. + self._specs = frozenset(parsed) + + # Store our prereleases value so we can use it later to determine if + # we accept prereleases or not. + self._prereleases = prereleases + + @property + def prereleases(self) -> Optional[bool]: + # If we have been given an explicit prerelease modifier, then we'll + # pass that through here. + if self._prereleases is not None: + return self._prereleases + + # If we don't have any specifiers, and we don't have a forced value, + # then we'll just return None since we don't know if this should have + # pre-releases or not. + if not self._specs: + return None + + # Otherwise we'll see if any of the given specifiers accept + # prereleases, if any of them do we'll return True, otherwise False. + return any(s.prereleases for s in self._specs) + + @prereleases.setter + def prereleases(self, value: bool) -> None: + self._prereleases = value + + def __repr__(self) -> str: + """A representation of the specifier set that shows all internal state. + + Note that the ordering of the individual specifiers within the set may not + match the input string. + + >>> SpecifierSet('>=1.0.0,!=2.0.0') + =1.0.0')> + >>> SpecifierSet('>=1.0.0,!=2.0.0', prereleases=False) + =1.0.0', prereleases=False)> + >>> SpecifierSet('>=1.0.0,!=2.0.0', prereleases=True) + =1.0.0', prereleases=True)> + """ + pre = ( + f", prereleases={self.prereleases!r}" + if self._prereleases is not None + else "" + ) + + return f"" + + def __str__(self) -> str: + """A string representation of the specifier set that can be round-tripped. + + Note that the ordering of the individual specifiers within the set may not + match the input string. + + >>> str(SpecifierSet(">=1.0.0,!=1.0.1")) + '!=1.0.1,>=1.0.0' + >>> str(SpecifierSet(">=1.0.0,!=1.0.1", prereleases=False)) + '!=1.0.1,>=1.0.0' + """ + return ",".join(sorted(str(s) for s in self._specs)) + + def __hash__(self) -> int: + return hash(self._specs) + + def __and__(self, other: Union["SpecifierSet", str]) -> "SpecifierSet": + """Return a SpecifierSet which is a combination of the two sets. + + :param other: The other object to combine with. + + >>> SpecifierSet(">=1.0.0,!=1.0.1") & '<=2.0.0,!=2.0.1' + =1.0.0')> + >>> SpecifierSet(">=1.0.0,!=1.0.1") & SpecifierSet('<=2.0.0,!=2.0.1') + =1.0.0')> + """ + if isinstance(other, str): + other = SpecifierSet(other) + elif not isinstance(other, SpecifierSet): + return NotImplemented + + specifier = SpecifierSet() + specifier._specs = frozenset(self._specs | other._specs) + + if self._prereleases is None and other._prereleases is not None: + specifier._prereleases = other._prereleases + elif self._prereleases is not None and other._prereleases is None: + specifier._prereleases = self._prereleases + elif self._prereleases == other._prereleases: + specifier._prereleases = self._prereleases + else: + raise ValueError( + "Cannot combine SpecifierSets with True and False prerelease " + "overrides." + ) + + return specifier + + def __eq__(self, other: object) -> bool: + """Whether or not the two SpecifierSet-like objects are equal. + + :param other: The other object to check against. + + The value of :attr:`prereleases` is ignored. + + >>> SpecifierSet(">=1.0.0,!=1.0.1") == SpecifierSet(">=1.0.0,!=1.0.1") + True + >>> (SpecifierSet(">=1.0.0,!=1.0.1", prereleases=False) == + ... SpecifierSet(">=1.0.0,!=1.0.1", prereleases=True)) + True + >>> SpecifierSet(">=1.0.0,!=1.0.1") == ">=1.0.0,!=1.0.1" + True + >>> SpecifierSet(">=1.0.0,!=1.0.1") == SpecifierSet(">=1.0.0") + False + >>> SpecifierSet(">=1.0.0,!=1.0.1") == SpecifierSet(">=1.0.0,!=1.0.2") + False + """ + if isinstance(other, (str, Specifier)): + other = SpecifierSet(str(other)) + elif not isinstance(other, SpecifierSet): + return NotImplemented + + return self._specs == other._specs + + def __len__(self) -> int: + """Returns the number of specifiers in this specifier set.""" + return len(self._specs) + + def __iter__(self) -> Iterator[Specifier]: + """ + Returns an iterator over all the underlying :class:`Specifier` instances + in this specifier set. + + >>> sorted(SpecifierSet(">=1.0.0,!=1.0.1"), key=str) + [, =1.0.0')>] + """ + return iter(self._specs) + + def __contains__(self, item: UnparsedVersion) -> bool: + """Return whether or not the item is contained in this specifier. + + :param item: The item to check for. + + This is used for the ``in`` operator and behaves the same as + :meth:`contains` with no ``prereleases`` argument passed. + + >>> "1.2.3" in SpecifierSet(">=1.0.0,!=1.0.1") + True + >>> Version("1.2.3") in SpecifierSet(">=1.0.0,!=1.0.1") + True + >>> "1.0.1" in SpecifierSet(">=1.0.0,!=1.0.1") + False + >>> "1.3.0a1" in SpecifierSet(">=1.0.0,!=1.0.1") + False + >>> "1.3.0a1" in SpecifierSet(">=1.0.0,!=1.0.1", prereleases=True) + True + """ + return self.contains(item) + + def contains( + self, + item: UnparsedVersion, + prereleases: Optional[bool] = None, + installed: Optional[bool] = None, + ) -> bool: + """Return whether or not the item is contained in this SpecifierSet. + + :param item: + The item to check for, which can be a version string or a + :class:`Version` instance. + :param prereleases: + Whether or not to match prereleases with this SpecifierSet. If set to + ``None`` (the default), it uses :attr:`prereleases` to determine + whether or not prereleases are allowed. + + >>> SpecifierSet(">=1.0.0,!=1.0.1").contains("1.2.3") + True + >>> SpecifierSet(">=1.0.0,!=1.0.1").contains(Version("1.2.3")) + True + >>> SpecifierSet(">=1.0.0,!=1.0.1").contains("1.0.1") + False + >>> SpecifierSet(">=1.0.0,!=1.0.1").contains("1.3.0a1") + False + >>> SpecifierSet(">=1.0.0,!=1.0.1", prereleases=True).contains("1.3.0a1") + True + >>> SpecifierSet(">=1.0.0,!=1.0.1").contains("1.3.0a1", prereleases=True) + True + """ + # Ensure that our item is a Version instance. + if not isinstance(item, Version): + item = Version(item) + + # Determine if we're forcing a prerelease or not, if we're not forcing + # one for this particular filter call, then we'll use whatever the + # SpecifierSet thinks for whether or not we should support prereleases. + if prereleases is None: + prereleases = self.prereleases + + # We can determine if we're going to allow pre-releases by looking to + # see if any of the underlying items supports them. If none of them do + # and this item is a pre-release then we do not allow it and we can + # short circuit that here. + # Note: This means that 1.0.dev1 would not be contained in something + # like >=1.0.devabc however it would be in >=1.0.debabc,>0.0.dev0 + if not prereleases and item.is_prerelease: + return False + + if installed and item.is_prerelease: + item = Version(item.base_version) + + # We simply dispatch to the underlying specs here to make sure that the + # given version is contained within all of them. + # Note: This use of all() here means that an empty set of specifiers + # will always return True, this is an explicit design decision. + return all(s.contains(item, prereleases=prereleases) for s in self._specs) + + def filter( + self, iterable: Iterable[UnparsedVersionVar], prereleases: Optional[bool] = None + ) -> Iterator[UnparsedVersionVar]: + """Filter items in the given iterable, that match the specifiers in this set. + + :param iterable: + An iterable that can contain version strings and :class:`Version` instances. + The items in the iterable will be filtered according to the specifier. + :param prereleases: + Whether or not to allow prereleases in the returned iterator. If set to + ``None`` (the default), it will be intelligently decide whether to allow + prereleases or not (based on the :attr:`prereleases` attribute, and + whether the only versions matching are prereleases). + + This method is smarter than just ``filter(SpecifierSet(...).contains, [...])`` + because it implements the rule from :pep:`440` that a prerelease item + SHOULD be accepted if no other versions match the given specifier. + + >>> list(SpecifierSet(">=1.2.3").filter(["1.2", "1.3", "1.5a1"])) + ['1.3'] + >>> list(SpecifierSet(">=1.2.3").filter(["1.2", "1.3", Version("1.4")])) + ['1.3', ] + >>> list(SpecifierSet(">=1.2.3").filter(["1.2", "1.5a1"])) + [] + >>> list(SpecifierSet(">=1.2.3").filter(["1.3", "1.5a1"], prereleases=True)) + ['1.3', '1.5a1'] + >>> list(SpecifierSet(">=1.2.3", prereleases=True).filter(["1.3", "1.5a1"])) + ['1.3', '1.5a1'] + + An "empty" SpecifierSet will filter items based on the presence of prerelease + versions in the set. + + >>> list(SpecifierSet("").filter(["1.3", "1.5a1"])) + ['1.3'] + >>> list(SpecifierSet("").filter(["1.5a1"])) + ['1.5a1'] + >>> list(SpecifierSet("", prereleases=True).filter(["1.3", "1.5a1"])) + ['1.3', '1.5a1'] + >>> list(SpecifierSet("").filter(["1.3", "1.5a1"], prereleases=True)) + ['1.3', '1.5a1'] + """ + # Determine if we're forcing a prerelease or not, if we're not forcing + # one for this particular filter call, then we'll use whatever the + # SpecifierSet thinks for whether or not we should support prereleases. + if prereleases is None: + prereleases = self.prereleases + + # If we have any specifiers, then we want to wrap our iterable in the + # filter method for each one, this will act as a logical AND amongst + # each specifier. + if self._specs: + for spec in self._specs: + iterable = spec.filter(iterable, prereleases=bool(prereleases)) + return iter(iterable) + # If we do not have any specifiers, then we need to have a rough filter + # which will filter out any pre-releases, unless there are no final + # releases. + else: + filtered: List[UnparsedVersionVar] = [] + found_prereleases: List[UnparsedVersionVar] = [] + + for item in iterable: + parsed_version = _coerce_version(item) + + # Store any item which is a pre-release for later unless we've + # already found a final version or we are accepting prereleases + if parsed_version.is_prerelease and not prereleases: + if not filtered: + found_prereleases.append(item) + else: + filtered.append(item) + + # If we've found no items except for pre-releases, then we'll go + # ahead and use the pre-releases + if not filtered and found_prereleases and prereleases is None: + return iter(found_prereleases) + + return iter(filtered) diff --git a/gyp/pylib/packaging/tags.py b/gyp/pylib/packaging/tags.py new file mode 100644 index 0000000000..37f33b1ef8 --- /dev/null +++ b/gyp/pylib/packaging/tags.py @@ -0,0 +1,553 @@ +# This file is dual licensed under the terms of the Apache License, Version +# 2.0, and the BSD License. See the LICENSE file in the root of this repository +# for complete details. + +import logging +import platform +import struct +import subprocess +import sys +import sysconfig +from importlib.machinery import EXTENSION_SUFFIXES +from typing import ( + Dict, + FrozenSet, + Iterable, + Iterator, + List, + Optional, + Sequence, + Tuple, + Union, + cast, +) + +from . import _manylinux, _musllinux + +logger = logging.getLogger(__name__) + +PythonVersion = Sequence[int] +MacVersion = Tuple[int, int] + +INTERPRETER_SHORT_NAMES: Dict[str, str] = { + "python": "py", # Generic. + "cpython": "cp", + "pypy": "pp", + "ironpython": "ip", + "jython": "jy", +} + + +_32_BIT_INTERPRETER = struct.calcsize("P") == 4 + + +class Tag: + """ + A representation of the tag triple for a wheel. + + Instances are considered immutable and thus are hashable. Equality checking + is also supported. + """ + + __slots__ = ["_interpreter", "_abi", "_platform", "_hash"] + + def __init__(self, interpreter: str, abi: str, platform: str) -> None: + self._interpreter = interpreter.lower() + self._abi = abi.lower() + self._platform = platform.lower() + # The __hash__ of every single element in a Set[Tag] will be evaluated each time + # that a set calls its `.disjoint()` method, which may be called hundreds of + # times when scanning a page of links for packages with tags matching that + # Set[Tag]. Pre-computing the value here produces significant speedups for + # downstream consumers. + self._hash = hash((self._interpreter, self._abi, self._platform)) + + @property + def interpreter(self) -> str: + return self._interpreter + + @property + def abi(self) -> str: + return self._abi + + @property + def platform(self) -> str: + return self._platform + + def __eq__(self, other: object) -> bool: + if not isinstance(other, Tag): + return NotImplemented + + return ( + (self._hash == other._hash) # Short-circuit ASAP for perf reasons. + and (self._platform == other._platform) + and (self._abi == other._abi) + and (self._interpreter == other._interpreter) + ) + + def __hash__(self) -> int: + return self._hash + + def __str__(self) -> str: + return f"{self._interpreter}-{self._abi}-{self._platform}" + + def __repr__(self) -> str: + return f"<{self} @ {id(self)}>" + + +def parse_tag(tag: str) -> FrozenSet[Tag]: + """ + Parses the provided tag (e.g. `py3-none-any`) into a frozenset of Tag instances. + + Returning a set is required due to the possibility that the tag is a + compressed tag set. + """ + tags = set() + interpreters, abis, platforms = tag.split("-") + for interpreter in interpreters.split("."): + for abi in abis.split("."): + for platform_ in platforms.split("."): + tags.add(Tag(interpreter, abi, platform_)) + return frozenset(tags) + + +def _get_config_var(name: str, warn: bool = False) -> Union[int, str, None]: + value: Union[int, str, None] = sysconfig.get_config_var(name) + if value is None and warn: + logger.debug( + "Config variable '%s' is unset, Python ABI tag may be incorrect", name + ) + return value + + +def _normalize_string(string: str) -> str: + return string.replace(".", "_").replace("-", "_").replace(" ", "_") + + +def _abi3_applies(python_version: PythonVersion) -> bool: + """ + Determine if the Python version supports abi3. + + PEP 384 was first implemented in Python 3.2. + """ + return len(python_version) > 1 and tuple(python_version) >= (3, 2) + + +def _cpython_abis(py_version: PythonVersion, warn: bool = False) -> List[str]: + py_version = tuple(py_version) # To allow for version comparison. + abis = [] + version = _version_nodot(py_version[:2]) + debug = pymalloc = ucs4 = "" + with_debug = _get_config_var("Py_DEBUG", warn) + has_refcount = hasattr(sys, "gettotalrefcount") + # Windows doesn't set Py_DEBUG, so checking for support of debug-compiled + # extension modules is the best option. + # https://github.com/pypa/pip/issues/3383#issuecomment-173267692 + has_ext = "_d.pyd" in EXTENSION_SUFFIXES + if with_debug or (with_debug is None and (has_refcount or has_ext)): + debug = "d" + if py_version < (3, 8): + with_pymalloc = _get_config_var("WITH_PYMALLOC", warn) + if with_pymalloc or with_pymalloc is None: + pymalloc = "m" + if py_version < (3, 3): + unicode_size = _get_config_var("Py_UNICODE_SIZE", warn) + if unicode_size == 4 or ( + unicode_size is None and sys.maxunicode == 0x10FFFF + ): + ucs4 = "u" + elif debug: + # Debug builds can also load "normal" extension modules. + # We can also assume no UCS-4 or pymalloc requirement. + abis.append(f"cp{version}") + abis.insert( + 0, + "cp{version}{debug}{pymalloc}{ucs4}".format( + version=version, debug=debug, pymalloc=pymalloc, ucs4=ucs4 + ), + ) + return abis + + +def cpython_tags( + python_version: Optional[PythonVersion] = None, + abis: Optional[Iterable[str]] = None, + platforms: Optional[Iterable[str]] = None, + *, + warn: bool = False, +) -> Iterator[Tag]: + """ + Yields the tags for a CPython interpreter. + + The tags consist of: + - cp-- + - cp-abi3- + - cp-none- + - cp-abi3- # Older Python versions down to 3.2. + + If python_version only specifies a major version then user-provided ABIs and + the 'none' ABItag will be used. + + If 'abi3' or 'none' are specified in 'abis' then they will be yielded at + their normal position and not at the beginning. + """ + if not python_version: + python_version = sys.version_info[:2] + + interpreter = f"cp{_version_nodot(python_version[:2])}" + + if abis is None: + if len(python_version) > 1: + abis = _cpython_abis(python_version, warn) + else: + abis = [] + abis = list(abis) + # 'abi3' and 'none' are explicitly handled later. + for explicit_abi in ("abi3", "none"): + try: + abis.remove(explicit_abi) + except ValueError: + pass + + platforms = list(platforms or platform_tags()) + for abi in abis: + for platform_ in platforms: + yield Tag(interpreter, abi, platform_) + if _abi3_applies(python_version): + yield from (Tag(interpreter, "abi3", platform_) for platform_ in platforms) + yield from (Tag(interpreter, "none", platform_) for platform_ in platforms) + + if _abi3_applies(python_version): + for minor_version in range(python_version[1] - 1, 1, -1): + for platform_ in platforms: + interpreter = "cp{version}".format( + version=_version_nodot((python_version[0], minor_version)) + ) + yield Tag(interpreter, "abi3", platform_) + + +def _generic_abi() -> List[str]: + """ + Return the ABI tag based on EXT_SUFFIX. + """ + # The following are examples of `EXT_SUFFIX`. + # We want to keep the parts which are related to the ABI and remove the + # parts which are related to the platform: + # - linux: '.cpython-310-x86_64-linux-gnu.so' => cp310 + # - mac: '.cpython-310-darwin.so' => cp310 + # - win: '.cp310-win_amd64.pyd' => cp310 + # - win: '.pyd' => cp37 (uses _cpython_abis()) + # - pypy: '.pypy38-pp73-x86_64-linux-gnu.so' => pypy38_pp73 + # - graalpy: '.graalpy-38-native-x86_64-darwin.dylib' + # => graalpy_38_native + + ext_suffix = _get_config_var("EXT_SUFFIX", warn=True) + if not isinstance(ext_suffix, str) or ext_suffix[0] != ".": + raise SystemError("invalid sysconfig.get_config_var('EXT_SUFFIX')") + parts = ext_suffix.split(".") + if len(parts) < 3: + # CPython3.7 and earlier uses ".pyd" on Windows. + return _cpython_abis(sys.version_info[:2]) + soabi = parts[1] + if soabi.startswith("cpython"): + # non-windows + abi = "cp" + soabi.split("-")[1] + elif soabi.startswith("cp"): + # windows + abi = soabi.split("-")[0] + elif soabi.startswith("pypy"): + abi = "-".join(soabi.split("-")[:2]) + elif soabi.startswith("graalpy"): + abi = "-".join(soabi.split("-")[:3]) + elif soabi: + # pyston, ironpython, others? + abi = soabi + else: + return [] + return [_normalize_string(abi)] + + +def generic_tags( + interpreter: Optional[str] = None, + abis: Optional[Iterable[str]] = None, + platforms: Optional[Iterable[str]] = None, + *, + warn: bool = False, +) -> Iterator[Tag]: + """ + Yields the tags for a generic interpreter. + + The tags consist of: + - -- + + The "none" ABI will be added if it was not explicitly provided. + """ + if not interpreter: + interp_name = interpreter_name() + interp_version = interpreter_version(warn=warn) + interpreter = "".join([interp_name, interp_version]) + if abis is None: + abis = _generic_abi() + else: + abis = list(abis) + platforms = list(platforms or platform_tags()) + if "none" not in abis: + abis.append("none") + for abi in abis: + for platform_ in platforms: + yield Tag(interpreter, abi, platform_) + + +def _py_interpreter_range(py_version: PythonVersion) -> Iterator[str]: + """ + Yields Python versions in descending order. + + After the latest version, the major-only version will be yielded, and then + all previous versions of that major version. + """ + if len(py_version) > 1: + yield f"py{_version_nodot(py_version[:2])}" + yield f"py{py_version[0]}" + if len(py_version) > 1: + for minor in range(py_version[1] - 1, -1, -1): + yield f"py{_version_nodot((py_version[0], minor))}" + + +def compatible_tags( + python_version: Optional[PythonVersion] = None, + interpreter: Optional[str] = None, + platforms: Optional[Iterable[str]] = None, +) -> Iterator[Tag]: + """ + Yields the sequence of tags that are compatible with a specific version of Python. + + The tags consist of: + - py*-none- + - -none-any # ... if `interpreter` is provided. + - py*-none-any + """ + if not python_version: + python_version = sys.version_info[:2] + platforms = list(platforms or platform_tags()) + for version in _py_interpreter_range(python_version): + for platform_ in platforms: + yield Tag(version, "none", platform_) + if interpreter: + yield Tag(interpreter, "none", "any") + for version in _py_interpreter_range(python_version): + yield Tag(version, "none", "any") + + +def _mac_arch(arch: str, is_32bit: bool = _32_BIT_INTERPRETER) -> str: + if not is_32bit: + return arch + + if arch.startswith("ppc"): + return "ppc" + + return "i386" + + +def _mac_binary_formats(version: MacVersion, cpu_arch: str) -> List[str]: + formats = [cpu_arch] + if cpu_arch == "x86_64": + if version < (10, 4): + return [] + formats.extend(["intel", "fat64", "fat32"]) + + elif cpu_arch == "i386": + if version < (10, 4): + return [] + formats.extend(["intel", "fat32", "fat"]) + + elif cpu_arch == "ppc64": + # TODO: Need to care about 32-bit PPC for ppc64 through 10.2? + if version > (10, 5) or version < (10, 4): + return [] + formats.append("fat64") + + elif cpu_arch == "ppc": + if version > (10, 6): + return [] + formats.extend(["fat32", "fat"]) + + if cpu_arch in {"arm64", "x86_64"}: + formats.append("universal2") + + if cpu_arch in {"x86_64", "i386", "ppc64", "ppc", "intel"}: + formats.append("universal") + + return formats + + +def mac_platforms( + version: Optional[MacVersion] = None, arch: Optional[str] = None +) -> Iterator[str]: + """ + Yields the platform tags for a macOS system. + + The `version` parameter is a two-item tuple specifying the macOS version to + generate platform tags for. The `arch` parameter is the CPU architecture to + generate platform tags for. Both parameters default to the appropriate value + for the current system. + """ + version_str, _, cpu_arch = platform.mac_ver() + if version is None: + version = cast("MacVersion", tuple(map(int, version_str.split(".")[:2]))) + if version == (10, 16): + # When built against an older macOS SDK, Python will report macOS 10.16 + # instead of the real version. + version_str = subprocess.run( + [ + sys.executable, + "-sS", + "-c", + "import platform; print(platform.mac_ver()[0])", + ], + check=True, + env={"SYSTEM_VERSION_COMPAT": "0"}, + stdout=subprocess.PIPE, + text=True, + ).stdout + version = cast("MacVersion", tuple(map(int, version_str.split(".")[:2]))) + else: + version = version + if arch is None: + arch = _mac_arch(cpu_arch) + else: + arch = arch + + if (10, 0) <= version and version < (11, 0): + # Prior to Mac OS 11, each yearly release of Mac OS bumped the + # "minor" version number. The major version was always 10. + for minor_version in range(version[1], -1, -1): + compat_version = 10, minor_version + binary_formats = _mac_binary_formats(compat_version, arch) + for binary_format in binary_formats: + yield "macosx_{major}_{minor}_{binary_format}".format( + major=10, minor=minor_version, binary_format=binary_format + ) + + if version >= (11, 0): + # Starting with Mac OS 11, each yearly release bumps the major version + # number. The minor versions are now the midyear updates. + for major_version in range(version[0], 10, -1): + compat_version = major_version, 0 + binary_formats = _mac_binary_formats(compat_version, arch) + for binary_format in binary_formats: + yield "macosx_{major}_{minor}_{binary_format}".format( + major=major_version, minor=0, binary_format=binary_format + ) + + if version >= (11, 0): + # Mac OS 11 on x86_64 is compatible with binaries from previous releases. + # Arm64 support was introduced in 11.0, so no Arm binaries from previous + # releases exist. + # + # However, the "universal2" binary format can have a + # macOS version earlier than 11.0 when the x86_64 part of the binary supports + # that version of macOS. + if arch == "x86_64": + for minor_version in range(16, 3, -1): + compat_version = 10, minor_version + binary_formats = _mac_binary_formats(compat_version, arch) + for binary_format in binary_formats: + yield "macosx_{major}_{minor}_{binary_format}".format( + major=compat_version[0], + minor=compat_version[1], + binary_format=binary_format, + ) + else: + for minor_version in range(16, 3, -1): + compat_version = 10, minor_version + binary_format = "universal2" + yield "macosx_{major}_{minor}_{binary_format}".format( + major=compat_version[0], + minor=compat_version[1], + binary_format=binary_format, + ) + + +def _linux_platforms(is_32bit: bool = _32_BIT_INTERPRETER) -> Iterator[str]: + linux = _normalize_string(sysconfig.get_platform()) + if not linux.startswith("linux_"): + # we should never be here, just yield the sysconfig one and return + yield linux + return + if is_32bit: + if linux == "linux_x86_64": + linux = "linux_i686" + elif linux == "linux_aarch64": + linux = "linux_armv8l" + _, arch = linux.split("_", 1) + archs = {"armv8l": ["armv8l", "armv7l"]}.get(arch, [arch]) + yield from _manylinux.platform_tags(archs) + yield from _musllinux.platform_tags(archs) + for arch in archs: + yield f"linux_{arch}" + + +def _generic_platforms() -> Iterator[str]: + yield _normalize_string(sysconfig.get_platform()) + + +def platform_tags() -> Iterator[str]: + """ + Provides the platform tags for this installation. + """ + if platform.system() == "Darwin": + return mac_platforms() + elif platform.system() == "Linux": + return _linux_platforms() + else: + return _generic_platforms() + + +def interpreter_name() -> str: + """ + Returns the name of the running interpreter. + + Some implementations have a reserved, two-letter abbreviation which will + be returned when appropriate. + """ + name = sys.implementation.name + return INTERPRETER_SHORT_NAMES.get(name) or name + + +def interpreter_version(*, warn: bool = False) -> str: + """ + Returns the version of the running interpreter. + """ + version = _get_config_var("py_version_nodot", warn=warn) + if version: + version = str(version) + else: + version = _version_nodot(sys.version_info[:2]) + return version + + +def _version_nodot(version: PythonVersion) -> str: + return "".join(map(str, version)) + + +def sys_tags(*, warn: bool = False) -> Iterator[Tag]: + """ + Returns the sequence of tag triples for the running interpreter. + + The order of the sequence corresponds to priority order for the + interpreter, from most to least important. + """ + + interp_name = interpreter_name() + if interp_name == "cp": + yield from cpython_tags(warn=warn) + else: + yield from generic_tags() + + if interp_name == "pp": + interp = "pp3" + elif interp_name == "cp": + interp = "cp" + interpreter_version(warn=warn) + else: + interp = None + yield from compatible_tags(interpreter=interp) diff --git a/gyp/pylib/packaging/utils.py b/gyp/pylib/packaging/utils.py new file mode 100644 index 0000000000..c2c2f75aa8 --- /dev/null +++ b/gyp/pylib/packaging/utils.py @@ -0,0 +1,172 @@ +# This file is dual licensed under the terms of the Apache License, Version +# 2.0, and the BSD License. See the LICENSE file in the root of this repository +# for complete details. + +import re +from typing import FrozenSet, NewType, Tuple, Union, cast + +from .tags import Tag, parse_tag +from .version import InvalidVersion, Version + +BuildTag = Union[Tuple[()], Tuple[int, str]] +NormalizedName = NewType("NormalizedName", str) + + +class InvalidName(ValueError): + """ + An invalid distribution name; users should refer to the packaging user guide. + """ + + +class InvalidWheelFilename(ValueError): + """ + An invalid wheel filename was found, users should refer to PEP 427. + """ + + +class InvalidSdistFilename(ValueError): + """ + An invalid sdist filename was found, users should refer to the packaging user guide. + """ + + +# Core metadata spec for `Name` +_validate_regex = re.compile( + r"^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$", re.IGNORECASE +) +_canonicalize_regex = re.compile(r"[-_.]+") +_normalized_regex = re.compile(r"^([a-z0-9]|[a-z0-9]([a-z0-9-](?!--))*[a-z0-9])$") +# PEP 427: The build number must start with a digit. +_build_tag_regex = re.compile(r"(\d+)(.*)") + + +def canonicalize_name(name: str, *, validate: bool = False) -> NormalizedName: + if validate and not _validate_regex.match(name): + raise InvalidName(f"name is invalid: {name!r}") + # This is taken from PEP 503. + value = _canonicalize_regex.sub("-", name).lower() + return cast(NormalizedName, value) + + +def is_normalized_name(name: str) -> bool: + return _normalized_regex.match(name) is not None + + +def canonicalize_version( + version: Union[Version, str], *, strip_trailing_zero: bool = True +) -> str: + """ + This is very similar to Version.__str__, but has one subtle difference + with the way it handles the release segment. + """ + if isinstance(version, str): + try: + parsed = Version(version) + except InvalidVersion: + # Legacy versions cannot be normalized + return version + else: + parsed = version + + parts = [] + + # Epoch + if parsed.epoch != 0: + parts.append(f"{parsed.epoch}!") + + # Release segment + release_segment = ".".join(str(x) for x in parsed.release) + if strip_trailing_zero: + # NB: This strips trailing '.0's to normalize + release_segment = re.sub(r"(\.0)+$", "", release_segment) + parts.append(release_segment) + + # Pre-release + if parsed.pre is not None: + parts.append("".join(str(x) for x in parsed.pre)) + + # Post-release + if parsed.post is not None: + parts.append(f".post{parsed.post}") + + # Development release + if parsed.dev is not None: + parts.append(f".dev{parsed.dev}") + + # Local version segment + if parsed.local is not None: + parts.append(f"+{parsed.local}") + + return "".join(parts) + + +def parse_wheel_filename( + filename: str, +) -> Tuple[NormalizedName, Version, BuildTag, FrozenSet[Tag]]: + if not filename.endswith(".whl"): + raise InvalidWheelFilename( + f"Invalid wheel filename (extension must be '.whl'): {filename}" + ) + + filename = filename[:-4] + dashes = filename.count("-") + if dashes not in (4, 5): + raise InvalidWheelFilename( + f"Invalid wheel filename (wrong number of parts): {filename}" + ) + + parts = filename.split("-", dashes - 2) + name_part = parts[0] + # See PEP 427 for the rules on escaping the project name. + if "__" in name_part or re.match(r"^[\w\d._]*$", name_part, re.UNICODE) is None: + raise InvalidWheelFilename(f"Invalid project name: {filename}") + name = canonicalize_name(name_part) + + try: + version = Version(parts[1]) + except InvalidVersion as e: + raise InvalidWheelFilename( + f"Invalid wheel filename (invalid version): {filename}" + ) from e + + if dashes == 5: + build_part = parts[2] + build_match = _build_tag_regex.match(build_part) + if build_match is None: + raise InvalidWheelFilename( + f"Invalid build number: {build_part} in '{filename}'" + ) + build = cast(BuildTag, (int(build_match.group(1)), build_match.group(2))) + else: + build = () + tags = parse_tag(parts[-1]) + return (name, version, build, tags) + + +def parse_sdist_filename(filename: str) -> Tuple[NormalizedName, Version]: + if filename.endswith(".tar.gz"): + file_stem = filename[: -len(".tar.gz")] + elif filename.endswith(".zip"): + file_stem = filename[: -len(".zip")] + else: + raise InvalidSdistFilename( + f"Invalid sdist filename (extension must be '.tar.gz' or '.zip'):" + f" {filename}" + ) + + # We are requiring a PEP 440 version, which cannot contain dashes, + # so we split on the last dash. + name_part, sep, version_part = file_stem.rpartition("-") + if not sep: + raise InvalidSdistFilename(f"Invalid sdist filename: {filename}") + + name = canonicalize_name(name_part) + + try: + version = Version(version_part) + except InvalidVersion as e: + raise InvalidSdistFilename( + f"Invalid sdist filename (invalid version): {filename}" + ) from e + + return (name, version) diff --git a/gyp/pylib/packaging/version.py b/gyp/pylib/packaging/version.py new file mode 100644 index 0000000000..5faab9bd0d --- /dev/null +++ b/gyp/pylib/packaging/version.py @@ -0,0 +1,563 @@ +# This file is dual licensed under the terms of the Apache License, Version +# 2.0, and the BSD License. See the LICENSE file in the root of this repository +# for complete details. +""" +.. testsetup:: + + from packaging.version import parse, Version +""" + +import itertools +import re +from typing import Any, Callable, NamedTuple, Optional, SupportsInt, Tuple, Union + +from ._structures import Infinity, InfinityType, NegativeInfinity, NegativeInfinityType + +__all__ = ["VERSION_PATTERN", "parse", "Version", "InvalidVersion"] + +LocalType = Tuple[Union[int, str], ...] + +CmpPrePostDevType = Union[InfinityType, NegativeInfinityType, Tuple[str, int]] +CmpLocalType = Union[ + NegativeInfinityType, + Tuple[Union[Tuple[int, str], Tuple[NegativeInfinityType, Union[int, str]]], ...], +] +CmpKey = Tuple[ + int, + Tuple[int, ...], + CmpPrePostDevType, + CmpPrePostDevType, + CmpPrePostDevType, + CmpLocalType, +] +VersionComparisonMethod = Callable[[CmpKey, CmpKey], bool] + + +class _Version(NamedTuple): + epoch: int + release: Tuple[int, ...] + dev: Optional[Tuple[str, int]] + pre: Optional[Tuple[str, int]] + post: Optional[Tuple[str, int]] + local: Optional[LocalType] + + +def parse(version: str) -> "Version": + """Parse the given version string. + + >>> parse('1.0.dev1') + + + :param version: The version string to parse. + :raises InvalidVersion: When the version string is not a valid version. + """ + return Version(version) + + +class InvalidVersion(ValueError): + """Raised when a version string is not a valid version. + + >>> Version("invalid") + Traceback (most recent call last): + ... + packaging.version.InvalidVersion: Invalid version: 'invalid' + """ + + +class _BaseVersion: + _key: Tuple[Any, ...] + + def __hash__(self) -> int: + return hash(self._key) + + # Please keep the duplicated `isinstance` check + # in the six comparisons hereunder + # unless you find a way to avoid adding overhead function calls. + def __lt__(self, other: "_BaseVersion") -> bool: + if not isinstance(other, _BaseVersion): + return NotImplemented + + return self._key < other._key + + def __le__(self, other: "_BaseVersion") -> bool: + if not isinstance(other, _BaseVersion): + return NotImplemented + + return self._key <= other._key + + def __eq__(self, other: object) -> bool: + if not isinstance(other, _BaseVersion): + return NotImplemented + + return self._key == other._key + + def __ge__(self, other: "_BaseVersion") -> bool: + if not isinstance(other, _BaseVersion): + return NotImplemented + + return self._key >= other._key + + def __gt__(self, other: "_BaseVersion") -> bool: + if not isinstance(other, _BaseVersion): + return NotImplemented + + return self._key > other._key + + def __ne__(self, other: object) -> bool: + if not isinstance(other, _BaseVersion): + return NotImplemented + + return self._key != other._key + + +# Deliberately not anchored to the start and end of the string, to make it +# easier for 3rd party code to reuse +_VERSION_PATTERN = r""" + v? + (?: + (?:(?P[0-9]+)!)? # epoch + (?P[0-9]+(?:\.[0-9]+)*) # release segment + (?P
                                          # pre-release
+            [-_\.]?
+            (?Palpha|a|beta|b|preview|pre|c|rc)
+            [-_\.]?
+            (?P[0-9]+)?
+        )?
+        (?P                                         # post release
+            (?:-(?P[0-9]+))
+            |
+            (?:
+                [-_\.]?
+                (?Ppost|rev|r)
+                [-_\.]?
+                (?P[0-9]+)?
+            )
+        )?
+        (?P                                          # dev release
+            [-_\.]?
+            (?Pdev)
+            [-_\.]?
+            (?P[0-9]+)?
+        )?
+    )
+    (?:\+(?P[a-z0-9]+(?:[-_\.][a-z0-9]+)*))?       # local version
+"""
+
+VERSION_PATTERN = _VERSION_PATTERN
+"""
+A string containing the regular expression used to match a valid version.
+
+The pattern is not anchored at either end, and is intended for embedding in larger
+expressions (for example, matching a version number as part of a file name). The
+regular expression should be compiled with the ``re.VERBOSE`` and ``re.IGNORECASE``
+flags set.
+
+:meta hide-value:
+"""
+
+
+class Version(_BaseVersion):
+    """This class abstracts handling of a project's versions.
+
+    A :class:`Version` instance is comparison aware and can be compared and
+    sorted using the standard Python interfaces.
+
+    >>> v1 = Version("1.0a5")
+    >>> v2 = Version("1.0")
+    >>> v1
+    
+    >>> v2
+    
+    >>> v1 < v2
+    True
+    >>> v1 == v2
+    False
+    >>> v1 > v2
+    False
+    >>> v1 >= v2
+    False
+    >>> v1 <= v2
+    True
+    """
+
+    _regex = re.compile(r"^\s*" + VERSION_PATTERN + r"\s*$", re.VERBOSE | re.IGNORECASE)
+    _key: CmpKey
+
+    def __init__(self, version: str) -> None:
+        """Initialize a Version object.
+
+        :param version:
+            The string representation of a version which will be parsed and normalized
+            before use.
+        :raises InvalidVersion:
+            If the ``version`` does not conform to PEP 440 in any way then this
+            exception will be raised.
+        """
+
+        # Validate the version and parse it into pieces
+        match = self._regex.search(version)
+        if not match:
+            raise InvalidVersion(f"Invalid version: '{version}'")
+
+        # Store the parsed out pieces of the version
+        self._version = _Version(
+            epoch=int(match.group("epoch")) if match.group("epoch") else 0,
+            release=tuple(int(i) for i in match.group("release").split(".")),
+            pre=_parse_letter_version(match.group("pre_l"), match.group("pre_n")),
+            post=_parse_letter_version(
+                match.group("post_l"), match.group("post_n1") or match.group("post_n2")
+            ),
+            dev=_parse_letter_version(match.group("dev_l"), match.group("dev_n")),
+            local=_parse_local_version(match.group("local")),
+        )
+
+        # Generate a key which will be used for sorting
+        self._key = _cmpkey(
+            self._version.epoch,
+            self._version.release,
+            self._version.pre,
+            self._version.post,
+            self._version.dev,
+            self._version.local,
+        )
+
+    def __repr__(self) -> str:
+        """A representation of the Version that shows all internal state.
+
+        >>> Version('1.0.0')
+        
+        """
+        return f""
+
+    def __str__(self) -> str:
+        """A string representation of the version that can be rounded-tripped.
+
+        >>> str(Version("1.0a5"))
+        '1.0a5'
+        """
+        parts = []
+
+        # Epoch
+        if self.epoch != 0:
+            parts.append(f"{self.epoch}!")
+
+        # Release segment
+        parts.append(".".join(str(x) for x in self.release))
+
+        # Pre-release
+        if self.pre is not None:
+            parts.append("".join(str(x) for x in self.pre))
+
+        # Post-release
+        if self.post is not None:
+            parts.append(f".post{self.post}")
+
+        # Development release
+        if self.dev is not None:
+            parts.append(f".dev{self.dev}")
+
+        # Local version segment
+        if self.local is not None:
+            parts.append(f"+{self.local}")
+
+        return "".join(parts)
+
+    @property
+    def epoch(self) -> int:
+        """The epoch of the version.
+
+        >>> Version("2.0.0").epoch
+        0
+        >>> Version("1!2.0.0").epoch
+        1
+        """
+        return self._version.epoch
+
+    @property
+    def release(self) -> Tuple[int, ...]:
+        """The components of the "release" segment of the version.
+
+        >>> Version("1.2.3").release
+        (1, 2, 3)
+        >>> Version("2.0.0").release
+        (2, 0, 0)
+        >>> Version("1!2.0.0.post0").release
+        (2, 0, 0)
+
+        Includes trailing zeroes but not the epoch or any pre-release / development /
+        post-release suffixes.
+        """
+        return self._version.release
+
+    @property
+    def pre(self) -> Optional[Tuple[str, int]]:
+        """The pre-release segment of the version.
+
+        >>> print(Version("1.2.3").pre)
+        None
+        >>> Version("1.2.3a1").pre
+        ('a', 1)
+        >>> Version("1.2.3b1").pre
+        ('b', 1)
+        >>> Version("1.2.3rc1").pre
+        ('rc', 1)
+        """
+        return self._version.pre
+
+    @property
+    def post(self) -> Optional[int]:
+        """The post-release number of the version.
+
+        >>> print(Version("1.2.3").post)
+        None
+        >>> Version("1.2.3.post1").post
+        1
+        """
+        return self._version.post[1] if self._version.post else None
+
+    @property
+    def dev(self) -> Optional[int]:
+        """The development number of the version.
+
+        >>> print(Version("1.2.3").dev)
+        None
+        >>> Version("1.2.3.dev1").dev
+        1
+        """
+        return self._version.dev[1] if self._version.dev else None
+
+    @property
+    def local(self) -> Optional[str]:
+        """The local version segment of the version.
+
+        >>> print(Version("1.2.3").local)
+        None
+        >>> Version("1.2.3+abc").local
+        'abc'
+        """
+        if self._version.local:
+            return ".".join(str(x) for x in self._version.local)
+        else:
+            return None
+
+    @property
+    def public(self) -> str:
+        """The public portion of the version.
+
+        >>> Version("1.2.3").public
+        '1.2.3'
+        >>> Version("1.2.3+abc").public
+        '1.2.3'
+        >>> Version("1.2.3+abc.dev1").public
+        '1.2.3'
+        """
+        return str(self).split("+", 1)[0]
+
+    @property
+    def base_version(self) -> str:
+        """The "base version" of the version.
+
+        >>> Version("1.2.3").base_version
+        '1.2.3'
+        >>> Version("1.2.3+abc").base_version
+        '1.2.3'
+        >>> Version("1!1.2.3+abc.dev1").base_version
+        '1!1.2.3'
+
+        The "base version" is the public version of the project without any pre or post
+        release markers.
+        """
+        parts = []
+
+        # Epoch
+        if self.epoch != 0:
+            parts.append(f"{self.epoch}!")
+
+        # Release segment
+        parts.append(".".join(str(x) for x in self.release))
+
+        return "".join(parts)
+
+    @property
+    def is_prerelease(self) -> bool:
+        """Whether this version is a pre-release.
+
+        >>> Version("1.2.3").is_prerelease
+        False
+        >>> Version("1.2.3a1").is_prerelease
+        True
+        >>> Version("1.2.3b1").is_prerelease
+        True
+        >>> Version("1.2.3rc1").is_prerelease
+        True
+        >>> Version("1.2.3dev1").is_prerelease
+        True
+        """
+        return self.dev is not None or self.pre is not None
+
+    @property
+    def is_postrelease(self) -> bool:
+        """Whether this version is a post-release.
+
+        >>> Version("1.2.3").is_postrelease
+        False
+        >>> Version("1.2.3.post1").is_postrelease
+        True
+        """
+        return self.post is not None
+
+    @property
+    def is_devrelease(self) -> bool:
+        """Whether this version is a development release.
+
+        >>> Version("1.2.3").is_devrelease
+        False
+        >>> Version("1.2.3.dev1").is_devrelease
+        True
+        """
+        return self.dev is not None
+
+    @property
+    def major(self) -> int:
+        """The first item of :attr:`release` or ``0`` if unavailable.
+
+        >>> Version("1.2.3").major
+        1
+        """
+        return self.release[0] if len(self.release) >= 1 else 0
+
+    @property
+    def minor(self) -> int:
+        """The second item of :attr:`release` or ``0`` if unavailable.
+
+        >>> Version("1.2.3").minor
+        2
+        >>> Version("1").minor
+        0
+        """
+        return self.release[1] if len(self.release) >= 2 else 0
+
+    @property
+    def micro(self) -> int:
+        """The third item of :attr:`release` or ``0`` if unavailable.
+
+        >>> Version("1.2.3").micro
+        3
+        >>> Version("1").micro
+        0
+        """
+        return self.release[2] if len(self.release) >= 3 else 0
+
+
+def _parse_letter_version(
+    letter: Optional[str], number: Union[str, bytes, SupportsInt, None]
+) -> Optional[Tuple[str, int]]:
+
+    if letter:
+        # We consider there to be an implicit 0 in a pre-release if there is
+        # not a numeral associated with it.
+        if number is None:
+            number = 0
+
+        # We normalize any letters to their lower case form
+        letter = letter.lower()
+
+        # We consider some words to be alternate spellings of other words and
+        # in those cases we want to normalize the spellings to our preferred
+        # spelling.
+        if letter == "alpha":
+            letter = "a"
+        elif letter == "beta":
+            letter = "b"
+        elif letter in ["c", "pre", "preview"]:
+            letter = "rc"
+        elif letter in ["rev", "r"]:
+            letter = "post"
+
+        return letter, int(number)
+    if not letter and number:
+        # We assume if we are given a number, but we are not given a letter
+        # then this is using the implicit post release syntax (e.g. 1.0-1)
+        letter = "post"
+
+        return letter, int(number)
+
+    return None
+
+
+_local_version_separators = re.compile(r"[\._-]")
+
+
+def _parse_local_version(local: Optional[str]) -> Optional[LocalType]:
+    """
+    Takes a string like abc.1.twelve and turns it into ("abc", 1, "twelve").
+    """
+    if local is not None:
+        return tuple(
+            part.lower() if not part.isdigit() else int(part)
+            for part in _local_version_separators.split(local)
+        )
+    return None
+
+
+def _cmpkey(
+    epoch: int,
+    release: Tuple[int, ...],
+    pre: Optional[Tuple[str, int]],
+    post: Optional[Tuple[str, int]],
+    dev: Optional[Tuple[str, int]],
+    local: Optional[LocalType],
+) -> CmpKey:
+
+    # When we compare a release version, we want to compare it with all of the
+    # trailing zeros removed. So we'll use a reverse the list, drop all the now
+    # leading zeros until we come to something non zero, then take the rest
+    # re-reverse it back into the correct order and make it a tuple and use
+    # that for our sorting key.
+    _release = tuple(
+        reversed(list(itertools.dropwhile(lambda x: x == 0, reversed(release))))
+    )
+
+    # We need to "trick" the sorting algorithm to put 1.0.dev0 before 1.0a0.
+    # We'll do this by abusing the pre segment, but we _only_ want to do this
+    # if there is not a pre or a post segment. If we have one of those then
+    # the normal sorting rules will handle this case correctly.
+    if pre is None and post is None and dev is not None:
+        _pre: CmpPrePostDevType = NegativeInfinity
+    # Versions without a pre-release (except as noted above) should sort after
+    # those with one.
+    elif pre is None:
+        _pre = Infinity
+    else:
+        _pre = pre
+
+    # Versions without a post segment should sort before those with one.
+    if post is None:
+        _post: CmpPrePostDevType = NegativeInfinity
+
+    else:
+        _post = post
+
+    # Versions without a development segment should sort after those with one.
+    if dev is None:
+        _dev: CmpPrePostDevType = Infinity
+
+    else:
+        _dev = dev
+
+    if local is None:
+        # Versions without a local segment should sort before those with one.
+        _local: CmpLocalType = NegativeInfinity
+    else:
+        # Versions with a local segment need that segment parsed to implement
+        # the sorting rules in PEP440.
+        # - Alpha numeric segments sort before numeric segments
+        # - Alpha numeric segments sort lexicographically
+        # - Numeric segments sort numerically
+        # - Shorter versions sort before longer versions when the prefixes
+        #   match exactly
+        _local = tuple(
+            (i, "") if isinstance(i, int) else (NegativeInfinity, i) for i in local
+        )
+
+    return epoch, _release, _pre, _post, _dev, _local
diff --git a/gyp/pyproject.toml b/gyp/pyproject.toml
index d8a5451520..0c25d0b3c1 100644
--- a/gyp/pyproject.toml
+++ b/gyp/pyproject.toml
@@ -4,14 +4,16 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "gyp-next"
-version = "0.14.0"
+version = "0.16.1"
 authors = [
   { name="Node.js contributors", email="ryzokuken@disroot.org" },
 ]
 description = "A fork of the GYP build system for use in the Node.js projects"
 readme = "README.md"
 license = { file="LICENSE" }
-requires-python = ">=3.6"
+requires-python = ">=3.8"
+# The Python module "packaging" is vendored in the "pylib/packaging" directory to support Python >= 3.12.
+# dependencies = ["packaging>=23.1"]  # Uncomment this line if the vendored version is removed.
 classifiers = [
     "Development Status :: 3 - Alpha",
     "Environment :: Console",
@@ -20,15 +22,14 @@ classifiers = [
     "Natural Language :: English",
     "Programming Language :: Python",
     "Programming Language :: Python :: 3",
-    "Programming Language :: Python :: 3.6",
-    "Programming Language :: Python :: 3.7",
     "Programming Language :: Python :: 3.8",
     "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
 ]
 
 [project.optional-dependencies]
-dev = ["flake8", "pytest"]
+dev = ["flake8", "ruff", "pytest"]
 
 [project.scripts]
 gyp = "gyp:script_main"
@@ -36,6 +37,83 @@ gyp = "gyp:script_main"
 [project.urls]
 "Homepage" = "https://github.com/nodejs/gyp-next"
 
+[tool.ruff]
+select = [
+  "C4",   # flake8-comprehensions
+  "C90",  # McCabe cyclomatic complexity
+  "DTZ",  # flake8-datetimez
+  "E",    # pycodestyle
+  "F",    # Pyflakes
+  "G",    # flake8-logging-format
+  "ICN",  # flake8-import-conventions
+  "INT",  # flake8-gettext
+  "PL",   # Pylint
+  "PYI",  # flake8-pyi
+  "RSE",  # flake8-raise
+  "RUF",  # Ruff-specific rules
+  "T10",  # flake8-debugger
+  "TCH",  # flake8-type-checking
+  "TID",  # flake8-tidy-imports
+  "UP",   # pyupgrade
+  "W",    # pycodestyle
+  "YTT",  # flake8-2020
+  # "A",    # flake8-builtins
+  # "ANN",  # flake8-annotations
+  # "ARG",  # flake8-unused-arguments
+  # "B",    # flake8-bugbear
+  # "BLE",  # flake8-blind-except
+  # "COM",  # flake8-commas
+  # "D",    # pydocstyle
+  # "DJ",   # flake8-django
+  # "EM",   # flake8-errmsg
+  # "ERA",  # eradicate
+  # "EXE",  # flake8-executable
+  # "FBT",  # flake8-boolean-trap
+  # "I",    # isort
+  # "INP",  # flake8-no-pep420
+  # "ISC",  # flake8-implicit-str-concat
+  # "N",    # pep8-naming
+  # "NPY",  # NumPy-specific rules
+  # "PD",   # pandas-vet
+  # "PGH",  # pygrep-hooks
+  # "PIE",  # flake8-pie
+  # "PT",   # flake8-pytest-style
+  # "PTH",  # flake8-use-pathlib
+  # "Q",    # flake8-quotes
+  # "RET",  # flake8-return
+  # "S",    # flake8-bandit
+  # "SIM",  # flake8-simplify
+  # "SLF",  # flake8-self
+  # "T20",  # flake8-print
+  # "TRY",  # tryceratops
+]
+ignore = [
+  "E721",
+  "PLC1901",
+  "PLR0402",
+  "PLR1714",
+  "PLR2004",
+  "PLR5501",
+  "PLW0603",
+  "PLW2901",
+  "PYI024",
+  "RUF005",
+  "RUF012",
+  "UP031",
+]
+extend-exclude = ["pylib/packaging"]
+line-length = 88
+target-version = "py37"
+
+[tool.ruff.mccabe]
+max-complexity = 101
+
+[tool.ruff.pylint]
+max-args = 11
+max-branches = 108
+max-returns = 10
+max-statements = 286
+
 [tool.setuptools]
 package-dir = {"" = "pylib"}
 packages = ["gyp", "gyp.generator"]
diff --git a/gyp/tools/pretty_sln.py b/gyp/tools/pretty_sln.py
index 6ca0cd12a7..cf0638a23d 100755
--- a/gyp/tools/pretty_sln.py
+++ b/gyp/tools/pretty_sln.py
@@ -34,10 +34,10 @@ def BuildProject(project, built, projects, deps):
 
 def ParseSolution(solution_file):
     # All projects, their clsid and paths.
-    projects = dict()
+    projects = {}
 
     # A list of dependencies associated with a project.
-    dependencies = dict()
+    dependencies = {}
 
     # Regular expressions that matches the SLN format.
     # The first line of a project definition.
diff --git a/gyp/tools/pretty_vcproj.py b/gyp/tools/pretty_vcproj.py
index 00d32debda..72c65d7fc4 100755
--- a/gyp/tools/pretty_vcproj.py
+++ b/gyp/tools/pretty_vcproj.py
@@ -21,7 +21,7 @@
 
 __author__ = "nsylvain (Nicolas Sylvain)"
 ARGUMENTS = None
-REPLACEMENTS = dict()
+REPLACEMENTS = {}
 
 
 def cmp(x, y):

From 21a7249b40d8f95e7721e450fd18764adb1648a7 Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Fri, 27 Oct 2023 20:51:20 -0700
Subject: [PATCH 390/551] chore: add check engines script to CI (#2922)

---
 .github/scripts/check-engines.js | 41 ++++++++++++++++++++++++++++++++
 .github/workflows/tests.yml      | 17 +++++++++++++
 package.json                     |  2 +-
 3 files changed, 59 insertions(+), 1 deletion(-)
 create mode 100644 .github/scripts/check-engines.js

diff --git a/.github/scripts/check-engines.js b/.github/scripts/check-engines.js
new file mode 100644
index 0000000000..80da7a03ed
--- /dev/null
+++ b/.github/scripts/check-engines.js
@@ -0,0 +1,41 @@
+const { join } = require('path')
+const semver = require('semver')
+const Arborist = require('@npmcli/arborist')
+
+const run = async (path, useEngines) => {
+  const pkgPath = join(path, 'package.json')
+  const pkg = require(pkgPath)
+
+  const engines = useEngines || pkg.engines.node
+
+  const arb = new Arborist({ path })
+  const tree = await arb.loadActual({ forceActual: true })
+  const deps = await tree.querySelectorAll(`#${pkg.name} > .prod:attr(engines, [node])`)
+
+  const invalid = []
+  for (const dep of deps) {
+    const depEngines = dep.target.package.engines.node
+    if (!semver.subset(engines, depEngines)) {
+      invalid.push({
+        name: `${dep.name}@${dep.version}`,
+        location: dep.location,
+        engines: depEngines
+      })
+    }
+  }
+
+  if (invalid.length) {
+    const msg = 'The following production dependencies are not compatible with ' +
+`\`engines.node: ${engines}\` found in \`${pkgPath}\`:\n` + invalid.map((dep) => [
+  `${dep.name}:`,
+  `  engines.node: ${dep.engines}`,
+  `  location: ${dep.location}`
+    ].join('\n')).join('\n')
+    throw new Error(msg)
+  }
+}
+
+run(process.cwd(), ...process.argv.slice(2)).then(() => console.log('Success')).catch((err) => {
+  console.error(err)
+  process.exitCode = 1
+})
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 310b4f06ea..29525b8a73 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -18,6 +18,23 @@ jobs:
     - uses: actions/checkout@v4
     - run: pip install --user ruff
     - run: ruff --output-format=github --select="E,F,PLC,PLE,UP,W,YTT" --ignore="E721,PLC1901,S101,UP031" --target-version=py38 .
+  Engines:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout Repository
+      uses: actions/checkout@v3
+    - name: Use Node.js 20.x
+      uses: actions/setup-node@v3
+      with:
+        node-version: 20.x
+    - name: Install Dependencies
+      run: |
+        npm install --no-progress
+    - name: Check Engines
+      run: |
+        # TODO: move this to its own action
+        npm install @npmcli/arborist@7 semver@7 --no-save
+        node .github/scripts/check-engines.js
   Tests:
     needs: Lint_Python  # Lint_Python takes ~5 seconds, so wait for it to pass before running the full matrix of tests.
     strategy:
diff --git a/package.json b/package.json
index d861261002..a8fd8a6fff 100644
--- a/package.json
+++ b/package.json
@@ -44,7 +44,7 @@
     "standard": "^17.0.0"
   },
   "scripts": {
-    "lint": "standard */*.js test/**/*.js",
+    "lint": "standard \"*/*.js\" \"test/**/*.js\" \".github/**/*.js\"",
     "test": "npm run lint && mocha --timeout 15000 --reporter=test/reporter.js test/test-download.js test/test-*"
   }
 }

From 4bef1ecc7554097d92beb397fbe1a546c5227545 Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Fri, 27 Oct 2023 21:12:39 -0700
Subject: [PATCH 391/551] deps: glob@10.3.10 (#2926)

---
 lib/build.js | 3 +--
 package.json | 2 +-
 2 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/lib/build.js b/lib/build.js
index 0bcc9bea19..c0cb9f3904 100644
--- a/lib/build.js
+++ b/lib/build.js
@@ -1,9 +1,8 @@
 'use strict'
 
 const fs = require('graceful-fs').promises
-const { promisify } = require('util')
 const path = require('path')
-const glob = promisify(require('glob'))
+const { glob } = require('glob')
 const log = require('./log')
 const which = require('which')
 const win = process.platform === 'win32'
diff --git a/package.json b/package.json
index a8fd8a6fff..99415b8790 100644
--- a/package.json
+++ b/package.json
@@ -24,7 +24,7 @@
   "dependencies": {
     "env-paths": "^2.2.0",
     "exponential-backoff": "^3.1.1",
-    "glob": "^8.0.3",
+    "glob": "^10.3.10",
     "graceful-fs": "^4.2.6",
     "make-fetch-happen": "^11.0.3",
     "nopt": "^7.0.0",

From 059bb6fd41bb50955a9efbd97887773d60d53221 Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Fri, 27 Oct 2023 21:12:49 -0700
Subject: [PATCH 392/551] deps: make-fetch-happen@13.0.0 (#2927)

---
 package.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package.json b/package.json
index 99415b8790..644ae7334c 100644
--- a/package.json
+++ b/package.json
@@ -26,7 +26,7 @@
     "exponential-backoff": "^3.1.1",
     "glob": "^10.3.10",
     "graceful-fs": "^4.2.6",
-    "make-fetch-happen": "^11.0.3",
+    "make-fetch-happen": "^13.0.0",
     "nopt": "^7.0.0",
     "proc-log": "^3.0.0",
     "semver": "^7.3.5",

From e38825531403aabeae7abe58e76867f31b832f36 Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Fri, 27 Oct 2023 21:12:59 -0700
Subject: [PATCH 393/551] deps: which@4.0.0 (#2928)

---
 package.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package.json b/package.json
index 644ae7334c..c2345c146f 100644
--- a/package.json
+++ b/package.json
@@ -31,7 +31,7 @@
     "proc-log": "^3.0.0",
     "semver": "^7.3.5",
     "tar": "^6.1.2",
-    "which": "^3.0.0"
+    "which": "^4.0.0"
   },
   "engines": {
     "node": "^14.17.0 || ^16.13.0 || >=18.0.0"

From 1b3bd341b40f384988d03207ce8187e93ba609bc Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Fri, 27 Oct 2023 21:18:49 -0700
Subject: [PATCH 394/551] feat!: drop node 14 support (#2929)

BREAKING CHANGE: `node-gyp` now supports node `^16.14.0 || >=18.0.0`
---
 package.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package.json b/package.json
index c2345c146f..409e228001 100644
--- a/package.json
+++ b/package.json
@@ -34,7 +34,7 @@
     "which": "^4.0.0"
   },
   "engines": {
-    "node": "^14.17.0 || ^16.13.0 || >=18.0.0"
+    "node": "^16.14.0 || >=18.0.0"
   },
   "devDependencies": {
     "bindings": "^1.5.0",

From 355622f4aac3bd3056b9e03aac5fa2f42a4b3576 Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Fri, 27 Oct 2023 15:52:13 -0700
Subject: [PATCH 395/551] feat: convert all internal functions to async/await

BREAKING CHANGE: All internal functions have been coverted to return
promises and no longer accept callbacks. This is not a breaking change
for users but may be breaking to consumers of `node-gyp` if you are
requiring internal functions directly.
---
 bin/node-gyp.js                |  34 +-
 lib/build.js                   |  12 +-
 lib/clean.js                   |   6 +-
 lib/configure.js               | 241 ++++++-------
 lib/create-config-gypi.js      |   6 +-
 lib/find-python.js             | 181 ++++------
 lib/find-visualstudio.js       | 135 ++++----
 lib/install.js                 |  58 ++--
 lib/list.js                    |  25 +-
 lib/node-gyp.js                |  15 +-
 lib/rebuild.js                 |   3 +-
 lib/remove.js                  |   6 +-
 lib/util.js                    |  65 ++--
 test/process-exec-sync.js      | 140 --------
 test/test-addon.js             |   4 +-
 test/test-configure-python.js  |  11 +-
 test/test-download.js          |   3 +-
 test/test-find-python.js       | 171 ++++-----
 test/test-find-visualstudio.js | 617 ++++++++++++++++-----------------
 test/test-install.js           |  95 ++---
 20 files changed, 767 insertions(+), 1061 deletions(-)
 delete mode 100644 test/process-exec-sync.js

diff --git a/bin/node-gyp.js b/bin/node-gyp.js
index 3441973da4..a6f3e73a50 100755
--- a/bin/node-gyp.js
+++ b/bin/node-gyp.js
@@ -68,7 +68,7 @@ if (dir) {
   }
 }
 
-function run () {
+async function run () {
   const command = prog.todo.shift()
   if (!command) {
     // done!
@@ -77,30 +77,28 @@ function run () {
     return
   }
 
-  prog.commands[command.name](command.args, function (err) {
-    if (err) {
-      log.error(command.name + ' error')
-      log.error('stack', err.stack)
-      errorMessage()
-      log.error('not ok')
-      return process.exit(1)
-    }
+  try {
+    const args = await prog.commands[command.name](command.args) ?? []
+
     if (command.name === 'list') {
-      const versions = arguments[1]
-      if (versions.length > 0) {
-        versions.forEach(function (version) {
-          console.log(version)
-        })
+      if (args.length) {
+        args.forEach((version) => console.log(version))
       } else {
         console.log('No node development files installed. Use `node-gyp install` to install a version.')
       }
-    } else if (arguments.length >= 2) {
-      console.log.apply(console, [].slice.call(arguments, 1))
+    } else if (args.length >= 1) {
+      console.log(...args.slice(1))
     }
 
     // now run the next command in the queue
-    process.nextTick(run)
-  })
+    return run()
+  } catch (err) {
+    log.error(command.name + ' error')
+    log.error('stack', err.stack)
+    errorMessage()
+    log.error('not ok')
+    return process.exit(1)
+  }
 }
 
 process.on('exit', function (code) {
diff --git a/lib/build.js b/lib/build.js
index c0cb9f3904..6b8d84d3ed 100644
--- a/lib/build.js
+++ b/lib/build.js
@@ -202,13 +202,7 @@ async function build (gyp, argv) {
     await new Promise((resolve, reject) => proc.on('exit', async (code, signal) => {
       if (buildBinsDir) {
         // Clean up the build-time dependency symlinks:
-        if (fs.rm) {
-          // fs.rm is only available in Node 14+
-          await fs.rm(buildBinsDir, { recursive: true })
-        } else {
-          // Only used for old node, as recursive rmdir is deprecated in Node 14+
-          await fs.rmdir(buildBinsDir, { recursive: true })
-        }
+        await fs.rm(buildBinsDir, { recursive: true })
       }
 
       if (code !== 0) {
@@ -222,7 +216,5 @@ async function build (gyp, argv) {
   }
 }
 
-module.exports = function (gyp, argv, callback) {
-  build(gyp, argv).then(callback.bind(undefined, null), callback)
-}
+module.exports = build
 module.exports.usage = 'Invokes `' + (win ? 'msbuild' : 'make') + '` and builds the module'
diff --git a/lib/clean.js b/lib/clean.js
index 844b47aed1..523f8016ca 100644
--- a/lib/clean.js
+++ b/lib/clean.js
@@ -1,6 +1,6 @@
 'use strict'
 
-const fs = require('fs/promises')
+const fs = require('graceful-fs').promises
 const log = require('./log')
 
 async function clean (gyp, argv) {
@@ -11,7 +11,5 @@ async function clean (gyp, argv) {
   await fs.rm(buildDir, { recursive: true, force: true })
 }
 
-module.exports = function (gyp, argv, callback) {
-  clean(gyp, argv).then(callback.bind(undefined, null), callback)
-}
+module.exports = clean
 module.exports.usage = 'Removes any generated build files and the "out" dir'
diff --git a/lib/configure.js b/lib/configure.js
index 01c3f497ad..8ccfc3fad5 100644
--- a/lib/configure.js
+++ b/lib/configure.js
@@ -1,6 +1,6 @@
 'use strict'
 
-const fs = require('graceful-fs')
+const { openSync, closeSync, promises: fs } = require('graceful-fs')
 const path = require('path')
 const log = require('./log')
 const os = require('os')
@@ -8,40 +8,28 @@ const processRelease = require('./process-release')
 const win = process.platform === 'win32'
 const findNodeDirectory = require('./find-node-directory')
 const createConfigGypi = require('./create-config-gypi')
-const msgFormat = require('util').format
+const { format: msgFormat } = require('util')
 const findPython = require('./find-python')
-let findVisualStudio
-if (win) {
-  findVisualStudio = require('./find-visualstudio')
-}
+const findVisualStudio = win ? require('./find-visualstudio') : null
 
-function configure (gyp, argv, callback) {
-  let python
+async function configure (gyp, argv) {
   const buildDir = path.resolve('build')
   const configNames = ['config.gypi', 'common.gypi']
   const configs = []
   let nodeDir
   const release = processRelease(argv, gyp, process.version, process.release)
 
-  findPython(gyp.opts.python, function (err, found) {
-    if (err) {
-      callback(err)
-    } else {
-      python = found
-      getNodeDir()
-    }
-  })
+  const python = await findPython(gyp.opts.python)
+  return getNodeDir()
 
-  function getNodeDir () {
+  async function getNodeDir () {
     // 'python' should be set by now
     process.env.PYTHON = python
 
     if (gyp.opts.nodedir) {
       // --nodedir was specified. use that for the dev files
       nodeDir = gyp.opts.nodedir.replace(/^~/, os.homedir())
-
       log.verbose('get node dir', 'compiling against specified --nodedir dev files: %s', nodeDir)
-      createBuildDir()
     } else {
       // if no --nodedir specified, ensure node dependencies are installed
       if ('v' + release.version !== process.version) {
@@ -54,87 +42,66 @@ function configure (gyp, argv, callback) {
 
       if (!release.semver) {
         // could not parse the version string with semver
-        return callback(new Error('Invalid version number: ' + release.version))
+        throw new Error('Invalid version number: ' + release.version)
       }
 
       // If the tarball option is set, always remove and reinstall the headers
       // into devdir. Otherwise only install if they're not already there.
       gyp.opts.ensure = !gyp.opts.tarball
 
-      gyp.commands.install([release.version], function (err) {
-        if (err) {
-          return callback(err)
-        }
-        log.verbose('get node dir', 'target node version installed:', release.versionDir)
-        nodeDir = path.resolve(gyp.devDir, release.versionDir)
-        createBuildDir()
-      })
+      await gyp.commands.install([release.version])
+
+      log.verbose('get node dir', 'target node version installed:', release.versionDir)
+      nodeDir = path.resolve(gyp.devDir, release.versionDir)
     }
+
+    return createBuildDir()
   }
 
-  function createBuildDir () {
+  async function createBuildDir () {
     log.verbose('build dir', 'attempting to create "build" dir: %s', buildDir)
 
-    fs.mkdir(buildDir, { recursive: true }, function (err, isNew) {
-      if (err) {
-        return callback(err)
-      }
-      log.verbose(
-        'build dir', '"build" dir needed to be created?', isNew ? 'Yes' : 'No'
-      )
-      if (win) {
-        findVisualStudio(release.semver, gyp.opts['msvs-version'],
-          createConfigFile)
-      } else {
-        createConfigFile()
-      }
-    })
+    const isNew = await fs.mkdir(buildDir, { recursive: true })
+    log.verbose(
+      'build dir', '"build" dir needed to be created?', isNew ? 'Yes' : 'No'
+    )
+    const vsInfo = win ? await findVisualStudio(release.semver, gyp.opts['msvs-version']) : null
+    return createConfigFile(vsInfo)
   }
 
-  function createConfigFile (err, vsInfo) {
-    if (err) {
-      return callback(err)
-    }
-    if (process.platform === 'win32') {
+  async function createConfigFile (vsInfo) {
+    if (win) {
       process.env.GYP_MSVS_VERSION = Math.min(vsInfo.versionYear, 2015)
       process.env.GYP_MSVS_OVERRIDE_PATH = vsInfo.path
     }
-    createConfigGypi({ gyp, buildDir, nodeDir, vsInfo, python }).then(configPath => {
-      configs.push(configPath)
-      findConfigs()
-    }).catch(err => {
-      callback(err)
-    })
+    const configPath = await createConfigGypi({ gyp, buildDir, nodeDir, vsInfo, python })
+    configs.push(configPath)
+    return findConfigs()
   }
 
-  function findConfigs () {
+  async function findConfigs () {
     const name = configNames.shift()
     if (!name) {
       return runGyp()
     }
-    const fullPath = path.resolve(name)
 
+    const fullPath = path.resolve(name)
     log.verbose(name, 'checking for gypi file: %s', fullPath)
-    fs.stat(fullPath, function (err) {
-      if (err) {
-        if (err.code === 'ENOENT') {
-          findConfigs() // check next gypi filename
-        } else {
-          callback(err)
-        }
-      } else {
-        log.verbose(name, 'found gypi file')
-        configs.push(fullPath)
-        findConfigs()
+    try {
+      await fs.stat(fullPath)
+      log.verbose(name, 'found gypi file')
+      configs.push(fullPath)
+    } catch (err) {
+      // ENOENT will check next gypi filename
+      if (err.code !== 'ENOENT') {
+        throw err
       }
-    })
-  }
-
-  function runGyp (err) {
-    if (err) {
-      return callback(err)
     }
 
+    return findConfigs()
+  }
+
+  async function runGyp () {
     if (!~argv.indexOf('-f') && !~argv.indexOf('--format')) {
       if (win) {
         log.verbose('gyp', 'gyp format was not specified; forcing "msvs"')
@@ -189,7 +156,7 @@ function configure (gyp, argv, callback) {
       } else {
         const msg = msgFormat('Could not find node.%s file in %s', ext, nodeRootDir)
         log.error(logprefix, 'Could not find exports file')
-        return callback(new Error(msg))
+        throw new Error(msg)
       }
     }
 
@@ -227,7 +194,7 @@ function configure (gyp, argv, callback) {
       } else if (release.version.split('.')[0] >= 16) {
         // zoslib is only shipped in Node v16 and above.
         log.error(logprefix, msg)
-        return callback(new Error(msg))
+        throw new Error(msg)
       }
     }
 
@@ -235,79 +202,79 @@ function configure (gyp, argv, callback) {
     const gypScript = path.resolve(__dirname, '..', 'gyp', 'gyp_main.py')
     const addonGypi = path.resolve(__dirname, '..', 'addon.gypi')
     let commonGypi = path.resolve(nodeDir, 'include/node/common.gypi')
-    fs.stat(commonGypi, function (err) {
-      if (err) {
-        commonGypi = path.resolve(nodeDir, 'common.gypi')
-      }
+    try {
+      await fs.stat(commonGypi)
+    } catch (err) {
+      commonGypi = path.resolve(nodeDir, 'common.gypi')
+    }
 
-      let outputDir = 'build'
-      if (win) {
-        // Windows expects an absolute path
-        outputDir = buildDir
-      }
-      const nodeGypDir = path.resolve(__dirname, '..')
-
-      let nodeLibFile = path.join(nodeDir,
-        !gyp.opts.nodedir ? '<(target_arch)' : '$(Configuration)',
-        release.name + '.lib')
-
-      argv.push('-I', addonGypi)
-      argv.push('-I', commonGypi)
-      argv.push('-Dlibrary=shared_library')
-      argv.push('-Dvisibility=default')
-      argv.push('-Dnode_root_dir=' + nodeDir)
-      if (process.platform === 'aix' || process.platform === 'os390' || process.platform === 'os400') {
-        argv.push('-Dnode_exp_file=' + nodeExpFile)
-        if (process.platform === 'os390' && zoslibIncDir) {
-          argv.push('-Dzoslib_include_dir=' + zoslibIncDir)
-        }
-      }
-      argv.push('-Dnode_gyp_dir=' + nodeGypDir)
+    let outputDir = 'build'
+    if (win) {
+      // Windows expects an absolute path
+      outputDir = buildDir
+    }
+    const nodeGypDir = path.resolve(__dirname, '..')
 
-      // Do this to keep Cygwin environments happy, else the unescaped '\' gets eaten up,
-      // resulting in bad paths, Ex c:parentFolderfolderanotherFolder instead of c:\parentFolder\folder\anotherFolder
-      if (win) {
-        nodeLibFile = nodeLibFile.replace(/\\/g, '\\\\')
+    let nodeLibFile = path.join(nodeDir,
+      !gyp.opts.nodedir ? '<(target_arch)' : '$(Configuration)',
+      release.name + '.lib')
+
+    argv.push('-I', addonGypi)
+    argv.push('-I', commonGypi)
+    argv.push('-Dlibrary=shared_library')
+    argv.push('-Dvisibility=default')
+    argv.push('-Dnode_root_dir=' + nodeDir)
+    if (process.platform === 'aix' || process.platform === 'os390' || process.platform === 'os400') {
+      argv.push('-Dnode_exp_file=' + nodeExpFile)
+      if (process.platform === 'os390' && zoslibIncDir) {
+        argv.push('-Dzoslib_include_dir=' + zoslibIncDir)
       }
-      argv.push('-Dnode_lib_file=' + nodeLibFile)
-      argv.push('-Dmodule_root_dir=' + process.cwd())
-      argv.push('-Dnode_engine=' +
+    }
+    argv.push('-Dnode_gyp_dir=' + nodeGypDir)
+
+    // Do this to keep Cygwin environments happy, else the unescaped '\' gets eaten up,
+    // resulting in bad paths, Ex c:parentFolderfolderanotherFolder instead of c:\parentFolder\folder\anotherFolder
+    if (win) {
+      nodeLibFile = nodeLibFile.replace(/\\/g, '\\\\')
+    }
+    argv.push('-Dnode_lib_file=' + nodeLibFile)
+    argv.push('-Dmodule_root_dir=' + process.cwd())
+    argv.push('-Dnode_engine=' +
         (gyp.opts.node_engine || process.jsEngine || 'v8'))
-      argv.push('--depth=.')
-      argv.push('--no-parallel')
+    argv.push('--depth=.')
+    argv.push('--no-parallel')
 
-      // tell gyp to write the Makefile/Solution files into output_dir
-      argv.push('--generator-output', outputDir)
+    // tell gyp to write the Makefile/Solution files into output_dir
+    argv.push('--generator-output', outputDir)
 
-      // tell make to write its output into the same dir
-      argv.push('-Goutput_dir=.')
+    // tell make to write its output into the same dir
+    argv.push('-Goutput_dir=.')
 
-      // enforce use of the "binding.gyp" file
-      argv.unshift('binding.gyp')
+    // enforce use of the "binding.gyp" file
+    argv.unshift('binding.gyp')
 
-      // execute `gyp` from the current target nodedir
-      argv.unshift(gypScript)
+    // execute `gyp` from the current target nodedir
+    argv.unshift(gypScript)
 
-      // make sure python uses files that came with this particular node package
-      const pypath = [path.join(__dirname, '..', 'gyp', 'pylib')]
-      if (process.env.PYTHONPATH) {
-        pypath.push(process.env.PYTHONPATH)
-      }
-      process.env.PYTHONPATH = pypath.join(win ? ';' : ':')
+    // make sure python uses files that came with this particular node package
+    const pypath = [path.join(__dirname, '..', 'gyp', 'pylib')]
+    if (process.env.PYTHONPATH) {
+      pypath.push(process.env.PYTHONPATH)
+    }
+    process.env.PYTHONPATH = pypath.join(win ? ';' : ':')
 
+    await new Promise((resolve, reject) => {
       const cp = gyp.spawn(python, argv)
-      cp.on('exit', onCpExit)
+      cp.on('exit', (code) => {
+        if (code !== 0) {
+          reject(new Error('`gyp` failed with exit code: ' + code))
+        } else {
+          // we're done
+          resolve()
+        }
+      })
     })
   }
-
-  function onCpExit (code) {
-    if (code !== 0) {
-      callback(new Error('`gyp` failed with exit code: ' + code))
-    } else {
-      // we're done
-      callback()
-    }
-  }
 }
 
 /**
@@ -320,13 +287,13 @@ function findAccessibleSync (logprefix, dir, candidates) {
     const candidate = path.resolve(dir, candidates[next])
     let fd
     try {
-      fd = fs.openSync(candidate, 'r')
+      fd = openSync(candidate, 'r')
     } catch (e) {
       // this candidate was not found or not readable, do nothing
       log.silly(logprefix, 'Could not open %s: %s', candidate, e.message)
       continue
     }
-    fs.closeSync(fd)
+    closeSync(fd)
     log.silly(logprefix, 'Found readable %s', candidate)
     return candidate
   }
diff --git a/lib/create-config-gypi.js b/lib/create-config-gypi.js
index 8687379385..a9c427e51d 100644
--- a/lib/create-config-gypi.js
+++ b/lib/create-config-gypi.js
@@ -1,6 +1,6 @@
 'use strict'
 
-const fs = require('graceful-fs')
+const fs = require('graceful-fs').promises
 const log = require('./log')
 const path = require('path')
 
@@ -24,7 +24,7 @@ async function getBaseConfigGypi ({ gyp, nodeDir }) {
   if (shouldReadConfigGypi && nodeDir) {
     try {
       const baseConfigGypiPath = path.resolve(nodeDir, 'include/node/config.gypi')
-      const baseConfigGypi = await fs.promises.readFile(baseConfigGypiPath)
+      const baseConfigGypi = await fs.readFile(baseConfigGypiPath)
       return parseConfigGypi(baseConfigGypi.toString())
     } catch (err) {
       log.warn('read config.gypi', err.message)
@@ -138,7 +138,7 @@ async function createConfigGypi ({ gyp, buildDir, nodeDir, vsInfo, python }) {
 
   const json = JSON.stringify(config, boolsToString, 2)
   log.verbose('build/' + configFilename, 'writing out config file: %s', configPath)
-  await fs.promises.writeFile(configPath, [prefix, json, ''].join('\n'))
+  await fs.writeFile(configPath, [prefix, json, ''].join('\n'))
 
   return configPath
 }
diff --git a/lib/find-python.js b/lib/find-python.js
index 3288839b50..f01d1dd454 100644
--- a/lib/find-python.js
+++ b/lib/find-python.js
@@ -2,8 +2,8 @@
 
 const log = require('./log')
 const semver = require('semver')
-const cp = require('child_process')
-const extend = require('util')._extend // eslint-disable-line
+const { _extend: extend } = require('util') // eslint-disable-line n/no-deprecated-api
+const { execFile } = require('./util')
 const win = process.platform === 'win32'
 
 const systemDrive = process.env.SystemDrive || 'C:'
@@ -38,8 +38,7 @@ function getOsUserInfo () {
   } catch (e) {}
 }
 
-function PythonFinder (configPython, callback) {
-  this.callback = callback
+function PythonFinder (configPython) {
   this.configPython = configPython
   this.errorLog = []
 }
@@ -51,7 +50,7 @@ PythonFinder.prototype = {
   semverRange: '>=3.6.0',
 
   // These can be overridden for testing:
-  execFile: cp.execFile,
+  execFile,
   env: process.env,
   win,
   pyLauncher: 'py.exe',
@@ -66,11 +65,10 @@ PythonFinder.prototype = {
 
   // Find Python by trying a sequence of possibilities.
   // Ignore errors, keep trying until Python is found.
-  findPython: function findPython () {
-    const SKIP = 0; const FAIL = 1
-    const toCheck = getChecks.apply(this)
-
-    function getChecks () {
+  findPython: async function findPython () {
+    const SKIP = 0
+    const FAIL = 1
+    const toCheck = (() => {
       if (this.env.NODE_GYP_FORCE_PYTHON) {
         return [{
           before: () => {
@@ -79,8 +77,7 @@ PythonFinder.prototype = {
             this.addLog('- process.env.NODE_GYP_FORCE_PYTHON is ' +
               `"${this.env.NODE_GYP_FORCE_PYTHON}"`)
           },
-          check: this.checkCommand,
-          arg: this.env.NODE_GYP_FORCE_PYTHON
+          check: () => this.checkCommand(this.env.NODE_GYP_FORCE_PYTHON)
         }]
       }
 
@@ -97,8 +94,7 @@ PythonFinder.prototype = {
             this.addLog('- "--python=" or "npm config get python" is ' +
               `"${this.configPython}"`)
           },
-          check: this.checkCommand,
-          arg: this.configPython
+          check: () => this.checkCommand(this.configPython)
         },
         {
           before: () => {
@@ -111,8 +107,7 @@ PythonFinder.prototype = {
               'variable PYTHON')
             this.addLog(`- process.env.PYTHON is "${this.env.PYTHON}"`)
           },
-          check: this.checkCommand,
-          arg: this.env.PYTHON
+          check: () => this.checkCommand(this.env.PYTHON)
         }
       ]
 
@@ -122,20 +117,18 @@ PythonFinder.prototype = {
             this.addLog(
               'checking if the py launcher can be used to find Python 3')
           },
-          check: this.checkPyLauncher
+          check: () => this.checkPyLauncher()
         })
       }
 
       checks.push(...[
         {
           before: () => { this.addLog('checking if "python3" can be used') },
-          check: this.checkCommand,
-          arg: 'python3'
+          check: () => this.checkCommand('python3')
         },
         {
           before: () => { this.addLog('checking if "python" can be used') },
-          check: this.checkCommand,
-          arg: 'python'
+          check: () => this.checkCommand('python')
         }
       ])
 
@@ -143,49 +136,37 @@ PythonFinder.prototype = {
         for (let i = 0; i < this.winDefaultLocations.length; ++i) {
           const location = this.winDefaultLocations[i]
           checks.push({
-            before: () => {
-              this.addLog('checking if Python is ' +
-                `${location}`)
-            },
-            check: this.checkExecPath,
-            arg: location
+            before: () => this.addLog(`checking if Python is ${location}`),
+            check: () => this.checkExecPath(location)
           })
         }
       }
 
       return checks
-    }
-
-    function runChecks (err) {
-      this.log.silly('runChecks: err = %j', (err && err.stack) || err)
-
-      const check = toCheck.shift()
-      if (!check) {
-        return this.fail()
-      }
+    })()
 
-      const before = check.before.apply(this)
+    for (const check of toCheck) {
+      const before = check.before()
       if (before === SKIP) {
-        return runChecks.apply(this)
+        continue
       }
       if (before === FAIL) {
         return this.fail()
       }
-
-      const args = [runChecks.bind(this)]
-      if (check.arg) {
-        args.unshift(check.arg)
+      try {
+        return await check.check()
+      } catch (err) {
+        this.log.silly('runChecks: err = %j', (err && err.stack) || err)
       }
-      check.check.apply(this, args)
     }
 
-    runChecks.apply(this)
+    return this.fail()
   },
 
   // Check if command is a valid Python to use.
   // Will exit the Python finder on success.
   // If on Windows, run in a CMD shell to support BAT/CMD launchers.
-  checkCommand: function checkCommand (command, errorCallback) {
+  checkCommand: async function checkCommand (command) {
     let exec = command
     let args = this.argsExecutable
     let shell = false
@@ -197,18 +178,18 @@ PythonFinder.prototype = {
     }
 
     this.log.verbose(`- executing "${command}" to get executable path`)
-    this.run(exec, args, shell, function (err, execPath) {
-      // Possible outcomes:
-      // - Error: not in PATH, not executable or execution fails
-      // - Gibberish: the next command to check version will fail
-      // - Absolute path to executable
-      if (err) {
-        this.addLog(`- "${command}" is not in PATH or produced an error`)
-        return errorCallback(err)
-      }
+    // Possible outcomes:
+    // - Error: not in PATH, not executable or execution fails
+    // - Gibberish: the next command to check version will fail
+    // - Absolute path to executable
+    try {
+      const execPath = await this.run(exec, args, shell)
       this.addLog(`- executable path is "${execPath}"`)
-      this.checkExecPath(execPath, errorCallback)
-    }.bind(this))
+      return this.checkExecPath(execPath)
+    } catch (err) {
+      this.addLog(`- "${command}" is not in PATH or produced an error`)
+      throw err
+    }
   },
 
   // Check if the py launcher can find a valid Python to use.
@@ -221,37 +202,31 @@ PythonFinder.prototype = {
   // the first command line argument. Since "py.exe -3" would be an invalid
   // executable for "execFile", we have to use the launcher to figure out
   // where the actual "python.exe" executable is located.
-  checkPyLauncher: function checkPyLauncher (errorCallback) {
-    this.log.verbose(
-      `- executing "${this.pyLauncher}" to get Python 3 executable path`)
-    this.run(this.pyLauncher, ['-3', ...this.argsExecutable], false,
-      function (err, execPath) {
-      // Possible outcomes: same as checkCommand
-        if (err) {
-          this.addLog(
-            `- "${this.pyLauncher}" is not in PATH or produced an error`)
-          return errorCallback(err)
-        }
-        this.addLog(`- executable path is "${execPath}"`)
-        this.checkExecPath(execPath, errorCallback)
-      }.bind(this))
+  checkPyLauncher: async function checkPyLauncher () {
+    this.log.verbose(`- executing "${this.pyLauncher}" to get Python 3 executable path`)
+    // Possible outcomes: same as checkCommand
+    try {
+      const execPath = await this.run(this.pyLauncher, ['-3', ...this.argsExecutable], false)
+      this.addLog(`- executable path is "${execPath}"`)
+      return this.checkExecPath(execPath)
+    } catch (err) {
+      this.addLog(`- "${this.pyLauncher}" is not in PATH or produced an error`)
+      throw err
+    }
   },
 
   // Check if a Python executable is the correct version to use.
   // Will exit the Python finder on success.
-  checkExecPath: function checkExecPath (execPath, errorCallback) {
+  checkExecPath: async function checkExecPath (execPath) {
     this.log.verbose(`- executing "${execPath}" to get version`)
-    this.run(execPath, this.argsVersion, false, function (err, version) {
-      // Possible outcomes:
-      // - Error: executable can not be run (likely meaning the command wasn't
-      //   a Python executable and the previous command produced gibberish)
-      // - Gibberish: somehow the last command produced an executable path,
-      //   this will fail when verifying the version
-      // - Version of the Python executable
-      if (err) {
-        this.addLog(`- "${execPath}" could not be run`)
-        return errorCallback(err)
-      }
+    // Possible outcomes:
+    // - Error: executable can not be run (likely meaning the command wasn't
+    //   a Python executable and the previous command produced gibberish)
+    // - Gibberish: somehow the last command produced an executable path,
+    //   this will fail when verifying the version
+    // - Version of the Python executable
+    try {
+      const version = await this.run(execPath, this.argsVersion, false)
       this.addLog(`- version is "${version}"`)
 
       const range = new semver.Range(this.semverRange)
@@ -262,21 +237,22 @@ PythonFinder.prototype = {
         this.log.silly('range.test() threw:\n%s', err.stack)
         this.addLog(`- "${execPath}" does not have a valid version`)
         this.addLog('- is it a Python executable?')
-        return errorCallback(err)
+        throw err
       }
-
       if (!valid) {
         this.addLog(`- version is ${version} - should be ${this.semverRange}`)
         this.addLog('- THIS VERSION OF PYTHON IS NOT SUPPORTED')
-        return errorCallback(new Error(
-          `Found unsupported Python version ${version}`))
+        throw new Error(`Found unsupported Python version ${version}`)
       }
-      this.succeed(execPath, version)
-    }.bind(this))
+      return this.succeed(execPath, version)
+    } catch (err) {
+      this.addLog(`- "${execPath}" could not be run`)
+      throw err
+    }
   },
 
   // Run an executable or shell command, trimming the output.
-  run: function run (exec, args, shell, callback) {
+  run: async function run (exec, args, shell) {
     const env = extend({}, this.env)
     env.TERM = 'dumb'
     const opts = { env, shell }
@@ -285,27 +261,20 @@ PythonFinder.prototype = {
     this.log.silly('execFile: args = %j', args)
     this.log.silly('execFile: opts = %j', opts)
     try {
-      this.execFile(exec, args, opts, execFileCallback.bind(this))
-    } catch (err) {
-      this.log.silly('execFile: threw:\n%s', err.stack)
-      return callback(err)
-    }
-
-    function execFileCallback (err, stdout, stderr) {
+      const [err, stdout, stderr] = await this.execFile(exec, args, opts)
       this.log.silly('execFile result: err = %j', (err && err.stack) || err)
       this.log.silly('execFile result: stdout = %j', stdout)
       this.log.silly('execFile result: stderr = %j', stderr)
-      if (err) {
-        return callback(err)
-      }
-      const execPath = stdout.trim()
-      callback(null, execPath)
+      return stdout.trim()
+    } catch (err) {
+      this.log.silly('execFile: threw:\n%s', err.stack)
+      throw err
     }
   },
 
   succeed: function succeed (execPath, version) {
     this.log.info(`using Python version ${version} found at "${execPath}"`)
-    process.nextTick(this.callback.bind(null, null, execPath))
+    return execPath
   },
 
   fail: function fail () {
@@ -333,15 +302,11 @@ PythonFinder.prototype = {
     ].join('\n')
 
     this.log.error(`\n${errorLog}\n\n${info}\n`)
-    process.nextTick(this.callback.bind(null, new Error(
-      'Could not find any Python installation to use')))
+    throw new Error('Could not find any Python installation to use')
   }
 }
 
-function findPython (configPython, callback) {
-  const finder = new PythonFinder(configPython, callback)
-  finder.findPython()
-}
+const findPython = async (configPython) => new PythonFinder(configPython).findPython()
 
 module.exports = findPython
 module.exports.test = {
diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js
index 45ef7d3fb1..7df976a2db 100644
--- a/lib/find-visualstudio.js
+++ b/lib/find-visualstudio.js
@@ -1,21 +1,13 @@
 'use strict'
 
 const log = require('./log')
-const execFile = require('child_process').execFile
-const fs = require('fs')
-const path = require('path').win32
-const regSearchKeys = require('./util').regSearchKeys
-
-function findVisualStudio (nodeSemver, configMsvsVersion, callback) {
-  const finder = new VisualStudioFinder(nodeSemver, configMsvsVersion,
-    callback)
-  finder.findVisualStudio()
-}
+const { existsSync } = require('fs')
+const { win32: path } = require('path')
+const { regSearchKeys, execFile } = require('./util')
 
-function VisualStudioFinder (nodeSemver, configMsvsVersion, callback) {
+function VisualStudioFinder (nodeSemver, configMsvsVersion) {
   this.nodeSemver = nodeSemver
   this.configMsvsVersion = configMsvsVersion
-  this.callback = callback
   this.errorLog = []
   this.validVersions = []
 }
@@ -32,7 +24,7 @@ VisualStudioFinder.prototype = {
     this.errorLog.push(message)
   },
 
-  findVisualStudio: function findVisualStudio () {
+  findVisualStudio: async function findVisualStudio () {
     this.configVersionYear = null
     this.configPath = null
     if (this.configMsvsVersion) {
@@ -59,29 +51,27 @@ VisualStudioFinder.prototype = {
       this.addLog('VCINSTALLDIR not set, not running in VS Command Prompt')
     }
 
-    this.findVisualStudio2017OrNewer((info) => {
+    const checks = [
+      () => this.findVisualStudio2017OrNewer(),
+      () => this.findVisualStudio2015(),
+      () => this.findVisualStudio2013()
+    ]
+
+    for (const check of checks) {
+      const info = await check()
       if (info) {
         return this.succeed(info)
       }
-      this.findVisualStudio2015((info) => {
-        if (info) {
-          return this.succeed(info)
-        }
-        this.findVisualStudio2013((info) => {
-          if (info) {
-            return this.succeed(info)
-          }
-          this.fail()
-        })
-      })
-    })
+    }
+
+    return this.fail()
   },
 
   succeed: function succeed (info) {
     this.log.info(`using VS${info.versionYear} (${info.version}) found at:` +
                   `\n"${info.path}"` +
                   '\nrun with --verbose for detailed information')
-    process.nextTick(this.callback.bind(null, null, info))
+    return info
   },
 
   fail: function fail () {
@@ -118,13 +108,12 @@ VisualStudioFinder.prototype = {
     ].join('\n')
 
     this.log.error(`\n${errorLog}\n\n${infoLog}\n`)
-    process.nextTick(this.callback.bind(null, new Error(
-      'Could not find any Visual Studio installation to use')))
+    throw new Error('Could not find any Visual Studio installation to use')
   },
 
   // Invoke the PowerShell script to get information about Visual Studio 2017
   // or newer installations
-  findVisualStudio2017OrNewer: function findVisualStudio2017OrNewer (cb) {
+  findVisualStudio2017OrNewer: async function findVisualStudio2017OrNewer () {
     const ps = path.join(process.env.SystemRoot, 'System32',
       'WindowsPowerShell', 'v1.0', 'powershell.exe')
     const csFile = path.join(__dirname, 'Find-VisualStudio.cs')
@@ -137,22 +126,19 @@ VisualStudioFinder.prototype = {
     ]
 
     this.log.silly('Running', ps, psArgs)
-    const child = execFile(ps, psArgs, { encoding: 'utf8' },
-      (err, stdout, stderr) => {
-        this.parseData(err, stdout, stderr, cb)
-      })
-    child.stdin.end()
+    const [err, stdout, stderr] = await execFile(ps, psArgs, { encoding: 'utf8' })
+    return this.parseData(err, stdout, stderr)
   },
 
   // Parse the output of the PowerShell script and look for an installation
   // of Visual Studio 2017 or newer to use
-  parseData: function parseData (err, stdout, stderr, cb) {
+  parseData: function parseData (err, stdout, stderr) {
     this.log.silly('PS stderr = %j', stderr)
 
     const failPowershell = () => {
       this.addLog(
         'could not use PowerShell to find Visual Studio 2017 or newer, try re-running with \'--loglevel silly\' for more details')
-      cb(null)
+      return null
     }
 
     if (err) {
@@ -228,12 +214,12 @@ VisualStudioFinder.prototype = {
         continue
       }
 
-      return cb(info)
+      return info
     }
 
     this.addLog(
       'could not find a version of Visual Studio 2017 or newer to use')
-    cb(null)
+    return null
   },
 
   // Helper - process version information
@@ -266,7 +252,7 @@ VisualStudioFinder.prototype = {
   },
 
   msBuildPathExists: function msBuildPathExists (path) {
-    return fs.existsSync(path)
+    return existsSync(path)
   },
 
   // Helper - process MSBuild information
@@ -356,11 +342,11 @@ VisualStudioFinder.prototype = {
   },
 
   // Find an installation of Visual Studio 2015 to use
-  findVisualStudio2015: function findVisualStudio2015 (cb) {
+  findVisualStudio2015: async function findVisualStudio2015 () {
     if (this.nodeSemver.major >= 19) {
       this.addLog(
         'not looking for VS2015 as it is only supported up to Node.js 18')
-      return cb(null)
+      return null
     }
     return this.findOldVS({
       version: '14.0',
@@ -368,15 +354,15 @@ VisualStudioFinder.prototype = {
       versionMinor: 0,
       versionYear: 2015,
       toolset: 'v140'
-    }, cb)
+    })
   },
 
   // Find an installation of Visual Studio 2013 to use
-  findVisualStudio2013: function findVisualStudio2013 (cb) {
+  findVisualStudio2013: async function findVisualStudio2013 () {
     if (this.nodeSemver.major >= 9) {
       this.addLog(
         'not looking for VS2013 as it is only supported up to Node.js 8')
-      return cb(null)
+      return null
     }
     return this.findOldVS({
       version: '12.0',
@@ -384,47 +370,44 @@ VisualStudioFinder.prototype = {
       versionMinor: 0,
       versionYear: 2013,
       toolset: 'v120'
-    }, cb)
+    })
   },
 
   // Helper - common code for VS2013 and VS2015
-  findOldVS: function findOldVS (info, cb) {
+  findOldVS: async function findOldVS (info) {
     const regVC7 = ['HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7',
       'HKLM\\Software\\Wow6432Node\\Microsoft\\VisualStudio\\SxS\\VC7']
     const regMSBuild = 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions'
 
     this.addLog(`looking for Visual Studio ${info.versionYear}`)
-    this.regSearchKeys(regVC7, info.version, [], (err, res) => {
-      if (err) {
-        this.addLog('- not found')
-        return cb(null)
-      }
-
+    try {
+      let res = await this.regSearchKeys(regVC7, info.version, [])
       const vsPath = path.resolve(res, '..')
       this.addLog(`- found in "${vsPath}"`)
-
       const msBuildRegOpts = process.arch === 'ia32' ? [] : ['/reg:32']
-      this.regSearchKeys([`${regMSBuild}\\${info.version}`],
-        'MSBuildToolsPath', msBuildRegOpts, (err, res) => {
-          if (err) {
-            this.addLog(
-              '- could not find MSBuild in registry for this version')
-            return cb(null)
-          }
-
-          const msBuild = path.join(res, 'MSBuild.exe')
-          this.addLog(`- MSBuild in "${msBuild}"`)
-
-          if (!this.checkConfigVersion(info.versionYear, vsPath)) {
-            return cb(null)
-          }
-
-          info.path = vsPath
-          info.msBuild = msBuild
-          info.sdk = null
-          cb(info)
-        })
-    })
+
+      try {
+        res = await this.regSearchKeys([`${regMSBuild}\\${info.version}`], 'MSBuildToolsPath', msBuildRegOpts)
+      } catch (err) {
+        this.addLog('- could not find MSBuild in registry for this version')
+        return null
+      }
+
+      const msBuild = path.join(res, 'MSBuild.exe')
+      this.addLog(`- MSBuild in "${msBuild}"`)
+
+      if (!this.checkConfigVersion(info.versionYear, vsPath)) {
+        return null
+      }
+
+      info.path = vsPath
+      info.msBuild = msBuild
+      info.sdk = null
+      return info
+    } catch (err) {
+      this.addLog('- not found')
+      return null
+    }
   },
 
   // After finding a usable version of Visual Studio:
@@ -455,6 +438,8 @@ VisualStudioFinder.prototype = {
   }
 }
 
+const findVisualStudio = async (nodeSemver, configMsvsVersion) => new VisualStudioFinder(nodeSemver, configMsvsVersion).findVisualStudio()
+
 module.exports = findVisualStudio
 module.exports.test = {
   VisualStudioFinder,
diff --git a/lib/install.js b/lib/install.js
index 7ac34d2b83..b338102c95 100644
--- a/lib/install.js
+++ b/lib/install.js
@@ -1,26 +1,20 @@
 'use strict'
 
-const fs = require('graceful-fs')
+const { createWriteStream, promises: fs } = require('graceful-fs')
 const os = require('os')
 const { backOff } = require('exponential-backoff')
-const rm = require('rimraf')
 const tar = require('tar')
 const path = require('path')
-const util = require('util')
-const stream = require('stream')
+const { Transform, promises: { pipeline } } = require('stream')
 const crypto = require('crypto')
 const log = require('./log')
 const semver = require('semver')
 const fetch = require('make-fetch-happen')
 const processRelease = require('./process-release')
 const win = process.platform === 'win32'
-const streamPipeline = util.promisify(stream.pipeline)
 
-/**
- * @param {typeof import('graceful-fs')} fs
- */
-
-async function install (fs, gyp, argv) {
+async function install (gyp, argv) {
+  console.log()
   const release = processRelease(argv, gyp, process.version, process.release)
   // Detecting target_arch based on logic from create-cnfig-gyp.js. Used on Windows only.
   const arch = win ? (gyp.opts.target_arch || gyp.opts.arch || process.arch || 'ia32') : ''
@@ -60,7 +54,7 @@ async function install (fs, gyp, argv) {
   if (gyp.opts.ensure) {
     log.verbose('install', '--ensure was passed, so won\'t reinstall if already installed')
     try {
-      await fs.promises.stat(devDir)
+      await fs.stat(devDir)
     } catch (err) {
       if (err.code === 'ENOENT') {
         log.verbose('install', 'version not already installed, continuing with install', release.version)
@@ -78,7 +72,7 @@ async function install (fs, gyp, argv) {
     const installVersionFile = path.resolve(devDir, 'installVersion')
     let installVersion = 0
     try {
-      const ver = await fs.promises.readFile(installVersionFile, 'ascii')
+      const ver = await fs.readFile(installVersionFile, 'ascii')
       installVersion = parseInt(ver, 10) || 0
     } catch (err) {
       if (err.code !== 'ENOENT') {
@@ -100,7 +94,7 @@ async function install (fs, gyp, argv) {
       log.verbose('on Windows; need to check node.lib')
       const nodeLibPath = path.resolve(devDir, arch, 'node.lib')
       try {
-        await fs.promises.stat(nodeLibPath)
+        await fs.stat(nodeLibPath)
       } catch (err) {
         if (err.code === 'ENOENT') {
           log.verbose('install', `version not already installed for ${arch}, continuing with install`, release.version)
@@ -126,12 +120,12 @@ async function install (fs, gyp, argv) {
 
   async function copyDirectory (src, dest) {
     try {
-      await fs.promises.stat(src)
+      await fs.stat(src)
     } catch {
       throw new Error(`Missing source directory for copy: ${src}`)
     }
-    await fs.promises.mkdir(dest, { recursive: true })
-    const entries = await fs.promises.readdir(src, { withFileTypes: true })
+    await fs.mkdir(dest, { recursive: true })
+    const entries = await fs.readdir(src, { withFileTypes: true })
     for (const entry of entries) {
       if (entry.isDirectory()) {
         await copyDirectory(path.join(src, entry.name), path.join(dest, entry.name))
@@ -140,12 +134,12 @@ async function install (fs, gyp, argv) {
         // Windows so use an exponential backoff to resolve collisions
         await backOff(async () => {
           try {
-            await fs.promises.copyFile(path.join(src, entry.name), path.join(dest, entry.name))
+            await fs.copyFile(path.join(src, entry.name), path.join(dest, entry.name))
           } catch (err) {
             // if ensure, check if file already exists and that's good enough
             if (gyp.opts.ensure && err.code === 'EBUSY') {
               try {
-                await fs.promises.stat(path.join(dest, entry.name))
+                await fs.stat(path.join(dest, entry.name))
                 return
               } catch {}
             }
@@ -163,7 +157,7 @@ async function install (fs, gyp, argv) {
 
     // first create the dir for the node dev files
     try {
-      const created = await fs.promises.mkdir(devDir, { recursive: true })
+      const created = await fs.mkdir(devDir, { recursive: true })
 
       if (created) {
         log.verbose('created devDir', created)
@@ -208,7 +202,7 @@ async function install (fs, gyp, argv) {
     // on Windows there can be file errors from tar if parallel installs
     // are happening (not uncommon with multiple native modules) so
     // extract the tarball to a temp directory first and then copy over
-    const tarExtractDir = win ? await fs.promises.mkdtemp(path.join(os.tmpdir(), 'node-gyp-tmp-')) : devDir
+    const tarExtractDir = win ? await fs.mkdtemp(path.join(os.tmpdir(), 'node-gyp-tmp-')) : devDir
 
     try {
       if (shouldDownloadTarball) {
@@ -228,7 +222,7 @@ async function install (fs, gyp, argv) {
               throw new Error(`${res.status} response downloading ${release.tarballUrl}`)
             }
 
-            await streamPipeline(
+            await pipeline(
               res.body,
               // content checksum
               new ShaSum((_, checksum) => {
@@ -267,7 +261,7 @@ async function install (fs, gyp, argv) {
       // need to download node.lib
         ...(win ? [downloadNodeLib()] : []),
         // write the "installVersion" file
-        fs.promises.writeFile(installVersionPath, gyp.package.installVersion + '\n'),
+        fs.writeFile(installVersionPath, gyp.package.installVersion + '\n'),
         // Only download SHASUMS.txt if we downloaded something in need of SHA verification
         ...(!tarPath || win ? [downloadShasums()] : [])
       ])
@@ -289,7 +283,7 @@ async function install (fs, gyp, argv) {
       if (tarExtractDir !== devDir) {
         try {
           // try to cleanup temp dir
-          await util.promisify(rm)(tarExtractDir)
+          await fs.rm(tarExtractDir, { recursive: true })
         } catch {
           log.warn('failed to clean up temp tarball extract directory')
         }
@@ -329,7 +323,7 @@ async function install (fs, gyp, argv) {
       log.verbose(name, 'dir', dir)
       log.verbose(name, 'url', libUrl)
 
-      await fs.promises.mkdir(dir, { recursive: true })
+      await fs.mkdir(dir, { recursive: true })
       log.verbose('streaming', name, 'to:', targetLibPath)
 
       const res = await download(gyp, libUrl)
@@ -339,13 +333,13 @@ async function install (fs, gyp, argv) {
         throw new Error(`${res.status} status code downloading ${name}`)
       }
 
-      return streamPipeline(
+      return pipeline(
         res.body,
         new ShaSum((_, checksum) => {
           contentShasums[libPath] = checksum
           log.verbose('content checksum', libPath, checksum)
         }),
-        fs.createWriteStream(targetLibPath)
+        createWriteStream(targetLibPath)
       )
     } // downloadNodeLib()
   } // go()
@@ -363,7 +357,7 @@ async function install (fs, gyp, argv) {
   async function rollback (err) {
     log.warn('install', 'got an error, rolling back install')
     // roll-back the install if anything went wrong
-    await util.promisify(gyp.commands.remove)([release.versionDir])
+    await gyp.commands.remove([release.versionDir])
     throw err
   }
 
@@ -394,11 +388,11 @@ async function install (fs, gyp, argv) {
       log.verbose('tmpdir == cwd', 'automatically will remove dev files after to save disk space')
       gyp.todo.push({ name: 'remove', args: argv })
     }
-    return util.promisify(gyp.commands.install)([noretry].concat(argv))
+    return gyp.commands.install([noretry].concat(argv))
   }
 }
 
-class ShaSum extends stream.Transform {
+class ShaSum extends Transform {
   constructor (callback) {
     super()
     this._callback = callback
@@ -442,14 +436,12 @@ async function download (gyp, url) {
 async function readCAFile (filename) {
   // The CA file can contain multiple certificates so split on certificate
   // boundaries.  [\S\s]*? is used to match everything including newlines.
-  const ca = await fs.promises.readFile(filename, 'utf8')
+  const ca = await fs.readFile(filename, 'utf8')
   const re = /(-----BEGIN CERTIFICATE-----[\S\s]*?-----END CERTIFICATE-----)/g
   return ca.match(re)
 }
 
-module.exports = function (gyp, argv, callback) {
-  install(fs, gyp, argv).then(callback.bind(undefined, null), callback)
-}
+module.exports = install
 module.exports.test = {
   download,
   install,
diff --git a/lib/list.js b/lib/list.js
index d93370018f..36889ad4f7 100644
--- a/lib/list.js
+++ b/lib/list.js
@@ -1,26 +1,25 @@
 'use strict'
 
-const fs = require('graceful-fs')
+const fs = require('graceful-fs').promises
 const log = require('./log')
 
-function list (gyp, args, callback) {
+async function list (gyp, args) {
   const devDir = gyp.devDir
   log.verbose('list', 'using node-gyp dir:', devDir)
 
-  fs.readdir(devDir, onreaddir)
-
-  function onreaddir (err, versions) {
-    if (err && err.code !== 'ENOENT') {
-      return callback(err)
+  let versions = []
+  try {
+    const dir = await fs.readdir(devDir)
+    if (Array.isArray(dir)) {
+      versions = dir.filter((v) => v !== 'current')
     }
-
-    if (Array.isArray(versions)) {
-      versions = versions.filter(function (v) { return v !== 'current' })
-    } else {
-      versions = []
+  } catch (err) {
+    if (err && err.code !== 'ENOENT') {
+      throw err
     }
-    callback(null, versions)
   }
+
+  return versions
 }
 
 module.exports = list
diff --git a/lib/node-gyp.js b/lib/node-gyp.js
index 392a0ecfa6..c2ce69051f 100644
--- a/lib/node-gyp.js
+++ b/lib/node-gyp.js
@@ -5,7 +5,7 @@ const nopt = require('nopt')
 const log = require('./log')
 const childProcess = require('child_process')
 const EE = require('events').EventEmitter
-const inherits = require('util').inherits
+const { inherits } = require('util')
 const commands = [
   // Module build commands
   'build',
@@ -27,17 +27,12 @@ function gyp () {
 }
 
 function Gyp () {
-  const self = this
-
   this.devDir = ''
-  this.commands = {}
 
-  commands.forEach(function (command) {
-    self.commands[command] = function (argv, callback) {
-      log.verbose('command', command, argv)
-      return require('./' + command)(self, argv, callback)
-    }
-  })
+  this.commands = commands.reduce((acc, command) => {
+    acc[command] = (argv) => require('./' + command)(this, argv)
+    return acc
+  }, {})
 }
 inherits(Gyp, EE)
 exports.Gyp = Gyp
diff --git a/lib/rebuild.js b/lib/rebuild.js
index a1c5b27cbe..609817665e 100644
--- a/lib/rebuild.js
+++ b/lib/rebuild.js
@@ -1,12 +1,11 @@
 'use strict'
 
-function rebuild (gyp, argv, callback) {
+async function rebuild (gyp, argv) {
   gyp.todo.push(
     { name: 'clean', args: [] }
     , { name: 'configure', args: argv }
     , { name: 'build', args: [] }
   )
-  process.nextTick(callback)
 }
 
 module.exports = rebuild
diff --git a/lib/remove.js b/lib/remove.js
index 9561cb95e5..7efdb01a66 100644
--- a/lib/remove.js
+++ b/lib/remove.js
@@ -1,6 +1,6 @@
 'use strict'
 
-const fs = require('fs/promises')
+const fs = require('graceful-fs').promises
 const path = require('path')
 const log = require('./log')
 const semver = require('semver')
@@ -39,7 +39,5 @@ async function remove (gyp, argv) {
   await fs.rm(versionPath, { recursive: true, force: true })
 }
 
-module.exports = function (gyp, argv, callback) {
-  remove(gyp, argv).then(callback.bind(undefined, null), callback)
-}
+module.exports = remove
 module.exports.usage = 'Removes the node development files for the specified version'
diff --git a/lib/util.js b/lib/util.js
index 5950aa25b4..ef2bfb0827 100644
--- a/lib/util.js
+++ b/lib/util.js
@@ -1,50 +1,55 @@
 'use strict'
 
 const log = require('./log')
-const execFile = require('child_process').execFile
+const cp = require('child_process')
 const path = require('path')
 
-function regGetValue (key, value, addOpts, cb) {
+const execFile = async (...args) => new Promise((resolve) => {
+  const child = cp.execFile(...args, (...a) => resolve(a))
+  child.stdin.end()
+})
+
+async function regGetValue (key, value, addOpts) {
   const outReValue = value.replace(/\W/g, '.')
   const outRe = new RegExp(`^\\s+${outReValue}\\s+REG_\\w+\\s+(\\S.*)$`, 'im')
   const reg = path.join(process.env.SystemRoot, 'System32', 'reg.exe')
   const regArgs = ['query', key, '/v', value].concat(addOpts)
 
   log.silly('reg', 'running', reg, regArgs)
-  const child = execFile(reg, regArgs, { encoding: 'utf8' },
-    function (err, stdout, stderr) {
-      log.silly('reg', 'reg.exe stdout = %j', stdout)
-      if (err || stderr.trim() !== '') {
-        log.silly('reg', 'reg.exe err = %j', err && (err.stack || err))
-        log.silly('reg', 'reg.exe stderr = %j', stderr)
-        return cb(err, stderr)
-      }
-
-      const result = outRe.exec(stdout)
-      if (!result) {
-        log.silly('reg', 'error parsing stdout')
-        return cb(new Error('Could not parse output of reg.exe'))
-      }
-      log.silly('reg', 'found: %j', result[1])
-      cb(null, result[1])
-    })
-  child.stdin.end()
+  const [err, stdout, stderr] = await execFile(reg, regArgs, { encoding: 'utf8' })
+
+  log.silly('reg', 'reg.exe stdout = %j', stdout)
+  if (err || stderr.trim() !== '') {
+    log.silly('reg', 'reg.exe err = %j', err && (err.stack || err))
+    log.silly('reg', 'reg.exe stderr = %j', stderr)
+    if (err) {
+      throw err
+    }
+    throw new Error(stderr)
+  }
+
+  const result = outRe.exec(stdout)
+  if (!result) {
+    log.silly('reg', 'error parsing stdout')
+    throw new Error('Could not parse output of reg.exe')
+  }
+
+  log.silly('reg', 'found: %j', result[1])
+  return result[1]
 }
 
-function regSearchKeys (keys, value, addOpts, cb) {
-  let i = 0
-  const search = () => {
-    log.silly('reg-search', 'looking for %j in %j', value, keys[i])
-    regGetValue(keys[i], value, addOpts, (err, res) => {
-      ++i
-      if (err && i < keys.length) { return search() }
-      cb(err, res)
-    })
+async function regSearchKeys (keys, value, addOpts) {
+  for (const key of keys) {
+    try {
+      return await regGetValue(key, value, addOpts)
+    } catch {
+      continue
+    }
   }
-  search()
 }
 
 module.exports = {
+  execFile,
   regGetValue,
   regSearchKeys
 }
diff --git a/test/process-exec-sync.js b/test/process-exec-sync.js
deleted file mode 100644
index 0a9002cc80..0000000000
--- a/test/process-exec-sync.js
+++ /dev/null
@@ -1,140 +0,0 @@
-'use strict'
-
-const fs = require('graceful-fs')
-const childProcess = require('child_process')
-
-function startsWith (str, search, pos) {
-  if (String.prototype.startsWith) {
-    return str.startsWith(search, pos)
-  }
-
-  return str.substr(!pos || pos < 0 ? 0 : +pos, search.length) === search
-}
-
-function processExecSync (file, args, options) {
-  let error, command
-  command = makeCommand(file, args)
-
-  /*
-    this function emulates child_process.execSync for legacy node <= 0.10.x
-    derived from https://github.com/gvarsanyi/sync-exec/blob/master/js/sync-exec.js
-  */
-
-  options = options || {}
-  // init timeout
-  const timeout = Date.now() + options.timeout
-  // init tmpdir
-  let osTempBase = '/tmp'
-  const os = determineOS()
-  osTempBase = '/tmp'
-
-  if (process.env.TMP) {
-    osTempBase = process.env.TMP
-  }
-
-  if (osTempBase[osTempBase.length - 1] !== '/') {
-    osTempBase += '/'
-  }
-
-  const tmpdir = osTempBase + 'processExecSync.' + Date.now() + Math.random()
-  fs.mkdirSync(tmpdir)
-
-  // init command
-  if (os === 'linux') {
-    command = '(' + command + ' > ' + tmpdir + '/stdout 2> ' + tmpdir +
-      '/stderr); echo $? > ' + tmpdir + '/status'
-  } else {
-    command = '(' + command + ' > ' + tmpdir + '/stdout 2> ' + tmpdir +
-      '/stderr) | echo %errorlevel% > ' + tmpdir + '/status | exit'
-  }
-
-  // init child
-  const child = childProcess.exec(command, options)
-
-  const maxTry = 100000 // increases the test time by 6 seconds on win-2016-node-0.10
-  let tryCount = 0
-  while (tryCount < maxTry) {
-    try {
-      const x = fs.readFileSync(tmpdir + '/status')
-      if (x.toString() === '0') {
-        break
-      }
-    } catch (ignore) {}
-    tryCount++
-    if (Date.now() > timeout) {
-      error = child
-      break
-    }
-  }
-
-  ['stdout', 'stderr', 'status'].forEach(function (file) {
-    child[file] = fs.readFileSync(tmpdir + '/' + file, options.encoding)
-    setTimeout(unlinkFile, 500, tmpdir + '/' + file)
-  })
-
-  child.status = Number(child.status)
-  if (child.status !== 0) {
-    error = child
-  }
-
-  try {
-    fs.rmdirSync(tmpdir)
-  } catch (ignore) {}
-  if (error) {
-    throw error
-  }
-  return child.stdout
-}
-
-function makeCommand (file, args) {
-  let command, quote
-  command = file
-  if (args.length > 0) {
-    for (const i in args) {
-      command = command + ' '
-      if (args[i][0] === '-') {
-        command = command + args[i]
-      } else {
-        if (!quote) {
-          command = command + '"'
-          quote = true
-        }
-        command = command + args[i]
-        if (quote) {
-          if (args.length === (parseInt(i) + 1)) {
-            command = command + '"'
-          }
-        }
-      }
-    }
-  }
-  return command
-}
-
-function determineOS () {
-  let os = ''
-  let tmpVar = ''
-  if (process.env.OSTYPE) {
-    tmpVar = process.env.OSTYPE
-  } else if (process.env.OS) {
-    tmpVar = process.env.OS
-  } else {
-    // default is linux
-    tmpVar = 'linux'
-  }
-
-  if (startsWith(tmpVar, 'linux')) {
-    os = 'linux'
-  }
-  if (startsWith(tmpVar, 'win')) {
-    os = 'win'
-  }
-
-  return os
-}
-
-function unlinkFile (file) {
-  fs.unlinkSync(file)
-}
-
-module.exports = processExecSync
diff --git a/test/test-addon.js b/test/test-addon.js
index 373647dc4c..0a0b9d56c5 100644
--- a/test/test-addon.js
+++ b/test/test-addon.js
@@ -4,12 +4,10 @@ const { describe, it } = require('mocha')
 const assert = require('assert')
 const path = require('path')
 const fs = require('graceful-fs')
-const childProcess = require('child_process')
+const { execFileSync, execFile } = require('child_process')
 const os = require('os')
 const addonPath = path.resolve(__dirname, 'node_modules', 'hello_world')
 const nodeGyp = path.resolve(__dirname, '..', 'bin', 'node-gyp.js')
-const execFileSync = childProcess.execFileSync || require('./process-exec-sync')
-const execFile = childProcess.execFile
 
 function runHello (hostProcess) {
   if (!hostProcess) {
diff --git a/test/test-configure-python.js b/test/test-configure-python.js
index dcce3e5de2..fd8e8d24ef 100644
--- a/test/test-configure-python.js
+++ b/test/test-configure-python.js
@@ -9,13 +9,12 @@ const log = require('../lib/log')
 const requireInject = require('require-inject')
 const configure = requireInject('../lib/configure', {
   'graceful-fs': {
-    openSync: function () { return 0 },
-    closeSync: function () { },
-    writeFile: function (file, data, cb) { cb() },
-    stat: function (file, cb) { cb(null, {}) },
-    mkdir: function (dir, options, cb) { cb() },
+    openSync: () => 0,
+    closeSync: () => {},
     promises: {
-      writeFile: function (file, data) { return Promise.resolve(null) }
+      stat: async () => ({}),
+      mkdir: async () => {},
+      writeFile: async () => {}
     }
   }
 })
diff --git a/test/test-download.js b/test/test-download.js
index 0a13df520b..5b65641f5d 100644
--- a/test/test-download.js
+++ b/test/test-download.js
@@ -4,7 +4,6 @@ const { describe, it, after } = require('mocha')
 const assert = require('assert')
 const fs = require('fs/promises')
 const path = require('path')
-const util = require('util')
 const http = require('http')
 const https = require('https')
 const install = require('../lib/install')
@@ -176,7 +175,7 @@ describe('download', function () {
     prog.parseArgv([])
     prog.devDir = devDir
     log.level = 'warn'
-    await util.promisify(install)(prog, [])
+    await install(prog, [])
 
     const data = await fs.readFile(path.join(expectedDir, 'installVersion'), 'utf8')
     assert.strictEqual(data, '11\n', 'correct installVersion')
diff --git a/test/test-find-python.js b/test/test-find-python.js
index e1880bea7a..a00ae94057 100644
--- a/test/test-find-python.js
+++ b/test/test-find-python.js
@@ -4,22 +4,16 @@ delete process.env.PYTHON
 
 const { describe, it } = require('mocha')
 const assert = require('assert')
-const findPython = require('../lib/find-python')
-const execFile = require('child_process').execFile
-const PythonFinder = findPython.test.PythonFinder
+const { test: { PythonFinder, findPython: testFindPython } } = require('../lib/find-python')
+const { execFile } = require('../lib/util')
 
 describe('find-python', function () {
-  it('find python', function () {
-    findPython.test.findPython(null, function (err, found) {
-      assert.strictEqual(err, null)
-      const proc = execFile(found, ['-V'], function (err, stdout, stderr) {
-        assert.strictEqual(err, null)
-        assert.ok(/Python 3/.test(stdout))
-        assert.strictEqual(stderr, '')
-      })
-      proc.stdout.setEncoding('utf-8')
-      proc.stderr.setEncoding('utf-8')
-    })
+  it('find python', async function () {
+    const found = await testFindPython(null)
+    const [err, stdout, stderr] = await execFile(found, ['-V'], { encoding: 'utf-8' })
+    assert.strictEqual(err, null)
+    assert.ok(/Python 3/.test(stdout))
+    assert.strictEqual(stderr, '')
   })
 
   function poison (object, property) {
@@ -36,109 +30,105 @@ describe('find-python', function () {
     Object.defineProperty(object, property, descriptor)
   }
 
-  function TestPythonFinder () {
-    PythonFinder.apply(this, arguments)
-  }
+  function TestPythonFinder () { PythonFinder.apply(this, arguments) }
   TestPythonFinder.prototype = Object.create(PythonFinder.prototype)
   delete TestPythonFinder.prototype.env.NODE_GYP_FORCE_PYTHON
+  const findPython = async (f) => {
+    try {
+      return { err: null, python: await f.findPython() }
+    } catch (err) {
+      return { err, python: null }
+    }
+  }
 
-  it('find python - python', function () {
-    const f = new TestPythonFinder('python', done)
-    f.execFile = function (program, args, opts, cb) {
-      f.execFile = function (program, args, opts, cb) {
+  it('find python - python', async function () {
+    const f = new TestPythonFinder('python')
+    f.execFile = async function (program, args, opts) {
+      f.execFile = async function (program, args, opts) {
         poison(f, 'execFile')
         assert.strictEqual(program, '/path/python')
         assert.ok(/sys\.version_info/.test(args[1]))
-        cb(null, '3.9.1')
+        return [null, '3.9.1']
       }
-      assert.strictEqual(program,
-        process.platform === 'win32' ? '"python"' : 'python')
+      assert.strictEqual(program, process.platform === 'win32' ? '"python"' : 'python')
       assert.ok(/sys\.executable/.test(args[1]))
-      cb(null, '/path/python')
+      return [null, '/path/python']
     }
-    f.findPython()
 
-    function done (err, python) {
-      assert.strictEqual(err, null)
-      assert.strictEqual(python, '/path/python')
-    }
+    const { err, python } = await findPython(f)
+    assert.strictEqual(err, null)
+    assert.strictEqual(python, '/path/python')
   })
 
-  it('find python - python too old', function () {
-    const f = new TestPythonFinder(null, done)
-    f.execFile = function (program, args, opts, cb) {
+  it('find python - python too old', async function () {
+    const f = new TestPythonFinder(null)
+    f.execFile = async function (program, args, opts) {
       if (/sys\.executable/.test(args[args.length - 1])) {
-        cb(null, '/path/python')
+        return [null, '/path/python']
       } else if (/sys\.version_info/.test(args[args.length - 1])) {
-        cb(null, '2.3.4')
+        return [null, '2.3.4']
       } else {
         assert.fail()
       }
     }
-    f.findPython()
 
-    function done (err) {
-      assert.ok(/Could not find any Python/.test(err))
-      assert.ok(/not supported/i.test(f.errorLog))
-    }
+    const { err } = await findPython(f)
+    assert.ok(/Could not find any Python/.test(err))
+    assert.ok(/not supported/i.test(f.errorLog))
   })
 
-  it('find python - no python', function () {
-    const f = new TestPythonFinder(null, done)
-    f.execFile = function (program, args, opts, cb) {
+  it('find python - no python', async function () {
+    const f = new TestPythonFinder(null)
+    f.execFile = async function (program, args, opts) {
       if (/sys\.executable/.test(args[args.length - 1])) {
-        cb(new Error('not found'))
+        throw new Error('not found')
       } else if (/sys\.version_info/.test(args[args.length - 1])) {
-        cb(new Error('not a Python executable'))
+        throw new Error('not a Python executable')
       } else {
         assert.fail()
       }
     }
-    f.findPython()
 
-    function done (err) {
-      assert.ok(/Could not find any Python/.test(err))
-      assert.ok(/not in PATH/.test(f.errorLog))
-    }
+    const { err } = await findPython(f)
+    assert.ok(/Could not find any Python/.test(err))
+    assert.ok(/not in PATH/.test(f.errorLog))
   })
 
-  it('find python - no python2, no python, unix', function () {
-    const f = new TestPythonFinder(null, done)
+  it('find python - no python2, no python, unix', async function () {
+    const f = new TestPythonFinder(null)
     f.checkPyLauncher = assert.fail
     f.win = false
 
-    f.execFile = function (program, args, opts, cb) {
+    f.execFile = async function (program, args, opts) {
       if (/sys\.executable/.test(args[args.length - 1])) {
-        cb(new Error('not found'))
+        throw new Error('not found')
       } else {
         assert.fail()
       }
     }
-    f.findPython()
 
-    function done (err) {
-      assert.ok(/Could not find any Python/.test(err))
-      assert.ok(/not in PATH/.test(f.errorLog))
-    }
+    const { err } = await findPython(f)
+    assert.ok(/Could not find any Python/.test(err))
+    assert.ok(/not in PATH/.test(f.errorLog))
   })
 
-  it('find python - no python, use python launcher', function () {
-    const f = new TestPythonFinder(null, done)
+  it('find python - no python, use python launcher', async function () {
+    const f = new TestPythonFinder(null)
     f.win = true
 
-    f.execFile = function (program, args, opts, cb) {
+    f.execFile = async function (program, args, opts) {
       if (program === 'py.exe') {
         assert.notStrictEqual(args.indexOf('-3'), -1)
         assert.notStrictEqual(args.indexOf('-c'), -1)
-        return cb(null, 'Z:\\snake.exe')
+        return [null, 'Z:\\snake.exe']
       }
       if (/sys\.executable/.test(args[args.length - 1])) {
-        cb(new Error('not found'))
+        throw new Error('not found')
       } else if (f.winDefaultLocations.includes(program)) {
-        cb(new Error('not found'))
+        throw new Error('not found')
       } else if (/sys\.version_info/.test(args[args.length - 1])) {
         if (program === 'Z:\\snake.exe') {
-          cb(null, '3.9.0')
+          return [null, '3.9.0']
         } else {
           assert.fail()
         }
@@ -146,58 +136,49 @@ describe('find-python', function () {
         assert.fail()
       }
     }
-    f.findPython()
-
-    function done (err, python) {
-      assert.strictEqual(err, null)
-      assert.strictEqual(python, 'Z:\\snake.exe')
-    }
+    const { err, python } = await findPython(f)
+    assert.strictEqual(err, null)
+    assert.strictEqual(python, 'Z:\\snake.exe')
   })
 
-  it('find python - no python, no python launcher, good guess', function () {
-    const f = new TestPythonFinder(null, done)
+  it('find python - no python, no python launcher, good guess', async function () {
+    const f = new TestPythonFinder(null)
     f.win = true
     const expectedProgram = f.winDefaultLocations[0]
 
-    f.execFile = function (program, args, opts, cb) {
+    f.execFile = async function (program, args, opts) {
       if (program === 'py.exe') {
-        return cb(new Error('not found'))
+        throw new Error('not found')
       }
       if (/sys\.executable/.test(args[args.length - 1])) {
-        cb(new Error('not found'))
+        throw new Error('not found')
       } else if (program === expectedProgram &&
                  /sys\.version_info/.test(args[args.length - 1])) {
-        cb(null, '3.7.3')
+        return [null, '3.7.3']
       } else {
         assert.fail()
       }
     }
-    f.findPython()
-
-    function done (err, python) {
-      assert.strictEqual(err, null)
-      assert.ok(python === expectedProgram)
-    }
+    const { err, python } = await findPython(f)
+    assert.strictEqual(err, null)
+    assert.ok(python === expectedProgram)
   })
 
-  it('find python - no python, no python launcher, bad guess', function () {
-    const f = new TestPythonFinder(null, done)
+  it('find python - no python, no python launcher, bad guess', async function () {
+    const f = new TestPythonFinder(null)
     f.win = true
 
-    f.execFile = function (program, args, opts, cb) {
+    f.execFile = async function (program, args, opts) {
       if (/sys\.executable/.test(args[args.length - 1])) {
-        cb(new Error('not found'))
+        throw new Error('not found')
       } else if (/sys\.version_info/.test(args[args.length - 1])) {
-        cb(new Error('not a Python executable'))
+        throw new Error('not a Python executable')
       } else {
         assert.fail()
       }
     }
-    f.findPython()
-
-    function done (err) {
-      assert.ok(/Could not find any Python/.test(err))
-      assert.ok(/not in PATH/.test(f.errorLog))
-    }
+    const { err } = await findPython(f)
+    assert.ok(/Could not find any Python/.test(err))
+    assert.ok(/not in PATH/.test(f.errorLog))
   })
 })
diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js
index def1fc81f7..cb065e52cf 100644
--- a/test/test-find-visualstudio.js
+++ b/test/test-find-visualstudio.js
@@ -4,8 +4,7 @@ const { describe, it } = require('mocha')
 const assert = require('assert')
 const fs = require('fs')
 const path = require('path')
-const findVisualStudio = require('../lib/find-visualstudio')
-const VisualStudioFinder = findVisualStudio.test.VisualStudioFinder
+const { test: { VisualStudioFinder } } = require('../lib/find-visualstudio')
 
 const semverV1 = { major: 1, minor: 0, patch: 0 }
 
@@ -28,26 +27,22 @@ function poison (object, property) {
 function TestVisualStudioFinder () { VisualStudioFinder.apply(this, arguments) }
 TestVisualStudioFinder.prototype = Object.create(VisualStudioFinder.prototype)
 
+const findVisualStudio = async (finder) => {
+  try {
+    return { err: null, info: await finder.findVisualStudio() }
+  } catch (err) {
+    return { err, info: null }
+  }
+}
+
 describe('find-visualstudio', function () {
-  it('VS2013', function () {
-    const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => {
-      assert.strictEqual(err, null)
-      assert.deepStrictEqual(info, {
-        msBuild: 'C:\\MSBuild12\\MSBuild.exe',
-        path: 'C:\\VS2013',
-        sdk: null,
-        toolset: 'v120',
-        version: '12.0',
-        versionMajor: 12,
-        versionMinor: 0,
-        versionYear: 2013
-      })
-    })
+  it('VS2013', async function () {
+    const finder = new TestVisualStudioFinder(semverV1, null)
 
-    finder.findVisualStudio2017OrNewer = (cb) => {
-      finder.parseData(new Error(), '', '', cb)
+    finder.findVisualStudio2017OrNewer = async () => {
+      return finder.parseData(new Error(), '', '')
     }
-    finder.regSearchKeys = (keys, value, addOpts, cb) => {
+    finder.regSearchKeys = async (keys, value, addOpts) => {
       for (let i = 0; i < keys.length; ++i) {
         const fullName = `${keys[i]}\\${value}`
         switch (fullName) {
@@ -56,35 +51,44 @@ describe('find-visualstudio', function () {
             continue
           case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\12.0':
             assert.ok(true, `expected search for registry value ${fullName}`)
-            return cb(null, 'C:\\VS2013\\VC\\')
+            return 'C:\\VS2013\\VC\\'
           case 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions\\12.0\\MSBuildToolsPath':
             assert.ok(true, `expected search for registry value ${fullName}`)
-            return cb(null, 'C:\\MSBuild12\\')
+            return 'C:\\MSBuild12\\'
           default:
             assert.fail(`unexpected search for registry value ${fullName}`)
         }
       }
-      return cb(new Error())
+      throw new Error()
     }
-    finder.findVisualStudio()
+
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info, {
+      msBuild: 'C:\\MSBuild12\\MSBuild.exe',
+      path: 'C:\\VS2013',
+      sdk: null,
+      toolset: 'v120',
+      version: '12.0',
+      versionMajor: 12,
+      versionMinor: 0,
+      versionYear: 2013
+    })
   })
 
-  it('VS2013 should not be found on new node versions', function () {
+  it('VS2013 should not be found on new node versions', async function () {
     const finder = new TestVisualStudioFinder({
       major: 10,
       minor: 0,
       patch: 0
-    }, null, (err, info) => {
-      assert.ok(/find .* Visual Studio/i.test(err), 'expect error')
-      assert.ok(!info, 'no data')
-    })
+    }, null)
 
-    finder.findVisualStudio2017OrNewer = (cb) => {
+    finder.findVisualStudio2017OrNewer = async () => {
       const file = path.join(__dirname, 'fixtures', 'VS_2017_Unusable.txt')
       const data = fs.readFileSync(file)
-      finder.parseData(null, data, '', cb)
+      return finder.parseData(null, data, '')
     }
-    finder.regSearchKeys = (keys, value, addOpts, cb) => {
+    finder.regSearchKeys = async (keys, value, addOpts) => {
       for (let i = 0; i < keys.length; ++i) {
         const fullName = `${keys[i]}\\${value}`
         switch (fullName) {
@@ -95,49 +99,51 @@ describe('find-visualstudio', function () {
             assert.fail(`unexpected search for registry value ${fullName}`)
         }
       }
-      return cb(new Error())
+      throw new Error()
     }
-    finder.findVisualStudio()
-  })
-
-  it('VS2015', function () {
-    const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => {
-      assert.strictEqual(err, null)
-      assert.deepStrictEqual(info, {
-        msBuild: 'C:\\MSBuild14\\MSBuild.exe',
-        path: 'C:\\VS2015',
-        sdk: null,
-        toolset: 'v140',
-        version: '14.0',
-        versionMajor: 14,
-        versionMinor: 0,
-        versionYear: 2015
-      })
-    })
 
-    finder.findVisualStudio2017OrNewer = (cb) => {
-      finder.parseData(new Error(), '', '', cb)
+    const { err, info } = await findVisualStudio(finder)
+    assert.ok(/find .* Visual Studio/i.test(err), 'expect error')
+    assert.ok(!info, 'no data')
+  })
+
+  it('VS2015', async function () {
+    const finder = new TestVisualStudioFinder(semverV1, null)
+
+    finder.findVisualStudio2017OrNewer = async () => {
+      return finder.parseData(new Error(), '', '')
     }
-    finder.regSearchKeys = (keys, value, addOpts, cb) => {
+    finder.regSearchKeys = async (keys, value, addOpts) => {
       for (let i = 0; i < keys.length; ++i) {
         const fullName = `${keys[i]}\\${value}`
         switch (fullName) {
           case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0':
             assert.ok(true, `expected search for registry value ${fullName}`)
-            return cb(null, 'C:\\VS2015\\VC\\')
+            return 'C:\\VS2015\\VC\\'
           case 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions\\14.0\\MSBuildToolsPath':
             assert.ok(true, `expected search for registry value ${fullName}`)
-            return cb(null, 'C:\\MSBuild14\\')
+            return 'C:\\MSBuild14\\'
           default:
             assert.fail(`unexpected search for registry value ${fullName}`)
         }
       }
-      return cb(new Error())
+      throw new Error()
     }
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info, {
+      msBuild: 'C:\\MSBuild14\\MSBuild.exe',
+      path: 'C:\\VS2015',
+      sdk: null,
+      toolset: 'v140',
+      version: '14.0',
+      versionMajor: 14,
+      versionMinor: 0,
+      versionYear: 2015
+    })
   })
 
-  it('error from PowerShell', function () {
+  it('error from PowerShell', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null, null)
 
     finder.parseData(new Error(), '', '', (info) => {
@@ -146,7 +152,7 @@ describe('find-visualstudio', function () {
     })
   })
 
-  it('empty output from PowerShell', function () {
+  it('empty output from PowerShell', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null, null)
 
     finder.parseData(null, '', '', (info) => {
@@ -155,7 +161,7 @@ describe('find-visualstudio', function () {
     })
   })
 
-  it('output from PowerShell not JSON', function () {
+  it('output from PowerShell not JSON', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null, null)
 
     finder.parseData(null, 'AAAABBBB', '', (info) => {
@@ -164,7 +170,7 @@ describe('find-visualstudio', function () {
     })
   })
 
-  it('wrong JSON from PowerShell', function () {
+  it('wrong JSON from PowerShell', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null, null)
 
     finder.parseData(null, '{}', '', (info) => {
@@ -173,7 +179,7 @@ describe('find-visualstudio', function () {
     })
   })
 
-  it('empty JSON from PowerShell', function () {
+  it('empty JSON from PowerShell', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null, null)
 
     finder.parseData(null, '[]', '', (info) => {
@@ -182,7 +188,7 @@ describe('find-visualstudio', function () {
     })
   })
 
-  it('future version', function () {
+  it('future version', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null, null)
 
     finder.parseData(null, JSON.stringify([{
@@ -200,7 +206,7 @@ describe('find-visualstudio', function () {
     })
   })
 
-  it('single unusable VS2017', function () {
+  it('single unusable VS2017', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null, null)
 
     const file = path.join(__dirname, 'fixtures', 'VS_2017_Unusable.txt')
@@ -212,204 +218,197 @@ describe('find-visualstudio', function () {
     })
   })
 
-  it('minimal VS2017 Build Tools', function () {
-    const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => {
-      assert.strictEqual(err, null)
-      assert.deepStrictEqual(info, {
-        msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\' +
-          'BuildTools\\MSBuild\\15.0\\Bin\\MSBuild.exe',
-        path:
-          'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTools',
-        sdk: '10.0.17134.0',
-        toolset: 'v141',
-        version: '15.9.28307.665',
-        versionMajor: 15,
-        versionMinor: 9,
-        versionYear: 2017
-      })
-    })
+  it('minimal VS2017 Build Tools', async function () {
+    const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
-    finder.findVisualStudio2017OrNewer = (cb) => {
+    finder.findVisualStudio2017OrNewer = async () => {
       const file = path.join(__dirname, 'fixtures',
         'VS_2017_BuildTools_minimal.txt')
       const data = fs.readFileSync(file)
-      finder.parseData(null, data, '', cb)
+      return finder.parseData(null, data, '')
     }
-    finder.findVisualStudio()
-  })
-
-  it('VS2017 Community with C++ workload', function () {
-    const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => {
-      assert.strictEqual(err, null)
-      assert.deepStrictEqual(info, {
-        msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\' +
-          'Community\\MSBuild\\15.0\\Bin\\MSBuild.exe',
-        path:
-          'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community',
-        sdk: '10.0.17763.0',
-        toolset: 'v141',
-        version: '15.9.28307.665',
-        versionMajor: 15,
-        versionMinor: 9,
-        versionYear: 2017
-      })
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info, {
+      msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\' +
+        'BuildTools\\MSBuild\\15.0\\Bin\\MSBuild.exe',
+      path:
+        'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTools',
+      sdk: '10.0.17134.0',
+      toolset: 'v141',
+      version: '15.9.28307.665',
+      versionMajor: 15,
+      versionMinor: 9,
+      versionYear: 2017
     })
+  })
+
+  it('VS2017 Community with C++ workload', async function () {
+    const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
-    finder.findVisualStudio2017OrNewer = (cb) => {
+    finder.findVisualStudio2017OrNewer = async () => {
       const file = path.join(__dirname, 'fixtures',
         'VS_2017_Community_workload.txt')
       const data = fs.readFileSync(file)
-      finder.parseData(null, data, '', cb)
+      return finder.parseData(null, data, '')
     }
-    finder.findVisualStudio()
-  })
-
-  it('VS2017 Express', function () {
-    const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => {
-      assert.strictEqual(err, null)
-      assert.deepStrictEqual(info, {
-        msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\' +
-          'WDExpress\\MSBuild\\15.0\\Bin\\MSBuild.exe',
-        path:
-          'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\WDExpress',
-        sdk: '10.0.17763.0',
-        toolset: 'v141',
-        version: '15.9.28307.858',
-        versionMajor: 15,
-        versionMinor: 9,
-        versionYear: 2017
-      })
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info, {
+      msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\' +
+        'Community\\MSBuild\\15.0\\Bin\\MSBuild.exe',
+      path:
+        'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community',
+      sdk: '10.0.17763.0',
+      toolset: 'v141',
+      version: '15.9.28307.665',
+      versionMajor: 15,
+      versionMinor: 9,
+      versionYear: 2017
     })
+  })
+
+  it('VS2017 Express', async function () {
+    const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
-    finder.findVisualStudio2017OrNewer = (cb) => {
+    finder.findVisualStudio2017OrNewer = async () => {
       const file = path.join(__dirname, 'fixtures', 'VS_2017_Express.txt')
       const data = fs.readFileSync(file)
-      finder.parseData(null, data, '', cb)
+      return finder.parseData(null, data, '')
     }
-    finder.findVisualStudio()
-  })
-
-  it('VS2019 Preview with C++ workload', function () {
-    const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => {
-      assert.strictEqual(err, null)
-      assert.deepStrictEqual(info, {
-        msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' +
-          'Preview\\MSBuild\\Current\\Bin\\MSBuild.exe',
-        path:
-          'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Preview',
-        sdk: '10.0.17763.0',
-        toolset: 'v142',
-        version: '16.0.28608.199',
-        versionMajor: 16,
-        versionMinor: 0,
-        versionYear: 2019
-      })
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info, {
+      msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\' +
+        'WDExpress\\MSBuild\\15.0\\Bin\\MSBuild.exe',
+      path:
+        'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\WDExpress',
+      sdk: '10.0.17763.0',
+      toolset: 'v141',
+      version: '15.9.28307.858',
+      versionMajor: 15,
+      versionMinor: 9,
+      versionYear: 2017
     })
+  })
+
+  it('VS2019 Preview with C++ workload', async function () {
+    const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
-    finder.findVisualStudio2017OrNewer = (cb) => {
+    finder.findVisualStudio2017OrNewer = async () => {
       const file = path.join(__dirname, 'fixtures',
         'VS_2019_Preview.txt')
       const data = fs.readFileSync(file)
-      finder.parseData(null, data, '', cb)
+      return finder.parseData(null, data, '')
     }
-    finder.findVisualStudio()
-  })
-
-  it('minimal VS2019 Build Tools', function () {
-    const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => {
-      assert.strictEqual(err, null)
-      assert.deepStrictEqual(info, {
-        msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' +
-          'BuildTools\\MSBuild\\Current\\Bin\\MSBuild.exe',
-        path:
-          'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools',
-        sdk: '10.0.17134.0',
-        toolset: 'v142',
-        version: '16.1.28922.388',
-        versionMajor: 16,
-        versionMinor: 1,
-        versionYear: 2019
-      })
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info, {
+      msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' +
+        'Preview\\MSBuild\\Current\\Bin\\MSBuild.exe',
+      path:
+        'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Preview',
+      sdk: '10.0.17763.0',
+      toolset: 'v142',
+      version: '16.0.28608.199',
+      versionMajor: 16,
+      versionMinor: 0,
+      versionYear: 2019
     })
+  })
+
+  it('minimal VS2019 Build Tools', async function () {
+    const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
-    finder.findVisualStudio2017OrNewer = (cb) => {
+    finder.findVisualStudio2017OrNewer = async () => {
       const file = path.join(__dirname, 'fixtures',
         'VS_2019_BuildTools_minimal.txt')
       const data = fs.readFileSync(file)
-      finder.parseData(null, data, '', cb)
+      return finder.parseData(null, data, '')
     }
-    finder.findVisualStudio()
-  })
-
-  it('VS2019 Community with C++ workload', function () {
-    const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => {
-      assert.strictEqual(err, null)
-      assert.deepStrictEqual(info, {
-        msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' +
-          'Community\\MSBuild\\Current\\Bin\\MSBuild.exe',
-        path:
-          'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community',
-        sdk: '10.0.17763.0',
-        toolset: 'v142',
-        version: '16.1.28922.388',
-        versionMajor: 16,
-        versionMinor: 1,
-        versionYear: 2019
-      })
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info, {
+      msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' +
+        'BuildTools\\MSBuild\\Current\\Bin\\MSBuild.exe',
+      path:
+        'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools',
+      sdk: '10.0.17134.0',
+      toolset: 'v142',
+      version: '16.1.28922.388',
+      versionMajor: 16,
+      versionMinor: 1,
+      versionYear: 2019
     })
+  })
+
+  it('VS2019 Community with C++ workload', async function () {
+    const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
-    finder.findVisualStudio2017OrNewer = (cb) => {
+    finder.findVisualStudio2017OrNewer = async () => {
       const file = path.join(__dirname, 'fixtures',
         'VS_2019_Community_workload.txt')
       const data = fs.readFileSync(file)
-      finder.parseData(null, data, '', cb)
+      return finder.parseData(null, data, '')
     }
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info, {
+      msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' +
+        'Community\\MSBuild\\Current\\Bin\\MSBuild.exe',
+      path:
+        'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community',
+      sdk: '10.0.17763.0',
+      toolset: 'v142',
+      version: '16.1.28922.388',
+      versionMajor: 16,
+      versionMinor: 1,
+      versionYear: 2019
+    })
   })
 
-  it('VS2022 Preview with C++ workload', function () {
+  it('VS2022 Preview with C++ workload', async function () {
     const msBuildPath = process.arch === 'arm64'
       ? 'C:\\Program Files\\Microsoft Visual Studio\\2022\\' +
         'Community\\MSBuild\\Current\\Bin\\arm64\\MSBuild.exe'
       : 'C:\\Program Files\\Microsoft Visual Studio\\2022\\' +
         'Community\\MSBuild\\Current\\Bin\\MSBuild.exe'
 
-    const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => {
-      assert.strictEqual(err, null)
-      assert.deepStrictEqual(info, {
-        msBuild: msBuildPath,
-        path:
-          'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community',
-        sdk: '10.0.22621.0',
-        toolset: 'v143',
-        version: '17.4.33213.308',
-        versionMajor: 17,
-        versionMinor: 4,
-        versionYear: 2022
-      })
-    })
+    const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
     finder.msBuildPathExists = (path) => {
       return true
     }
-    finder.findVisualStudio2017OrNewer = (cb) => {
+    finder.findVisualStudio2017OrNewer = async () => {
       const file = path.join(__dirname, 'fixtures',
         'VS_2022_Community_workload.txt')
       const data = fs.readFileSync(file)
-      finder.parseData(null, data, '', cb)
+      return finder.parseData(null, data, '')
     }
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info, {
+      msBuild: msBuildPath,
+      path:
+        'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community',
+      sdk: '10.0.22621.0',
+      toolset: 'v143',
+      version: '17.4.33213.308',
+      versionMajor: 17,
+      versionMinor: 4,
+      versionYear: 2022
+    })
   })
 
   function allVsVersions (finder) {
-    finder.findVisualStudio2017OrNewer = (cb) => {
+    finder.findVisualStudio2017OrNewer = async () => {
       const data0 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures',
         'VS_2017_Unusable.txt')))
       const data1 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures',
@@ -428,9 +427,9 @@ describe('find-visualstudio', function () {
         'VS_2022_Community_workload.txt')))
       const data = JSON.stringify(data0.concat(data1, data2, data3, data4,
         data5, data6, data7))
-      finder.parseData(null, data, '', cb)
+      return finder.parseData(null, data, '')
     }
-    finder.regSearchKeys = (keys, value, addOpts, cb) => {
+    finder.regSearchKeys = async (keys, value, addOpts) => {
       for (let i = 0; i < keys.length; ++i) {
         const fullName = `${keys[i]}\\${value}`
         switch (fullName) {
@@ -438,225 +437,201 @@ describe('find-visualstudio', function () {
           case 'HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7\\12.0':
             continue
           case 'HKLM\\Software\\Wow6432Node\\Microsoft\\VisualStudio\\SxS\\VC7\\12.0':
-            return cb(null, 'C:\\VS2013\\VC\\')
+            return 'C:\\VS2013\\VC\\'
           case 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions\\12.0\\MSBuildToolsPath':
-            return cb(null, 'C:\\MSBuild12\\')
+            return 'C:\\MSBuild12\\'
           case 'HKLM\\Software\\Wow6432Node\\Microsoft\\VisualStudio\\SxS\\VC7\\14.0':
-            return cb(null, 'C:\\VS2015\\VC\\')
+            return 'C:\\VS2015\\VC\\'
           case 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions\\14.0\\MSBuildToolsPath':
-            return cb(null, 'C:\\MSBuild14\\')
+            return 'C:\\MSBuild14\\'
           default:
             assert.fail(`unexpected search for registry value ${fullName}`)
         }
       }
-      return cb(new Error())
+      throw new Error()
     }
   }
 
-  it('fail when looking for invalid path', function () {
-    const finder = new TestVisualStudioFinder(semverV1, 'AABB', (err, info) => {
-      assert.ok(/find .* Visual Studio/i.test(err), 'expect error')
-      assert.ok(!info, 'no data')
-    })
+  it('fail when looking for invalid path', async function () {
+    const finder = new TestVisualStudioFinder(semverV1, 'AABB')
 
     allVsVersions(finder)
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.ok(/find .* Visual Studio/i.test(err), 'expect error')
+    assert.ok(!info, 'no data')
   })
 
-  it('look for VS2013 by version number', function () {
-    const finder = new TestVisualStudioFinder(semverV1, '2013', (err, info) => {
-      assert.strictEqual(err, null)
-      assert.deepStrictEqual(info.versionYear, 2013)
-    })
+  it('look for VS2013 by version number', async function () {
+    const finder = new TestVisualStudioFinder(semverV1, '2013')
 
     allVsVersions(finder)
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info.versionYear, 2013)
   })
 
-  it('look for VS2013 by installation path', function () {
-    const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2013',
-      (err, info) => {
-        assert.strictEqual(err, null)
-        assert.deepStrictEqual(info.path, 'C:\\VS2013')
-      })
+  it('look for VS2013 by installation path', async function () {
+    const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2013')
 
     allVsVersions(finder)
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info.path, 'C:\\VS2013')
   })
 
-  it('look for VS2015 by version number', function () {
-    const finder = new TestVisualStudioFinder(semverV1, '2015', (err, info) => {
-      assert.strictEqual(err, null)
-      assert.deepStrictEqual(info.versionYear, 2015)
-    })
+  it('look for VS2015 by version number', async function () {
+    const finder = new TestVisualStudioFinder(semverV1, '2015')
 
     allVsVersions(finder)
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info.versionYear, 2015)
   })
 
-  it('look for VS2015 by installation path', function () {
-    const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2015',
-      (err, info) => {
-        assert.strictEqual(err, null)
-        assert.deepStrictEqual(info.path, 'C:\\VS2015')
-      })
+  it('look for VS2015 by installation path', async function () {
+    const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2015')
 
     allVsVersions(finder)
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info.path, 'C:\\VS2015')
   })
 
-  it('look for VS2017 by version number', function () {
-    const finder = new TestVisualStudioFinder(semverV1, '2017', (err, info) => {
-      assert.strictEqual(err, null)
-      assert.deepStrictEqual(info.versionYear, 2017)
-    })
+  it('look for VS2017 by version number', async function () {
+    const finder = new TestVisualStudioFinder(semverV1, '2017')
 
     allVsVersions(finder)
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info.versionYear, 2017)
   })
 
-  it('look for VS2017 by installation path', function () {
+  it('look for VS2017 by installation path', async function () {
     const finder = new TestVisualStudioFinder(semverV1,
-      'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community',
-      (err, info) => {
-        assert.strictEqual(err, null)
-        assert.deepStrictEqual(info.path,
-          'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community')
-      })
+      'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community')
 
     allVsVersions(finder)
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info.path,
+      'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community')
   })
 
-  it('look for VS2019 by version number', function () {
-    const finder = new TestVisualStudioFinder(semverV1, '2019', (err, info) => {
-      assert.strictEqual(err, null)
-      assert.deepStrictEqual(info.versionYear, 2019)
-    })
+  it('look for VS2019 by version number', async function () {
+    const finder = new TestVisualStudioFinder(semverV1, '2019')
 
     allVsVersions(finder)
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info.versionYear, 2019)
   })
 
-  it('look for VS2019 by installation path', function () {
+  it('look for VS2019 by installation path', async function () {
     const finder = new TestVisualStudioFinder(semverV1,
-      'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools',
-      (err, info) => {
-        assert.strictEqual(err, null)
-        assert.deepStrictEqual(info.path,
-          'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools')
-      })
+      'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools')
 
     allVsVersions(finder)
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info.path,
+      'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools')
   })
 
-  it('look for VS2022 by version number', function () {
-    const finder = new TestVisualStudioFinder(semverV1, '2022', (err, info) => {
-      assert.strictEqual(err, null)
-      assert.deepStrictEqual(info.versionYear, 2022)
-    })
+  it('look for VS2022 by version number', async function () {
+    const finder = new TestVisualStudioFinder(semverV1, '2022')
 
     finder.msBuildPathExists = (path) => {
       return true
     }
 
     allVsVersions(finder)
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info.versionYear, 2022)
   })
 
-  it('msvs_version match should be case insensitive', function () {
+  it('msvs_version match should be case insensitive', async function () {
     const finder = new TestVisualStudioFinder(semverV1,
-      'c:\\program files (x86)\\microsoft visual studio\\2019\\BUILDTOOLS',
-      (err, info) => {
-        assert.strictEqual(err, null)
-        assert.deepStrictEqual(info.path,
-          'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools')
-      })
+      'c:\\program files (x86)\\microsoft visual studio\\2019\\BUILDTOOLS')
 
     allVsVersions(finder)
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info.path,
+      'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools')
   })
 
-  it('latest version should be found by default', function () {
-    const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => {
-      assert.strictEqual(err, null)
-      assert.deepStrictEqual(info.versionYear, 2022)
-    })
+  it('latest version should be found by default', async function () {
+    const finder = new TestVisualStudioFinder(semverV1, null)
 
     finder.msBuildPathExists = (path) => {
       return true
     }
 
     allVsVersions(finder)
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info.versionYear, 2022)
   })
 
-  it('run on a usable VS Command Prompt', function () {
+  it('run on a usable VS Command Prompt', async function () {
     process.env.VCINSTALLDIR = 'C:\\VS2015\\VC'
     // VSINSTALLDIR is not defined on Visual C++ Build Tools 2015
     delete process.env.VSINSTALLDIR
 
-    const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => {
-      assert.strictEqual(err, null)
-      assert.deepStrictEqual(info.path, 'C:\\VS2015')
-    })
+    const finder = new TestVisualStudioFinder(semverV1, null)
 
     allVsVersions(finder)
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info.path, 'C:\\VS2015')
   })
 
-  it('VCINSTALLDIR match should be case insensitive', function () {
+  it('VCINSTALLDIR match should be case insensitive', async function () {
     process.env.VCINSTALLDIR =
       'c:\\program files (x86)\\microsoft visual studio\\2019\\BUILDTOOLS\\VC'
 
-    const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => {
-      assert.strictEqual(err, null)
-      assert.deepStrictEqual(info.path,
-        'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools')
-    })
+    const finder = new TestVisualStudioFinder(semverV1, null)
 
     allVsVersions(finder)
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info.path,
+      'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools')
   })
 
-  it('run on a unusable VS Command Prompt', function () {
+  it('run on a unusable VS Command Prompt', async function () {
     process.env.VCINSTALLDIR =
       'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildToolsUnusable\\VC'
 
-    const finder = new TestVisualStudioFinder(semverV1, null, (err, info) => {
-      assert.ok(/find .* Visual Studio/i.test(err), 'expect error')
-      assert.ok(!info, 'no data')
-    })
+    const finder = new TestVisualStudioFinder(semverV1, null)
 
     allVsVersions(finder)
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.ok(/find .* Visual Studio/i.test(err), 'expect error')
+    assert.ok(!info, 'no data')
   })
 
-  it('run on a VS Command Prompt with matching msvs_version', function () {
+  it('run on a VS Command Prompt with matching msvs_version', async function () {
     process.env.VCINSTALLDIR = 'C:\\VS2015\\VC'
 
-    const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2015',
-      (err, info) => {
-        assert.strictEqual(err, null)
-        assert.deepStrictEqual(info.path, 'C:\\VS2015')
-      })
+    const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2015')
 
     allVsVersions(finder)
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info.path, 'C:\\VS2015')
   })
 
-  it('run on a VS Command Prompt with mismatched msvs_version', function () {
+  it('run on a VS Command Prompt with mismatched msvs_version', async function () {
     process.env.VCINSTALLDIR =
       'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC'
 
-    const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2015',
-      (err, info) => {
-        assert.ok(/find .* Visual Studio/i.test(err), 'expect error')
-        assert.ok(!info, 'no data')
-      })
+    const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2015')
 
     allVsVersions(finder)
-    finder.findVisualStudio()
+    const { err, info } = await findVisualStudio(finder)
+    assert.ok(/find .* Visual Studio/i.test(err), 'expect error')
+    assert.ok(!info, 'no data')
   })
 })
diff --git a/test/test-install.js b/test/test-install.js
index af83ad9442..3fc39b7c2f 100644
--- a/test/test-install.js
+++ b/test/test-install.js
@@ -1,52 +1,56 @@
 'use strict'
 
 const { describe, it, after } = require('mocha')
-const { rm } = require('fs/promises')
+const { rm, mkdtemp } = require('fs/promises')
+const { createWriteStream } = require('fs')
 const assert = require('assert')
 const path = require('path')
 const os = require('os')
-const util = require('util')
-const { test: { download, install } } = require('../lib/install')
-const gyp = require('../lib/node-gyp')
 const semver = require('semver')
-const stream = require('stream')
-const streamPipeline = util.promisify(stream.pipeline)
+const { pipeline: streamPipeline } = require('stream/promises')
+const requireInject = require('require-inject')
+const gyp = require('../lib/node-gyp')
+
+const createInstall = (mocks = {}) => requireInject('../lib/install', mocks).test
+const { download, install } = createInstall()
 
 describe('install', function () {
   it('EACCES retry once', async () => {
-    const fs = {
-      promises: {
-        stat (_) {
-          const err = new Error()
-          err.code = 'EACCES'
-          assert.ok(true)
-          throw err
+    let statCalled = 0
+    const mockInstall = createInstall({
+      'graceful-fs': {
+        promises: {
+          stat (_) {
+            const err = new Error()
+            err.code = 'EACCES'
+            statCalled++
+            throw err
+          }
         }
       }
-    }
-
+    })
     const Gyp = {
       devDir: __dirname,
       opts: {
         ensure: true
       },
       commands: {
-        install (argv, cb) {
-          install(fs, Gyp, argv).then(cb, cb)
-        },
-        remove (_, cb) {
-          cb()
-        }
+        install: (...args) => mockInstall.install(Gyp, ...args),
+        remove: async () => {}
       }
     }
 
+    let err
     try {
-      await install(fs, Gyp, [])
-    } catch (err) {
+      await Gyp.commands.install([])
+    } catch (e) {
+      err = e
+    }
+
+    assert.ok(err)
+    assert.equal(statCalled, 2)
+    if (/"pre" versions of node cannot be installed/.test(err.message)) {
       assert.ok(true)
-      if (/"pre" versions of node cannot be installed/.test(err.message)) {
-        assert.ok(true)
-      }
     }
   })
 
@@ -56,7 +60,7 @@ describe('install', function () {
     semver.prerelease(process.version) !== null ||
     semver.satisfies(process.version, '<10')
 
-  async function parallelInstallsTest (test, fs, devDir, prog) {
+  async function parallelInstallsTest (test, devDir, prog) {
     if (skipParallelInstallTests) {
       return test.skip('Skipping parallel installs test due to test environment configuration')
     }
@@ -69,52 +73,49 @@ describe('install', function () {
     await rm(expectedDir, { recursive: true, force: true })
 
     await Promise.all([
-      install(fs, prog, []),
-      install(fs, prog, []),
-      install(fs, prog, []),
-      install(fs, prog, []),
-      install(fs, prog, []),
-      install(fs, prog, []),
-      install(fs, prog, []),
-      install(fs, prog, []),
-      install(fs, prog, []),
-      install(fs, prog, [])
+      install(prog, []),
+      install(prog, []),
+      install(prog, []),
+      install(prog, []),
+      install(prog, []),
+      install(prog, []),
+      install(prog, []),
+      install(prog, []),
+      install(prog, []),
+      install(prog, [])
     ])
   }
 
   it('parallel installs (ensure=true)', async function () {
     this.timeout(600000)
 
-    const fs = require('graceful-fs')
-    const devDir = await util.promisify(fs.mkdtemp)(path.join(os.tmpdir(), 'node-gyp-test-'))
+    const devDir = await mkdtemp(path.join(os.tmpdir(), 'node-gyp-test-'))
 
     const prog = gyp()
     prog.parseArgv([])
     prog.devDir = devDir
     prog.opts.ensure = true
 
-    await parallelInstallsTest(this, fs, devDir, prog)
+    await parallelInstallsTest(this, devDir, prog)
   })
 
   it('parallel installs (ensure=false)', async function () {
     this.timeout(600000)
 
-    const fs = require('graceful-fs')
-    const devDir = await util.promisify(fs.mkdtemp)(path.join(os.tmpdir(), 'node-gyp-test-'))
+    const devDir = await mkdtemp(path.join(os.tmpdir(), 'node-gyp-test-'))
 
     const prog = gyp()
     prog.parseArgv([])
     prog.devDir = devDir
     prog.opts.ensure = false
 
-    await parallelInstallsTest(this, fs, devDir, prog)
+    await parallelInstallsTest(this, devDir, prog)
   })
 
   it('parallel installs (tarball)', async function () {
     this.timeout(600000)
 
-    const fs = require('graceful-fs')
-    const devDir = await util.promisify(fs.mkdtemp)(path.join(os.tmpdir(), 'node-gyp-test-'))
+    const devDir = await mkdtemp(path.join(os.tmpdir(), 'node-gyp-test-'))
 
     const prog = gyp()
     prog.parseArgv([])
@@ -123,9 +124,9 @@ describe('install', function () {
 
     await streamPipeline(
       (await download(prog, `https://nodejs.org/dist/${process.version}/node-${process.version}.tar.gz`)).body,
-      fs.createWriteStream(prog.opts.tarball)
+      createWriteStream(prog.opts.tarball)
     )
 
-    await parallelInstallsTest(this, fs, devDir, prog)
+    await parallelInstallsTest(this, devDir, prog)
   })
 })

From d52997e975b9da6e0cea3d9b99873e9ddc768679 Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Fri, 27 Oct 2023 16:40:36 -0700
Subject: [PATCH 396/551] feat: convert internal classes from util.inherits to
 classes

BREAKING CHANGE: the `Gyp` class exported is now created using
ECMAScript classes and therefore might have small differences to classes
that were previously created with `util.inherits`.
---
 lib/configure.js               |   4 +-
 lib/find-python.js             |  83 +++++----
 lib/find-visualstudio.js       |  86 +++++----
 lib/node-gyp.js                | 311 ++++++++++++++++-----------------
 test/common.js                 |  14 ++
 test/test-find-python.js       |  59 +++----
 test/test-find-visualstudio.js |  86 ++++-----
 7 files changed, 303 insertions(+), 340 deletions(-)

diff --git a/lib/configure.js b/lib/configure.js
index 8ccfc3fad5..8d59e33497 100644
--- a/lib/configure.js
+++ b/lib/configure.js
@@ -9,8 +9,8 @@ const win = process.platform === 'win32'
 const findNodeDirectory = require('./find-node-directory')
 const createConfigGypi = require('./create-config-gypi')
 const { format: msgFormat } = require('util')
-const findPython = require('./find-python')
-const findVisualStudio = win ? require('./find-visualstudio') : null
+const { findPython } = require('./find-python')
+const { findVisualStudio } = win ? require('./find-visualstudio') : {}
 
 async function configure (gyp, argv) {
   const buildDir = path.resolve('build')
diff --git a/lib/find-python.js b/lib/find-python.js
index f01d1dd454..615da57bb8 100644
--- a/lib/find-python.js
+++ b/lib/find-python.js
@@ -2,10 +2,15 @@
 
 const log = require('./log')
 const semver = require('semver')
-const { _extend: extend } = require('util') // eslint-disable-line n/no-deprecated-api
 const { execFile } = require('./util')
 const win = process.platform === 'win32'
 
+function getOsUserInfo () {
+  try {
+    return require('os').userInfo().username
+  } catch {}
+}
+
 const systemDrive = process.env.SystemDrive || 'C:'
 const username = process.env.USERNAME || process.env.USER || getOsUserInfo()
 const localAppData = process.env.LOCALAPPDATA || `${systemDrive}\\${username}\\AppData\\Local`
@@ -32,40 +37,36 @@ for (const majorMinor of ['311', '310', '39', '38']) {
   }
 }
 
-function getOsUserInfo () {
-  try {
-    return require('os').userInfo().username
-  } catch (e) {}
-}
-
-function PythonFinder (configPython) {
-  this.configPython = configPython
-  this.errorLog = []
-}
+class PythonFinder {
+  static findPython = (...args) => new PythonFinder(...args).findPython()
 
-PythonFinder.prototype = {
-  log: log.withPrefix('find Python'),
-  argsExecutable: ['-c', 'import sys; print(sys.executable);'],
-  argsVersion: ['-c', 'import sys; print("%s.%s.%s" % sys.version_info[:3]);'],
-  semverRange: '>=3.6.0',
+  log = log.withPrefix('find Python')
+  argsExecutable = ['-c', 'import sys; print(sys.executable);']
+  argsVersion = ['-c', 'import sys; print("%s.%s.%s" % sys.version_info[:3]);']
+  semverRange = '>=3.6.0'
 
   // These can be overridden for testing:
-  execFile,
-  env: process.env,
-  win,
-  pyLauncher: 'py.exe',
-  winDefaultLocations: winDefaultLocationsArray,
+  execFile = execFile
+  env = process.env
+  win = win
+  pyLauncher = 'py.exe'
+  winDefaultLocations = winDefaultLocationsArray
+
+  constructor (configPython) {
+    this.configPython = configPython
+    this.errorLog = []
+  }
 
   // Logs a message at verbose level, but also saves it to be displayed later
   // at error level if an error occurs. This should help diagnose the problem.
-  addLog: function addLog (message) {
+  addLog (message) {
     this.log.verbose(message)
     this.errorLog.push(message)
-  },
+  }
 
   // Find Python by trying a sequence of possibilities.
   // Ignore errors, keep trying until Python is found.
-  findPython: async function findPython () {
+  async findPython () {
     const SKIP = 0
     const FAIL = 1
     const toCheck = (() => {
@@ -161,12 +162,12 @@ PythonFinder.prototype = {
     }
 
     return this.fail()
-  },
+  }
 
   // Check if command is a valid Python to use.
   // Will exit the Python finder on success.
   // If on Windows, run in a CMD shell to support BAT/CMD launchers.
-  checkCommand: async function checkCommand (command) {
+  async checkCommand (command) {
     let exec = command
     let args = this.argsExecutable
     let shell = false
@@ -190,7 +191,7 @@ PythonFinder.prototype = {
       this.addLog(`- "${command}" is not in PATH or produced an error`)
       throw err
     }
-  },
+  }
 
   // Check if the py launcher can find a valid Python to use.
   // Will exit the Python finder on success.
@@ -202,7 +203,7 @@ PythonFinder.prototype = {
   // the first command line argument. Since "py.exe -3" would be an invalid
   // executable for "execFile", we have to use the launcher to figure out
   // where the actual "python.exe" executable is located.
-  checkPyLauncher: async function checkPyLauncher () {
+  async checkPyLauncher () {
     this.log.verbose(`- executing "${this.pyLauncher}" to get Python 3 executable path`)
     // Possible outcomes: same as checkCommand
     try {
@@ -213,11 +214,11 @@ PythonFinder.prototype = {
       this.addLog(`- "${this.pyLauncher}" is not in PATH or produced an error`)
       throw err
     }
-  },
+  }
 
   // Check if a Python executable is the correct version to use.
   // Will exit the Python finder on success.
-  checkExecPath: async function checkExecPath (execPath) {
+  async checkExecPath (execPath) {
     this.log.verbose(`- executing "${execPath}" to get version`)
     // Possible outcomes:
     // - Error: executable can not be run (likely meaning the command wasn't
@@ -249,11 +250,11 @@ PythonFinder.prototype = {
       this.addLog(`- "${execPath}" could not be run`)
       throw err
     }
-  },
+  }
 
   // Run an executable or shell command, trimming the output.
-  run: async function run (exec, args, shell) {
-    const env = extend({}, this.env)
+  async run (exec, args, shell) {
+    const env = Object.assign({}, this.env)
     env.TERM = 'dumb'
     const opts = { env, shell }
 
@@ -270,14 +271,14 @@ PythonFinder.prototype = {
       this.log.silly('execFile: threw:\n%s', err.stack)
       throw err
     }
-  },
+  }
 
-  succeed: function succeed (execPath, version) {
+  succeed (execPath, version) {
     this.log.info(`using Python version ${version} found at "${execPath}"`)
     return execPath
-  },
+  }
 
-  fail: function fail () {
+  fail () {
     const errorLog = this.errorLog.join('\n')
 
     const pathExample = this.win
@@ -306,10 +307,4 @@ PythonFinder.prototype = {
   }
 }
 
-const findPython = async (configPython) => new PythonFinder(configPython).findPython()
-
-module.exports = findPython
-module.exports.test = {
-  PythonFinder,
-  findPython
-}
+module.exports = PythonFinder
diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js
index 7df976a2db..b57770259a 100644
--- a/lib/find-visualstudio.js
+++ b/lib/find-visualstudio.js
@@ -5,26 +5,28 @@ const { existsSync } = require('fs')
 const { win32: path } = require('path')
 const { regSearchKeys, execFile } = require('./util')
 
-function VisualStudioFinder (nodeSemver, configMsvsVersion) {
-  this.nodeSemver = nodeSemver
-  this.configMsvsVersion = configMsvsVersion
-  this.errorLog = []
-  this.validVersions = []
-}
+class VisualStudioFinder {
+  static findVisualStudio = (...args) => new VisualStudioFinder(...args).findVisualStudio()
+
+  log = log.withPrefix('find VS')
 
-VisualStudioFinder.prototype = {
-  log: log.withPrefix('find VS'),
+  regSearchKeys = regSearchKeys
 
-  regSearchKeys,
+  constructor (nodeSemver, configMsvsVersion) {
+    this.nodeSemver = nodeSemver
+    this.configMsvsVersion = configMsvsVersion
+    this.errorLog = []
+    this.validVersions = []
+  }
 
   // Logs a message at verbose level, but also saves it to be displayed later
   // at error level if an error occurs. This should help diagnose the problem.
-  addLog: function addLog (message) {
+  addLog (message) {
     this.log.verbose(message)
     this.errorLog.push(message)
-  },
+  }
 
-  findVisualStudio: async function findVisualStudio () {
+  async findVisualStudio () {
     this.configVersionYear = null
     this.configPath = null
     if (this.configMsvsVersion) {
@@ -65,16 +67,16 @@ VisualStudioFinder.prototype = {
     }
 
     return this.fail()
-  },
+  }
 
-  succeed: function succeed (info) {
+  succeed (info) {
     this.log.info(`using VS${info.versionYear} (${info.version}) found at:` +
                   `\n"${info.path}"` +
                   '\nrun with --verbose for detailed information')
     return info
-  },
+  }
 
-  fail: function fail () {
+  fail () {
     if (this.configMsvsVersion && this.envVcInstallDir) {
       this.errorLog.push(
         'msvs_version does not match this VS Command Prompt or the',
@@ -109,11 +111,11 @@ VisualStudioFinder.prototype = {
 
     this.log.error(`\n${errorLog}\n\n${infoLog}\n`)
     throw new Error('Could not find any Visual Studio installation to use')
-  },
+  }
 
   // Invoke the PowerShell script to get information about Visual Studio 2017
   // or newer installations
-  findVisualStudio2017OrNewer: async function findVisualStudio2017OrNewer () {
+  async findVisualStudio2017OrNewer () {
     const ps = path.join(process.env.SystemRoot, 'System32',
       'WindowsPowerShell', 'v1.0', 'powershell.exe')
     const csFile = path.join(__dirname, 'Find-VisualStudio.cs')
@@ -128,11 +130,11 @@ VisualStudioFinder.prototype = {
     this.log.silly('Running', ps, psArgs)
     const [err, stdout, stderr] = await execFile(ps, psArgs, { encoding: 'utf8' })
     return this.parseData(err, stdout, stderr)
-  },
+  }
 
   // Parse the output of the PowerShell script and look for an installation
   // of Visual Studio 2017 or newer to use
-  parseData: function parseData (err, stdout, stderr) {
+  parseData (err, stdout, stderr) {
     this.log.silly('PS stderr = %j', stderr)
 
     const failPowershell = () => {
@@ -220,10 +222,10 @@ VisualStudioFinder.prototype = {
     this.addLog(
       'could not find a version of Visual Studio 2017 or newer to use')
     return null
-  },
+  }
 
   // Helper - process version information
-  getVersionInfo: function getVersionInfo (info) {
+  getVersionInfo (info) {
     const match = /^(\d+)\.(\d+)\..*/.exec(info.version)
     if (!match) {
       this.log.silly('- failed to parse version:', info.version)
@@ -249,14 +251,14 @@ VisualStudioFinder.prototype = {
     }
     this.log.silly('- unsupported version:', ret.versionMajor)
     return {}
-  },
+  }
 
-  msBuildPathExists: function msBuildPathExists (path) {
+  msBuildPathExists (path) {
     return existsSync(path)
-  },
+  }
 
   // Helper - process MSBuild information
-  getMSBuild: function getMSBuild (info, versionYear) {
+  getMSBuild (info, versionYear) {
     const pkg = 'Microsoft.VisualStudio.VC.MSBuild.Base'
     const msbuildPath = path.join(info.path, 'MSBuild', 'Current', 'Bin', 'MSBuild.exe')
     const msbuildPathArm64 = path.join(info.path, 'MSBuild', 'Current', 'Bin', 'arm64', 'MSBuild.exe')
@@ -280,10 +282,10 @@ VisualStudioFinder.prototype = {
       return msbuildPath
     }
     return null
-  },
+  }
 
   // Helper - process toolset information
-  getToolset: function getToolset (info, versionYear) {
+  getToolset (info, versionYear) {
     const pkg = 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64'
     const express = 'Microsoft.VisualStudio.WDExpress'
 
@@ -304,10 +306,10 @@ VisualStudioFinder.prototype = {
     }
     this.log.silly('- invalid versionYear:', versionYear)
     return null
-  },
+  }
 
   // Helper - process Windows SDK information
-  getSDK: function getSDK (info) {
+  getSDK (info) {
     const win8SDK = 'Microsoft.VisualStudio.Component.Windows81SDK'
     const win10SDKPrefix = 'Microsoft.VisualStudio.Component.Windows10SDK.'
     const win11SDKPrefix = 'Microsoft.VisualStudio.Component.Windows11SDK.'
@@ -339,10 +341,10 @@ VisualStudioFinder.prototype = {
       return '8.1'
     }
     return null
-  },
+  }
 
   // Find an installation of Visual Studio 2015 to use
-  findVisualStudio2015: async function findVisualStudio2015 () {
+  async findVisualStudio2015 () {
     if (this.nodeSemver.major >= 19) {
       this.addLog(
         'not looking for VS2015 as it is only supported up to Node.js 18')
@@ -355,10 +357,10 @@ VisualStudioFinder.prototype = {
       versionYear: 2015,
       toolset: 'v140'
     })
-  },
+  }
 
   // Find an installation of Visual Studio 2013 to use
-  findVisualStudio2013: async function findVisualStudio2013 () {
+  async findVisualStudio2013 () {
     if (this.nodeSemver.major >= 9) {
       this.addLog(
         'not looking for VS2013 as it is only supported up to Node.js 8')
@@ -371,10 +373,10 @@ VisualStudioFinder.prototype = {
       versionYear: 2013,
       toolset: 'v120'
     })
-  },
+  }
 
   // Helper - common code for VS2013 and VS2015
-  findOldVS: async function findOldVS (info) {
+  async findOldVS (info) {
     const regVC7 = ['HKLM\\Software\\Microsoft\\VisualStudio\\SxS\\VC7',
       'HKLM\\Software\\Wow6432Node\\Microsoft\\VisualStudio\\SxS\\VC7']
     const regMSBuild = 'HKLM\\Software\\Microsoft\\MSBuild\\ToolsVersions'
@@ -408,14 +410,14 @@ VisualStudioFinder.prototype = {
       this.addLog('- not found')
       return null
     }
-  },
+  }
 
   // After finding a usable version of Visual Studio:
   // - add it to validVersions to be displayed at the end if a specific
   //   version was requested and not found;
   // - check if this is the version that was requested.
   // - check if this matches the Visual Studio Command Prompt
-  checkConfigVersion: function checkConfigVersion (versionYear, vsPath) {
+  checkConfigVersion (versionYear, vsPath) {
     this.validVersions.push(versionYear)
     this.validVersions.push(vsPath)
 
@@ -438,10 +440,4 @@ VisualStudioFinder.prototype = {
   }
 }
 
-const findVisualStudio = async (nodeSemver, configMsvsVersion) => new VisualStudioFinder(nodeSemver, configMsvsVersion).findVisualStudio()
-
-module.exports = findVisualStudio
-module.exports.test = {
-  VisualStudioFinder,
-  findVisualStudio
-}
+module.exports = VisualStudioFinder
diff --git a/lib/node-gyp.js b/lib/node-gyp.js
index c2ce69051f..5e25bf996f 100644
--- a/lib/node-gyp.js
+++ b/lib/node-gyp.js
@@ -4,8 +4,8 @@ const path = require('path')
 const nopt = require('nopt')
 const log = require('./log')
 const childProcess = require('child_process')
-const EE = require('events').EventEmitter
-const { inherits } = require('util')
+const { EventEmitter } = require('events')
+
 const commands = [
   // Module build commands
   'build',
@@ -17,191 +17,172 @@ const commands = [
   'list',
   'remove'
 ]
-const aliases = {
-  ls: 'list',
-  rm: 'remove'
-}
 
-function gyp () {
-  return new Gyp()
-}
+class Gyp extends EventEmitter {
+  /**
+   * Export the contents of the package.json.
+   */
+  package = require('../package.json')
+
+  /**
+   * nopt configuration definitions
+   */
+  configDefs = {
+    help: Boolean, // everywhere
+    arch: String, // 'configure'
+    cafile: String, // 'install'
+    debug: Boolean, // 'build'
+    directory: String, // bin
+    make: String, // 'build'
+    'msvs-version': String, // 'configure'
+    ensure: Boolean, // 'install'
+    solution: String, // 'build' (windows only)
+    proxy: String, // 'install'
+    noproxy: String, // 'install'
+    devdir: String, // everywhere
+    nodedir: String, // 'configure'
+    loglevel: String, // everywhere
+    python: String, // 'configure'
+    'dist-url': String, // 'install'
+    tarball: String, // 'install'
+    jobs: String, // 'build'
+    thin: String, // 'configure'
+    'force-process-config': Boolean // 'configure'
+  }
 
-function Gyp () {
-  this.devDir = ''
+  /**
+   * nopt shorthands
+   */
+  shorthands = {
+    release: '--no-debug',
+    C: '--directory',
+    debug: '--debug',
+    j: '--jobs',
+    silly: '--loglevel=silly',
+    verbose: '--loglevel=verbose',
+    silent: '--loglevel=silent'
+  }
 
-  this.commands = commands.reduce((acc, command) => {
-    acc[command] = (argv) => require('./' + command)(this, argv)
-    return acc
-  }, {})
-}
-inherits(Gyp, EE)
-exports.Gyp = Gyp
-const proto = Gyp.prototype
-
-/**
- * Export the contents of the package.json.
- */
-
-proto.package = require('../package.json')
-
-/**
- * nopt configuration definitions
- */
-
-proto.configDefs = {
-  help: Boolean, // everywhere
-  arch: String, // 'configure'
-  cafile: String, // 'install'
-  debug: Boolean, // 'build'
-  directory: String, // bin
-  make: String, // 'build'
-  'msvs-version': String, // 'configure'
-  ensure: Boolean, // 'install'
-  solution: String, // 'build' (windows only)
-  proxy: String, // 'install'
-  noproxy: String, // 'install'
-  devdir: String, // everywhere
-  nodedir: String, // 'configure'
-  loglevel: String, // everywhere
-  python: String, // 'configure'
-  'dist-url': String, // 'install'
-  tarball: String, // 'install'
-  jobs: String, // 'build'
-  thin: String, // 'configure'
-  'force-process-config': Boolean // 'configure'
-}
+  /**
+   * expose the command aliases for the bin file to use.
+   */
+  aliases = {
+    ls: 'list',
+    rm: 'remove'
+  }
 
-/**
- * nopt shorthands
- */
-
-proto.shorthands = {
-  release: '--no-debug',
-  C: '--directory',
-  debug: '--debug',
-  j: '--jobs',
-  silly: '--loglevel=silly',
-  verbose: '--loglevel=verbose',
-  silent: '--loglevel=silent'
-}
+  constructor (...args) {
+    super(...args)
 
-/**
- * expose the command aliases for the bin file to use.
- */
+    this.devDir = ''
 
-proto.aliases = aliases
+    this.commands = commands.reduce((acc, command) => {
+      acc[command] = (argv) => require('./' + command)(this, argv)
+      return acc
+    }, {})
 
-/**
- * Parses the given argv array and sets the 'opts',
- * 'argv' and 'command' properties.
- */
+    Object.defineProperty(this, 'version', {
+      enumerable: true,
+      get: function () { return this.package.version }
+    })
+  }
 
-proto.parseArgv = function parseOpts (argv) {
-  this.opts = nopt(this.configDefs, this.shorthands, argv)
-  this.argv = this.opts.argv.remain.slice()
+  /**
+   * Parses the given argv array and sets the 'opts',
+   * 'argv' and 'command' properties.
+   */
+  parseArgv (argv) {
+    this.opts = nopt(this.configDefs, this.shorthands, argv)
+    this.argv = this.opts.argv.remain.slice()
 
-  const commands = this.todo = []
+    const commands = this.todo = []
 
-  // create a copy of the argv array with aliases mapped
-  argv = this.argv.map(function (arg) {
+    // create a copy of the argv array with aliases mapped
+    argv = this.argv.map((arg) => {
     // is this an alias?
-    if (arg in this.aliases) {
-      arg = this.aliases[arg]
-    }
-    return arg
-  }, this)
-
-  // process the mapped args into "command" objects ("name" and "args" props)
-  argv.slice().forEach(function (arg) {
-    if (arg in this.commands) {
-      const args = argv.splice(0, argv.indexOf(arg))
-      argv.shift()
-      if (commands.length > 0) {
-        commands[commands.length - 1].args = args
+      if (arg in this.aliases) {
+        arg = this.aliases[arg]
       }
-      commands.push({ name: arg, args: [] })
+      return arg
+    })
+
+    // process the mapped args into "command" objects ("name" and "args" props)
+    argv.slice().forEach((arg) => {
+      if (arg in this.commands) {
+        const args = argv.splice(0, argv.indexOf(arg))
+        argv.shift()
+        if (commands.length > 0) {
+          commands[commands.length - 1].args = args
+        }
+        commands.push({ name: arg, args: [] })
+      }
+    })
+    if (commands.length > 0) {
+      commands[commands.length - 1].args = argv.splice(0)
     }
-  }, this)
-  if (commands.length > 0) {
-    commands[commands.length - 1].args = argv.splice(0)
-  }
 
-  // support for inheriting config env variables from npm
-  const npmConfigPrefix = 'npm_config_'
-  Object.keys(process.env).forEach(function (name) {
-    if (name.indexOf(npmConfigPrefix) !== 0) {
-      return
-    }
-    const val = process.env[name]
-    if (name === npmConfigPrefix + 'loglevel') {
-      log.logger.level = val
-    } else {
+    // support for inheriting config env variables from npm
+    const npmConfigPrefix = 'npm_config_'
+    Object.keys(process.env).forEach((name) => {
+      if (name.indexOf(npmConfigPrefix) !== 0) {
+        return
+      }
+      const val = process.env[name]
+      if (name === npmConfigPrefix + 'loglevel') {
+        log.logger.level = val
+      } else {
       // add the user-defined options to the config
-      name = name.substring(npmConfigPrefix.length)
-      // gyp@741b7f1 enters an infinite loop when it encounters
-      // zero-length options so ensure those don't get through.
-      if (name) {
+        name = name.substring(npmConfigPrefix.length)
+        // gyp@741b7f1 enters an infinite loop when it encounters
+        // zero-length options so ensure those don't get through.
+        if (name) {
         // convert names like force_process_config to force-process-config
-        if (name.includes('_')) {
-          name = name.replace(/_/g, '-')
+          if (name.includes('_')) {
+            name = name.replace(/_/g, '-')
+          }
+          this.opts[name] = val
         }
-        this.opts[name] = val
       }
-    }
-  }, this)
+    })
 
-  if (this.opts.loglevel) {
-    log.logger.level = this.opts.loglevel
+    if (this.opts.loglevel) {
+      log.logger.level = this.opts.loglevel
+    }
+    log.resume()
   }
-  log.resume()
-}
-
-/**
- * Spawns a child process and emits a 'spawn' event.
- */
 
-proto.spawn = function spawn (command, args, opts) {
-  if (!opts) {
-    opts = {}
-  }
-  if (!opts.silent && !opts.stdio) {
-    opts.stdio = [0, 1, 2]
+  /**
+   * Spawns a child process and emits a 'spawn' event.
+   */
+  spawn (command, args, opts) {
+    if (!opts) {
+      opts = {}
+    }
+    if (!opts.silent && !opts.stdio) {
+      opts.stdio = [0, 1, 2]
+    }
+    const cp = childProcess.spawn(command, args, opts)
+    log.info('spawn', command)
+    log.info('spawn args', args)
+    return cp
   }
-  const cp = childProcess.spawn(command, args, opts)
-  log.info('spawn', command)
-  log.info('spawn args', args)
-  return cp
-}
 
-/**
- * Returns the usage instructions for node-gyp.
- */
-
-proto.usage = function usage () {
-  const str = [
-    '',
-    '  Usage: node-gyp  [options]',
-    '',
-    '  where  is one of:',
-    commands.map(function (c) {
-      return '    - ' + c + ' - ' + require('./' + c).usage
-    }).join('\n'),
-    '',
-    'node-gyp@' + this.version + '  ' + path.resolve(__dirname, '..'),
-    'node@' + process.versions.node
-  ].join('\n')
-  return str
+  /**
+   * Returns the usage instructions for node-gyp.
+   */
+  usage () {
+    return [
+      '',
+      '  Usage: node-gyp  [options]',
+      '',
+      '  where  is one of:',
+      commands.map((c) => '    - ' + c + ' - ' + require('./' + c).usage).join('\n'),
+      '',
+      'node-gyp@' + this.version + '  ' + path.resolve(__dirname, '..'),
+      'node@' + process.versions.node
+    ].join('\n')
+  }
 }
 
-/**
- * Version number getter.
- */
-
-Object.defineProperty(proto, 'version', {
-  get: function () {
-    return this.package.version
-  },
-  enumerable: true
-})
-
-module.exports = exports = gyp
+module.exports = () => new Gyp()
+module.exports.Gyp = Gyp
diff --git a/test/common.js b/test/common.js
index b714ee2902..f7e2e2a34c 100644
--- a/test/common.js
+++ b/test/common.js
@@ -1,3 +1,17 @@
 const envPaths = require('env-paths')
 
 module.exports.devDir = () => envPaths('node-gyp', { suffix: '' }).cache
+
+module.exports.poison = (object, property) => {
+  function fail () {
+    console.error(Error(`Property ${property} should not have been accessed.`))
+    process.abort()
+  }
+  const descriptor = {
+    configurable: false,
+    enumerable: false,
+    get: fail,
+    set: fail
+  }
+  Object.defineProperty(object, property, descriptor)
+}
diff --git a/test/test-find-python.js b/test/test-find-python.js
index a00ae94057..c5fa35d4bc 100644
--- a/test/test-find-python.js
+++ b/test/test-find-python.js
@@ -4,42 +4,33 @@ delete process.env.PYTHON
 
 const { describe, it } = require('mocha')
 const assert = require('assert')
-const { test: { PythonFinder, findPython: testFindPython } } = require('../lib/find-python')
+const PythonFinder = require('../lib/find-python')
 const { execFile } = require('../lib/util')
+const { poison } = require('./common')
 
-describe('find-python', function () {
-  it('find python', async function () {
-    const found = await testFindPython(null)
-    const [err, stdout, stderr] = await execFile(found, ['-V'], { encoding: 'utf-8' })
-    assert.strictEqual(err, null)
-    assert.ok(/Python 3/.test(stdout))
-    assert.strictEqual(stderr, '')
-  })
-
-  function poison (object, property) {
-    function fail () {
-      console.error(Error(`Property ${property} should not have been accessed.`))
-      process.abort()
-    }
-    const descriptor = {
-      configurable: false,
-      enumerable: false,
-      get: fail,
-      set: fail
-    }
-    Object.defineProperty(object, property, descriptor)
+class TestPythonFinder extends PythonFinder {
+  constructor (...args) {
+    super(...args)
+    delete this.env.NODE_GYP_FORCE_PYTHON
   }
 
-  function TestPythonFinder () { PythonFinder.apply(this, arguments) }
-  TestPythonFinder.prototype = Object.create(PythonFinder.prototype)
-  delete TestPythonFinder.prototype.env.NODE_GYP_FORCE_PYTHON
-  const findPython = async (f) => {
+  async findPython () {
     try {
-      return { err: null, python: await f.findPython() }
+      return { err: null, python: await super.findPython() }
     } catch (err) {
       return { err, python: null }
     }
   }
+}
+
+describe('find-python', function () {
+  it('find python', async function () {
+    const found = await PythonFinder.findPython(null)
+    const [err, stdout, stderr] = await execFile(found, ['-V'], { encoding: 'utf-8' })
+    assert.strictEqual(err, null)
+    assert.ok(/Python 3/.test(stdout))
+    assert.strictEqual(stderr, '')
+  })
 
   it('find python - python', async function () {
     const f = new TestPythonFinder('python')
@@ -55,7 +46,7 @@ describe('find-python', function () {
       return [null, '/path/python']
     }
 
-    const { err, python } = await findPython(f)
+    const { err, python } = await f.findPython()
     assert.strictEqual(err, null)
     assert.strictEqual(python, '/path/python')
   })
@@ -72,7 +63,7 @@ describe('find-python', function () {
       }
     }
 
-    const { err } = await findPython(f)
+    const { err } = await f.findPython()
     assert.ok(/Could not find any Python/.test(err))
     assert.ok(/not supported/i.test(f.errorLog))
   })
@@ -89,7 +80,7 @@ describe('find-python', function () {
       }
     }
 
-    const { err } = await findPython(f)
+    const { err } = await f.findPython()
     assert.ok(/Could not find any Python/.test(err))
     assert.ok(/not in PATH/.test(f.errorLog))
   })
@@ -107,7 +98,7 @@ describe('find-python', function () {
       }
     }
 
-    const { err } = await findPython(f)
+    const { err } = await f.findPython()
     assert.ok(/Could not find any Python/.test(err))
     assert.ok(/not in PATH/.test(f.errorLog))
   })
@@ -136,7 +127,7 @@ describe('find-python', function () {
         assert.fail()
       }
     }
-    const { err, python } = await findPython(f)
+    const { err, python } = await f.findPython()
     assert.strictEqual(err, null)
     assert.strictEqual(python, 'Z:\\snake.exe')
   })
@@ -159,7 +150,7 @@ describe('find-python', function () {
         assert.fail()
       }
     }
-    const { err, python } = await findPython(f)
+    const { err, python } = await f.findPython()
     assert.strictEqual(err, null)
     assert.ok(python === expectedProgram)
   })
@@ -177,7 +168,7 @@ describe('find-python', function () {
         assert.fail()
       }
     }
-    const { err } = await findPython(f)
+    const { err } = await f.findPython()
     assert.ok(/Could not find any Python/.test(err))
     assert.ok(/not in PATH/.test(f.errorLog))
   })
diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js
index cb065e52cf..08e9438c45 100644
--- a/test/test-find-visualstudio.js
+++ b/test/test-find-visualstudio.js
@@ -4,34 +4,20 @@ const { describe, it } = require('mocha')
 const assert = require('assert')
 const fs = require('fs')
 const path = require('path')
-const { test: { VisualStudioFinder } } = require('../lib/find-visualstudio')
+const VisualStudioFinder = require('../lib/find-visualstudio')
+const { poison } = require('./common')
 
 const semverV1 = { major: 1, minor: 0, patch: 0 }
 
 delete process.env.VCINSTALLDIR
 
-function poison (object, property) {
-  function fail () {
-    console.error(Error(`Property ${property} should not have been accessed.`))
-    process.abort()
-  }
-  const descriptor = {
-    configurable: false,
-    enumerable: false,
-    get: fail,
-    set: fail
-  }
-  Object.defineProperty(object, property, descriptor)
-}
-
-function TestVisualStudioFinder () { VisualStudioFinder.apply(this, arguments) }
-TestVisualStudioFinder.prototype = Object.create(VisualStudioFinder.prototype)
-
-const findVisualStudio = async (finder) => {
-  try {
-    return { err: null, info: await finder.findVisualStudio() }
-  } catch (err) {
-    return { err, info: null }
+class TestVisualStudioFinder extends VisualStudioFinder {
+  async findVisualStudio () {
+    try {
+      return { err: null, info: await super.findVisualStudio() }
+    } catch (err) {
+      return { err, info: null }
+    }
   }
 }
 
@@ -62,7 +48,7 @@ describe('find-visualstudio', function () {
       throw new Error()
     }
 
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info, {
       msBuild: 'C:\\MSBuild12\\MSBuild.exe',
@@ -102,7 +88,7 @@ describe('find-visualstudio', function () {
       throw new Error()
     }
 
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.ok(/find .* Visual Studio/i.test(err), 'expect error')
     assert.ok(!info, 'no data')
   })
@@ -129,7 +115,7 @@ describe('find-visualstudio', function () {
       }
       throw new Error()
     }
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info, {
       msBuild: 'C:\\MSBuild14\\MSBuild.exe',
@@ -228,7 +214,7 @@ describe('find-visualstudio', function () {
       const data = fs.readFileSync(file)
       return finder.parseData(null, data, '')
     }
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info, {
       msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\' +
@@ -254,7 +240,7 @@ describe('find-visualstudio', function () {
       const data = fs.readFileSync(file)
       return finder.parseData(null, data, '')
     }
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info, {
       msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\' +
@@ -279,7 +265,7 @@ describe('find-visualstudio', function () {
       const data = fs.readFileSync(file)
       return finder.parseData(null, data, '')
     }
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info, {
       msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\' +
@@ -305,7 +291,7 @@ describe('find-visualstudio', function () {
       const data = fs.readFileSync(file)
       return finder.parseData(null, data, '')
     }
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info, {
       msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' +
@@ -331,7 +317,7 @@ describe('find-visualstudio', function () {
       const data = fs.readFileSync(file)
       return finder.parseData(null, data, '')
     }
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info, {
       msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' +
@@ -357,7 +343,7 @@ describe('find-visualstudio', function () {
       const data = fs.readFileSync(file)
       return finder.parseData(null, data, '')
     }
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info, {
       msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' +
@@ -392,7 +378,7 @@ describe('find-visualstudio', function () {
       const data = fs.readFileSync(file)
       return finder.parseData(null, data, '')
     }
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info, {
       msBuild: msBuildPath,
@@ -456,7 +442,7 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, 'AABB')
 
     allVsVersions(finder)
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.ok(/find .* Visual Studio/i.test(err), 'expect error')
     assert.ok(!info, 'no data')
   })
@@ -465,7 +451,7 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, '2013')
 
     allVsVersions(finder)
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info.versionYear, 2013)
   })
@@ -474,7 +460,7 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2013')
 
     allVsVersions(finder)
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info.path, 'C:\\VS2013')
   })
@@ -483,7 +469,7 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, '2015')
 
     allVsVersions(finder)
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info.versionYear, 2015)
   })
@@ -492,7 +478,7 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2015')
 
     allVsVersions(finder)
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info.path, 'C:\\VS2015')
   })
@@ -501,7 +487,7 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, '2017')
 
     allVsVersions(finder)
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info.versionYear, 2017)
   })
@@ -511,7 +497,7 @@ describe('find-visualstudio', function () {
       'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community')
 
     allVsVersions(finder)
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info.path,
       'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community')
@@ -521,7 +507,7 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, '2019')
 
     allVsVersions(finder)
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info.versionYear, 2019)
   })
@@ -531,7 +517,7 @@ describe('find-visualstudio', function () {
       'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools')
 
     allVsVersions(finder)
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info.path,
       'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools')
@@ -545,7 +531,7 @@ describe('find-visualstudio', function () {
     }
 
     allVsVersions(finder)
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info.versionYear, 2022)
   })
@@ -555,7 +541,7 @@ describe('find-visualstudio', function () {
       'c:\\program files (x86)\\microsoft visual studio\\2019\\BUILDTOOLS')
 
     allVsVersions(finder)
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info.path,
       'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools')
@@ -569,7 +555,7 @@ describe('find-visualstudio', function () {
     }
 
     allVsVersions(finder)
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info.versionYear, 2022)
   })
@@ -582,7 +568,7 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
 
     allVsVersions(finder)
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info.path, 'C:\\VS2015')
   })
@@ -594,7 +580,7 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
 
     allVsVersions(finder)
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info.path,
       'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools')
@@ -607,7 +593,7 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
 
     allVsVersions(finder)
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.ok(/find .* Visual Studio/i.test(err), 'expect error')
     assert.ok(!info, 'no data')
   })
@@ -618,7 +604,7 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2015')
 
     allVsVersions(finder)
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
     assert.deepStrictEqual(info.path, 'C:\\VS2015')
   })
@@ -630,7 +616,7 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, 'C:\\VS2015')
 
     allVsVersions(finder)
-    const { err, info } = await findVisualStudio(finder)
+    const { err, info } = await finder.findVisualStudio()
     assert.ok(/find .* Visual Studio/i.test(err), 'expect error')
     assert.ok(!info, 'no data')
   })

From 4e493d4fb262d12ac52c84979071ccc79e666a1a Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Sat, 28 Oct 2023 14:13:10 -0700
Subject: [PATCH 397/551] chore: misc testing fixes (#2930)

* chore: misc test fixes

* Sort test runs by os first

* Use cross-env for test env var

* Try sorting matrix params

* Make FAST_TEST the default and rename to FULL_TEST

* Separate helper functions to not need to export test obj in files
---
 .github/PULL_REQUEST_TEMPLATE.md  |   2 +-
 .github/workflows/tests.yml       |  28 +++++--
 bin/node-gyp.js                   |  10 +--
 lib/configure.js                  |  33 +-------
 lib/create-config-gypi.js         |   4 +-
 lib/download.js                   |  39 ++++++++++
 lib/install.js                    |  41 +---------
 lib/log.js                        |  10 ++-
 lib/util.js                       |  30 +++++++-
 package.json                      |   3 +-
 test/common.js                    |   9 ++-
 test/reporter.js                  |  75 ------------------
 test/test-addon.js                | 108 +++++++++++---------------
 test/test-configure-python.js     |   6 +-
 test/test-create-config-gypi.js   |   3 +-
 test/test-download.js             |  27 +++----
 test/test-find-accessible-sync.js |  16 ++--
 test/test-install.js              | 124 ++++++++++++------------------
 18 files changed, 239 insertions(+), 329 deletions(-)
 create mode 100644 lib/download.js
 delete mode 100644 test/reporter.js

diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 779897573a..da362b8ccf 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -7,7 +7,7 @@ Contributor guide: https://github.com/nodejs/node/blob/main/CONTRIBUTING.md
 ##### Checklist
 
 
-- [ ] `npm install && npm test` passes
+- [ ] `npm install && npm run lint && npm test` passes
 - [ ] tests are included 
 - [ ] documentation is changed or added
 - [ ] commit message follows [commit guidelines](https://github.com/googleapis/release-please#how-should-i-write-my-commits)
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 29525b8a73..4aa041df9e 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -18,11 +18,24 @@ jobs:
     - uses: actions/checkout@v4
     - run: pip install --user ruff
     - run: ruff --output-format=github --select="E,F,PLC,PLE,UP,W,YTT" --ignore="E721,PLC1901,S101,UP031" --target-version=py38 .
+  Lint_JS:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout Repository
+      uses: actions/checkout@v4
+    - name: Use Node.js 20.x
+      uses: actions/setup-node@v3
+      with:
+        node-version: 20.x
+    - name: Install Dependencies
+      run: npm install --no-progress
+    - name: Lint
+      run: npm run lint
   Engines:
     runs-on: ubuntu-latest
     steps:
     - name: Checkout Repository
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
     - name: Use Node.js 20.x
       uses: actions/setup-node@v3
       with:
@@ -41,10 +54,11 @@ jobs:
       fail-fast: false
       max-parallel: 15
       matrix:
-        node: [16.x, 18.x, 20.x]
+        os: [macos, ubuntu, windows]
         python: ["3.8", "3.10", "3.12"]
-        os: [macos-latest, ubuntu-latest, windows-latest]
-    runs-on: ${{ matrix.os }}
+        node: [16.x, 18.x, 20.x]
+    name: ${{ matrix.os }} - ${{ matrix.python }} - ${{ matrix.node }}
+    runs-on: ${{ matrix.os }}-latest
     steps:
       - name: Checkout Repository
         uses: actions/checkout@v4
@@ -63,7 +77,7 @@ jobs:
           npm install --no-progress
           pip install pytest
       - name: Set Windows environment
-        if: startsWith(matrix.os, 'windows')
+        if: matrix.os == 'windows'
         run: |
           echo 'GYP_MSVS_VERSION=2015' >> $Env:GITHUB_ENV
           echo 'GYP_MSVS_OVERRIDE_PATH=C:\\Dummy' >> $Env:GITHUB_ENV
@@ -77,7 +91,11 @@ jobs:
         if: runner.os != 'Windows'
         shell: bash
         run: npm test --python="${pythonLocation}/python"
+        env:
+          FULL_TEST: ${{ (matrix.node == '20.x' && matrix.python == '3.12') && '1' || '0' }}
       - name: Run tests (Windows)
         if: runner.os == 'Windows'
         shell: pwsh
         run: npm run test --python="${env:pythonLocation}\\python.exe"
+        env:
+          FULL_TEST: ${{ (matrix.node == '20.x' && matrix.python == '3.12') && '1' || '0' }}
diff --git a/bin/node-gyp.js b/bin/node-gyp.js
index a6f3e73a50..f8317b47b3 100755
--- a/bin/node-gyp.js
+++ b/bin/node-gyp.js
@@ -32,9 +32,9 @@ if (prog.devDir) {
 
 if (prog.todo.length === 0) {
   if (~process.argv.indexOf('-v') || ~process.argv.indexOf('--version')) {
-    console.log('v%s', prog.version)
+    log.stdout('v%s', prog.version)
   } else {
-    console.log('%s', prog.usage())
+    log.stdout('%s', prog.usage())
   }
   process.exit(0)
 }
@@ -82,12 +82,12 @@ async function run () {
 
     if (command.name === 'list') {
       if (args.length) {
-        args.forEach((version) => console.log(version))
+        args.forEach((version) => log.stdout(version))
       } else {
-        console.log('No node development files installed. Use `node-gyp install` to install a version.')
+        log.stdout('No node development files installed. Use `node-gyp install` to install a version.')
       }
     } else if (args.length >= 1) {
-      console.log(...args.slice(1))
+      log.stdout(...args.slice(1))
     }
 
     // now run the next command in the queue
diff --git a/lib/configure.js b/lib/configure.js
index 8d59e33497..38fd2b9718 100644
--- a/lib/configure.js
+++ b/lib/configure.js
@@ -1,14 +1,14 @@
 'use strict'
 
-const { openSync, closeSync, promises: fs } = require('graceful-fs')
+const { promises: fs } = require('graceful-fs')
 const path = require('path')
 const log = require('./log')
 const os = require('os')
 const processRelease = require('./process-release')
 const win = process.platform === 'win32'
 const findNodeDirectory = require('./find-node-directory')
-const createConfigGypi = require('./create-config-gypi')
-const { format: msgFormat } = require('util')
+const { createConfigGypi } = require('./create-config-gypi')
+const { format: msgFormat, findAccessibleSync } = require('util')
 const { findPython } = require('./find-python')
 const { findVisualStudio } = win ? require('./find-visualstudio') : {}
 
@@ -277,32 +277,5 @@ async function configure (gyp, argv) {
   }
 }
 
-/**
- * Returns the first file or directory from an array of candidates that is
- * readable by the current user, or undefined if none of the candidates are
- * readable.
- */
-function findAccessibleSync (logprefix, dir, candidates) {
-  for (let next = 0; next < candidates.length; next++) {
-    const candidate = path.resolve(dir, candidates[next])
-    let fd
-    try {
-      fd = openSync(candidate, 'r')
-    } catch (e) {
-      // this candidate was not found or not readable, do nothing
-      log.silly(logprefix, 'Could not open %s: %s', candidate, e.message)
-      continue
-    }
-    closeSync(fd)
-    log.silly(logprefix, 'Found readable %s', candidate)
-    return candidate
-  }
-
-  return undefined
-}
-
 module.exports = configure
-module.exports.test = {
-  findAccessibleSync
-}
 module.exports.usage = 'Generates ' + (win ? 'MSVC project files' : 'a Makefile') + ' for the current module'
diff --git a/lib/create-config-gypi.js b/lib/create-config-gypi.js
index a9c427e51d..d598dea6e2 100644
--- a/lib/create-config-gypi.js
+++ b/lib/create-config-gypi.js
@@ -143,8 +143,8 @@ async function createConfigGypi ({ gyp, buildDir, nodeDir, vsInfo, python }) {
   return configPath
 }
 
-module.exports = createConfigGypi
-module.exports.test = {
+module.exports = {
+  createConfigGypi,
   parseConfigGypi,
   getCurrentConfigGypi
 }
diff --git a/lib/download.js b/lib/download.js
new file mode 100644
index 0000000000..ed0aa37f44
--- /dev/null
+++ b/lib/download.js
@@ -0,0 +1,39 @@
+const fetch = require('make-fetch-happen')
+const { promises: fs } = require('graceful-fs')
+const log = require('./log')
+
+async function download (gyp, url) {
+  log.http('GET', url)
+
+  const requestOpts = {
+    headers: {
+      'User-Agent': `node-gyp v${gyp.version} (node ${process.version})`,
+      Connection: 'keep-alive'
+    },
+    proxy: gyp.opts.proxy,
+    noProxy: gyp.opts.noproxy
+  }
+
+  const cafile = gyp.opts.cafile
+  if (cafile) {
+    requestOpts.ca = await readCAFile(cafile)
+  }
+
+  const res = await fetch(url, requestOpts)
+  log.http(res.status, res.url)
+
+  return res
+}
+
+async function readCAFile (filename) {
+  // The CA file can contain multiple certificates so split on certificate
+  // boundaries.  [\S\s]*? is used to match everything including newlines.
+  const ca = await fs.readFile(filename, 'utf8')
+  const re = /(-----BEGIN CERTIFICATE-----[\S\s]*?-----END CERTIFICATE-----)/g
+  return ca.match(re)
+}
+
+module.exports = {
+  download,
+  readCAFile
+}
diff --git a/lib/install.js b/lib/install.js
index b338102c95..7196a31629 100644
--- a/lib/install.js
+++ b/lib/install.js
@@ -9,12 +9,13 @@ const { Transform, promises: { pipeline } } = require('stream')
 const crypto = require('crypto')
 const log = require('./log')
 const semver = require('semver')
-const fetch = require('make-fetch-happen')
+const { download } = require('./download')
 const processRelease = require('./process-release')
+
 const win = process.platform === 'win32'
 
 async function install (gyp, argv) {
-  console.log()
+  log.stdout()
   const release = processRelease(argv, gyp, process.version, process.release)
   // Detecting target_arch based on logic from create-cnfig-gyp.js. Used on Windows only.
   const arch = win ? (gyp.opts.target_arch || gyp.opts.arch || process.arch || 'ia32') : ''
@@ -410,41 +411,5 @@ class ShaSum extends Transform {
   }
 }
 
-async function download (gyp, url) {
-  log.http('GET', url)
-
-  const requestOpts = {
-    headers: {
-      'User-Agent': `node-gyp v${gyp.version} (node ${process.version})`,
-      Connection: 'keep-alive'
-    },
-    proxy: gyp.opts.proxy,
-    noProxy: gyp.opts.noproxy
-  }
-
-  const cafile = gyp.opts.cafile
-  if (cafile) {
-    requestOpts.ca = await readCAFile(cafile)
-  }
-
-  const res = await fetch(url, requestOpts)
-  log.http(res.status, res.url)
-
-  return res
-}
-
-async function readCAFile (filename) {
-  // The CA file can contain multiple certificates so split on certificate
-  // boundaries.  [\S\s]*? is used to match everything including newlines.
-  const ca = await fs.readFile(filename, 'utf8')
-  const re = /(-----BEGIN CERTIFICATE-----[\S\s]*?-----END CERTIFICATE-----)/g
-  return ca.match(re)
-}
-
 module.exports = install
-module.exports.test = {
-  download,
-  install,
-  readCAFile
-}
 module.exports.usage = 'Install node development files for the specified node version.'
diff --git a/lib/log.js b/lib/log.js
index 47fa8a3372..6841719aba 100644
--- a/lib/log.js
+++ b/lib/log.js
@@ -73,11 +73,11 @@ class Logger {
     style: { fg: 'red', bg: 'black' }
   }]
 
-  constructor () {
+  constructor (stream) {
     process.on('log', (...args) => this.#onLog(...args))
     this.#levels = new Map(this.#levels.map((level, index) => [level.id, { ...level, index }]))
     this.level = 'info'
-    this.stream = process.stderr
+    this.stream = stream
     procLog.pause()
   }
 
@@ -158,8 +158,12 @@ class Logger {
   }
 }
 
+// used to suppress logs in tests
+const NULL_LOGGER = !!process.env.NODE_GYP_NULL_LOGGER
+
 module.exports = {
-  logger: new Logger(),
+  logger: new Logger(NULL_LOGGER ? null : process.stderr),
+  stdout: NULL_LOGGER ? () => {} : (...args) => console.log(...args),
   withPrefix,
   ...procLog
 }
diff --git a/lib/util.js b/lib/util.js
index ef2bfb0827..3f6aeeb7dc 100644
--- a/lib/util.js
+++ b/lib/util.js
@@ -1,8 +1,9 @@
 'use strict'
 
-const log = require('./log')
 const cp = require('child_process')
 const path = require('path')
+const { openSync, closeSync } = require('graceful-fs')
+const log = require('./log')
 
 const execFile = async (...args) => new Promise((resolve) => {
   const child = cp.execFile(...args, (...a) => resolve(a))
@@ -48,8 +49,33 @@ async function regSearchKeys (keys, value, addOpts) {
   }
 }
 
+/**
+ * Returns the first file or directory from an array of candidates that is
+ * readable by the current user, or undefined if none of the candidates are
+ * readable.
+ */
+function findAccessibleSync (logprefix, dir, candidates) {
+  for (let next = 0; next < candidates.length; next++) {
+    const candidate = path.resolve(dir, candidates[next])
+    let fd
+    try {
+      fd = openSync(candidate, 'r')
+    } catch (e) {
+      // this candidate was not found or not readable, do nothing
+      log.silly(logprefix, 'Could not open %s: %s', candidate, e.message)
+      continue
+    }
+    closeSync(fd)
+    log.silly(logprefix, 'Found readable %s', candidate)
+    return candidate
+  }
+
+  return undefined
+}
+
 module.exports = {
   execFile,
   regGetValue,
-  regSearchKeys
+  regSearchKeys,
+  findAccessibleSync
 }
diff --git a/package.json b/package.json
index 409e228001..dadbf1d25a 100644
--- a/package.json
+++ b/package.json
@@ -38,6 +38,7 @@
   },
   "devDependencies": {
     "bindings": "^1.5.0",
+    "cross-env": "^7.0.3",
     "mocha": "^10.2.0",
     "nan": "^2.14.2",
     "require-inject": "^1.4.4",
@@ -45,6 +46,6 @@
   },
   "scripts": {
     "lint": "standard \"*/*.js\" \"test/**/*.js\" \".github/**/*.js\"",
-    "test": "npm run lint && mocha --timeout 15000 --reporter=test/reporter.js test/test-download.js test/test-*"
+    "test": "cross-env NODE_GYP_NULL_LOGGER=true mocha --timeout 15000 test/test-download.js test/test-*"
   }
 }
diff --git a/test/common.js b/test/common.js
index f7e2e2a34c..aef6bacfe1 100644
--- a/test/common.js
+++ b/test/common.js
@@ -1,6 +1,7 @@
 const envPaths = require('env-paths')
+const semver = require('semver')
 
-module.exports.devDir = () => envPaths('node-gyp', { suffix: '' }).cache
+module.exports.devDir = envPaths('node-gyp', { suffix: '' }).cache
 
 module.exports.poison = (object, property) => {
   function fail () {
@@ -15,3 +16,9 @@ module.exports.poison = (object, property) => {
   }
   Object.defineProperty(object, property, descriptor)
 }
+
+// Only run full test suite when instructed and on a non-prerelease version of node
+module.exports.FULL_TEST =
+  process.env.FULL_TEST === '1' &&
+  process.release.name === 'node' &&
+  semver.prerelease(process.version) === null
diff --git a/test/reporter.js b/test/reporter.js
deleted file mode 100644
index 9964b1b5d5..0000000000
--- a/test/reporter.js
+++ /dev/null
@@ -1,75 +0,0 @@
-const Mocha = require('mocha')
-
-class Reporter {
-  constructor (runner) {
-    this.failedTests = []
-
-    runner.on(Mocha.Runner.constants.EVENT_RUN_BEGIN, () => {
-      console.log('Starting tests')
-    })
-
-    runner.on(Mocha.Runner.constants.EVENT_RUN_END, () => {
-      console.log('Tests finished')
-      console.log()
-      console.log('****************')
-      console.log('* TESTS REPORT *')
-      console.log('****************')
-      console.log()
-      console.log(`Executed ${runner.stats.suites} suites with ${runner.stats.tests} tests in ${runner.stats.duration} ms`)
-      console.log(`  Passed: ${runner.stats.passes}`)
-      console.log(`  Skipped: ${runner.stats.pending}`)
-      console.log(`  Failed: ${runner.stats.failures}`)
-      if (this.failedTests.length > 0) {
-        console.log()
-        console.log('  Failed test details')
-        this.failedTests.forEach((failedTest, index) => {
-          console.log()
-          console.log(`    ${index + 1}.'${failedTest.test.fullTitle()}'`)
-          console.log(`      Name: ${failedTest.error.name}`)
-          console.log(`      Message: ${failedTest.error.message}`)
-          console.log(`      Code: ${failedTest.error.code}`)
-          console.log(`      Stack: ${failedTest.error.stack}`)
-        })
-      }
-      console.log()
-    })
-
-    runner.on(Mocha.Runner.constants.EVENT_SUITE_BEGIN, (suite) => {
-      if (suite.root) {
-        return
-      }
-      console.log(`Starting suite '${suite.title}'`)
-    })
-
-    runner.on(Mocha.Runner.constants.EVENT_SUITE_END, (suite) => {
-      if (suite.root) {
-        return
-      }
-      console.log(`Suite '${suite.title}' finished`)
-      console.log()
-    })
-
-    runner.on(Mocha.Runner.constants.EVENT_TEST_BEGIN, (test) => {
-      console.log(`Starting test '${test.title}'`)
-    })
-
-    runner.on(Mocha.Runner.constants.EVENT_TEST_PASS, (test) => {
-      console.log(`Test '${test.title}' passed in ${test.duration} ms`)
-    })
-
-    runner.on(Mocha.Runner.constants.EVENT_TEST_PENDING, (test) => {
-      console.log(`Test '${test.title}' skipped in ${test.duration} ms`)
-    })
-
-    runner.on(Mocha.Runner.constants.EVENT_TEST_FAIL, (test, error) => {
-      this.failedTests.push({ test, error })
-      console.log(`Test '${test.title}' failed in ${test.duration} ms with ${error}`)
-    })
-
-    runner.on(Mocha.Runner.constants.EVENT_TEST_END, (test) => {
-      console.log()
-    })
-  }
-}
-
-module.exports = Reporter
diff --git a/test/test-addon.js b/test/test-addon.js
index 0a0b9d56c5..b8d90b65bb 100644
--- a/test/test-addon.js
+++ b/test/test-addon.js
@@ -4,66 +4,66 @@ const { describe, it } = require('mocha')
 const assert = require('assert')
 const path = require('path')
 const fs = require('graceful-fs')
-const { execFileSync, execFile } = require('child_process')
 const os = require('os')
+const cp = require('child_process')
+const util = require('../lib/util')
+
 const addonPath = path.resolve(__dirname, 'node_modules', 'hello_world')
 const nodeGyp = path.resolve(__dirname, '..', 'bin', 'node-gyp.js')
 
-function runHello (hostProcess) {
-  if (!hostProcess) {
-    hostProcess = process.execPath
-  }
+const execFileSync = (...args) => cp.execFileSync(...args).toString().trim()
+
+const execFile = async (cmd) => {
+  const [err,, stderr] = await util.execFile(process.execPath, cmd, {
+    env: { ...process.env, NODE_GYP_NULL_LOGGER: undefined },
+    encoding: 'utf-8'
+  })
+  return [err, stderr.toString().trim().split(/\r?\n/)]
+}
+
+function runHello (hostProcess = process.execPath) {
   const testCode = "console.log(require('hello_world').hello())"
-  return execFileSync(hostProcess, ['-e', testCode], { cwd: __dirname }).toString()
+  return execFileSync(hostProcess, ['-e', testCode], { cwd: __dirname })
 }
 
 function getEncoding () {
   const code = 'import locale;print(locale.getdefaultlocale()[1])'
-  return execFileSync('python', ['-c', code]).toString().trim()
+  return execFileSync('python', ['-c', code])
 }
 
 function checkCharmapValid () {
-  let data
   try {
-    data = execFileSync('python', ['fixtures/test-charmap.py'],
-      { cwd: __dirname })
-  } catch (err) {
+    const data = execFileSync('python', ['fixtures/test-charmap.py'], { cwd: __dirname })
+    return data.split('\n').pop() === 'True'
+  } catch {
     return false
   }
-  const lines = data.toString().trim().split('\n')
-  return lines.pop() === 'True'
 }
 
 describe('addon', function () {
   this.timeout(300000)
 
-  it('build simple addon', function (done) {
+  it('build simple addon', async function () {
     // Set the loglevel otherwise the output disappears when run via 'npm test'
     const cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose']
-    const proc = execFile(process.execPath, cmd, function (err, stdout, stderr) {
-      const logLines = stderr.toString().trim().split(/\r?\n/)
-      const lastLine = logLines[logLines.length - 1]
-      assert.strictEqual(err, null)
-      assert.strictEqual(lastLine, 'gyp info ok', 'should end in ok')
-      assert.strictEqual(runHello().trim(), 'world')
-      done()
-    })
-    proc.stdout.setEncoding('utf-8')
-    proc.stderr.setEncoding('utf-8')
+    const [err, logLines] = await execFile(cmd)
+    const lastLine = logLines[logLines.length - 1]
+    assert.strictEqual(err, null)
+    assert.strictEqual(lastLine, 'gyp info ok', 'should end in ok')
+    assert.strictEqual(runHello(), 'world')
   })
 
-  it('build simple addon in path with non-ascii characters', function (done) {
+  it('build simple addon in path with non-ascii characters', async function () {
     if (!checkCharmapValid()) {
       return this.skip('python console app can\'t encode non-ascii character.')
     }
 
-    const testDirNames = {
+    // Select non-ascii characters by current encoding
+    const testDirName = {
       cp936: '文件夹',
       cp1252: 'Latīna',
       cp932: 'フォルダ'
-    }
-    // Select non-ascii characters by current encoding
-    const testDirName = testDirNames[getEncoding()]
+    }[getEncoding()]
     // If encoding is UTF-8 or other then no need to test
     if (!testDirName) {
       return this.skip('no need to test')
@@ -105,46 +105,30 @@ describe('addon', function () {
       '--loglevel=verbose',
       '-nodedir=' + testNodeDir
     ]
-    const proc = execFile(process.execPath, cmd, function (err, stdout, stderr) {
-      try {
-        fs.unlink(testNodeDir)
-      } catch (err) {
-        assert.fail(err)
-      }
-
-      const logLines = stderr.toString().trim().split(/\r?\n/)
-      const lastLine = logLines[logLines.length - 1]
-      assert.strictEqual(err, null)
-      assert.strictEqual(lastLine, 'gyp info ok', 'should end in ok')
-      assert.strictEqual(runHello().trim(), 'world')
-      done()
-    })
-    proc.stdout.setEncoding('utf-8')
-    proc.stderr.setEncoding('utf-8')
-  })
-
-  it('addon works with renamed host executable', function (done) {
-    // No `fs.copyFileSync` before node8.
-    if (process.version.substr(1).split('.')[0] < 8) {
-      return this.skip('skipping test for old node version')
+    const [err, logLines] = await execFile(cmd)
+    try {
+      fs.unlink(testNodeDir)
+    } catch (err) {
+      assert.fail(err)
     }
+    const lastLine = logLines[logLines.length - 1]
+    assert.strictEqual(err, null)
+    assert.strictEqual(lastLine, 'gyp info ok', 'should end in ok')
+    assert.strictEqual(runHello(), 'world')
+  })
 
+  it('addon works with renamed host executable', async function () {
     this.timeout(300000)
 
     const notNodePath = path.join(os.tmpdir(), 'notnode' + path.extname(process.execPath))
     fs.copyFileSync(process.execPath, notNodePath)
 
     const cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose']
-    const proc = execFile(process.execPath, cmd, function (err, stdout, stderr) {
-      const logLines = stderr.toString().trim().split(/\r?\n/)
-      const lastLine = logLines[logLines.length - 1]
-      assert.strictEqual(err, null)
-      assert.strictEqual(lastLine, 'gyp info ok', 'should end in ok')
-      assert.strictEqual(runHello(notNodePath).trim(), 'world')
-      fs.unlinkSync(notNodePath)
-      done()
-    })
-    proc.stdout.setEncoding('utf-8')
-    proc.stderr.setEncoding('utf-8')
+    const [err, logLines] = await execFile(cmd)
+    const lastLine = logLines[logLines.length - 1]
+    assert.strictEqual(err, null)
+    assert.strictEqual(lastLine, 'gyp info ok', 'should end in ok')
+    assert.strictEqual(runHello(notNodePath), 'world')
+    fs.unlinkSync(notNodePath)
   })
 })
diff --git a/test/test-configure-python.js b/test/test-configure-python.js
index fd8e8d24ef..eee230a496 100644
--- a/test/test-configure-python.js
+++ b/test/test-configure-python.js
@@ -3,10 +3,10 @@
 const { describe, it } = require('mocha')
 const assert = require('assert')
 const path = require('path')
-const devDir = require('./common').devDir()
+const { devDir } = require('./common')
 const gyp = require('../lib/node-gyp')
-const log = require('../lib/log')
 const requireInject = require('require-inject')
+
 const configure = requireInject('../lib/configure', {
   'graceful-fs': {
     openSync: () => 0,
@@ -19,8 +19,6 @@ const configure = requireInject('../lib/configure', {
   }
 })
 
-log.logger.stream = null
-
 const EXPECTED_PYPATH = path.join(__dirname, '..', 'gyp', 'pylib')
 const SEPARATOR = process.platform === 'win32' ? ';' : ':'
 const SPAWN_RESULT = cb => ({ on: function () { cb() } })
diff --git a/test/test-create-config-gypi.js b/test/test-create-config-gypi.js
index 725819b6aa..3c77b87859 100644
--- a/test/test-create-config-gypi.js
+++ b/test/test-create-config-gypi.js
@@ -4,8 +4,7 @@ const path = require('path')
 const { describe, it } = require('mocha')
 const assert = require('assert')
 const gyp = require('../lib/node-gyp')
-const createConfigGypi = require('../lib/create-config-gypi')
-const { parseConfigGypi, getCurrentConfigGypi } = createConfigGypi.test
+const { parseConfigGypi, getCurrentConfigGypi } = require('../lib/create-config-gypi')
 
 describe('create-config-gypi', function () {
   it('config.gypi with no options', async function () {
diff --git a/test/test-download.js b/test/test-download.js
index 5b65641f5d..e8e9df4a76 100644
--- a/test/test-download.js
+++ b/test/test-download.js
@@ -7,14 +7,11 @@ const path = require('path')
 const http = require('http')
 const https = require('https')
 const install = require('../lib/install')
-const semver = require('semver')
-const devDir = require('./common').devDir()
+const { download, readCAFile } = require('../lib/download')
+const { FULL_TEST, devDir } = require('./common')
 const gyp = require('../lib/node-gyp')
-const log = require('../lib/log')
 const certs = require('./fixtures/certs')
 
-log.logger.stream = null
-
 describe('download', function () {
   it('download over http', async function () {
     const server = http.createServer((req, res) => {
@@ -32,7 +29,7 @@ describe('download', function () {
       version: '42'
     }
     const url = `http://${host}:${port}`
-    const res = await install.test.download(gyp, url)
+    const res = await download(gyp, url)
     assert.strictEqual(await res.text(), 'ok')
   })
 
@@ -42,7 +39,7 @@ describe('download', function () {
     const cert = certs['server.crt']
     const key = certs['server.key']
     await fs.writeFile(cafile, cacontents, 'utf8')
-    const ca = await install.test.readCAFile(cafile)
+    const ca = await readCAFile(cafile)
 
     assert.strictEqual(ca.length, 1)
 
@@ -67,7 +64,7 @@ describe('download', function () {
       version: '42'
     }
     const url = `https://${host}:${port}`
-    const res = await install.test.download(gyp, url)
+    const res = await download(gyp, url)
     assert.strictEqual(await res.text(), 'ok')
   })
 
@@ -98,7 +95,7 @@ describe('download', function () {
       version: '42'
     }
     const url = `http://${host}:${port}`
-    const res = await install.test.download(gyp, url)
+    const res = await download(gyp, url)
     assert.strictEqual(await res.text(), 'proxy ok')
   })
 
@@ -129,7 +126,7 @@ describe('download', function () {
       version: '42'
     }
     const url = `http://${host}:${port}`
-    const res = await install.test.download(gyp, url)
+    const res = await download(gyp, url)
     assert.strictEqual(await res.text(), 'ok')
   })
 
@@ -138,7 +135,7 @@ describe('download', function () {
       opts: { cafile: 'no.such.file' }
     }
     try {
-      await install.test.download(gyp, {}, 'http://bad/')
+      await download(gyp, {}, 'http://bad/')
     } catch (e) {
       assert.ok(/no.such.file/.test(e.message))
     }
@@ -151,7 +148,7 @@ describe('download', function () {
     after(async () => {
       await fs.unlink(cafile)
     })
-    const cas = await install.test.readCAFile(path.join(__dirname, 'fixtures/ca-bundle.crt'))
+    const cas = await readCAFile(path.join(__dirname, 'fixtures/ca-bundle.crt'))
     assert.strictEqual(cas.length, 2)
     assert.notStrictEqual(cas[0], cas[1])
   })
@@ -159,10 +156,7 @@ describe('download', function () {
   // only run this test if we are running a version of Node with predictable version path behavior
 
   it('download headers (actual)', async function () {
-    if (process.env.FAST_TEST ||
-        process.release.name !== 'node' ||
-        semver.prerelease(process.version) !== null ||
-        semver.satisfies(process.version, '<10')) {
+    if (!FULL_TEST) {
       return this.skip('Skipping actual download of headers due to test environment configuration')
     }
 
@@ -174,7 +168,6 @@ describe('download', function () {
     const prog = gyp()
     prog.parseArgv([])
     prog.devDir = devDir
-    log.level = 'warn'
     await install(prog, [])
 
     const data = await fs.readFile(path.join(expectedDir, 'installVersion'), 'utf8')
diff --git a/test/test-find-accessible-sync.js b/test/test-find-accessible-sync.js
index 9ec12dbb33..0c8a5ddf07 100644
--- a/test/test-find-accessible-sync.js
+++ b/test/test-find-accessible-sync.js
@@ -4,7 +4,7 @@ const { describe, it } = require('mocha')
 const assert = require('assert')
 const path = require('path')
 const requireInject = require('require-inject')
-const configure = requireInject('../lib/configure', {
+const { findAccessibleSync } = requireInject('../lib/util', {
   'graceful-fs': {
     closeSync: function () { return undefined },
     openSync: function (path) {
@@ -31,43 +31,43 @@ const readableFiles = [
 describe('find-accessible-sync', function () {
   it('find accessible - empty array', function () {
     const candidates = []
-    const found = configure.test.findAccessibleSync('test', dir, candidates)
+    const found = findAccessibleSync('test', dir, candidates)
     assert.strictEqual(found, undefined)
   })
 
   it('find accessible - single item array, readable', function () {
     const candidates = [readableFile]
-    const found = configure.test.findAccessibleSync('test', dir, candidates)
+    const found = findAccessibleSync('test', dir, candidates)
     assert.strictEqual(found, path.resolve(dir, readableFile))
   })
 
   it('find accessible - single item array, readable in subdir', function () {
     const candidates = [readableFileInDir]
-    const found = configure.test.findAccessibleSync('test', dir, candidates)
+    const found = findAccessibleSync('test', dir, candidates)
     assert.strictEqual(found, path.resolve(dir, readableFileInDir))
   })
 
   it('find accessible - single item array, unreadable', function () {
     const candidates = ['unreadable_file']
-    const found = configure.test.findAccessibleSync('test', dir, candidates)
+    const found = findAccessibleSync('test', dir, candidates)
     assert.strictEqual(found, undefined)
   })
 
   it('find accessible - multi item array, no matches', function () {
     const candidates = ['non_existent_file', 'unreadable_file']
-    const found = configure.test.findAccessibleSync('test', dir, candidates)
+    const found = findAccessibleSync('test', dir, candidates)
     assert.strictEqual(found, undefined)
   })
 
   it('find accessible - multi item array, single match', function () {
     const candidates = ['non_existent_file', readableFile]
-    const found = configure.test.findAccessibleSync('test', dir, candidates)
+    const found = findAccessibleSync('test', dir, candidates)
     assert.strictEqual(found, path.resolve(dir, readableFile))
   })
 
   it('find accessible - multi item array, return first match', function () {
     const candidates = ['non_existent_file', anotherReadableFile, readableFile]
-    const found = configure.test.findAccessibleSync('test', dir, candidates)
+    const found = findAccessibleSync('test', dir, candidates)
     assert.strictEqual(found, path.resolve(dir, anotherReadableFile))
   })
 })
diff --git a/test/test-install.js b/test/test-install.js
index 3fc39b7c2f..d6694270bb 100644
--- a/test/test-install.js
+++ b/test/test-install.js
@@ -1,23 +1,22 @@
 'use strict'
 
-const { describe, it, after } = require('mocha')
+const { describe, it, afterEach, beforeEach } = require('mocha')
 const { rm, mkdtemp } = require('fs/promises')
 const { createWriteStream } = require('fs')
 const assert = require('assert')
 const path = require('path')
 const os = require('os')
-const semver = require('semver')
 const { pipeline: streamPipeline } = require('stream/promises')
 const requireInject = require('require-inject')
+const { FULL_TEST } = require('./common')
 const gyp = require('../lib/node-gyp')
-
-const createInstall = (mocks = {}) => requireInject('../lib/install', mocks).test
-const { download, install } = createInstall()
+const install = require('../lib/install')
+const { download } = require('../lib/download')
 
 describe('install', function () {
   it('EACCES retry once', async () => {
     let statCalled = 0
-    const mockInstall = createInstall({
+    const mockInstall = requireInject('../lib/install', {
       'graceful-fs': {
         promises: {
           stat (_) {
@@ -35,7 +34,7 @@ describe('install', function () {
         ensure: true
       },
       commands: {
-        install: (...args) => mockInstall.install(Gyp, ...args),
+        install: (...args) => mockInstall(Gyp, ...args),
         remove: async () => {}
       }
     }
@@ -54,79 +53,58 @@ describe('install', function () {
     }
   })
 
-  // only run these tests if we are running a version of Node with predictable version path behavior
-  const skipParallelInstallTests = process.env.FAST_TEST ||
-    process.release.name !== 'node' ||
-    semver.prerelease(process.version) !== null ||
-    semver.satisfies(process.version, '<10')
-
-  async function parallelInstallsTest (test, devDir, prog) {
-    if (skipParallelInstallTests) {
-      return test.skip('Skipping parallel installs test due to test environment configuration')
-    }
+  describe('parallel', function () {
+    let prog
 
-    after(async () => {
-      await rm(devDir, { recursive: true, force: true })
+    beforeEach(async () => {
+      prog = gyp()
+      prog.parseArgv([])
+      prog.devDir = await mkdtemp(path.join(os.tmpdir(), 'node-gyp-test-'))
     })
 
-    const expectedDir = path.join(devDir, process.version.replace(/^v/, ''))
-    await rm(expectedDir, { recursive: true, force: true })
-
-    await Promise.all([
-      install(prog, []),
-      install(prog, []),
-      install(prog, []),
-      install(prog, []),
-      install(prog, []),
-      install(prog, []),
-      install(prog, []),
-      install(prog, []),
-      install(prog, []),
-      install(prog, [])
-    ])
-  }
-
-  it('parallel installs (ensure=true)', async function () {
-    this.timeout(600000)
-
-    const devDir = await mkdtemp(path.join(os.tmpdir(), 'node-gyp-test-'))
-
-    const prog = gyp()
-    prog.parseArgv([])
-    prog.devDir = devDir
-    prog.opts.ensure = true
-
-    await parallelInstallsTest(this, devDir, prog)
-  })
-
-  it('parallel installs (ensure=false)', async function () {
-    this.timeout(600000)
-
-    const devDir = await mkdtemp(path.join(os.tmpdir(), 'node-gyp-test-'))
-
-    const prog = gyp()
-    prog.parseArgv([])
-    prog.devDir = devDir
-    prog.opts.ensure = false
-
-    await parallelInstallsTest(this, devDir, prog)
-  })
+    afterEach(async () => {
+      await rm(prog.devDir, { recursive: true, force: true })
+      prog = null
+    })
 
-  it('parallel installs (tarball)', async function () {
-    this.timeout(600000)
+    const runIt = (name, fn) => {
+      // only run these tests if we are running a version of Node with predictable version path behavior
+      if (!FULL_TEST) {
+        return it.skip('Skipping parallel installs test due to test environment configuration')
+      }
 
-    const devDir = await mkdtemp(path.join(os.tmpdir(), 'node-gyp-test-'))
+      return it(name, async function () {
+        this.timeout(600000)
+        await fn.call(this)
+        const expectedDir = path.join(prog.devDir, process.version.replace(/^v/, ''))
+        await rm(expectedDir, { recursive: true, force: true })
+        await Promise.all([
+          install(prog, []),
+          install(prog, []),
+          install(prog, []),
+          install(prog, []),
+          install(prog, []),
+          install(prog, []),
+          install(prog, []),
+          install(prog, []),
+          install(prog, []),
+          install(prog, [])
+        ])
+      })
+    }
 
-    const prog = gyp()
-    prog.parseArgv([])
-    prog.devDir = devDir
-    prog.opts.tarball = path.join(devDir, 'node-headers.tar.gz')
+    runIt('ensure=true', async function () {
+      prog.opts.ensure = true
+    })
 
-    await streamPipeline(
-      (await download(prog, `https://nodejs.org/dist/${process.version}/node-${process.version}.tar.gz`)).body,
-      createWriteStream(prog.opts.tarball)
-    )
+    runIt('ensure=false', async function () {
+      prog.opts.ensure = false
+    })
 
-    await parallelInstallsTest(this, devDir, prog)
+    runIt('tarball', async function () {
+      prog.opts.tarball = path.join(prog.devDir, 'node-headers.tar.gz')
+      const dl = await download(prog, `https://nodejs.org/dist/${process.version}/node-${process.version}.tar.gz`)
+      await streamPipeline(dl.body, createWriteStream(prog.opts.tarball))
+    })
   })
 })

From 864a979930cf0ef5ad64bc887b901fa8955d058f Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Sat, 28 Oct 2023 16:07:44 -0700
Subject: [PATCH 398/551] feat!: use .npmignore file to limit which files are
 published (#2921)

* feat!: use package.json files to limit which files are published

Fixes: #2372

* Use npmignore instead of package.json#files

* Add update-gyp.py to npmignore

* Add install to pack test

* Use output var for pack dir

* Move existing .gitignore entries to .npmignore

* Sort git and npm ignores

* Update and cleanup workflows
---
 .github/workflows/tests.yml         | 78 +++++++++++++++++++++--------
 .github/workflows/visual-studio.yml |  5 +-
 .gitignore                          |  7 +--
 .npmignore                          | 15 ++++++
 4 files changed, 76 insertions(+), 29 deletions(-)
 create mode 100644 .npmignore

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 4aa041df9e..fde4aacfc1 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -1,5 +1,5 @@
 # https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources
-# TODO: Line 48, enable pytest --doctest-modules
+# TODO: add `python -m pytest --doctest-modules`
 
 name: Tests
 on:
@@ -12,44 +12,82 @@ permissions:
   contents: read # to fetch code (actions/checkout)
 
 jobs:
-  Lint_Python:
+  lint-python:
+    name: Lint Python
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v4
     - run: pip install --user ruff
     - run: ruff --output-format=github --select="E,F,PLC,PLE,UP,W,YTT" --ignore="E721,PLC1901,S101,UP031" --target-version=py38 .
-  Lint_JS:
+  
+  lint-js:
+    name: Lint JS
     runs-on: ubuntu-latest
     steps:
     - name: Checkout Repository
       uses: actions/checkout@v4
     - name: Use Node.js 20.x
-      uses: actions/setup-node@v3
+      uses: actions/setup-node@v4
       with:
         node-version: 20.x
     - name: Install Dependencies
-      run: npm install --no-progress
+      run: npm install
     - name: Lint
       run: npm run lint
-  Engines:
+
+  check-engines:
+    name: Check Engines
     runs-on: ubuntu-latest
     steps:
     - name: Checkout Repository
       uses: actions/checkout@v4
     - name: Use Node.js 20.x
-      uses: actions/setup-node@v3
+      uses: actions/setup-node@v4
       with:
         node-version: 20.x
     - name: Install Dependencies
-      run: |
-        npm install --no-progress
+      run: npm install
     - name: Check Engines
       run: |
         # TODO: move this to its own action
         npm install @npmcli/arborist@7 semver@7 --no-save
         node .github/scripts/check-engines.js
-  Tests:
-    needs: Lint_Python  # Lint_Python takes ~5 seconds, so wait for it to pass before running the full matrix of tests.
+
+  test-pack:
+    name: Test Pack
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout Repository
+      uses: actions/checkout@v4
+    - name: Use Node.js 20.x
+      uses: actions/setup-node@v4
+      with:
+        node-version: 20.x
+    - name: Update npm
+      run: npm install npm@latest -g
+    - name: Install Dependencies
+      run: npm install
+    - name: Pack
+      id: pack
+      env:
+        NODE_GYP_TEMP_DIR: '${{ runner.temp }}/node-gyp'
+      run: |
+        mkdir -p $NODE_GYP_TEMP_DIR
+        npm pack
+        tar xzf *.tgz -C $NODE_GYP_TEMP_DIR --strip-components=1
+        cp -r test/ $NODE_GYP_TEMP_DIR/test/
+        echo "dir=$NODE_GYP_TEMP_DIR" >> "$GITHUB_OUTPUT"
+    - name: Test
+      working-directory: ${{ steps.pack.outputs.dir }}
+      env:
+        FULL_TEST: '1'
+      run: |
+        npm install
+        npm test
+
+  tests:
+    # lint-python takes ~5 seconds, so wait for it to pass before running the full matrix of tests.
+    needs: [lint-python] 
     strategy:
       fail-fast: false
       max-parallel: 15
@@ -63,7 +101,7 @@ jobs:
       - name: Checkout Repository
         uses: actions/checkout@v4
       - name: Use Node.js ${{ matrix.node }}
-        uses: actions/setup-node@v3
+        uses: actions/setup-node@v4
         with:
           node-version: ${{ matrix.node }}
       - name: Use Python ${{ matrix.python }}
@@ -74,26 +112,22 @@ jobs:
           PYTHON_VERSION: ${{ matrix.python }}  # Why do this?
       - name: Install Dependencies
         run: |
-          npm install --no-progress
+          npm install
           pip install pytest
-      - name: Set Windows environment
-        if: matrix.os == 'windows'
+      - name: Set Windows Env
+        if: runner.os == 'Windows'
         run: |
           echo 'GYP_MSVS_VERSION=2015' >> $Env:GITHUB_ENV
           echo 'GYP_MSVS_OVERRIDE_PATH=C:\\Dummy' >> $Env:GITHUB_ENV
-      - name: Run Python tests
+      - name: Run Python Tests
         run: python -m pytest
-      # - name: Run doctests with pytest
-      #   run: python -m pytest --doctest-modules
-      - name: Environment Information
-        run: npx envinfo
-      - name: Run Node tests (macOS or Linux)
+      - name: Run Tests (macOS or Linux)
         if: runner.os != 'Windows'
         shell: bash
         run: npm test --python="${pythonLocation}/python"
         env:
           FULL_TEST: ${{ (matrix.node == '20.x' && matrix.python == '3.12') && '1' || '0' }}
-      - name: Run tests (Windows)
+      - name: Run Tests (Windows)
         if: runner.os == 'Windows'
         shell: pwsh
         run: npm run test --python="${env:pythonLocation}\\python.exe"
diff --git a/.github/workflows/visual-studio.yml b/.github/workflows/visual-studio.yml
index 8b796256da..19993a57f9 100644
--- a/.github/workflows/visual-studio.yml
+++ b/.github/workflows/visual-studio.yml
@@ -26,10 +26,7 @@ jobs:
       - name: Checkout Repository
         uses: actions/checkout@v4
       - name: Install Dependencies
-        run: |
-          npm install --no-progress
-      - name: Environment Information
-        run: npx envinfo
+        run: npm install
       - name: Run Node tests
         shell: pwsh
         run: |
diff --git a/.gitignore b/.gitignore
index e906ca7280..e588699b7a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,7 +1,8 @@
+.ncu
+.nyc_output
 *.swp
 gyp/test
 node_modules
-test/.node-gyp
-.ncu
-.nyc_output
+node-gyp-*.tgz
 package-lock.json
+test/.node-gyp
diff --git a/.npmignore b/.npmignore
new file mode 100644
index 0000000000..8f95594d44
--- /dev/null
+++ b/.npmignore
@@ -0,0 +1,15 @@
+.ncu
+.nyc_output
+*.swp
+/.github/
+/docs/
+/gyp/.github/
+/gyp/*.md
+/gyp/AUTHORS
+/gyp/test
+/gyp/tools/
+/node-gyp-*.tgz
+/test/
+/test/.node-gyp
+/update-gyp.py
+node-gyp-*.tgz

From 3032e1061cc2b7b49f83c397d385bafddc6b0214 Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Sat, 28 Oct 2023 16:24:46 -0700
Subject: [PATCH 399/551] chore: run tests after release please PR

---
 .github/workflows/release-please.yml | 96 +++++++++++++++-------------
 .github/workflows/tests.yml          |  1 +
 2 files changed, 53 insertions(+), 44 deletions(-)

diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml
index cef38bcc06..f942e74122 100644
--- a/.github/workflows/release-please.yml
+++ b/.github/workflows/release-please.yml
@@ -4,57 +4,65 @@ on:
   push:
     branches:
       - main
+  pull_request:
 
 jobs:
   release-please:
+    outputs:
+      pr: ${{ steps.release.outputs.pr }}
     permissions:
       contents: write # to create release commit (google-github-actions/release-please-action)
       pull-requests: write # to create release PR (google-github-actions/release-please-action)
 
     runs-on: ubuntu-latest
     steps:
-      - uses: google-github-actions/release-please-action@v2
-        id: release
-        with:
-          package-name: node-gyp
-          release-type: node
-          changelog-types: >
-            [{"type":"feat","section":"Features","hidden":false},
-            {"type":"fix","section":"Bug Fixes","hidden":false},
-            {"type":"bin","section":"Core","hidden":false},
-            {"type":"gyp","section":"Core","hidden":false},
-            {"type":"lib","section":"Core","hidden":false},
-            {"type":"src","section":"Core","hidden":false},
-            {"type":"test","section":"Tests","hidden":false},
-            {"type":"build","section":"Core","hidden":false},
-            {"type":"clean","section":"Core","hidden":false},
-            {"type":"configure","section":"Core","hidden":false},
-            {"type":"install","section":"Core","hidden":false},
-            {"type":"list","section":"Core","hidden":false},
-            {"type":"rebuild","section":"Core","hidden":false},
-            {"type":"remove","section":"Core","hidden":false},
-            {"type":"deps","section":"Core","hidden":false},
-            {"type":"python","section":"Core","hidden":false},
-            {"type":"lin","section":"Core","hidden":false},
-            {"type":"linux","section":"Core","hidden":false},
-            {"type":"mac","section":"Core","hidden":false},
-            {"type":"macos","section":"Core","hidden":false},
-            {"type":"win","section":"Core","hidden":false},
-            {"type":"windows","section":"Core","hidden":false},
-            {"type":"zos","section":"Core","hidden":false},
-            {"type":"doc","section":"Doc","hidden":false},
-            {"type":"docs","section":"Doc","hidden":false},
-            {"type":"readme","section":"Doc","hidden":false},
-            {"type":"chore","section":"Miscellaneous","hidden":false},
-            {"type":"refactor","section":"Miscellaneous","hidden":false},
-            {"type":"ci","section":"Miscellaneous","hidden":false},
-            {"type":"meta","section":"Miscellaneous","hidden":false}]
+    - uses: google-github-actions/release-please-action@v2
+      id: release
+      with:
+        package-name: node-gyp
+        release-type: node
+        changelog-types: >
+          [{"type":"feat","section":"Features","hidden":false},
+          {"type":"fix","section":"Bug Fixes","hidden":false},
+          {"type":"bin","section":"Core","hidden":false},
+          {"type":"gyp","section":"Core","hidden":false},
+          {"type":"lib","section":"Core","hidden":false},
+          {"type":"src","section":"Core","hidden":false},
+          {"type":"test","section":"Tests","hidden":false},
+          {"type":"build","section":"Core","hidden":false},
+          {"type":"clean","section":"Core","hidden":false},
+          {"type":"configure","section":"Core","hidden":false},
+          {"type":"install","section":"Core","hidden":false},
+          {"type":"list","section":"Core","hidden":false},
+          {"type":"rebuild","section":"Core","hidden":false},
+          {"type":"remove","section":"Core","hidden":false},
+          {"type":"deps","section":"Core","hidden":false},
+          {"type":"python","section":"Core","hidden":false},
+          {"type":"lin","section":"Core","hidden":false},
+          {"type":"linux","section":"Core","hidden":false},
+          {"type":"mac","section":"Core","hidden":false},
+          {"type":"macos","section":"Core","hidden":false},
+          {"type":"win","section":"Core","hidden":false},
+          {"type":"windows","section":"Core","hidden":false},
+          {"type":"zos","section":"Core","hidden":false},
+          {"type":"doc","section":"Doc","hidden":false},
+          {"type":"docs","section":"Doc","hidden":false},
+          {"type":"readme","section":"Doc","hidden":false},
+          {"type":"chore","section":"Miscellaneous","hidden":false},
+          {"type":"refactor","section":"Miscellaneous","hidden":false},
+          {"type":"ci","section":"Miscellaneous","hidden":false},
+          {"type":"meta","section":"Miscellaneous","hidden":false}]
+        # Standard Conventional Commits: `feat` and `fix`
+        # node-gyp subdirectories: `bin`, `gyp`, `lib`, `src`, `test`
+        # node-gyp subcommands: `build`, `clean`, `configure`, `install`, `list`, `rebuild`, `remove`
+        # Core abstract category: `deps`
+        # Languages/platforms: `python`, `lin`, `linux`, `mac`, `macos`, `win`, `window`, `zos`
+        # Documentation: `doc`, `docs`, `readme`
+        # Standard Conventional Commits: `chore` (under "Miscellaneous")
+        # Miscellaneous abstract categories: `refactor`, `ci`, `meta`
 
-          # Standard Conventional Commits: `feat` and `fix`
-          # node-gyp subdirectories: `bin`, `gyp`, `lib`, `src`, `test`
-          # node-gyp subcommands: `build`, `clean`, `configure`, `install`, `list`, `rebuild`, `remove`
-          # Core abstract category: `deps`
-          # Languages/platforms: `python`, `lin`, `linux`, `mac`, `macos`, `win`, `window`, `zos`
-          # Documentation: `doc`, `docs`, `readme`
-          # Standard Conventional Commits: `chore` (under "Miscellaneous")
-          # Miscellaneous abstract categories: `refactor`, `ci`, `meta`
+  test:
+    name: Release Test
+    needs: [ release-please ]
+    if: needs.release-please.outputs.pr
+    uses: ./.github/workflows/tests.yml
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index fde4aacfc1..5017bcd9f8 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -7,6 +7,7 @@ on:
     branches: [ main ]
   pull_request:
     branches: [ main ]
+  workflow_call:
 
 permissions:
   contents: read # to fetch code (actions/checkout)

From 4c302cad0eba96f3dfc2bb3d8908c3b1ad48bf43 Mon Sep 17 00:00:00 2001
From: "github-actions[bot]"
 <41898282+github-actions[bot]@users.noreply.github.com>
Date: Sat, 28 Oct 2023 16:47:34 -0700
Subject: [PATCH 400/551] chore: release 10.0.0 (#2920)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
---
 CHANGELOG.md | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 package.json |  2 +-
 2 files changed, 65 insertions(+), 1 deletion(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index edf79f7025..6473e09605 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,69 @@
 # Changelog
 
+## [10.0.0](https://www.github.com/nodejs/node-gyp/compare/v9.4.0...v10.0.0) (2023-10-28)
+
+
+### ⚠ BREAKING CHANGES
+
+* use .npmignore file to limit which files are published (#2921)
+* the `Gyp` class exported is now created using ECMAScript classes and therefore might have small differences to classes that were previously created with `util.inherits`.
+* All internal functions have been coverted to return promises and no longer accept callbacks. This is not a breaking change for users but may be breaking to consumers of `node-gyp` if you are requiring internal functions directly.
+* `node-gyp` now supports node `^16.14.0 || >=18.0.0`
+* update engines.node to ^14.17.0 || ^16.13.0 || >=18.0.0
+
+### Features
+
+* convert all internal functions to async/await ([355622f](https://www.github.com/nodejs/node-gyp/commit/355622f4aac3bd3056b9e03aac5fa2f42a4b3576))
+* convert internal classes from util.inherits to classes ([d52997e](https://www.github.com/nodejs/node-gyp/commit/d52997e975b9da6e0cea3d9b99873e9ddc768679))
+* drop node 14 support ([#2929](https://www.github.com/nodejs/node-gyp/issues/2929)) ([1b3bd34](https://www.github.com/nodejs/node-gyp/commit/1b3bd341b40f384988d03207ce8187e93ba609bc))
+* drop rimraf dependency ([4a50fe3](https://www.github.com/nodejs/node-gyp/commit/4a50fe31574217c4b2a798fc72b19947a64ceea1))
+* **gyp:** update gyp to v0.16.1 ([#2923](https://www.github.com/nodejs/node-gyp/issues/2923)) ([707927c](https://www.github.com/nodejs/node-gyp/commit/707927cd579205ef2b4b17e61c1cce24c056b452))
+* replace npmlog with proc-log ([4a50fe3](https://www.github.com/nodejs/node-gyp/commit/4a50fe31574217c4b2a798fc72b19947a64ceea1))
+* update engines.node to ^14.17.0 || ^16.13.0 || >=18.0.0 ([4a50fe3](https://www.github.com/nodejs/node-gyp/commit/4a50fe31574217c4b2a798fc72b19947a64ceea1))
+* use .npmignore file to limit which files are published ([#2921](https://www.github.com/nodejs/node-gyp/issues/2921)) ([864a979](https://www.github.com/nodejs/node-gyp/commit/864a979930cf0ef5ad64bc887b901fa8955d058f))
+
+
+### Bug Fixes
+
+* create Python symlink only during builds, and clean it up after ([#2721](https://www.github.com/nodejs/node-gyp/issues/2721)) ([0f1f667](https://www.github.com/nodejs/node-gyp/commit/0f1f667b737d21905e283df100a2cb639993562a))
+* promisify build command ([4a50fe3](https://www.github.com/nodejs/node-gyp/commit/4a50fe31574217c4b2a798fc72b19947a64ceea1))
+* use fs/promises in favor of fs.promises ([4a50fe3](https://www.github.com/nodejs/node-gyp/commit/4a50fe31574217c4b2a798fc72b19947a64ceea1))
+
+
+### Tests
+
+* increase mocha timeout ([#2887](https://www.github.com/nodejs/node-gyp/issues/2887)) ([445c28f](https://www.github.com/nodejs/node-gyp/commit/445c28fabc5fbdf9c3bb3341fb70660a3530f6ad))
+* update expired certs ([#2908](https://www.github.com/nodejs/node-gyp/issues/2908)) ([5746691](https://www.github.com/nodejs/node-gyp/commit/5746691a36f7b37019d4b8d4e9616aec43d20410))
+
+
+### Doc
+
+* Add note about Python symlinks (PR 2362) to CHANGELOG.md for 9.1.0 ([#2783](https://www.github.com/nodejs/node-gyp/issues/2783)) ([b3d41ae](https://www.github.com/nodejs/node-gyp/commit/b3d41aeb737ddd54cc292f363abc561dcc0a614e))
+* README.md Do not hardcode the supported versions of Python ([#2880](https://www.github.com/nodejs/node-gyp/issues/2880)) ([bb93b94](https://www.github.com/nodejs/node-gyp/commit/bb93b946a9c74934b59164deb52128cf913c97d5))
+* update applicable GitHub links from master to main ([#2843](https://www.github.com/nodejs/node-gyp/issues/2843)) ([d644ce4](https://www.github.com/nodejs/node-gyp/commit/d644ce48311edf090d0e920ad449e5766c757933))
+* Update windows installation instructions in README.md ([#2882](https://www.github.com/nodejs/node-gyp/issues/2882)) ([c9caa2e](https://www.github.com/nodejs/node-gyp/commit/c9caa2ecf3c7deae68444ce8fabb32d2dca651cd))
+
+
+### Core
+
+* find python checks order changed on windows ([#2872](https://www.github.com/nodejs/node-gyp/issues/2872)) ([b030555](https://www.github.com/nodejs/node-gyp/commit/b030555cdb754d9c23906e7e707115cd077bbf76))
+* glob@10.3.10 ([#2926](https://www.github.com/nodejs/node-gyp/issues/2926)) ([4bef1ec](https://www.github.com/nodejs/node-gyp/commit/4bef1ecc7554097d92beb397fbe1a546c5227545))
+* glob@8.0.3 ([4a50fe3](https://www.github.com/nodejs/node-gyp/commit/4a50fe31574217c4b2a798fc72b19947a64ceea1))
+* make-fetch-happen@13.0.0 ([#2927](https://www.github.com/nodejs/node-gyp/issues/2927)) ([059bb6f](https://www.github.com/nodejs/node-gyp/commit/059bb6fd41bb50955a9efbd97887773d60d53221))
+* nopt@^7.0.0 ([4a50fe3](https://www.github.com/nodejs/node-gyp/commit/4a50fe31574217c4b2a798fc72b19947a64ceea1))
+* standard@17.0.0 and fix linting errors ([4a50fe3](https://www.github.com/nodejs/node-gyp/commit/4a50fe31574217c4b2a798fc72b19947a64ceea1))
+* which@3.0.0 ([4a50fe3](https://www.github.com/nodejs/node-gyp/commit/4a50fe31574217c4b2a798fc72b19947a64ceea1))
+* which@4.0.0 ([#2928](https://www.github.com/nodejs/node-gyp/issues/2928)) ([e388255](https://www.github.com/nodejs/node-gyp/commit/e38825531403aabeae7abe58e76867f31b832f36))
+
+
+### Miscellaneous
+
+* add check engines script to CI ([#2922](https://www.github.com/nodejs/node-gyp/issues/2922)) ([21a7249](https://www.github.com/nodejs/node-gyp/commit/21a7249b40d8f95e7721e450fd18764adb1648a7))
+* empty commit to add changelog entries from [#2770](https://www.github.com/nodejs/node-gyp/issues/2770) ([4a50fe3](https://www.github.com/nodejs/node-gyp/commit/4a50fe31574217c4b2a798fc72b19947a64ceea1))
+* GitHub Workflows security hardening ([#2740](https://www.github.com/nodejs/node-gyp/issues/2740)) ([26683e9](https://www.github.com/nodejs/node-gyp/commit/26683e993df038fb94d89f2276f3535e4522d79a))
+* misc testing fixes ([#2930](https://www.github.com/nodejs/node-gyp/issues/2930)) ([4e493d4](https://www.github.com/nodejs/node-gyp/commit/4e493d4fb262d12ac52c84979071ccc79e666a1a))
+* run tests after release please PR ([3032e10](https://www.github.com/nodejs/node-gyp/commit/3032e1061cc2b7b49f83c397d385bafddc6b0214))
+
 ## [9.4.0](https://www.github.com/nodejs/node-gyp/compare/v9.3.1...v9.4.0) (2023-06-12)
 
 
diff --git a/package.json b/package.json
index dadbf1d25a..62139ca33b 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "9.4.0",
+  "version": "10.0.0",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From 7de1f5f32d550d26d48fe4f76aed5866744edcba Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Sun, 29 Oct 2023 10:29:40 -0700
Subject: [PATCH 401/551] chore: add parallel test logging

---
 test/test-install.js | 17 +++++------------
 1 file changed, 5 insertions(+), 12 deletions(-)

diff --git a/test/test-install.js b/test/test-install.js
index d6694270bb..8d9951431d 100644
--- a/test/test-install.js
+++ b/test/test-install.js
@@ -75,21 +75,14 @@ describe('install', function () {
 
       return it(name, async function () {
         this.timeout(600000)
+        const start = Date.now()
         await fn.call(this)
         const expectedDir = path.join(prog.devDir, process.version.replace(/^v/, ''))
         await rm(expectedDir, { recursive: true, force: true })
-        await Promise.all([
-          install(prog, []),
-          install(prog, []),
-          install(prog, []),
-          install(prog, []),
-          install(prog, []),
-          install(prog, []),
-          install(prog, []),
-          install(prog, []),
-          install(prog, []),
-          install(prog, [])
-        ])
+        await Promise.all(new Array(10).fill(0).map(async (_, i) => {
+          await install(prog,[])
+          console.log(`${' '.repeat(8)}${name} ${(i + 1).toString().padEnd(2, ' ')} (${Date.now() - start}ms)`)
+        }))
       })
     }
 

From 4e0ed992566f43abc6e988af091ad07fde04acbf Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Sun, 29 Oct 2023 10:45:50 -0700
Subject: [PATCH 402/551] chore: lint fixes

---
 test/test-install.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test/test-install.js b/test/test-install.js
index 8d9951431d..b578f3dd7f 100644
--- a/test/test-install.js
+++ b/test/test-install.js
@@ -80,7 +80,7 @@ describe('install', function () {
         const expectedDir = path.join(prog.devDir, process.version.replace(/^v/, ''))
         await rm(expectedDir, { recursive: true, force: true })
         await Promise.all(new Array(10).fill(0).map(async (_, i) => {
-          await install(prog,[])
+          await install(prog, [])
           console.log(`${' '.repeat(8)}${name} ${(i + 1).toString().padEnd(2, ' ')} (${Date.now() - start}ms)`)
         }))
       })

From a68586a67d0af238300662cc062422b42820044d Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Sun, 29 Oct 2023 11:06:34 -0700
Subject: [PATCH 403/551] chore: use platform specific timeouts in tests

---
 test/common.js        |  9 +++++++++
 test/test-addon.js    | 18 ++++++++----------
 test/test-download.js |  4 ++--
 test/test-install.js  |  4 ++--
 4 files changed, 21 insertions(+), 14 deletions(-)

diff --git a/test/common.js b/test/common.js
index aef6bacfe1..489502d4dd 100644
--- a/test/common.js
+++ b/test/common.js
@@ -22,3 +22,12 @@ module.exports.FULL_TEST =
   process.env.FULL_TEST === '1' &&
   process.release.name === 'node' &&
   semver.prerelease(process.version) === null
+
+module.exports.platformTimeout = (def, obj) => {
+  for (const [key, value] of Object.entries(obj)) {
+    if (process.platform === key) {
+      return value * 60 * 1000
+    }
+  }
+  return def * 60 * 1000
+}
diff --git a/test/test-addon.js b/test/test-addon.js
index b8d90b65bb..1f4d95ed7e 100644
--- a/test/test-addon.js
+++ b/test/test-addon.js
@@ -7,6 +7,7 @@ const fs = require('graceful-fs')
 const os = require('os')
 const cp = require('child_process')
 const util = require('../lib/util')
+const { platformTimeout } = require('./common')
 
 const addonPath = path.resolve(__dirname, 'node_modules', 'hello_world')
 const nodeGyp = path.resolve(__dirname, '..', 'bin', 'node-gyp.js')
@@ -41,9 +42,9 @@ function checkCharmapValid () {
 }
 
 describe('addon', function () {
-  this.timeout(300000)
-
   it('build simple addon', async function () {
+    this.timeout(platformTimeout(1, { win32: 5 }))
+
     // Set the loglevel otherwise the output disappears when run via 'npm test'
     const cmd = [nodeGyp, 'rebuild', '-C', addonPath, '--loglevel=verbose']
     const [err, logLines] = await execFile(cmd)
@@ -69,15 +70,14 @@ describe('addon', function () {
       return this.skip('no need to test')
     }
 
-    this.timeout(300000)
+    this.timeout(platformTimeout(1, { win32: 5 }))
 
     let data
     const configPath = path.join(addonPath, 'build', 'config.gypi')
     try {
       data = fs.readFileSync(configPath, 'utf8')
     } catch (err) {
-      assert.fail(err)
-      return
+      return assert.fail(err)
     }
     const config = JSON.parse(data.replace(/#.+\n/, ''))
     const nodeDir = config.variables.nodedir
@@ -89,11 +89,9 @@ describe('addon', function () {
       switch (err.code) {
         case 'EEXIST': break
         case 'EPERM':
-          assert.fail(err, null, 'Please try to running console as an administrator')
-          return
+          return assert.fail(err, null, 'Please try to running console as an administrator')
         default:
-          assert.fail(err)
-          return
+          return assert.fail(err)
       }
     }
 
@@ -118,7 +116,7 @@ describe('addon', function () {
   })
 
   it('addon works with renamed host executable', async function () {
-    this.timeout(300000)
+    this.timeout(platformTimeout(1, { win32: 5 }))
 
     const notNodePath = path.join(os.tmpdir(), 'notnode' + path.extname(process.execPath))
     fs.copyFileSync(process.execPath, notNodePath)
diff --git a/test/test-download.js b/test/test-download.js
index e8e9df4a76..1d0f3ab286 100644
--- a/test/test-download.js
+++ b/test/test-download.js
@@ -8,7 +8,7 @@ const http = require('http')
 const https = require('https')
 const install = require('../lib/install')
 const { download, readCAFile } = require('../lib/download')
-const { FULL_TEST, devDir } = require('./common')
+const { FULL_TEST, devDir, platformTimeout } = require('./common')
 const gyp = require('../lib/node-gyp')
 const certs = require('./fixtures/certs')
 
@@ -160,7 +160,7 @@ describe('download', function () {
       return this.skip('Skipping actual download of headers due to test environment configuration')
     }
 
-    this.timeout(300000)
+    this.timeout(platformTimeout(1, { win32: 5 }))
 
     const expectedDir = path.join(devDir, process.version.replace(/^v/, ''))
     await fs.rm(expectedDir, { recursive: true, force: true })
diff --git a/test/test-install.js b/test/test-install.js
index b578f3dd7f..de91b26ad8 100644
--- a/test/test-install.js
+++ b/test/test-install.js
@@ -8,7 +8,7 @@ const path = require('path')
 const os = require('os')
 const { pipeline: streamPipeline } = require('stream/promises')
 const requireInject = require('require-inject')
-const { FULL_TEST } = require('./common')
+const { FULL_TEST, platformTimeout } = require('./common')
 const gyp = require('../lib/node-gyp')
 const install = require('../lib/install')
 const { download } = require('../lib/download')
@@ -74,7 +74,7 @@ describe('install', function () {
       }
 
       return it(name, async function () {
-        this.timeout(600000)
+        this.timeout(platformTimeout(1, { win32: 20 }))
         const start = Date.now()
         await fn.call(this)
         const expectedDir = path.join(prog.devDir, process.version.replace(/^v/, ''))

From b39e6819aa9e2c45107d6e60a4913ca036ebfbfd Mon Sep 17 00:00:00 2001
From: Richard Lau 
Date: Thu, 2 Nov 2023 14:42:09 +0000
Subject: [PATCH 404/551] fix: use local `util` for `findAccessibleSync()`

The `findAccessibleSync()` function is in the local `util` module
instead of Node.js' builtin `util` module.
---
 lib/configure.js | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/configure.js b/lib/configure.js
index 38fd2b9718..8da41a849d 100644
--- a/lib/configure.js
+++ b/lib/configure.js
@@ -8,7 +8,8 @@ const processRelease = require('./process-release')
 const win = process.platform === 'win32'
 const findNodeDirectory = require('./find-node-directory')
 const { createConfigGypi } = require('./create-config-gypi')
-const { format: msgFormat, findAccessibleSync } = require('util')
+const { format: msgFormat } = require('util')
+const { findAccessibleSync } = require('./util')
 const { findPython } = require('./find-python')
 const { findVisualStudio } = win ? require('./find-visualstudio') : {}
 

From da19158e7a02c574d4f6d3d367ee264cb08d47ec Mon Sep 17 00:00:00 2001
From: "github-actions[bot]"
 <41898282+github-actions[bot]@users.noreply.github.com>
Date: Thu, 2 Nov 2023 16:14:57 +0000
Subject: [PATCH 405/551] chore: release 10.0.1

---
 CHANGELOG.md | 14 ++++++++++++++
 package.json |  2 +-
 2 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6473e09605..98315add5e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,19 @@
 # Changelog
 
+### [10.0.1](https://www.github.com/nodejs/node-gyp/compare/v10.0.0...v10.0.1) (2023-11-02)
+
+
+### Bug Fixes
+
+* use local `util` for `findAccessibleSync()` ([b39e681](https://www.github.com/nodejs/node-gyp/commit/b39e6819aa9e2c45107d6e60a4913ca036ebfbfd))
+
+
+### Miscellaneous
+
+* add parallel test logging ([7de1f5f](https://www.github.com/nodejs/node-gyp/commit/7de1f5f32d550d26d48fe4f76aed5866744edcba))
+* lint fixes ([4e0ed99](https://www.github.com/nodejs/node-gyp/commit/4e0ed992566f43abc6e988af091ad07fde04acbf))
+* use platform specific timeouts in tests ([a68586a](https://www.github.com/nodejs/node-gyp/commit/a68586a67d0af238300662cc062422b42820044d))
+
 ## [10.0.0](https://www.github.com/nodejs/node-gyp/compare/v9.4.0...v10.0.0) (2023-10-28)
 
 
diff --git a/package.json b/package.json
index 62139ca33b..80c63f2e72 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "10.0.0",
+  "version": "10.0.1",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From b42e7966177f006f3d1aab1d27885d8372c8ed01 Mon Sep 17 00:00:00 2001
From: Mike McCready <66998419+MikeMcC399@users.noreply.github.com>
Date: Thu, 2 Nov 2023 19:51:24 +0100
Subject: [PATCH 406/551] docs: remove outdated update engines.node reference
 in 10.0.0 changelog

---
 CHANGELOG.md | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 98315add5e..bcd167d9bc 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -23,7 +23,6 @@
 * the `Gyp` class exported is now created using ECMAScript classes and therefore might have small differences to classes that were previously created with `util.inherits`.
 * All internal functions have been coverted to return promises and no longer accept callbacks. This is not a breaking change for users but may be breaking to consumers of `node-gyp` if you are requiring internal functions directly.
 * `node-gyp` now supports node `^16.14.0 || >=18.0.0`
-* update engines.node to ^14.17.0 || ^16.13.0 || >=18.0.0
 
 ### Features
 
@@ -706,11 +705,11 @@ Republish of v5.0.6 with unnecessary tarball removed from pack file.
 * [[`94c39c604e`](https://github.com/nodejs/node-gyp/commit/94c39c604e)] - **gyp**: fix ninja build failure (GYP patch) (Daniel Bevenius) [nodejs/node#12484](https://github.com/nodejs/node/pull/12484)
 * [[`e8ea74e0fa`](https://github.com/nodejs/node-gyp/commit/e8ea74e0fa)] - **tools**: patch gyp to avoid xcrun errors (Ujjwal Sharma) [nodejs/node#21520](https://github.com/nodejs/node/pull/21520)
 * [[`ea9aff44f2`](https://github.com/nodejs/node-gyp/commit/ea9aff44f2)] - **tools**: fix "the the" typos in comments (Masashi Hirano) [nodejs/node#20716](https://github.com/nodejs/node/pull/20716)
-* [[`207e5aa4fd`](https://github.com/nodejs/node-gyp/commit/207e5aa4fd)] - **gyp**: implement LD/LDXX for ninja and FIPS (Sam Roberts) 
+* [[`207e5aa4fd`](https://github.com/nodejs/node-gyp/commit/207e5aa4fd)] - **gyp**: implement LD/LDXX for ninja and FIPS (Sam Roberts)
 * [[`b416c5f4b7`](https://github.com/nodejs/node-gyp/commit/b416c5f4b7)] - **gyp**: enable cctest to use objects (gyp part) (Daniel Bevenius) [nodejs/node#12450](https://github.com/nodejs/node/pull/12450)
 * [[`40692d016b`](https://github.com/nodejs/node-gyp/commit/40692d016b)] - **gyp**: add compile\_commands.json gyp generator (Ben Noordhuis) [nodejs/node#12450](https://github.com/nodejs/node/pull/12450)
 * [[`fc3c4e2b10`](https://github.com/nodejs/node-gyp/commit/fc3c4e2b10)] - **gyp**: float gyp patch for long filenames (Anna Henningsen) [nodejs/node#7963](https://github.com/nodejs/node/pull/7963)
-* [[`8aedbfdef6`](https://github.com/nodejs/node-gyp/commit/8aedbfdef6)] - **gyp**: backport GYP fix to fix AIX shared suffix (Stewart Addison) 
+* [[`8aedbfdef6`](https://github.com/nodejs/node-gyp/commit/8aedbfdef6)] - **gyp**: backport GYP fix to fix AIX shared suffix (Stewart Addison)
 * [[`6cd84b84fc`](https://github.com/nodejs/node-gyp/commit/6cd84b84fc)] - **test**: formatting and minor fixes for execFileSync replacement (Rod Vagg) [#1521](https://github.com/nodejs/node-gyp/pull/1521)
 * [[`60e421363f`](https://github.com/nodejs/node-gyp/commit/60e421363f)] - **test**: added test/processExecSync.js for when execFileSync is not available. (Rohit Hazra) [#1492](https://github.com/nodejs/node-gyp/pull/1492)
 * [[`969447c5bd`](https://github.com/nodejs/node-gyp/commit/969447c5bd)] - **deps**: bump request to 2.8.7, fixes heok/hawk issues (Rohit Hazra) [#1492](https://github.com/nodejs/node-gyp/pull/1492)
@@ -772,7 +771,7 @@ Republish of v5.0.6 with unnecessary tarball removed from pack file.
 
 ## v3.5.0 2017-01-10
 
-* [[`762d19a39e`](https://github.com/nodejs/node-gyp/commit/762d19a39e)] - \[doc\] merge History.md and CHANGELOG.md (Rod Vagg) 
+* [[`762d19a39e`](https://github.com/nodejs/node-gyp/commit/762d19a39e)] - \[doc\] merge History.md and CHANGELOG.md (Rod Vagg)
 * [[`80fc5c3d31`](https://github.com/nodejs/node-gyp/commit/80fc5c3d31)] - Fix deprecated dependency warning (Simone Primarosa) [#1069](https://github.com/nodejs/node-gyp/pull/1069)
 * [[`05c44944fd`](https://github.com/nodejs/node-gyp/commit/05c44944fd)] - Open the build file with universal-newlines mode (Guy Margalit) [#1053](https://github.com/nodejs/node-gyp/pull/1053)
 * [[`37ae7be114`](https://github.com/nodejs/node-gyp/commit/37ae7be114)] - Try python launcher when stock python is python 3. (Ben Noordhuis) [#992](https://github.com/nodejs/node-gyp/pull/992)
@@ -829,7 +828,7 @@ Republish of v5.0.6 with unnecessary tarball removed from pack file.
 * [[`0e2dfda1f3`](https://github.com/nodejs/node-gyp/commit/0e2dfda1f3)] - Fix test/test-options when run through `npm test`. (Ben Noordhuis) [#755](https://github.com/nodejs/node-gyp/pull/755)
 * [[`9bfa0876b4`](https://github.com/nodejs/node-gyp/commit/9bfa0876b4)] - Add support for AIX (Michael Dawson) [#753](https://github.com/nodejs/node-gyp/pull/753)
 * [[`a8d441a0a2`](https://github.com/nodejs/node-gyp/commit/a8d441a0a2)] - Update README for Windows 10 support. (Jason Williams) [#766](https://github.com/nodejs/node-gyp/pull/766)
-* [[`d1d6015276`](https://github.com/nodejs/node-gyp/commit/d1d6015276)] - Update broken links and switch to HTTPS. (andrew morton) 
+* [[`d1d6015276`](https://github.com/nodejs/node-gyp/commit/d1d6015276)] - Update broken links and switch to HTTPS. (andrew morton)
 
 ## v3.1.0 2015-11-14
 

From cff9ac2c3083769a383e00bc60b91562f03116e3 Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Mon, 6 Nov 2023 09:35:09 -0700
Subject: [PATCH 407/551] chore: only run release please on push

And then run release-please tests on release PRs
---
 .github/workflows/release-please.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml
index f942e74122..de4cb087a7 100644
--- a/.github/workflows/release-please.yml
+++ b/.github/workflows/release-please.yml
@@ -13,7 +13,7 @@ jobs:
     permissions:
       contents: write # to create release commit (google-github-actions/release-please-action)
       pull-requests: write # to create release PR (google-github-actions/release-please-action)
-
+    if: github.event_name == 'push'
     runs-on: ubuntu-latest
     steps:
     - uses: google-github-actions/release-please-action@v2
@@ -64,5 +64,5 @@ jobs:
   test:
     name: Release Test
     needs: [ release-please ]
-    if: needs.release-please.outputs.pr
+    if: needs.release-please.outputs.pr || startsWith(github.head_ref, 'release-v')
     uses: ./.github/workflows/tests.yml

From ae8478ec32d9b2fa71b591ac22cdf867ef2e9a7d Mon Sep 17 00:00:00 2001
From: Artur Yapparov 
Date: Mon, 25 Dec 2023 16:07:56 +0300
Subject: [PATCH 408/551] docs: remove outdated Node versions from readme
 (#2955)

Refs: https://github.com/nodejs/node-gyp/issues/2953
---
 README.md | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/README.md b/README.md
index f46ee06308..a4aa6f893a 100644
--- a/README.md
+++ b/README.md
@@ -11,9 +11,8 @@ addons.
 
 Note that `node-gyp` is _not_ used to build Node.js itself.
 
-Multiple target versions of Node.js are supported (i.e. `0.8`, ..., `4`, `5`, `6`,
-etc.), regardless of what version of Node.js is actually installed on your system
-(`node-gyp` downloads the necessary development files or headers for the target version).
+All current and LTS target versions of Node.js are supported. Depending on what version of Node.js is actually installed on your system
+`node-gyp` downloads the necessary development files or headers for the target version. List of stable Node.js versions can be found on [Node.js website](https://nodejs.org/en/about/previous-releases).
 
 ## Features
 

From 7f58bc8144edf8beb44736d68386044b450e1885 Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Wed, 17 Jan 2024 07:52:35 +0100
Subject: [PATCH 409/551] Keep GitHub Actions up to date with Dependabot

* https://docs.github.com/en/code-security/dependabot/working-with-dependabot/keeping-your-actions-up-to-date-with-dependabot
---
 .github/dependabot.yml | 8 ++++++++
 1 file changed, 8 insertions(+)
 create mode 100644 .github/dependabot.yml

diff --git a/.github/dependabot.yml b/.github/dependabot.yml
new file mode 100644
index 0000000000..15e494ec86
--- /dev/null
+++ b/.github/dependabot.yml
@@ -0,0 +1,8 @@
+# Keep GitHub Actions up to date with Dependabot...
+# https://docs.github.com/en/code-security/dependabot/working-with-dependabot/keeping-your-actions-up-to-date-with-dependabot
+version: 2
+updates:
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "daily"

From 3f0df7e9334e49e8c7f6fdbbb9e1e6c5a8cca53b Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Wed, 17 Jan 2024 08:15:15 +0100
Subject: [PATCH 410/551] build(deps): bump actions/setup-python from 4 to 5
 (#2960)

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4 to 5.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 .github/workflows/tests.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 5017bcd9f8..6835cceed5 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -106,7 +106,7 @@ jobs:
         with:
           node-version: ${{ matrix.node }}
       - name: Use Python ${{ matrix.python }}
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python }}
         env:

From b1f1808bfff0d51e6d3eb696ab6a5b89b7b9630c Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Sat, 20 Jan 2024 20:11:30 +0100
Subject: [PATCH 411/551] build(deps): bump
 google-github-actions/release-please-action (#2961)

Bumps [google-github-actions/release-please-action](https://github.com/google-github-actions/release-please-action) from 2 to 4.
- [Release notes](https://github.com/google-github-actions/release-please-action/releases)
- [Changelog](https://github.com/google-github-actions/release-please-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/google-github-actions/release-please-action/compare/v2...v4)

---
updated-dependencies:
- dependency-name: google-github-actions/release-please-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 .github/workflows/release-please.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml
index de4cb087a7..114666d043 100644
--- a/.github/workflows/release-please.yml
+++ b/.github/workflows/release-please.yml
@@ -16,7 +16,7 @@ jobs:
     if: github.event_name == 'push'
     runs-on: ubuntu-latest
     steps:
-    - uses: google-github-actions/release-please-action@v2
+    - uses: google-github-actions/release-please-action@v4
       id: release
       with:
         package-name: node-gyp

From c24cead6e3991a3c5a612f0d333c4b026541b6d9 Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Thu, 25 Jan 2024 07:32:33 +0100
Subject: [PATCH 412/551] Revert "build(deps): bump
 google-github-actions/release-please-action from 2 to 4" (#2967)

---
 .github/workflows/release-please.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml
index 114666d043..de4cb087a7 100644
--- a/.github/workflows/release-please.yml
+++ b/.github/workflows/release-please.yml
@@ -16,7 +16,7 @@ jobs:
     if: github.event_name == 'push'
     runs-on: ubuntu-latest
     steps:
-    - uses: google-github-actions/release-please-action@v4
+    - uses: google-github-actions/release-please-action@v2
       id: release
       with:
         package-name: node-gyp

From a87d0bf458e57d9a8c8c113250bf6a9da952016d Mon Sep 17 00:00:00 2001
From: Davee 
Date: Thu, 25 Jan 2024 21:23:49 +0100
Subject: [PATCH 413/551] refine readme Visual C++ Build Environment text for
 clarity (#2965)

* refine readme text for clarity

* refine readme Visual C++ build env for clarity
---
 README.md | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/README.md b/README.md
index a4aa6f893a..d43d617b78 100644
--- a/README.md
+++ b/README.md
@@ -49,9 +49,7 @@ Install the current [version of Python](https://devguide.python.org/versions/) f
 [Microsoft Store](https://apps.microsoft.com/store/search?publisher=Python+Software+Foundation).
 
 Install tools and configuration manually:
-   * Install Visual C++ Build Environment: [Visual Studio Build Tools](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools)
-   (using "Visual C++ build tools" if using a version older than VS2019, otherwise use "Desktop development with C++" workload) or [Visual Studio Community](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=Community)
-   (using the "Desktop development with C++" workload)
+   * Install Visual C++ Build Environment: For Visual Studio 2019 or later, use the `Desktop development with C++` workload from [Visual Studio Community](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=Community).  For a version older than Visual Studio 2019, install [Visual Studio Build Tools](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) with the `Visual C++ build tools` option.
 
    If the above steps didn't work for you, please visit [Microsoft's Node.js Guidelines for Windows](https://github.com/Microsoft/nodejs-guidelines/blob/master/windows-environment.md#compiling-native-addon-modules) for additional tips.
 

From 7705a22f31a62076e9f8429780a459f4ad71ea4c Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Mon, 29 Jan 2024 19:12:57 +0100
Subject: [PATCH 414/551] docs: npm is currently v10 (#2970)

---
 docs/Updating-npm-bundled-node-gyp.md | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/docs/Updating-npm-bundled-node-gyp.md b/docs/Updating-npm-bundled-node-gyp.md
index 5759add3fe..7f5794ba05 100644
--- a/docs/Updating-npm-bundled-node-gyp.md
+++ b/docs/Updating-npm-bundled-node-gyp.md
@@ -14,7 +14,7 @@ This means that while `node-gyp` doesn't get installed into your `$PATH` by defa
 attempt to `npm install` a native add-on.
 
 Sometimes, you may need to update npm's internal node-gyp to a newer version than what is installed. A simple `npm install -g node-gyp`
-_won't_ do the trick since npm will still continue to use its internal copy over the global one.
+_won't_ do the trick since npm will continue to use its internal copy over the global one.
 
 So instead:
 
@@ -29,7 +29,7 @@ npm --version
 
 Unix is easy. Just run the following command.
 
-If your npm is version ___7 or 8___, do:
+If your npm is version ___7 or higher___, do:
 ```bash
 $ npm explore npm/node_modules/@npmcli/run-script -g -- npm_config_global=false npm install node-gyp@latest
 ```
@@ -43,10 +43,9 @@ If the command fails with a permissions error, please try `sudo` and then the co
 
 ## Windows
 
-Windows is a bit trickier, since `npm` might be installed to the "Program Files" directory, which needs admin privileges in order to
-modify on current Windows. Therefore, run the following commands __inside a `cmd.exe` started with "Run as Administrator"__:
+Windows is a bit trickier, since `npm` might be installed in the "Program Files" directory, which needs admin privileges to modify current Windows. Therefore, run the following commands __inside a `cmd.exe` started with "Run as Administrator"__:
 
-First we need to find the location of `node`. If you don't already know the location that `node.exe` got installed to, then run:
+First, we need to find the location of `node`. If you don't already know the location that `node.exe` got installed to, then run:
 ```bash
 $ where node
 ```
@@ -56,7 +55,7 @@ Now `cd` to the directory that `node.exe` is contained in e.g.:
 $ cd "C:\Program Files\nodejs"
 ```
 
-If your npm version is ___7 or 8___, do:
+If your npm version is ___7 or higher___, do:
 ```bash
 cd node_modules\npm\node_modules\@npmcli\run-script
 ```

From 329873141f0d3e3787d3c006801431da04e4ed0c Mon Sep 17 00:00:00 2001
From: Michael Dawson 
Date: Mon, 29 Jan 2024 14:03:30 -0500
Subject: [PATCH 415/551] src: add support for locally installed headers
 (#2964)

Some linux distros allow headers to be installed through
tools like rpm. If the runtime sets
process.config.variables.use_prefix_to_find_headers, look
for matching headers based on the directory set for the
prefix in process.config.variables.prefix

Signed-off-by: Michael Dawson 
Co-authored-by: Luke Karrys 
---
 lib/configure.js               |  28 +++++++-
 test/test-configure-nodedir.js | 123 +++++++++++++++++++++++++++++++++
 test/test-configure-python.js  |   1 +
 3 files changed, 151 insertions(+), 1 deletion(-)
 create mode 100644 test/test-configure-nodedir.js

diff --git a/lib/configure.js b/lib/configure.js
index 8da41a849d..e4b8c94e3d 100644
--- a/lib/configure.js
+++ b/lib/configure.js
@@ -1,6 +1,6 @@
 'use strict'
 
-const { promises: fs } = require('graceful-fs')
+const { promises: fs, readFileSync } = require('graceful-fs')
 const path = require('path')
 const log = require('./log')
 const os = require('os')
@@ -13,6 +13,10 @@ const { findAccessibleSync } = require('./util')
 const { findPython } = require('./find-python')
 const { findVisualStudio } = win ? require('./find-visualstudio') : {}
 
+const majorRe = /^#define NODE_MAJOR_VERSION (\d+)/m
+const minorRe = /^#define NODE_MINOR_VERSION (\d+)/m
+const patchRe = /^#define NODE_PATCH_VERSION (\d+)/m
+
 async function configure (gyp, argv) {
   const buildDir = path.resolve('build')
   const configNames = ['config.gypi', 'common.gypi']
@@ -27,6 +31,28 @@ async function configure (gyp, argv) {
     // 'python' should be set by now
     process.env.PYTHON = python
 
+    if (!gyp.opts.nodedir &&
+        process.config.variables.use_prefix_to_find_headers) {
+      // check if the headers can be found using the prefix specified
+      // at build time. Use them if they match the version expected
+      const prefix = process.config.variables.node_prefix
+      let availVersion
+      try {
+        const nodeVersionH = readFileSync(path.join(prefix,
+          'include', 'node', 'node_version.h'), { encoding: 'utf8' })
+        const major = nodeVersionH.match(majorRe)[1]
+        const minor = nodeVersionH.match(minorRe)[1]
+        const patch = nodeVersionH.match(patchRe)[1]
+        availVersion = major + '.' + minor + '.' + patch
+      } catch {}
+      if (availVersion === release.version) {
+        // ok version matches, use the headers
+        gyp.opts.nodedir = prefix
+        log.verbose('using local node headers based on prefix',
+          'setting nodedir to ' + gyp.opts.nodedir)
+      }
+    }
+
     if (gyp.opts.nodedir) {
       // --nodedir was specified. use that for the dev files
       nodeDir = gyp.opts.nodedir.replace(/^~/, os.homedir())
diff --git a/test/test-configure-nodedir.js b/test/test-configure-nodedir.js
new file mode 100644
index 0000000000..a6debded06
--- /dev/null
+++ b/test/test-configure-nodedir.js
@@ -0,0 +1,123 @@
+'use strict'
+
+const { describe, it } = require('mocha')
+const assert = require('assert')
+const path = require('path')
+const os = require('os')
+const gyp = require('../lib/node-gyp')
+const requireInject = require('require-inject')
+const semver = require('semver')
+
+const versionSemver = semver.parse(process.version)
+
+const configure = requireInject('../lib/configure', {
+  'graceful-fs': {
+    openSync: () => 0,
+    closeSync: () => {},
+    existsSync: () => true,
+    readFileSync: () => '#define NODE_MAJOR_VERSION ' + versionSemver.major + '\n' +
+        '#define NODE_MINOR_VERSION ' + versionSemver.minor + '\n' +
+        '#define NODE_PATCH_VERSION ' + versionSemver.patch + '\n',
+    promises: {
+      stat: async () => ({}),
+      mkdir: async () => {},
+      writeFile: async () => {}
+    }
+  }
+})
+
+const configure2 = requireInject('../lib/configure', {
+  'graceful-fs': {
+    openSync: () => 0,
+    closeSync: () => {},
+    existsSync: () => true,
+    readFileSync: () => '#define NODE_MAJOR_VERSION 8\n' +
+        '#define NODE_MINOR_VERSION 0\n' +
+        '#define NODE_PATCH_VERSION 0\n',
+    promises: {
+      stat: async () => ({}),
+      mkdir: async () => {},
+      writeFile: async () => {}
+    }
+  }
+})
+
+const SPAWN_RESULT = cb => ({ on: function () { cb() } })
+
+const driveLetter = os.platform() === 'win32' ? `${process.cwd().split(path.sep)[0]}` : ''
+function checkTargetPath (target, value) {
+  let targetPath = path.join(path.sep, target, 'include',
+    'node', 'common.gypi')
+  if (process.platform === 'win32') {
+    targetPath = driveLetter + targetPath
+  }
+
+  return targetPath.localeCompare(value) === 0
+}
+
+describe('configure-nodedir', function () {
+  it('configure nodedir with node-gyp command line', function (done) {
+    const prog = gyp()
+    prog.parseArgv(['dummy_prog', 'dummy_script', '--nodedir=' + path.sep + 'usr'])
+
+    prog.spawn = function (program, args) {
+      for (let i = 0; i < args.length; i++) {
+        if (checkTargetPath('usr', args[i])) {
+          return SPAWN_RESULT(done)
+        }
+      };
+      assert.fail()
+    }
+    configure(prog, [], assert.fail)
+  })
+
+  if (process.config.variables.use_prefix_to_find_headers) {
+    it('use-prefix-to-find-headers build time option - match', function (done) {
+      const prog = gyp()
+      prog.parseArgv(['dummy_prog', 'dummy_script'])
+
+      prog.spawn = function (program, args) {
+        for (let i = 0; i < args.length; i++) {
+          const nodedir = process.config.variables.node_prefix
+          if (checkTargetPath(nodedir, args[i])) {
+            return SPAWN_RESULT(done)
+          }
+        };
+        assert.fail()
+      }
+      configure(prog, [], assert.fail)
+    })
+
+    it('use-prefix-to-find-headers build time option - no match', function (done) {
+      const prog = gyp()
+      prog.parseArgv(['dummy_prog', 'dummy_script'])
+
+      prog.spawn = function (program, args) {
+        for (let i = 0; i < args.length; i++) {
+          const nodedir = process.config.variables.node_prefix
+          if (checkTargetPath(nodedir, args[i])) {
+            assert.fail()
+          }
+        };
+        return SPAWN_RESULT(done)
+      }
+      configure2(prog, [], assert.fail)
+    })
+
+    it('use-prefix-to-find-headers build time option, target specified', function (done) {
+      const prog = gyp()
+      prog.parseArgv(['dummy_prog', 'dummy_script', '--target=8.0.0'])
+
+      prog.spawn = function (program, args) {
+        for (let i = 0; i < args.length; i++) {
+          const nodedir = process.config.variables.node_prefix
+          if (checkTargetPath(nodedir, args[i])) {
+            assert.fail()
+          }
+        };
+        return SPAWN_RESULT(done)
+      }
+      configure(prog, [], assert.fail)
+    })
+  }
+})
diff --git a/test/test-configure-python.js b/test/test-configure-python.js
index eee230a496..094e79182c 100644
--- a/test/test-configure-python.js
+++ b/test/test-configure-python.js
@@ -11,6 +11,7 @@ const configure = requireInject('../lib/configure', {
   'graceful-fs': {
     openSync: () => 0,
     closeSync: () => {},
+    existsSync: () => {},
     promises: {
       stat: async () => ({}),
       mkdir: async () => {},

From 109e3d4245504a7b75c99f578e1203c0ef4b518e Mon Sep 17 00:00:00 2001
From: Jaroslav <375519+jarig@users.noreply.github.com>
Date: Fri, 9 Feb 2024 03:33:54 +0200
Subject: [PATCH 416/551] feat: improve visual studio detection (#2957)

Detect visual studio installation using the VSSetup module and
Get-VSSetupInstance method. It works even in systems with the
Constrained language mode of PowerShell.
---
 README.md                                     |   3 +
 lib/find-visualstudio.js                      |  87 +++-
 .../VSSetup_VS_2019_Professional_workload.txt | 235 +++++++++++
 .../VSSetup_VS_2022_VS2019_workload.txt       | 371 ++++++++++++++++++
 .../VSSetup_VS_2022_multiple_install.txt      | 369 +++++++++++++++++
 test/fixtures/VSSetup_VS_2022_workload.txt    | 136 +++++++
 .../VSSetup_VS_2022_workload_missing_sdk.txt  | 152 +++++++
 test/test-find-visualstudio.js                | 215 ++++++++--
 8 files changed, 1517 insertions(+), 51 deletions(-)
 create mode 100644 test/fixtures/VSSetup_VS_2019_Professional_workload.txt
 create mode 100644 test/fixtures/VSSetup_VS_2022_VS2019_workload.txt
 create mode 100644 test/fixtures/VSSetup_VS_2022_multiple_install.txt
 create mode 100644 test/fixtures/VSSetup_VS_2022_workload.txt
 create mode 100644 test/fixtures/VSSetup_VS_2022_workload_missing_sdk.txt

diff --git a/README.md b/README.md
index d43d617b78..9e4c608e20 100644
--- a/README.md
+++ b/README.md
@@ -57,6 +57,9 @@ Install tools and configuration manually:
 
    To use the native ARM64 C++ compiler on Windows on ARM, ensure that you have Visual Studio 2022 [17.4 or later](https://devblogs.microsoft.com/visualstudio/arm64-visual-studio-is-officially-here/) installed.
 
+It's advised to install following Powershell module: [VSSetup](https://github.com/microsoft/vssetup.powershell) using `Install-Module VSSetup -Scope CurrentUser`.
+This will make Visual Studio detection logic to use more flexible and accessible method, avoiding Powershell's `ConstrainedLanguage` mode.
+
 ### Configuring Python Dependency
 
 `node-gyp` requires that you have installed a [supported version of Python](https://devguide.python.org/versions/).
diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js
index b57770259a..f0710fbe9b 100644
--- a/lib/find-visualstudio.js
+++ b/lib/find-visualstudio.js
@@ -54,6 +54,7 @@ class VisualStudioFinder {
     }
 
     const checks = [
+      () => this.findVisualStudio2017OrNewerUsingSetupModule(),
       () => this.findVisualStudio2017OrNewer(),
       () => this.findVisualStudio2015(),
       () => this.findVisualStudio2013()
@@ -113,6 +114,52 @@ class VisualStudioFinder {
     throw new Error('Could not find any Visual Studio installation to use')
   }
 
+  async findVisualStudio2017OrNewerUsingSetupModule () {
+    const ps = path.join(process.env.SystemRoot, 'System32',
+      'WindowsPowerShell', 'v1.0', 'powershell.exe')
+    const vcInstallDir = this.envVcInstallDir
+
+    const checkModuleArgs = [
+      '-NoProfile',
+      '-Command',
+      '&{@(Get-Module -ListAvailable -Name VSSetup).Version.ToString()}'
+    ]
+    this.log.silly('Running', ps, checkModuleArgs)
+    const [cErr] = await this.execFile(ps, checkModuleArgs)
+    if (cErr) {
+      this.addLog('VSSetup module doesn\'t seem to exist. You can install it via: "Install-Module VSSetup -Scope CurrentUser"')
+      this.log.silly('VSSetup error = %j', cErr && (cErr.stack || cErr))
+      return null
+    }
+    const filterArg = vcInstallDir !== undefined ? `| where {$_.InstallationPath -eq '${vcInstallDir}' }` : ''
+    const psArgs = [
+      '-NoProfile',
+      '-Command',
+      `&{Get-VSSetupInstance ${filterArg} | ConvertTo-Json -Depth 3}`
+    ]
+
+    this.log.silly('Running', ps, psArgs)
+    const [err, stdout, stderr] = await this.execFile(ps, psArgs)
+    let parsedData = this.parseData(err, stdout, stderr)
+    if (parsedData === null) {
+      return null
+    }
+    this.log.silly('Parsed data', parsedData)
+    if (!Array.isArray(parsedData)) {
+      // if there are only 1 result, then Powershell will output non-array
+      parsedData = [parsedData]
+    }
+    // normalize output
+    parsedData = parsedData.map((info) => {
+      info.path = info.InstallationPath
+      info.version = `${info.InstallationVersion.Major}.${info.InstallationVersion.Minor}.${info.InstallationVersion.Build}.${info.InstallationVersion.Revision}`
+      info.packages = info.Packages.map((p) => p.Id)
+      return info
+    })
+    // pass for further processing
+    return this.processData(parsedData)
+  }
+
   // Invoke the PowerShell script to get information about Visual Studio 2017
   // or newer installations
   async findVisualStudio2017OrNewer () {
@@ -128,24 +175,35 @@ class VisualStudioFinder {
     ]
 
     this.log.silly('Running', ps, psArgs)
-    const [err, stdout, stderr] = await execFile(ps, psArgs, { encoding: 'utf8' })
-    return this.parseData(err, stdout, stderr)
+    const [err, stdout, stderr] = await this.execFile(ps, psArgs)
+    const parsedData = this.parseData(err, stdout, stderr, { checkIsArray: true })
+    if (parsedData === null) {
+      return null
+    }
+    return this.processData(parsedData)
   }
 
-  // Parse the output of the PowerShell script and look for an installation
-  // of Visual Studio 2017 or newer to use
-  parseData (err, stdout, stderr) {
+  // Parse the output of the PowerShell script, make sanity checks
+  parseData (err, stdout, stderr, sanityCheckOptions) {
+    const defaultOptions = {
+      checkIsArray: false
+    }
+
+    // Merging provided options with the default options
+    const sanityOptions = { ...defaultOptions, ...sanityCheckOptions }
+
     this.log.silly('PS stderr = %j', stderr)
 
-    const failPowershell = () => {
+    const failPowershell = (failureDetails) => {
       this.addLog(
-        'could not use PowerShell to find Visual Studio 2017 or newer, try re-running with \'--loglevel silly\' for more details')
+        `could not use PowerShell to find Visual Studio 2017 or newer, try re-running with '--loglevel silly' for more details. \n
+        Failure details: ${failureDetails}`)
       return null
     }
 
     if (err) {
       this.log.silly('PS err = %j', err && (err.stack || err))
-      return failPowershell()
+      return failPowershell(`${err}`.substring(0, 40))
     }
 
     let vsInfo
@@ -157,11 +215,16 @@ class VisualStudioFinder {
       return failPowershell()
     }
 
-    if (!Array.isArray(vsInfo)) {
+    if (sanityOptions.checkIsArray && !Array.isArray(vsInfo)) {
       this.log.silly('PS stdout = %j', stdout)
-      return failPowershell()
+      return failPowershell('Expected array as output of the PS script')
     }
+    return vsInfo
+  }
 
+  // Process parsed data containing information about VS installations
+  // Look for the required parts, extract and output them back
+  processData (vsInfo) {
     vsInfo = vsInfo.map((info) => {
       this.log.silly(`processing installation: "${info.path}"`)
       info.path = path.resolve(info.path)
@@ -438,6 +501,10 @@ class VisualStudioFinder {
 
     return true
   }
+
+  async execFile (exec, args) {
+    return await execFile(exec, args, { encoding: 'utf8' })
+  }
 }
 
 module.exports = VisualStudioFinder
diff --git a/test/fixtures/VSSetup_VS_2019_Professional_workload.txt b/test/fixtures/VSSetup_VS_2019_Professional_workload.txt
new file mode 100644
index 0000000000..efafa41707
--- /dev/null
+++ b/test/fixtures/VSSetup_VS_2019_Professional_workload.txt
@@ -0,0 +1,235 @@
+[
+    {
+        "InstanceId":  "2619cf21",
+        "InstallationName":  "VisualStudio/16.11.33+34407.143",
+        "InstallationPath":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Professional",
+        "InstallationVersion":  {
+                                    "Major":  16,
+                                    "Minor":  11,
+                                    "Build":  34407,
+                                    "Revision":  143,
+                                    "MajorRevision":  0,
+                                    "MinorRevision":  143
+                                },
+        "InstallDate":  "\/Date(1706804943503)\/",
+        "State":  4294967295,
+        "DisplayName":  "Visual Studio Professional 2019",
+        "Description":  "Professional IDE best suited to small teams",
+        "ProductPath":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Professional\\Common7\\IDE\\devenv.exe",
+        "Product":  {
+                        "Id":  "Microsoft.VisualStudio.Product.Professional",
+                        "Version":  {
+                                        "Major":  16,
+                                        "Minor":  11,
+                                        "Build":  34407,
+                                        "Revision":  143,
+                                        "MajorRevision":  0,
+                                        "MinorRevision":  143
+                                    },
+                        "Chip":  null,
+                        "Branch":  null,
+                        "Type":  "Product",
+                        "IsExtension":  false,
+                        "UniqueId":  "Microsoft.VisualStudio.Product.Professional,version=16.11.34407.143"
+                    },
+        "Packages":  [
+                         {
+                             "Id":  "Microsoft.VisualStudio.Product.Professional",
+                             "Version":  "16.11.34407.143",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Product",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.Product.Professional,version=16.11.34407.143"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.Component.VC.14.29.16.10.ATL",
+                             "Version":  "16.11.31314.313",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Component",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.Component.VC.14.29.16.10.ATL,version=16.11.31314.313"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.VC.MSBuild.X64.v142",
+                             "Version":  "16.11.31503.54",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Vsix",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.VC.MSBuild.X64.v142,version=16.11.31503.54"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.VC.MSBuild.X64",
+                             "Version":  "16.11.31503.54",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Vsix",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.VC.MSBuild.X64,version=16.11.31503.54"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.VC.MSBuild.x86.v142",
+                             "Version":  "16.11.31503.54",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Vsix",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.VC.MSBuild.x86.v142,version=16.11.31503.54"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.VC.MSBuild.X86",
+                             "Version":  "16.11.31503.54",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Vsix",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.VC.MSBuild.X86,version=16.11.31503.54"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.VC.MSBuild.Base",
+                             "Version":  "16.11.31829.152",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Vsix",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.VC.MSBuild.Base,version=16.11.31829.152"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.VC.MSBuild.Base.Resources",
+                             "Version":  "16.11.31829.152",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Vsix",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.VC.MSBuild.Base.Resources,version=16.11.31829.152,language=en-US"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.Branding.Professional",
+                             "Version":  "16.11.31605.320",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Vsix",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.Branding.Professional,version=16.11.31605.320,language=en-US"
+                         },
+                         {
+                            "Id":  "Microsoft.VisualStudio.Component.Windows10SDK.19041",
+                            "Version":  "16.10.31205.252",
+                            "Chip":  null,
+                            "Branch":  null,
+                            "Type":  "Component",
+                            "IsExtension":  false,
+                            "UniqueId":  "Microsoft.VisualStudio.Component.Windows10SDK.19041,version=16.10.31205.252"
+                        },
+                        {
+                            "Id":  "Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
+                            "Version":  "16.11.32406.258",
+                            "Chip":  null,
+                            "Branch":  null,
+                            "Type":  "Component",
+                            "IsExtension":  false,
+                            "UniqueId":  "Microsoft.VisualStudio.Component.VC.Tools.x86.x64,version=16.11.32406.258"
+                        }
+                     ],
+        "Properties":  [
+                           {
+                               "Key":  "CampaignId",
+                               "Value":  "09"
+                           },
+                           {
+                               "Key":  "SetupEngineFilePath",
+                               "Value":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\setup.exe"
+                           },
+                           {
+                               "Key":  "Nickname",
+                               "Value":  ""
+                           },
+                           {
+                               "Key":  "ChannelManifestId",
+                               "Value":  "VisualStudio.16.Release/16.11.33+34407.143"
+                           }
+                       ],
+        "Errors":  null,
+        "EnginePath":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\resources\\app\\ServiceHub\\Services\\Microsoft.VisualStudio.Setup.Service",
+        "IsComplete":  true,
+        "IsLaunchable":  true,
+        "CatalogInfo":  [
+                            {
+                                "Key":  "Id",
+                                "Value":  "VisualStudio/16.11.33+34407.143"
+                            },
+                            {
+                                "Key":  "BuildBranch",
+                                "Value":  "d16.11"
+                            },
+                            {
+                                "Key":  "BuildVersion",
+                                "Value":  "16.11.34407.143"
+                            },
+                            {
+                                "Key":  "LocalBuild",
+                                "Value":  "build-lab"
+                            },
+                            {
+                                "Key":  "ManifestName",
+                                "Value":  "VisualStudio"
+                            },
+                            {
+                                "Key":  "ManifestType",
+                                "Value":  "installer"
+                            },
+                            {
+                                "Key":  "ProductDisplayVersion",
+                                "Value":  "16.11.33"
+                            },
+                            {
+                                "Key":  "ProductLine",
+                                "Value":  "Dev16"
+                            },
+                            {
+                                "Key":  "ProductLineVersion",
+                                "Value":  "2019"
+                            },
+                            {
+                                "Key":  "ProductMilestone",
+                                "Value":  "RTW"
+                            },
+                            {
+                                "Key":  "ProductMilestoneIsPreRelease",
+                                "Value":  "False"
+                            },
+                            {
+                                "Key":  "ProductName",
+                                "Value":  "Visual Studio"
+                            },
+                            {
+                                "Key":  "ProductPatchVersion",
+                                "Value":  "33"
+                            },
+                            {
+                                "Key":  "ProductPreReleaseMilestoneSuffix",
+                                "Value":  "1.0"
+                            },
+                            {
+                                "Key":  "ProductSemanticVersion",
+                                "Value":  "16.11.33+34407.143"
+                            },
+                            {
+                                "Key":  "RequiredEngineVersion",
+                                "Value":  "2.11.72.18200"
+                            }
+                        ],
+        "IsPrerelease":  false,
+        "PSPath":  "Microsoft.PowerShell.Core\\FileSystem::C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
+        "UpdateDate":  "2024-01-09T19:19:11.0115234Z",
+        "ResolvedInstallationPath":  "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
+        "ChannelId":  "VisualStudio.17.Release",
+        "InstalledChannelId":  "VisualStudio.17.Release",
+        "ChannelUri":  "https://aka.ms/vs/17/release/channel",
+        "InstalledChannelUri":  "https://aka.ms/vs/17/release/channel",
+        "ReleaseNotes":  "https://docs.microsoft.com/en-us/visualstudio/releases/2022/release-notes-v17.8#17.8.4",
+        "ThirdPartyNotices":  "https://go.microsoft.com/fwlink/?LinkId=661288"
+    }
+]
diff --git a/test/fixtures/VSSetup_VS_2022_VS2019_workload.txt b/test/fixtures/VSSetup_VS_2022_VS2019_workload.txt
new file mode 100644
index 0000000000..a6524171ae
--- /dev/null
+++ b/test/fixtures/VSSetup_VS_2022_VS2019_workload.txt
@@ -0,0 +1,371 @@
+[
+    {
+    "InstanceId":  "621862c0",
+    "InstallationName":  "VisualStudio/17.8.3+34330.188",
+    "InstallationPath":  "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
+    "InstallationVersion":  {
+                                "Major":  17,
+                                "Minor":  8,
+                                "Build":  34330,
+                                "Revision":  188,
+                                "MajorRevision":  0,
+                                "MinorRevision":  188
+                            },
+    "InstallDate":  "\/Date(1703254955000)\/",
+    "State":  4294967295,
+    "DisplayName":  "Visual Studio Enterprise 2022",
+    "Description":  "Scalable, end-to-end solution for teams of any size",
+    "ProductPath":  "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\Common7\\IDE\\devenv.exe",
+    "Product":  {
+                    "Id":  "Microsoft.VisualStudio.Product.Enterprise",
+                    "Version":  {
+                                    "Major":  17,
+                                    "Minor":  8,
+                                    "Build":  34330,
+                                    "Revision":  188,
+                                    "MajorRevision":  0,
+                                    "MinorRevision":  188
+                                },
+                    "Chip":  "x64",
+                    "Branch":  null,
+                    "Type":  "Product",
+                    "IsExtension":  false,
+                    "UniqueId":  "Microsoft.VisualStudio.Product.Enterprise,version=17.8.34330.188,chip=x64"
+                },
+    "Packages":  [
+                     {
+                         "Id":  "Microsoft.VisualStudio.Product.Enterprise",
+                         "Version":  "17.8.34330.188",
+                         "Chip":  "x64",
+                         "Branch":  null,
+                         "Type":  "Product",
+                         "IsExtension":  false,
+                         "UniqueId":  "Microsoft.VisualStudio.Product.Enterprise,version=17.8.34330.188,chip=x64"
+                     },
+                     {
+                         "Id":  "Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
+                         "Version":  "17.8.34129.139",
+                         "Chip":  null,
+                         "Branch":  null,
+                         "Type":  "Component",
+                         "IsExtension":  false,
+                         "UniqueId":  "Microsoft.VisualStudio.Component.VC.Tools.x86.x64,version=17.8.34129.139"
+                     },
+                     {
+                         "Id":  "Microsoft.VisualStudio.Component.Windows11SDK.22000",
+                         "Version":  "17.8.34129.139",
+                         "Chip":  null,
+                         "Branch":  null,
+                         "Type":  "Component",
+                         "IsExtension":  false,
+                         "UniqueId":  "Microsoft.VisualStudio.Component.Windows11SDK.22000,version=17.8.34129.139"
+                     },
+                     {
+                         "Id":  "Win11SDK_10.0.22000",
+                         "Version":  "10.0.22000.4",
+                         "Chip":  null,
+                         "Branch":  null,
+                         "Type":  "Exe",
+                         "IsExtension":  false,
+                         "UniqueId":  "Win11SDK_10.0.22000,version=10.0.22000.4"
+                     },
+                     {
+                         "Id":  "Microsoft.VisualStudio.Component.Windows10SDK.20348",
+                         "Version":  "17.8.34129.139",
+                         "Chip":  null,
+                         "Branch":  null,
+                         "Type":  "Component",
+                         "IsExtension":  false,
+                         "UniqueId":  "Microsoft.VisualStudio.Component.Windows10SDK.20348,version=17.8.34129.139"
+                     },
+                     {
+                         "Id":  "Win10SDK_10.0.20348",
+                         "Version":  "10.0.20348.3",
+                         "Chip":  null,
+                         "Branch":  null,
+                         "Type":  "Exe",
+                         "IsExtension":  false,
+                         "UniqueId":  "Win10SDK_10.0.20348,version=10.0.20348.3"
+                     }
+                 ],
+    "Properties":  [
+                       {
+                           "Key":  "CampaignId",
+                           "Value":  ""
+                       },
+                       {
+                           "Key":  "SetupEngineFilePath",
+                           "Value":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\setup.exe"
+                       },
+                       {
+                           "Key":  "Nickname",
+                           "Value":  ""
+                       },
+                       {
+                           "Key":  "ChannelManifestId",
+                           "Value":  "VisualStudio.17.Release/17.8.3+34330.188"
+                       }
+                   ],
+    "Errors":  null,
+    "EnginePath":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\resources\\app\\ServiceHub\\Services\\Microsoft.VisualStudio.Setup.Service",
+    "IsComplete":  true,
+    "IsLaunchable":  true,
+    "CatalogInfo":  [
+                        {
+                            "Key":  "Id",
+                            "Value":  "VisualStudio/17.8.3+34330.188"
+                        },
+                        {
+                            "Key":  "BuildBranch",
+                            "Value":  "d17.8"
+                        },
+                        {
+                            "Key":  "BuildVersion",
+                            "Value":  "17.8.34330.188"
+                        }
+                    ],
+    "IsPrerelease":  false,
+    "PSPath":  "Microsoft.PowerShell.Core\\FileSystem::C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
+    "UpdateDate":  "2023-12-22T14:22:35.1818213Z",
+    "ResolvedInstallationPath":  "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
+    "ChannelId":  "VisualStudio.17.Release",
+    "InstalledChannelId":  "VisualStudio.17.Release",
+    "ChannelUri":  "https://aka.ms/vs/17/release/channel",
+    "InstalledChannelUri":  "https://aka.ms/vs/17/release/channel",
+    "ReleaseNotes":  "https://docs.microsoft.com/en-us/visualstudio/releases/2022/release-notes-v17.8#17.8.3",
+    "ThirdPartyNotices":  "https://go.microsoft.com/fwlink/?LinkId=661288"
+},
+{
+        "InstanceId":  "2619cf21",
+        "InstallationName":  "VisualStudio/16.11.33+34407.143",
+        "InstallationPath":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Professional",
+        "InstallationVersion":  {
+                                    "Major":  16,
+                                    "Minor":  11,
+                                    "Build":  34407,
+                                    "Revision":  143,
+                                    "MajorRevision":  0,
+                                    "MinorRevision":  143
+                                },
+        "InstallDate":  "\/Date(1706804943503)\/",
+        "State":  4294967295,
+        "DisplayName":  "Visual Studio Professional 2019",
+        "Description":  "Professional IDE best suited to small teams",
+        "ProductPath":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Professional\\Common7\\IDE\\devenv.exe",
+        "Product":  {
+                        "Id":  "Microsoft.VisualStudio.Product.Professional",
+                        "Version":  {
+                                        "Major":  16,
+                                        "Minor":  11,
+                                        "Build":  34407,
+                                        "Revision":  143,
+                                        "MajorRevision":  0,
+                                        "MinorRevision":  143
+                                    },
+                        "Chip":  null,
+                        "Branch":  null,
+                        "Type":  "Product",
+                        "IsExtension":  false,
+                        "UniqueId":  "Microsoft.VisualStudio.Product.Professional,version=16.11.34407.143"
+                    },
+        "Packages":  [
+                         {
+                             "Id":  "Microsoft.VisualStudio.Product.Professional",
+                             "Version":  "16.11.34407.143",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Product",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.Product.Professional,version=16.11.34407.143"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.Component.VC.14.29.16.10.ATL",
+                             "Version":  "16.11.31314.313",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Component",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.Component.VC.14.29.16.10.ATL,version=16.11.31314.313"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.VC.MSBuild.X64.v142",
+                             "Version":  "16.11.31503.54",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Vsix",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.VC.MSBuild.X64.v142,version=16.11.31503.54"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.VC.MSBuild.X64",
+                             "Version":  "16.11.31503.54",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Vsix",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.VC.MSBuild.X64,version=16.11.31503.54"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.VC.MSBuild.x86.v142",
+                             "Version":  "16.11.31503.54",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Vsix",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.VC.MSBuild.x86.v142,version=16.11.31503.54"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.VC.MSBuild.X86",
+                             "Version":  "16.11.31503.54",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Vsix",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.VC.MSBuild.X86,version=16.11.31503.54"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.VC.MSBuild.Base",
+                             "Version":  "16.11.31829.152",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Vsix",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.VC.MSBuild.Base,version=16.11.31829.152"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.VC.MSBuild.Base.Resources",
+                             "Version":  "16.11.31829.152",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Vsix",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.VC.MSBuild.Base.Resources,version=16.11.31829.152,language=en-US"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.Branding.Professional",
+                             "Version":  "16.11.31605.320",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Vsix",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.Branding.Professional,version=16.11.31605.320,language=en-US"
+                         },
+                         {
+                            "Id":  "Microsoft.VisualStudio.Component.Windows10SDK.19041",
+                            "Version":  "16.10.31205.252",
+                            "Chip":  null,
+                            "Branch":  null,
+                            "Type":  "Component",
+                            "IsExtension":  false,
+                            "UniqueId":  "Microsoft.VisualStudio.Component.Windows10SDK.19041,version=16.10.31205.252"
+                        },
+                        {
+                            "Id":  "Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
+                            "Version":  "16.11.32406.258",
+                            "Chip":  null,
+                            "Branch":  null,
+                            "Type":  "Component",
+                            "IsExtension":  false,
+                            "UniqueId":  "Microsoft.VisualStudio.Component.VC.Tools.x86.x64,version=16.11.32406.258"
+                        }
+                     ],
+        "Properties":  [
+                           {
+                               "Key":  "CampaignId",
+                               "Value":  "09"
+                           },
+                           {
+                               "Key":  "SetupEngineFilePath",
+                               "Value":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\setup.exe"
+                           },
+                           {
+                               "Key":  "Nickname",
+                               "Value":  ""
+                           },
+                           {
+                               "Key":  "ChannelManifestId",
+                               "Value":  "VisualStudio.16.Release/16.11.33+34407.143"
+                           }
+                       ],
+        "Errors":  null,
+        "EnginePath":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\resources\\app\\ServiceHub\\Services\\Microsoft.VisualStudio.Setup.Service",
+        "IsComplete":  true,
+        "IsLaunchable":  true,
+        "CatalogInfo":  [
+                            {
+                                "Key":  "Id",
+                                "Value":  "VisualStudio/16.11.33+34407.143"
+                            },
+                            {
+                                "Key":  "BuildBranch",
+                                "Value":  "d16.11"
+                            },
+                            {
+                                "Key":  "BuildVersion",
+                                "Value":  "16.11.34407.143"
+                            },
+                            {
+                                "Key":  "LocalBuild",
+                                "Value":  "build-lab"
+                            },
+                            {
+                                "Key":  "ManifestName",
+                                "Value":  "VisualStudio"
+                            },
+                            {
+                                "Key":  "ManifestType",
+                                "Value":  "installer"
+                            },
+                            {
+                                "Key":  "ProductDisplayVersion",
+                                "Value":  "16.11.33"
+                            },
+                            {
+                                "Key":  "ProductLine",
+                                "Value":  "Dev16"
+                            },
+                            {
+                                "Key":  "ProductLineVersion",
+                                "Value":  "2019"
+                            },
+                            {
+                                "Key":  "ProductMilestone",
+                                "Value":  "RTW"
+                            },
+                            {
+                                "Key":  "ProductMilestoneIsPreRelease",
+                                "Value":  "False"
+                            },
+                            {
+                                "Key":  "ProductName",
+                                "Value":  "Visual Studio"
+                            },
+                            {
+                                "Key":  "ProductPatchVersion",
+                                "Value":  "33"
+                            },
+                            {
+                                "Key":  "ProductPreReleaseMilestoneSuffix",
+                                "Value":  "1.0"
+                            },
+                            {
+                                "Key":  "ProductSemanticVersion",
+                                "Value":  "16.11.33+34407.143"
+                            },
+                            {
+                                "Key":  "RequiredEngineVersion",
+                                "Value":  "2.11.72.18200"
+                            }
+                        ],
+        "IsPrerelease":  false,
+        "PSPath":  "Microsoft.PowerShell.Core\\FileSystem::C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
+        "UpdateDate":  "2024-01-09T19:19:11.0115234Z",
+        "ResolvedInstallationPath":  "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
+        "ChannelId":  "VisualStudio.17.Release",
+        "InstalledChannelId":  "VisualStudio.17.Release",
+        "ChannelUri":  "https://aka.ms/vs/17/release/channel",
+        "InstalledChannelUri":  "https://aka.ms/vs/17/release/channel",
+        "ReleaseNotes":  "https://docs.microsoft.com/en-us/visualstudio/releases/2022/release-notes-v17.8#17.8.4",
+        "ThirdPartyNotices":  "https://go.microsoft.com/fwlink/?LinkId=661288"
+    }
+]
diff --git a/test/fixtures/VSSetup_VS_2022_multiple_install.txt b/test/fixtures/VSSetup_VS_2022_multiple_install.txt
new file mode 100644
index 0000000000..2bfd67a190
--- /dev/null
+++ b/test/fixtures/VSSetup_VS_2022_multiple_install.txt
@@ -0,0 +1,369 @@
+[
+    {
+        "InstanceId":  "621862c0",
+        "InstallationName":  "VisualStudio/17.8.3+34330.188",
+        "InstallationPath":  "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
+        "InstallationVersion":  {
+                                    "Major":  17,
+                                    "Minor":  8,
+                                    "Build":  34330,
+                                    "Revision":  188,
+                                    "MajorRevision":  0,
+                                    "MinorRevision":  188
+                                },
+        "InstallDate":  "\/Date(1703254955000)\/",
+        "State":  4294967295,
+        "DisplayName":  "Visual Studio Enterprise 2022",
+        "Description":  "Scalable, end-to-end solution for teams of any size",
+        "ProductPath":  "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\Common7\\IDE\\devenv.exe",
+        "Product":  {
+                        "Id":  "Microsoft.VisualStudio.Product.Enterprise",
+                        "Version":  {
+                                        "Major":  17,
+                                        "Minor":  8,
+                                        "Build":  34330,
+                                        "Revision":  188,
+                                        "MajorRevision":  0,
+                                        "MinorRevision":  188
+                                    },
+                        "Chip":  "x64",
+                        "Branch":  null,
+                        "Type":  "Product",
+                        "IsExtension":  false,
+                        "UniqueId":  "Microsoft.VisualStudio.Product.Enterprise,version=17.8.34330.188,chip=x64"
+                    },
+        "Packages":  [
+                         {
+                            "Id":  "Microsoft.VisualStudio.Product.Enterprise",
+                            "Version":  "17.8.34330.188",
+                            "Chip":  "x64",
+                            "Branch":  null,
+                            "Type":  "Product",
+                            "IsExtension":  false,
+                            "UniqueId":  "Microsoft.VisualStudio.Product.Enterprise,version=17.8.34330.188,chip=x64"
+                        },
+                        {
+                            "Id":  "Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
+                            "Version":  "17.8.34129.139",
+                            "Chip":  null,
+                            "Branch":  null,
+                            "Type":  "Component",
+                            "IsExtension":  false,
+                            "UniqueId":  "Microsoft.VisualStudio.Component.VC.Tools.x86.x64,version=17.8.34129.139"
+                        },
+                        {
+                            "Id":  "Microsoft.VisualStudio.Component.Windows11SDK.22000",
+                            "Version":  "17.8.34129.139",
+                            "Chip":  null,
+                            "Branch":  null,
+                            "Type":  "Component",
+                            "IsExtension":  false,
+                            "UniqueId":  "Microsoft.VisualStudio.Component.Windows11SDK.22000,version=17.8.34129.139"
+                        },
+                        {
+                            "Id":  "Win11SDK_10.0.22000",
+                            "Version":  "10.0.22000.4",
+                            "Chip":  null,
+                            "Branch":  null,
+                            "Type":  "Exe",
+                            "IsExtension":  false,
+                            "UniqueId":  "Win11SDK_10.0.22000,version=10.0.22000.4"
+                        },
+                        {
+                            "Id":  "Microsoft.VisualStudio.Component.Windows10SDK.20348",
+                            "Version":  "17.8.34129.139",
+                            "Chip":  null,
+                            "Branch":  null,
+                            "Type":  "Component",
+                            "IsExtension":  false,
+                            "UniqueId":  "Microsoft.VisualStudio.Component.Windows10SDK.20348,version=17.8.34129.139"
+                        },
+                        {
+                            "Id":  "Win10SDK_10.0.20348",
+                            "Version":  "10.0.20348.3",
+                            "Chip":  null,
+                            "Branch":  null,
+                            "Type":  "Exe",
+                            "IsExtension":  false,
+                            "UniqueId":  "Win10SDK_10.0.20348,version=10.0.20348.3"
+                        }
+                     ],
+        "Properties":  [
+                           {
+                               "Key":  "CampaignId",
+                               "Value":  ""
+                           },
+                           {
+                               "Key":  "SetupEngineFilePath",
+                               "Value":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\setup.exe"
+                           },
+                           {
+                               "Key":  "Nickname",
+                               "Value":  ""
+                           },
+                           {
+                               "Key":  "ChannelManifestId",
+                               "Value":  "VisualStudio.17.Release/17.8.3+34330.188"
+                           }
+                       ],
+        "Errors":  null,
+        "EnginePath":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\resources\\app\\ServiceHub\\Services\\Microsoft.VisualStudio.Setup.Service",
+        "IsComplete":  true,
+        "IsLaunchable":  true,
+        "CatalogInfo":  [
+                            {
+                                "Key":  "Id",
+                                "Value":  "VisualStudio/17.8.3+34330.188"
+                            },
+                            {
+                                "Key":  "BuildBranch",
+                                "Value":  "d17.8"
+                            },
+                            {
+                                "Key":  "BuildVersion",
+                                "Value":  "17.8.34330.188"
+                            },
+                            {
+                                "Key":  "LocalBuild",
+                                "Value":  "build-lab"
+                            },
+                            {
+                                "Key":  "ManifestName",
+                                "Value":  "VisualStudio"
+                            },
+                            {
+                                "Key":  "ManifestType",
+                                "Value":  "installer"
+                            },
+                            {
+                                "Key":  "ProductDisplayVersion",
+                                "Value":  "17.8.3"
+                            },
+                            {
+                                "Key":  "ProductLine",
+                                "Value":  "Dev17"
+                            },
+                            {
+                                "Key":  "ProductLineVersion",
+                                "Value":  "2022"
+                            },
+                            {
+                                "Key":  "ProductMilestone",
+                                "Value":  "RTW"
+                            },
+                            {
+                                "Key":  "ProductMilestoneIsPreRelease",
+                                "Value":  "False"
+                            },
+                            {
+                                "Key":  "ProductName",
+                                "Value":  "Visual Studio"
+                            },
+                            {
+                                "Key":  "ProductPatchVersion",
+                                "Value":  "3"
+                            },
+                            {
+                                "Key":  "ProductPreReleaseMilestoneSuffix",
+                                "Value":  "1.0"
+                            },
+                            {
+                                "Key":  "ProductSemanticVersion",
+                                "Value":  "17.8.3+34330.188"
+                            },
+                            {
+                                "Key":  "RequiredEngineVersion",
+                                "Value":  "3.8.2112.61926"
+                            }
+                        ],
+        "IsPrerelease":  false,
+        "PSPath":  "Microsoft.PowerShell.Core\\FileSystem::C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
+        "UpdateDate":  "2023-12-22T14:22:35.1818213Z",
+        "ResolvedInstallationPath":  "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
+        "ChannelId":  "VisualStudio.17.Release",
+        "InstalledChannelId":  "VisualStudio.17.Release",
+        "ChannelUri":  "https://aka.ms/vs/17/release/channel",
+        "InstalledChannelUri":  "https://aka.ms/vs/17/release/channel",
+        "ReleaseNotes":  "https://docs.microsoft.com/en-us/visualstudio/releases/2022/release-notes-v17.8#17.8.3",
+        "ThirdPartyNotices":  "https://go.microsoft.com/fwlink/?LinkId=661288"
+    },
+    {
+        "InstanceId":  "dd50c6cc",
+        "InstallationName":  "VisualStudio/17.8.3+34330.188",
+        "InstallationPath":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools",
+        "InstallationVersion":  {
+                                    "Major":  17,
+                                    "Minor":  8,
+                                    "Build":  34330,
+                                    "Revision":  188,
+                                    "MajorRevision":  0,
+                                    "MinorRevision":  188
+                                },
+        "InstallDate":  "\/Date(1703262914503)\/",
+        "State":  4294967295,
+        "DisplayName":  "Visual Studio Build Tools 2022",
+        "Description":  "The Visual Studio Build Tools allows you to build native and managed MSBuild-based applications without requiring the Visual Studio IDE. There are options to install the Visual C++ compilers and libraries, MFC, ATL, and C++/CLI support.",
+        "ProductPath":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\Common7\\Tools\\LaunchDevCmd.bat",
+        "Product":  {
+                        "Id":  "Microsoft.VisualStudio.Product.BuildTools",
+                        "Version":  {
+                                        "Major":  17,
+                                        "Minor":  8,
+                                        "Build":  34330,
+                                        "Revision":  188,
+                                        "MajorRevision":  0,
+                                        "MinorRevision":  188
+                                    },
+                        "Chip":  null,
+                        "Branch":  null,
+                        "Type":  "Product",
+                        "IsExtension":  false,
+                        "UniqueId":  "Microsoft.VisualStudio.Product.BuildTools,version=17.8.34330.188"
+                    },
+        "Packages":  [
+                         {
+                             "Id":  "Microsoft.VisualStudio.Product.BuildTools",
+                             "Version":  "17.8.34330.188",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Product",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.Product.BuildTools,version=17.8.34330.188"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.Workload.MSBuildTools",
+                             "Version":  "17.8.34129.139",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Workload",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.Workload.MSBuildTools,version=17.8.34129.139"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.NuGet.BuildTools",
+                             "Version":  "17.0.60800.131",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Vsix",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.NuGet.BuildTools,version=17.0.60800.131"
+                         },
+                         {
+                             "Id":  "Microsoft.Build.UnGAC",
+                             "Version":  "17.8.3.2351904",
+                             "Chip":  "neutral",
+                             "Branch":  null,
+                             "Type":  "Exe",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.Build.UnGAC,version=17.8.3.2351904,chip=neutral,language=neutral"
+                         },
+                         {
+                             "Id":  "Microsoft.VisualStudio.VC.Icons",
+                             "Version":  "17.8.34129.139",
+                             "Chip":  null,
+                             "Branch":  null,
+                             "Type":  "Vsix",
+                             "IsExtension":  false,
+                             "UniqueId":  "Microsoft.VisualStudio.VC.Icons,version=17.8.34129.139"
+                         }
+                     ],
+        "Properties":  [
+                           {
+                               "Key":  "CampaignId",
+                               "Value":  "09"
+                           },
+                           {
+                               "Key":  "SetupEngineFilePath",
+                               "Value":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\setup.exe"
+                           },
+                           {
+                               "Key":  "Nickname",
+                               "Value":  "2"
+                           },
+                           {
+                               "Key":  "ChannelManifestId",
+                               "Value":  "VisualStudio.17.Release/17.8.3+34330.188"
+                           }
+                       ],
+        "Errors":  null,
+        "EnginePath":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\resources\\app\\ServiceHub\\Services\\Microsoft.VisualStudio.Setup.Service",
+        "IsComplete":  true,
+        "IsLaunchable":  true,
+        "CatalogInfo":  [
+                            {
+                                "Key":  "Id",
+                                "Value":  "VisualStudio/17.8.3+34330.188"
+                            },
+                            {
+                                "Key":  "BuildBranch",
+                                "Value":  "d17.8"
+                            },
+                            {
+                                "Key":  "BuildVersion",
+                                "Value":  "17.8.34330.188"
+                            },
+                            {
+                                "Key":  "LocalBuild",
+                                "Value":  "build-lab"
+                            },
+                            {
+                                "Key":  "ManifestName",
+                                "Value":  "VisualStudio"
+                            },
+                            {
+                                "Key":  "ManifestType",
+                                "Value":  "installer"
+                            },
+                            {
+                                "Key":  "ProductDisplayVersion",
+                                "Value":  "17.8.3"
+                            },
+                            {
+                                "Key":  "ProductLine",
+                                "Value":  "Dev17"
+                            },
+                            {
+                                "Key":  "ProductLineVersion",
+                                "Value":  "2022"
+                            },
+                            {
+                                "Key":  "ProductMilestone",
+                                "Value":  "RTW"
+                            },
+                            {
+                                "Key":  "ProductMilestoneIsPreRelease",
+                                "Value":  "False"
+                            },
+                            {
+                                "Key":  "ProductName",
+                                "Value":  "Visual Studio"
+                            },
+                            {
+                                "Key":  "ProductPatchVersion",
+                                "Value":  "3"
+                            },
+                            {
+                                "Key":  "ProductPreReleaseMilestoneSuffix",
+                                "Value":  "1.0"
+                            },
+                            {
+                                "Key":  "ProductSemanticVersion",
+                                "Value":  "17.8.3+34330.188"
+                            },
+                            {
+                                "Key":  "RequiredEngineVersion",
+                                "Value":  "3.8.2112.61926"
+                            }
+                        ],
+        "IsPrerelease":  false,
+        "PSPath":  "Microsoft.PowerShell.Core\\FileSystem::C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
+        "UpdateDate":  "2023-12-22T14:22:35.1818213Z",
+        "ResolvedInstallationPath":  "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
+        "ChannelId":  "VisualStudio.17.Release",
+        "InstalledChannelId":  "VisualStudio.17.Release",
+        "ChannelUri":  "https://aka.ms/vs/17/release/channel",
+        "InstalledChannelUri":  "https://aka.ms/vs/17/release/channel",
+        "ReleaseNotes":  "https://docs.microsoft.com/en-us/visualstudio/releases/2022/release-notes-v17.8#17.8.3",
+        "ThirdPartyNotices":  "https://go.microsoft.com/fwlink/?LinkId=661288"
+    }
+]
diff --git a/test/fixtures/VSSetup_VS_2022_workload.txt b/test/fixtures/VSSetup_VS_2022_workload.txt
new file mode 100644
index 0000000000..3e1abdc30d
--- /dev/null
+++ b/test/fixtures/VSSetup_VS_2022_workload.txt
@@ -0,0 +1,136 @@
+{
+    "InstanceId":  "621862c0",
+    "InstallationName":  "VisualStudio/17.8.3+34330.188",
+    "InstallationPath":  "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
+    "InstallationVersion":  {
+                                "Major":  17,
+                                "Minor":  8,
+                                "Build":  34330,
+                                "Revision":  188,
+                                "MajorRevision":  0,
+                                "MinorRevision":  188
+                            },
+    "InstallDate":  "\/Date(1703254955000)\/",
+    "State":  4294967295,
+    "DisplayName":  "Visual Studio Enterprise 2022",
+    "Description":  "Scalable, end-to-end solution for teams of any size",
+    "ProductPath":  "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\Common7\\IDE\\devenv.exe",
+    "Product":  {
+                    "Id":  "Microsoft.VisualStudio.Product.Enterprise",
+                    "Version":  {
+                                    "Major":  17,
+                                    "Minor":  8,
+                                    "Build":  34330,
+                                    "Revision":  188,
+                                    "MajorRevision":  0,
+                                    "MinorRevision":  188
+                                },
+                    "Chip":  "x64",
+                    "Branch":  null,
+                    "Type":  "Product",
+                    "IsExtension":  false,
+                    "UniqueId":  "Microsoft.VisualStudio.Product.Enterprise,version=17.8.34330.188,chip=x64"
+                },
+    "Packages":  [
+                     {
+                         "Id":  "Microsoft.VisualStudio.Product.Enterprise",
+                         "Version":  "17.8.34330.188",
+                         "Chip":  "x64",
+                         "Branch":  null,
+                         "Type":  "Product",
+                         "IsExtension":  false,
+                         "UniqueId":  "Microsoft.VisualStudio.Product.Enterprise,version=17.8.34330.188,chip=x64"
+                     },
+                     {
+                         "Id":  "Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
+                         "Version":  "17.8.34129.139",
+                         "Chip":  null,
+                         "Branch":  null,
+                         "Type":  "Component",
+                         "IsExtension":  false,
+                         "UniqueId":  "Microsoft.VisualStudio.Component.VC.Tools.x86.x64,version=17.8.34129.139"
+                     },
+                     {
+                         "Id":  "Microsoft.VisualStudio.Component.Windows11SDK.22000",
+                         "Version":  "17.8.34129.139",
+                         "Chip":  null,
+                         "Branch":  null,
+                         "Type":  "Component",
+                         "IsExtension":  false,
+                         "UniqueId":  "Microsoft.VisualStudio.Component.Windows11SDK.22000,version=17.8.34129.139"
+                     },
+                     {
+                         "Id":  "Win11SDK_10.0.22000",
+                         "Version":  "10.0.22000.4",
+                         "Chip":  null,
+                         "Branch":  null,
+                         "Type":  "Exe",
+                         "IsExtension":  false,
+                         "UniqueId":  "Win11SDK_10.0.22000,version=10.0.22000.4"
+                     },
+                     {
+                         "Id":  "Microsoft.VisualStudio.Component.Windows10SDK.20348",
+                         "Version":  "17.8.34129.139",
+                         "Chip":  null,
+                         "Branch":  null,
+                         "Type":  "Component",
+                         "IsExtension":  false,
+                         "UniqueId":  "Microsoft.VisualStudio.Component.Windows10SDK.20348,version=17.8.34129.139"
+                     },
+                     {
+                         "Id":  "Win10SDK_10.0.20348",
+                         "Version":  "10.0.20348.3",
+                         "Chip":  null,
+                         "Branch":  null,
+                         "Type":  "Exe",
+                         "IsExtension":  false,
+                         "UniqueId":  "Win10SDK_10.0.20348,version=10.0.20348.3"
+                     }
+                 ],
+    "Properties":  [
+                       {
+                           "Key":  "CampaignId",
+                           "Value":  ""
+                       },
+                       {
+                           "Key":  "SetupEngineFilePath",
+                           "Value":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\setup.exe"
+                       },
+                       {
+                           "Key":  "Nickname",
+                           "Value":  ""
+                       },
+                       {
+                           "Key":  "ChannelManifestId",
+                           "Value":  "VisualStudio.17.Release/17.8.3+34330.188"
+                       }
+                   ],
+    "Errors":  null,
+    "EnginePath":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\resources\\app\\ServiceHub\\Services\\Microsoft.VisualStudio.Setup.Service",
+    "IsComplete":  true,
+    "IsLaunchable":  true,
+    "CatalogInfo":  [
+                        {
+                            "Key":  "Id",
+                            "Value":  "VisualStudio/17.8.3+34330.188"
+                        },
+                        {
+                            "Key":  "BuildBranch",
+                            "Value":  "d17.8"
+                        },
+                        {
+                            "Key":  "BuildVersion",
+                            "Value":  "17.8.34330.188"
+                        }
+                    ],
+    "IsPrerelease":  false,
+    "PSPath":  "Microsoft.PowerShell.Core\\FileSystem::C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
+    "UpdateDate":  "2023-12-22T14:22:35.1818213Z",
+    "ResolvedInstallationPath":  "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
+    "ChannelId":  "VisualStudio.17.Release",
+    "InstalledChannelId":  "VisualStudio.17.Release",
+    "ChannelUri":  "https://aka.ms/vs/17/release/channel",
+    "InstalledChannelUri":  "https://aka.ms/vs/17/release/channel",
+    "ReleaseNotes":  "https://docs.microsoft.com/en-us/visualstudio/releases/2022/release-notes-v17.8#17.8.3",
+    "ThirdPartyNotices":  "https://go.microsoft.com/fwlink/?LinkId=661288"
+}
diff --git a/test/fixtures/VSSetup_VS_2022_workload_missing_sdk.txt b/test/fixtures/VSSetup_VS_2022_workload_missing_sdk.txt
new file mode 100644
index 0000000000..8edee24607
--- /dev/null
+++ b/test/fixtures/VSSetup_VS_2022_workload_missing_sdk.txt
@@ -0,0 +1,152 @@
+{
+    "InstanceId":  "621862c0",
+    "InstallationName":  "VisualStudio/17.8.3+34330.188",
+    "InstallationPath":  "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
+    "InstallationVersion":  {
+                                "Major":  17,
+                                "Minor":  8,
+                                "Build":  34330,
+                                "Revision":  188,
+                                "MajorRevision":  0,
+                                "MinorRevision":  188
+                            },
+    "InstallDate":  "\/Date(1703254955000)\/",
+    "State":  4294967295,
+    "DisplayName":  "Visual Studio Enterprise 2022",
+    "Description":  "Scalable, end-to-end solution for teams of any size",
+    "ProductPath":  "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\Common7\\IDE\\devenv.exe",
+    "Product":  {
+                    "Id":  "Microsoft.VisualStudio.Product.Enterprise",
+                    "Version":  {
+                                    "Major":  17,
+                                    "Minor":  8,
+                                    "Build":  34330,
+                                    "Revision":  188,
+                                    "MajorRevision":  0,
+                                    "MinorRevision":  188
+                                },
+                    "Chip":  "x64",
+                    "Branch":  null,
+                    "Type":  "Product",
+                    "IsExtension":  false,
+                    "UniqueId":  "Microsoft.VisualStudio.Product.Enterprise,version=17.8.34330.188,chip=x64"
+                },
+    "Packages":  [
+                     {
+                         "Id":  "Microsoft.VisualStudio.Product.Enterprise",
+                         "Version":  "17.8.34330.188",
+                         "Chip":  "x64",
+                         "Branch":  null,
+                         "Type":  "Product",
+                         "IsExtension":  false,
+                         "UniqueId":  "Microsoft.VisualStudio.Product.Enterprise,version=17.8.34330.188,chip=x64"
+                     },
+                     {
+                         "Id":  "Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
+                         "Version":  "17.8.34129.139",
+                         "Chip":  null,
+                         "Branch":  null,
+                         "Type":  "Component",
+                         "IsExtension":  false,
+                         "UniqueId":  "Microsoft.VisualStudio.Component.VC.Tools.x86.x64,version=17.8.34129.139"
+                     },
+                 ],
+    "Properties":  [
+                       {
+                           "Key":  "CampaignId",
+                           "Value":  ""
+                       },
+                       {
+                           "Key":  "SetupEngineFilePath",
+                           "Value":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\setup.exe"
+                       },
+                       {
+                           "Key":  "Nickname",
+                           "Value":  ""
+                       },
+                       {
+                           "Key":  "ChannelManifestId",
+                           "Value":  "VisualStudio.17.Release/17.8.3+34330.188"
+                       }
+                   ],
+    "Errors":  null,
+    "EnginePath":  "C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\resources\\app\\ServiceHub\\Services\\Microsoft.VisualStudio.Setup.Service",
+    "IsComplete":  true,
+    "IsLaunchable":  true,
+    "CatalogInfo":  [
+                        {
+                            "Key":  "Id",
+                            "Value":  "VisualStudio/17.8.3+34330.188"
+                        },
+                        {
+                            "Key":  "BuildBranch",
+                            "Value":  "d17.8"
+                        },
+                        {
+                            "Key":  "BuildVersion",
+                            "Value":  "17.8.34330.188"
+                        },
+                        {
+                            "Key":  "LocalBuild",
+                            "Value":  "build-lab"
+                        },
+                        {
+                            "Key":  "ManifestName",
+                            "Value":  "VisualStudio"
+                        },
+                        {
+                            "Key":  "ManifestType",
+                            "Value":  "installer"
+                        },
+                        {
+                            "Key":  "ProductDisplayVersion",
+                            "Value":  "17.8.3"
+                        },
+                        {
+                            "Key":  "ProductLine",
+                            "Value":  "Dev17"
+                        },
+                        {
+                            "Key":  "ProductLineVersion",
+                            "Value":  "2022"
+                        },
+                        {
+                            "Key":  "ProductMilestone",
+                            "Value":  "RTW"
+                        },
+                        {
+                            "Key":  "ProductMilestoneIsPreRelease",
+                            "Value":  "False"
+                        },
+                        {
+                            "Key":  "ProductName",
+                            "Value":  "Visual Studio"
+                        },
+                        {
+                            "Key":  "ProductPatchVersion",
+                            "Value":  "3"
+                        },
+                        {
+                            "Key":  "ProductPreReleaseMilestoneSuffix",
+                            "Value":  "1.0"
+                        },
+                        {
+                            "Key":  "ProductSemanticVersion",
+                            "Value":  "17.8.3+34330.188"
+                        },
+                        {
+                            "Key":  "RequiredEngineVersion",
+                            "Value":  "3.8.2112.61926"
+                        }
+                    ],
+    "IsPrerelease":  false,
+    "PSPath":  "Microsoft.PowerShell.Core\\FileSystem::C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
+    "UpdateDate":  "2023-12-22T14:22:35.1818213Z",
+    "ResolvedInstallationPath":  "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
+    "ChannelId":  "VisualStudio.17.Release",
+    "InstalledChannelId":  "VisualStudio.17.Release",
+    "ChannelUri":  "https://aka.ms/vs/17/release/channel",
+    "InstalledChannelUri":  "https://aka.ms/vs/17/release/channel",
+    "ReleaseNotes":  "https://docs.microsoft.com/en-us/visualstudio/releases/2022/release-notes-v17.8#17.8.3",
+    "ThirdPartyNotices":  "https://go.microsoft.com/fwlink/?LinkId=661288"
+}
diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js
index 08e9438c45..ca40165a74 100644
--- a/test/test-find-visualstudio.js
+++ b/test/test-find-visualstudio.js
@@ -22,9 +22,18 @@ class TestVisualStudioFinder extends VisualStudioFinder {
 }
 
 describe('find-visualstudio', function () {
+  this.beforeAll(function () {
+    // Condition to skip the test suite
+    if (process.env.SystemRoot === undefined) {
+      process.env.SystemRoot = '/'
+    }
+  })
+
   it('VS2013', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
-
+    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
+      return null
+    }
     finder.findVisualStudio2017OrNewer = async () => {
       return finder.parseData(new Error(), '', '')
     }
@@ -69,10 +78,15 @@ describe('find-visualstudio', function () {
       patch: 0
     }, null)
 
+    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
+      return null
+    }
+
     finder.findVisualStudio2017OrNewer = async () => {
       const file = path.join(__dirname, 'fixtures', 'VS_2017_Unusable.txt')
       const data = fs.readFileSync(file)
-      return finder.parseData(null, data, '')
+      const vsInfo = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(vsInfo)
     }
     finder.regSearchKeys = async (keys, value, addOpts) => {
       for (let i = 0; i < keys.length; ++i) {
@@ -96,6 +110,10 @@ describe('find-visualstudio', function () {
   it('VS2015', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
 
+    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
+      return null
+    }
+
     finder.findVisualStudio2017OrNewer = async () => {
       return finder.parseData(new Error(), '', '')
     }
@@ -132,52 +150,49 @@ describe('find-visualstudio', function () {
   it('error from PowerShell', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null, null)
 
-    finder.parseData(new Error(), '', '', (info) => {
-      assert.ok(/use PowerShell/i.test(finder.errorLog[0]), 'expect error')
-      assert.ok(!info, 'no data')
-    })
+    const vsInfo = finder.parseData(new Error('Error msg'), '', '')
+    assert.ok(/use PowerShell/i.test(finder.errorLog[0]), `expect error, output: ${finder.errorLog[0]}`)
+    assert.ok(/error msg/i.test(finder.errorLog[0]), `expect error, output: ${finder.errorLog[0]}`)
+    assert.equal(vsInfo, null)
   })
 
   it('empty output from PowerShell', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null, null)
 
-    finder.parseData(null, '', '', (info) => {
-      assert.ok(/use PowerShell/i.test(finder.errorLog[0]), 'expect error')
-      assert.ok(!info, 'no data')
-    })
+    const vsInfo = finder.parseData(null, '', '', { checkIsArray: true })
+    assert.ok(/use PowerShell/i.test(finder.errorLog[0]), `expect error, output: ${finder.errorLog[0]}`)
+    assert.equal(vsInfo, null)
   })
 
   it('output from PowerShell not JSON', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null, null)
 
-    finder.parseData(null, 'AAAABBBB', '', (info) => {
-      assert.ok(/use PowerShell/i.test(finder.errorLog[0]), 'expect error')
-      assert.ok(!info, 'no data')
-    })
+    const vsInfo = finder.parseData(null, 'AAAABBBB', '', { checkIsArray: true })
+    assert.ok(/use PowerShell/i.test(finder.errorLog[0]), `expect error, output: ${finder.errorLog[0]}`)
+    assert.equal(vsInfo, null)
   })
 
   it('wrong JSON from PowerShell', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null, null)
 
-    finder.parseData(null, '{}', '', (info) => {
-      assert.ok(/use PowerShell/i.test(finder.errorLog[0]), 'expect error')
-      assert.ok(!info, 'no data')
-    })
+    const vsInfo = finder.parseData(null, '{}', '', { checkIsArray: true })
+    assert.ok(/use PowerShell/i.test(finder.errorLog[0]), `expect error, output: ${finder.errorLog[0]}`)
+    assert.ok(/expected array/i.test(finder.errorLog[0]), `expect error, output: ${finder.errorLog[0]}`)
+    assert.equal(vsInfo, null)
   })
 
   it('empty JSON from PowerShell', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null, null)
 
-    finder.parseData(null, '[]', '', (info) => {
-      assert.ok(/find .* Visual Studio/i.test(finder.errorLog[0]), 'expect error')
-      assert.ok(!info, 'no data')
-    })
+    const vsInfo = finder.parseData(null, '[]', '', { checkIsArray: true })
+    assert.equal(finder.errorLog.length, 0)
+    assert.equal(vsInfo.length, 0)
   })
 
   it('future version', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null, null)
 
-    finder.parseData(null, JSON.stringify([{
+    const vsInfo = finder.parseData(null, JSON.stringify([{
       packages: [
         'Microsoft.VisualStudio.Component.VC.Tools.x86.x64',
         'Microsoft.VisualStudio.Component.Windows10SDK.17763',
@@ -185,11 +200,9 @@ describe('find-visualstudio', function () {
       ],
       path: 'C:\\VS',
       version: '9999.9999.9999.9999'
-    }]), '', (info) => {
-      assert.ok(/unknown version/i.test(finder.errorLog[0]), 'expect error')
-      assert.ok(/find .* Visual Studio/i.test(finder.errorLog[1]), 'expect error')
-      assert.ok(!info, 'no data')
-    })
+    }]), '', { checkIsArray: true })
+    assert.equal(finder.errorLog.length, 0)
+    assert.equal(vsInfo[0].packages.length, 3)
   })
 
   it('single unusable VS2017', async function () {
@@ -197,22 +210,27 @@ describe('find-visualstudio', function () {
 
     const file = path.join(__dirname, 'fixtures', 'VS_2017_Unusable.txt')
     const data = fs.readFileSync(file)
-    finder.parseData(null, data, '', (info) => {
-      assert.ok(/checking/i.test(finder.errorLog[0]), 'expect error')
-      assert.ok(/find .* Visual Studio/i.test(finder.errorLog[2]), 'expect error')
-      assert.ok(!info, 'no data')
-    })
+    const vsInfo = finder.parseData(null, data, '', { checkIsArray: true })
+    assert.equal(finder.errorLog.length, 0)
+    assert.equal(vsInfo.length, 1)
+    assert.equal(vsInfo[0].InstallationPath, undefined)
+    assert.notEqual(vsInfo[0].path, undefined)
+    assert.ok(vsInfo[0].packages.length > 1)
   })
 
   it('minimal VS2017 Build Tools', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
+    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
+      return null
+    }
     finder.findVisualStudio2017OrNewer = async () => {
       const file = path.join(__dirname, 'fixtures',
         'VS_2017_BuildTools_minimal.txt')
       const data = fs.readFileSync(file)
-      return finder.parseData(null, data, '')
+      const vsInfo = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(vsInfo)
     }
     const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
@@ -234,11 +252,15 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
+    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
+      return null
+    }
     finder.findVisualStudio2017OrNewer = async () => {
       const file = path.join(__dirname, 'fixtures',
         'VS_2017_Community_workload.txt')
       const data = fs.readFileSync(file)
-      return finder.parseData(null, data, '')
+      const vsInfo = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(vsInfo)
     }
     const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
@@ -260,10 +282,14 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
+    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
+      return null
+    }
     finder.findVisualStudio2017OrNewer = async () => {
       const file = path.join(__dirname, 'fixtures', 'VS_2017_Express.txt')
       const data = fs.readFileSync(file)
-      return finder.parseData(null, data, '')
+      const vsInfo = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(vsInfo)
     }
     const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
@@ -285,11 +311,15 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
+    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
+      return null
+    }
     finder.findVisualStudio2017OrNewer = async () => {
       const file = path.join(__dirname, 'fixtures',
         'VS_2019_Preview.txt')
       const data = fs.readFileSync(file)
-      return finder.parseData(null, data, '')
+      const vsInfo = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(vsInfo)
     }
     const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
@@ -311,11 +341,15 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
+    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
+      return null
+    }
     finder.findVisualStudio2017OrNewer = async () => {
       const file = path.join(__dirname, 'fixtures',
         'VS_2019_BuildTools_minimal.txt')
       const data = fs.readFileSync(file)
-      return finder.parseData(null, data, '')
+      const vsInfo = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(vsInfo)
     }
     const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
@@ -337,11 +371,15 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
+    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
+      return null
+    }
     finder.findVisualStudio2017OrNewer = async () => {
       const file = path.join(__dirname, 'fixtures',
         'VS_2019_Community_workload.txt')
       const data = fs.readFileSync(file)
-      return finder.parseData(null, data, '')
+      const vsInfo = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(vsInfo)
     }
     const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
@@ -372,11 +410,15 @@ describe('find-visualstudio', function () {
     finder.msBuildPathExists = (path) => {
       return true
     }
+    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
+      return null
+    }
     finder.findVisualStudio2017OrNewer = async () => {
       const file = path.join(__dirname, 'fixtures',
         'VS_2022_Community_workload.txt')
       const data = fs.readFileSync(file)
-      return finder.parseData(null, data, '')
+      const vsInfo = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(vsInfo)
     }
     const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
@@ -393,7 +435,97 @@ describe('find-visualstudio', function () {
     })
   })
 
+  it('VSSetup: VS2022 with C++ workload without SDK', async function () {
+    const finder = new TestVisualStudioFinder(semverV1, null)
+    finder.msBuildPathExists = (path) => {
+      return true
+    }
+    finder.findVisualStudio2017OrNewer = async () => {
+      return null
+    }
+    finder.findOldVS = async (info) => {
+      return null
+    }
+    setupExecFixture(finder, 'VSSetup_VS_2022_workload_missing_sdk.txt')
+    const { err, info } = await finder.findVisualStudio()
+    assert.match(err.message, /could not find/i)
+    assert.strictEqual(info, null)
+  })
+
+  it('VSSetup: VS2019 with C++ workload', async function () {
+    await verifyVSSetupData('VSSetup_VS_2019_Professional_workload.txt', 'Professional', 2019,
+      '10.0.19041.0', 'v142', '16.11.34407.143', 'Program Files (x86)')
+  })
+
+  it('VSSetup: VS2022 with C++ workload', async function () {
+    await verifyVSSetupData('VSSetup_VS_2022_workload.txt', 'Enterprise', 2022,
+      '10.0.22000.0', 'v143', '17.8.34330.188', 'Program Files')
+  })
+
+  it('VSSetup: VS2022 and VS2019 with C++ workload', async function () {
+    await verifyVSSetupData('VSSetup_VS_2022_VS2019_workload.txt', 'Enterprise', 2022,
+      '10.0.22000.0', 'v143', '17.8.34330.188', 'Program Files')
+  })
+
+  it('VSSetup: VS2022 with multiple installations', async function () {
+    await verifyVSSetupData('VSSetup_VS_2022_multiple_install.txt', 'Enterprise', 2022,
+      '10.0.22000.0', 'v143', '17.8.34330.188', 'Program Files')
+  })
+
+  async function verifyVSSetupData (fixtureName, vsType, vsYear, sdkVersion, toolsetVersion, vsVersion, expectedProgramFilesPath) {
+    const msBuildPath = process.arch === 'arm64'
+      ? `C:\\${expectedProgramFilesPath}\\Microsoft Visual Studio\\${vsYear}\\` +
+        `${vsType}\\MSBuild\\Current\\Bin\\arm64\\MSBuild.exe`
+      : `C:\\${expectedProgramFilesPath}\\Microsoft Visual Studio\\${vsYear}\\` +
+        `${vsType}\\MSBuild\\Current\\Bin\\MSBuild.exe`
+
+    const finder = new TestVisualStudioFinder(semverV1, null)
+
+    poison(finder, 'regSearchKeys')
+    const expectedVSPath = `C:\\${expectedProgramFilesPath}\\Microsoft Visual Studio\\${vsYear}\\${vsType}`
+    finder.msBuildPathExists = (path) => {
+      if (path.startsWith(expectedVSPath) && path.endsWith('MSBuild.exe')) {
+        return true
+      }
+      return false
+    }
+    finder.findVisualStudio2017OrNewer = async () => {
+      throw new Error("findVisualStudio2017OrNewer shouldn't be called")
+    }
+    setupExecFixture(finder, fixtureName)
+    const { err, info } = await finder.findVisualStudio()
+    assert.strictEqual(err, null)
+    const vsVersionTokens = vsVersion.split('.')
+    assert.deepStrictEqual(info, {
+      msBuild: msBuildPath,
+      path:
+        `C:\\${expectedProgramFilesPath}\\Microsoft Visual Studio\\${vsYear}\\${vsType}`,
+      sdk: sdkVersion,
+      toolset: toolsetVersion,
+      version: vsVersion,
+      versionMajor: parseInt(vsVersionTokens[0]),
+      versionMinor: parseInt(vsVersionTokens[1]),
+      versionYear: vsYear
+    })
+  }
+
+  function setupExecFixture (finder, fixtureName) {
+    finder.execFile = async (exec, args) => {
+      if (args.length > 2 && args[2].includes('Get-Module')) {
+        return [null, '1.0.0', '']
+      } else if (args.length > 2 && args.at(-1).includes('Get-VSSetupInstance')) {
+        const file = path.join(__dirname, 'fixtures', fixtureName)
+        return [null, fs.readFileSync(file), '']
+      }
+      return [new Error(), '', '']
+    }
+  }
+
   function allVsVersions (finder) {
+    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
+      return null
+    }
+
     finder.findVisualStudio2017OrNewer = async () => {
       const data0 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures',
         'VS_2017_Unusable.txt')))
@@ -413,7 +545,8 @@ describe('find-visualstudio', function () {
         'VS_2022_Community_workload.txt')))
       const data = JSON.stringify(data0.concat(data1, data2, data3, data4,
         data5, data6, data7))
-      return finder.parseData(null, data, '')
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData)
     }
     finder.regSearchKeys = async (keys, value, addOpts) => {
       for (let i = 0; i < keys.length; ++i) {

From 391cc5b9b25cffe0cb2edcba3583414a771b4a15 Mon Sep 17 00:00:00 2001
From: Stefan Stojanovic 
Date: Wed, 6 Mar 2024 17:32:44 +0100
Subject: [PATCH 417/551] win: update supported vs versions (#2959)

Drop VS2017 support for Node.js v22 and above.

Refs: https://github.com/nodejs/build/pull/3603
Refs: https://github.com/nodejs/node/pull/45427
---
 lib/find-visualstudio.js       |  50 ++++++++--
 test/test-find-visualstudio.js | 173 +++++++++++++++++++--------------
 2 files changed, 142 insertions(+), 81 deletions(-)

diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js
index f0710fbe9b..8c5ae96127 100644
--- a/lib/find-visualstudio.js
+++ b/lib/find-visualstudio.js
@@ -54,8 +54,10 @@ class VisualStudioFinder {
     }
 
     const checks = [
-      () => this.findVisualStudio2017OrNewerUsingSetupModule(),
-      () => this.findVisualStudio2017OrNewer(),
+      () => this.findVisualStudio2019OrNewerUsingSetupModule(),
+      () => this.findVisualStudio2019OrNewer(),
+      () => this.findVisualStudio2017UsingSetupModule(),
+      () => this.findVisualStudio2017(),
       () => this.findVisualStudio2015(),
       () => this.findVisualStudio2013()
     ]
@@ -114,7 +116,20 @@ class VisualStudioFinder {
     throw new Error('Could not find any Visual Studio installation to use')
   }
 
-  async findVisualStudio2017OrNewerUsingSetupModule () {
+  async findVisualStudio2019OrNewerUsingSetupModule () {
+    return this.findNewVSUsingSetupModule([2019, 2022])
+  }
+
+  async findVisualStudio2017UsingSetupModule () {
+    if (this.nodeSemver.major >= 22) {
+      this.addLog(
+        'not looking for VS2017 as it is only supported up to Node.js 21')
+      return null
+    }
+    return this.findNewVSUsingSetupModule([2017])
+  }
+
+  async findNewVSUsingSetupModule (supportedYears) {
     const ps = path.join(process.env.SystemRoot, 'System32',
       'WindowsPowerShell', 'v1.0', 'powershell.exe')
     const vcInstallDir = this.envVcInstallDir
@@ -157,12 +172,28 @@ class VisualStudioFinder {
       return info
     })
     // pass for further processing
-    return this.processData(parsedData)
+    return this.processData(parsedData, supportedYears)
+  }
+
+  // Invoke the PowerShell script to get information about Visual Studio 2019
+  // or newer installations
+  async findVisualStudio2019OrNewer () {
+    return this.findNewVS([2019, 2022])
+  }
+
+  // Invoke the PowerShell script to get information about Visual Studio 2017
+  async findVisualStudio2017 () {
+    if (this.nodeSemver.major >= 22) {
+      this.addLog(
+        'not looking for VS2017 as it is only supported up to Node.js 21')
+      return null
+    }
+    return this.findNewVS([2017])
   }
 
   // Invoke the PowerShell script to get information about Visual Studio 2017
   // or newer installations
-  async findVisualStudio2017OrNewer () {
+  async findNewVS (supportedYears) {
     const ps = path.join(process.env.SystemRoot, 'System32',
       'WindowsPowerShell', 'v1.0', 'powershell.exe')
     const csFile = path.join(__dirname, 'Find-VisualStudio.cs')
@@ -180,7 +211,7 @@ class VisualStudioFinder {
     if (parsedData === null) {
       return null
     }
-    return this.processData(parsedData)
+    return this.processData(parsedData, supportedYears)
   }
 
   // Parse the output of the PowerShell script, make sanity checks
@@ -224,7 +255,7 @@ class VisualStudioFinder {
 
   // Process parsed data containing information about VS installations
   // Look for the required parts, extract and output them back
-  processData (vsInfo) {
+  processData (vsInfo, supportedYears) {
     vsInfo = vsInfo.map((info) => {
       this.log.silly(`processing installation: "${info.path}"`)
       info.path = path.resolve(info.path)
@@ -238,11 +269,12 @@ class VisualStudioFinder {
     this.log.silly('vsInfo:', vsInfo)
 
     // Remove future versions or errors parsing version number
+    // Also remove any unsupported versions
     vsInfo = vsInfo.filter((info) => {
-      if (info.versionYear) {
+      if (info.versionYear && supportedYears.indexOf(info.versionYear) !== -1) {
         return true
       }
-      this.addLog(`unknown version "${info.version}" found at "${info.path}"`)
+      this.addLog(`${info.versionYear ? 'unsupported' : 'unknown'} version "${info.version}" found at "${info.path}"`)
       return false
     })
 
diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js
index ca40165a74..2c3f4e1981 100644
--- a/test/test-find-visualstudio.js
+++ b/test/test-find-visualstudio.js
@@ -31,12 +31,8 @@ describe('find-visualstudio', function () {
 
   it('VS2013', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
-    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
-      return null
-    }
-    finder.findVisualStudio2017OrNewer = async () => {
-      return finder.parseData(new Error(), '', '')
-    }
+    finder.findNewVSUsingSetupModule = async () => null
+    finder.findNewVS = async () => null
     finder.regSearchKeys = async (keys, value, addOpts) => {
       for (let i = 0; i < keys.length; ++i) {
         const fullName = `${keys[i]}\\${value}`
@@ -78,15 +74,18 @@ describe('find-visualstudio', function () {
       patch: 0
     }, null)
 
-    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
-      return null
+    finder.findNewVSUsingSetupModule = async () => null
+    finder.findVisualStudio2019OrNewer = async () => {
+      const file = path.join(__dirname, 'fixtures', 'VS_2017_Unusable.txt')
+      const data = fs.readFileSync(file)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2019, 2022])
     }
-
-    finder.findVisualStudio2017OrNewer = async () => {
+    finder.findVisualStudio2017 = async () => {
       const file = path.join(__dirname, 'fixtures', 'VS_2017_Unusable.txt')
       const data = fs.readFileSync(file)
-      const vsInfo = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(vsInfo)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2017])
     }
     finder.regSearchKeys = async (keys, value, addOpts) => {
       for (let i = 0; i < keys.length; ++i) {
@@ -109,14 +108,8 @@ describe('find-visualstudio', function () {
 
   it('VS2015', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
-
-    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
-      return null
-    }
-
-    finder.findVisualStudio2017OrNewer = async () => {
-      return finder.parseData(new Error(), '', '')
-    }
+    finder.findNewVSUsingSetupModule = async () => null
+    finder.findNewVS = async () => null
     finder.regSearchKeys = async (keys, value, addOpts) => {
       for (let i = 0; i < keys.length; ++i) {
         const fullName = `${keys[i]}\\${value}`
@@ -222,15 +215,20 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
-    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
-      return null
+    finder.findNewVSUsingSetupModule = async () => null
+    finder.findVisualStudio2019OrNewer = async () => {
+      const file = path.join(__dirname, 'fixtures',
+        'VS_2017_BuildTools_minimal.txt')
+      const data = fs.readFileSync(file)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2019, 2022])
     }
-    finder.findVisualStudio2017OrNewer = async () => {
+    finder.findVisualStudio2017 = async () => {
       const file = path.join(__dirname, 'fixtures',
         'VS_2017_BuildTools_minimal.txt')
       const data = fs.readFileSync(file)
-      const vsInfo = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(vsInfo)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2017])
     }
     const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
@@ -252,15 +250,20 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
-    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
-      return null
+    finder.findNewVSUsingSetupModule = async () => null
+    finder.findVisualStudio2019OrNewer = async () => {
+      const file = path.join(__dirname, 'fixtures',
+        'VS_2017_Community_workload.txt')
+      const data = fs.readFileSync(file)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2019, 2022])
     }
-    finder.findVisualStudio2017OrNewer = async () => {
+    finder.findVisualStudio2017 = async () => {
       const file = path.join(__dirname, 'fixtures',
         'VS_2017_Community_workload.txt')
       const data = fs.readFileSync(file)
-      const vsInfo = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(vsInfo)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2017])
     }
     const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
@@ -282,14 +285,18 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
-    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
-      return null
+    finder.findNewVSUsingSetupModule = async () => null
+    finder.findVisualStudio2019OrNewer = async () => {
+      const file = path.join(__dirname, 'fixtures', 'VS_2017_Express.txt')
+      const data = fs.readFileSync(file)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2019, 2022])
     }
-    finder.findVisualStudio2017OrNewer = async () => {
+    finder.findVisualStudio2017 = async () => {
       const file = path.join(__dirname, 'fixtures', 'VS_2017_Express.txt')
       const data = fs.readFileSync(file)
-      const vsInfo = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(vsInfo)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2017])
     }
     const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
@@ -311,15 +318,20 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
-    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
-      return null
+    finder.findNewVSUsingSetupModule = async () => null
+    finder.findVisualStudio2019OrNewer = async () => {
+      const file = path.join(__dirname, 'fixtures',
+        'VS_2019_Preview.txt')
+      const data = fs.readFileSync(file)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2019, 2022])
     }
-    finder.findVisualStudio2017OrNewer = async () => {
+    finder.findVisualStudio2017 = async () => {
       const file = path.join(__dirname, 'fixtures',
         'VS_2019_Preview.txt')
       const data = fs.readFileSync(file)
-      const vsInfo = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(vsInfo)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2017])
     }
     const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
@@ -341,15 +353,20 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
-    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
-      return null
+    finder.findNewVSUsingSetupModule = async () => null
+    finder.findVisualStudio2019OrNewer = async () => {
+      const file = path.join(__dirname, 'fixtures',
+        'VS_2019_BuildTools_minimal.txt')
+      const data = fs.readFileSync(file)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2019, 2022])
     }
-    finder.findVisualStudio2017OrNewer = async () => {
+    finder.findVisualStudio2017 = async () => {
       const file = path.join(__dirname, 'fixtures',
         'VS_2019_BuildTools_minimal.txt')
       const data = fs.readFileSync(file)
-      const vsInfo = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(vsInfo)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2017])
     }
     const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
@@ -371,15 +388,20 @@ describe('find-visualstudio', function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
 
     poison(finder, 'regSearchKeys')
-    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
-      return null
+    finder.findNewVSUsingSetupModule = async () => null
+    finder.findVisualStudio2019OrNewer = async () => {
+      const file = path.join(__dirname, 'fixtures',
+        'VS_2019_Community_workload.txt')
+      const data = fs.readFileSync(file)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2019, 2022])
     }
-    finder.findVisualStudio2017OrNewer = async () => {
+    finder.findVisualStudio2017 = async () => {
       const file = path.join(__dirname, 'fixtures',
         'VS_2019_Community_workload.txt')
       const data = fs.readFileSync(file)
-      const vsInfo = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(vsInfo)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2017])
     }
     const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
@@ -410,15 +432,20 @@ describe('find-visualstudio', function () {
     finder.msBuildPathExists = (path) => {
       return true
     }
-    finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
-      return null
+    finder.findNewVSUsingSetupModule = async () => null
+    finder.findVisualStudio2019OrNewer = async () => {
+      const file = path.join(__dirname, 'fixtures',
+        'VS_2022_Community_workload.txt')
+      const data = fs.readFileSync(file)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2019, 2022])
     }
-    finder.findVisualStudio2017OrNewer = async () => {
+    finder.findVisualStudio2017 = async () => {
       const file = path.join(__dirname, 'fixtures',
         'VS_2022_Community_workload.txt')
       const data = fs.readFileSync(file)
-      const vsInfo = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(vsInfo)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2017])
     }
     const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
@@ -440,12 +467,8 @@ describe('find-visualstudio', function () {
     finder.msBuildPathExists = (path) => {
       return true
     }
-    finder.findVisualStudio2017OrNewer = async () => {
-      return null
-    }
-    finder.findOldVS = async (info) => {
-      return null
-    }
+    finder.findNewVS = async () => null
+    finder.findOldVS = async () => null
     setupExecFixture(finder, 'VSSetup_VS_2022_workload_missing_sdk.txt')
     const { err, info } = await finder.findVisualStudio()
     assert.match(err.message, /could not find/i)
@@ -489,8 +512,11 @@ describe('find-visualstudio', function () {
       }
       return false
     }
-    finder.findVisualStudio2017OrNewer = async () => {
-      throw new Error("findVisualStudio2017OrNewer shouldn't be called")
+    finder.findVisualStudio2019OrNewer = async () => {
+      throw new Error("findVisualStudio2019OrNewer shouldn't be called")
+    }
+    finder.findVisualStudio2017 = async () => {
+      throw new Error("findVisualStudio2017 shouldn't be called")
     }
     setupExecFixture(finder, fixtureName)
     const { err, info } = await finder.findVisualStudio()
@@ -525,8 +551,7 @@ describe('find-visualstudio', function () {
     finder.findVisualStudio2017OrNewerUsingSetupModule = async () => {
       return null
     }
-
-    finder.findVisualStudio2017OrNewer = async () => {
+    finder.findVisualStudio2017 = async () => {
       const data0 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures',
         'VS_2017_Unusable.txt')))
       const data1 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures',
@@ -535,18 +560,22 @@ describe('find-visualstudio', function () {
         'VS_2017_Community_workload.txt')))
       const data3 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures',
         'VS_2017_Express.txt')))
-      const data4 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures',
+      const data = JSON.stringify(data0.concat(data1, data2, data3))
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2017])
+    }
+    finder.findVisualStudio2019OrNewer = async () => {
+      const data0 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures',
         'VS_2019_Preview.txt')))
-      const data5 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures',
+      const data1 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures',
         'VS_2019_BuildTools_minimal.txt')))
-      const data6 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures',
+      const data2 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures',
         'VS_2019_Community_workload.txt')))
-      const data7 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures',
+      const data3 = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures',
         'VS_2022_Community_workload.txt')))
-      const data = JSON.stringify(data0.concat(data1, data2, data3, data4,
-        data5, data6, data7))
+      const data = JSON.stringify(data0.concat(data1, data2, data3))
       const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(parsedData)
+      return finder.processData(parsedData, [2019, 2022])
     }
     finder.regSearchKeys = async (keys, value, addOpts) => {
       for (let i = 0; i < keys.length; ++i) {

From 0035d8e9dc98b94f0bc8cd9023a6fa635003703e Mon Sep 17 00:00:00 2001
From: Ayushman Chhabra <14110965+ayushmanchhabra@users.noreply.github.com>
Date: Thu, 7 Mar 2024 02:33:52 +0530
Subject: [PATCH 418/551] chore: upgrade release please action from v2 to v4
 (#2982)

* chore(release-please-action): upgrade from v2 to v4

* chore(ci): remove target branch

* chore(release-please): disable `include-component-in-tag`

Co-authored-by: DeeDeeG 

* chore(release-please): format JSON properties per line

Co-authored-by: DeeDeeG 

* Remove last release sha

---------

Co-authored-by: DeeDeeG 
Co-authored-by: Luke Karrys 
---
 .github/workflows/release-please.yml | 36 +------------------------
 .release-please-manifest.json        |  3 +++
 release-please-config.json           | 40 ++++++++++++++++++++++++++++
 3 files changed, 44 insertions(+), 35 deletions(-)
 create mode 100644 .release-please-manifest.json
 create mode 100644 release-please-config.json

diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml
index de4cb087a7..8985302558 100644
--- a/.github/workflows/release-please.yml
+++ b/.github/workflows/release-please.yml
@@ -16,42 +16,8 @@ jobs:
     if: github.event_name == 'push'
     runs-on: ubuntu-latest
     steps:
-    - uses: google-github-actions/release-please-action@v2
+    - uses: google-github-actions/release-please-action@v4
       id: release
-      with:
-        package-name: node-gyp
-        release-type: node
-        changelog-types: >
-          [{"type":"feat","section":"Features","hidden":false},
-          {"type":"fix","section":"Bug Fixes","hidden":false},
-          {"type":"bin","section":"Core","hidden":false},
-          {"type":"gyp","section":"Core","hidden":false},
-          {"type":"lib","section":"Core","hidden":false},
-          {"type":"src","section":"Core","hidden":false},
-          {"type":"test","section":"Tests","hidden":false},
-          {"type":"build","section":"Core","hidden":false},
-          {"type":"clean","section":"Core","hidden":false},
-          {"type":"configure","section":"Core","hidden":false},
-          {"type":"install","section":"Core","hidden":false},
-          {"type":"list","section":"Core","hidden":false},
-          {"type":"rebuild","section":"Core","hidden":false},
-          {"type":"remove","section":"Core","hidden":false},
-          {"type":"deps","section":"Core","hidden":false},
-          {"type":"python","section":"Core","hidden":false},
-          {"type":"lin","section":"Core","hidden":false},
-          {"type":"linux","section":"Core","hidden":false},
-          {"type":"mac","section":"Core","hidden":false},
-          {"type":"macos","section":"Core","hidden":false},
-          {"type":"win","section":"Core","hidden":false},
-          {"type":"windows","section":"Core","hidden":false},
-          {"type":"zos","section":"Core","hidden":false},
-          {"type":"doc","section":"Doc","hidden":false},
-          {"type":"docs","section":"Doc","hidden":false},
-          {"type":"readme","section":"Doc","hidden":false},
-          {"type":"chore","section":"Miscellaneous","hidden":false},
-          {"type":"refactor","section":"Miscellaneous","hidden":false},
-          {"type":"ci","section":"Miscellaneous","hidden":false},
-          {"type":"meta","section":"Miscellaneous","hidden":false}]
         # Standard Conventional Commits: `feat` and `fix`
         # node-gyp subdirectories: `bin`, `gyp`, `lib`, `src`, `test`
         # node-gyp subcommands: `build`, `clean`, `configure`, `install`, `list`, `rebuild`, `remove`
diff --git a/.release-please-manifest.json b/.release-please-manifest.json
new file mode 100644
index 0000000000..31da23be6f
--- /dev/null
+++ b/.release-please-manifest.json
@@ -0,0 +1,3 @@
+{
+    ".": "10.0.1"
+}
diff --git a/release-please-config.json b/release-please-config.json
new file mode 100644
index 0000000000..94b8f8110e
--- /dev/null
+++ b/release-please-config.json
@@ -0,0 +1,40 @@
+{
+    "packages": {
+        ".": {
+            "include-component-in-tag": false,
+            "release-type": "node",
+            "changelog-sections": [
+                { "type": "feat", "section": "Features", "hidden": false },
+                { "type": "fix", "section": "Bug Fixes", "hidden": false },
+                { "type": "bin", "section": "Core", "hidden": false },
+                { "type": "gyp", "section": "Core", "hidden": false },
+                { "type": "lib", "section": "Core", "hidden": false },
+                { "type": "src", "section": "Core", "hidden": false },
+                { "type": "test", "section": "Tests", "hidden": false },
+                { "type": "build", "section": "Core", "hidden": false },
+                { "type": "clean", "section": "Core", "hidden": false },
+                { "type": "configure", "section": "Core", "hidden": false },
+                { "type": "install", "section": "Core", "hidden": false },
+                { "type": "list", "section": "Core", "hidden": false },
+                { "type": "rebuild", "section": "Core", "hidden": false },
+                { "type": "remove", "section": "Core", "hidden": false },
+                { "type": "deps", "section": "Core", "hidden": false },
+                { "type": "python", "section": "Core", "hidden": false },
+                { "type": "lin", "section": "Core", "hidden": false },
+                { "type": "linux", "section": "Core", "hidden": false },
+                { "type": "mac", "section": "Core", "hidden": false },
+                { "type": "macos", "section": "Core", "hidden": false },
+                { "type": "win", "section": "Core", "hidden": false },
+                { "type": "windows", "section": "Core", "hidden": false },
+                { "type": "zos", "section": "Core", "hidden": false },
+                { "type": "doc", "section": "Doc", "hidden": false },
+                { "type": "docs", "section": "Doc", "hidden": false },
+                { "type": "readme", "section": "Doc", "hidden": false },
+                { "type": "chore", "section": "Miscellaneous", "hidden": false },
+                { "type": "refactor", "section": "Miscellaneous", "hidden": false },
+                { "type": "ci", "section": "Miscellaneous", "hidden": false },
+                { "type": "meta", "section": "Miscellaneous", "hidden": false }
+            ]
+        }
+    }
+}

From fbf3fda875e1ef8b298ce306d59ce9d36aec0dbe Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Fri, 8 Mar 2024 15:59:51 -0700
Subject: [PATCH 419/551] Use new branch name format for release please CI
 tests

---
 .github/workflows/release-please.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml
index 8985302558..6b44256ea9 100644
--- a/.github/workflows/release-please.yml
+++ b/.github/workflows/release-please.yml
@@ -30,5 +30,5 @@ jobs:
   test:
     name: Release Test
     needs: [ release-please ]
-    if: needs.release-please.outputs.pr || startsWith(github.head_ref, 'release-v')
+    if: needs.release-please.outputs.pr || startsWith(github.head_ref, 'release-please--')
     uses: ./.github/workflows/tests.yml

From 1205fb084c0ed571429cf2ab12e885788be68e5f Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Wed, 13 Mar 2024 05:54:17 +0100
Subject: [PATCH 420/551] Fix lint: ruff check --select=UP032 --fix (#2994)

---
 gyp/pylib/gyp/generator/android.py |  6 +++---
 gyp/pylib/gyp/generator/gypsh.py   |  7 +++----
 gyp/pylib/gyp/generator/msvs.py    | 20 ++++++++------------
 gyp/pylib/gyp/input.py             | 10 ++++------
 gyp/pylib/gyp/msvs_emulation.py    |  9 +++------
 gyp/pyproject.toml                 |  8 ++++----
 6 files changed, 25 insertions(+), 35 deletions(-)

diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py
index d3c97c666d..9a79670214 100644
--- a/gyp/pylib/gyp/generator/android.py
+++ b/gyp/pylib/gyp/generator/android.py
@@ -739,9 +739,9 @@ def ComputeOutput(self, spec):
                     % (self.android_class, self.android_module)
                 )
             else:
-                path = "$(call intermediates-dir-for,{},{},,,$(GYP_VAR_PREFIX))".format(
-                    self.android_class,
-                    self.android_module,
+                path = (
+                    "$(call intermediates-dir-for,"
+                    f"{self.android_class},{self.android_module},,,$(GYP_VAR_PREFIX))"
                 )
 
         assert spec.get("product_dir") is None  # TODO: not supported?
diff --git a/gyp/pylib/gyp/generator/gypsh.py b/gyp/pylib/gyp/generator/gypsh.py
index 82a07ddc65..625b6d65ca 100644
--- a/gyp/pylib/gyp/generator/gypsh.py
+++ b/gyp/pylib/gyp/generator/gypsh.py
@@ -49,10 +49,9 @@ def GenerateOutput(target_list, target_dicts, data, params):
     # Use a banner that looks like the stock Python one and like what
     # code.interact uses by default, but tack on something to indicate what
     # locals are available, and identify gypsh.
-    banner = "Python {} on {}\nlocals.keys() = {}\ngypsh".format(
-        sys.version,
-        sys.platform,
-        repr(sorted(locals.keys())),
+    banner = (
+        f"Python {sys.version} on {sys.platform}\nlocals.keys() = "
+        f"{repr(sorted(locals.keys()))}\ngypsh"
     )
 
     code.interact(banner, local=locals)
diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py
index 13b0794b4d..6f0f8c1ab6 100644
--- a/gyp/pylib/gyp/generator/msvs.py
+++ b/gyp/pylib/gyp/generator/msvs.py
@@ -1778,11 +1778,9 @@ def _GetCopies(spec):
                 outer_dir = posixpath.split(src_bare)[1]
                 fixed_dst = _FixPath(dst)
                 full_dst = f'"{fixed_dst}\\{outer_dir}\\"'
-                cmd = 'mkdir {} 2>nul & cd "{}" && xcopy /e /f /y "{}" {}'.format(
-                    full_dst,
-                    _FixPath(base_dir),
-                    outer_dir,
-                    full_dst,
+                cmd = (
+                    f'mkdir {full_dst} 2>nul & cd "{_FixPath(base_dir)}" && '
+                    f'xcopy /e /f /y "{outer_dir}" {full_dst}'
                 )
                 copies.append(
                     (
@@ -1794,10 +1792,9 @@ def _GetCopies(spec):
                 )
             else:
                 fix_dst = _FixPath(cpy["destination"])
-                cmd = 'mkdir "{}" 2>nul & set ERRORLEVEL=0 & copy /Y "{}" "{}"'.format(
-                    fix_dst,
-                    _FixPath(src),
-                    _FixPath(dst),
+                cmd = (
+                    f'mkdir "{fix_dst}" 2>nul & set ERRORLEVEL=0 & '
+                    f'copy /Y "{_FixPath(src)}" "{_FixPath(dst)}"'
                 )
                 copies.append(([src], [dst], cmd, f"Copying {src} to {fix_dst}"))
     return copies
@@ -1899,9 +1896,8 @@ def _GetPlatformOverridesOfProject(spec):
     for config_name, c in spec["configurations"].items():
         config_fullname = _ConfigFullName(config_name, c)
         platform = c.get("msvs_target_platform", _ConfigPlatform(c))
-        fixed_config_fullname = "{}|{}".format(
-            _ConfigBaseName(config_name, _ConfigPlatform(c)),
-            platform,
+        fixed_config_fullname = (
+            f"{_ConfigBaseName(config_name, _ConfigPlatform(c))}|{platform}"
         )
         if spec["toolset"] == "host" and generator_supports_multiple_toolsets:
             fixed_config_fullname = f"{config_name}|x64"
diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py
index 8f39519dee..0b56c72750 100644
--- a/gyp/pylib/gyp/input.py
+++ b/gyp/pylib/gyp/input.py
@@ -1135,18 +1135,16 @@ def EvalCondition(condition, conditions_key, phase, variables, build_file):
         true_dict = condition[i + 1]
         if type(true_dict) is not dict:
             raise GypError(
-                "{} {} must be followed by a dictionary, not {}".format(
-                    conditions_key, cond_expr, type(true_dict)
-                )
+                f"{conditions_key} {cond_expr} must be followed by a dictionary, not "
+                f"{type(true_dict)}"
             )
         if len(condition) > i + 2 and type(condition[i + 2]) is dict:
             false_dict = condition[i + 2]
             i = i + 3
             if i != len(condition):
                 raise GypError(
-                    "{} {} has {} unexpected trailing items".format(
-                        conditions_key, cond_expr, len(condition) - i
-                    )
+                    f"{conditions_key} {cond_expr} has {len(condition) - i} "
+                    "unexpected trailing items"
                 )
         else:
             false_dict = None
diff --git a/gyp/pylib/gyp/msvs_emulation.py b/gyp/pylib/gyp/msvs_emulation.py
index 38fa21dd66..847d1b8dc1 100644
--- a/gyp/pylib/gyp/msvs_emulation.py
+++ b/gyp/pylib/gyp/msvs_emulation.py
@@ -830,17 +830,14 @@ def _GetLdManifestFlags(
                 ("VCLinkerTool", "UACUIAccess"), config, default="false"
             )
 
-            inner = """
+            inner = f"""
 
   
     
-      
+      
     
   
-""".format(
-                execution_level_map[execution_level],
-                ui_access,
-            )
+"""  # noqa: E501
         else:
             inner = ""
 
diff --git a/gyp/pyproject.toml b/gyp/pyproject.toml
index 0c25d0b3c1..7183e07d3c 100644
--- a/gyp/pyproject.toml
+++ b/gyp/pyproject.toml
@@ -38,7 +38,7 @@ gyp = "gyp:script_main"
 "Homepage" = "https://github.com/nodejs/gyp-next"
 
 [tool.ruff]
-select = [
+lint.select = [
   "C4",   # flake8-comprehensions
   "C90",  # McCabe cyclomatic complexity
   "DTZ",  # flake8-datetimez
@@ -87,7 +87,7 @@ select = [
   # "T20",  # flake8-print
   # "TRY",  # tryceratops
 ]
-ignore = [
+lint.ignore = [
   "E721",
   "PLC1901",
   "PLR0402",
@@ -105,10 +105,10 @@ extend-exclude = ["pylib/packaging"]
 line-length = 88
 target-version = "py37"
 
-[tool.ruff.mccabe]
+[tool.ruff.lint.mccabe]
 max-complexity = 101
 
-[tool.ruff.pylint]
+[tool.ruff.lint.pylint]
 max-args = 11
 max-branches = 108
 max-returns = 10

From c4729129daa9bb5204246b857826fb391ac961e1 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?H=C3=BCseyin=20A=C3=A7acak?=
 <110401522+huseyinacacak-janea@users.noreply.github.com>
Date: Wed, 13 Mar 2024 19:20:35 +0300
Subject: [PATCH 421/551] lib: print Python executable path using UTF-8 (#2995)

* lib: print Python executable path using UTF-8

The Python executable path may have non-ASCII characters, which can make
the print function fail if the environment encoding is different. This fixes
this issue by using stdout.buffer, which can be used with UTF-8 encoding
for the output, regardless of the environment encoding.

Fixes: https://github.com/nodejs/node-gyp/issues/2829

* fixup! lib: print Python executable path using UTF-8
---
 lib/find-python.js       |  2 +-
 test/test-find-python.js | 31 ++++++++++++++++++++++++++++++-
 2 files changed, 31 insertions(+), 2 deletions(-)

diff --git a/lib/find-python.js b/lib/find-python.js
index 615da57bb8..a71c00c2b6 100644
--- a/lib/find-python.js
+++ b/lib/find-python.js
@@ -41,7 +41,7 @@ class PythonFinder {
   static findPython = (...args) => new PythonFinder(...args).findPython()
 
   log = log.withPrefix('find Python')
-  argsExecutable = ['-c', 'import sys; print(sys.executable);']
+  argsExecutable = ['-c', 'import sys; sys.stdout.buffer.write(sys.executable.encode(\'utf-8\'));']
   argsVersion = ['-c', 'import sys; print("%s.%s.%s" % sys.version_info[:3]);']
   semverRange = '>=3.6.0'
 
diff --git a/test/test-find-python.js b/test/test-find-python.js
index c5fa35d4bc..dc7192809a 100644
--- a/test/test-find-python.js
+++ b/test/test-find-python.js
@@ -2,11 +2,14 @@
 
 delete process.env.PYTHON
 
-const { describe, it } = require('mocha')
+const { describe, it, after } = require('mocha')
 const assert = require('assert')
 const PythonFinder = require('../lib/find-python')
 const { execFile } = require('../lib/util')
 const { poison } = require('./common')
+const fs = require('fs')
+const path = require('path')
+const os = require('os')
 
 class TestPythonFinder extends PythonFinder {
   constructor (...args) {
@@ -32,6 +35,32 @@ describe('find-python', function () {
     assert.strictEqual(stderr, '')
   })
 
+  it('find python - encoding', async function () {
+    const found = await PythonFinder.findPython(null)
+    const testFolderPath = fs.mkdtempSync(path.join(os.tmpdir(), 'test-ü-'))
+    const testFilePath = path.join(testFolderPath, 'python.exe')
+    after(function () {
+      try {
+        fs.unlinkSync(testFilePath)
+        fs.rmdirSync(testFolderPath)
+      } catch {}
+    })
+
+    try {
+      fs.symlinkSync(found, testFilePath)
+    } catch (err) {
+      switch (err.code) {
+        case 'EPERM':
+          return assert.fail(err, null, 'Please try to run console as an administrator')
+        default:
+          return assert.fail(err)
+      }
+    }
+
+    const finder = new PythonFinder(testFilePath)
+    await assert.doesNotReject(finder.checkCommand(testFilePath))
+  })
+
   it('find python - python', async function () {
     const f = new TestPythonFinder('python')
     f.execFile = async function (program, args, opts) {

From f90ce122fe564be68368d0c0dec5dacd9e770233 Mon Sep 17 00:00:00 2001
From: "github-actions[bot]"
 <41898282+github-actions[bot]@users.noreply.github.com>
Date: Mon, 25 Mar 2024 11:04:31 -0700
Subject: [PATCH 422/551] chore(main): release 10.1.0 (#2989)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
---
 .release-please-manifest.json |  2 +-
 CHANGELOG.md                  | 29 +++++++++++++++++++++++++++++
 package.json                  |  2 +-
 3 files changed, 31 insertions(+), 2 deletions(-)

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
index 31da23be6f..1842506cfa 100644
--- a/.release-please-manifest.json
+++ b/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "10.0.1"
+    ".": "10.1.0"
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bcd167d9bc..9db4f9f952 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,34 @@
 # Changelog
 
+## [10.1.0](https://github.com/nodejs/node-gyp/compare/v10.0.1...v10.1.0) (2024-03-13)
+
+
+### Features
+
+* improve visual studio detection ([#2957](https://github.com/nodejs/node-gyp/issues/2957)) ([109e3d4](https://github.com/nodejs/node-gyp/commit/109e3d4245504a7b75c99f578e1203c0ef4b518e))
+
+
+### Core
+
+* add support for locally installed headers ([#2964](https://github.com/nodejs/node-gyp/issues/2964)) ([3298731](https://github.com/nodejs/node-gyp/commit/329873141f0d3e3787d3c006801431da04e4ed0c))
+* **deps:** bump actions/setup-python from 4 to 5 ([#2960](https://github.com/nodejs/node-gyp/issues/2960)) ([3f0df7e](https://github.com/nodejs/node-gyp/commit/3f0df7e9334e49e8c7f6fdbbb9e1e6c5a8cca53b))
+* **deps:** bump google-github-actions/release-please-action ([#2961](https://github.com/nodejs/node-gyp/issues/2961)) ([b1f1808](https://github.com/nodejs/node-gyp/commit/b1f1808bfff0d51e6d3eb696ab6a5b89b7b9630c))
+* print Python executable path using UTF-8 ([#2995](https://github.com/nodejs/node-gyp/issues/2995)) ([c472912](https://github.com/nodejs/node-gyp/commit/c4729129daa9bb5204246b857826fb391ac961e1))
+* update supported vs versions ([#2959](https://github.com/nodejs/node-gyp/issues/2959)) ([391cc5b](https://github.com/nodejs/node-gyp/commit/391cc5b9b25cffe0cb2edcba3583414a771b4a15))
+
+
+### Doc
+
+* npm is currently v10 ([#2970](https://github.com/nodejs/node-gyp/issues/2970)) ([7705a22](https://github.com/nodejs/node-gyp/commit/7705a22f31a62076e9f8429780a459f4ad71ea4c))
+* remove outdated Node versions from readme ([#2955](https://github.com/nodejs/node-gyp/issues/2955)) ([ae8478e](https://github.com/nodejs/node-gyp/commit/ae8478ec32d9b2fa71b591ac22cdf867ef2e9a7d))
+* remove outdated update engines.node reference in 10.0.0 changelog ([b42e796](https://github.com/nodejs/node-gyp/commit/b42e7966177f006f3d1aab1d27885d8372c8ed01))
+
+
+### Miscellaneous
+
+* only run release please on push ([cff9ac2](https://github.com/nodejs/node-gyp/commit/cff9ac2c3083769a383e00bc60b91562f03116e3))
+* upgrade release please action from v2 to v4 ([#2982](https://github.com/nodejs/node-gyp/issues/2982)) ([0035d8e](https://github.com/nodejs/node-gyp/commit/0035d8e9dc98b94f0bc8cd9023a6fa635003703e))
+
 ### [10.0.1](https://www.github.com/nodejs/node-gyp/compare/v10.0.0...v10.0.1) (2023-11-02)
 
 
diff --git a/package.json b/package.json
index 80c63f2e72..95f012fa5d 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "10.0.1",
+  "version": "10.1.0",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From 9fd7936f0d7232a8a79e6a7b6cbfb814d9042b13 Mon Sep 17 00:00:00 2001
From: SmallY <45689960+iamSmallY@users.noreply.github.com>
Date: Wed, 3 Apr 2024 17:29:34 +0800
Subject: [PATCH 423/551] docs: add the way to configuring Python dependency
 for Windows PowerShell (#2996)

---
 README.md | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/README.md b/README.md
index 9e4c608e20..2570084add 100644
--- a/README.md
+++ b/README.md
@@ -81,7 +81,8 @@ export npm_config_python=/path/to/executable/python
     Or on Windows:
 ```console
 py --list-paths  # To see the installed Python versions
-set npm_config_python=C:\path\to\python.exe
+set npm_config_python=C:\path\to\python.exe  # CMD
+$Env:npm_config_python="C:\path\to\python.exe"  # PowerShell
 ```
 
 3. If the `PYTHON` environment variable is set to the path of a Python executable,

From 7647a78bd7e4cad13179578249fdffe60a3ab72d Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Sat, 6 Apr 2024 00:03:30 +0200
Subject: [PATCH 424/551] Update tests.yml

---
 .github/workflows/tests.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 6835cceed5..c97a11a9c2 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -84,7 +84,7 @@ jobs:
         FULL_TEST: '1'
       run: |
         npm install
-        npm test
+        # npm test  # Why is this running Windows tests on macOS?!?
 
   tests:
     # lint-python takes ~5 seconds, so wait for it to pass before running the full matrix of tests.

From e1304e870bd5cee413a755677bc8dc289a04fda4 Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Sat, 6 Apr 2024 00:16:38 +0200
Subject: [PATCH 425/551] Revert

---
 .github/workflows/tests.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index c97a11a9c2..6835cceed5 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -84,7 +84,7 @@ jobs:
         FULL_TEST: '1'
       run: |
         npm install
-        # npm test  # Why is this running Windows tests on macOS?!?
+        npm test
 
   tests:
     # lint-python takes ~5 seconds, so wait for it to pass before running the full matrix of tests.

From a6b48fca9993e54d757cd110f6b41f8200d99ca4 Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Sun, 7 Apr 2024 11:59:14 +0200
Subject: [PATCH 426/551] docs: Installation -- Python >= v3.12 requires
 `node-gyp` >= v10 (#3010)

We are getting a lot of issues opened on this topic so add it to our installation instructions.
* #2869

> [!Important]
> Python >= v3.12 requires `node-gyp` >= v10
---
 README.md | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/README.md b/README.md
index 2570084add..03a2d40dc1 100644
--- a/README.md
+++ b/README.md
@@ -21,6 +21,9 @@ All current and LTS target versions of Node.js are supported. Depending on what
 
 ## Installation
 
+> [!Important]
+> Python >= v3.12 requires `node-gyp` >= v10
+
 You can install `node-gyp` using `npm`:
 
 ``` bash

From 77737e45db5cd7873f440c43154302e22ef21c85 Mon Sep 17 00:00:00 2001
From: Victor Ejike Nwosu <74430629+Eprince-hub@users.noreply.github.com>
Date: Sun, 7 Apr 2024 22:04:04 +0200
Subject: [PATCH 427/551] Add Chocolatey Guide for node-gyp on Windows (#3008)

* Add chocolatey install

* Revert unwanted diffs

* Add chocolatey guide

* Use bash for code block

* Update README.md

Co-authored-by: Christian Clauss 

* Move chocolatey bash to top

* Fix section identation

---------

Co-authored-by: Christian Clauss 
---
 README.md | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/README.md b/README.md
index 03a2d40dc1..474c59b458 100644
--- a/README.md
+++ b/README.md
@@ -48,10 +48,16 @@ Depending on your operating system, you will need to install:
 
 ### On Windows
 
-Install the current [version of Python](https://devguide.python.org/versions/) from the
-[Microsoft Store](https://apps.microsoft.com/store/search?publisher=Python+Software+Foundation).
+Install tools with [Chocolatey](https://chocolatey.org):
+``` bash
+choco install python visualstudio2022-workload-vctools -y
+```
+
+Or install and configure Python and Visual Studio tools manually:
+
+  * Install the current [version of Python](https://devguide.python.org/versions/) from the
+  [Microsoft Store](https://apps.microsoft.com/store/search?publisher=Python+Software+Foundation).
 
-Install tools and configuration manually:
    * Install Visual C++ Build Environment: For Visual Studio 2019 or later, use the `Desktop development with C++` workload from [Visual Studio Community](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=Community).  For a version older than Visual Studio 2019, install [Visual Studio Build Tools](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) with the `Visual C++ build tools` option.
 
    If the above steps didn't work for you, please visit [Microsoft's Node.js Guidelines for Windows](https://github.com/Microsoft/nodejs-guidelines/blob/master/windows-environment.md#compiling-native-addon-modules) for additional tips.

From 0bab6a071bbc7bafb0804eccf0e2e81e1815d481 Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Thu, 11 Apr 2024 17:34:49 +0200
Subject: [PATCH 428/551] GitHub Actions: npm test is failing Windows tests on
 M1 Macs (#3011)

* GitHub Actions: npm test is failing Windows tests on M1 Macs

`npm test` is running and failing Windows `find-visualstudio` tests on an M1 Mac!!!

These tests are not being run or not failing on Intel Macs.

* Only run "Find Visual Studio" tests on Windows

* Update test/test-find-visualstudio.js

Co-authored-by: Christian Clauss 

---------

Co-authored-by: Stefan Stojanovic 
---
 .github/workflows/tests.yml    | 14 +++++++++-----
 test/test-find-visualstudio.js |  6 ++++++
 2 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 6835cceed5..6c4541f53a 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -93,11 +93,15 @@ jobs:
       fail-fast: false
       max-parallel: 15
       matrix:
-        os: [macos, ubuntu, windows]
+        os: [macos-latest, ubuntu-latest, windows-latest]
         python: ["3.8", "3.10", "3.12"]
         node: [16.x, 18.x, 20.x]
+        include:  # `npm test` is running Windows find-visualstudio tests on an M1 Mac!!!
+          - os: macos-14
+            python: "3.12"
+            node: 20.x
     name: ${{ matrix.os }} - ${{ matrix.python }} - ${{ matrix.node }}
-    runs-on: ${{ matrix.os }}-latest
+    runs-on: ${{ matrix.os }}
     steps:
       - name: Checkout Repository
         uses: actions/checkout@v4
@@ -116,20 +120,20 @@ jobs:
           npm install
           pip install pytest
       - name: Set Windows Env
-        if: runner.os == 'Windows'
+        if: startsWith(matrix.os, 'windows')
         run: |
           echo 'GYP_MSVS_VERSION=2015' >> $Env:GITHUB_ENV
           echo 'GYP_MSVS_OVERRIDE_PATH=C:\\Dummy' >> $Env:GITHUB_ENV
       - name: Run Python Tests
         run: python -m pytest
       - name: Run Tests (macOS or Linux)
-        if: runner.os != 'Windows'
+        if: "!startsWith(matrix.os, 'windows')"
         shell: bash
         run: npm test --python="${pythonLocation}/python"
         env:
           FULL_TEST: ${{ (matrix.node == '20.x' && matrix.python == '3.12') && '1' || '0' }}
       - name: Run Tests (Windows)
-        if: runner.os == 'Windows'
+        if: startsWith(matrix.os, 'windows')
         shell: pwsh
         run: npm run test --python="${env:pythonLocation}\\python.exe"
         env:
diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js
index 2c3f4e1981..41dd837830 100644
--- a/test/test-find-visualstudio.js
+++ b/test/test-find-visualstudio.js
@@ -21,7 +21,13 @@ class TestVisualStudioFinder extends VisualStudioFinder {
   }
 }
 
+const shouldSkip = process.platform !== 'win32'
+
 describe('find-visualstudio', function () {
+  if (shouldSkip) {
+    return
+  }
+
   this.beforeAll(function () {
     // Condition to skip the test suite
     if (process.env.SystemRoot === undefined) {

From b22d5eef861892c968052ffc1c71b551f738163b Mon Sep 17 00:00:00 2001
From: Matt 
Date: Mon, 15 Apr 2024 17:03:22 -0700
Subject: [PATCH 429/551] deps: tar@6.2.1 (#3021)

---
 package.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package.json b/package.json
index 95f012fa5d..6a24a1aaf0 100644
--- a/package.json
+++ b/package.json
@@ -30,7 +30,7 @@
     "nopt": "^7.0.0",
     "proc-log": "^3.0.0",
     "semver": "^7.3.5",
-    "tar": "^6.1.2",
+    "tar": "^6.2.1",
     "which": "^4.0.0"
   },
   "engines": {

From 141aa6bf029e6f984be8ea98aaf985e5df894082 Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Mon, 15 Apr 2024 17:03:47 -0700
Subject: [PATCH 430/551] deps: proc-log@4.0.0 (#3022)

---
 lib/log.js   | 11 +++++------
 package.json |  2 +-
 2 files changed, 6 insertions(+), 7 deletions(-)

diff --git a/lib/log.js b/lib/log.js
index 6841719aba..36fa2487f5 100644
--- a/lib/log.js
+++ b/lib/log.js
@@ -1,12 +1,11 @@
 'use strict'
 
-const procLog = require('proc-log')
+const { log } = require('proc-log')
 const { format } = require('util')
 
 // helper to emit log messages with a predefined prefix
-const logLevels = Object.keys(procLog).filter((k) => typeof procLog[k] === 'function')
-const withPrefix = (prefix) => logLevels.reduce((acc, level) => {
-  acc[level] = (...args) => procLog[level](prefix, ...args)
+const withPrefix = (prefix) => log.LEVELS.reduce((acc, level) => {
+  acc[level] = (...args) => log[level](prefix, ...args)
   return acc
 }, {})
 
@@ -78,7 +77,7 @@ class Logger {
     this.#levels = new Map(this.#levels.map((level, index) => [level.id, { ...level, index }]))
     this.level = 'info'
     this.stream = stream
-    procLog.pause()
+    log.pause()
   }
 
   get stream () {
@@ -165,5 +164,5 @@ module.exports = {
   logger: new Logger(NULL_LOGGER ? null : process.stderr),
   stdout: NULL_LOGGER ? () => {} : (...args) => console.log(...args),
   withPrefix,
-  ...procLog
+  ...log
 }
diff --git a/package.json b/package.json
index 6a24a1aaf0..e9402d7923 100644
--- a/package.json
+++ b/package.json
@@ -28,7 +28,7 @@
     "graceful-fs": "^4.2.6",
     "make-fetch-happen": "^13.0.0",
     "nopt": "^7.0.0",
-    "proc-log": "^3.0.0",
+    "proc-log": "^4.1.0",
     "semver": "^7.3.5",
     "tar": "^6.2.1",
     "which": "^4.0.0"

From 93186f10c966b4148fc500e48f8cbffacccdfa3c Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Tue, 16 Apr 2024 02:05:08 +0200
Subject: [PATCH 431/551] docs: `node-pre-gyp` is no longer maintained (#3015)

* docs: `node-pre-gyp` is no longer maintained

* Update README.md

* Update README.md
---
 docs/README.md | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/docs/README.md b/docs/README.md
index 3b8c86179d..90cbedc491 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -1,4 +1,4 @@
-## Versions of `node-gyp` that are earlier than v9.x.x
+## Versions of `node-gyp` that are earlier than v10.x.x
 
 Please look thru your error log for the string `gyp info using node-gyp@` and if that version number is less than the [current release of node-gyp](https://github.com/nodejs/node-gyp/releases) then __please upgrade__ using [these instructions](https://github.com/nodejs/node-gyp/blob/main/docs/Updating-npm-bundled-node-gyp.md) and then try your command again.
 
@@ -12,8 +12,20 @@ npm install sass --save
 npm install --global node-sass@latest
 ```
 `node-sass` projects _may_ work by downgrading to Node.js v14 but [that release is end-of-life](https://github.com/nodejs/release#release-schedule).
-But in any case, please avoid opening new `node-sass` issues on this repo because we [cannot help much](https://github.com/nodejs/node-gyp/issues?q=is%3Aissue+label%3A%22Node+Sass+--%3E+Dart+Sass%22+).
 
-## Issues finding the installed Visual Studio
+In any case, please avoid opening new `node-sass` issues on this repo because we [cannot help much](https://github.com/nodejs/node-gyp/issues?q=is%3Aissue+label%3A%22Node+Sass+--%3E+Dart+Sass%22+).
 
-In cmd, [`npm config set msvs_version 20xx`](https://github.com/nodejs/node-gyp#on-windows) with ___xx___ matching your locally installed version of Visual Studio.
+## `node-pre-gyp` is no longer maintained
+
+* mapbox/node-pre-gyp#657
+
+Support in the `abi_crosswalk.json` file ends at Node.js v17 but [that release is end-of-life](https://github.com/nodejs/release#release-schedule).
+
+In any case, please avoid opening new `node-pre-gyp` issues on this repo because we [cannot help much](https://github.com/nodejs/node-gyp/issues?q=is%3Aissue+label%3A%22node-pre-gyp+is+unmaintained%22).
+
+Unsupported __WORKAROUND__ for versions of Node.js > v17
+```
+npm ci  # mapbox/node-pre-gyp
+npm run update-crosswalk
+# npm audit  # Currently fails on a `Severity: critical` issue
+```

From 323957b74e9586fb3fbfb2acad5040379c778de6 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?H=C3=BCseyin=20A=C3=A7acak?=
 <110401522+huseyinacacak-janea@users.noreply.github.com>
Date: Tue, 23 Apr 2024 16:46:11 +0300
Subject: [PATCH 432/551] win: add an arch check to VS 2019 (#3025)

---
 lib/find-visualstudio.js       | 6 +++++-
 test/test-find-visualstudio.js | 6 ------
 2 files changed, 5 insertions(+), 7 deletions(-)

diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js
index 8c5ae96127..a78e763480 100644
--- a/lib/find-visualstudio.js
+++ b/lib/find-visualstudio.js
@@ -363,7 +363,11 @@ class VisualStudioFinder {
         return path.join(info.path, 'MSBuild', '15.0', 'Bin', 'MSBuild.exe')
       }
       if (versionYear === 2019) {
-        return msbuildPath
+        if (process.arch === 'arm64' && this.msBuildPathExists(msbuildPathArm64)) {
+          return msbuildPathArm64
+        } else {
+          return msbuildPath
+        }
       }
     }
     /**
diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js
index 41dd837830..2c3f4e1981 100644
--- a/test/test-find-visualstudio.js
+++ b/test/test-find-visualstudio.js
@@ -21,13 +21,7 @@ class TestVisualStudioFinder extends VisualStudioFinder {
   }
 }
 
-const shouldSkip = process.platform !== 'win32'
-
 describe('find-visualstudio', function () {
-  if (shouldSkip) {
-    return
-  }
-
   this.beforeAll(function () {
     // Condition to skip the test suite
     if (process.env.SystemRoot === undefined) {

From c495083d991e26de74ec998110d6259675328d65 Mon Sep 17 00:00:00 2001
From: Sharad Chandran R 
Date: Fri, 31 May 2024 14:56:21 +0530
Subject: [PATCH 433/551] Update binding.gyp-files-in-the-wild.md (#3023)

Add node-oracledb binding.gyp file
---
 docs/binding.gyp-files-in-the-wild.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/docs/binding.gyp-files-in-the-wild.md b/docs/binding.gyp-files-in-the-wild.md
index 795d2fd2ec..be8efaddfe 100644
--- a/docs/binding.gyp-files-in-the-wild.md
+++ b/docs/binding.gyp-files-in-the-wild.md
@@ -39,6 +39,7 @@ To add to this page, just add the link to the project's `binding.gyp` file below
  * [node-osmium](https://github.com/osmcode/node-osmium/blob/master/binding.gyp)
  * [node-osrm](https://github.com/DennisOSRM/node-osrm)
  * [node-oracle](https://github.com/joeferner/node-oracle/blob/master/binding.gyp)
+ * [node-oracledb](https://github.com/oracle/node-oracledb/blob/main/binding.gyp)
  * [node-process-list](https://github.com/ReklatsMasters/node-process-list/blob/master/binding.gyp)
  * [node-nanomsg](https://github.com/nickdesaulniers/node-nanomsg/blob/master/binding.gyp)
  * [Ghostscript4JS](https://github.com/NickNaso/ghostscript4js/blob/master/binding.gyp)

From ea99fea83485dc5be04db01df9b2fdbe05319b8e Mon Sep 17 00:00:00 2001
From: Toyo Li 
Date: Wed, 12 Jun 2024 17:23:37 +0800
Subject: [PATCH 434/551] feat(gyp): update gyp to v0.18.1 (#3039)

* feat(gyp): update gyp to v0.18.1

* ci: setup ninja

* ci: visual-studio job use python 3.12
---
 .github/workflows/tests.yml                   |    1 +
 .github/workflows/visual-studio.yml           |    8 +-
 gyp/.github/dependabot.yml                    |   20 +
 gyp/.github/workflows/Python_tests.yml        |   11 +-
 gyp/.github/workflows/node-gyp.yml            |   14 +-
 gyp/.github/workflows/release-please.yml      |   93 +-
 gyp/.release-please-manifest.json             |    3 +
 gyp/CHANGELOG.md                              |   41 +
 gyp/CONTRIBUTING.md                           |    4 +
 gyp/README.md                                 |    2 +-
 gyp/data/ninja/build.ninja                    |    4 +
 gyp/docs/GypVsCMake.md                        |  116 ++
 gyp/docs/Hacking.md                           |   46 +
 gyp/docs/InputFormatReference.md              | 1080 +++++++++++++++++
 gyp/docs/LanguageSpecification.md             |  430 +++++++
 gyp/docs/README.md                            |   27 +
 gyp/docs/Testing.md                           |  450 +++++++
 gyp/docs/UserDocumentation.md                 |  965 +++++++++++++++
 gyp/pylib/gyp/MSVSSettings.py                 |    2 +
 gyp/pylib/gyp/common.py                       |   63 +-
 gyp/pylib/gyp/common_test.py                  |  103 +-
 gyp/pylib/gyp/generator/android.py            |    4 +-
 .../gyp/generator/compile_commands_json.py    |   10 +-
 gyp/pylib/gyp/generator/gypsh.py              |    2 +-
 gyp/pylib/gyp/generator/make.py               |   80 +-
 gyp/pylib/gyp/generator/msvs.py               |    9 +-
 gyp/pylib/gyp/generator/ninja.py              |   31 +
 gyp/pylib/gyp/generator/ninja_test.py         |   12 +
 gyp/pylib/gyp/input.py                        |   10 +-
 gyp/pylib/gyp/msvs_emulation.py               |    5 +-
 gyp/pylib/gyp/xcode_emulation.py              |   29 +-
 gyp/pylib/gyp/xcode_emulation_test.py         |   53 +
 gyp/pyproject.toml                            |   19 +-
 gyp/release-please-config.json                |   11 +
 34 files changed, 3669 insertions(+), 89 deletions(-)
 create mode 100644 gyp/.github/dependabot.yml
 create mode 100644 gyp/.release-please-manifest.json
 create mode 100644 gyp/data/ninja/build.ninja
 create mode 100644 gyp/docs/GypVsCMake.md
 create mode 100644 gyp/docs/Hacking.md
 create mode 100644 gyp/docs/InputFormatReference.md
 create mode 100644 gyp/docs/LanguageSpecification.md
 create mode 100644 gyp/docs/README.md
 create mode 100644 gyp/docs/Testing.md
 create mode 100644 gyp/docs/UserDocumentation.md
 create mode 100644 gyp/pylib/gyp/xcode_emulation_test.py
 create mode 100644 gyp/release-please-config.json

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 6c4541f53a..1e8e8bc8cb 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -115,6 +115,7 @@ jobs:
           python-version: ${{ matrix.python }}
         env:
           PYTHON_VERSION: ${{ matrix.python }}  # Why do this?
+      - uses: seanmiddleditch/gha-setup-ninja@v4
       - name: Install Dependencies
         run: |
           npm install
diff --git a/.github/workflows/visual-studio.yml b/.github/workflows/visual-studio.yml
index 19993a57f9..0f51f9c8d0 100644
--- a/.github/workflows/visual-studio.yml
+++ b/.github/workflows/visual-studio.yml
@@ -25,10 +25,12 @@ jobs:
     steps:
       - name: Checkout Repository
         uses: actions/checkout@v4
+      - name: Use Python 3.12
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
       - name: Install Dependencies
         run: npm install
       - name: Run Node tests
         shell: pwsh
-        run: |
-          $pythonLocation = (Get-Command python).Source
-          npm run test --python="${pythonLocation}" --msvs-version="${{ matrix.msvs-version }}"
+        run: npm run test --python="${env:pythonLocation}\\python.exe" --msvs-version="${{ matrix.msvs-version }}"
diff --git a/gyp/.github/dependabot.yml b/gyp/.github/dependabot.yml
new file mode 100644
index 0000000000..58d68276fc
--- /dev/null
+++ b/gyp/.github/dependabot.yml
@@ -0,0 +1,20 @@
+# Keep GitHub Actions up to date with Dependabot...
+# https://docs.github.com/en/code-security/dependabot/working-with-dependabot/keeping-your-actions-up-to-date-with-dependabot
+version: 2
+updates:
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    groups:
+      GitHub_Actions:
+        patterns:
+          - "*"  # Group all Actions updates into a single larger pull request
+    schedule:
+      interval: weekly
+  - package-ecosystem: "pip"
+    directory: "/"
+    groups:
+      pip:
+        patterns:
+          - "*"  # Group all pip updates into a single larger pull request
+    schedule:
+      interval: weekly
diff --git a/gyp/.github/workflows/Python_tests.yml b/gyp/.github/workflows/Python_tests.yml
index 049d5fe50c..5c86466378 100644
--- a/gyp/.github/workflows/Python_tests.yml
+++ b/gyp/.github/workflows/Python_tests.yml
@@ -15,15 +15,16 @@ jobs:
       fail-fast: false
       max-parallel: 5
       matrix:
-        os: [macos-latest, ubuntu-latest] # , windows-latest]
-        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
+        os: [macos-13, macos-14, ubuntu-latest] # , windows-latest]
+        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"]
     steps:
       - uses: actions/checkout@v4
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
           allow-prereleases: true
+      - uses: seanmiddleditch/gha-setup-ninja@v4
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip setuptools
@@ -35,3 +36,7 @@ jobs:
         run: pytest
       # - name: Run doctests with pytest
       #   run: pytest --doctest-modules
+      - name: Test CLI commands on a pipx install
+        run: |
+          pipx run --no-cache --spec ./ gyp --help
+          pipx run --no-cache --spec ./ gyp --version
diff --git a/gyp/.github/workflows/node-gyp.yml b/gyp/.github/workflows/node-gyp.yml
index ebe7497521..b960fcf2b7 100644
--- a/gyp/.github/workflows/node-gyp.yml
+++ b/gyp/.github/workflows/node-gyp.yml
@@ -10,9 +10,9 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: [macos-latest, ubuntu-latest, windows-latest]
-        python: ["3.8", "3.10", "3.12"]
-
+        node-version: ["22"]
+        os: [macos-13, macos-14, ubuntu-latest, windows-latest]
+        python-version: ["3.8", "3.10", "3.12", "3.13"]
     runs-on: ${{ matrix.os }}
     steps:
       - name: Clone gyp-next
@@ -24,12 +24,12 @@ jobs:
         with:
           repository: nodejs/node-gyp
           path: node-gyp
-      - uses: actions/setup-node@v3
+      - uses: actions/setup-node@v4
         with:
-          node-version: 18.x
-      - uses: actions/setup-python@v4
+          node-version: ${{ matrix.node-version }}
+      - uses: actions/setup-python@v5
         with:
-          python-version: ${{ matrix.python }}
+          python-version: ${{ matrix.python-version }}
           allow-prereleases: true
       - name: Install Python dependencies
         run: |
diff --git a/gyp/.github/workflows/release-please.yml b/gyp/.github/workflows/release-please.yml
index 665c4c48fe..95dadd53bb 100644
--- a/gyp/.github/workflows/release-please.yml
+++ b/gyp/.github/workflows/release-please.yml
@@ -7,10 +7,91 @@ name: release-please
 jobs:
   release-please:
     runs-on: ubuntu-latest
+    outputs:
+      release_created: ${{ steps.release.outputs.release_created }}
+      tag_name: ${{ steps.release.outputs.tag_name }}
+    permissions:
+      contents: write
+      pull-requests: write
     steps:
-      - uses: google-github-actions/release-please-action@v3
-        with:
-          token: ${{ secrets.GITHUB_TOKEN }}
-          release-type: python
-          package-name: gyp-next
-          bump-minor-pre-major: true
+      - uses: google-github-actions/release-please-action@v4
+        id: release
+
+  build:
+    name: Build distribution
+    needs:
+      - release-please
+    if: ${{ needs.release-please.outputs.release_created }}  # only publish on release
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: "3.12"
+    - name: Install pypa/build
+      run: >-
+        python3 -m pip install build --user
+    - name: Build a binary wheel and a source tarball
+      run: python3 -m build
+    - name: Store the distribution packages
+      uses: actions/upload-artifact@v4
+      with:
+        name: python-package-distributions
+        path: dist/
+
+  publish-to-pypi:
+    name: >-
+      Publish Python distribution to PyPI
+    needs:
+      - release-please
+      - build
+    if: ${{ needs.release-please.outputs.release_created }}  # only publish on release
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/gyp-next
+    permissions:
+      id-token: write # IMPORTANT: mandatory for trusted publishing
+    steps:
+    - name: Download all the dists
+      uses: actions/download-artifact@v4
+      with:
+        name: python-package-distributions
+        path: dist/
+    - name: Publish distribution to PyPI
+      uses: pypa/gh-action-pypi-publish@release/v1
+
+  github-release:
+    name: >-
+      Publish Python distribution to GitHub Release
+    needs:
+      - release-please
+      - build
+    if: ${{ needs.release-please.outputs.release_created }}  # only publish on release
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write # IMPORTANT: mandatory for making GitHub Releases
+      id-token: write # IMPORTANT: mandatory for sigstore
+    steps:
+    - name: Download all the dists
+      uses: actions/download-artifact@v4
+      with:
+        name: python-package-distributions
+        path: dist/
+    - name: Sign the dists with Sigstore
+      uses: sigstore/gh-action-sigstore-python@v2.1.1
+      with:
+        inputs: >-
+          ./dist/*.tar.gz
+          ./dist/*.whl
+    - name: Upload artifact signatures to GitHub Release
+      env:
+        GITHUB_TOKEN: ${{ github.token }}
+      # Upload to GitHub Release using the `gh` CLI.
+      # `dist/` contains the built packages, and the
+      # sigstore-produced signatures and certificates.
+      run: >-
+        gh release upload
+        ${{ needs.release-please.outputs.tag_name }} dist/**
+        --repo '${{ github.repository }}'
diff --git a/gyp/.release-please-manifest.json b/gyp/.release-please-manifest.json
new file mode 100644
index 0000000000..cbd0ca0683
--- /dev/null
+++ b/gyp/.release-please-manifest.json
@@ -0,0 +1,3 @@
+{
+    ".": "0.18.1"
+}
diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md
index 483943e013..8a8282a5f8 100644
--- a/gyp/CHANGELOG.md
+++ b/gyp/CHANGELOG.md
@@ -1,5 +1,46 @@
 # Changelog
 
+## [0.18.1](https://github.com/nodejs/gyp-next/compare/v0.18.0...v0.18.1) (2024-05-26)
+
+
+### Bug Fixes
+
+* **ci:** add Python 3.13 pre-release to test matrix ([#257](https://github.com/nodejs/gyp-next/issues/257)) ([8597203](https://github.com/nodejs/gyp-next/commit/8597203b687325c7516367135e026586279d0583))
+
+
+### Documentation
+
+* vendor docs from gyp.gsrc.io ([#254](https://github.com/nodejs/gyp-next/issues/254)) ([8d7ba6e](https://github.com/nodejs/gyp-next/commit/8d7ba6e784dedf1122a0456150c739d2a09ecf57))
+
+## [0.18.0](https://github.com/nodejs/gyp-next/compare/v0.17.0...v0.18.0) (2024-05-08)
+
+
+### Features
+
+* support language standard keys in msvs_settings ([#252](https://github.com/nodejs/gyp-next/issues/252)) ([322f6d5](https://github.com/nodejs/gyp-next/commit/322f6d5d5233967522f3e55c623a8e7d7281e024))
+
+## [0.17.0](https://github.com/nodejs/gyp-next/compare/v0.16.2...v0.17.0) (2024-04-29)
+
+
+### Features
+
+* generate compile_commands.json with ninja ([#228](https://github.com/nodejs/gyp-next/issues/228)) ([7b20b46](https://github.com/nodejs/gyp-next/commit/7b20b4673d8cf46ff61898eb19569007d55c854a))
+
+
+### Bug Fixes
+
+* failed to detect flavor if compiler path include white spaces ([#240](https://github.com/nodejs/gyp-next/issues/240)) ([f3b9753](https://github.com/nodejs/gyp-next/commit/f3b9753e7526377020e7d40e66b624db771cf84a))
+* support cross compiling for wasm with make generator ([#222](https://github.com/nodejs/gyp-next/issues/222)) ([de0e1c9](https://github.com/nodejs/gyp-next/commit/de0e1c9a5791d1bf4bc3103f878ab74814864ab4))
+* support empty dictionary keys in input ([#245](https://github.com/nodejs/gyp-next/issues/245)) ([178459f](https://github.com/nodejs/gyp-next/commit/178459ff343a2771d5f30f04467d2f032d6b3565))
+* update Ruff to 0.3.1 ([876ccaf](https://github.com/nodejs/gyp-next/commit/876ccaf5629e1b95e13aaa2b0eb6cbd08fa80593))
+
+## [0.16.2](https://github.com/nodejs/gyp-next/compare/v0.16.1...v0.16.2) (2024-03-07)
+
+
+### Bug Fixes
+
+* avoid quoting cflag name and parameter with space separator ([#223](https://github.com/nodejs/gyp-next/issues/223)) ([2b9703d](https://github.com/nodejs/gyp-next/commit/2b9703dbd5b3b8a935faf257c6103033b47bf8bf))
+
 ## [0.16.1](https://github.com/nodejs/gyp-next/compare/v0.16.0...v0.16.1) (2023-10-25)
 
 
diff --git a/gyp/CONTRIBUTING.md b/gyp/CONTRIBUTING.md
index 1a0bcde2b4..5237b216fb 100644
--- a/gyp/CONTRIBUTING.md
+++ b/gyp/CONTRIBUTING.md
@@ -1,5 +1,9 @@
 # Contributing to gyp-next
 
+## Start contributing
+
+Read the docs at [`./docs/Hacking.md`](./docs/Hacking.md) to get started.
+
 ## Code of Conduct
 
 This project is bound to the [Node.js Code of Conduct](https://github.com/nodejs/admin/blob/HEAD/CODE_OF_CONDUCT.md).
diff --git a/gyp/README.md b/gyp/README.md
index be1d7b9ebf..38792e1de4 100644
--- a/gyp/README.md
+++ b/gyp/README.md
@@ -1,7 +1,7 @@
 GYP can Generate Your Projects.
 ===================================
 
-Documents are available at [gyp.gsrc.io](https://gyp.gsrc.io), or you can check out ```md-pages``` branch to read those documents offline.
+Documents are available at [`./docs`](./docs).
 
 __gyp-next__ is [released](https://github.com/nodejs/gyp-next/releases) to the [__Python Packaging Index__](https://pypi.org/project/gyp-next) (PyPI) and can be installed with the command:
 * `python3 -m pip install gyp-next`
diff --git a/gyp/data/ninja/build.ninja b/gyp/data/ninja/build.ninja
new file mode 100644
index 0000000000..2400dbb1f0
--- /dev/null
+++ b/gyp/data/ninja/build.ninja
@@ -0,0 +1,4 @@
+rule cc
+  command = cc $in $out
+
+build my.out: cc my.in
diff --git a/gyp/docs/GypVsCMake.md b/gyp/docs/GypVsCMake.md
new file mode 100644
index 0000000000..6d659a6123
--- /dev/null
+++ b/gyp/docs/GypVsCMake.md
@@ -0,0 +1,116 @@
+# vs. CMake
+
+GYP was originally created to generate native IDE project files (Visual Studio, Xcode) for building [Chromium](http://www.chromim.org).
+
+The functionality of GYP is very similar to the [CMake](http://www.cmake.org)
+build tool.  Bradley Nelson wrote up the following description of why the team
+created GYP instead of using CMake.  The text below is copied from
+http://www.mail-archive.com/webkit-dev@lists.webkit.org/msg11029.html
+
+```
+
+Re: [webkit-dev] CMake as a build system?
+Bradley Nelson
+Mon, 19 Apr 2010 22:38:30 -0700
+
+Here's the innards of an email with a laundry list of stuff I came up with a
+while back on the gyp-developers list in response to Mike Craddick regarding
+what motivated gyp's development, since we were aware of cmake at the time
+(we'd even started a speculative port):
+
+
+I did an exploratory port of portions of Chromium to cmake (I think I got as
+far as net, base, sandbox, and part of webkit).
+There were a number of motivations, not all of which would apply to other
+projects. Also, some of the design of gyp was informed by experience at
+Google with large projects built wholly from source, leading to features
+absent from cmake, but not strictly required for Chromium.
+
+1. Ability to incrementally transition on Windows. It took us about 6 months
+to switch fully to gyp. Previous attempts to move to scons had taken a long
+time and failed, due to the requirement to transition while in flight. For a
+substantial period of time, we had a hybrid of checked in vcproj and gyp generated
+vcproj. To this day we still have a good number of GUIDs pinned in the gyp files,
+because different parts of our release pipeline have leftover assumptions
+regarding manipulating the raw sln/vcprojs. This transition occurred from
+the bottom up, largely because modules like base were easier to convert, and
+had a lower churn rate. During early stages of the transition, the majority
+of the team wasn't even aware they were using gyp, as it integrated into
+their existing workflow, and only affected modules that had been converted.
+
+2. Generation of a more 'normal' vcproj file. Gyp attempts, particularly on
+Windows, to generate vcprojs which resemble hand generated projects. It
+doesn't generate any Makefile type projects, but instead produces msvs
+Custom Build Steps and Custom Build Rules. This makes the resulting projects
+easier to understand from the IDE and avoids parts of the IDE that simply
+don't function correctly if you use Makefile projects. Our early hope with
+gyp was to support the least common denominator of features present in each
+of the platform specific project file formats, rather than falling back on
+generated Makefiles/shell scripts to emulate some common abstraction. CMake by
+comparison makes a good faith attempt to use native project features, but
+falls back on generated scripts in order to preserve the same semantics on
+each platforms.
+
+3. Abstraction on the level of project settings, rather than command line
+flags. In gyp's syntax you can add nearly any option present in a hand
+generated xcode/vcproj file. This allows you to use abstractions built into
+the IDEs rather than reverse engineering them possibly incorrectly for
+things like: manifest generation, precompiled headers, bundle generation.
+When somebody wants to use a particular menu option from msvs, I'm able to
+do a web search on the name of the setting from the IDE and provide them
+with a gyp stanza that does the equivalent. In many cases, not all project
+file constructs correspond to command line flags.
+
+4. Strong notion of module public/private interface. Gyp allows targets to
+publish a set of direct_dependent_settings, specifying things like
+include_dirs, defines, platforms specific settings, etc. This means that
+when module A depends on module B, it automatically acquires the right build
+settings without module A being filled with assumptions/knowledge of exactly
+how module B is built. Additionally, all of the transitive dependencies of
+module B are pulled in. This avoids their being a single top level view of
+the project, rather each gyp file expresses knowledge about its immediate
+neighbors. This keep local knowledge local. CMake effectively has a large
+shared global namespace.
+
+5. Cross platform generation. CMake is not able to generate all project
+files on all platforms. For example xcode projects cannot be generated from
+windows (cmake uses mac specific libraries to do project generation). This
+means that for instance generating a tarball containing pregenerated
+projects for all platforms is hard with Cmake (requires distribution to
+several machine types).
+
+6. Gyp has rudimentary cross compile support. Currently we've added enough
+functionality to gyp to support x86 -> arm cross compiles. Last I checked
+this functionality wasn't present in cmake. (This occurred later).
+
+
+That being said there are a number of drawbacks currently to gyp:
+
+1. Because platform specific settings are expressed at the project file
+level (rather than the command line level). Settings which might otherwise
+be shared in common between platforms (flags to gcc on mac/linux), end up
+being repeated twice. Though in fairness there is actually less sharing here
+than you'd think. include_dirs and defines actually represent 90% of what
+can be typically shared.
+
+2. CMake may be more mature, having been applied to a broader range of
+projects. There a number of 'tool modules' for cmake, which are shared in a
+common community.
+
+3. gyp currently makes some nasty assumptions about the availability of
+chromium's hermetic copy of cygwin on windows. This causes you to either
+have to special case a number of rules, or swallow this copy of cygwin as a
+build time dependency.
+
+4. CMake includes a fairly readable imperative language. Currently Gyp has a
+somewhat poorly specified declarative language (variable expansion happens
+in sometimes weird and counter-intuitive ways). In fairness though, gyp assumes
+that external python scripts can be used as an escape hatch. Also gyp avoids
+a lot of the things you'd need imperative code for, by having a nice target
+settings publication mechanism.
+
+5. (Feature/drawback depending on personal preference). Gyp's syntax is
+DEEPLY nested. It suffers from all of Lisp's advantages and drawbacks.
+
+-BradN
+```
diff --git a/gyp/docs/Hacking.md b/gyp/docs/Hacking.md
new file mode 100644
index 0000000000..89b3b8bea9
--- /dev/null
+++ b/gyp/docs/Hacking.md
@@ -0,0 +1,46 @@
+# Hacking
+
+## Getting the sources
+
+Git is required to hack on anything, you can set up a git clone of GYP
+as follows:
+
+```
+mkdir foo
+cd foo
+git clone git@github.com:nodejs/gyp-next.git
+cd gyp
+```
+
+(this will clone gyp underneath it into `foo/gyp`.
+`foo` can be any directory name you want. Once you've done that,
+you can use the repo like anything other Git repo.
+
+## Testing your change
+
+GYP has a suite of tests which you can run with the provided test driver
+to make sure your changes aren't breaking anything important.
+
+You run the test driver with e.g.
+
+``` sh
+$ python -m pip install --upgrade pip setuptools
+$ pip install --editable ".[dev]"
+$ python -m pytest
+```
+
+See [Testing](Testing.md) for more details on the test framework.
+
+Note that it can be handy to look at the project files output by the tests
+to diagnose problems. The easiest way to do that is by kindly asking the
+test driver to leave the temporary directories it creates in-place.
+This is done by setting the enviroment variable "PRESERVE", e.g.
+
+```
+set PRESERVE=all     # On Windows
+export PRESERVE=all  # On saner platforms.
+```
+
+## Reviewing your change
+
+All changes to GYP must be code reviewed before submission.
diff --git a/gyp/docs/InputFormatReference.md b/gyp/docs/InputFormatReference.md
new file mode 100644
index 0000000000..2b2c180f44
--- /dev/null
+++ b/gyp/docs/InputFormatReference.md
@@ -0,0 +1,1080 @@
+# Input Format Reference
+
+## Primitive Types
+
+The following primitive types are found within input files:
+
+  * String values, which may be represented by enclosing them in
+    `'single quotes'` or `"double quotes"`.  By convention, single
+    quotes are used.
+  * Integer values, which are represented in decimal without any special
+    decoration.  Integers are fairly rare in input files, but have a few
+    applications in boolean contexts, where the convention is to
+    represent true values with `1` and false with `0`.
+  * Lists, which are represented as a sequence of items separated by
+    commas (`,`) within square brackets (`[` and `]`).  A list may
+    contain any other primitive types, including other lists.
+    Generally, each item of a list must be of the same type as all other
+    items in the list, but in some cases (such as within `conditions`
+    sections), the list structure is more tightly specified.  A trailing
+    comma is permitted.
+
+    This example list contains three string values.
+
+      ```
+      [ 'Generate', 'Your', 'Projects', ]
+      ```
+
+  * Dictionaries, which map keys to values.  All keys are strings.
+    Values may be of any other primitive type, including other
+    dictionaries.  A dictionary is enclosed within curly braces (`{` and
+    `}`).  Keys precede values, separated by a colon (`:`).  Successive
+    dictionary entries are separated by commas (`,`).  A trailing comma
+    is permitted.  It is an error for keys to be duplicated within a
+    single dictionary as written in an input file, although keys may
+    replace other keys during [merging](#Merging).
+
+    This example dictionary maps each of three keys to different values.
+
+      ```
+      {
+        'inputs': ['version.c.in'],
+        'outputs': ['version.c'],
+        'process_outputs_as_sources': 1,
+      }
+      ```
+
+## Overall Structure
+
+A GYP input file is organized as structured data.  At the root scope of
+each `.gyp` or `.gypi` (include) file is a dictionary.  The keys and
+values of this dictionary, along with any descendants contained within
+the values, provide the data contained within the file.  This data is
+given meaning by interpreting specific key names and their associated
+values in specific ways (see [Settings Keys](#Settings_Keys)).
+
+### Comments (#)
+
+Within an input file, a comment is introduced by a pound sign (`#`) not
+within a string.  Any text following the pound sign, up until the end of
+the line, is treated as a comment.
+
+#### Example
+
+```
+{
+  'school_supplies': [
+    'Marble composition book',
+    'Sharp #2 pencil',
+    'Safety scissors',  # You still shouldn't run with these
+  ],
+}
+```
+
+In this example, the # in `'Sharp #2 pencil'` is not taken as
+introducing a comment because it occurs within a string, but the text
+after `'Safety scissors'` is treated as a comment having no impact on
+the data within the file.
+
+## Merging
+
+### Merge Basics (=, ?, +)
+
+Many operations on GYP input files occurs by merging dictionary and list
+items together.  During merge operations, it is important to recognize
+the distinction between source and destination values.  Items from the
+source value are merged into the destination, which leaves the source
+unchanged and the destination modified by the source.  A dictionary may
+only be merged into another dictionary, and a list may only be merged
+into another list.
+
+  * When merging a dictionary, for each key in the source:
+    * If the key does not exist in the destination dictionary, insert it
+      and copy the associated value directly.
+    * If the key does exist:
+      * If the associated value is a dictionary, perform the dictionary
+        merging procedure using the source's and destination's value
+        dictionaries.
+      * If the associated value is a list, perform the list merging
+        procedure using the source's and destination's value lists.
+      * If the associated value is a string or integer, the destination
+        value is replaced by the source value.
+  * When merging a list, merge according to the suffix appended to the
+    key name, if the list is a value within a dictionary.
+    * If the key ends with an equals sign (`=`), the policy is for the
+      source list to completely replace the destination list if it
+      exists.  _Mnemonic: `=` for assignment._
+    * If the key ends with a question mark (`?`), the policy is for the
+      source list to be set as the destination list only if the key is
+      not already present in the destination.  _Mnemonic: `?` for
+      conditional assignment_.
+    * If the key ends with a plus sign (`+`), the policy is for the
+      source list contents to be prepended to the destination list.
+      _Mnemonic: `+` for addition or concatenation._
+    * If the list key is undecorated, the policy is for the source list
+      contents to be appended to the destination list.  This is the
+      default list merge policy.
+
+#### Example
+
+Source dictionary:
+
+```
+{
+  'include_dirs+': [
+    'shared_stuff/public',
+  ],
+  'link_settings': {
+    'libraries': [
+      '-lshared_stuff',
+    ],
+  },
+  'test': 1,
+}
+```
+
+Destination dictionary:
+
+```
+{
+  'target_name': 'hello',
+  'sources': [
+    'kitty.cc',
+  ],
+  'include_dirs': [
+    'headers',
+  ],
+  'link_settings': {
+    'libraries': [
+      '-lm',
+    ],
+    'library_dirs': [
+      '/usr/lib',
+    ],
+  },
+  'test': 0,
+}
+```
+
+Merged dictionary:
+
+```
+{
+  'target_name': 'hello',
+  'sources': [
+    'kitty.cc',
+  ],
+  'include_dirs': [
+    'shared_stuff/public',  # Merged, list item prepended due to include_dirs+
+    'headers',
+  ],
+  'link_settings': {
+    'libraries': [
+      '-lm',
+      '-lshared_stuff',  # Merged, list item appended
+    ],
+    'library_dirs': [
+      '/usr/lib',
+    ],
+  },
+  'test': 1,  # Merged, int value replaced
+}
+```
+
+## Pathname Relativization
+
+In a `.gyp` or `.gypi` file, many string values are treated as pathnames
+relative to the file in which they are defined.
+
+String values associated with the following keys, or contained within
+lists associated with the following keys, are treated as pathnames:
+
+  * destination
+  * files
+  * include\_dirs
+  * inputs
+  * libraries
+  * outputs
+  * sources
+  * mac\_bundle\_resources
+  * mac\_framework\_dirs
+  * msvs\_cygwin\_dirs
+  * msvs\_props
+
+Additionally, string values associated with keys ending in the following
+suffixes, or contained within lists associated with keys ending in the
+following suffixes, are treated as pathnames:
+
+  * `_dir`
+  * `_dirs`
+  * `_file`
+  * `_files`
+  * `_path`
+  * `_paths`
+
+However, any string value beginning with any of these characters is
+excluded from pathname relativization:
+
+  * `/` for identifying absolute paths.
+  * `$` for introducing build system variable expansions.
+  * `-` to support specifying such items as `-llib`, meaning “library
+    `lib` in the library search path.”
+  * `<`, `>`, and `!` for GYP expansions.
+
+When merging such relative pathnames, they are adjusted so that they can
+remain valid relative pathnames, despite being relative to a new home.
+
+#### Example
+
+Source dictionary from `../build/common.gypi`:
+
+```
+{
+  'include_dirs': ['include'],  # Treated as relative to ../build
+  'libraries': ['-lz'],  # Not treated as a pathname, begins with a dash
+  'defines': ['NDEBUG'],  # defines does not contain pathnames
+}
+```
+
+Target dictionary, from `base.gyp`:
+
+```
+{
+  'sources': ['string_util.cc'],
+}
+```
+
+Merged dictionary:
+
+```
+{
+  'sources': ['string_util.cc'],
+  'include_dirs': ['../build/include'],
+  'libraries': ['-lz'],
+  'defines': ['NDEBUG'],
+}
+```
+
+Because of pathname relativization, after the merge is complete, all of
+the pathnames in the merged dictionary are valid relative to the
+directory containing `base.gyp`.
+
+## List Singletons
+
+Some list items are treated as singletons, and the list merge process
+will enforce special rules when merging them.  At present, any string
+item in a list that does not begin with a dash (`-`) is treated as a
+singleton, although **this is subject to change.**  When appending or
+prepending a singleton to a list, if the item is already in the list,
+only the earlier instance is retained in the merged list.
+
+#### Example
+
+Source dictionary:
+
+```
+{
+  'defines': [
+    'EXPERIMENT=1',
+    'NDEBUG',
+  ],
+}
+```
+
+Destination dictionary:
+
+```
+{
+  'defines': [
+    'NDEBUG',
+    'USE_THREADS',
+  ],
+}
+```
+
+Merged dictionary:
+
+```
+{
+  'defines': [
+    'NDEBUG',
+    'USE_THREADS',
+    'EXPERIMENT=1',  # Note that NDEBUG is not appended after this.
+  ],
+}
+```
+
+## Including Other Files
+
+If the `-I` (`--include`) argument was used to invoke GYP, any files
+specified will be implicitly merged into the root dictionary of all
+`.gyp` files.
+
+An [includes](#includes) section may be placed anywhere within a
+`.gyp` or `.gypi` (include) file.  `includes` sections contain lists of
+other files to include.  They are processed sequentially and merged into
+the enclosing dictionary at the point that the `includes` section was
+found.  `includes` sections at the root of a `.gyp` file dictionary are
+merged after any `-I` includes from the command line.
+
+[includes](#includes) sections are processed immediately after a file is
+loaded, even before [variable and conditional
+processing](#Variables_and_Conditionals), so it is not possible to
+include a file based on a [variable reference](#Variable_Expansions).
+While it would be useful to be able to include files based on variable
+expansions, it is most likely more useful to allow included files access
+to variables set by the files that included them.
+
+An [includes](#includes) section may, however, be placed within a
+[conditional](#Conditionals) section.  The included file itself will
+be loaded unconditionally, but its dictionary will be discarded if the
+associated condition is not true.
+
+## Variables and Conditionals
+
+### Variables
+
+There are three main types of variables within GYP.
+
+  * Predefined variables.  By convention, these are named with
+    `CAPITAL_LETTERS`.  Predefined variables are set automatically by
+    GYP.  They may be overridden, but it is not advisable to do so.  See
+    [Predefined Variables](#Predefined_Variables) for a list of
+    variables that GYP provides.
+  * User-defined variables.  Within any dictionary, a key named
+    `variables` can be provided, containing a mapping between variable
+    names (keys) and their contents (values), which may be strings,
+    integers, or lists of strings.  By convention, user-defined
+    variables are named with `lowercase_letters`.
+  * Automatic variables.  Within any dictionary, any key with a string
+    value has a corresponding automatic variable whose name is the same
+    as the key name with an underscore (`_`) prefixed.  For example, if
+    your dictionary contains `type: 'static_library'`, an automatic
+    variable named `_type` will be provided, and its value will be a
+    string, `'static_library'`.
+
+Variables are inherited from enclosing scopes.
+
+### Providing Default Values for Variables (%)
+
+Within a `variables` section, keys named with percent sign (`%`)
+suffixes mean that the variable should be set only if it is undefined at
+the time it is processed.  This can be used to provide defaults for
+variables that would otherwise be undefined, so that they may reliably
+be used in [variable expansion or conditional
+processing](#Variables_and_Conditionals).
+
+### Predefined Variables
+
+Each GYP generator module provides defaults for the following variables:
+
+  * `OS`: The name of the operating system that the generator produces
+    output for.  Common values for values for `OS` are:
+
+    * `'linux'`
+    * `'mac'`
+    * `'win'`
+
+    But other values may be encountered and this list should not be
+    considered exhaustive.  The `gypd` (debug) generator module does not
+    provide a predefined value for `OS`.  When invoking GYP with the
+    `gypd` module, if a value for `OS` is needed, it must be provided on
+    the command line, such as `gyp -f gypd -DOS=mac`.
+
+    GYP generators also provide defaults for these variables.  They may
+    be expressed in terms of variables used by the build system that
+    they generate for, often in `$(VARIABLE)` format.  For example, the
+    GYP `PRODUCT_DIR` variable maps to the Xcode `BUILT_PRODUCTS_DIR`
+    variable, so `PRODUCT_DIR` is defined by the Xcode generator as
+    `$(BUILT_PRODUCTS_DIR)`.
+  * `EXECUTABLE_PREFIX`: A prefix, if any, applied to executable names.
+    Usually this will be an empty string.
+  * `EXECUTABLE_SUFFIX`: A suffix, if any, applied to executable names.
+    On Windows, this will be `.exe`, elsewhere, it will usually be an
+    empty string.
+  * `INTERMEDIATE_DIR`: A directory that can be used to place
+    intermediate build results in.  `INTERMEDIATE_DIR` is only
+    guaranteed to be accessible within a single target (See targets).
+    This variable is most useful within the context of rules and actions
+    (See rules, See actions).  Compare with `SHARED_INTERMEDIATE_DIR`.
+  * `PRODUCT_DIR`: The directory in which the primary output of each
+    target, such as executables and libraries, is placed.
+  * `RULE_INPUT_ROOT`: The base name for the input file (e.g. "`foo`").
+    See Rules.
+  * `RULE_INPUT_EXT`: The file extension for the input file (e.g.
+    "`.cc`").  See Rules.
+  * `RULE_INPUT_NAME`: Full name of the input file (e.g. "`foo.cc`").
+    See Rules.
+  * `RULE_INPUT_PATH`: Full path to the input file (e.g.
+    "`/bar/foo.cc`").  See Rules.
+  * `SHARED_INTERMEDIATE_DIR`: A directory that can be used to place
+    intermediate build results in, and have them be accessible to other
+    targets.  Unlike `INTERMEDIATE_DIR`, each target in a project,
+    possibly spanning multiple `.gyp` files, shares the same
+    `SHARED_INTERMEDIATE_DIR`.
+
+The following additional predefined variables may be available under
+certain circumstances:
+
+  * `DEPTH`.  When GYP is invoked with a `--depth` argument, when
+    processing any `.gyp` file, `DEPTH` will be a relative path from the
+    `.gyp` file to the directory specified by the `--depth` argument.
+
+### User-Defined Variables
+
+A user-defined variable may be defined in terms of other variables, but
+not other variables that have definitions provided in the same scope.
+
+### Variable Expansions (<, >, <@, >@)
+
+GYP provides two forms of variable expansions, “early” or “pre”
+expansions, and “late,” “post,” or “target” expansions.  They have
+similar syntax, differing only in the character used to introduce them.
+
+  * Early expansions are introduced by a less-than (`<`) character.
+    _Mnemonic: the arrow points to the left, earlier on a timeline._
+  * Late expansions are introduced by a less-than (`>`) character.
+    _Mnemonic: the arrow points to the right, later on a timeline._
+
+The difference the two phases of expansion is described in [Early and
+Late Phases](#Early_and_Late_Phases).
+
+These characters were chosen based upon the requirement that they not
+conflict with the variable format used natively by build systems.  While
+the dollar sign (`$`) is the most natural fit for variable expansions,
+its use was ruled out because most build systems already use that
+character for their own variable expansions.  Using different characters
+means that no escaping mechanism was needed to differentiate between GYP
+variables and build system variables, and writing build system variables
+into GYP files is not cumbersome.
+
+Variables may contain lists or strings, and variable expansions may
+occur in list or string context.  There are variant forms of variable
+expansions that may be used to determine how each type of variable is to
+be expanded in each context.
+
+  * When a variable is referenced by `<(VAR)` or `>(VAR)`:
+    * If `VAR` is a string, the variable reference within the string is
+      replaced by variable's string value.
+    * If `VAR` is a list, the variable reference within the string is
+      replaced by a string containing the concatenation of all of the
+      variable’s list items.  Generally, the items are joined with
+      spaces between each, but the specific behavior is
+      generator-specific.  The precise encoding used by any generator
+      should be one that would allow each list item to be treated as a
+      separate argument when used as program arguments on the system
+      that the generator produces output for.
+  * When a variable is referenced by `<@(VAR)` or `>@(VAR)`:
+    * The expansion must occur in list context.
+    * The list item must be `'<@(VAR)'` or `'>@(VAR)'` exactly.
+    * If `VAR` is a list, each of its elements are inserted into the
+      list in which expansion is taking place, replacing the list item
+      containing the variable reference.
+    * If `VAR` is a string, the string is converted to a list which is
+      inserted into the list in which expansion is taking place as
+      above.  The conversion into a list is generator-specific, but
+      generally, spaces in the string are taken as separators between
+      list items.  The specific method of converting the string to a
+      list should be the inverse of the encoding method used to expand
+      list variables in string context, above.
+
+GYP treats references to undefined variables as errors.
+
+### Command Expansions (` form
+    of [variable expansions](#Variable_Expansions),
+    and on the `!` form of [command
+    expansions](#Command_Expansions_(!,_!@)).
+
+These two phases are provided because there are some circumstances in
+which each is desirable.
+
+The “early” phase is appropriate for most expansions and evaluations.
+“Early” expansions and evaluations may be performed anywhere within any
+`.gyp` or `.gypi` file.
+
+The “late” phase is appropriate when expansion or evaluation must be
+deferred until a specific section has been merged into target context.
+“Late” expansions and evaluations only occur within `targets` sections
+and their descendants.  The typical use case for a late-phase expansion
+is to provide, in some globally-included `.gypi` file, distinct
+behaviors depending on the specifics of a target.
+
+#### Example
+
+Given this input:
+
+```
+{
+  'target_defaults': {
+    'target_conditions': [
+      ['_type=="shared_library"', {'cflags': ['-fPIC']}],
+    ],
+  },
+  'targets': [
+    {
+      'target_name': 'sharing_is_caring',
+      'type': 'shared_library',
+    },
+    {
+      'target_name': 'static_in_the_attic',
+      'type': 'static_library',
+    },
+  ]
+}
+```
+
+The conditional needs to be evaluated only in target context; it is
+nonsense outside of target context because no `_type` variable is
+defined.  [target\_conditions](#target_conditions) allows evaluation
+to be deferred until after the [targets](#targets) sections are
+merged into their copies of [target\_defaults](#target_defaults).
+The resulting targets, after “late” phase processing:
+
+```
+{
+  'targets': [
+    {
+      'target_name': 'sharing_is_caring',
+      'type': 'shared_library',
+      'cflags': ['-fPIC'],
+    },
+    {
+      'target_name': 'static_in_the_attic',
+      'type': 'static_library',
+    },
+  ]
+}
+```
+
+### Expansion and Evaluation Performed Simultaneously
+
+During any expansion and evaluation phase, both expansion and evaluation
+are performed simultaneously.  The process for handling variable
+expansions and conditional evaluation within a dictionary is:
+
+  * Load [automatic variables](#Variables) (those with leading
+    underscores).
+  * If a [variables](#variables) section is present, recurse into its
+    dictionary.  This allows [conditionals](#Conditionals) to be
+    present within the `variables` dictionary.
+  * Load [Variables user-defined variables](#User-Defined) from the
+    [variables](#variables) section.
+  * For each string value in the dictionary, perform [variable
+    expansion](#Variable_Expansions) and, if operating
+    during the “late” phase, [command
+    expansions](#Command_Expansions).
+  * Reload [automatic variables](#Variables) and [Variables
+    user-defined variables](#User-Defined) because the variable
+    expansion step may have resulted in changes to the automatic
+    variables.
+  * If a [conditions](#conditions) or
+    [target\_conditions](#target_conditions) section (depending on
+    phase) is present, recurse into its dictionary.  This is done after
+    variable expansion so that conditionals may take advantage of
+    expanded automatic variables.
+  * Evaluate [conditionals](#Conditionals).
+  * Reload [automatic variables](#Variables) and [Variables
+    user-defined variables](#User-Defined) because the conditional
+    evaluation step may have resulted in changes to the automatic
+    variables.
+  * Recurse into child dictionaries or lists that have not yet been
+    processed.
+
+One quirk of this ordering is that you cannot expect a
+[variables](#variables) section within a dictionary’s
+[conditional](#Conditionals) to be effective in the dictionary
+itself, but the added variables will be effective in any child
+dictionaries or lists.  It is thought to be far more worthwhile to
+provide resolved [automatic variables](#Variables) to
+[conditional](#Conditionals) sections, though.  As a workaround, to
+conditionalize variable values, place a [conditions](#conditions) or
+[target\_conditions](#target_conditions) section within the
+[variables](#variables) section.
+
+## Dependencies and Dependents
+
+In GYP, “dependents” are targets that rely on other targets, called
+“dependencies.”  Dependents declare their reliance with a special
+section within their target dictionary,
+[dependencies](#dependencies).
+
+### Dependent Settings
+
+It is useful for targets to “advertise” settings to their dependents.
+For example, a target might require that all of its dependents add
+certain directories to their include paths, link against special
+libraries, or define certain preprocessor macros.  GYP allows these
+cases to be handled gracefully with “dependent settings” sections.
+There are three types of such sections:
+
+  * [direct\_dependent\_settings](#direct_dependent_settings), which
+    advertises settings to a target's direct dependents only.
+  * [all\_dependent\_settings](#all_dependnet_settings), which
+    advertises settings to all of a target's dependents, both direct and
+    indirect.
+  * [link\_settings](#link_settings), which contains settings that
+    should be applied when a target’s object files are used as linker
+    input.
+
+Furthermore, in some cases, a target needs to pass its dependencies’
+settings on to its own dependents.  This might happen when a target’s
+own public header files include header files provided by its dependency.
+[export\_dependent\_settings](#export_dependent_settings) allows a
+target to declare dependencies for which
+[direct\_dependent\_settings](#direct_dependent_settings) should be
+passed through to its own dependents.
+
+Dependent settings processing merges a copy of the relevant dependent
+settings dictionary from a dependency into its relevant dependent
+targets.
+
+In most instances,
+[direct\_dependent\_settings](#direct_dependent_settings) will be
+used.  There are very few cases where
+[all\_dependent\_settings](#all_dependent_settings) is actually
+correct; in most of the cases where it is tempting to use, it would be
+preferable to declare
+[export\_dependent\_settings](#export_dependent_settings).  Most
+[libraries](#libraries) and [library\_dirs](#library_dirs)
+sections should be placed within [link\_settings](#link_settings)
+sections.
+
+#### Example
+
+Given:
+
+```
+{
+  'targets': [
+    {
+      'target_name': 'cruncher',
+      'type': 'static_library',
+      'sources': ['cruncher.cc'],
+      'direct_dependent_settings': {
+        'include_dirs': ['.'],  # dependents need to find cruncher.h.
+      },
+      'link_settings': {
+        'libraries': ['-lm'],  # cruncher.cc does math.
+      },
+    },
+    {
+      'target_name': 'cruncher_test',
+      'type': 'executable',
+      'dependencies': ['cruncher'],
+      'sources': ['cruncher_test.cc'],
+    },
+  ],
+}
+```
+
+After dependent settings processing, the dictionary for `cruncher_test`
+will be:
+
+```
+{
+  'target_name': 'cruncher_test',
+  'type': 'executable',
+  'dependencies': ['cruncher'],  # implies linking against cruncher
+  'sources': ['cruncher_test.cc'],
+  'include_dirs': ['.']
+  'libraries': ['-lm'],
+},
+```
+
+If `cruncher` was declared as a `shared_library` instead of a
+`static_library`, the `cruncher_test` target would not contain `-lm`,
+but instead, `cruncher` itself would link against `-lm`.
+
+## Linking Dependencies
+
+The precise meaning of a dependency relationship varies with the
+[types](#type) of the [targets](#targets) at either end of the
+relationship.  In GYP, a dependency relationship can indicate two things
+about how targets relate to each other:
+
+  * Whether the dependent target needs to link against the dependency.
+  * Whether the dependency target needs to be built prior to the
+    dependent.  If the former case is true, this case must be true as
+    well.
+
+The analysis of the first item is complicated by the differences between
+static and shared libraries.
+
+  * Static libraries are simply collections of object files (`.o` or
+    `.obj`) that are used as inputs to a linker (`ld` or `link.exe`).
+    Static libraries don't link against other libraries, they’re
+    collected together and used when eventually linking a shared library
+    or executable.
+  * Shared libraries are linker output and must undergo symbol
+    resolution.  They must link against other libraries (static or
+    shared) in order to facilitate symbol resolution.  They may be used
+    as libraries in subsequent link steps.
+  * Executables are also linker output, and also undergo symbol
+    resolution.  Like shared libraries, they must link against static
+    and shared libraries to facilitate symbol resolution.  They may not
+    be reused as linker inputs in subsequent link steps.
+
+Accordingly, GYP performs an operation referred to as “static library
+dependency adjustment,” in which it makes each linker output target
+(shared libraries and executables) link against the static libraries it
+depends on, either directly or indirectly.  Because the linkable targets
+link against these static libraries, they are also made direct
+dependents of the static libraries.
+
+As part of this process, GYP is also able to remove the direct
+dependency relationships between two static library targets, as a
+dependent static library does not actually need to link against a
+dependency static library.  This removal facilitates speedier builds
+under some build systems, as they are now free to build the two targets
+in parallel.  The removal of this dependency is incorrect in some cases,
+such as when the dependency target contains [rules](#rules) or
+[actions](#actions) that generate header files required by the
+dependent target.  In such cases, the dependency target, the one
+providing the side-effect files, must declare itself as a
+[hard\_dependency](#hard_dependency).  This setting instructs GYP to
+not remove the dependency link between two static library targets in its
+generated output.
+
+## Loading Files to Resolve Dependencies
+
+When GYP runs, it loads all `.gyp` files needed to resolve dependencies
+found in [dependencies](#dependencies) sections.  These files are not
+merged into the files that reference them, but they may contain special
+sections that are merged into dependent target dictionaries.
+
+## Build Configurations
+
+Explain this.
+
+## List Filters
+
+GYP allows list items to be filtered by “exclusions” and “patterns.”
+Any list containing string values in a dictionary may have this
+filtering applied.  For the purposes of this section, a list modified by
+exclusions or patterns is referred to as a “base list”, in contrast to
+the “exclusion list” and “pattern list” that operates on it.
+
+  * For a base list identified by key name `key`, the `key!` list
+    provides exclusions.
+  * For a base list identified by key name `key`, the `key/` list
+    provides regular expression pattern-based filtering.
+
+Both `key!` and `key/` may be present.  The `key!` exclusion list will
+be processed first, followed by the `key/` pattern list.
+
+Exclusion lists are most powerful when used in conjunction with
+[conditionals](#Conditionals).
+
+## Exclusion Lists (!)
+
+An exclusion list provides a way to remove items from the related list
+based on exact matching.  Any item found in an exclusion list will be
+removed from the corresponding base list.
+
+#### Example
+
+This example excludes files from the `sources` based on the setting of
+the `OS` variable.
+
+```
+{
+  'sources:' [
+    'mac_util.mm',
+    'win_util.cc',
+  ],
+  'conditions': [
+    ['OS=="mac"', {'sources!': ['win_util.cc']}],
+    ['OS=="win"', {'sources!': ['mac_util.cc']}],
+  ],
+}
+```
+
+## Pattern Lists (/)
+
+Pattern lists are similar to, but more powerful than, [exclusion
+lists](#Exclusion_Lists_(!)).  Each item in a pattern list is itself
+a two-element list.  The first item is a string, either `'include'` or
+`'exclude'`, specifying the action to take.  The second item is a string
+specifying a regular expression.  Any item in the base list matching the
+regular expression pattern will either be included or excluded, based on
+the action specified.
+
+Items in a pattern list are processed in sequence, and an excluded item
+that is later included will not be removed from the list (unless it is
+subsequently excluded again.)
+
+Pattern lists are processed after [exclusion
+lists](#Exclusion_Lists_(!)), so it is possible for a pattern list to
+re-include items previously excluded by an exclusion list.
+
+Nothing is actually removed from a base list until all items in an
+[exclusion list](#Exclusion_Lists_(!)) and pattern list have been
+evaluated.  This allows items to retain their correct position relative
+to one another even after being excluded and subsequently included.
+
+#### Example
+
+In this example, a uniform naming scheme is adopted for
+platform-specific files.
+
+```
+{
+  'sources': [
+    'io_posix.cc',
+    'io_win.cc',
+    'launcher_mac.cc',
+    'main.cc',
+    'platform_util_linux.cc',
+    'platform_util_mac.mm',
+  ],
+  'sources/': [
+    ['exclude', '_win\\.cc$'],
+  ],
+  'conditions': [
+    ['OS!="linux"', {'sources/': [['exclude', '_linux\\.cc$']]}],
+    ['OS!="mac"', {'sources/': [['exclude', '_mac\\.cc|mm?$']]}],
+    ['OS=="win"', {'sources/': [
+      ['include', '_win\\.cc$'],
+      ['exclude', '_posix\\.cc$'],
+    ]}],
+  ],
+}
+```
+
+After the pattern list is applied, `sources` will have the following
+values, depending on the setting of `OS`:
+
+  * When `OS` is `linux`: `['io_posix.cc', 'main.cc',
+    'platform_util_linux.cc']`
+  * When `OS` is `mac`: `['io_posix.cc', 'launcher_mac.cc', 'main.cc',
+    'platform_util_mac.mm']`
+  * When `OS` is `win`: `['io_win.cc', 'main.cc',
+    'platform_util_win.cc']`
+
+Note that when `OS` is `win`, the `include` for `_win.cc` files is
+processed after the `exclude` matching the same pattern, because the
+`sources/` list participates in [merging](#Merging) during
+[conditional evaluation](#Conditonals) just like any other list
+would.  This guarantees that the `_win.cc` files, previously
+unconditionally excluded, will be re-included when `OS` is `win`.
+
+## Locating Excluded Items
+
+In some cases, a GYP generator needs to access to items that were
+excluded by an [exclusion list](#Exclusion_Lists_(!)) or [pattern
+list](#Pattern_Lists_(/)).  When GYP excludes items during processing
+of either of these list types, it places the results in an `_excluded`
+list.  In the example above, when `OS` is `mac`, `sources_excluded`
+would be set to `['io_win.cc', 'platform_util_linux.cc']`.  Some GYP
+generators use this feature to display excluded files in the project
+files they generate for the convenience of users, who may wish to refer
+to other implementations.
+
+## Processing Order
+
+GYP uses a defined and predictable order to execute the various steps
+performed between loading files and generating output.
+
+  * Load files.
+    * Load `.gyp` files.  Merge any [command-line
+      includes](#Including_Other_Files) into each `.gyp` file’s root
+      dictionary.  As [includes](#Including_Other_Files) are found,
+      load them as well and [merge](#Merging) them into the scope in
+      which the [includes](#includes) section was found.
+    * Perform [“early” or “pre”](#Early_and_Late_Phases) [variable
+      expansion and conditional
+      evaluation](#Variables_and_Conditionals).
+    * [Merge](#Merging) each [target’s](#targets) dictionary into
+      the `.gyp` file’s root [target\_defaults](#target_defaults)
+      dictionary.
+    * Scan each [target](#targets) for
+      [dependencies](#dependencies), and repeat the above steps for
+      any newly-referenced `.gyp` files not yet loaded.
+  * Scan each [target](#targets) for wildcard
+    [dependencies](#dependencies), expanding the wildcards.
+  * Process [dependent settings](#Dependent_Settings).  These
+    sections are processed, in order:
+    * [all\_dependent\_settings](#all_dependent_settings)
+    * [direct\_dependent\_settings](#direct_dependent_settings)
+    * [link\_dependent\_settings](#link_dependent_settings)
+  * Perform [static library dependency
+    adjustment](#Linking_Dependencies).
+  * Perform [“late,” “post,” or “target”](#Early_and_Late_Phases)
+    [variable expansion and conditional
+    evaluation](#Variables_and_Conditionals) on [target](#targets)
+    dictionaries.
+  * Merge [target](#targets) settings into
+    [configurations](#configurations) as appropriate.
+  * Process [exclusion and pattern
+    lists](#List_Exclusions_and_Patterns).
+
+## Settings Keys
+
+### Settings that may appear anywhere
+
+#### conditions
+
+_List of `condition` items_
+
+A `conditions` section introduces a subdictionary that is only merged
+into the enclosing scope based on the evaluation of a conditional
+expression.  Each `condition` within a `conditions` list is itself a
+list of at least two items:
+
+  1. A string containing the conditional expression itself.  Conditional
+  expressions may take the following forms:
+    * For string values, `var=="value"` and `var!="value"` to test
+      equality and inequality.  For example, `'OS=="linux"'` is true
+      when the `OS` variable is set to `"linux"`.
+    * For integer values, `var==value`, `var!=value`, `var=value`, and `var>value`, to test equality and
+      several common forms of inequality.  For example,
+      `'chromium_code==0'` is true when the `chromium_code` variable is
+      set to `0`.
+    * It is an error for a conditional expression to reference any
+      undefined variable.
+  1. A dictionary containing the subdictionary to be merged into the
+  enclosing scope if the conditional expression evaluates to true.
+
+These two items can be followed by any number of similar two items that
+will be evaluated if the previous conditional expression does not
+evaluate to true.
+
+An additional optional dictionary can be appended to this sequence of
+two items.  This optional dictionary will be merged into the enclosing
+scope if none of the conditional expressions evaluate to true.
+
+Within a `conditions` section, each item is processed sequentially, so
+it is possible to predict the order in which operations will occur.
+
+There is no restriction on nesting `conditions` sections.
+
+`conditions` sections are very similar to `target_conditions` sections.
+See target\_conditions.
+
+#### Example
+
+```
+{
+  'sources': [
+    'common.cc',
+  ],
+  'conditions': [
+    ['OS=="mac"', {'sources': ['mac_util.mm']}],
+    ['OS=="win"', {'sources': ['win_main.cc']}, {'sources': ['posix_main.cc']}],
+    ['OS=="mac"', {'sources': ['mac_impl.mm']},
+     'OS=="win"', {'sources': ['win_impl.cc']},
+     {'sources': ['default_impl.cc']}
+    ],
+  ],
+}
+```
+
+Given this input, the `sources` list will take on different values based
+on the `OS` variable.
+
+  * If `OS` is `"mac"`, `sources` will contain `['common.cc',
+    'mac_util.mm', 'posix_main.cc', 'mac_impl.mm']`.
+  * If `OS` is `"win"`, `sources` will contain `['common.cc',
+    'win_main.cc', 'win_impl.cc']`.
+  * If `OS` is any other value such as `"linux"`, `sources` will contain
+    `['common.cc', 'posix_main.cc', 'default_impl.cc']`.
diff --git a/gyp/docs/LanguageSpecification.md b/gyp/docs/LanguageSpecification.md
new file mode 100644
index 0000000000..178b8c8316
--- /dev/null
+++ b/gyp/docs/LanguageSpecification.md
@@ -0,0 +1,430 @@
+# Language Specification
+
+## Objective
+
+Create a tool for the Chromium project that generates native Visual Studio,
+Xcode and SCons and/or make build files from a platform-independent input
+format.  Make the input format as reasonably general as possible without
+spending extra time trying to "get everything right," except where not doing so
+would likely lead Chromium to an eventual dead end.  When in doubt, do what
+Chromium needs and don't worry about generalizing the solution.
+
+## Background
+
+Numerous other projects, both inside and outside Google, have tried to
+create a simple, universal cross-platform build representation that
+still allows sufficient per-platform flexibility to accommodate
+irreconcilable differences.  The fact that no obvious working candidate
+exists that meets Chromium's requirements indicates this is probably a
+tougher problem than it appears at first glance.  We aim to succeed by
+creating a tool that is highly specific to Chromium's specific use case,
+not to the general case of design a completely platform-independent tool
+for expressing any possible build.
+
+The Mac has the most sophisticated model for application development
+through an IDE.  Consequently, we will use the Xcode model as the
+starting point (the input file format must handle Chromium's use of
+Xcode seamlessly) and adapt the design as necessary for the other
+platforms.
+
+## Overview
+
+The overall design has the following characteristics:
+
+  * Input configurations are specified in files with the suffix `.gyp`.
+  * Each `.gyp` file specifies how to build the targets for the
+    "component" defined by that file.
+  * Each `.gyp` file generates one or more output files appropriate to
+    the platform:
+    * On Mac, a `.gyp` file generates one Xcode .xcodeproj bundle with
+      information about how its targets are built.
+    * On Windows, a `.gyp` file generates one Visual Studio .sln file,
+      and one Visual Studio .vcproj file per target.
+    * On Linux, a `.gyp` file generates one SCons file and/or one
+      Makefile per target
+  * The `.gyp` file syntax is a Python data structure.
+  * Use of arbitrary Python in `.gyp` files is forbidden.
+    * Use of eval() with restricted globals and locals on `.gyp` file
+      contents restricts the input to an evaluated expression, not
+      arbitrary Python statements.
+    * All input is expected to comply with JSON, with two exceptions:
+      the # character (not inside strings) begins a comment that lasts
+      until the end of the line, and trailing commas are permitted at
+      the end of list and dict contents.
+  * Input data is a dictionary of keywords and values.
+  * "Invalid" keywords on any given data structure are not illegal,
+    they're just ignored.
+    * TODO:  providing warnings on use of illegal keywords would help
+      users catch typos.  Figure out something nice to do with this.
+
+## Detailed Design
+
+Some up-front design principles/thoughts/TODOs:
+
+  * Re-use keywords consistently.
+  * Keywords that allow configuration of a platform-specific concept get
+    prefixed appropriately:
+    * Examples:  `msvs_disabled_warnings`, `xcode_framework_dirs`
+  * The input syntax is declarative and data-driven.
+    * This gets enforced by using Python `eval()` (which only evaluates
+      an expression) instead of `exec` (which executes arbitrary python)
+  * Semantic meanings of specific keyword values get deferred until all
+    are read and the configuration is being evaluated to spit out the
+    appropriate file(s)
+  * Source file lists:
+    * Are flat lists.  Any imposed ordering within the `.gyp` file (e.g.
+      alphabetically) is purely by convention and for developer
+      convenience.  When source files are linked or archived together,
+      it is expected that this will occur in the order that files are
+      listed in the `.gyp` file.
+    * Source file lists contain no mechanism for by-hand folder
+      configuration (`Filter` tags in Visual Studio, `Groups` in Xcode)
+    * A folder hierarchy is created automatically that mirrors the file
+      system
+
+### Example
+
+```
+{
+  'target_defaults': {
+    'defines': [
+      'U_STATIC_IMPLEMENTATION',
+      ['LOGFILE', 'foo.log',],
+    ],
+    'include_dirs': [
+      '..',
+    ],
+  },
+  'targets': [
+    {
+      'target_name': 'foo',
+      'type': 'static_library',
+      'sources': [
+        'foo/src/foo.cc',
+        'foo/src/foo_main.cc',
+      ],
+      'include_dirs': [
+         'foo',
+         'foo/include',
+      ],
+      'conditions': [
+         [ 'OS==mac', { 'sources': [ 'platform_test_mac.mm' ] } ]
+      ],
+      'direct_dependent_settings': {
+        'defines': [
+          'UNIT_TEST',
+        ],
+        'include_dirs': [
+          'foo',
+          'foo/include',
+        ],
+      },
+    },
+  ],
+}
+```
+
+### Structural Elements
+
+### Top-level Dictionary
+
+This is the single dictionary in the `.gyp` file that defines the
+targets and how they're to be built.
+
+The following keywords are meaningful within the top-level dictionary
+definition:
+
+| *Keyword*         | *Description*     |
+|:------------------|:------------------|
+| `conditions`      | A conditional section that may contain other items that can be present in a top-level dictionary, on a conditional basis.  See the "Conditionals" section below. |
+| `includes`        | A list of `.gypi` files to be included in the top-level dictionary. |
+| `target_defaults` | A dictionary of default settings to be inherited by all targets in the top-level dictionary.  See the "Settings keywords" section below. |
+| `targets`         | A list of target specifications.  See the "targets" below. |
+| `variables`       | A dictionary containing variable definitions.  Each key in this dictionary is the name of a variable, and each value must be a string value that the variable is to be set to. |
+
+### targets
+
+A list of dictionaries defining targets to be built by the files
+generated from this `.gyp` file.
+
+Targets may contain `includes`, `conditions`, and `variables` sections
+as permitted in the root dictionary. The following additional keywords
+have structural meaning for target definitions:
+
+| *Keyword*         | *Description*     |
+|:---------------------------- |:------------------------------------------|
+| `actions`                    | A list of special custom actions to perform on a specific input file, or files, to produce output files.  See the "Actions" section below. |
+| `all_dependent_settings`     | A dictionary of settings to be applied to all dependents of the target, transitively.  This includes direct dependents and the entire set of their dependents, and so on.  This section may contain anything found within a `target` dictionary, except `configurations`, `target_name`, and `type` sections.  Compare `direct_dependent_settings` and `link_settings`. |
+| `configurations`             | A list of dictionaries defining build configurations for the target.  See the "Configurations" section below.  |
+| `copies`                     | A list of copy actions to perform. See the "Copies" section below. |
+| `defines`                    | A list of preprocesor definitions to be passed on the command line to the C/C++ compiler (via `-D` or `/D` options). |
+| `dependencies`               | A list of targets on which this target depends.  Targets in other `.gyp` files are specified as `../path/to/other.gyp:target_we_want`. |
+| `direct_dependent_settings`  | A dictionary of settings to be applied to other targets that depend on this target.  These settings will only be applied to direct dependents.  This section may contain anything found within a `target` dictionary, except `configurations`, `target_name`, and `type` sections.  Compare with `all_dependent_settings` and `link_settings`. |
+| `include_dirs`               | A list of include directories to be passed on the command line to the C/C++ compiler (via `-I` or `/I` options). |
+| `libraries`                  | A list of list of libraries (and/or frameworks) on which this target depends. |
+| `link_settings`              | A dictionary of settings to be applied to targets in which this target's contents are linked.  `executable` and `shared_library` targets are linkable, so if they depend on a non-linkable target such as a `static_library`, they will adopt its `link_settings`.  This section can contain anything found within a `target` dictionary, except `configurations`, `target_name`, and `type` sections.  Compare `all_dependent_settings` and `direct_dependent_settings`. |
+| `rules`                      | A special custom action to perform on a list of input files, to produce output files.  See the "Rules" section below. |
+| `sources`                    | A list of source files that are used to build this target or which should otherwise show up in the IDE for this target.  In practice, we expect this list to be a union of all files necessary to build the target on all platforms, as well as other related files that aren't actually used for building, like README files. |
+| `target_conditions`          | Like `conditions`, but evaluation is delayed until the settings have been merged into an actual target.  `target_conditions` may be used to place conditionals into a `target_defaults` section but have them still depend on specific target settings. |
+| `target_name`                | The name of a target being defined. |
+| `type`                       | The type of target being defined. This field currently supports `executable`, `static_library`, `shared_library`, and `none`.  The `none` target type is useful when producing output which is not linked. For example, converting raw translation files into resources or documentation into platform specific help files. |
+| `msvs_props`                 | A list of Visual Studio property sheets (`.vsprops` files) to be used to build the target. |
+| `xcode_config_file`          | An Xcode configuration (`.xcconfig` file) to be used to build the target. |
+| `xcode_framework_dirs`       | A list of framework directories be used to build the target. |
+
+You can affect the way that lists/dictionaries are merged together (for
+example the way a list in target\_defaults interacts with the same named
+list in the target itself) with a couple of special characters, which
+are covered in [Merge
+Basics](InputFormatReference#Merge_Basics_(=,_?,_+).md) and [List
+Filters](InputFormatReference#List_Filters.md) on the
+InputFormatReference page.
+
+### configurations
+
+`configurations` sections may be found within `targets` or
+`target_defaults` sections.  The `configurations` section is a list of
+dictionaries specifying different build configurations.  Because
+configurations are implemented as lists, it is not currently possible to
+override aspects of configurations that are imported into a target from
+a `target_defaults` section.
+
+NOTE: It is extremely important that each target within a project define
+the same set of configurations.  This continues to apply even when a
+project spans across multiple `.gyp` files.
+
+A configuration dictionary may contain anything that can be found within
+a target dictionary, except for `actions`, `all_dependent_settings`,
+`configurations`, `dependencies`, `direct_dependent_settings`,
+`libraries`, `link_settings`, `sources`, `target_name`, and `type`.
+
+Configuration dictionaries may also contain these elements:
+
+| *Keyword*            | *Description*                                       |
+|:---------------------|:----------------------------------------------------|
+| `configuration_name` | Required attribute.  The name of the configuration. |
+
+### Conditionals
+
+Conditionals may appear within any dictionary in a `.gyp` file.  There
+are two tpes of conditionals, which differ only in the timing of their
+processing.  `conditons` sections are processed shortly after loading
+`.gyp` files, and `target_conditons` sections are processed after all
+dependencies have been computed.
+
+A conditional section is introduced with a `conditions` or
+`target_conditions` dictionary keyword, and is composed of a list.  Each
+list contains two or three elements.  The first two elements, which are
+always required, are the conditional expression to evaluate and a
+dictionary containing settings to merge into the dictionary containing
+the `conditions` or `target_conditions` section if the expression
+evaluates to true.  The third, optional, list element is a dictionary to
+merge if the expression evaluates to false.
+
+The `eval()` of the expression string takes place in the context of
+global and/or local dictionaries that constructed from the `.gyp` input
+data, and overrides the `__builtin__` dictionary, to prevent the
+execution of arbitrary Python code.
+
+### Actions
+
+An `actions` section provides a list of custom build actions to perform
+on inputs, producing outputs.  The `actions` section is organized as a
+list.  Each item in the list is a dictionary having the following form:
+
+| *Keyword*     | *Type* | *Description*                |
+|:--------------|:-------|:-----------------------------|
+| `action_name` | string | The name of the action.  Depending on how actions are implemented in the various generators, some may desire or require this property to be set to a unique name; others may ignore this property entirely. |
+| `inputs`      | list   | A list of pathnames treated as inputs to the custom action. |
+| `outputs`     | list   | A list of pathnames that the custom action produces. |
+| `action`      | list   | A command line invocation used to produce `outputs` from `inputs`.  For maximum cross-platform compatibility, invocations that require a Python interpreter should be specified with a first element `"python"`.  This will enable generators for environments with specialized Python installations to be able to perform the action in an appropriate Python environment. |
+| `message`     | string | A message to be displayed to the user by the build system when the action is run. |
+
+Build environments will compare `inputs` and `outputs`.  If any `output`
+is missing or is outdated relative to any `input`, the custom action
+will be invoked.  If all `outputs` are present and newer than all
+`inputs`, the `outputs` are considered up-to-date and the action need
+not be invoked.
+
+Actions are implemented in Xcode as shell script build phases performed
+prior to the compilation phase.  In the Visual Studio generator, actions
+appear files with a `FileConfiguration` containing a custom
+`VCCustomBuildTool` specifying the remainder of the inputs, the outputs,
+and the action.
+
+Combined with variable expansions, actions can be quite powerful.  Here
+is an example action that leverages variable expansions to minimize
+duplication of pathnames:
+
+```
+      'sources': [
+        # libraries.cc is generated by the js2c action below.
+        '<(INTERMEDIATE_DIR)/libraries.cc',
+      ],
+      'actions': [
+        {
+          'variables': {
+            'core_library_files': [
+              'src/runtime.js',
+              'src/v8natives.js',
+              'src/macros.py',
+            ],
+          },
+          'action_name': 'js2c',
+          'inputs': [
+            'tools/js2c.py',
+            '<@(core_library_files)',
+          ],
+          'outputs': [
+            '<(INTERMEDIATE_DIR)/libraries.cc',
+            '<(INTERMEDIATE_DIR)/libraries-empty.cc',
+          ],
+          'action': ['python', 'tools/js2c.py', '<@(_outputs)', 'CORE', '<@(core_library_files)'],
+        },
+      ],
+```
+
+### Rules
+
+A `rules` section provides custom build action to perform on inputs, producing
+outputs.  The `rules` section is organized as a list.  Each item in the list is
+a dictionary having the following form:
+
+| *Keyword*   | *Type* | *Description*                            |
+|:------------|:-------|:-----------------------------------------|
+| `rule_name` | string | The name of the rule.  Depending on how Rules are implemented in the various generators, some may desire or require this property to be set to a unique name; others may ignore this property entirely. |
+| `extension` | string | All source files of the current target with the given extension will be treated successively as inputs to the rule. |
+| `inputs`    | list   | Additional dependencies of the rule. |
+| `outputs`   | list   | A list of pathnames that the rule produces. Has access to `RULE_INPUT_` variables (see below). |
+| `action`    | list   | A command line invocation used to produce `outputs` from `inputs`.  For maximum cross-platform compatibility, invocations that require a Python interpreter should be specified with a first element `"python"`.  This will enable generators for environments with specialized Python installations to be able to perform the action in an appropriate Python environment. Has access to `RULE_INPUT_` variables (see below). |
+| `message`   | string | A message to be displayed to the user by the build system when the action is run. Has access to `RULE_INPUT_` variables (see below). |
+
+There are several variables available to `outputs`, `action`, and `message`.
+
+|  *Variable*          | *Description*                       |
+|:---------------------|:------------------------------------|
+| `RULE_INPUT_PATH`    | The full path to the current input. |
+| `RULE_INPUT_DIRNAME` | The directory of the current input. |
+| `RULE_INPUT_NAME`    | The file name of the current input. |
+| `RULE_INPUT_ROOT`    | The file name of the current input without extension. |
+| `RULE_INPUT_EXT`     | The file name extension of the current input. |
+
+Rules can be thought of as Action generators. For each source selected
+by `extension` an special action is created. This action starts out with
+the same `inputs`, `outputs`, `action`, and `message` as the rule. The
+source is added to the action's `inputs`. The `outputs`, `action`, and
+`message` are then handled the same but with the additional variables.
+If the `_output` variable is used in the `action` or `message` the
+`RULE_INPUT_` variables in `output` will be expanded for the current
+source.
+
+### Copies
+
+A `copies` section provides a simple means of copying files.  The
+`copies` section is organized as a list.  Each item in the list is a
+dictionary having the following form:
+
+| *Keyword*     | *Type* | *Description*                 |
+|:--------------|:-------|:------------------------------|
+| `destination` | string | The directory into which the `files` will be copied. |
+| `files`       | list   | A list of files to be copied. |
+
+The copies will be created in `destination` and have the same file name
+as the file they are copied from. Even if the `files` are from multiple
+directories they will all be copied into the `destination` directory.
+Each `destination` file has an implicit build dependency on the file it
+is copied from.
+
+### Generated Xcode .pbxproj Files
+
+We derive the following things in a `project.pbxproj` plist file within
+an `.xcodeproj` bundle from the above input file formats as follows:
+
+  * `Group hierarchy`: This is generated in a fixed format with contents
+    derived from the input files. There is no provision for the user to
+    specify additional groups or create a custom hierarchy.
+    * `Configuration group`: This will be used with the
+      `xcode_config_file` property above, if needed.
+    * `Source group`: The union of the `sources` lists of all `targets`
+      after applying appropriate `conditions`.  The resulting list is
+      sorted and put into a group hierarchy that matches the layout of
+      the directory tree on disk, with a root of // (the top of the
+      hierarchy).
+    * `Frameworks group`: Taken directly from `libraries` value for the
+      target, after applying appropriate conditions.
+    * `Projects group`: References to other `.xcodeproj` bundles that
+      are needed by the `.xcodeproj` in which the group is contained.
+    * `Products group`: Output from the various targets.
+  * `Project References`:
+  * `Project Configurations`:
+    * Per-`.xcodeproj` file settings are not supported, all settings are
+      applied at the target level.
+  * `Targets`:
+    * `Phases`: Copy sources, link with libraries/frameworks, ...
+    * `Target Configurations`: Specified by input.
+    * `Dependencies`: (local and remote)
+
+### Generated Visual Studio .vcproj Files
+
+We derive the following sections in a `.vcproj` file from the above
+input file formats as follows:
+
+  * `VisualStudioProject`:
+    * `Platforms`:
+    * `ToolFiles`:
+    * `Configurations`:
+      * `Configuration`:
+    * `References`:
+    * `Files`:
+      * `Filter`:
+      * `File`:
+        * `FileConfiguration`:
+          * `Tool`:
+    * `Globals`:
+
+### Generated Visual Studio .sln Files
+
+We derive the following sections in a `.sln` file from the above input
+file formats as follows:
+
+  * `Projects`:
+    * `WebsiteProperties`:
+    * `ProjectDependencies`:
+  * `Global`:
+    * `SolutionConfigurationPlatforms`:
+    * `ProjectConfigurationPlatforms`:
+    * `SolutionProperties`:
+    * `NestedProjects`:
+
+## Caveats
+
+Notes/Question from very first prototype draft of the language.
+Make sure these issues are addressed somewhere before deleting.
+
+  * Libraries are easy, application abstraction is harder
+    * Applications involves resource compilation
+    * Applications involve many inputs
+    * Applications include transitive closure of dependencies
+  * Specific use cases like cc\_library
+    * Mac compiles more than just .c/.cpp files (specifically, .m and .mm
+      files)
+    * Compiler options vary by:
+      * File type
+      * Target type
+      * Individual file
+    * Files may have custom settings per file per platform, but we probably
+      don't care or need to support this in gyp.
+  * Will all linked non-Chromium projects always use the same versions of every
+    subsystem?
+  * Variants are difficult.  We've identified the following variants (some
+    specific to Chromium, some typical of other projects in the same ballpark):
+    * Target platform
+    * V8 vs. JSC
+    * Debug vs. Release
+    * Toolchain (VS version, gcc, version)
+    * Host platform
+    * L10N
+    * Vendor
+    * Purify / Valgrind
+  * Will everyone upgrade VS at once?
+  * What does a dylib dependency mean?
diff --git a/gyp/docs/README.md b/gyp/docs/README.md
new file mode 100644
index 0000000000..5f9b6a59ce
--- /dev/null
+++ b/gyp/docs/README.md
@@ -0,0 +1,27 @@
+# Generate Your Projects (gyp-next)
+
+GYP is a Meta-Build system: a build system that generates other build systems.
+
+* [User documentation](./UserDocumentation.md)
+* [Input Format Reference](./InputFormatReference.md)
+* [Language specification](./LanguageSpecification.md)
+* [Hacking](./Hacking.md)
+* [Testing](./Testing.md)
+* [GYP vs. CMake](./GypVsCMake.md)
+
+GYP is intended to support large projects that need to be built on multiple
+platforms (e.g., Mac, Windows, Linux), and where it is important that
+the project can be built using the IDEs that are popular on each platform
+as if the project is a "native" one.
+
+It can be used to generate XCode projects, Visual Studio projects, Ninja
+build files, and Makefiles. In each case GYP's goal is to replicate as
+closely as possible the way one would set up a native build of the project
+using the IDE.
+
+GYP can also be used to generate "hybrid" projects that provide the IDE
+scaffolding for a nice user experience but call out to Ninja to do the actual
+building (which is usually much faster than the native build systems of the
+IDEs).
+
+For more information on GYP, click on the links above.
diff --git a/gyp/docs/Testing.md b/gyp/docs/Testing.md
new file mode 100644
index 0000000000..baeb65f944
--- /dev/null
+++ b/gyp/docs/Testing.md
@@ -0,0 +1,450 @@
+# Testing
+
+NOTE: this document is outdated and needs to be updated. Read with your own discretion.
+
+## Introduction
+
+This document describes the GYP testing infrastructure,
+as provided by the `TestGyp.py` module.
+
+These tests emphasize testing the _behavior_ of the
+various GYP-generated build configurations:
+Visual Studio, Xcode, SCons, Make, etc.
+The goal is _not_ to test the output of the GYP generators by,
+for example, comparing a GYP-generated Makefile
+against a set of known "golden" Makefiles
+(although the testing infrastructure could
+be used to write those kinds of tests).
+The idea is that the generated build configuration files
+could be completely written to add a feature or fix a bug
+so long as they continue to support the functional behaviors
+defined by the tests:  building programs, shared libraries, etc.
+
+## "Hello, world!" GYP test configuration
+
+Here is an actual test configuration,
+a simple build of a C program to print `"Hello, world!"`.
+
+```
+  $ ls -l test/hello
+  total 20
+  -rw-r--r-- 1 knight knight 312 Jul 30 20:22 gyptest-all.py
+  -rw-r--r-- 1 knight knight 307 Jul 30 20:22 gyptest-default.py
+  -rwxr-xr-x 1 knight knight 326 Jul 30 20:22 gyptest-target.py
+  -rw-r--r-- 1 knight knight  98 Jul 30 20:22 hello.c
+  -rw-r--r-- 1 knight knight 142 Jul 30 20:22 hello.gyp
+  $
+```
+
+The `gyptest-*.py` files are three separate tests (test scripts)
+that use this configuration.  The first one, `gyptest-all.py`,
+looks like this:
+
+```
+  #!/usr/bin/env python
+
+  """
+  Verifies simplest-possible build of a "Hello, world!" program
+  using an explicit build target of 'all'.
+  """
+
+  import TestGyp
+
+  test = TestGyp.TestGyp()
+
+  test.run_gyp('hello.gyp')
+
+  test.build_all('hello.gyp')
+
+  test.run_built_executable('hello', stdout="Hello, world!\n")
+
+  test.pass_test()
+```
+
+The test script above runs GYP against the specified input file
+(`hello.gyp`) to generate a build configuration.
+It then tries to build the `'all'` target
+(or its equivalent) using the generated build configuration.
+Last, it verifies that the build worked as expected
+by running the executable program (`hello`)
+that was just presumably built by the generated configuration,
+and verifies that the output from the program
+matches the expected `stdout` string (`"Hello, world!\n"`).
+
+Which configuration is generated
+(i.e., which build tool to test)
+is specified when the test is run;
+see the next section.
+
+Surrounding the functional parts of the test
+described above are the header,
+which should be basically the same for each test
+(modulo a different description in the docstring):
+
+```
+  #!/usr/bin/env python
+
+  """
+  Verifies simplest-possible build of a "Hello, world!" program
+  using an explicit build target of 'all'.
+  """
+
+  import TestGyp
+
+  test = TestGyp.TestGyp()
+```
+
+Similarly, the footer should be the same in every test:
+
+```
+  test.pass_test()
+```
+
+## Running tests
+
+Test scripts are run by the `gyptest.py` script.
+You can specify (an) explicit test script(s) to run:
+
+```
+  $ python gyptest.py test/hello/gyptest-all.py
+  PYTHONPATH=/home/knight/src/gyp/trunk/test/lib
+  TESTGYP_FORMAT=scons
+  /usr/bin/python test/hello/gyptest-all.py
+  PASSED
+  $
+```
+
+If you specify a directory, all test scripts
+(scripts prefixed with `gyptest-`) underneath
+the directory will be run:
+
+```
+  $ python gyptest.py test/hello
+  PYTHONPATH=/home/knight/src/gyp/trunk/test/lib
+  TESTGYP_FORMAT=scons
+  /usr/bin/python test/hello/gyptest-all.py
+  PASSED
+  /usr/bin/python test/hello/gyptest-default.py
+  PASSED
+  /usr/bin/python test/hello/gyptest-target.py
+  PASSED
+  $
+```
+
+Or you can specify the `-a` option to run all scripts
+in the tree:
+
+```
+  $ python gyptest.py -a
+  PYTHONPATH=/home/knight/src/gyp/trunk/test/lib
+  TESTGYP_FORMAT=scons
+  /usr/bin/python test/configurations/gyptest-configurations.py
+  PASSED
+  /usr/bin/python test/defines/gyptest-defines.py
+  PASSED
+      .
+      .
+      .
+      .
+  /usr/bin/python test/variables/gyptest-commands.py
+  PASSED
+  $
+```
+
+If any tests fail during the run,
+the `gyptest.py` script will report them in a
+summary at the end.
+
+## Debugging tests
+
+Tests that create intermediate output do so under the gyp/out/testworkarea
+directory. On test completion, intermediate output is cleaned up. To preserve
+this output, set the environment variable PRESERVE=1. This can be handy to
+inspect intermediate data when debugging a test.
+
+You can also set PRESERVE\_PASS=1, PRESERVE\_FAIL=1 or PRESERVE\_NO\_RESULT=1
+to preserve output for tests that fall into one of those categories.
+
+# Specifying the format (build tool) to use
+
+By default, the `gyptest.py` script will generate configurations for
+the "primary" supported build tool for the platform you're on:
+Visual Studio on Windows,
+Xcode on Mac,
+and (currently) SCons on Linux.
+An alternate format (build tool) may be specified
+using the `-f` option:
+
+```
+  $ python gyptest.py -f make test/hello/gyptest-all.py
+  PYTHONPATH=/home/knight/src/gyp/trunk/test/lib
+  TESTGYP_FORMAT=make
+  /usr/bin/python test/hello/gyptest-all.py
+  PASSED
+  $
+```
+
+Multiple tools may be specified in a single pass as
+a comma-separated list:
+
+```
+  $ python gyptest.py -f make,scons test/hello/gyptest-all.py
+  PYTHONPATH=/home/knight/src/gyp/trunk/test/lib
+  TESTGYP_FORMAT=make
+  /usr/bin/python test/hello/gyptest-all.py
+  PASSED
+  TESTGYP_FORMAT=scons
+  /usr/bin/python test/hello/gyptest-all.py
+  PASSED
+  $
+```
+
+## Test script functions and methods
+
+The `TestGyp` class contains a lot of functionality
+intended to make it easy to write tests.
+This section describes the most useful pieces for GYP testing.
+
+(The `TestGyp` class is actually a subclass of more generic
+`TestCommon` and `TestCmd` base classes
+that contain even more functionality than is
+described here.)
+
+### Initialization
+
+The standard initialization formula is:
+
+```
+  import TestGyp
+  test = TestGyp.TestGyp()
+```
+
+This copies the contents of the directory tree in which
+the test script lives to a temporary directory for execution,
+and arranges for the temporary directory's removal on exit.
+
+By default, any comparisons of output or file contents
+must be exact matches for the test to pass.
+If you need to use regular expressions for matches,
+a useful alternative initialization is:
+
+```
+  import TestGyp
+  test = TestGyp.TestGyp(match = TestGyp.match_re,
+                         diff = TestGyp.diff_re)`
+```
+
+### Running GYP
+
+The canonical invocation is to simply specify the `.gyp` file to be executed:
+
+```
+  test.run_gyp('file.gyp')
+```
+
+Additional GYP arguments may be specified:
+
+```
+  test.run_gyp('file.gyp', arguments=['arg1', 'arg2', ...])
+```
+
+To execute GYP from a subdirectory (where, presumably, the specified file
+lives):
+
+```
+  test.run_gyp('file.gyp', chdir='subdir')
+```
+
+### Running the build tool
+
+Running the build tool requires passing in a `.gyp` file, which may be used to
+calculate the name of a specific build configuration file (such as a MSVS
+solution file corresponding to the `.gyp` file).
+
+There are several different `.build_*()` methods for invoking different types
+of builds.
+
+To invoke a build tool with an explicit `all` target (or equivalent):
+
+```
+  test.build_all('file.gyp')
+```
+
+To invoke a build tool with its default behavior (for example, executing `make`
+with no targets specified):
+
+```
+  test.build_default('file.gyp')
+```
+
+To invoke a build tool with an explicit specified target:
+
+```
+  test.build_target('file.gyp', 'target')
+```
+
+### Running executables
+
+The most useful method executes a program built by the GYP-generated
+configuration:
+
+```
+  test.run_built_executable('program')
+```
+
+The `.run_built_executable()` method will account for the actual built target
+output location for the build tool being tested, as well as tack on any
+necessary executable file suffix for the platform (for example `.exe` on
+Windows).
+
+`stdout=` and `stderr=` keyword arguments specify expected standard output and
+error output, respectively.  Failure to match these (if specified) will cause
+the test to fail.  An explicit `None` value will suppress that verification:
+
+```
+  test.run_built_executable('program',
+                            stdout="expect this output\n",
+							stderr=None)
+```
+
+Note that the default values are `stdout=None` and `stderr=''` (that is, no
+check for standard output, and error output must be empty).
+
+Arbitrary executables (not necessarily those built by GYP) can be executed with
+the lower-level `.run()` method:
+
+```
+  test.run('program')
+```
+
+The program must be in the local directory (that is, the temporary directory
+for test execution) or be an absolute path name.
+
+### Fetching command output
+
+```
+  test.stdout()
+```
+
+Returns the standard output from the most recent executed command (including
+`.run_gyp()`, `.build_*()`, or `.run*()` methods).
+
+```
+  test.stderr()
+```
+
+Returns the error output from the most recent executed command (including
+`.run_gyp()`, `.build_*()`, or `.run*()` methods).
+
+### Verifying existence or non-existence of files or directories
+
+```
+  test.must_exist('file_or_dir')
+```
+
+Verifies that the specified file or directory exists, and fails the test if it
+doesn't.
+
+```
+  test.must_not_exist('file_or_dir')
+```
+
+Verifies that the specified file or directory does not exist, and fails the
+test if it does.
+
+### Verifying file contents
+
+```
+  test.must_match('file', 'expected content\n')
+```
+
+Verifies that the content of the specified file match the expected string, and
+fails the test if it does not.  By default, the match must be exact, but
+line-by-line regular expressions may be used if the `TestGyp` object was
+initialized with `TestGyp.match_re`.
+
+```
+  test.must_not_match('file', 'expected content\n')
+```
+
+Verifies that the content of the specified file does _not_ match the expected
+string, and fails the test if it does.  By default, the match must be exact,
+but line-by-line regular expressions may be used if the `TestGyp` object was
+initialized with `TestGyp.match_re`.
+
+```
+  test.must_contain('file', 'substring')
+```
+
+Verifies that the specified file contains the specified substring, and fails
+the test if it does not.
+
+```
+  test.must_not_contain('file', 'substring')
+```
+
+Verifies that the specified file does not contain the specified substring, and
+fails the test if it does.
+
+```
+  test.must_contain_all_lines(output, lines)
+```
+
+Verifies that the output string contains all of the "lines" in the specified
+list of lines.  In practice, the lines can be any substring and need not be
+`\n`-terminaed lines per se.  If any line is missing, the test fails.
+
+```
+  test.must_not_contain_any_lines(output, lines)
+```
+
+Verifies that the output string does _not_ contain any of the "lines" in the
+specified list of lines.  In practice, the lines can be any substring and need
+not be `\n`-terminaed lines per se.  If any line exists in the output string,
+the test fails.
+
+```
+  test.must_contain_any_line(output, lines)
+```
+
+Verifies that the output string contains at least one of the "lines" in the
+specified list of lines.  In practice, the lines can be any substring and need
+not be `\n`-terminaed lines per se.  If none of the specified lines is present,
+the test fails.
+
+### Reading file contents
+
+```
+  test.read('file')
+```
+
+Returns the contents of the specified file.  Directory elements contained in a
+list will be joined:
+
+```
+  test.read(['subdir', 'file'])
+```
+
+### Test success or failure
+
+```
+  test.fail_test()
+```
+
+Fails the test, reporting `FAILED` on standard output and exiting with an exit
+status of `1`.
+
+```
+  test.pass_test()
+```
+
+Passes the test, reporting `PASSED` on standard output and exiting with an exit
+status of `0`.
+
+```
+  test.no_result()
+```
+
+Indicates the test had no valid result (i.e., the conditions could not be
+tested because of an external factor like a full file system).  Reports `NO
+RESULT` on standard output and exits with a status of `2`.
diff --git a/gyp/docs/UserDocumentation.md b/gyp/docs/UserDocumentation.md
new file mode 100644
index 0000000000..808f37a1a9
--- /dev/null
+++ b/gyp/docs/UserDocumentation.md
@@ -0,0 +1,965 @@
+# User Documentation
+
+## Introduction
+
+This document is intended to provide a user-level guide to GYP.  The
+emphasis here is on how to use GYP to accomplish specific tasks, not on
+the complete technical language specification.  (For that, see the
+[LanguageSpecification](LanguageSpecification.md).)
+
+The document below starts with some overviews to provide context: an
+overview of the structure of a `.gyp` file itself, an overview of a
+typical executable-program target in a `.gyp` file, an an overview of a
+typical library target in a `.gyp` file.
+
+After the overviews, there are examples of `gyp` patterns for different
+common use cases.
+
+## Skeleton of a typical Chromium .gyp file
+
+Here is the skeleton of a typical `.gyp` file in the Chromium tree:
+
+```
+  {
+    'variables': {
+      .
+      .
+      .
+    },
+    'includes': [
+      '../build/common.gypi',
+    ],
+    'target_defaults': {
+      .
+      .
+      .
+    },
+    'targets': [
+      {
+        'target_name': 'target_1',
+          .
+          .
+          .
+      },
+      {
+        'target_name': 'target_2',
+          .
+          .
+          .
+      },
+    ],
+    'conditions': [
+      ['OS=="linux"', {
+        'targets': [
+          {
+            'target_name': 'linux_target_3',
+              .
+              .
+              .
+          },
+        ],
+      }],
+      ['OS=="win"', {
+        'targets': [
+          {
+            'target_name': 'windows_target_4',
+              .
+              .
+              .
+          },
+        ],
+      }, { # OS != "win"
+        'targets': [
+          {
+            'target_name': 'non_windows_target_5',
+              .
+              .
+              .
+          },
+      }],
+    ],
+  }
+```
+
+The entire file just contains a Python dictionary.  (It's actually JSON,
+with two small Pythonic deviations: comments are introduced with `#`,
+and a `,` (comma)) is legal after the last element in a list or
+dictionary.)
+
+The top-level pieces in the `.gyp` file are as follows:
+
+`'variables'`:  Definitions of variables that can be interpolated and
+used in various other parts of the file.
+
+`'includes'`:  A list of of other files that will be included in this
+file.  By convention, included files have the suffix `.gypi` (gyp
+include).
+
+`'target_defaults'`:  Settings that will apply to _all_ of the targets
+defined in this `.gyp` file.
+
+`'targets'`:  The list of targets for which this `.gyp` file can
+generate builds.  Each target is a dictionary that contains settings
+describing all the information necessary to build the target.
+
+`'conditions'`:  A list of condition specifications that can modify the
+contents of the items in the global dictionary defined by this `.gyp`
+file based on the values of different variablwes.  As implied by the
+above example, the most common use of a `conditions` section in the
+top-level dictionary is to add platform-specific targets to the
+`targets` list.
+
+## Skeleton of a typical executable target in a .gyp file
+
+The most straightforward target is probably a simple executable program.
+Here is an example `executable` target that demonstrates the features
+that should cover most simple uses of gyp:
+
+```
+  {
+    'targets': [
+      {
+        'target_name': 'foo',
+        'type': 'executable',
+        'msvs_guid': '5ECEC9E5-8F23-47B6-93E0-C3B328B3BE65',
+        'dependencies': [
+          'xyzzy',
+          '../bar/bar.gyp:bar',
+        ],
+        'defines': [
+          'DEFINE_FOO',
+          'DEFINE_A_VALUE=value',
+        ],
+        'include_dirs': [
+          '..',
+        ],
+        'sources': [
+          'file1.cc',
+          'file2.cc',
+        ],
+        'conditions': [
+          ['OS=="linux"', {
+            'defines': [
+              'LINUX_DEFINE',
+            ],
+            'include_dirs': [
+              'include/linux',
+            ],
+          }],
+          ['OS=="win"', {
+            'defines': [
+              'WINDOWS_SPECIFIC_DEFINE',
+            ],
+          }, { # OS != "win",
+            'defines': [
+              'NON_WINDOWS_DEFINE',
+            ],
+          }]
+        ],
+      },
+    ],
+  }
+```
+
+The top-level settings in the target include:
+
+`'target_name'`: The name by which the target should be known, which
+should be unique across all `.gyp` files.  This name will be used as the
+project name in the generated Visual Studio solution, as the target name
+in the generated XCode configuration, and as the alias for building this
+target from the command line of the generated SCons configuration.
+
+`'type'`: Set to `executable`, logically enough.
+
+`'msvs_guid'`: THIS IS ONLY TRANSITIONAL.  This is a hard-coded GUID
+values that will be used in the generated Visual Studio solution
+file(s).  This allows us to check in a `chrome.sln` file that
+interoperates with gyp-generated project files.  Once everything in
+Chromium is being generated by gyp, it will no longer be important that
+the GUIDs stay constant across invocations, and we'll likely get rid of
+these settings,
+
+`'dependencies'`: This lists other targets that this target depends on.
+The gyp-generated files will guarantee that the other targets are built
+before this target.  Any library targets in the `dependencies` list will
+be linked with this target.  The various settings (`defines`,
+`include_dirs`, etc.) listed in the `direct_dependent_settings` sections
+of the targets in this list will be applied to how _this_ target is
+built and linked.  See the more complete discussion of
+`direct_dependent_settings`, below.
+
+`'defines'`: The C preprocessor definitions that will be passed in on
+compilation command lines (using `-D` or `/D` options).
+
+`'include_dirs'`: The directories in which included header files live.
+These will be passed in on compilation command lines (using `-I` or `/I`
+options).
+
+`'sources'`: The source files for this target.
+
+`'conditions'`: A block of conditions that will be evaluated to update
+the different settings in the target dictionary.
+
+## Skeleton of a typical library target in a .gyp file
+
+The vast majority of targets are libraries.  Here is an example of a
+library target including the additional features that should cover most
+needs of libraries:
+
+```
+  {
+    'targets': [
+      {
+        'target_name': 'foo',
+        'type': '<(library)'
+        'msvs_guid': '5ECEC9E5-8F23-47B6-93E0-C3B328B3BE65',
+        'dependencies': [
+          'xyzzy',
+          '../bar/bar.gyp:bar',
+        ],
+        'defines': [
+          'DEFINE_FOO',
+          'DEFINE_A_VALUE=value',
+        ],
+        'include_dirs': [
+          '..',
+        ],
+        'direct_dependent_settings': {
+          'defines': [
+            'DEFINE_FOO',
+            'DEFINE_ADDITIONAL',
+          ],
+          'linkflags': [
+          ],
+        },
+        'export_dependent_settings': [
+          '../bar/bar.gyp:bar',
+        ],
+        'sources': [
+          'file1.cc',
+          'file2.cc',
+        ],
+        'conditions': [
+          ['OS=="linux"', {
+            'defines': [
+              'LINUX_DEFINE',
+            ],
+            'include_dirs': [
+              'include/linux',
+            ],
+          ],
+          ['OS=="win"', {
+            'defines': [
+              'WINDOWS_SPECIFIC_DEFINE',
+            ],
+          }, { # OS != "win",
+            'defines': [
+              'NON_WINDOWS_DEFINE',
+            ],
+          }]
+        ],
+    ],
+  }
+```
+
+The possible entries in a library target are largely the same as those
+that can be specified for an executable target (`defines`,
+`include_dirs`, etc.).  The differences include:
+
+`'type'`: This should almost always be set to '<(library)', which allows
+the user to define at gyp time whether libraries are to be built static
+or shared.  (On Linux, at least, linking with shared libraries saves
+significant link time.) If it's necessary to pin down the type of
+library to be built, the `type` can be set explicitly to
+`static_library` or `shared_library`.
+
+`'direct_dependent_settings'`: This defines the settings that will be
+applied to other targets that _directly depend_ on this target--that is,
+that list _this_ target in their `'dependencies'` setting.  This is
+where you list the `defines`, `include_dirs`, `cflags` and `linkflags`
+that other targets that compile or link against this target need to
+build consistently.
+
+`'export_dependent_settings'`: This lists the targets whose
+`direct_dependent_settings` should be "passed on" to other targets that
+use (depend on) this target.  `TODO:  expand on this description.`
+
+## Use Cases
+
+These use cases are intended to cover the most common actions performed
+by developers using GYP.
+
+Note that these examples are _not_ fully-functioning, self-contained
+examples (or else they'd be way too long).  Each example mostly contains
+just the keywords and settings relevant to the example, with perhaps a
+few extra keywords for context.  The intent is to try to show the
+specific pieces you need to pay attention to when doing something.
+[NOTE:  if practical use shows that these examples are confusing without
+additional context, please add what's necessary to clarify things.]
+
+### Add new source files
+
+There are similar but slightly different patterns for adding a
+platform-independent source file vs. adding a source file that only
+builds on some of the supported platforms.
+
+#### Add a source file that builds on all platforms
+
+**Simplest possible case**: You are adding a file(s) that builds on all
+platforms.
+
+Just add the file(s) to the `sources` list of the appropriate dictionary
+in the `targets` list:
+
+```
+  {
+    'targets': [
+      {
+        'target_name': 'my_target',
+        'type': 'executable',
+        'sources': [
+          '../other/file_1.cc',
+          'new_file.cc',
+          'subdir/file3.cc',
+        ],
+      },
+    ],
+  },
+```
+
+File path names are relative to the directory in which the `.gyp` file lives.
+
+Keep the list sorted alphabetically (unless there's a really, really,
+_really_ good reason not to).
+
+#### Add a platform-specific source file
+
+##### Your platform-specific file is named `*_linux.{ext}`, `*_mac.{ext}`, `*_posix.{ext}` or `*_win.{ext}`
+
+The simplest way to add a platform-specific source file, assuming you're
+adding a completely new file and get to name it, is to use one of the
+following standard suffixes:
+
+  * `_linux`  (e.g. `foo_linux.cc`)
+  * `_mac`    (e.g. `foo_mac.cc`)
+  * `_posix`  (e.g. `foo_posix.cc`)
+  * `_win`    (e.g. `foo_win.cc`)
+
+Simply add the file to the `sources` list of the appropriate dict within
+the `targets` list, like you would any other source file.
+
+```
+  {
+    'targets': [
+      {
+        'target_name': 'foo',
+        'type': 'executable',
+        'sources': [
+          'independent.cc',
+          'specific_win.cc',
+        ],
+      },
+    ],
+  },
+```
+
+The Chromium `.gyp` files all have appropriate `conditions` entries to
+filter out the files that aren't appropriate for the current platform.
+In the above example, the `specific_win.cc` file will be removed
+automatically from the source-list on non-Windows builds.
+
+##### Your platform-specific file does not use an already-defined pattern
+
+If your platform-specific file does not contain a
+`*_{linux,mac,posix,win}` substring (or some other pattern that's
+already in the `conditions` for the target), and you can't change the
+file name, there are two patterns that can be used.
+
+**Prefererred**:  Add the file to the `sources` list of the appropriate
+dictionary within the `targets` list.  Add an appropriate `conditions`
+section to exclude the specific files name:
+
+```
+  {
+    'targets': [
+      {
+        'target_name': 'foo',
+        'type': 'executable',
+        'sources': [
+          'linux_specific.cc',
+        ],
+        'conditions': [
+          ['OS != "linux"', {
+            'sources!': [
+              # Linux-only; exclude on other platforms.
+              'linux_specific.cc',
+            ]
+          }[,
+        ],
+      },
+    ],
+  },
+```
+
+Despite the duplicate listing, the above is generally preferred because
+the `sources` list contains a useful global list of all sources on all
+platforms with consistent sorting on all platforms.
+
+**Non-preferred**: In some situations, however, it might make sense to
+list a platform-specific file only in a `conditions` section that
+specifically _includes_ it in the `sources` list:
+
+```
+  {
+    'targets': [
+      {
+        'target_name': 'foo',
+        'type': 'executable',
+        'sources': [],
+        ['OS == "linux"', {
+          'sources': [
+            # Only add to sources list on Linux.
+            'linux_specific.cc',
+          ]
+        }],
+      },
+    ],
+  },
+```
+
+The above two examples end up generating equivalent builds, with the
+small exception that the `sources` lists will list the files in
+different orders.  (The first example defines explicitly where
+`linux_specific.cc` appears in the list--perhaps in in the
+middle--whereas the second example will always tack it on to the end of
+the list.)
+
+**Including or excluding files using patterns**: There are more
+complicated ways to construct a `sources` list based on patterns.  See
+`TODO` below.
+
+### Add a new executable
+
+An executable program is probably the most straightforward type of
+target, since all it typically needs is a list of source files, some
+compiler/linker settings (probably varied by platform), and some library
+targets on which it depends and which must be used in the final link.
+
+#### Add an executable that builds on all platforms
+
+Add a dictionary defining the new executable target to the `targets`
+list in the appropriate `.gyp` file.  Example:
+
+```
+  {
+    'targets': [
+      {
+        'target_name': 'new_unit_tests',
+        'type': 'executable',
+        'defines': [
+          'FOO',
+        ],
+        'include_dirs': [
+          '..',
+        ],
+        'dependencies': [
+          'other_target_in_this_file',
+          'other_gyp2:target_in_other_gyp2',
+        ],
+        'sources': [
+          'new_additional_source.cc',
+          'new_unit_tests.cc',
+        ],
+      },
+    ],
+  }
+```
+
+#### Add a platform-specific executable
+
+Add a dictionary defining the new executable target to the `targets`
+list within an appropriate `conditions` block for the platform.  The
+`conditions` block should be a sibling to the top-level `targets` list:
+
+```
+  {
+    'targets': [
+    ],
+    'conditions': [
+      ['OS=="win"', {
+        'targets': [
+          {
+            'target_name': 'new_unit_tests',
+            'type': 'executable',
+            'defines': [
+              'FOO',
+            ],
+            'include_dirs': [
+              '..',
+            ],
+            'dependencies': [
+              'other_target_in_this_file',
+              'other_gyp2:target_in_other_gyp2',
+            ],
+            'sources': [
+              'new_additional_source.cc',
+              'new_unit_tests.cc',
+            ],
+          },
+        ],
+      }],
+    ],
+  }
+```
+
+### Add settings to a target
+
+There are several different types of settings that can be defined for
+any given target.
+
+#### Add new preprocessor definitions (`-D` or `/D` flags)
+
+New preprocessor definitions are added by the `defines` setting:
+
+```
+  {
+    'targets': [
+      {
+        'target_name': 'existing_target',
+        'defines': [
+          'FOO',
+          'BAR=some_value',
+        ],
+      },
+    ],
+  },
+```
+
+These may be specified directly in a target's settings, as in the above
+example, or in a `conditions` section.
+
+#### Add a new include directory (`-I` or `/I` flags)
+
+New include directories are added by the `include_dirs` setting:
+
+```
+  {
+    'targets': [
+      {
+        'target_name': 'existing_target',
+        'include_dirs': [
+          '..',
+          'include',
+        ],
+      },
+    ],
+  },
+```
+
+These may be specified directly in a target's settings, as in the above
+example, or in a `conditions` section.
+
+#### Add new compiler flags
+
+Specific compiler flags can be added with the `cflags` setting:
+
+```
+  {
+    'targets': [
+      {
+        'target_name': 'existing_target',
+        'conditions': [
+          ['OS=="win"', {
+            'cflags': [
+              '/WX',
+            ],
+          }, { # OS != "win"
+            'cflags': [
+              '-Werror',
+            ],
+          }],
+        ],
+      },
+    ],
+  },
+```
+
+Because these flags will be specific to the actual compiler involved,
+they will almost always be only set within a `conditions` section.
+
+#### Add new linker flags
+
+Setting linker flags is OS-specific. On linux and most non-mac posix
+systems, they can be added with the `ldflags` setting:
+
+```
+  {
+    'targets': [
+      {
+        'target_name': 'existing_target',
+        'conditions': [
+          ['OS=="linux"', {
+            'ldflags': [
+              '-pthread',
+            ],
+          }],
+        ],
+      },
+    ],
+  },
+```
+
+Because these flags will be specific to the actual linker involved,
+they will almost always be only set within a `conditions` section.
+
+On OS X, linker settings are set via `xcode_settings`, on Windows via
+`msvs_settings`.
+
+#### Exclude settings on a platform
+
+Any given settings keyword (`defines`, `include_dirs`, etc.) has a
+corresponding form with a trailing `!` (exclamation point) to remove
+values from a setting.  One useful example of this is to remove the
+Linux `-Werror` flag from the global settings defined in
+`build/common.gypi`:
+
+```
+  {
+    'targets': [
+      {
+        'target_name': 'third_party_target',
+        'conditions': [
+          ['OS=="linux"', {
+            'cflags!': [
+              '-Werror',
+            ],
+          }],
+        ],
+      },
+    ],
+  },
+```
+
+### Cross-compiling
+
+GYP has some (relatively limited) support for cross-compiling.
+
+If the variable `GYP_CROSSCOMPILE` or one of the toolchain-related
+variables (like `CC_host` or `CC_target`) is set, GYP will think that
+you wish to do a cross-compile.
+
+When cross-compiling, each target can be part of a "host" build, a
+"target" build, or both. By default, the target is assumed to be (only)
+part of the "target" build. The 'toolsets' property can be set on a
+target to change the default.
+
+A target's dependencies are assumed to match the build type (so, if A
+depends on B, by default that means that a target build of A depends on
+a target build of B). You can explicitly depend on targets across
+toolchains by specifying "#host" or "#target" in the dependencies list.
+If GYP is not doing a cross-compile, the "#host" and "#target" will be
+stripped as needed, so nothing breaks.
+
+### Add a new library
+
+TODO:  write intro
+
+#### Add a library that builds on all platforms
+
+Add the a dictionary defining the new library target to the `targets`
+list in the appropriate `.gyp` file.  Example:
+
+```
+  {
+    'targets': [
+      {
+        'target_name': 'new_library',
+        'type': '<(library)',
+        'defines': [
+          'FOO',
+          'BAR=some_value',
+        ],
+        'include_dirs': [
+          '..',
+        ],
+        'dependencies': [
+          'other_target_in_this_file',
+          'other_gyp2:target_in_other_gyp2',
+        ],
+        'direct_dependent_settings': {
+          'include_dirs': '.',
+        },
+        'export_dependent_settings': [
+          'other_target_in_this_file',
+        ],
+        'sources': [
+          'new_additional_source.cc',
+          'new_library.cc',
+        ],
+      },
+    ],
+  }
+```
+
+The use of the `<(library)` variable above should be the default `type`
+setting for most library targets, as it allows the developer to choose,
+at `gyp` time, whether to build with static or shared libraries.
+(Building with shared libraries saves a _lot_ of link time on Linux.)
+
+It may be necessary to build a specific library as a fixed type.  Is so,
+the `type` field can be hard-wired appropriately.  For a static library:
+
+```
+        'type': 'static_library',
+```
+
+For a shared library:
+
+```
+        'type': 'shared_library',
+```
+
+#### Add a platform-specific library
+
+Add a dictionary defining the new library target to the `targets` list
+within a `conditions` block that's a sibling to the top-level `targets`
+list:
+
+```
+  {
+    'targets': [
+    ],
+    'conditions': [
+      ['OS=="win"', {
+        'targets': [
+          {
+            'target_name': 'new_library',
+            'type': '<(library)',
+            'defines': [
+              'FOO',
+              'BAR=some_value',
+            ],
+            'include_dirs': [
+              '..',
+            ],
+            'dependencies': [
+              'other_target_in_this_file',
+              'other_gyp2:target_in_other_gyp2',
+            ],
+            'direct_dependent_settings': {
+              'include_dirs': '.',
+            },
+            'export_dependent_settings': [
+              'other_target_in_this_file',
+            ],
+            'sources': [
+              'new_additional_source.cc',
+              'new_library.cc',
+            ],
+          },
+        ],
+      }],
+    ],
+  }
+```
+
+### Dependencies between targets
+
+GYP provides useful primitives for establishing dependencies between
+targets, which need to be configured in the following situations.
+
+#### Linking with another library target
+
+```
+  {
+    'targets': [
+      {
+        'target_name': 'foo',
+        'dependencies': [
+          'libbar',
+        ],
+      },
+      {
+        'target_name': 'libbar',
+        'type': '<(library)',
+        'sources': [
+        ],
+      },
+    ],
+  }
+```
+
+Note that if the library target is in a different `.gyp` file, you have
+to specify the path to other `.gyp` file, relative to this `.gyp` file's
+directory:
+
+```
+  {
+    'targets': [
+      {
+        'target_name': 'foo',
+        'dependencies': [
+          '../bar/bar.gyp:libbar',
+        ],
+      },
+    ],
+  }
+```
+
+Adding a library often involves updating multiple `.gyp` files, adding
+the target to the approprate `.gyp` file (possibly a newly-added `.gyp`
+file), and updating targets in the other `.gyp` files that depend on
+(link with) the new library.
+
+#### Compiling with necessary flags for a library target dependency
+
+We need to build a library (often a third-party library) with specific
+preprocessor definitions or command-line flags, and need to ensure that
+targets that depend on the library build with the same settings.  This
+situation is handled by a `direct_dependent_settings` block:
+
+```
+  {
+    'targets': [
+      {
+        'target_name': 'foo',
+        'type': 'executable',
+        'dependencies': [
+          'libbar',
+        ],
+      },
+      {
+        'target_name': 'libbar',
+        'type': '<(library)',
+        'defines': [
+          'LOCAL_DEFINE_FOR_LIBBAR',
+          'DEFINE_TO_USE_LIBBAR',
+        ],
+        'include_dirs': [
+          '..',
+          'include/libbar',
+        ],
+        'direct_dependent_settings': {
+          'defines': [
+            'DEFINE_TO_USE_LIBBAR',
+          ],
+          'include_dirs': [
+            'include/libbar',
+          ],
+        },
+      },
+    ],
+  }
+```
+
+In the above example, the sources of the `foo` executable will be
+compiled with the options `-DDEFINE_TO_USE_LIBBAR -Iinclude/libbar`,
+because of those settings' being listed in the
+`direct_dependent_settings` block.
+
+Note that these settings will likely need to be replicated in the
+settings for the library target itsef, so that the library will build
+with the same options.  This does not prevent the target from defining
+additional options for its "internal" use when compiling its own source
+files.  (In the above example, these are the `LOCAL_DEFINE_FOR_LIBBAR`
+define, and the `..` entry in the `include_dirs` list.)
+
+#### When a library depends on an additional library at final link time
+
+```
+  {
+    'targets': [
+      {
+        'target_name': 'foo',
+        'type': 'executable',
+        'dependencies': [
+          'libbar',
+        ],
+      },
+      {
+        'target_name': 'libbar',
+        'type': '<(library)',
+        'dependencies': [
+          'libother'
+        ],
+        'export_dependent_settings': [
+          'libother'
+        ],
+      },
+      {
+        'target_name': 'libother',
+        'type': '<(library)',
+        'direct_dependent_settings': {
+          'defines': [
+            'DEFINE_FOR_LIBOTHER',
+          ],
+          'include_dirs': [
+            'include/libother',
+          ],
+        },
+      },
+    ],
+  }
+```
+
+### Support for Mac OS X bundles
+
+gyp supports building bundles on OS X (.app, .framework, .bundle, etc).
+Here is an example of this:
+
+```
+    {
+      'target_name': 'test_app',
+      'product_name': 'Test App Gyp',
+      'type': 'executable',
+      'mac_bundle': 1,
+      'sources': [
+        'main.m',
+        'TestAppAppDelegate.h',
+        'TestAppAppDelegate.m',
+      ],
+      'mac_bundle_resources': [
+        'TestApp/English.lproj/InfoPlist.strings',
+        'TestApp/English.lproj/MainMenu.xib',
+      ],
+      'link_settings': {
+        'libraries': [
+          '$(SDKROOT)/System/Library/Frameworks/Cocoa.framework',
+        ],
+      },
+      'xcode_settings': {
+        'INFOPLIST_FILE': 'TestApp/TestApp-Info.plist',
+      },
+    },
+```
+
+The `mac_bundle` key tells gyp that this target should be a bundle.
+`executable` targets get extension `.app` by default, `shared_library`
+targets get `.framework` – but you can change the bundle extensions by
+setting `product_extension` if you want. Files listed in
+`mac_bundle_resources` will be copied to the bundle's `Resource` folder
+of the bundle. You can also set
+`process_outputs_as_mac_bundle_resources` to 1 in actions and rules to
+let the output of actions and rules be added to that folder (similar to
+`process_outputs_as_sources`). If `product_name` is not set, the bundle
+will be named after `target_name`as usual.
+
+### Move files (refactoring)
+
+TODO
+
+### Custom build steps
+
+TODO
+
+#### Adding an explicit build step to generate specific files
+
+TODO
+
+#### Adding a rule to handle files with a new suffix
+
+TODO
+
+### Build flavors
+
+TODO
diff --git a/gyp/pylib/gyp/MSVSSettings.py b/gyp/pylib/gyp/MSVSSettings.py
index 93633dbca1..ac87f572b2 100644
--- a/gyp/pylib/gyp/MSVSSettings.py
+++ b/gyp/pylib/gyp/MSVSSettings.py
@@ -793,6 +793,8 @@ def _ValidateSettings(validators, settings, stderr):
     _compile, "CompileAsManaged", _Enumeration([], new=["false", "true"])
 )  # /clr
 _MSBuildOnly(_compile, "CreateHotpatchableImage", _boolean)  # /hotpatch
+_MSBuildOnly(_compile, "LanguageStandard", _string)
+_MSBuildOnly(_compile, "LanguageStandard_C", _string)
 _MSBuildOnly(_compile, "MultiProcessorCompilation", _boolean)  # /MP
 _MSBuildOnly(_compile, "PreprocessOutputPath", _string)  # /Fi
 _MSBuildOnly(_compile, "ProcessorNumber", _integer)  # the number of processors
diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py
index b73a0c55b1..762ae02109 100644
--- a/gyp/pylib/gyp/common.py
+++ b/gyp/pylib/gyp/common.py
@@ -9,6 +9,7 @@
 import tempfile
 import sys
 import subprocess
+import shlex
 
 from collections.abc import MutableSet
 
@@ -422,8 +423,54 @@ def EnsureDirExists(path):
     except OSError:
         pass
 
+def GetCrossCompilerPredefines():  # -> dict
+    cmd = []
+
+    # shlex.split() will eat '\' in posix mode, but
+    # setting posix=False will preserve extra '"' cause CreateProcess fail on Windows
+    # this makes '\' in %CC_target% and %CFLAGS% work
+    def replace_sep(s):
+        return s.replace(os.sep, "/") if os.sep != "/" else s
+
+    if CC := os.environ.get("CC_target") or os.environ.get("CC"):
+        cmd += shlex.split(replace_sep(CC))
+        if CFLAGS := os.environ.get("CFLAGS"):
+            cmd += shlex.split(replace_sep(CFLAGS))
+    elif CXX := os.environ.get("CXX_target") or os.environ.get("CXX"):
+        cmd += shlex.split(replace_sep(CXX))
+        if CXXFLAGS := os.environ.get("CXXFLAGS"):
+            cmd += shlex.split(replace_sep(CXXFLAGS))
+    else:
+        return {}
 
-def GetFlavor(params):
+    if sys.platform == "win32":
+        fd, input = tempfile.mkstemp(suffix=".c")
+        real_cmd = [*cmd, "-dM", "-E", "-x", "c", input]
+        try:
+            os.close(fd)
+            stdout = subprocess.run(
+                real_cmd, shell=True,
+                capture_output=True, check=True
+            ).stdout
+        finally:
+            os.unlink(input)
+    else:
+        input = "/dev/null"
+        real_cmd = [*cmd, "-dM", "-E", "-x", "c", input]
+        stdout = subprocess.run(
+            real_cmd, shell=False,
+            capture_output=True, check=True
+        ).stdout
+
+    defines = {}
+    lines = stdout.decode("utf-8").replace("\r\n", "\n").split("\n")
+    for line in lines:
+        if (line or "").startswith("#define "):
+            _, key, *value = line.split(" ")
+            defines[key] = " ".join(value)
+    return defines
+
+def GetFlavorByPlatform():
     """Returns |params.flavor| if it's set, the system's default flavor else."""
     flavors = {
         "cygwin": "win",
@@ -431,8 +478,6 @@ def GetFlavor(params):
         "darwin": "mac",
     }
 
-    if "flavor" in params:
-        return params["flavor"]
     if sys.platform in flavors:
         return flavors[sys.platform]
     if sys.platform.startswith("sunos"):
@@ -452,6 +497,18 @@ def GetFlavor(params):
 
     return "linux"
 
+def GetFlavor(params):
+    if "flavor" in params:
+        return params["flavor"]
+
+    defines = GetCrossCompilerPredefines()
+    if "__EMSCRIPTEN__" in defines:
+        return "emscripten"
+    if "__wasm__" in defines:
+        return "wasi" if "__wasi__" in defines else "wasm"
+
+    return GetFlavorByPlatform()
+
 
 def CopyTool(flavor, out_path, generator_flags={}):
     """Finds (flock|mac|win)_tool.gyp in the gyp directory and copies it
diff --git a/gyp/pylib/gyp/common_test.py b/gyp/pylib/gyp/common_test.py
index 05344085ad..b6c4cccc1a 100755
--- a/gyp/pylib/gyp/common_test.py
+++ b/gyp/pylib/gyp/common_test.py
@@ -9,7 +9,8 @@
 import gyp.common
 import unittest
 import sys
-
+import os
+from unittest.mock import patch, MagicMock
 
 class TestTopologicallySorted(unittest.TestCase):
     def test_Valid(self):
@@ -24,9 +25,8 @@ def test_Valid(self):
         def GetEdge(node):
             return tuple(graph[node])
 
-        self.assertEqual(
-            gyp.common.TopologicallySorted(graph.keys(), GetEdge), ["a", "c", "d", "b"]
-        )
+        assert gyp.common.TopologicallySorted(
+            graph.keys(), GetEdge) == ["a", "c", "d", "b"]
 
     def test_Cycle(self):
         """Test that an exception is thrown on a cyclic graph."""
@@ -58,7 +58,7 @@ def tearDown(self):
 
     def assertFlavor(self, expected, argument, param):
         sys.platform = argument
-        self.assertEqual(expected, gyp.common.GetFlavor(param))
+        assert expected == gyp.common.GetFlavor(param)
 
     def test_platform_default(self):
         self.assertFlavor("freebsd", "freebsd9", {})
@@ -73,6 +73,99 @@ def test_platform_default(self):
     def test_param(self):
         self.assertFlavor("foobar", "linux2", {"flavor": "foobar"})
 
+    class MockCommunicate:
+        def __init__(self, stdout):
+            self.stdout = stdout
+
+        def decode(self, encoding):
+            return self.stdout
+
+    @patch("os.close")
+    @patch("os.unlink")
+    @patch("tempfile.mkstemp")
+    def test_GetCrossCompilerPredefines(self, mock_mkstemp, mock_unlink, mock_close):
+        mock_close.return_value = None
+        mock_unlink.return_value = None
+        mock_mkstemp.return_value = (0, "temp.c")
+
+        def mock_run(env, defines_stdout, expected_cmd):
+            with patch("subprocess.run") as mock_run:
+                mock_process = MagicMock()
+                mock_process.returncode = 0
+                mock_process.stdout = TestGetFlavor.MockCommunicate(defines_stdout)
+                mock_run.return_value = mock_process
+                expected_input = "temp.c" if sys.platform == "win32" else "/dev/null"
+                with patch.dict(os.environ, env):
+                    defines = gyp.common.GetCrossCompilerPredefines()
+                    flavor = gyp.common.GetFlavor({})
+                if env.get("CC_target"):
+                    mock_run.assert_called_with(
+                        [
+                            *expected_cmd,
+                            "-dM", "-E", "-x", "c", expected_input
+                        ],
+                        shell=sys.platform == "win32",
+                        capture_output=True, check=True)
+                return [defines, flavor]
+
+        [defines1, _] = mock_run({}, "", [])
+        assert {} == defines1
+
+        [defines2, flavor2] = mock_run(
+            { "CC_target": "/opt/wasi-sdk/bin/clang" },
+            "#define __wasm__ 1\n#define __wasi__ 1\n",
+            ["/opt/wasi-sdk/bin/clang"]
+        )
+        assert { "__wasm__": "1", "__wasi__": "1" } == defines2
+        assert flavor2 == "wasi"
+
+        [defines3, flavor3] = mock_run(
+            { "CC_target": "/opt/wasi-sdk/bin/clang --target=wasm32" },
+            "#define __wasm__ 1\n",
+            ["/opt/wasi-sdk/bin/clang", "--target=wasm32"]
+        )
+        assert { "__wasm__": "1" } == defines3
+        assert flavor3 == "wasm"
+
+        [defines4, flavor4] = mock_run(
+            { "CC_target": "/emsdk/upstream/emscripten/emcc" },
+            "#define __EMSCRIPTEN__ 1\n",
+            ["/emsdk/upstream/emscripten/emcc"]
+        )
+        assert { "__EMSCRIPTEN__": "1" } == defines4
+        assert flavor4 == "emscripten"
+
+        # Test path which include white space
+        [defines5, flavor5] = mock_run(
+            {
+                "CC_target": "\"/Users/Toyo Li/wasi-sdk/bin/clang\" -O3",
+                "CFLAGS": "--target=wasm32-wasi-threads -pthread"
+            },
+            "#define __wasm__ 1\n#define __wasi__ 1\n#define _REENTRANT 1\n",
+            [
+                "/Users/Toyo Li/wasi-sdk/bin/clang",
+                "-O3",
+                "--target=wasm32-wasi-threads",
+                "-pthread"
+            ]
+        )
+        assert {
+            "__wasm__": "1",
+            "__wasi__": "1",
+            "_REENTRANT": "1"
+        } == defines5
+        assert flavor5 == "wasi"
+
+        original_sep = os.sep
+        os.sep = "\\"
+        [defines6, flavor6] = mock_run(
+            { "CC_target": "\"C:\\Program Files\\wasi-sdk\\clang.exe\"" },
+            "#define __wasm__ 1\n#define __wasi__ 1\n",
+            ["C:/Program Files/wasi-sdk/clang.exe"]
+        )
+        os.sep = original_sep
+        assert { "__wasm__": "1", "__wasi__": "1" } == defines6
+        assert flavor6 == "wasi"
 
 if __name__ == "__main__":
     unittest.main()
diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py
index 9a79670214..2a63f412db 100644
--- a/gyp/pylib/gyp/generator/android.py
+++ b/gyp/pylib/gyp/generator/android.py
@@ -740,8 +740,8 @@ def ComputeOutput(self, spec):
                 )
             else:
                 path = (
-                    "$(call intermediates-dir-for,"
-                    f"{self.android_class},{self.android_module},,,$(GYP_VAR_PREFIX))"
+                    f"$(call intermediates-dir-for,{self.android_class},"
+                    f"{self.android_module},,,$(GYP_VAR_PREFIX))"
                 )
 
         assert spec.get("product_dir") is None  # TODO: not supported?
diff --git a/gyp/pylib/gyp/generator/compile_commands_json.py b/gyp/pylib/gyp/generator/compile_commands_json.py
index 0ffa3bb598..5d7f14da96 100644
--- a/gyp/pylib/gyp/generator/compile_commands_json.py
+++ b/gyp/pylib/gyp/generator/compile_commands_json.py
@@ -108,10 +108,14 @@ def GenerateOutput(target_list, target_dicts, data, params):
         cwd = os.path.dirname(build_file)
         AddCommandsForTarget(cwd, target, params, per_config_commands)
 
+    output_dir = None
     try:
-        output_dir = params["options"].generator_output
-    except (AttributeError, KeyError):
-        output_dir = params["generator_flags"].get("output_dir", "out")
+        # generator_output can be `None` on Windows machines, or even not
+        # defined in other cases
+        output_dir = params.get("options").generator_output
+    except AttributeError:
+        pass
+    output_dir = output_dir or params["generator_flags"].get("output_dir", "out")
     for configuration_name, commands in per_config_commands.items():
         filename = os.path.join(output_dir, configuration_name, "compile_commands.json")
         gyp.common.EnsureDirExists(filename)
diff --git a/gyp/pylib/gyp/generator/gypsh.py b/gyp/pylib/gyp/generator/gypsh.py
index 625b6d65ca..8dfb1f1645 100644
--- a/gyp/pylib/gyp/generator/gypsh.py
+++ b/gyp/pylib/gyp/generator/gypsh.py
@@ -51,7 +51,7 @@ def GenerateOutput(target_list, target_dicts, data, params):
     # locals are available, and identify gypsh.
     banner = (
         f"Python {sys.version} on {sys.platform}\nlocals.keys() = "
-        f"{repr(sorted(locals.keys()))}\ngypsh"
+        f"{sorted(locals.keys())!r}\ngypsh"
     )
 
     code.interact(banner, local=locals)
diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py
index 1b9974948e..392d900914 100644
--- a/gyp/pylib/gyp/generator/make.py
+++ b/gyp/pylib/gyp/generator/make.py
@@ -25,6 +25,7 @@
 import os
 import re
 import subprocess
+import sys
 import gyp
 import gyp.common
 import gyp.xcode_emulation
@@ -378,7 +379,7 @@ def CalculateGeneratorInputInfo(params):
 CXXFLAGS.target ?= $(CPPFLAGS) $(CXXFLAGS)
 LINK.target ?= %(LINK.target)s
 LDFLAGS.target ?= $(LDFLAGS)
-AR.target ?= $(AR)
+AR.target ?= %(AR.target)s
 PLI.target ?= %(PLI.target)s
 
 # C++ apps need to be linked with g++.
@@ -442,13 +443,21 @@ def CalculateGeneratorInputInfo(params):
 define fixup_dep
 # The depfile may not exist if the input file didn't have any #includes.
 touch $(depfile).raw
-# Fixup path as in (1).
-sed -e "s|^$(notdir $@)|$@|" $(depfile).raw >> $(depfile)
+# Fixup path as in (1).""" +
+    (r"""
+sed -e "s|^$(notdir $@)|$@|" -re 's/\\\\([^$$])/\/\1/g' $(depfile).raw >> $(depfile)"""
+    if sys.platform == 'win32' else r"""
+sed -e "s|^$(notdir $@)|$@|" $(depfile).raw >> $(depfile)""") +
+    r"""
 # Add extra rules as in (2).
 # We remove slashes and replace spaces with new lines;
 # remove blank lines;
-# delete the first line and append a colon to the remaining lines.
-sed -e 's|\\||' -e 'y| |\n|' $(depfile).raw |\
+# delete the first line and append a colon to the remaining lines.""" +
+    ("""
+sed -e 's/\\\\\\\\$$//' -e 's/\\\\\\\\/\\//g' -e 'y| |\\n|' $(depfile).raw |\\"""
+    if sys.platform == 'win32' else """
+sed -e 's|\\\\||' -e 'y| |\\n|' $(depfile).raw |\\""") +
+    r"""
   grep -v '^$$'                             |\
   sed -e 1d -e 's|$$|:|'                     \
     >> $(depfile)
@@ -724,6 +733,10 @@ def QuoteIfNecessary(string):
         string = '"' + string.replace('"', '\\"') + '"'
     return string
 
+def replace_sep(string):
+    if sys.platform == 'win32':
+        string = string.replace('\\\\', '/').replace('\\', '/')
+    return string
 
 def StringToMakefileVariable(string):
     """Convert a string to a value that is acceptable as a make variable name."""
@@ -859,7 +872,7 @@ def Write(
             self.output = self.ComputeMacBundleOutput(spec)
             self.output_binary = self.ComputeMacBundleBinaryOutput(spec)
         else:
-            self.output = self.output_binary = self.ComputeOutput(spec)
+            self.output = self.output_binary = replace_sep(self.ComputeOutput(spec))
 
         self.is_standalone_static_library = bool(
             spec.get("standalone_static_library", 0)
@@ -985,7 +998,7 @@ def WriteSubMake(self, output_filename, makefile_path, targets, build_dir):
         # sub-project dir (see test/subdirectory/gyptest-subdir-all.py).
         self.WriteLn(
             "export builddir_name ?= %s"
-            % os.path.join(os.path.dirname(output_filename), build_dir)
+            % replace_sep(os.path.join(os.path.dirname(output_filename), build_dir))
         )
         self.WriteLn(".PHONY: all")
         self.WriteLn("all:")
@@ -2063,7 +2076,7 @@ def WriteList(self, value_list, variable=None, prefix="", quoter=QuoteIfNecessar
         """
         values = ""
         if value_list:
-            value_list = [quoter(prefix + value) for value in value_list]
+            value_list = [replace_sep(quoter(prefix + value)) for value in value_list]
             values = " \\\n\t" + " \\\n\t".join(value_list)
         self.fp.write(f"{variable} :={values}\n\n")
 
@@ -2369,10 +2382,12 @@ def WriteAutoRegenerationRule(params, root_makefile, makefile_name, build_files)
         "\t$(call do_cmd,regen_makefile)\n\n"
         % {
             "makefile_name": makefile_name,
-            "deps": " ".join(SourceifyAndQuoteSpaces(bf) for bf in build_files),
-            "cmd": gyp.common.EncodePOSIXShellList(
-                [gyp_binary, "-fmake"] + gyp.RegenerateFlags(options) + build_files_args
+            "deps": replace_sep(
+                " ".join(SourceifyAndQuoteSpaces(bf) for bf in build_files)
             ),
+            "cmd": replace_sep(gyp.common.EncodePOSIXShellList(
+                [gyp_binary, "-fmake"] + gyp.RegenerateFlags(options) + build_files_args
+            )),
         }
     )
 
@@ -2435,33 +2450,52 @@ def CalculateMakefilePath(build_file, base_name):
         makefile_path = os.path.join(
             options.toplevel_dir, options.generator_output, makefile_name
         )
-        srcdir = gyp.common.RelativePath(srcdir, options.generator_output)
+        srcdir = replace_sep(gyp.common.RelativePath(srcdir, options.generator_output))
         srcdir_prefix = "$(srcdir)/"
 
     flock_command = "flock"
     copy_archive_arguments = "-af"
     makedep_arguments = "-MMD"
+
+    # wasm-ld doesn't support --start-group/--end-group
+    link_commands = LINK_COMMANDS_LINUX
+    if flavor in ["wasi", "wasm"]:
+        link_commands = link_commands.replace(' -Wl,--start-group', '').replace(
+            ' -Wl,--end-group', ''
+        )
+
+    CC_target = replace_sep(GetEnvironFallback(("CC_target", "CC"), "$(CC)"))
+    AR_target = replace_sep(GetEnvironFallback(("AR_target", "AR"), "$(AR)"))
+    CXX_target = replace_sep(GetEnvironFallback(("CXX_target", "CXX"), "$(CXX)"))
+    LINK_target = replace_sep(GetEnvironFallback(("LINK_target", "LINK"), "$(LINK)"))
+    PLI_target = replace_sep(GetEnvironFallback(("PLI_target", "PLI"), "pli"))
+    CC_host = replace_sep(GetEnvironFallback(("CC_host", "CC"), "gcc"))
+    AR_host = replace_sep(GetEnvironFallback(("AR_host", "AR"), "ar"))
+    CXX_host = replace_sep(GetEnvironFallback(("CXX_host", "CXX"), "g++"))
+    LINK_host = replace_sep(GetEnvironFallback(("LINK_host", "LINK"), "$(CXX.host)"))
+    PLI_host = replace_sep(GetEnvironFallback(("PLI_host", "PLI"), "pli"))
+
     header_params = {
         "default_target": default_target,
         "builddir": builddir_name,
         "default_configuration": default_configuration,
         "flock": flock_command,
         "flock_index": 1,
-        "link_commands": LINK_COMMANDS_LINUX,
+        "link_commands": link_commands,
         "extra_commands": "",
         "srcdir": srcdir,
         "copy_archive_args": copy_archive_arguments,
         "makedep_args": makedep_arguments,
-        "CC.target": GetEnvironFallback(("CC_target", "CC"), "$(CC)"),
-        "AR.target": GetEnvironFallback(("AR_target", "AR"), "$(AR)"),
-        "CXX.target": GetEnvironFallback(("CXX_target", "CXX"), "$(CXX)"),
-        "LINK.target": GetEnvironFallback(("LINK_target", "LINK"), "$(LINK)"),
-        "PLI.target": GetEnvironFallback(("PLI_target", "PLI"), "pli"),
-        "CC.host": GetEnvironFallback(("CC_host", "CC"), "gcc"),
-        "AR.host": GetEnvironFallback(("AR_host", "AR"), "ar"),
-        "CXX.host": GetEnvironFallback(("CXX_host", "CXX"), "g++"),
-        "LINK.host": GetEnvironFallback(("LINK_host", "LINK"), "$(CXX.host)"),
-        "PLI.host": GetEnvironFallback(("PLI_host", "PLI"), "pli"),
+        "CC.target": CC_target,
+        "AR.target": AR_target,
+        "CXX.target": CXX_target,
+        "LINK.target": LINK_target,
+        "PLI.target": PLI_target,
+        "CC.host": CC_host,
+        "AR.host": AR_host,
+        "CXX.host": CXX_host,
+        "LINK.host": LINK_host,
+        "PLI.host": PLI_host,
     }
     if flavor == "mac":
         flock_command = "./gyp-mac-tool flock"
diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py
index 6f0f8c1ab6..6b5b24acc0 100644
--- a/gyp/pylib/gyp/generator/msvs.py
+++ b/gyp/pylib/gyp/generator/msvs.py
@@ -1779,8 +1779,8 @@ def _GetCopies(spec):
                 fixed_dst = _FixPath(dst)
                 full_dst = f'"{fixed_dst}\\{outer_dir}\\"'
                 cmd = (
-                    f'mkdir {full_dst} 2>nul & cd "{_FixPath(base_dir)}" && '
-                    f'xcopy /e /f /y "{outer_dir}" {full_dst}'
+                    f'mkdir {full_dst} 2>nul & cd "{_FixPath(base_dir)}" '
+                    f'&& xcopy /e /f /y "{outer_dir}" {full_dst}'
                 )
                 copies.append(
                     (
@@ -1896,9 +1896,8 @@ def _GetPlatformOverridesOfProject(spec):
     for config_name, c in spec["configurations"].items():
         config_fullname = _ConfigFullName(config_name, c)
         platform = c.get("msvs_target_platform", _ConfigPlatform(c))
-        fixed_config_fullname = (
-            f"{_ConfigBaseName(config_name, _ConfigPlatform(c))}|{platform}"
-        )
+        base_name = _ConfigBaseName(config_name, _ConfigPlatform(c))
+        fixed_config_fullname = f"{base_name}|{platform}"
         if spec["toolset"] == "host" and generator_supports_multiple_toolsets:
             fixed_config_fullname = f"{config_name}|x64"
         config_platform_overrides[config_fullname] = fixed_config_fullname
diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py
index 8ba341e96d..0146c49962 100644
--- a/gyp/pylib/gyp/generator/ninja.py
+++ b/gyp/pylib/gyp/generator/ninja.py
@@ -11,6 +11,7 @@
 import os.path
 import re
 import signal
+import shutil
 import subprocess
 import sys
 import gyp
@@ -2210,6 +2211,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name
     options = params["options"]
     flavor = gyp.common.GetFlavor(params)
     generator_flags = params.get("generator_flags", {})
+    generate_compile_commands = generator_flags.get("compile_commands", False)
 
     # build_dir: relative path from source root to our output files.
     # e.g. "out/Debug"
@@ -2878,6 +2880,35 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name
 
     master_ninja_file.close()
 
+    if generate_compile_commands:
+        compile_db = GenerateCompileDBWithNinja(toplevel_build)
+        compile_db_file = OpenOutput(
+            os.path.join(toplevel_build, "compile_commands.json")
+        )
+        compile_db_file.write(json.dumps(compile_db, indent=2))
+        compile_db_file.close()
+
+
+def GenerateCompileDBWithNinja(path, targets=["all"]):
+    """Generates a compile database using ninja.
+
+    Args:
+        path: The build directory to generate a compile database for.
+        targets: Additional targets to pass to ninja.
+
+    Returns:
+        List of the contents of the compile database.
+    """
+    ninja_path = shutil.which("ninja")
+    if ninja_path is None:
+        raise Exception("ninja not found in PATH")
+    json_compile_db = subprocess.check_output(
+        [ninja_path, "-C", path]
+        + targets
+        + ["-t", "compdb", "cc", "cxx", "objc", "objcxx"]
+    )
+    return json.loads(json_compile_db)
+
 
 def PerformBuild(data, configurations, params):
     options = params["options"]
diff --git a/gyp/pylib/gyp/generator/ninja_test.py b/gyp/pylib/gyp/generator/ninja_test.py
index 7d180685b2..15cddfdf24 100644
--- a/gyp/pylib/gyp/generator/ninja_test.py
+++ b/gyp/pylib/gyp/generator/ninja_test.py
@@ -6,6 +6,7 @@
 
 """ Unit tests for the ninja.py file. """
 
+from pathlib import Path
 import sys
 import unittest
 
@@ -50,6 +51,17 @@ def test_BinaryNamesLinux(self):
             writer.ComputeOutputFileName(spec, "static_library").endswith(".a")
         )
 
+    def test_GenerateCompileDBWithNinja(self):
+        build_dir = (
+            Path(__file__).resolve().parent.parent.parent.parent / "data" / "ninja"
+        )
+        compile_db = ninja.GenerateCompileDBWithNinja(build_dir)
+        assert len(compile_db) == 1
+        assert compile_db[0]["directory"] == str(build_dir)
+        assert compile_db[0]["command"] == "cc my.in my.out"
+        assert compile_db[0]["file"] == "my.in"
+        assert compile_db[0]["output"] == "my.out"
+
 
 if __name__ == "__main__":
     unittest.main()
diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py
index 0b56c72750..7150269cda 100644
--- a/gyp/pylib/gyp/input.py
+++ b/gyp/pylib/gyp/input.py
@@ -1135,16 +1135,16 @@ def EvalCondition(condition, conditions_key, phase, variables, build_file):
         true_dict = condition[i + 1]
         if type(true_dict) is not dict:
             raise GypError(
-                f"{conditions_key} {cond_expr} must be followed by a dictionary, not "
-                f"{type(true_dict)}"
+                f"{conditions_key} {cond_expr} must be followed by a dictionary, "
+                f"not {type(true_dict)}"
             )
         if len(condition) > i + 2 and type(condition[i + 2]) is dict:
             false_dict = condition[i + 2]
             i = i + 3
             if i != len(condition):
                 raise GypError(
-                    f"{conditions_key} {cond_expr} has {len(condition) - i} "
-                    "unexpected trailing items"
+                    f"{conditions_key} {cond_expr} has "
+                    f"{len(condition) - i} unexpected trailing items"
                 )
         else:
             false_dict = None
@@ -2536,6 +2536,8 @@ def ProcessListFiltersInDict(name, the_dict):
     lists = []
     del_lists = []
     for key, value in the_dict.items():
+        if not key:
+            continue
         operation = key[-1]
         if operation not in {"!", "/"}:
             continue
diff --git a/gyp/pylib/gyp/msvs_emulation.py b/gyp/pylib/gyp/msvs_emulation.py
index 847d1b8dc1..adda5a0273 100644
--- a/gyp/pylib/gyp/msvs_emulation.py
+++ b/gyp/pylib/gyp/msvs_emulation.py
@@ -830,14 +830,15 @@ def _GetLdManifestFlags(
                 ("VCLinkerTool", "UACUIAccess"), config, default="false"
             )
 
+            level = execution_level_map[execution_level]
             inner = f"""
 
   
     
-      
+      
     
   
-"""  # noqa: E501
+"""
         else:
             inner = ""
 
diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py
index 29caf1ce7f..5f2c097f63 100644
--- a/gyp/pylib/gyp/xcode_emulation.py
+++ b/gyp/pylib/gyp/xcode_emulation.py
@@ -579,7 +579,8 @@ def GetCflags(self, configname, arch=None):
 
         sdk_root = self._SdkPath()
         if "SDKROOT" in self._Settings() and sdk_root:
-            cflags.append("-isysroot %s" % sdk_root)
+            cflags.append("-isysroot")
+            cflags.append(sdk_root)
 
         if self.header_map_path:
             cflags.append("-I%s" % self.header_map_path)
@@ -664,7 +665,8 @@ def GetCflags(self, configname, arch=None):
                 # TODO: Supporting fat binaries will be annoying.
                 self._WarnUnimplemented("ARCHS")
                 archs = ["i386"]
-            cflags.append("-arch " + archs[0])
+            cflags.append("-arch")
+            cflags.append(archs[0])
 
             if archs[0] in ("i386", "x86_64"):
                 if self._Test("GCC_ENABLE_SSE3_EXTENSIONS", "YES", default="NO"):
@@ -924,17 +926,15 @@ def GetLdflags(self, configname, product_dir, gyp_to_build_path, arch=None):
         self._AppendPlatformVersionMinFlags(ldflags)
 
         if "SDKROOT" in self._Settings() and self._SdkPath():
-            ldflags.append("-isysroot " + self._SdkPath())
+            ldflags.append("-isysroot")
+            ldflags.append(self._SdkPath())
 
         for library_path in self._Settings().get("LIBRARY_SEARCH_PATHS", []):
             ldflags.append("-L" + gyp_to_build_path(library_path))
 
         if "ORDER_FILE" in self._Settings():
-            ldflags.append(
-                "-Wl,-order_file "
-                + "-Wl,"
-                + gyp_to_build_path(self._Settings()["ORDER_FILE"])
-            )
+            ldflags.append("-Wl,-order_file")
+            ldflags.append("-Wl," + gyp_to_build_path(self._Settings()["ORDER_FILE"]))
 
         if not gyp.common.CrossCompileRequested():
             if arch is not None:
@@ -946,7 +946,9 @@ def GetLdflags(self, configname, product_dir, gyp_to_build_path, arch=None):
                 # TODO: Supporting fat binaries will be annoying.
                 self._WarnUnimplemented("ARCHS")
                 archs = ["i386"]
-            ldflags.append("-arch " + archs[0])
+            # Avoid quoting the space between -arch and the arch name
+            ldflags.append("-arch")
+            ldflags.append(archs[0])
 
         # Xcode adds the product directory by default.
         # Rewrite -L. to -L./ to work around http://www.openradar.me/25313838
@@ -954,7 +956,8 @@ def GetLdflags(self, configname, product_dir, gyp_to_build_path, arch=None):
 
         install_name = self.GetInstallName()
         if install_name and self.spec["type"] != "loadable_module":
-            ldflags.append("-install_name " + install_name.replace(" ", r"\ "))
+            ldflags.append("-install_name")
+            ldflags.append(install_name.replace(" ", r"\ "))
 
         for rpath in self._Settings().get("LD_RUNPATH_SEARCH_PATHS", []):
             ldflags.append("-Wl,-rpath," + rpath)
@@ -971,7 +974,8 @@ def GetLdflags(self, configname, product_dir, gyp_to_build_path, arch=None):
             platform_root = self._XcodePlatformPath(configname)
             if sdk_root and platform_root:
                 ldflags.append("-F" + platform_root + "/Developer/Library/Frameworks/")
-                ldflags.append("-framework XCTest")
+                ldflags.append("-framework")
+                ldflags.append("XCTest")
 
         is_extension = self._IsIosAppExtension() or self._IsIosWatchKitExtension()
         if sdk_root and is_extension:
@@ -987,7 +991,8 @@ def GetLdflags(self, configname, product_dir, gyp_to_build_path, arch=None):
                     + "/System/Library/PrivateFrameworks/PlugInKit.framework/PlugInKit"
                 )
             else:
-                ldflags.append("-e _NSExtensionMain")
+                ldflags.append("-e")
+                ldflags.append("_NSExtensionMain")
             ldflags.append("-fapplication-extension")
 
         self._Appendf(ldflags, "CLANG_CXX_LIBRARY", "-stdlib=%s")
diff --git a/gyp/pylib/gyp/xcode_emulation_test.py b/gyp/pylib/gyp/xcode_emulation_test.py
new file mode 100644
index 0000000000..98b02320d5
--- /dev/null
+++ b/gyp/pylib/gyp/xcode_emulation_test.py
@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+
+"""Unit tests for the xcode_emulation.py file."""
+
+from gyp.xcode_emulation import XcodeSettings
+import sys
+import unittest
+
+
+class TestXcodeSettings(unittest.TestCase):
+    def setUp(self):
+        if sys.platform != "darwin":
+            self.skipTest("This test only runs on macOS")
+
+    def test_GetCflags(self):
+        target = {
+            "type": "static_library",
+            "configurations": {
+                "Release": {},
+            },
+        }
+        configuration_name = "Release"
+        xcode_settings = XcodeSettings(target)
+        cflags = xcode_settings.GetCflags(configuration_name, "arm64")
+
+        # Do not quote `-arch arm64` with spaces in one string.
+        self.assertEqual(
+            cflags,
+            ["-fasm-blocks", "-mpascal-strings", "-Os", "-gdwarf-2", "-arch", "arm64"],
+        )
+
+    def GypToBuildPath(self, path):
+        return path
+
+    def test_GetLdflags(self):
+        target = {
+            "type": "static_library",
+            "configurations": {
+                "Release": {},
+            },
+        }
+        configuration_name = "Release"
+        xcode_settings = XcodeSettings(target)
+        ldflags = xcode_settings.GetLdflags(
+            configuration_name, "PRODUCT_DIR", self.GypToBuildPath, "arm64"
+        )
+
+        # Do not quote `-arch arm64` with spaces in one string.
+        self.assertEqual(ldflags, ["-arch", "arm64", "-LPRODUCT_DIR"])
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/gyp/pyproject.toml b/gyp/pyproject.toml
index 7183e07d3c..def9858e44 100644
--- a/gyp/pyproject.toml
+++ b/gyp/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "gyp-next"
-version = "0.16.1"
+version = "0.18.1"
 authors = [
   { name="Node.js contributors", email="ryzokuken@disroot.org" },
 ]
@@ -12,8 +12,7 @@ description = "A fork of the GYP build system for use in the Node.js projects"
 readme = "README.md"
 license = { file="LICENSE" }
 requires-python = ">=3.8"
-# The Python module "packaging" is vendored in the "pylib/packaging" directory to support Python >= 3.12.
-# dependencies = ["packaging>=23.1"]  # Uncomment this line if the vendored version is removed.
+dependencies = ["packaging>=24.0", "setuptools>=69.5.1"]
 classifiers = [
     "Development Status :: 3 - Alpha",
     "Environment :: Console",
@@ -29,7 +28,7 @@ classifiers = [
 ]
 
 [project.optional-dependencies]
-dev = ["flake8", "ruff", "pytest"]
+dev = ["pytest", "ruff"]
 
 [project.scripts]
 gyp = "gyp:script_main"
@@ -38,7 +37,12 @@ gyp = "gyp:script_main"
 "Homepage" = "https://github.com/nodejs/gyp-next"
 
 [tool.ruff]
-lint.select = [
+extend-exclude = ["pylib/packaging"]
+line-length = 88
+target-version = "py37"
+
+[tool.ruff.lint]
+select = [
   "C4",   # flake8-comprehensions
   "C90",  # McCabe cyclomatic complexity
   "DTZ",  # flake8-datetimez
@@ -87,7 +91,7 @@ lint.select = [
   # "T20",  # flake8-print
   # "TRY",  # tryceratops
 ]
-lint.ignore = [
+ignore = [
   "E721",
   "PLC1901",
   "PLR0402",
@@ -101,9 +105,6 @@ lint.ignore = [
   "RUF012",
   "UP031",
 ]
-extend-exclude = ["pylib/packaging"]
-line-length = 88
-target-version = "py37"
 
 [tool.ruff.lint.mccabe]
 max-complexity = 101
diff --git a/gyp/release-please-config.json b/gyp/release-please-config.json
new file mode 100644
index 0000000000..b6cad32a2d
--- /dev/null
+++ b/gyp/release-please-config.json
@@ -0,0 +1,11 @@
+{
+    "last-release-sha": "78756421b0d7bb335992a9c7d26ba3cc8b619708",
+    "packages": {
+        ".": {
+          "release-type": "python",
+          "package-name": "gyp-next",
+          "bump-minor-pre-major": true,
+          "include-component-in-tag": false
+        }
+    }
+}

From 6318d2b210224415ff5932c2863e6cc14d4583dc Mon Sep 17 00:00:00 2001
From: Toyo Li 
Date: Fri, 28 Jun 2024 22:36:30 +0800
Subject: [PATCH 435/551] feat: support `rebuild` and `build` for
 cross-compiling Node-API module to wasm on Windows (#2974)

---
 .github/workflows/tests.yml               |   4 +-
 lib/build.js                              |  33 ++++---
 lib/configure.js                          |  24 ++++-
 test/node_modules/hello_napi/binding.gyp  |   8 ++
 test/node_modules/hello_napi/common.gypi  | 110 ++++++++++++++++++++++
 test/node_modules/hello_napi/hello.c      |  54 +++++++++++
 test/node_modules/hello_napi/hello.js     |  57 +++++++++++
 test/node_modules/hello_napi/package.json |  11 +++
 test/test-windows-make.js                 | 108 +++++++++++++++++++++
 9 files changed, 392 insertions(+), 17 deletions(-)
 create mode 100644 test/node_modules/hello_napi/binding.gyp
 create mode 100644 test/node_modules/hello_napi/common.gypi
 create mode 100644 test/node_modules/hello_napi/hello.c
 create mode 100644 test/node_modules/hello_napi/hello.js
 create mode 100644 test/node_modules/hello_napi/package.json
 create mode 100644 test/test-windows-make.js

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 1e8e8bc8cb..96b08ae3e1 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -135,7 +135,7 @@ jobs:
           FULL_TEST: ${{ (matrix.node == '20.x' && matrix.python == '3.12') && '1' || '0' }}
       - name: Run Tests (Windows)
         if: startsWith(matrix.os, 'windows')
-        shell: pwsh
-        run: npm run test --python="${env:pythonLocation}\\python.exe"
+        shell: bash # Building wasm on Windows requires using make generator, it only works in bash
+        run: npm run test --python="${pythonLocation}\\python.exe"
         env:
           FULL_TEST: ${{ (matrix.node == '20.x' && matrix.python == '3.12') && '1' || '0' }}
diff --git a/lib/build.js b/lib/build.js
index 6b8d84d3ed..e1f49bb6ff 100644
--- a/lib/build.js
+++ b/lib/build.js
@@ -1,6 +1,7 @@
 'use strict'
 
-const fs = require('graceful-fs').promises
+const gracefulFs = require('graceful-fs')
+const fs = gracefulFs.promises
 const path = require('path')
 const { glob } = require('glob')
 const log = require('./log')
@@ -85,59 +86,65 @@ async function build (gyp, argv) {
   async function findSolutionFile () {
     const files = await glob('build/*.sln')
     if (files.length === 0) {
-      throw new Error('Could not find *.sln file. Did you run "configure"?')
+      if (gracefulFs.existsSync('build/Makefile') || (await glob('build/*.mk')).length !== 0) {
+        command = makeCommand
+        await doWhich(false)
+        return
+      } else {
+        throw new Error('Could not find *.sln file or Makefile. Did you run "configure"?')
+      }
     }
     guessedSolution = files[0]
     log.verbose('found first Solution file', guessedSolution)
-    await doWhich()
+    await doWhich(true)
   }
 
   /**
    * Uses node-which to locate the msbuild / make executable.
    */
 
-  async function doWhich () {
+  async function doWhich (msvs) {
     // On Windows use msbuild provided by node-gyp configure
-    if (win) {
+    if (msvs) {
       if (!config.variables.msbuild_path) {
         throw new Error('MSBuild is not set, please run `node-gyp configure`.')
       }
       command = config.variables.msbuild_path
       log.verbose('using MSBuild:', command)
-      await doBuild()
+      await doBuild(msvs)
       return
     }
 
     // First make sure we have the build command in the PATH
     const execPath = await which(command)
     log.verbose('`which` succeeded for `' + command + '`', execPath)
-    await doBuild()
+    await doBuild(msvs)
   }
 
   /**
    * Actually spawn the process and compile the module.
    */
 
-  async function doBuild () {
+  async function doBuild (msvs) {
     // Enable Verbose build
     const verbose = log.logger.isVisible('verbose')
     let j
 
-    if (!win && verbose) {
+    if (!msvs && verbose) {
       argv.push('V=1')
     }
 
-    if (win && !verbose) {
+    if (msvs && !verbose) {
       argv.push('/clp:Verbosity=minimal')
     }
 
-    if (win) {
+    if (msvs) {
       // Turn off the Microsoft logo on Windows
       argv.push('/nologo')
     }
 
     // Specify the build type, Release by default
-    if (win) {
+    if (msvs) {
       // Convert .gypi config target_arch to MSBuild /Platform
       // Since there are many ways to state '32-bit Intel', default to it.
       // N.B. msbuild's Condition string equality tests are case-insensitive.
@@ -173,7 +180,7 @@ async function build (gyp, argv) {
       }
     }
 
-    if (win) {
+    if (msvs) {
       // did the user specify their own .sln file?
       const hasSln = argv.some(function (arg) {
         return path.extname(arg) === '.sln'
diff --git a/lib/configure.js b/lib/configure.js
index e4b8c94e3d..ee672cfbf2 100644
--- a/lib/configure.js
+++ b/lib/configure.js
@@ -92,8 +92,28 @@ async function configure (gyp, argv) {
     log.verbose(
       'build dir', '"build" dir needed to be created?', isNew ? 'Yes' : 'No'
     )
-    const vsInfo = win ? await findVisualStudio(release.semver, gyp.opts['msvs-version']) : null
-    return createConfigFile(vsInfo)
+    if (win) {
+      let usingMakeGenerator = false
+      for (let i = argv.length - 1; i >= 0; --i) {
+        const arg = argv[i]
+        if (arg === '-f' || arg === '--format') {
+          const format = argv[i + 1]
+          if (typeof format === 'string' && format.startsWith('make')) {
+            usingMakeGenerator = true
+            break
+          }
+        } else if (arg.startsWith('--format=make')) {
+          usingMakeGenerator = true
+          break
+        }
+      }
+      let vsInfo = {}
+      if (!usingMakeGenerator) {
+        vsInfo = await findVisualStudio(release.semver, gyp.opts['msvs-version'])
+      }
+      return createConfigFile(vsInfo)
+    }
+    return createConfigFile(null)
   }
 
   async function createConfigFile (vsInfo) {
diff --git a/test/node_modules/hello_napi/binding.gyp b/test/node_modules/hello_napi/binding.gyp
new file mode 100644
index 0000000000..cb6a3bdc22
--- /dev/null
+++ b/test/node_modules/hello_napi/binding.gyp
@@ -0,0 +1,8 @@
+{
+  "targets": [
+    {
+      "target_name": "hello",
+      "sources": [ "hello.c" ],
+    }
+  ]
+}
diff --git a/test/node_modules/hello_napi/common.gypi b/test/node_modules/hello_napi/common.gypi
new file mode 100644
index 0000000000..f3c53bbad9
--- /dev/null
+++ b/test/node_modules/hello_napi/common.gypi
@@ -0,0 +1,110 @@
+{
+  'variables': {
+    # OS: 'emscripten' | 'wasi' | 'unknown' | 'wasm'
+    'clang': 1,
+    'target_arch%': 'wasm32',
+    'stack_size%': 1048576,
+    'initial_memory%': 16777216,
+    'max_memory%': 2147483648,
+  },
+
+  'target_defaults': {
+    'type': 'executable',
+
+    'defines': [
+      'BUILDING_NODE_EXTENSION',
+      '__STDC_FORMAT_MACROS',
+    ],
+
+    'cflags': [
+      '-Wall',
+      '-Wextra',
+      '-Wno-unused-parameter',
+      '--target=wasm32-unknown-unknown',
+    ],
+    'cflags_cc': [
+      '-fno-rtti',
+      '-fno-exceptions',
+      '-std=c++17'
+    ],
+    'ldflags': [
+      '--target=wasm32-unknown-unknown',
+    ],
+
+    'xcode_settings': {
+      # WARNING_CFLAGS == cflags
+      # OTHER_CFLAGS == cflags_c
+      # OTHER_CPLUSPLUSFLAGS == cflags_cc
+      # OTHER_LDFLAGS == ldflags
+
+      'CLANG_CXX_LANGUAGE_STANDARD': 'c++17',
+      'GCC_ENABLE_CPP_RTTI': 'NO',
+      'GCC_ENABLE_CPP_EXCEPTIONS': 'NO',
+      'WARNING_CFLAGS': [
+        '-Wall',
+        '-Wextra',
+        '-Wno-unused-parameter',
+        '--target=wasm32-unknown-unknown'
+      ],
+      'OTHER_LDFLAGS': [ '--target=wasm32-unknown-unknown' ],
+    },
+
+    'default_configuration': 'Release',
+    'configurations': {
+      'Debug': {
+        'defines': [ 'DEBUG', '_DEBUG' ],
+        'cflags': [ '-g', '-O0' ],
+        'ldflags': [ '-g', '-O0' ],
+        'xcode_settings': {
+          'WARNING_CFLAGS': [ '-g', '-O0' ],
+          'OTHER_LDFLAGS': [ '-g', '-O0' ],
+        },
+      },
+      'Release': {
+        'cflags': [ '-O3' ],
+        'ldflags': [ '-O3', '-Wl,--strip-debug' ],
+        'xcode_settings': {
+          'WARNING_CFLAGS': [ '-O3' ],
+          'OTHER_LDFLAGS': [ '-O3', '-Wl,--strip-debug' ],
+        },
+      }
+    },
+
+    'target_conditions': [
+      ['_type=="executable"', {
+
+        'product_extension': 'wasm',
+
+        'ldflags': [
+          '-Wl,--export-dynamic',
+          '-Wl,--export=napi_register_wasm_v1',
+          '-Wl,--export-if-defined=node_api_module_get_api_version_v1',
+          '-Wl,--import-undefined',
+          '-Wl,--export-table',
+          '-Wl,-zstack-size=<(stack_size)',
+          '-Wl,--initial-memory=<(initial_memory)',
+          '-Wl,--max-memory=<(max_memory)',
+          '-nostdlib',
+          '-Wl,--no-entry',
+        ],
+        'xcode_settings': {
+          'OTHER_LDFLAGS': [
+            '-Wl,--export-dynamic',
+            '-Wl,--export=napi_register_wasm_v1',
+            '-Wl,--export-if-defined=node_api_module_get_api_version_v1',
+            '-Wl,--import-undefined',
+            '-Wl,--export-table',
+            '-Wl,-zstack-size=<(stack_size)',
+            '-Wl,--initial-memory=<(initial_memory)',
+            '-Wl,--max-memory=<(max_memory)',
+            '-nostdlib',
+            '-Wl,--no-entry',
+          ],
+        },
+        'defines': [
+          'PAGESIZE=65536'
+        ],
+      }],
+    ],
+  }
+}
diff --git a/test/node_modules/hello_napi/hello.c b/test/node_modules/hello_napi/hello.c
new file mode 100644
index 0000000000..2c03dc156f
--- /dev/null
+++ b/test/node_modules/hello_napi/hello.c
@@ -0,0 +1,54 @@
+#include 
+
+#if !defined(__wasm__) || (defined(__EMSCRIPTEN__) || defined(__wasi__))
+#include 
+#include 
+#else
+#define assert(x) do { if (!(x)) { __builtin_trap(); } } while (0)
+
+
+__attribute__((__import_module__("napi")))
+int napi_create_string_utf8(void* env,
+                            const char* str,
+                            size_t length,
+                            void** result);
+
+__attribute__((__import_module__("napi")))
+int napi_create_function(void* env,
+                        const char* utf8name,
+                        size_t length,
+                        void* cb,
+                        void* data,
+                        void** result);
+
+__attribute__((__import_module__("napi")))
+int napi_set_named_property(void* env,
+                            void* object,
+                            const char* utf8name,
+                            void* value);
+#ifdef __cplusplus
+#define EXTERN_C extern "C" {
+#else
+#define EXTERN_C
+#endif
+#define NAPI_MODULE_INIT() \
+  EXTERN_C __attribute__((visibility("default"))) void* napi_register_wasm_v1(void* env, void* exports)
+
+typedef void* napi_env;
+typedef void* napi_value;
+typedef void* napi_callback_info;
+#endif
+
+static napi_value hello(napi_env env, napi_callback_info info) {
+  napi_value greeting = NULL;
+  assert(0 == napi_create_string_utf8(env, "world", -1, &greeting));
+  return greeting;
+}
+
+NAPI_MODULE_INIT() {
+  napi_value hello_function = NULL;
+  assert(0 == napi_create_function(env, "hello", -1,
+                                         hello, NULL, &hello_function));
+  assert(0 == napi_set_named_property(env, exports, "hello", hello_function));
+  return exports;
+}
diff --git a/test/node_modules/hello_napi/hello.js b/test/node_modules/hello_napi/hello.js
new file mode 100644
index 0000000000..00084f851b
--- /dev/null
+++ b/test/node_modules/hello_napi/hello.js
@@ -0,0 +1,57 @@
+const path = require('path')
+const fs = require('fs')
+
+const addon = (function () {
+  const entry = (() => {
+    try {
+      return require.resolve('./build/Release/hello.node')
+    } catch (_) {
+      return require.resolve('./build/Release/hello.wasm')
+    }
+  })()
+  
+  const ext = path.extname(entry)
+  if (ext === '.node') {
+    return require(entry)
+  }
+
+  if (ext === '.wasm') {
+    const values = [undefined, undefined, null, false, true, global, {}]
+    const module = new WebAssembly.Module(fs.readFileSync(entry))
+    const instance = new WebAssembly.Instance(module, {
+      napi: {
+        napi_create_string_utf8: (env, str, len, ret) => {
+          let end = str
+          const buffer = new Uint8Array(instance.exports.memory.buffer)
+          while (buffer[end]) end++
+          values.push(new TextDecoder().decode(buffer.slice(str, end)))
+          new DataView(instance.exports.memory.buffer).setInt32(ret, values.length - 1, true)
+          return 0
+        },
+        napi_create_function: (env, name, len, fn, data, ret) => {
+          values.push(function () {
+            return values[instance.exports.__indirect_function_table.get(fn)(env, 0)]
+          })
+          new DataView(instance.exports.memory.buffer).setInt32(ret, values.length - 1, true)
+          return 0
+        },
+        napi_set_named_property: (env, obj, key, val) => {
+          const buffer = new Uint8Array(instance.exports.memory.buffer)
+          let end = key
+          while (buffer[end]) end++
+          const k = new TextDecoder().decode(buffer.slice(key, end))
+          values[obj][k] = values[val]
+          return 0
+        }
+      }
+    })
+    const newExports = values[instance.exports.napi_register_wasm_v1(1, 6)]
+    if (newExports) {
+      values[6] = newExports
+    }
+    return values[6]
+  }
+  throw new Error('Failed to initialize Node-API wasm module')
+})()
+
+exports.hello = function() { return addon.hello() }
diff --git a/test/node_modules/hello_napi/package.json b/test/node_modules/hello_napi/package.json
new file mode 100644
index 0000000000..6cdb71b21f
--- /dev/null
+++ b/test/node_modules/hello_napi/package.json
@@ -0,0 +1,11 @@
+{
+  "name": "hello_napi",
+  "version": "0.0.0",
+  "description": "Node.js Addons Example #2",
+  "main": "hello.js",
+  "private": true,
+  "scripts": {
+    "test": "node hello.js"
+  },
+  "gypfile": true
+}
diff --git a/test/test-windows-make.js b/test/test-windows-make.js
new file mode 100644
index 0000000000..41cefc3035
--- /dev/null
+++ b/test/test-windows-make.js
@@ -0,0 +1,108 @@
+'use strict'
+
+const { describe, it } = require('mocha')
+const assert = require('assert')
+const path = require('path')
+const gracefulFs = require('graceful-fs')
+const cp = require('child_process')
+const util = require('../lib/util')
+const { platformTimeout } = require('./common')
+
+const addonPath = path.resolve(__dirname, 'node_modules', 'hello_napi')
+const nodeGyp = path.resolve(__dirname, '..', 'bin', 'node-gyp.js')
+
+const execFileSync = (...args) => cp.execFileSync(...args).toString().trim()
+
+const execFile = async (cmd, env) => {
+  const [err,, stderr] = await util.execFile(process.execPath, cmd, {
+    env: {
+      ...process.env,
+      NODE_GYP_NULL_LOGGER: undefined,
+      ...env
+    },
+    encoding: 'utf-8'
+  })
+  return [err, stderr.toString().trim().split(/\r?\n/)]
+}
+
+function runHello (hostProcess = process.execPath) {
+  const testCode = "console.log(require('hello_napi').hello())"
+  return execFileSync(hostProcess, ['--experimental-wasi-unstable-preview1', '-e', testCode], { cwd: __dirname })
+}
+
+function executable (name) {
+  return name + (process.platform === 'win32' ? '.exe' : '')
+}
+
+function getEnv (target) {
+  const env = {
+    GYP_CROSSCOMPILE: '1',
+    AR_host: 'ar',
+    CC_host: 'clang',
+    CXX_host: 'clang++'
+  }
+  if (target === 'emscripten') {
+    env.AR_target = 'emar'
+    env.CC_target = 'emcc'
+    env.CXX_target = 'em++'
+  } else if (target === 'wasi') {
+    env.AR_target = path.resolve(__dirname, '..', process.env.WASI_SDK_PATH, 'bin', executable('ar'))
+    env.CC_target = path.resolve(__dirname, '..', process.env.WASI_SDK_PATH, 'bin', executable('clang'))
+    env.CXX_target = path.resolve(__dirname, '..', process.env.WASI_SDK_PATH, 'bin', executable('clang++'))
+  } else if (target === 'wasm') {
+    env.AR_target = path.resolve(__dirname, '..', process.env.WASI_SDK_PATH, 'bin', executable('ar'))
+    env.CC_target = path.resolve(__dirname, '..', process.env.WASI_SDK_PATH, 'bin', executable('clang'))
+    env.CXX_target = path.resolve(__dirname, '..', process.env.WASI_SDK_PATH, 'bin', executable('clang++'))
+    env.CFLAGS = '--target=wasm32'
+  } else if (target === 'win-clang') {
+    let vsdir = 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise'
+    if (!gracefulFs.existsSync(vsdir)) {
+      vsdir = 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community'
+    }
+    const llvmBin = 'VC\\Tools\\Llvm\\x64\\bin'
+    env.AR_target = path.join(vsdir, llvmBin, 'llvm-ar.exe')
+    env.CC_target = path.join(vsdir, llvmBin, 'clang.exe')
+    env.CXX_target = path.join(vsdir, llvmBin, 'clang++.exe')
+    env.CFLAGS = '--target=wasm32'
+  }
+  return env
+}
+
+function quote (path) {
+  if (path.includes(' ')) {
+    return `"${path}"`
+  }
+}
+
+describe('windows-cross-compile', function () {
+  it('build simple node-api addon', async function () {
+    if (process.platform !== 'win32') {
+      return this.skip('This test is only for windows')
+    }
+    const env = getEnv('win-clang')
+    if (!gracefulFs.existsSync(env.CC_target)) {
+      return this.skip('Visual Studio Clang is not installed')
+    }
+
+    // handle bash whitespace
+    env.AR_target = quote(env.AR_target)
+    env.CC_target = quote(env.CC_target)
+    env.CXX_target = quote(env.CXX_target)
+    this.timeout(platformTimeout(1, { win32: 5 }))
+
+    const cmd = [
+      nodeGyp,
+      'rebuild',
+      '-C', addonPath,
+      '--loglevel=verbose',
+      `--nodedir=${addonPath}`,
+      '--arch=wasm32',
+      '--', '-f', 'make'
+    ]
+    const [err, logLines] = await execFile(cmd, env)
+    const lastLine = logLines[logLines.length - 1]
+    assert.strictEqual(err, null)
+    assert.strictEqual(lastLine, 'gyp info ok', 'should end in ok')
+    assert.strictEqual(runHello(), 'world')
+  })
+})

From 10f6730be660e7a38be8a12111937e37fcf74834 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Fri, 28 Jun 2024 15:40:58 +0100
Subject: [PATCH 436/551] build(deps): bump seanmiddleditch/gha-setup-ninja
 from 4 to 5 (#3041)

Bumps [seanmiddleditch/gha-setup-ninja](https://github.com/seanmiddleditch/gha-setup-ninja) from 4 to 5.
- [Release notes](https://github.com/seanmiddleditch/gha-setup-ninja/releases)
- [Commits](https://github.com/seanmiddleditch/gha-setup-ninja/compare/v4...v5)

---
updated-dependencies:
- dependency-name: seanmiddleditch/gha-setup-ninja
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 .github/workflows/tests.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 96b08ae3e1..d2746c1dc5 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -115,7 +115,7 @@ jobs:
           python-version: ${{ matrix.python }}
         env:
           PYTHON_VERSION: ${{ matrix.python }}  # Why do this?
-      - uses: seanmiddleditch/gha-setup-ninja@v4
+      - uses: seanmiddleditch/gha-setup-ninja@v5
       - name: Install Dependencies
         run: |
           npm install

From d38af2e0c2a81b12cd221b1f8517fb89e609d62c Mon Sep 17 00:00:00 2001
From: Chengzhong Wu 
Date: Tue, 9 Jul 2024 23:13:40 +0800
Subject: [PATCH 437/551] feat: allow VCINSTALLDIR to specify a portable
 instance (#3036)

---
 lib/find-visualstudio.js       | 46 +++++++++++++++++++++++++++++++++-
 test/test-find-visualstudio.js | 25 ++++++++++++++++++
 2 files changed, 70 insertions(+), 1 deletion(-)

diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js
index a78e763480..2dc1930fd7 100644
--- a/lib/find-visualstudio.js
+++ b/lib/find-visualstudio.js
@@ -54,8 +54,10 @@ class VisualStudioFinder {
     }
 
     const checks = [
+      () => this.findVisualStudio2019OrNewerFromSpecifiedLocation(),
       () => this.findVisualStudio2019OrNewerUsingSetupModule(),
       () => this.findVisualStudio2019OrNewer(),
+      () => this.findVisualStudio2017FromSpecifiedLocation(),
       () => this.findVisualStudio2017UsingSetupModule(),
       () => this.findVisualStudio2017(),
       () => this.findVisualStudio2015(),
@@ -116,6 +118,48 @@ class VisualStudioFinder {
     throw new Error('Could not find any Visual Studio installation to use')
   }
 
+  async findVisualStudio2019OrNewerFromSpecifiedLocation () {
+    return this.findVSFromSpecifiedLocation([2019, 2022])
+  }
+
+  async findVisualStudio2017FromSpecifiedLocation () {
+    if (this.nodeSemver.major >= 22) {
+      this.addLog(
+        'not looking for VS2017 as it is only supported up to Node.js 21')
+      return null
+    }
+    return this.findVSFromSpecifiedLocation([2017])
+  }
+
+  async findVSFromSpecifiedLocation (supportedYears) {
+    if (!this.envVcInstallDir) {
+      return null
+    }
+    const info = {
+      path: path.resolve(this.envVcInstallDir),
+      // Assume the version specified by the user is correct.
+      // Since Visual Studio 2015, the Developer Command Prompt sets the
+      // VSCMD_VER environment variable which contains the version information
+      // for Visual Studio.
+      // https://learn.microsoft.com/en-us/visualstudio/ide/reference/command-prompt-powershell?view=vs-2022
+      version: process.env.VSCMD_VER,
+      packages: [
+        'Microsoft.VisualStudio.Component.VC.Tools.x86.x64',
+        // Assume MSBuild exists. It will be checked in processing.
+        'Microsoft.VisualStudio.VC.MSBuild.Base'
+      ]
+    }
+
+    // Is there a better way to get SDK information?
+    const envWindowsSDKVersion = process.env.WindowsSDKVersion
+    const sdkVersionMatched = envWindowsSDKVersion?.match(/^(\d+)\.(\d+)\.(\d+)\..*/)
+    if (sdkVersionMatched) {
+      info.packages.push(`Microsoft.VisualStudio.Component.Windows10SDK.${sdkVersionMatched[3]}.Desktop`)
+    }
+    // pass for further processing
+    return this.processData([info], supportedYears)
+  }
+
   async findVisualStudio2019OrNewerUsingSetupModule () {
     return this.findNewVSUsingSetupModule([2019, 2022])
   }
@@ -321,7 +365,7 @@ class VisualStudioFinder {
 
   // Helper - process version information
   getVersionInfo (info) {
-    const match = /^(\d+)\.(\d+)\..*/.exec(info.version)
+    const match = /^(\d+)\.(\d+)(?:\..*)?/.exec(info.version)
     if (!match) {
       this.log.silly('- failed to parse version:', info.version)
       return {}
diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js
index 2c3f4e1981..9cad33eb4e 100644
--- a/test/test-find-visualstudio.js
+++ b/test/test-find-visualstudio.js
@@ -782,4 +782,29 @@ describe('find-visualstudio', function () {
     assert.ok(/find .* Visual Studio/i.test(err), 'expect error')
     assert.ok(!info, 'no data')
   })
+
+  it('run on a portable VS Command Prompt with sufficient environs', async function () {
+    process.env.VCINSTALLDIR = 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC'
+    process.env.VSCMD_VER = '16.0'
+    process.env.WindowsSDKVersion = '10.0.17763.0'
+
+    const finder = new TestVisualStudioFinder(semverV1, null)
+
+    allVsVersions(finder)
+    const { err, info } = await finder.findVisualStudio()
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info, {
+      msBuild: 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\' +
+        'Community\\MSBuild\\Current\\Bin\\MSBuild.exe',
+      path:
+        'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community',
+      sdk: '10.0.17763.0',
+      toolset: 'v142',
+      // Assume version in the environ is correct.
+      version: '16.0',
+      versionMajor: 16,
+      versionMinor: 0,
+      versionYear: 2019
+    })
+  })
 })

From b3916d5b25704a53e89be16b500036a14bdc5060 Mon Sep 17 00:00:00 2001
From: Chengzhong Wu 
Date: Wed, 10 Jul 2024 01:03:10 +0800
Subject: [PATCH 438/551] chore: fix ruff command (#3044)

---
 .github/workflows/tests.yml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index d2746c1dc5..78f6b42d75 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -19,8 +19,8 @@ jobs:
     steps:
     - uses: actions/checkout@v4
     - run: pip install --user ruff
-    - run: ruff --output-format=github --select="E,F,PLC,PLE,UP,W,YTT" --ignore="E721,PLC1901,S101,UP031" --target-version=py38 .
-  
+    - run: ruff check --output-format=github --select="E,F,PLC,PLE,UP,W,YTT" --ignore="E721,PLC1901,S101,UP031" --target-version=py38 .
+
   lint-js:
     name: Lint JS
     runs-on: ubuntu-latest
@@ -88,7 +88,7 @@ jobs:
 
   tests:
     # lint-python takes ~5 seconds, so wait for it to pass before running the full matrix of tests.
-    needs: [lint-python] 
+    needs: [lint-python]
     strategy:
       fail-fast: false
       max-parallel: 15

From 08c91d08adfcf1f24e7ff3e14b454da27bb6be25 Mon Sep 17 00:00:00 2001
From: "github-actions[bot]"
 <41898282+github-actions[bot]@users.noreply.github.com>
Date: Wed, 10 Jul 2024 13:05:32 +0100
Subject: [PATCH 439/551] chore(main): release 10.2.0 (#3006)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Chengzhong Wu 
Co-authored-by: Christian Clauss 
---
 .release-please-manifest.json |  2 +-
 CHANGELOG.md                  | 29 +++++++++++++++++++++++++++++
 package.json                  |  2 +-
 3 files changed, 31 insertions(+), 2 deletions(-)

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
index 1842506cfa..01db3293b9 100644
--- a/.release-please-manifest.json
+++ b/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "10.1.0"
+    ".": "10.2.0"
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9db4f9f952..1d4e418532 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,34 @@
 # Changelog
 
+## [10.2.0](https://github.com/nodejs/node-gyp/compare/v10.1.0...v10.2.0) (2024-07-09)
+
+
+### Features
+
+* allow VCINSTALLDIR to specify a portable instance ([#3036](https://github.com/nodejs/node-gyp/issues/3036)) ([d38af2e](https://github.com/nodejs/node-gyp/commit/d38af2e0c2a81b12cd221b1f8517fb89e609d62c))
+* **gyp:** update gyp to v0.18.1 ([#3039](https://github.com/nodejs/node-gyp/issues/3039)) ([ea99fea](https://github.com/nodejs/node-gyp/commit/ea99fea83485dc5be04db01df9b2fdbe05319b8e))
+* support `rebuild` and `build` for cross-compiling Node-API module to wasm on Windows ([#2974](https://github.com/nodejs/node-gyp/issues/2974)) ([6318d2b](https://github.com/nodejs/node-gyp/commit/6318d2b210224415ff5932c2863e6cc14d4583dc))
+
+
+### Core
+
+* add an arch check to VS 2019 ([#3025](https://github.com/nodejs/node-gyp/issues/3025)) ([323957b](https://github.com/nodejs/node-gyp/commit/323957b74e9586fb3fbfb2acad5040379c778de6))
+* **deps:** bump seanmiddleditch/gha-setup-ninja from 4 to 5 ([#3041](https://github.com/nodejs/node-gyp/issues/3041)) ([10f6730](https://github.com/nodejs/node-gyp/commit/10f6730be660e7a38be8a12111937e37fcf74834))
+* proc-log@4.0.0 ([#3022](https://github.com/nodejs/node-gyp/issues/3022)) ([141aa6b](https://github.com/nodejs/node-gyp/commit/141aa6bf029e6f984be8ea98aaf985e5df894082))
+* tar@6.2.1 ([#3021](https://github.com/nodejs/node-gyp/issues/3021)) ([b22d5ee](https://github.com/nodejs/node-gyp/commit/b22d5eef861892c968052ffc1c71b551f738163b))
+
+
+### Doc
+
+* `node-pre-gyp` is no longer maintained ([#3015](https://github.com/nodejs/node-gyp/issues/3015)) ([93186f1](https://github.com/nodejs/node-gyp/commit/93186f10c966b4148fc500e48f8cbffacccdfa3c))
+* add the way to configuring Python dependency for Windows PowerShell ([#2996](https://github.com/nodejs/node-gyp/issues/2996)) ([9fd7936](https://github.com/nodejs/node-gyp/commit/9fd7936f0d7232a8a79e6a7b6cbfb814d9042b13))
+* Installation -- Python >= v3.12 requires `node-gyp` >= v10 ([#3010](https://github.com/nodejs/node-gyp/issues/3010)) ([a6b48fc](https://github.com/nodejs/node-gyp/commit/a6b48fca9993e54d757cd110f6b41f8200d99ca4))
+
+
+### Miscellaneous
+
+* fix ruff command ([#3044](https://github.com/nodejs/node-gyp/issues/3044)) ([b3916d5](https://github.com/nodejs/node-gyp/commit/b3916d5b25704a53e89be16b500036a14bdc5060))
+
 ## [10.1.0](https://github.com/nodejs/node-gyp/compare/v10.0.1...v10.1.0) (2024-03-13)
 
 
diff --git a/package.json b/package.json
index e9402d7923..8e2ea42510 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "10.1.0",
+  "version": "10.2.0",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From d1ed3d4dc3a53b8ccab4093d002e43945bbece0e Mon Sep 17 00:00:00 2001
From: Sebastian Mellen 
Date: Fri, 19 Jul 2024 14:37:34 -0400
Subject: [PATCH 440/551] fix(ci): use correct release-please-action domain
 after organization url was changed (#3032)

Co-authored-by: Luke Karrys 
---
 .github/workflows/release-please.yml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml
index 6b44256ea9..e6ea15e871 100644
--- a/.github/workflows/release-please.yml
+++ b/.github/workflows/release-please.yml
@@ -11,12 +11,12 @@ jobs:
     outputs:
       pr: ${{ steps.release.outputs.pr }}
     permissions:
-      contents: write # to create release commit (google-github-actions/release-please-action)
-      pull-requests: write # to create release PR (google-github-actions/release-please-action)
+      contents: write # to create release commit (googleapis/release-please-action)
+      pull-requests: write # to create release PR (googleapis/release-please-action)
     if: github.event_name == 'push'
     runs-on: ubuntu-latest
     steps:
-    - uses: google-github-actions/release-please-action@v4
+    - uses: googleapis/release-please-action@v4
       id: release
         # Standard Conventional Commits: `feat` and `fix`
         # node-gyp subdirectories: `bin`, `gyp`, `lib`, `src`, `test`

From d22e2eb080807c6290533a67249c343a7605a989 Mon Sep 17 00:00:00 2001
From: Jivthesh M R 
Date: Sat, 20 Jul 2024 00:12:58 +0530
Subject: [PATCH 441/551] meta: add links to Code of Conduct from root file
 (#2196)

---
 CODE_OF_CONDUCT.md | 4 ++++
 1 file changed, 4 insertions(+)
 create mode 100644 CODE_OF_CONDUCT.md

diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md
new file mode 100644
index 0000000000..4c21140559
--- /dev/null
+++ b/CODE_OF_CONDUCT.md
@@ -0,0 +1,4 @@
+# Code of Conduct
+
+* [Node.js Code of Conduct](https://github.com/nodejs/admin/blob/master/CODE_OF_CONDUCT.md)
+* [Node.js Moderation Policy](https://github.com/nodejs/admin/blob/master/Moderation-Policy.md)

From 831984736393a3ea8417efec5255f95d53a70785 Mon Sep 17 00:00:00 2001
From: Chengzhong Wu 
Date: Mon, 22 Jul 2024 17:39:31 +0800
Subject: [PATCH 442/551] chore: publish to npm with release-please (#3051)

---
 .github/workflows/release-please.yml | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml
index e6ea15e871..a68a589513 100644
--- a/.github/workflows/release-please.yml
+++ b/.github/workflows/release-please.yml
@@ -32,3 +32,17 @@ jobs:
     needs: [ release-please ]
     if: needs.release-please.outputs.pr || startsWith(github.head_ref, 'release-please--')
     uses: ./.github/workflows/tests.yml
+
+  npm-publish:
+    needs: release-please
+    if: ${{ needs.release-please.outputs.release_created }}
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: lts/*
+          registry-url: 'https://registry.npmjs.org'
+      - run: npm publish --access public
+        env:
+          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

From e6f4ede10cca28e9edeaa85d7830914c5d1499c7 Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Tue, 23 Jul 2024 06:50:30 +0200
Subject: [PATCH 443/551] chore(ci) test on Node.js v22 and not v16 (#3052)

---
 .github/workflows/tests.yml | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 78f6b42d75..147ecde6c9 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -27,10 +27,10 @@ jobs:
     steps:
     - name: Checkout Repository
       uses: actions/checkout@v4
-    - name: Use Node.js 20.x
+    - name: Use Node.js 22.x
       uses: actions/setup-node@v4
       with:
-        node-version: 20.x
+        node-version: 22.x
     - name: Install Dependencies
       run: npm install
     - name: Lint
@@ -42,10 +42,10 @@ jobs:
     steps:
     - name: Checkout Repository
       uses: actions/checkout@v4
-    - name: Use Node.js 20.x
+    - name: Use Node.js 22.x
       uses: actions/setup-node@v4
       with:
-        node-version: 20.x
+        node-version: 22.x
     - name: Install Dependencies
       run: npm install
     - name: Check Engines
@@ -60,10 +60,10 @@ jobs:
     steps:
     - name: Checkout Repository
       uses: actions/checkout@v4
-    - name: Use Node.js 20.x
+    - name: Use Node.js 22.x
       uses: actions/setup-node@v4
       with:
-        node-version: 20.x
+        node-version: 22.x
     - name: Update npm
       run: npm install npm@latest -g
     - name: Install Dependencies
@@ -95,11 +95,11 @@ jobs:
       matrix:
         os: [macos-latest, ubuntu-latest, windows-latest]
         python: ["3.8", "3.10", "3.12"]
-        node: [16.x, 18.x, 20.x]
-        include:  # `npm test` is running Windows find-visualstudio tests on an M1 Mac!!!
-          - os: macos-14
+        node: [18.x, 20.x, 22.x]
+        include:  # `npm test` runs Windows find-visualstudio tests on an Intel Mac!!!
+          - os: macos-13
             python: "3.12"
-            node: 20.x
+            node: 22.x
     name: ${{ matrix.os }} - ${{ matrix.python }} - ${{ matrix.node }}
     runs-on: ${{ matrix.os }}
     steps:
@@ -132,10 +132,10 @@ jobs:
         shell: bash
         run: npm test --python="${pythonLocation}/python"
         env:
-          FULL_TEST: ${{ (matrix.node == '20.x' && matrix.python == '3.12') && '1' || '0' }}
+          FULL_TEST: ${{ (matrix.node == '22.x' && matrix.python == '3.12') && '1' || '0' }}
       - name: Run Tests (Windows)
         if: startsWith(matrix.os, 'windows')
         shell: bash # Building wasm on Windows requires using make generator, it only works in bash
         run: npm run test --python="${pythonLocation}\\python.exe"
         env:
-          FULL_TEST: ${{ (matrix.node == '20.x' && matrix.python == '3.12') && '1' || '0' }}
+          FULL_TEST: ${{ (matrix.node == '22.x' && matrix.python == '3.12') && '1' || '0' }}

From 88260bf86aeb4c39959b78104a5edc3dc88d3aef Mon Sep 17 00:00:00 2001
From: Stefan Stojanovic 
Date: Fri, 29 Nov 2024 11:56:42 +0100
Subject: [PATCH 444/551] feat: prohibit compiling with ClangCL on Windows
 (#3098)

Disables trying to compile with ClangCL on Windows. Was required to enable native tests in the Node.js CI for the ClangCL produced binaries.

Refs: https://github.com/nodejs/node/pull/55784
---
 lib/create-config-gypi.js | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/create-config-gypi.js b/lib/create-config-gypi.js
index d598dea6e2..01a820e9f2 100644
--- a/lib/create-config-gypi.js
+++ b/lib/create-config-gypi.js
@@ -96,6 +96,9 @@ async function getCurrentConfigGypi ({ gyp, nodeDir, vsInfo, python }) {
       }
     }
     variables.msbuild_path = vsInfo.msBuild
+    if (config.variables.clang === 1) {
+      config.variables.clang = 0
+    }
   }
 
   // loop through the rest of the opts and add the unknown ones as variables.

From 9afaf00e7ab547fd76c6890582abff9739effa24 Mon Sep 17 00:00:00 2001
From: "github-actions[bot]"
 <41898282+github-actions[bot]@users.noreply.github.com>
Date: Fri, 29 Nov 2024 23:56:38 +0800
Subject: [PATCH 445/551] chore(main): release 10.3.0 (#3050)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
---
 .release-please-manifest.json |  2 +-
 CHANGELOG.md                  | 18 ++++++++++++++++++
 package.json                  |  2 +-
 3 files changed, 20 insertions(+), 2 deletions(-)

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
index 01db3293b9..2ff3edc9c8 100644
--- a/.release-please-manifest.json
+++ b/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "10.2.0"
+    ".": "10.3.0"
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1d4e418532..96d9f18b0f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,23 @@
 # Changelog
 
+## [10.3.0](https://github.com/nodejs/node-gyp/compare/v10.2.0...v10.3.0) (2024-11-29)
+
+
+### Features
+
+* prohibit compiling with ClangCL on Windows ([#3098](https://github.com/nodejs/node-gyp/issues/3098)) ([88260bf](https://github.com/nodejs/node-gyp/commit/88260bf86aeb4c39959b78104a5edc3dc88d3aef))
+
+
+### Bug Fixes
+
+* **ci:** use correct release-please-action domain after organization url was changed ([#3032](https://github.com/nodejs/node-gyp/issues/3032)) ([d1ed3d4](https://github.com/nodejs/node-gyp/commit/d1ed3d4dc3a53b8ccab4093d002e43945bbece0e))
+
+
+### Miscellaneous
+
+* add links to Code of Conduct from root file ([#2196](https://github.com/nodejs/node-gyp/issues/2196)) ([d22e2eb](https://github.com/nodejs/node-gyp/commit/d22e2eb080807c6290533a67249c343a7605a989))
+* publish to npm with release-please ([#3051](https://github.com/nodejs/node-gyp/issues/3051)) ([8319847](https://github.com/nodejs/node-gyp/commit/831984736393a3ea8417efec5255f95d53a70785))
+
 ## [10.2.0](https://github.com/nodejs/node-gyp/compare/v10.1.0...v10.2.0) (2024-07-09)
 
 
diff --git a/package.json b/package.json
index 8e2ea42510..52ae22aa70 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "10.2.0",
+  "version": "10.3.0",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From 6dded88065872a32f44114e60731ba4b701ec057 Mon Sep 17 00:00:00 2001
From: Chengzhong Wu 
Date: Mon, 2 Dec 2024 20:31:57 +0000
Subject: [PATCH 446/551] chore: fix npm-publish dependencies and add
 provenance (#3099)

---
 .github/workflows/release-please.yml | 35 ++++++++++++----------------
 .github/workflows/tests.yml          |  3 ++-
 2 files changed, 17 insertions(+), 21 deletions(-)

diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml
index a68a589513..2bef09ec91 100644
--- a/.github/workflows/release-please.yml
+++ b/.github/workflows/release-please.yml
@@ -4,45 +4,40 @@ on:
   push:
     branches:
       - main
-  pull_request:
 
 jobs:
   release-please:
     outputs:
-      pr: ${{ steps.release.outputs.pr }}
+      release_created:  ${{ steps.release.outputs.release_created }}
     permissions:
       contents: write # to create release commit (googleapis/release-please-action)
       pull-requests: write # to create release PR (googleapis/release-please-action)
-    if: github.event_name == 'push'
     runs-on: ubuntu-latest
     steps:
-    - uses: googleapis/release-please-action@v4
-      id: release
-        # Standard Conventional Commits: `feat` and `fix`
-        # node-gyp subdirectories: `bin`, `gyp`, `lib`, `src`, `test`
-        # node-gyp subcommands: `build`, `clean`, `configure`, `install`, `list`, `rebuild`, `remove`
-        # Core abstract category: `deps`
-        # Languages/platforms: `python`, `lin`, `linux`, `mac`, `macos`, `win`, `window`, `zos`
-        # Documentation: `doc`, `docs`, `readme`
-        # Standard Conventional Commits: `chore` (under "Miscellaneous")
-        # Miscellaneous abstract categories: `refactor`, `ci`, `meta`
-
-  test:
-    name: Release Test
-    needs: [ release-please ]
-    if: needs.release-please.outputs.pr || startsWith(github.head_ref, 'release-please--')
-    uses: ./.github/workflows/tests.yml
+      - uses: googleapis/release-please-action@v4
+        id: release
+          # Standard Conventional Commits: `feat` and `fix`
+          # node-gyp subdirectories: `bin`, `gyp`, `lib`, `src`, `test`
+          # node-gyp subcommands: `build`, `clean`, `configure`, `install`, `list`, `rebuild`, `remove`
+          # Core abstract category: `deps`
+          # Languages/platforms: `python`, `lin`, `linux`, `mac`, `macos`, `win`, `window`, `zos`
+          # Documentation: `doc`, `docs`, `readme`
+          # Standard Conventional Commits: `chore` (under "Miscellaneous")
+          # Miscellaneous abstract categories: `refactor`, `ci`, `meta`
 
   npm-publish:
     needs: release-please
     if: ${{ needs.release-please.outputs.release_created }}
     runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      id-token: write # to generate npm provenance statements
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-node@v4
         with:
           node-version: lts/*
           registry-url: 'https://registry.npmjs.org'
-      - run: npm publish --access public
+      - run: npm publish --provenance --access public
         env:
           NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 147ecde6c9..c1557d5587 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -19,7 +19,8 @@ jobs:
     steps:
     - uses: actions/checkout@v4
     - run: pip install --user ruff
-    - run: ruff check --output-format=github --select="E,F,PLC,PLE,UP,W,YTT" --ignore="E721,PLC1901,S101,UP031" --target-version=py38 .
+    # Excluding `/gyp` directory as it is been checked in https://github.com/nodejs/gyp-next/ already
+    - run: ruff check --output-format=github --extend-exclude=gyp --select="E,F,PLC,PLE,UP,W,YTT" --ignore="E721,PLC1901,S101,UP031" --target-version=py38 .
 
   lint-js:
     name: Lint JS

From 2852eae33e57ccbf808eeb33a685a50d7968cbe1 Mon Sep 17 00:00:00 2001
From: "github-actions[bot]"
 <41898282+github-actions[bot]@users.noreply.github.com>
Date: Mon, 2 Dec 2024 13:51:48 -0700
Subject: [PATCH 447/551] chore(main): release 10.3.1 (#3100)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
---
 .release-please-manifest.json | 2 +-
 CHANGELOG.md                  | 7 +++++++
 package.json                  | 2 +-
 3 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
index 2ff3edc9c8..9e5fec54a4 100644
--- a/.release-please-manifest.json
+++ b/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "10.3.0"
+    ".": "10.3.1"
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 96d9f18b0f..ce66ab155d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,12 @@
 # Changelog
 
+## [10.3.1](https://github.com/nodejs/node-gyp/compare/v10.3.0...v10.3.1) (2024-12-02)
+
+
+### Miscellaneous
+
+* fix npm-publish dependencies and add provenance ([#3099](https://github.com/nodejs/node-gyp/issues/3099)) ([6dded88](https://github.com/nodejs/node-gyp/commit/6dded88065872a32f44114e60731ba4b701ec057))
+
 ## [10.3.0](https://github.com/nodejs/node-gyp/compare/v10.2.0...v10.3.0) (2024-11-29)
 
 
diff --git a/package.json b/package.json
index 52ae22aa70..57142c4284 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "10.3.0",
+  "version": "10.3.1",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From 0e6b6f8bea615cf031d76ecff9102a38e5474c72 Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Tue, 3 Dec 2024 03:04:29 -0700
Subject: [PATCH 448/551] feat!: drop node 16 support (#3102)

`node-gyp` is now compatible with Node `^18.17.0 || >=20.5.0`
---
 package.json | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/package.json b/package.json
index 57142c4284..0b65301d4d 100644
--- a/package.json
+++ b/package.json
@@ -26,20 +26,20 @@
     "exponential-backoff": "^3.1.1",
     "glob": "^10.3.10",
     "graceful-fs": "^4.2.6",
-    "make-fetch-happen": "^13.0.0",
-    "nopt": "^7.0.0",
-    "proc-log": "^4.1.0",
+    "make-fetch-happen": "^14.0.3",
+    "nopt": "^8.0.0",
+    "proc-log": "^5.0.0",
     "semver": "^7.3.5",
-    "tar": "^6.2.1",
-    "which": "^4.0.0"
+    "tar": "^7.4.3",
+    "which": "^5.0.0"
   },
   "engines": {
-    "node": "^16.14.0 || >=18.0.0"
+    "node": "^18.17.0 || >=20.5.0"
   },
   "devDependencies": {
     "bindings": "^1.5.0",
     "cross-env": "^7.0.3",
-    "mocha": "^10.2.0",
+    "mocha": "^11.0.1",
     "nan": "^2.14.2",
     "require-inject": "^1.4.4",
     "standard": "^17.0.0"

From a13017807d0ae7da8fa076b0bcf23153af7c60a6 Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Tue, 3 Dec 2024 10:45:30 -0700
Subject: [PATCH 449/551] chore: migrate from standard to neostandard (#3103)

---
 eslint.config.js | 3 +++
 package.json     | 7 ++++---
 2 files changed, 7 insertions(+), 3 deletions(-)
 create mode 100644 eslint.config.js

diff --git a/eslint.config.js b/eslint.config.js
new file mode 100644
index 0000000000..5212dc93d5
--- /dev/null
+++ b/eslint.config.js
@@ -0,0 +1,3 @@
+'use strict'
+
+module.exports = require('neostandard')({})
diff --git a/package.json b/package.json
index 0b65301d4d..d3ecfaee20 100644
--- a/package.json
+++ b/package.json
@@ -39,13 +39,14 @@
   "devDependencies": {
     "bindings": "^1.5.0",
     "cross-env": "^7.0.3",
+    "eslint": "^9.16.0",
     "mocha": "^11.0.1",
     "nan": "^2.14.2",
-    "require-inject": "^1.4.4",
-    "standard": "^17.0.0"
+    "neostandard": "^0.11.9",
+    "require-inject": "^1.4.4"
   },
   "scripts": {
-    "lint": "standard \"*/*.js\" \"test/**/*.js\" \".github/**/*.js\"",
+    "lint": "eslint \"*/*.js\" \"test/**/*.js\" \".github/**/*.js\"",
     "test": "cross-env NODE_GYP_NULL_LOGGER=true mocha --timeout 15000 test/test-download.js test/test-*"
   }
 }

From bf9168ef3d6a65ff834f4f8715e0a3f6a1914205 Mon Sep 17 00:00:00 2001
From: "github-actions[bot]"
 <41898282+github-actions[bot]@users.noreply.github.com>
Date: Wed, 4 Dec 2024 14:49:27 -0700
Subject: [PATCH 450/551] chore(main): release 11.0.0 (#3104)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
---
 .release-please-manifest.json |  2 +-
 CHANGELOG.md                  | 16 ++++++++++++++++
 package.json                  |  2 +-
 3 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
index 9e5fec54a4..26a3463a2e 100644
--- a/.release-please-manifest.json
+++ b/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "10.3.1"
+    ".": "11.0.0"
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ce66ab155d..8374a920b7 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,21 @@
 # Changelog
 
+## [11.0.0](https://github.com/nodejs/node-gyp/compare/v10.3.1...v11.0.0) (2024-12-03)
+
+
+### ⚠ BREAKING CHANGES
+
+* drop node 16 support ([#3102](https://github.com/nodejs/node-gyp/issues/3102))
+
+### Features
+
+* drop node 16 support ([#3102](https://github.com/nodejs/node-gyp/issues/3102)) ([0e6b6f8](https://github.com/nodejs/node-gyp/commit/0e6b6f8bea615cf031d76ecff9102a38e5474c72))
+
+
+### Miscellaneous
+
+* migrate from standard to neostandard ([#3103](https://github.com/nodejs/node-gyp/issues/3103)) ([a130178](https://github.com/nodejs/node-gyp/commit/a13017807d0ae7da8fa076b0bcf23153af7c60a6))
+
 ## [10.3.1](https://github.com/nodejs/node-gyp/compare/v10.3.0...v10.3.1) (2024-12-02)
 
 
diff --git a/package.json b/package.json
index d3ecfaee20..4a1cfb0eb1 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "10.3.1",
+  "version": "11.0.0",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From b9d10a5a37081e2a731937e43eca52c83609e7f5 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=E6=9D=A8=E5=BF=85=E8=B5=9E?= <348063288@qq.com>
Date: Sat, 14 Dec 2024 01:45:59 +0800
Subject: [PATCH 451/551] fix: try libnode.dll first in load_exe_hook (#2834)

---
 src/win_delay_load_hook.cc | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/src/win_delay_load_hook.cc b/src/win_delay_load_hook.cc
index 169f8029f1..c6e80aa313 100644
--- a/src/win_delay_load_hook.cc
+++ b/src/win_delay_load_hook.cc
@@ -28,7 +28,9 @@ static FARPROC WINAPI load_exe_hook(unsigned int event, DelayLoadInfo* info) {
   if (_stricmp(info->szDll, HOST_BINARY) != 0)
     return NULL;
 
-  m = GetModuleHandle(NULL);
+  // try for libnode.dll to compat node.js that using 'vcbuild.bat dll'
+  m = GetModuleHandle("libnode.dll");
+  if (m == NULL) m = GetModuleHandle(NULL);
   return (FARPROC) m;
 }
 

From b899faed56270d3d8496da7576b5750b264c2c21 Mon Sep 17 00:00:00 2001
From: Kagami Sascha Rosylight 
Date: Fri, 13 Dec 2024 18:48:59 +0100
Subject: [PATCH 452/551] fix: Find VC.Tools.ARM64 on arm64 machine (#3075)

---
 lib/find-visualstudio.js                      |  18 +-
 .../VS_2022_BuildTools_arm64_only.txt         | 423 ++++++++++++++++++
 test/test-find-visualstudio.js                |  38 ++
 3 files changed, 475 insertions(+), 4 deletions(-)
 create mode 100644 test/fixtures/VS_2022_BuildTools_arm64_only.txt

diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js
index 2dc1930fd7..e9aa7fafdc 100644
--- a/lib/find-visualstudio.js
+++ b/lib/find-visualstudio.js
@@ -145,6 +145,7 @@ class VisualStudioFinder {
       version: process.env.VSCMD_VER,
       packages: [
         'Microsoft.VisualStudio.Component.VC.Tools.x86.x64',
+        'Microsoft.VisualStudio.Component.VC.Tools.ARM64',
         // Assume MSBuild exists. It will be checked in processing.
         'Microsoft.VisualStudio.VC.MSBuild.Base'
       ]
@@ -429,12 +430,21 @@ class VisualStudioFinder {
 
   // Helper - process toolset information
   getToolset (info, versionYear) {
-    const pkg = 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64'
+    const vcToolsArm64 = 'VC.Tools.ARM64'
+    const pkgArm64 = `Microsoft.VisualStudio.Component.${vcToolsArm64}`
+    const vcToolsX64 = 'VC.Tools.x86.x64'
+    const pkgX64 = `Microsoft.VisualStudio.Component.${vcToolsX64}`
     const express = 'Microsoft.VisualStudio.WDExpress'
 
-    if (info.packages.indexOf(pkg) !== -1) {
-      this.log.silly('- found VC.Tools.x86.x64')
-    } else if (info.packages.indexOf(express) !== -1) {
+    if (process.arch === 'arm64' && info.packages.includes(pkgArm64)) {
+      this.log.silly(`- found ${vcToolsArm64}`)
+    } else if (info.packages.includes(pkgX64)) {
+      if (process.arch === 'arm64') {
+        this.addLog(`- found ${vcToolsX64} on ARM64 platform. Expect less performance and/or link failure with ARM64 binary.`)
+      } else {
+        this.log.silly(`- found ${vcToolsX64}`)
+      }
+    } else if (info.packages.includes(express)) {
       this.log.silly('- found Visual Studio Express (looking for toolset)')
     } else {
       return null
diff --git a/test/fixtures/VS_2022_BuildTools_arm64_only.txt b/test/fixtures/VS_2022_BuildTools_arm64_only.txt
new file mode 100644
index 0000000000..39e4590f34
--- /dev/null
+++ b/test/fixtures/VS_2022_BuildTools_arm64_only.txt
@@ -0,0 +1,423 @@
+[
+  {
+    "path": "C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools",
+    "version": "17.11.35222.181",
+    "packages": [
+      "Microsoft.VisualStudio.Component.VC.Tools.ARM64",
+      "Microsoft.VisualStudio.VC.MSBuild.v170.ARM64.v143",
+      "Microsoft.VisualStudio.VC.MSBuild.v170.ARM64",
+      "Microsoft.VS.VC.vcvars.arm64.Shortcuts",
+      "Microsoft.VisualCpp.CA.Ext.Hostx64.TargetARM64",
+      "Microsoft.VC.14.41.17.11.CA.Ext.Hostx64.TargetARM64.base",
+      "Microsoft.VC.14.41.17.11.CA.Ext.Hostx64.TargetARM64.Res.base",
+      "Microsoft.VisualCpp.CA.Ext.Hostx86.TargetARM64",
+      "Microsoft.VC.14.41.17.11.CA.Ext.Hostx86.TargetARM64.base",
+      "Microsoft.VC.14.41.17.11.CA.Ext.Hostx86.TargetARM64.Res.base",
+      "Microsoft.VisualCpp.CA.Ext.HostARM64.TargetARM64",
+      "Microsoft.VC.14.41.17.11.CA.Ext.HostARM64.TargetARM64.base",
+      "Microsoft.VC.14.41.17.11.CA.Ext.HostARM64.TargetARM64.Res.base",
+      "Microsoft.VisualCpp.Tools.Hostx86.Targetarm64",
+      "Microsoft.VC.14.41.17.11.Tools.Hostx86.Targetarm64.base",
+      "Microsoft.VC.14.41.17.11.Tools.HostX86.TargetARM64.Res.base",
+      "Microsoft.VisualCpp.Tools.HostARM64.TargetARM64",
+      "Microsoft.VC.14.41.17.11.Tools.HostARM64.TargetARM64.base",
+      "Microsoft.VC.14.41.17.11.Tools.HostARM64.TargetARM64.Res.base",
+      "Microsoft.VisualCpp.CRT.Redist.ARM64.OneCore.Desktop",
+      "Microsoft.VC.14.40.17.10.CRT.Redist.ARM64.OneCore.Desktop.base",
+      "Microsoft.VisualCpp.CRT.Redist.ARM64",
+      "Microsoft.VC.14.40.17.10.CRT.Redist.ARM64.base",
+      "Microsoft.VisualCpp.CRT.ARM64.OneCore.Desktop",
+      "Microsoft.VC.14.41.17.11.CRT.ARM64.OneCore.Desktop.base",
+      "Microsoft.VC.14.41.17.11.CRT.ARM64.OneCore.Desktop.debug.base",
+      "Microsoft.VisualCpp.CRT.ARM64.Store",
+      "Microsoft.VC.14.41.17.11.CRT.ARM64.Store.base",
+      "Microsoft.VisualCpp.CRT.ARM64.Desktop",
+      "Microsoft.VC.14.41.17.11.CRT.ARM64.Desktop.base",
+      "Microsoft.VC.14.41.17.11.CRT.ARM64.Desktop.debug.base",
+      "Microsoft.VisualStudio.PackageGroup.VC.Tools.x64.ARM64",
+      "Microsoft.VisualCpp.Tools.Core",
+      "Microsoft.VisualCpp.PGO.ARM64",
+      "Microsoft.VC.14.41.17.11.PGO.ARM64.base",
+      "Microsoft.VisualCpp.Premium.Tools.Hostx86.Targetarm64",
+      "Microsoft.VC.14.41.17.11.Premium.Tools.Hostx86.Targetarm64.base",
+      "Microsoft.VC.14.41.17.11.Prem.HostX86.TargetARM64.Res.base",
+      "Microsoft.VisualCpp.Premium.Tools.HostX64.TargetARM64",
+      "Microsoft.VC.14.41.17.11.Premium.Tools.HostX64.TargetARM64.base",
+      "Microsoft.VC.14.41.17.11.Prem.HostX64.TargetARM64.Res.base",
+      "Microsoft.VisualCpp.Premium.Tools.ARM64.Base",
+      "Microsoft.VC.14.41.17.11.Premium.Tools.ARM64.Base.base",
+      "Microsoft.VisualCpp.Tools.HostX64.TargetARM64",
+      "Microsoft.VC.14.41.17.11.Tools.HostX64.TargetARM64.base",
+      "Microsoft.VC.14.41.17.11.Tools.HostX64.TargetARM64.Res.base",
+      "Microsoft.VC.14.41.17.11.Props.ARM64",
+      "Microsoft.VisualStudio.TestTools.TestWIExtension",
+      "Microsoft.VisualStudio.TestTools.TestPlatform.IDE",
+      "Microsoft.VisualStudio.VC.Ide.Pro",
+      "Microsoft.VisualStudio.VC.Ide.Pro.Resources",
+      "Microsoft.VisualStudio.VC.Templates.General",
+      "Microsoft.VisualStudio.VC.Templates.General.Resources",
+      "Microsoft.VisualStudio.VC.Items.Pro",
+      "Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Reduced",
+      "Microsoft.VisualStudio.VC.Ide.MDD",
+      "Microsoft.VisualStudio.PackageGroup.Core",
+      "Microsoft.VisualStudio.CodeSense.Community",
+      "Microsoft.VisualStudio.TestTools.TeamFoundationClient",
+      "Microsoft.PackageGroup.ClientDiagnostics",
+      "Microsoft.VisualStudio.AppResponsiveness",
+      "Microsoft.VisualStudio.AppResponsiveness.Targeted",
+      "Microsoft.VisualStudio.AppResponsiveness.Resources",
+      "Microsoft.VisualStudio.ClientDiagnostics",
+      "Microsoft.VisualStudio.ClientDiagnostics.Targeted",
+      "Microsoft.VisualStudio.Component.VC.Llvm.Clang",
+      "Microsoft.VisualStudio.VC.Llvm.Clang",
+      "Microsoft.VisualStudio.ClientDiagnostics.Resources",
+      "Microsoft.VisualStudio.PackageGroup.CommunityCore",
+      "Microsoft.VisualStudio.ProjectSystem.Full",
+      "Microsoft.VisualStudio.Product.BuildTools",
+      "Microsoft.VisualStudio.VC.Ide.Linux.ConnectionManager",
+      "Microsoft.VisualStudio.VC.Ide.Linux.ConnectionManager.Resources",
+      "Microsoft.VisualStudio.PackageGroup.TestTools.Core",
+      "Microsoft.VC.14.41.17.11.Props.x86",
+      "Microsoft.VC.14.41.17.11.Props",
+      "Microsoft.VisualCpp.RuntimeDebug.14",
+      "Microsoft.VisualCpp.RuntimeDebug.14.ARM64",
+      "Microsoft.VC.14.41.17.11.Tools.HostX86.TargetX86.Res.base",
+      "Microsoft.VisualCpp.Tools.Core.Resources",
+      "Microsoft.VisualCpp.Tools.Core.x86",
+      "Microsoft.VC.14.41.17.11.Tools.Core.Props",
+      "Microsoft.VisualCpp.Tools.Common.Utils",
+      "Microsoft.VisualCpp.Tools.Common.Utils.Resources",
+      "Microsoft.VisualCpp.DIA.SDK",
+      "Microsoft.VisualCpp.Servicing.DIASDK",
+      "Microsoft.VisualCpp.CRT.x86.Desktop",
+      "Microsoft.VisualStudio.Workload.VCTools",
+      "Microsoft.VC.14.41.17.11.CRT.x64.Desktop.base",
+      "Microsoft.VisualCpp.CRT.Source",
+      "Microsoft.VC.14.41.17.11.CRT.Source.base",
+      "Microsoft.VisualCpp.CRT.Headers",
+      "Microsoft.VC.14.41.17.11.CRT.Headers.Resources",
+      "Microsoft.VisualStudio.Component.VC.Llvm.ClangToolset",
+      "Microsoft.VisualCpp.CRT.Redist.Resources",
+      "Microsoft.VC.14.41.17.11.CRT.Headers.Resources.base",
+      "Microsoft.VisualStudio.VC.MSBuild.Llvm",
+      "Microsoft.VisualStudio.VC.MSBuild.Llvm.Resources",
+      "Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core",
+      "Microsoft.VisualStudio.PackageGroup.TestTools.Native",
+      "Microsoft.VisualStudio.VC.Templates.UnitTest",
+      "Microsoft.VisualStudio.VC.Templates.UnitTest.Resources",
+      "Microsoft.VisualCpp.ATL.ARM64",
+      "Microsoft.VC.14.41.17.11.ATL.ARM64.base",
+      "Microsoft.VisualStudio.Community.VB.Targeted",
+      "Microsoft.VisualStudio.Community.VB.Neutral",
+      "Microsoft.VisualStudio.Community.CSharp.Targeted",
+      "Microsoft.VisualStudio.Community.CSharp.Neutral",
+      "Microsoft.VisualStudio.Community.ProductArch.TargetedExtra",
+      "Microsoft.VisualStudio.VC.MSBuild.Base",
+      "Microsoft.VisualStudio.VC.MSBuild.Base.Resources",
+      "Microsoft.VisualStudio.VC.Templates.Desktop",
+      "Microsoft.VisualStudio.Component.VC.CoreIde",
+      "Microsoft.VisualCpp.ATL.ARM64.Spectre",
+      "Microsoft.VisualStudio.Component.VC.ATL.ARM64",
+      "Microsoft.VisualCpp.CRT.x86.Store",
+      "Microsoft.VC.14.41.17.11.ATL.ARM64.Spectre.base",
+      "Microsoft.VC.14.41.17.11.CRT.x86.Store.base",
+      "Microsoft.VC.14.41.17.11.Props.ARM64.Spectre",
+      "Microsoft.VisualStudio.Component.VC.ATL.ARM64.Spectre",
+      "Microsoft.VisualStudio.Community.x64",
+      "Microsoft.VisualStudio.Community.ProductArch.Targeted",
+      "Microsoft.VisualStudio.Community.ProductArch.NeutralExtra",
+      "Microsoft.IntelliTrace.CollectorCab",
+      "Microsoft.VisualStudio.Community.VB.Resources.Targeted",
+      "Microsoft.VisualStudio.Community.VB.Resources.Neutral",
+      "Microsoft.VisualStudio.Community.CSharp.Resources.Targeted",
+      "Microsoft.VisualStudio.Community.CSharp.Resources.Neutral",
+      "Microsoft.VisualStudio.Community.ProductArch.Resources.Targeted",
+      "Microsoft.VisualStudio.Community.ProductArch.Resources.NeutralExtra",
+      "Microsoft.VisualStudio.Community.ProductArch.Resources.Neutral",
+      "Microsoft.VisualStudio.WebSiteProject.DTE",
+      "Microsoft.VisualStudio.Diagnostics.AspNetHelper",
+      "Microsoft.MSHtml",
+      "Microsoft.VisualStudio.Platform.CallHierarchy",
+      "Microsoft.VisualStudio.Community.ProductArch.Neutral",
+      "Microsoft.VisualStudio.Community.Msi.Resources",
+      "Microsoft.VisualStudio.Community.Msi",
+      "Microsoft.VisualStudio.Community.Shared.Msi",
+      "Microsoft.VisualStudio.MinShell.Interop.Msi",
+      "Microsoft.VisualStudio.MinShell.Interop.Shared.Msi",
+      "Microsoft.VisualStudio.PackageGroup.CoreEditor",
+      "Microsoft.VisualStudio.ScriptedHost",
+      "Microsoft.VisualStudio.ScriptedHost.Targeted",
+      "Microsoft.VisualStudio.VirtualTree",
+      "Microsoft.VisualStudio.PackageGroup.Progression",
+      "Microsoft.VisualStudio.PerformanceProvider",
+      "Microsoft.VisualStudio.GraphModel",
+      "Microsoft.VisualStudio.GraphProvider",
+      "Microsoft.VisualStudio.TextMateGrammars",
+      "Microsoft.VisualStudio.Platform.Markdown",
+      "Microsoft.VisualStudio.PackageGroup.TeamExplorer.Common",
+      "Microsoft.VisualStudio.PackageGroup.ServiceHub",
+      "Microsoft.ServiceHub.Node",
+      "Microsoft.ServiceHub.Managed",
+      "Microsoft.VisualStudio.OpenFolder.VSIX",
+      "Microsoft.VisualStudio.FileHandler.Msi",
+      "Microsoft.VisualStudio.FileHandler.Msi",
+      "Microsoft.VisualStudio.PackageGroup.MinShell",
+      "Microsoft.VisualStudio.MinShell.Msi",
+      "Microsoft.VisualStudio.MinShell.Shared.Msi",
+      "Microsoft.VisualStudio.MinShell.Msi.Resources",
+      "Microsoft.VisualStudio.MinShell.Interop",
+      "Microsoft.VisualStudio.Log",
+      "Microsoft.VisualStudio.Log.Targeted",
+      "Microsoft.VisualStudio.Log.Resources",
+      "Microsoft.VisualStudio.Finalizer",
+      "Microsoft.VisualStudio.CoreEditor",
+      "Microsoft.VisualStudio.Platform.NavigateTo",
+      "Microsoft.VisualStudio.Connected",
+      "Microsoft.VisualStudio.Identity",
+      "Microsoft.Developer.IdentityServiceGS",
+      "SQLitePCLRaw",
+      "SQLitePCLRaw.Targeted",
+      "Microsoft.VisualStudio.Connected.Auto",
+      "Microsoft.VisualStudio.Connected.Auto.Resources",
+      "Microsoft.VisualStudio.Connected.Resources",
+      "Microsoft.VisualStudio.VC.Ide.x64",
+      "Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Express",
+      "Microsoft.VisualStudio.PackageGroup.Debugger.Script",
+      "Microsoft.VisualStudio.Debugger.Script",
+      "Microsoft.VisualStudio.Debugger.Script.Resources",
+      "Microsoft.VisualStudio.Debugger.Script.Remote",
+      "Microsoft.WebView2",
+      "Microsoft.VisualStudio.Debugger.Script.Remote",
+      "Microsoft.VisualStudio.Debugger.Script.Remote.Resources",
+      "Microsoft.VisualStudio.Debugger.Script.Remote.Resources",
+      "Microsoft.VisualStudio.VC.Ide.WinXPlus",
+      "Microsoft.VisualStudio.VC.Ide.Dskx",
+      "Microsoft.VisualStudio.VC.Ide.Dskx.Resources",
+      "Microsoft.VisualStudio.VC.Ide.Base",
+      "Microsoft.VisualStudio.VC.Ide.LanguageService",
+      "Microsoft.VisualStudio.VC.Copilot.Setup",
+      "Microsoft.VisualStudio.VC.Ide.VCPkgDatabase",
+      "Microsoft.VisualStudio.VC.Ide.ResourceEditor",
+      "Microsoft.VisualStudio.VC.Ide.ResourceEditor.Resources",
+      "Microsoft.VisualStudio.VC.Ide.Core",
+      "Microsoft.VisualStudio.VisualC.Utilities",
+      "Microsoft.VisualStudio.VisualC.Utilities.Resources",
+      "Microsoft.VisualStudio.VC.Ide.ProjectSystem",
+      "Microsoft.VisualStudio.VC.Ide.ProjectSystem.Resources",
+      "Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine",
+      "Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine.Resources",
+      "Microsoft.VisualStudio.VC.Ide.LanguageService.Resources",
+      "Microsoft.VisualStudio.VC.Llvm.Base",
+      "CoreEditorFonts",
+      "Microsoft.VisualStudio.VC.Ide.Base.Resources",
+      "Microsoft.VisualStudio.Component.TextTemplating",
+      "Microsoft.VisualStudio.PackageGroup.Debugger.Core",
+      "Microsoft.VisualStudio.Debugger.BrokeredServices",
+      "Microsoft.VisualStudio.Debugger.VSCodeDebuggerHost",
+      "Microsoft.VisualStudio.Debugger.AzureAttach",
+      "Microsoft.VisualStudio.Web.Azure.Common",
+      "Microsoft.WebTools.Shared",
+      "Microsoft.WebTools.DotNet.Core.ItemTemplates",
+      "Microsoft.VisualStudio.PackageGroup.Debugger.TimeTravel.Replay",
+      "Microsoft.VisualStudio.VC.Ide.Debugger",
+      "Microsoft.VisualStudio.VC.Ide.Debugger.Concord",
+      "Microsoft.VisualStudio.VC.Ide.Debugger.Concord.Resources",
+      "Microsoft.VisualStudio.VC.Ide.Debugger.Resources",
+      "Microsoft.VisualStudio.VC.Ide.Common",
+      "Microsoft.VisualStudio.VC.Ide.Common.Resources",
+      "Microsoft.VisualStudio.Debugger.CollectionAgents",
+      "Microsoft.VisualStudio.Debugger.Parallel",
+      "Microsoft.VisualStudio.Debugger.Parallel.Resources",
+      "Microsoft.VisualStudio.Debugger.Managed",
+      "Microsoft.DiaSymReader",
+      "Microsoft.CodeAnalysis.ExpressionEvaluator",
+      "Microsoft.VisualStudio.Debugger.Concord.Managed",
+      "Microsoft.VisualStudio.Debugger.Concord.Managed.Resources",
+      "Microsoft.VisualStudio.Debugger.Managed.Resources",
+      "Microsoft.VisualStudio.Debugger.TargetComposition",
+      "Microsoft.VisualStudio.Debugger.TargetComposition.Remote.arm64",
+      "Microsoft.VisualStudio.Debugger.TargetComposition.Remote",
+      "Microsoft.VisualStudio.Debugger.TargetComposition.Remote",
+      "Microsoft.VisualStudio.Debugger.Remote",
+      "Microsoft.VisualStudio.Debugger.Concord.Remote",
+      "Microsoft.VisualStudio.Debugger.Concord.Remote.Resources",
+      "Microsoft.VisualStudio.VC.Ide.LanguageService.Dependencies",
+      "Microsoft.VisualStudio.Debugger.Remote",
+      "Microsoft.VisualStudio.Debugger.Remote.ARM64",
+      "Microsoft.VisualStudio.Debugger.Concord.Remote.ARM64",
+      "Microsoft.VisualStudio.Debugger.Concord.Remote.Resources.ARM64",
+      "Microsoft.VisualStudio.Debugger.Remote.ARM",
+      "Microsoft.VisualStudio.Debugger.Concord.Remote.ARM",
+      "Microsoft.VisualStudio.Debugger.Concord.Remote.Resources.ARM",
+      "Microsoft.VisualStudio.Debugger.Remote.Resources.ARM",
+      "Microsoft.VisualStudio.Debugger.Remote.Resources.ARM64",
+      "Microsoft.VisualStudio.Debugger.Concord.Remote",
+      "Microsoft.VisualStudio.Debugger.Concord.Remote.Resources",
+      "Microsoft.VisualStudio.Debugger.Remote.Resources",
+      "Microsoft.VisualStudio.Debugger.Remote.Resources",
+      "Microsoft.VisualStudio.Debugger",
+      "Microsoft.VisualStudio.AzureSDK",
+      "Microsoft.VisualStudio.Editors",
+      "Microsoft.VisualStudio.VC.MSVCDis",
+      "Microsoft.IntelliTrace.DiagnosticsHub",
+      "Microsoft.VisualStudio.MinShell",
+      "Microsoft.VisualStudio.Copilot.Contracts",
+      "Microsoft.VisualStudio.Licensing",
+      "Microsoft.VisualStudio.IdentityDependencies",
+      "Microsoft.VisualStudio.GitHubProtocolHandler.Msi",
+      "Microsoft.VisualStudio.VsWebProtocolSelector.Msi",
+      "Microsoft.VisualStudio.Extensibility.Container",
+      "Microsoft.VisualStudio.LanguageServer",
+      "Microsoft.VisualStudio.MefHosting",
+      "Microsoft.VisualStudio.Initializer",
+      "Microsoft.VisualStudio.ExtensionManager",
+      "Microsoft.VisualStudio.ExtensionManager.Auto",
+      "Microsoft.VisualStudio.Platform.Editor",
+      "Microsoft.VisualStudio.MinShell.Targeted",
+      "Microsoft.VisualStudio.Devenv.Config",
+      "Microsoft.VisualStudio.MinShell.Resources",
+      "Microsoft.VisualStudio.UIInternal.Guide",
+      "Microsoft.VisualStudio.UIInternal",
+      "Microsoft.VisualStudio.UIInternal.Resources",
+      "Microsoft.VisualStudio.CoreDotNet",
+      "Microsoft.VisualStudio.MinShell.Auto",
+      "Microsoft.VisualStudio.MinShell.Auto.Resources",
+      "Microsoft.VisualStudio.Debugger.Concord",
+      "Microsoft.VisualStudio.Debugger.Concord.Resources",
+      "Microsoft.VisualStudio.Debugger.Resources",
+      "Microsoft.DiaSymReader.PortablePdb",
+      "Microsoft.VisualStudio.PerfLib",
+      "Microsoft.VisualStudio.Debugger.Package.DiagHub.Client",
+      "Microsoft.VisualStudio.Debugger.Remote.DiagnosticsHub.Client",
+      "Microsoft.VisualStudio.Debugger.Remote.DiagnosticsHub.Client",
+      "Microsoft.VisualStudio.Debugger.Remote.DiagnosticsHub.Client",
+      "Microsoft.VisualStudio.TextTemplating.MSBuild",
+      "Microsoft.VisualStudio.TextTemplating.Integration",
+      "Microsoft.VisualStudio.TextTemplating.Core",
+      "Microsoft.VisualStudio.PackageGroup.Roslyn.LanguageServices",
+      "Microsoft.CodeAnalysis.VisualStudio.Setup",
+      "Microsoft.VisualStudio.TextTemplating.Integration.Resources",
+      "Microsoft.VC.14.41.17.11.Props.IFC",
+      "Microsoft.VisualStudio.Component.VC.ASAN",
+      "Microsoft.VisualCpp.ASAN.X86",
+      "Microsoft.VC.14.41.17.11.ASAN.X86.base",
+      "Microsoft.VC.14.41.17.11.ASAN.X64.base",
+      "Microsoft.VC.14.41.17.11.ASAN.Headers.base",
+      "Microsoft.VC.14.41.17.11.Props.ATLMFC",
+      "Microsoft.VisualCpp.ATL.Source",
+      "Microsoft.VC.14.41.17.11.ATL.Source.base",
+      "Microsoft.VisualCpp.ATL.Headers",
+      "Microsoft.VC.14.41.17.11.ATL.Headers.base",
+      "Microsoft.VC.14.41.17.11.Servicing.ATL",
+      "Microsoft.VisualStudio.Component.TestTools.BuildTools",
+      "Microsoft.VisualStudio.PackageGroup.TestTools.BuildTools",
+      "Microsoft.VisualStudio.PackageGroup.TestTools.CodeCoverage",
+      "Microsoft.VisualStudio.TestTools.DynamicCodeCoverage",
+      "Microsoft.CodeCoverage.Console.Targeted",
+      "Microsoft.VisualStudio.TestTools.TestPlatform.V1.CLI",
+      "Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.V2.CLI",
+      "Microsoft.VisualStudio.TestTools.TestPlatform.V2.CLI",
+      "Microsoft.VisualStudio.Component.Windows11SDK.22621",
+      "Microsoft.VisualStudio.VC.UnitTest.Desktop.Build.Core",
+      "Microsoft.VisualStudio.TestTools.TestPlatform.V1.CPP",
+      "Microsoft.VisualStudio.Component.VC.Redist.14.Latest",
+      "Microsoft.VC.14.41.17.11.CA.Rulesets.base",
+      "Microsoft.VC.14.41.17.11.Servicing.CARulesets",
+      "Microsoft.VC.14.41.17.11.Servicing.CAExtensions",
+      "Microsoft.VC.14.41.17.11.Tools.HostX64.TargetX86.base",
+      "Microsoft.VC.14.41.17.11.Tools.HostX64.TargetX86.Res.base",
+      "Microsoft.VisualCpp.Tools.HostX64.TargetX64",
+      "Microsoft.VC.14.41.17.11.Tools.HostX64.TargetX64.base",
+      "Microsoft.VC.14.41.17.11.Tools.HostX64.TargetX64.Res.base",
+      "Microsoft.VisualCpp.Tools.HostARM64.TargetX86",
+      "Microsoft.VC.14.41.17.11.Tools.HostARM64.TargetX86.base",
+      "Microsoft.VC.14.41.17.11.Tools.HostARM64.TargetX86.Res.base",
+      "Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86",
+      "Microsoft.VC.14.41.17.11.Premium.Tools.HostX86.TargetX86.base",
+      "Microsoft.VisualStudio.Component.VC.Modules.ARM64",
+      "Microsoft.VC.14.41.17.11.Prem.HostX86.TargetX86.Res.base",
+      "Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64",
+      "Microsoft.VC.14.41.17.11.Premium.Tools.HostX64.TargetX64.base",
+      "Microsoft.VC.14.41.17.11.Prem.HostX64.TargetX64.Res.base",
+      "Microsoft.Build.Arm64",
+      "Microsoft.Build.UnGAC",
+      "Microsoft.VisualStudio.VC.Icons",
+      "Microsoft.VisualCpp.CRT.x86.OneCore.Desktop",
+      "Microsoft.VC.14.41.17.11.CRT.x86.OneCore.Desktop.base",
+      "Microsoft.VisualCpp.CRT.x64.Store",
+      "Microsoft.VC.14.41.17.11.CRT.x64.Store.base",
+      "Microsoft.VisualStudio.InstrumentationEngine.ARM64",
+      "Microsoft.VisualStudio.InstrumentationEngine",
+      "Microsoft.VisualCpp.CRT.x64.OneCore.Desktop",
+      "Microsoft.VC.14.41.17.11.CRT.x64.OneCore.Desktop.base",
+      "Microsoft.VC.14.41.17.11.Tools.HostX86.TargetX64.base",
+      "Microsoft.VC.14.41.17.11.Props.x64",
+      "Microsoft.VC.14.41.17.11.Tools.Hostx86.Targetx64.Res.base",
+      "Win11SDK_10.0.22621",
+      "Microsoft.VisualCpp.Tools.HostX86.TargetX86",
+      "Microsoft.VC.14.41.17.11.Tools.HostX86.TargetX86.base",
+      "Microsoft.VC.14.41.17.11.Servicing.Compilers",
+      "Microsoft.VisualCpp.Redist.14.Latest",
+      "Microsoft.VisualCpp.Redist.14.Latest",
+      "Microsoft.VisualStudio.LiveShareApi",
+      "Microsoft.VisualStudio.ProjectSystem.Query",
+      "Microsoft.VisualStudio.Component.VC.Tools.ARM64EC",
+      "Microsoft.VisualCpp.CRT.ARM64EC.Store",
+      "Microsoft.VisualStudio.VC.MSBuild.v170.ARM64EC.v143",
+      "Microsoft.VC.14.41.17.11.CRT.ARM64EC.Store.base",
+      "Microsoft.VisualStudio.VC.MSBuild.v170.ARM64EC",
+      "Microsoft.VC.14.41.17.11.CRT.x86.Desktop.base",
+      "Microsoft.VisualCpp.CRT.x64.Desktop",
+      "Microsoft.VisualStudio.ProjectSystem",
+      "Microsoft.VisualStudio.Community.x86",
+      "Microsoft.VisualCpp.IFC.arm64",
+      "Microsoft.VisualCpp.Redist.14",
+      "Microsoft.VisualCpp.Redist.14",
+      "Microsoft.VisualCpp.Servicing.Redist",
+      "Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.IDE",
+      "Microsoft.VC.14.41.17.11.CRT.Headers.base",
+      "Microsoft.VC.14.41.17.11.Servicing.CrtHeaders",
+      "Microsoft.VC.14.41.17.11.Servicing",
+      "Microsoft.VisualStudio.Component.VC.CoreBuildTools",
+      "Microsoft.VisualStudio.VC.vcvars",
+      "Microsoft.VS.VC.vcvars.x86.Shortcuts",
+      "Microsoft.Windows.UniversalCRT.Redistributable.Msi",
+      "Microsoft.VS.VC.vcvars.x64.Shortcuts",
+      "Microsoft.VS.VC.vcvars.arm64_x64.Shortcuts",
+      "Microsoft.VisualStudio.Component.Windows10SDK",
+      "Microsoft.VisualStudio.VC.MSBuild.v170.x86.v143",
+      "Microsoft.VisualStudio.VC.MSBuild.v170.X86",
+      "Microsoft.VisualStudio.VC.MSBuild.v170.X64.v143",
+      "Microsoft.VisualStudio.VC.MSBuild.v170.X64",
+      "Microsoft.VisualStudio.VC.MSBuild.v170.ARM.v143",
+      "Microsoft.VisualStudio.VC.MSBuild.v170.ARM",
+      "Microsoft.VisualStudio.VC.MSBuild.v170.Base",
+      "Microsoft.VisualStudio.VC.MSBuild.v170.Base.Resources",
+      "Microsoft.VisualStudio.Workload.MSBuildTools",
+      "Microsoft.VisualStudio.Component.CoreBuildTools",
+      "Microsoft.VisualStudio.Setup.Configuration",
+      "Microsoft.VisualStudio.PackageGroup.Setup.Common",
+      "Microsoft.VisualStudio.Setup.WMIProvider",
+      "Microsoft.VisualStudio.Setup.Configuration.Interop",
+      "Microsoft.VisualStudio.PackageGroup.VsDevCmd",
+      "Microsoft.VisualStudio.VsDevCmd.Ext.NetFxSdk",
+      "Microsoft.VisualStudio.VsDevCmd.Core.WinSdk",
+      "Microsoft.VisualStudio.VsDevCmd.Core.DotNet",
+      "Microsoft.VisualStudio.VC.DevCmd",
+      "Microsoft.VisualStudio.VC.DevCmd.Resources",
+      "Microsoft.VisualStudio.BuildTools.Resources",
+      "Microsoft.VisualStudio.Net.Eula.Resources",
+      "Microsoft.Build.Dependencies",
+      "Microsoft.NuGet.Build.Tasks.Setup",
+      "Microsoft.Build.FileTracker.Msi",
+      "Microsoft.Component.MSBuild",
+      "Microsoft.PythonTools.BuildCore.Vsix",
+      "Microsoft.VisualStudio.Component.Roslyn.Compiler",
+      "Microsoft.CodeAnalysis.Compilers",
+      "Microsoft.VisualStudio.NativeImageSupport",
+      "Microsoft.Build",
+      "Microsoft.VisualStudio.PackageGroup.NuGet",
+      "Microsoft.VisualStudio.NuGet.BuildTools"
+    ]
+  }
+]
diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js
index 9cad33eb4e..e53b6678ed 100644
--- a/test/test-find-visualstudio.js
+++ b/test/test-find-visualstudio.js
@@ -462,6 +462,44 @@ describe('find-visualstudio', function () {
     })
   })
 
+  it('VS2022 Build Tools with ARM64 MSVC only', async function () {
+    if (process.arch !== 'arm64') {
+      return
+    }
+
+    const msBuildPath = 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\' +
+        'BuildTools\\MSBuild\\Current\\Bin\\arm64\\MSBuild.exe'
+
+    const finder = new TestVisualStudioFinder(semverV1, null)
+
+    poison(finder, 'regSearchKeys')
+    poison(finder, 'findVisualStudio2017')
+    finder.msBuildPathExists = (path) => {
+      return true
+    }
+    finder.findNewVSUsingSetupModule = async () => null
+    finder.findVisualStudio2019OrNewer = async () => {
+      const file = path.join(__dirname, 'fixtures',
+        'VS_2022_BuildTools_arm64_only.txt')
+      const data = fs.readFileSync(file)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2019, 2022])
+    }
+    const { err, info } = await finder.findVisualStudio()
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info, {
+      msBuild: msBuildPath,
+      path:
+        'C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools',
+      sdk: '10.0.22621.0',
+      toolset: 'v143',
+      version: '17.11.35222.181',
+      versionMajor: 17,
+      versionMinor: 11,
+      versionYear: 2022
+    })
+  })
+
   it('VSSetup: VS2022 with C++ workload without SDK', async function () {
     const finder = new TestVisualStudioFinder(semverV1, null)
     finder.msBuildPathExists = (path) => {

From 94448fcd9f090814bce1c4361471dae199dc2e82 Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Thu, 2 Jan 2025 20:41:47 +0100
Subject: [PATCH 453/551] chore: Use astral-sh/ruff-action@v3 to run the Python
 linter (#3114)

* Use pipx instead of pip to install the ruff linter

* PLC0206

* astral-sh/ruff-action@v3
---
 .github/workflows/tests.yml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index c1557d5587..ea55f0a1b1 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -18,9 +18,9 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v4
-    - run: pip install --user ruff
-    # Excluding `/gyp` directory as it is been checked in https://github.com/nodejs/gyp-next/ already
-    - run: ruff check --output-format=github --extend-exclude=gyp --select="E,F,PLC,PLE,UP,W,YTT" --ignore="E721,PLC1901,S101,UP031" --target-version=py38 .
+    - uses: astral-sh/ruff-action@v3
+      with:
+        args: "check --select=E,F,PLC,PLE,UP,W,YTT --ignore=E721,PLC0206,PLC1901,S101,UP031 --target-version=py39"
 
   lint-js:
     name: Lint JS

From 44cea2e09e84bdafef7f5fad38989c17777b587f Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Mon, 13 Jan 2025 17:33:44 +0100
Subject: [PATCH 454/551] Test on GitHub Actions `windows-2025` image (#3116)

* DRAFT: Test on GitHub Actions windows-2025 image

* actions/runner-images#11228

* Windows 2025 currently ships with Visual Studio 2022

* Revert changes to lib/find-visualstudio.js

* Revert changes to lib/find-visualstudio.js

* Revert ALL changes to lib/find-visualstudio.js
---
 .github/workflows/tests.yml         | 19 +++++++++++--------
 .github/workflows/visual-studio.yml |  2 ++
 2 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index ea55f0a1b1..b02171e707 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -92,14 +92,17 @@ jobs:
     needs: [lint-python]
     strategy:
       fail-fast: false
-      max-parallel: 15
+      max-parallel: 11
       matrix:
         os: [macos-latest, ubuntu-latest, windows-latest]
-        python: ["3.8", "3.10", "3.12"]
+        python: ["3.9", "3.11", "3.13"]
         node: [18.x, 20.x, 22.x]
-        include:  # `npm test` runs Windows find-visualstudio tests on an Intel Mac!!!
+        include:
           - os: macos-13
-            python: "3.12"
+            python: "3.13"
+            node: 22.x
+          - os: windows-2025
+            python: "3.13"
             node: 22.x
     name: ${{ matrix.os }} - ${{ matrix.python }} - ${{ matrix.node }}
     runs-on: ${{ matrix.os }}
@@ -129,14 +132,14 @@ jobs:
       - name: Run Python Tests
         run: python -m pytest
       - name: Run Tests (macOS or Linux)
-        if: "!startsWith(matrix.os, 'windows')"
+        if: runner.os != 'Windows'
         shell: bash
         run: npm test --python="${pythonLocation}/python"
         env:
-          FULL_TEST: ${{ (matrix.node == '22.x' && matrix.python == '3.12') && '1' || '0' }}
+          FULL_TEST: ${{ (matrix.node == '22.x' && matrix.python == '3.13') && '1' || '0' }}
       - name: Run Tests (Windows)
-        if: startsWith(matrix.os, 'windows')
+        if: runner.os == 'Windows'
         shell: bash # Building wasm on Windows requires using make generator, it only works in bash
         run: npm run test --python="${pythonLocation}\\python.exe"
         env:
-          FULL_TEST: ${{ (matrix.node == '22.x' && matrix.python == '3.12') && '1' || '0' }}
+          FULL_TEST: ${{ (matrix.node == '22.x' && matrix.python == '3.13') && '1' || '0' }}
diff --git a/.github/workflows/visual-studio.yml b/.github/workflows/visual-studio.yml
index 0f51f9c8d0..3412837422 100644
--- a/.github/workflows/visual-studio.yml
+++ b/.github/workflows/visual-studio.yml
@@ -21,6 +21,8 @@ jobs:
             msvs-verison: 2019
           - os: windows-2022
             msvs-version: 2022
+          - os: windows-2025
+            msvs-version: 2022  # Fix this when Visual Studio 2025 is released
     runs-on: ${{ matrix.os }}
     steps:
       - name: Checkout Repository

From e3f9a7756f65a7f4e50799017b3dc51d5bc195b2 Mon Sep 17 00:00:00 2001
From: Chengzhong Wu 
Date: Tue, 28 Jan 2025 21:07:07 +0000
Subject: [PATCH 455/551] chore: add gyp-next updater (#3105)

* chore: add gyp-next updater

Co-authored-by: Christian Clauss 
---
 .github/workflows/release-please.yml  |  5 ++-
 .github/workflows/update-gyp-next.yml | 51 +++++++++++++++++++++++++++
 update-gyp.py                         | 11 ++++--
 3 files changed, 62 insertions(+), 5 deletions(-)
 create mode 100644 .github/workflows/update-gyp-next.yml

diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml
index 2bef09ec91..5c6a1ec615 100644
--- a/.github/workflows/release-please.yml
+++ b/.github/workflows/release-please.yml
@@ -9,13 +9,12 @@ jobs:
   release-please:
     outputs:
       release_created:  ${{ steps.release.outputs.release_created }}
-    permissions:
-      contents: write # to create release commit (googleapis/release-please-action)
-      pull-requests: write # to create release PR (googleapis/release-please-action)
     runs-on: ubuntu-latest
     steps:
       - uses: googleapis/release-please-action@v4
         id: release
+        with:
+          token: ${{ secrets.GH_USER_TOKEN }}
           # Standard Conventional Commits: `feat` and `fix`
           # node-gyp subdirectories: `bin`, `gyp`, `lib`, `src`, `test`
           # node-gyp subcommands: `build`, `clean`, `configure`, `install`, `list`, `rebuild`, `remove`
diff --git a/.github/workflows/update-gyp-next.yml b/.github/workflows/update-gyp-next.yml
new file mode 100644
index 0000000000..f43be6c70c
--- /dev/null
+++ b/.github/workflows/update-gyp-next.yml
@@ -0,0 +1,51 @@
+name: Update gyp-next
+
+on:
+  schedule:
+    # Run once a week at 12:00 AM UTC on Sunday.
+    - cron: 0 0 * * *
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+env:
+  NODE_VERSION: lts/*
+
+jobs:
+  update-gyp-next:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+
+      - uses: actions/github-script@v7
+        id: get-gyp-next-version
+        with:
+          script: |
+            const result = await github.rest.repos.getLatestRelease({
+              owner: 'nodejs',
+              repo: 'gyp-next',
+            });
+            return result.data.tag_name
+          result-encoding: string
+
+      - name: Update gyp-next
+        run: |
+          python update-gyp.py --no-commit ${{ steps.get-gyp-next-version.outputs.result }}
+
+      - name: Open or update PR for the gyp-next update
+        uses: gr2m/create-or-update-pull-request-action@v1
+        with:
+          branch: actions/update-gyp-next
+          author: Node.js GitHub Bot 
+          title: 'feat: update gyp-next to ${{ steps.get-gyp-next-version.outputs.result }}'
+          commit-message: 'feat: update gyp-next to ${{ steps.get-gyp-next-version.outputs.result }}'
+          update-pull-request-title-and-body: true
+          body: >
+            This is an automated update of the gyp-next to
+            https://github.com/nodejs/gyp-next/releases/tag/${{ steps.get-gyp-next-version.outputs.result }}.
+        env:
+          GITHUB_TOKEN: ${{ secrets.GH_USER_TOKEN }}
diff --git a/update-gyp.py b/update-gyp.py
index 70e2d10028..c65da41472 100755
--- a/update-gyp.py
+++ b/update-gyp.py
@@ -13,6 +13,10 @@
 CHECKOUT_GYP_PATH = os.path.join(CHECKOUT_PATH, "gyp")
 
 parser = argparse.ArgumentParser()
+parser.add_argument("--no-commit",
+    action="store_true",
+    dest="no_commit",
+    help="do not run git-commit")
 parser.add_argument("tag", help="gyp tag to update to")
 args = parser.parse_args()
 
@@ -60,5 +64,8 @@ def safe_extract(tar, path=".", members=None, *, numeric_owner=False):
             os.path.join(unzip_target, os.listdir(unzip_target)[0]), CHECKOUT_GYP_PATH
         )
 
-subprocess.check_output(["git", "add", "gyp"], cwd=CHECKOUT_PATH)
-subprocess.check_output(["git", "commit", "-m", "feat(gyp): update gyp to " + args.tag])
+if not args.no_commit:
+  subprocess.check_output(["git", "add", "gyp"], cwd=CHECKOUT_PATH)
+  subprocess.check_output([
+    "git", "commit", "-m", f"feat(gyp): update gyp to {args.tag}"
+  ])

From 2530f51cec3ba595184e5bcb7fe1245e240beb59 Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Sun, 2 Feb 2025 11:12:49 +0100
Subject: [PATCH 456/551] chore: Test on Ubuntu-24.04-arm and Node.js v23
 (#3121)

---
 .github/workflows/tests.yml | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index b02171e707..f68abd41ef 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -94,16 +94,19 @@ jobs:
       fail-fast: false
       max-parallel: 11
       matrix:
-        os: [macos-latest, ubuntu-latest, windows-latest]
+        os: [windows-latest, macos-latest, ubuntu-latest]
         python: ["3.9", "3.11", "3.13"]
-        node: [18.x, 20.x, 22.x]
+        node: [18.x, 20.x, 22.x, 23.x]
         include:
           - os: macos-13
             python: "3.13"
-            node: 22.x
+            node: 23.x
+          - os: ubuntu-24.04-arm 
+            python: "3.13"
+            node: 23.x
           - os: windows-2025
             python: "3.13"
-            node: 22.x
+            node: 23.x
     name: ${{ matrix.os }} - ${{ matrix.python }} - ${{ matrix.node }}
     runs-on: ${{ matrix.os }}
     steps:
@@ -119,13 +122,13 @@ jobs:
           python-version: ${{ matrix.python }}
         env:
           PYTHON_VERSION: ${{ matrix.python }}  # Why do this?
-      - uses: seanmiddleditch/gha-setup-ninja@v5
+      - uses: seanmiddleditch/gha-setup-ninja@v6
       - name: Install Dependencies
         run: |
           npm install
           pip install pytest
       - name: Set Windows Env
-        if: startsWith(matrix.os, 'windows')
+        if: runner.os == 'Windows'
         run: |
           echo 'GYP_MSVS_VERSION=2015' >> $Env:GITHUB_ENV
           echo 'GYP_MSVS_OVERRIDE_PATH=C:\\Dummy' >> $Env:GITHUB_ENV

From 504250e5e3e27c6ef6dcfcaa744b36e1a99c1be8 Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Mon, 10 Feb 2025 05:12:24 -0500
Subject: [PATCH 457/551] feat: update gyp-next to v0.19.1 (#3122)

---
 gyp/.github/workflows/node-gyp.yml            |   5 +-
 gyp/.github/workflows/nodejs-windows.yml      |  32 -----
 gyp/.github/workflows/nodejs.yml              |  51 ++++++++
 .../{Python_tests.yml => python_tests.yml}    |   7 +-
 gyp/.github/workflows/release-please.yml      |   2 +-
 gyp/.release-please-manifest.json             |   2 +-
 gyp/CHANGELOG.md                              |  29 +++++
 gyp/docs/Hacking.md                           |   2 +-
 gyp/docs/LanguageSpecification.md             |   6 +-
 gyp/docs/Testing.md                           |   6 +-
 gyp/docs/UserDocumentation.md                 |   8 +-
 gyp/pylib/gyp/MSVSSettings.py                 |   2 +-
 gyp/pylib/gyp/MSVSVersion.py                  |   2 +-
 gyp/pylib/gyp/__init__.py                     |  33 +++--
 gyp/pylib/gyp/generator/analyzer.py           |   2 +-
 gyp/pylib/gyp/generator/android.py            |   2 +-
 gyp/pylib/gyp/generator/cmake.py              |  12 +-
 gyp/pylib/gyp/generator/make.py               |  32 ++---
 gyp/pylib/gyp/generator/msvs.py               |  25 ++--
 gyp/pylib/gyp/generator/ninja.py              |  29 +++--
 gyp/pylib/gyp/generator/xcode.py              |   2 +-
 gyp/pylib/gyp/input.py                        | 118 +++++++++---------
 gyp/pylib/gyp/xcode_emulation.py              |  15 ++-
 gyp/pylib/gyp/xcodeproj_file.py               |  20 +--
 gyp/pylib/packaging/metadata.py               |  14 +--
 gyp/pyproject.toml                            |   3 +-
 gyp/tools/pretty_sln.py                       |   4 +-
 gyp/tools/pretty_vcproj.py                    |   8 +-
 28 files changed, 270 insertions(+), 203 deletions(-)
 delete mode 100644 gyp/.github/workflows/nodejs-windows.yml
 create mode 100644 gyp/.github/workflows/nodejs.yml
 rename gyp/.github/workflows/{Python_tests.yml => python_tests.yml} (90%)

diff --git a/gyp/.github/workflows/node-gyp.yml b/gyp/.github/workflows/node-gyp.yml
index b960fcf2b7..4382979fc3 100644
--- a/gyp/.github/workflows/node-gyp.yml
+++ b/gyp/.github/workflows/node-gyp.yml
@@ -1,12 +1,11 @@
 name: node-gyp integration
 on:
   push:
-    branches: [ main ]
   pull_request:
-    branches: [ main ]
   workflow_dispatch:
+
 jobs:
-  integration:
+  node-gyp-integration:
     strategy:
       fail-fast: false
       matrix:
diff --git a/gyp/.github/workflows/nodejs-windows.yml b/gyp/.github/workflows/nodejs-windows.yml
deleted file mode 100644
index 3f52ff9ce7..0000000000
--- a/gyp/.github/workflows/nodejs-windows.yml
+++ /dev/null
@@ -1,32 +0,0 @@
-name: Node.js Windows integration
-
-on:
-  push:
-    branches: [ main ]
-  pull_request:
-    branches: [ main ]
-  workflow_dispatch:
-
-jobs:
-  build-windows:
-    runs-on: windows-latest
-    steps:
-      - name: Clone gyp-next
-        uses: actions/checkout@v4
-        with:
-          path: gyp-next
-      - name: Clone nodejs/node
-        uses: actions/checkout@v4
-        with:
-          repository: nodejs/node
-          path: node
-      - name: Install deps
-        run: choco install nasm
-      - name: Replace gyp in Node.js
-        run: |
-          rm -Recurse node/tools/gyp
-          cp -Recurse gyp-next node/tools/gyp
-      - name: Build Node.js
-        run: |
-          cd node
-          ./vcbuild.bat
diff --git a/gyp/.github/workflows/nodejs.yml b/gyp/.github/workflows/nodejs.yml
new file mode 100644
index 0000000000..53935d4069
--- /dev/null
+++ b/gyp/.github/workflows/nodejs.yml
@@ -0,0 +1,51 @@
+name: Node.js integration
+on:
+  push:
+  pull_request:
+  workflow_dispatch:
+
+jobs:
+  nodejs-integration:
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [macos-13, macos-latest, ubuntu-latest, windows-latest]
+        python: ["3.8", "3.10", "3.12", "3.13"]
+
+    runs-on: ${{ matrix.os }}
+    steps:
+      - name: Clone gyp-next
+        uses: actions/checkout@v4
+        with:
+          path: gyp-next
+      - name: Clone nodejs/node
+        uses: actions/checkout@v4
+        with:
+          repository: nodejs/node
+          path: node
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+          allow-prereleases: true
+      - name: Replace gyp in Node.js
+        shell: bash
+        run: |
+          rm -rf node/tools/gyp
+          cp -r gyp-next node/tools/gyp
+
+      # macOS and Linux
+      - name: Run configure
+        if: runner.os != 'Windows'
+        run: |
+          cd node
+          ./configure
+
+      # Windows
+      - name: Install deps
+        if: runner.os == 'Windows'
+        run: choco install nasm
+      - name: Run configure
+        if: runner.os == 'Windows'
+        run: |
+          cd node
+          ./vcbuild.bat nobuild
diff --git a/gyp/.github/workflows/Python_tests.yml b/gyp/.github/workflows/python_tests.yml
similarity index 90%
rename from gyp/.github/workflows/Python_tests.yml
rename to gyp/.github/workflows/python_tests.yml
index 5c86466378..507d4d2d75 100644
--- a/gyp/.github/workflows/Python_tests.yml
+++ b/gyp/.github/workflows/python_tests.yml
@@ -4,10 +4,9 @@
 name: Python_tests
 on:
   push:
-    branches: [ main ]
   pull_request:
-    branches: [ main ]
   workflow_dispatch:
+
 jobs:
   Python_tests:
     runs-on: ${{ matrix.os }}
@@ -24,14 +23,14 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
           allow-prereleases: true
-      - uses: seanmiddleditch/gha-setup-ninja@v4
+      - uses: seanmiddleditch/gha-setup-ninja@v5
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip setuptools
           pip install --editable ".[dev]"
       - run: ./gyp -V && ./gyp --version && gyp -V && gyp --version
       - name: Lint with ruff  # See pyproject.toml for settings
-        run: ruff --output-format=github .
+        run: ruff check --output-format=github .
       - name: Test with pytest  # See pyproject.toml for settings
         run: pytest
       # - name: Run doctests with pytest
diff --git a/gyp/.github/workflows/release-please.yml b/gyp/.github/workflows/release-please.yml
index 95dadd53bb..a96a88fe68 100644
--- a/gyp/.github/workflows/release-please.yml
+++ b/gyp/.github/workflows/release-please.yml
@@ -80,7 +80,7 @@ jobs:
         name: python-package-distributions
         path: dist/
     - name: Sign the dists with Sigstore
-      uses: sigstore/gh-action-sigstore-python@v2.1.1
+      uses: sigstore/gh-action-sigstore-python@v3.0.0
       with:
         inputs: >-
           ./dist/*.tar.gz
diff --git a/gyp/.release-please-manifest.json b/gyp/.release-please-manifest.json
index cbd0ca0683..1f9113816b 100644
--- a/gyp/.release-please-manifest.json
+++ b/gyp/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "0.18.1"
+    ".": "0.19.1"
 }
diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md
index 8a8282a5f8..22257ab93d 100644
--- a/gyp/CHANGELOG.md
+++ b/gyp/CHANGELOG.md
@@ -1,5 +1,34 @@
 # Changelog
 
+## [0.19.1](https://github.com/nodejs/gyp-next/compare/v0.19.0...v0.19.1) (2024-12-09)
+
+
+### Bug Fixes
+
+* fixup for break in EscapeForCString ([#274](https://github.com/nodejs/gyp-next/issues/274)) ([610f661](https://github.com/nodejs/gyp-next/commit/610f661da877a358c8b3cbc106b528fb1d0b8095))
+
+## [0.19.0](https://github.com/nodejs/gyp-next/compare/v0.18.3...v0.19.0) (2024-12-03)
+
+
+### Features
+
+* provide escaped version of `PRODUCT_DIR_ABS` ([#271](https://github.com/nodejs/gyp-next/issues/271)) ([3bf3b1c](https://github.com/nodejs/gyp-next/commit/3bf3b1cda26f16c645e0fdd5582ffbf49d9a2580))
+
+## [0.18.3](https://github.com/nodejs/gyp-next/compare/v0.18.2...v0.18.3) (2024-10-08)
+
+
+### Bug Fixes
+
+* enable pch for clang on windows ([#268](https://github.com/nodejs/gyp-next/issues/268)) ([cc5838c](https://github.com/nodejs/gyp-next/commit/cc5838c4e9260bf459d71de53fbb2eebd1a6f508))
+
+## [0.18.2](https://github.com/nodejs/gyp-next/compare/v0.18.1...v0.18.2) (2024-09-23)
+
+
+### Bug Fixes
+
+* do not assume that /usr/bin/env exists on macOS ([#216](https://github.com/nodejs/gyp-next/issues/216)) ([706d04a](https://github.com/nodejs/gyp-next/commit/706d04aba5bd18f311dc56f84720e99f64c73466))
+* fix E721 lint errors ([#206](https://github.com/nodejs/gyp-next/issues/206)) ([d1299a4](https://github.com/nodejs/gyp-next/commit/d1299a49d313eccabecf97ccb56fc033afad39ad))
+
 ## [0.18.1](https://github.com/nodejs/gyp-next/compare/v0.18.0...v0.18.1) (2024-05-26)
 
 
diff --git a/gyp/docs/Hacking.md b/gyp/docs/Hacking.md
index 89b3b8bea9..b00783bd36 100644
--- a/gyp/docs/Hacking.md
+++ b/gyp/docs/Hacking.md
@@ -34,7 +34,7 @@ See [Testing](Testing.md) for more details on the test framework.
 Note that it can be handy to look at the project files output by the tests
 to diagnose problems. The easiest way to do that is by kindly asking the
 test driver to leave the temporary directories it creates in-place.
-This is done by setting the enviroment variable "PRESERVE", e.g.
+This is done by setting the environment variable "PRESERVE", e.g.
 
 ```
 set PRESERVE=all     # On Windows
diff --git a/gyp/docs/LanguageSpecification.md b/gyp/docs/LanguageSpecification.md
index 178b8c8316..f8fff097ab 100644
--- a/gyp/docs/LanguageSpecification.md
+++ b/gyp/docs/LanguageSpecification.md
@@ -157,7 +157,7 @@ have structural meaning for target definitions:
 | `all_dependent_settings`     | A dictionary of settings to be applied to all dependents of the target, transitively.  This includes direct dependents and the entire set of their dependents, and so on.  This section may contain anything found within a `target` dictionary, except `configurations`, `target_name`, and `type` sections.  Compare `direct_dependent_settings` and `link_settings`. |
 | `configurations`             | A list of dictionaries defining build configurations for the target.  See the "Configurations" section below.  |
 | `copies`                     | A list of copy actions to perform. See the "Copies" section below. |
-| `defines`                    | A list of preprocesor definitions to be passed on the command line to the C/C++ compiler (via `-D` or `/D` options). |
+| `defines`                    | A list of preprocessor definitions to be passed on the command line to the C/C++ compiler (via `-D` or `/D` options). |
 | `dependencies`               | A list of targets on which this target depends.  Targets in other `.gyp` files are specified as `../path/to/other.gyp:target_we_want`. |
 | `direct_dependent_settings`  | A dictionary of settings to be applied to other targets that depend on this target.  These settings will only be applied to direct dependents.  This section may contain anything found within a `target` dictionary, except `configurations`, `target_name`, and `type` sections.  Compare with `all_dependent_settings` and `link_settings`. |
 | `include_dirs`               | A list of include directories to be passed on the command line to the C/C++ compiler (via `-I` or `/I` options). |
@@ -208,8 +208,8 @@ Configuration dictionaries may also contain these elements:
 
 Conditionals may appear within any dictionary in a `.gyp` file.  There
 are two tpes of conditionals, which differ only in the timing of their
-processing.  `conditons` sections are processed shortly after loading
-`.gyp` files, and `target_conditons` sections are processed after all
+processing.  `conditions` sections are processed shortly after loading
+`.gyp` files, and `target_conditions` sections are processed after all
 dependencies have been computed.
 
 A conditional section is introduced with a `conditions` or
diff --git a/gyp/docs/Testing.md b/gyp/docs/Testing.md
index baeb65f944..a52031e888 100644
--- a/gyp/docs/Testing.md
+++ b/gyp/docs/Testing.md
@@ -392,7 +392,7 @@ fails the test if it does.
 
 Verifies that the output string contains all of the "lines" in the specified
 list of lines.  In practice, the lines can be any substring and need not be
-`\n`-terminaed lines per se.  If any line is missing, the test fails.
+`\n`-terminated lines per se.  If any line is missing, the test fails.
 
 ```
   test.must_not_contain_any_lines(output, lines)
@@ -400,7 +400,7 @@ list of lines.  In practice, the lines can be any substring and need not be
 
 Verifies that the output string does _not_ contain any of the "lines" in the
 specified list of lines.  In practice, the lines can be any substring and need
-not be `\n`-terminaed lines per se.  If any line exists in the output string,
+not be `\n`-terminated lines per se.  If any line exists in the output string,
 the test fails.
 
 ```
@@ -409,7 +409,7 @@ the test fails.
 
 Verifies that the output string contains at least one of the "lines" in the
 specified list of lines.  In practice, the lines can be any substring and need
-not be `\n`-terminaed lines per se.  If none of the specified lines is present,
+not be `\n`-terminated lines per se.  If none of the specified lines is present,
 the test fails.
 
 ### Reading file contents
diff --git a/gyp/docs/UserDocumentation.md b/gyp/docs/UserDocumentation.md
index 808f37a1a9..b9d412e1c8 100644
--- a/gyp/docs/UserDocumentation.md
+++ b/gyp/docs/UserDocumentation.md
@@ -104,7 +104,7 @@ describing all the information necessary to build the target.
 
 `'conditions'`:  A list of condition specifications that can modify the
 contents of the items in the global dictionary defined by this `.gyp`
-file based on the values of different variablwes.  As implied by the
+file based on the values of different variables.  As implied by the
 above example, the most common use of a `conditions` section in the
 top-level dictionary is to add platform-specific targets to the
 `targets` list.
@@ -375,7 +375,7 @@ If your platform-specific file does not contain a
 already in the `conditions` for the target), and you can't change the
 file name, there are two patterns that can be used.
 
-**Prefererred**:  Add the file to the `sources` list of the appropriate
+**Preferred**:  Add the file to the `sources` list of the appropriate
 dictionary within the `targets` list.  Add an appropriate `conditions`
 section to exclude the specific files name:
 
@@ -807,7 +807,7 @@ directory:
 ```
 
 Adding a library often involves updating multiple `.gyp` files, adding
-the target to the approprate `.gyp` file (possibly a newly-added `.gyp`
+the target to the appropriate `.gyp` file (possibly a newly-added `.gyp`
 file), and updating targets in the other `.gyp` files that depend on
 (link with) the new library.
 
@@ -858,7 +858,7 @@ because of those settings' being listed in the
 `direct_dependent_settings` block.
 
 Note that these settings will likely need to be replicated in the
-settings for the library target itsef, so that the library will build
+settings for the library target itself, so that the library will build
 with the same options.  This does not prevent the target from defining
 additional options for its "internal" use when compiling its own source
 files.  (In the above example, these are the `LOCAL_DEFINE_FOR_LIBBAR`
diff --git a/gyp/pylib/gyp/MSVSSettings.py b/gyp/pylib/gyp/MSVSSettings.py
index ac87f572b2..fea6e67286 100644
--- a/gyp/pylib/gyp/MSVSSettings.py
+++ b/gyp/pylib/gyp/MSVSSettings.py
@@ -171,7 +171,7 @@ def ValidateMSBuild(self, value):
         int(value, self._msbuild_base)
 
     def ConvertToMSBuild(self, value):
-        msbuild_format = (self._msbuild_base == 10) and "%d" or "0x%04x"
+        msbuild_format = ((self._msbuild_base == 10) and "%d") or "0x%04x"
         return msbuild_format % int(value)
 
 
diff --git a/gyp/pylib/gyp/MSVSVersion.py b/gyp/pylib/gyp/MSVSVersion.py
index 8d7f21e82d..1b35362922 100644
--- a/gyp/pylib/gyp/MSVSVersion.py
+++ b/gyp/pylib/gyp/MSVSVersion.py
@@ -69,7 +69,7 @@ def UsesVcxproj(self):
 
     def ProjectExtension(self):
         """Returns the file extension for the project."""
-        return self.uses_vcxproj and ".vcxproj" or ".vcproj"
+        return (self.uses_vcxproj and ".vcxproj") or ".vcproj"
 
     def Path(self):
         """Returns the path to Visual Studio installation."""
diff --git a/gyp/pylib/gyp/__init__.py b/gyp/pylib/gyp/__init__.py
index d6cc01307d..8933d0c4f7 100755
--- a/gyp/pylib/gyp/__init__.py
+++ b/gyp/pylib/gyp/__init__.py
@@ -4,7 +4,7 @@
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
-
+from __future__ import annotations
 import copy
 import gyp.input
 import argparse
@@ -24,6 +24,18 @@
 DEBUG_VARIABLES = "variables"
 DEBUG_INCLUDES = "includes"
 
+def EscapeForCString(string: bytes | str) -> str:
+    if isinstance(string, str):
+        string = string.encode(encoding='utf8')
+
+    backslash_or_double_quote = {ord('\\'), ord('"')}
+    result = ''
+    for char in string:
+        if char in backslash_or_double_quote or not 32 <= char < 127:
+            result += '\\%03o' % char
+        else:
+            result += chr(char)
+    return result
 
 def DebugOutput(mode, message, *args):
     if "all" in gyp.debug or mode in gyp.debug:
@@ -106,18 +118,19 @@ def Load(
 
     output_dir = params["options"].generator_output or params["options"].toplevel_dir
     if default_variables["GENERATOR"] == "ninja":
-        default_variables.setdefault(
-            "PRODUCT_DIR_ABS",
-            os.path.join(
-                output_dir, "out", default_variables.get("build_type", "default")
-            ),
+        product_dir_abs = os.path.join(
+            output_dir, "out", default_variables.get("build_type", "default")
         )
     else:
-        default_variables.setdefault(
-            "PRODUCT_DIR_ABS",
-            os.path.join(output_dir, default_variables["CONFIGURATION_NAME"]),
+        product_dir_abs = os.path.join(
+            output_dir, default_variables["CONFIGURATION_NAME"]
         )
 
+    default_variables.setdefault("PRODUCT_DIR_ABS", product_dir_abs)
+    default_variables.setdefault(
+        "PRODUCT_DIR_ABS_CSTR", EscapeForCString(product_dir_abs)
+    )
+
     # Give the generator the opportunity to set additional variables based on
     # the params it will receive in the output phase.
     if getattr(generator, "CalculateVariables", None):
@@ -253,7 +266,7 @@ def Noop(value):
     for name, metadata in options._regeneration_metadata.items():
         opt = metadata["opt"]
         value = getattr(options, name)
-        value_predicate = metadata["type"] == "path" and FixPath or Noop
+        value_predicate = (metadata["type"] == "path" and FixPath) or Noop
         action = metadata["action"]
         env_name = metadata["env_name"]
         if action == "append":
diff --git a/gyp/pylib/gyp/generator/analyzer.py b/gyp/pylib/gyp/generator/analyzer.py
index 1334f2fca9..64573ad2cc 100644
--- a/gyp/pylib/gyp/generator/analyzer.py
+++ b/gyp/pylib/gyp/generator/analyzer.py
@@ -699,7 +699,7 @@ def find_matching_test_target_names(self):
         ) & set(self._root_targets)
         if matching_test_targets_contains_all:
             # Remove any of the targets for all that were not explicitly supplied,
-            # 'all' is subsequentely added to the matching names below.
+            # 'all' is subsequently added to the matching names below.
             matching_test_targets = list(
                 set(matching_test_targets) & set(test_targets_no_all)
             )
diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py
index 2a63f412db..64da385e6a 100644
--- a/gyp/pylib/gyp/generator/android.py
+++ b/gyp/pylib/gyp/generator/android.py
@@ -769,7 +769,7 @@ def ExtractIncludesFromCFlags(self, cflags):
         Args:
           cflags: A list of compiler flags, which may be mixed with "-I.."
         Returns:
-          A tuple of lists: (clean_clfags, include_paths). "-I.." is trimmed.
+          A tuple of lists: (clean_cflags, include_paths). "-I.." is trimmed.
         """
         clean_cflags = []
         include_paths = []
diff --git a/gyp/pylib/gyp/generator/cmake.py b/gyp/pylib/gyp/generator/cmake.py
index 320a891aa8..8720a3daf3 100644
--- a/gyp/pylib/gyp/generator/cmake.py
+++ b/gyp/pylib/gyp/generator/cmake.py
@@ -251,7 +251,7 @@ def WriteActions(target_name, actions, extra_sources, extra_deps, path_to_gyp, o
     target_name: the name of the CMake target being generated.
     actions: the Gyp 'actions' dict for this target.
     extra_sources: [(, )] to append with generated source files.
-    extra_deps: [] to append with generated targets.
+    extra_deps: [] to append with generated targets.
     path_to_gyp: relative path from CMakeLists.txt being generated to
         the Gyp file in which the target being generated is defined.
   """
@@ -340,7 +340,7 @@ def WriteRules(target_name, rules, extra_sources, extra_deps, path_to_gyp, outpu
     target_name: the name of the CMake target being generated.
     actions: the Gyp 'actions' dict for this target.
     extra_sources: [(, )] to append with generated source files.
-    extra_deps: [] to append with generated targets.
+    extra_deps: [] to append with generated targets.
     path_to_gyp: relative path from CMakeLists.txt being generated to
         the Gyp file in which the target being generated is defined.
   """
@@ -457,7 +457,7 @@ def WriteCopies(target_name, copies, extra_deps, path_to_gyp, output):
   Args:
     target_name: the name of the CMake target being generated.
     actions: the Gyp 'actions' dict for this target.
-    extra_deps: [] to append with generated targets.
+    extra_deps: [] to append with generated targets.
     path_to_gyp: relative path from CMakeLists.txt being generated to
         the Gyp file in which the target being generated is defined.
   """
@@ -603,7 +603,7 @@ class CMakeNamer:
   """
 
     def __init__(self, target_list):
-        self.cmake_target_base_names_conficting = set()
+        self.cmake_target_base_names_conflicting = set()
 
         cmake_target_base_names_seen = set()
         for qualified_target in target_list:
@@ -612,11 +612,11 @@ def __init__(self, target_list):
             if cmake_target_base_name not in cmake_target_base_names_seen:
                 cmake_target_base_names_seen.add(cmake_target_base_name)
             else:
-                self.cmake_target_base_names_conficting.add(cmake_target_base_name)
+                self.cmake_target_base_names_conflicting.add(cmake_target_base_name)
 
     def CreateCMakeTargetName(self, qualified_target):
         base_name = CreateCMakeTargetBaseName(qualified_target)
-        if base_name in self.cmake_target_base_names_conficting:
+        if base_name in self.cmake_target_base_names_conflicting:
             return CreateCMakeTargetFullName(qualified_target)
         return base_name
 
diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py
index 392d900914..634da8973c 100644
--- a/gyp/pylib/gyp/generator/make.py
+++ b/gyp/pylib/gyp/generator/make.py
@@ -208,7 +208,7 @@ def CalculateGeneratorInputInfo(params):
 
 LINK_COMMANDS_MAC = """\
 quiet_cmd_alink = LIBTOOL-STATIC $@
-cmd_alink = rm -f $@ && ./gyp-mac-tool filter-libtool libtool $(GYP_LIBTOOLFLAGS) -static -o $@ $(filter %.o,$^)
+cmd_alink = rm -f $@ && %(python)s gyp-mac-tool filter-libtool libtool $(GYP_LIBTOOLFLAGS) -static -o $@ $(filter %%.o,$^)
 
 quiet_cmd_link = LINK($(TOOLSET)) $@
 cmd_link = $(LINK.$(TOOLSET)) $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o "$@" $(LD_INPUTS) $(LIBS)
@@ -218,7 +218,7 @@ def CalculateGeneratorInputInfo(params):
 
 quiet_cmd_solink_module = SOLINK_MODULE($(TOOLSET)) $@
 cmd_solink_module = $(LINK.$(TOOLSET)) -bundle $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ $(filter-out FORCE_DO_CMD, $^) $(LIBS)
-"""  # noqa: E501
+""" % {'python': sys.executable}  # noqa: E501
 
 LINK_COMMANDS_ANDROID = """\
 quiet_cmd_alink = AR($(TOOLSET)) $@
@@ -609,14 +609,14 @@ def CalculateGeneratorInputInfo(params):
 # Use $(4) for the command, since $(2) and $(3) are used as flag by do_cmd
 # already.
 quiet_cmd_mac_tool = MACTOOL $(4) $<
-cmd_mac_tool = ./gyp-mac-tool $(4) $< "$@"
+cmd_mac_tool = %(python)s gyp-mac-tool $(4) $< "$@"
 
 quiet_cmd_mac_package_framework = PACKAGE FRAMEWORK $@
-cmd_mac_package_framework = ./gyp-mac-tool package-framework "$@" $(4)
+cmd_mac_package_framework = %(python)s gyp-mac-tool package-framework "$@" $(4)
 
 quiet_cmd_infoplist = INFOPLIST $@
 cmd_infoplist = $(CC.$(TOOLSET)) -E -P -Wno-trigraphs -x c $(INFOPLIST_DEFINES) "$<" -o "$@"
-"""  # noqa: E501
+""" % {'python': sys.executable}  # noqa: E501
 
 
 def WriteRootHeaderSuffixRules(writer):
@@ -788,7 +788,7 @@ def __init__(self, generator_flags, flavor):
         self.suffix_rules_objdir2 = {}
 
         # Generate suffix rules for all compilable extensions.
-        for ext in COMPILABLE_EXTENSIONS:
+        for ext, value in COMPILABLE_EXTENSIONS.items():
             # Suffix rules for source folder.
             self.suffix_rules_srcdir.update(
                 {
@@ -797,7 +797,7 @@ def __init__(self, generator_flags, flavor):
 $(obj).$(TOOLSET)/$(TARGET)/%%.o: $(srcdir)/%%%s FORCE_DO_CMD
 \t@$(call do_cmd,%s,1)
 """
-                        % (ext, COMPILABLE_EXTENSIONS[ext])
+                        % (ext, value)
                     )
                 }
             )
@@ -810,7 +810,7 @@ def __init__(self, generator_flags, flavor):
 $(obj).$(TOOLSET)/$(TARGET)/%%.o: $(obj).$(TOOLSET)/%%%s FORCE_DO_CMD
 \t@$(call do_cmd,%s,1)
 """
-                        % (ext, COMPILABLE_EXTENSIONS[ext])
+                        % (ext, value)
                     )
                 }
             )
@@ -821,7 +821,7 @@ def __init__(self, generator_flags, flavor):
 $(obj).$(TOOLSET)/$(TARGET)/%%.o: $(obj)/%%%s FORCE_DO_CMD
 \t@$(call do_cmd,%s,1)
 """
-                        % (ext, COMPILABLE_EXTENSIONS[ext])
+                        % (ext, value)
                     )
                 }
             )
@@ -1779,13 +1779,13 @@ def WriteTarget(
             # using ":=".
             self.WriteSortedXcodeEnv(self.output, self.GetSortedXcodePostbuildEnv())
 
-            for configname in target_postbuilds:
+            for configname, value in target_postbuilds.items():
                 self.WriteLn(
                     "%s: TARGET_POSTBUILDS_%s := %s"
                     % (
                         QuoteSpaces(self.output),
                         configname,
-                        gyp.common.EncodePOSIXShellList(target_postbuilds[configname]),
+                        gyp.common.EncodePOSIXShellList(value),
                     )
                 )
 
@@ -1834,7 +1834,7 @@ def WriteTarget(
             # Since this target depends on binary and resources which are in
             # nested subfolders, the framework directory will be older than
             # its dependencies usually. To prevent this rule from executing
-            # on every build (expensive, especially with postbuilds), expliclity
+            # on every build (expensive, especially with postbuilds), explicitly
             # update the time on the framework directory.
             self.WriteLn("\t@touch -c %s" % QuoteSpaces(self.output))
 
@@ -2498,7 +2498,7 @@ def CalculateMakefilePath(build_file, base_name):
         "PLI.host": PLI_host,
     }
     if flavor == "mac":
-        flock_command = "./gyp-mac-tool flock"
+        flock_command = "%s gyp-mac-tool flock" % sys.executable
         header_params.update(
             {
                 "flock": flock_command,
@@ -2548,7 +2548,7 @@ def CalculateMakefilePath(build_file, base_name):
         header_params.update(
             {
                 "copy_archive_args": copy_archive_arguments,
-                "flock": "./gyp-flock-tool flock",
+                "flock": "%s gyp-flock-tool flock" % sys.executable,
                 "flock_index": 2,
             }
         )
@@ -2564,7 +2564,7 @@ def CalculateMakefilePath(build_file, base_name):
             {
                 "copy_archive_args": copy_archive_arguments,
                 "link_commands": LINK_COMMANDS_AIX,
-                "flock": "./gyp-flock-tool flock",
+                "flock": "%s gyp-flock-tool flock" % sys.executable,
                 "flock_index": 2,
             }
         )
@@ -2574,7 +2574,7 @@ def CalculateMakefilePath(build_file, base_name):
             {
                 "copy_archive_args": copy_archive_arguments,
                 "link_commands": LINK_COMMANDS_OS400,
-                "flock": "./gyp-flock-tool flock",
+                "flock": "%s gyp-flock-tool flock" % sys.executable,
                 "flock_index": 2,
             }
         )
diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py
index 6b5b24acc0..bea6e64348 100644
--- a/gyp/pylib/gyp/generator/msvs.py
+++ b/gyp/pylib/gyp/generator/msvs.py
@@ -276,7 +276,7 @@ def _ToolAppend(tools, tool_name, setting, value, only_if_unset=False):
 def _ToolSetOrAppend(tools, tool_name, setting, value, only_if_unset=False):
     # TODO(bradnelson): ugly hack, fix this more generally!!!
     if "Directories" in setting or "Dependencies" in setting:
-        if type(value) == str:
+        if isinstance(value, str):
             value = value.replace("/", "\\")
         else:
             value = [i.replace("/", "\\") for i in value]
@@ -288,7 +288,7 @@ def _ToolSetOrAppend(tools, tool_name, setting, value, only_if_unset=False):
     if tool.get(setting):
         if only_if_unset:
             return
-        if type(tool[setting]) == list and type(value) == list:
+        if isinstance(tool[setting], list) and isinstance(value, list):
             tool[setting] += value
         else:
             raise TypeError(
@@ -1423,7 +1423,7 @@ def _ConvertToolsToExpectedForm(tools):
         # Collapse settings with lists.
         settings_fixed = {}
         for setting, value in settings.items():
-            if type(value) == list:
+            if isinstance(value, list):
                 if (
                     tool == "VCLinkerTool" and setting == "AdditionalDependencies"
                 ) or setting == "AdditionalOptions":
@@ -1816,7 +1816,7 @@ def _DictsToFolders(base_path, bucket, flat):
     # Convert to folders recursively.
     children = []
     for folder, contents in bucket.items():
-        if type(contents) == dict:
+        if isinstance(contents, dict):
             folder_children = _DictsToFolders(
                 os.path.join(base_path, folder), contents, flat
             )
@@ -1838,9 +1838,10 @@ def _CollapseSingles(parent, node):
     # Recursively explorer the tree of dicts looking for projects which are
     # the sole item in a folder which has the same name as the project. Bring
     # such projects up one level.
-    if type(node) == dict and len(node) == 1 and next(iter(node)) == parent + ".vcproj":
+    if (isinstance(node, dict) and len(node) == 1 and
+        next(iter(node)) == parent + ".vcproj"):
         return node[next(iter(node))]
-    if type(node) != dict:
+    if not isinstance(node, dict):
         return node
     for child in node:
         node[child] = _CollapseSingles(child, node[child])
@@ -1860,7 +1861,7 @@ def _GatherSolutionFolders(sln_projects, project_objects, flat):
     # Walk down from the top until we hit a folder that has more than one entry.
     # In practice, this strips the top-level "src/" dir from the hierarchy in
     # the solution.
-    while len(root) == 1 and type(root[next(iter(root))]) == dict:
+    while len(root) == 1 and isinstance(root[next(iter(root))], dict):
         root = root[next(iter(root))]
     # Collapse singles.
     root = _CollapseSingles("", root)
@@ -3274,7 +3275,7 @@ def _GetMSBuildPropertyGroup(spec, label, properties):
     num_configurations = len(spec["configurations"])
 
     def GetEdges(node):
-        # Use a definition of edges such that user_of_variable -> used_varible.
+        # Use a definition of edges such that user_of_variable -> used_variable.
         # This happens to be easier in this case, since a variable's
         # definition contains all variables it references in a single string.
         edges = set()
@@ -3411,7 +3412,11 @@ def _FinalizeMSBuildSettings(spec, configuration):
     )
     # Turn on precompiled headers if appropriate.
     if precompiled_header:
-        precompiled_header = os.path.split(precompiled_header)[1]
+        # While MSVC works with just file name eg. "v8_pch.h", ClangCL requires
+        # the full path eg. "tools/msvs/pch/v8_pch.h" to find the file.
+        # P.S. Only ClangCL defines msbuild_toolset, for MSVC it is None.
+        if configuration.get("msbuild_toolset") != 'ClangCL':
+            precompiled_header = os.path.split(precompiled_header)[1]
         _ToolAppend(msbuild_settings, "ClCompile", "PrecompiledHeader", "Use")
         _ToolAppend(
             msbuild_settings, "ClCompile", "PrecompiledHeaderFile", precompiled_header
@@ -3441,7 +3446,7 @@ def _FinalizeMSBuildSettings(spec, configuration):
 
 
 def _GetValueFormattedForMSBuild(tool_name, name, value):
-    if type(value) == list:
+    if isinstance(value, list):
         # For some settings, VS2010 does not automatically extends the settings
         # TODO(jeanluc) Is this what we want?
         if name in [
diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py
index 0146c49962..ae3dded9b4 100644
--- a/gyp/pylib/gyp/generator/ninja.py
+++ b/gyp/pylib/gyp/generator/ninja.py
@@ -2595,9 +2595,9 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name
             "alink",
             description="LIBTOOL-STATIC $out, POSTBUILDS",
             command="rm -f $out && "
-            "./gyp-mac-tool filter-libtool libtool $libtool_flags "
+            "%s gyp-mac-tool filter-libtool libtool $libtool_flags "
             "-static -o $out $in"
-            "$postbuilds",
+            "$postbuilds" % sys.executable,
         )
         master_ninja.rule(
             "lipo",
@@ -2698,41 +2698,44 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name
         master_ninja.rule(
             "copy_infoplist",
             description="COPY INFOPLIST $in",
-            command="$env ./gyp-mac-tool copy-info-plist $in $out $binary $keys",
+            command="$env %s gyp-mac-tool copy-info-plist $in $out $binary $keys"
+            % sys.executable,
         )
         master_ninja.rule(
             "merge_infoplist",
             description="MERGE INFOPLISTS $in",
-            command="$env ./gyp-mac-tool merge-info-plist $out $in",
+            command="$env %s gyp-mac-tool merge-info-plist $out $in" % sys.executable,
         )
         master_ninja.rule(
             "compile_xcassets",
             description="COMPILE XCASSETS $in",
-            command="$env ./gyp-mac-tool compile-xcassets $keys $in",
+            command="$env %s gyp-mac-tool compile-xcassets $keys $in" % sys.executable,
         )
         master_ninja.rule(
             "compile_ios_framework_headers",
             description="COMPILE HEADER MAPS AND COPY FRAMEWORK HEADERS $in",
-            command="$env ./gyp-mac-tool compile-ios-framework-header-map $out "
-            "$framework $in && $env ./gyp-mac-tool "
-            "copy-ios-framework-headers $framework $copy_headers",
+            command="$env %(python)s gyp-mac-tool compile-ios-framework-header-map "
+            "$out $framework $in && $env %(python)s gyp-mac-tool "
+            "copy-ios-framework-headers $framework $copy_headers"
+            % {'python': sys.executable},
         )
         master_ninja.rule(
             "mac_tool",
             description="MACTOOL $mactool_cmd $in",
-            command="$env ./gyp-mac-tool $mactool_cmd $in $out $binary",
+            command="$env %s gyp-mac-tool $mactool_cmd $in $out $binary"
+            % sys.executable,
         )
         master_ninja.rule(
             "package_framework",
             description="PACKAGE FRAMEWORK $out, POSTBUILDS",
-            command="./gyp-mac-tool package-framework $out $version$postbuilds "
-            "&& touch $out",
+            command="%s gyp-mac-tool package-framework $out $version$postbuilds "
+            "&& touch $out" % sys.executable,
         )
         master_ninja.rule(
             "package_ios_framework",
             description="PACKAGE IOS FRAMEWORK $out, POSTBUILDS",
-            command="./gyp-mac-tool package-ios-framework $out $postbuilds "
-            "&& touch $out",
+            command="%s gyp-mac-tool package-ios-framework $out $postbuilds "
+            "&& touch $out" % sys.executable,
         )
     if flavor == "win":
         master_ninja.rule(
diff --git a/gyp/pylib/gyp/generator/xcode.py b/gyp/pylib/gyp/generator/xcode.py
index 1ac672c387..c3c000c4ef 100644
--- a/gyp/pylib/gyp/generator/xcode.py
+++ b/gyp/pylib/gyp/generator/xcode.py
@@ -959,7 +959,7 @@ def GenerateOutput(target_list, target_dicts, data, params):
         #         would-be additional inputs are newer than the output.  Modifying
         #         the source tree - even just modification times - feels dirty.
         # 6564240 Xcode "custom script" build rules always dump all environment
-        #         variables.  This is a low-prioroty problem and is not a
+        #         variables.  This is a low-priority problem and is not a
         #         show-stopper.
         rules_by_ext = {}
         for rule in spec_rules:
diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py
index 7150269cda..5e71fdace0 100644
--- a/gyp/pylib/gyp/input.py
+++ b/gyp/pylib/gyp/input.py
@@ -242,7 +242,7 @@ def LoadOneBuildFile(build_file_path, data, aux_data, includes, is_target, check
         gyp.common.ExceptionAppend(e, "while reading " + build_file_path)
         raise
 
-    if type(build_file_data) is not dict:
+    if not isinstance(build_file_data, dict):
         raise GypError("%s does not evaluate to a dictionary." % build_file_path)
 
     data[build_file_path] = build_file_data
@@ -303,20 +303,20 @@ def LoadBuildFileIncludesIntoDict(
 
     # Recurse into subdictionaries.
     for k, v in subdict.items():
-        if type(v) is dict:
+        if isinstance(v, dict):
             LoadBuildFileIncludesIntoDict(v, subdict_path, data, aux_data, None, check)
-        elif type(v) is list:
+        elif isinstance(v, list):
             LoadBuildFileIncludesIntoList(v, subdict_path, data, aux_data, check)
 
 
 # This recurses into lists so that it can look for dicts.
 def LoadBuildFileIncludesIntoList(sublist, sublist_path, data, aux_data, check):
     for item in sublist:
-        if type(item) is dict:
+        if isinstance(item, dict):
             LoadBuildFileIncludesIntoDict(
                 item, sublist_path, data, aux_data, None, check
             )
-        elif type(item) is list:
+        elif isinstance(item, list):
             LoadBuildFileIncludesIntoList(item, sublist_path, data, aux_data, check)
 
 
@@ -350,9 +350,9 @@ def ProcessToolsetsInDict(data):
         data["targets"] = new_target_list
     if "conditions" in data:
         for condition in data["conditions"]:
-            if type(condition) is list:
+            if isinstance(condition, list):
                 for condition_dict in condition[1:]:
-                    if type(condition_dict) is dict:
+                    if isinstance(condition_dict, dict):
                         ProcessToolsetsInDict(condition_dict)
 
 
@@ -694,7 +694,7 @@ def IsStrCanonicalInt(string):
 
   The canonical form is such that str(int(string)) == string.
   """
-    if type(string) is str:
+    if isinstance(string, str):
         # This function is called a lot so for maximum performance, avoid
         # involving regexps which would otherwise make the code much
         # shorter. Regexps would need twice the time of this function.
@@ -744,7 +744,7 @@ def IsStrCanonicalInt(string):
 
 def FixupPlatformCommand(cmd):
     if sys.platform == "win32":
-        if type(cmd) is list:
+        if isinstance(cmd, list):
             cmd = [re.sub("^cat ", "type ", cmd[0])] + cmd[1:]
         else:
             cmd = re.sub("^cat ", "type ", cmd)
@@ -870,7 +870,8 @@ def ExpandVariables(input, phase, variables, build_file):
         # This works around actions/rules which have more inputs than will
         # fit on the command line.
         if file_list:
-            contents_list = contents if type(contents) is list else contents.split(" ")
+            contents_list = (contents if isinstance(contents, list)
+                             else contents.split(" "))
             replacement = contents_list[0]
             if os.path.isabs(replacement):
                 raise GypError('| cannot handle absolute paths, got "%s"' % replacement)
@@ -1011,7 +1012,7 @@ def ExpandVariables(input, phase, variables, build_file):
 
         if isinstance(replacement, bytes) and not isinstance(replacement, str):
             replacement = replacement.decode("utf-8")  # done on Python 3 only
-        if type(replacement) is list:
+        if isinstance(replacement, list):
             for item in replacement:
                 if isinstance(item, bytes) and not isinstance(item, str):
                     item = item.decode("utf-8")  # done on Python 3 only
@@ -1042,7 +1043,7 @@ def ExpandVariables(input, phase, variables, build_file):
             # Expanding in list context.  It's guaranteed that there's only one
             # replacement to do in |input_str| and that it's this replacement.  See
             # above.
-            if type(replacement) is list:
+            if isinstance(replacement, list):
                 # If it's already a list, make a copy.
                 output = replacement[:]
             else:
@@ -1051,7 +1052,7 @@ def ExpandVariables(input, phase, variables, build_file):
         else:
             # Expanding in string context.
             encoded_replacement = ""
-            if type(replacement) is list:
+            if isinstance(replacement, list):
                 # When expanding a list into string context, turn the list items
                 # into a string in a way that will work with a subprocess call.
                 #
@@ -1081,8 +1082,8 @@ def ExpandVariables(input, phase, variables, build_file):
         # expanding local variables (variables defined in the same
         # variables block as this one).
         gyp.DebugOutput(gyp.DEBUG_VARIABLES, "Found output %r, recursing.", output)
-        if type(output) is list:
-            if output and type(output[0]) is list:
+        if isinstance(output, list):
+            if output and isinstance(output[0], list):
                 # Leave output alone if it's a list of lists.
                 # We don't want such lists to be stringified.
                 pass
@@ -1097,7 +1098,7 @@ def ExpandVariables(input, phase, variables, build_file):
             output = ExpandVariables(output, phase, variables, build_file)
 
     # Convert all strings that are canonically-represented integers into integers.
-    if type(output) is list:
+    if isinstance(output, list):
         for index, outstr in enumerate(output):
             if IsStrCanonicalInt(outstr):
                 output[index] = int(outstr)
@@ -1115,7 +1116,7 @@ def ExpandVariables(input, phase, variables, build_file):
 def EvalCondition(condition, conditions_key, phase, variables, build_file):
     """Returns the dict that should be used or None if the result was
   that nothing should be used."""
-    if type(condition) is not list:
+    if not isinstance(condition, list):
         raise GypError(conditions_key + " must be a list")
     if len(condition) < 2:
         # It's possible that condition[0] won't work in which case this
@@ -1133,12 +1134,12 @@ def EvalCondition(condition, conditions_key, phase, variables, build_file):
     while i < len(condition):
         cond_expr = condition[i]
         true_dict = condition[i + 1]
-        if type(true_dict) is not dict:
+        if not isinstance(true_dict, dict):
             raise GypError(
                 f"{conditions_key} {cond_expr} must be followed by a dictionary, "
                 f"not {type(true_dict)}"
             )
-        if len(condition) > i + 2 and type(condition[i + 2]) is dict:
+        if len(condition) > i + 2 and isinstance(condition[i + 2], dict):
             false_dict = condition[i + 2]
             i = i + 3
             if i != len(condition):
@@ -1239,7 +1240,7 @@ def ProcessConditionsInDict(the_dict, phase, variables, build_file):
         )
 
         if merge_dict is not None:
-            # Expand variables and nested conditinals in the merge_dict before
+            # Expand variables and nested conditionals in the merge_dict before
             # merging it.
             ProcessVariablesAndConditionsInDict(
                 merge_dict, phase, variables, build_file
@@ -1320,7 +1321,7 @@ def ProcessVariablesAndConditionsInDict(
 
     for key, value in the_dict.items():
         # Skip "variables", which was already processed if present.
-        if key != "variables" and type(value) is str:
+        if key != "variables" and isinstance(value, str):
             expanded = ExpandVariables(value, phase, variables, build_file)
             if type(expanded) not in (str, int):
                 raise ValueError(
@@ -1383,21 +1384,21 @@ def ProcessVariablesAndConditionsInDict(
     for key, value in the_dict.items():
         # Skip "variables" and string values, which were already processed if
         # present.
-        if key == "variables" or type(value) is str:
+        if key == "variables" or isinstance(value, str):
             continue
-        if type(value) is dict:
+        if isinstance(value, dict):
             # Pass a copy of the variables dict so that subdicts can't influence
             # parents.
             ProcessVariablesAndConditionsInDict(
                 value, phase, variables, build_file, key
             )
-        elif type(value) is list:
+        elif isinstance(value, list):
             # The list itself can't influence the variables dict, and
             # ProcessVariablesAndConditionsInList will make copies of the variables
             # dict if it needs to pass it to something that can influence it.  No
             # copy is necessary here.
             ProcessVariablesAndConditionsInList(value, phase, variables, build_file)
-        elif type(value) is not int:
+        elif not isinstance(value, int):
             raise TypeError("Unknown type " + value.__class__.__name__ + " for " + key)
 
 
@@ -1406,17 +1407,17 @@ def ProcessVariablesAndConditionsInList(the_list, phase, variables, build_file):
     index = 0
     while index < len(the_list):
         item = the_list[index]
-        if type(item) is dict:
+        if isinstance(item, dict):
             # Make a copy of the variables dict so that it won't influence anything
             # outside of its own scope.
             ProcessVariablesAndConditionsInDict(item, phase, variables, build_file)
-        elif type(item) is list:
+        elif isinstance(item, list):
             ProcessVariablesAndConditionsInList(item, phase, variables, build_file)
-        elif type(item) is str:
+        elif isinstance(item, str):
             expanded = ExpandVariables(item, phase, variables, build_file)
             if type(expanded) in (str, int):
                 the_list[index] = expanded
-            elif type(expanded) is list:
+            elif isinstance(expanded, list):
                 the_list[index : index + 1] = expanded
                 index += len(expanded)
 
@@ -1431,7 +1432,7 @@ def ProcessVariablesAndConditionsInList(the_list, phase, variables, build_file):
                     + " at "
                     + index
                 )
-        elif type(item) is not int:
+        elif not isinstance(item, int):
             raise TypeError(
                 "Unknown type " + item.__class__.__name__ + " at index " + index
             )
@@ -2232,18 +2233,18 @@ def is_in_set_or_list(x, s, items):
             # The cheap and easy case.
             to_item = MakePathRelative(to_file, fro_file, item) if is_paths else item
 
-            if not (type(item) is str and item.startswith("-")):
+            if not (isinstance(item, str) and item.startswith("-")):
                 # Any string that doesn't begin with a "-" is a singleton - it can
                 # only appear once in a list, to be enforced by the list merge append
                 # or prepend.
                 singleton = True
-        elif type(item) is dict:
+        elif isinstance(item, dict):
             # Make a copy of the dictionary, continuing to look for paths to fix.
             # The other intelligent aspects of merge processing won't apply because
             # item is being merged into an empty dict.
             to_item = {}
             MergeDicts(to_item, item, to_file, fro_file)
-        elif type(item) is list:
+        elif isinstance(item, list):
             # Recurse, making a copy of the list.  If the list contains any
             # descendant dicts, path fixing will occur.  Note that here, custom
             # values for is_paths and append are dropped; those are only to be
@@ -2312,12 +2313,12 @@ def MergeDicts(to, fro, to_file, fro_file):
                 to[k] = MakePathRelative(to_file, fro_file, v)
             else:
                 to[k] = v
-        elif type(v) is dict:
+        elif isinstance(v, dict):
             # Recurse, guaranteeing copies will be made of objects that require it.
             if k not in to:
                 to[k] = {}
             MergeDicts(to[k], v, to_file, fro_file)
-        elif type(v) is list:
+        elif isinstance(v, list):
             # Lists in dicts can be merged with different policies, depending on
             # how the key in the "from" dict (k, the from-key) is written.
             #
@@ -2361,7 +2362,7 @@ def MergeDicts(to, fro, to_file, fro_file):
                     # If the key ends in "?", the list will only be merged if it doesn't
                     # already exist.
                     continue
-                elif type(to[list_base]) is not list:
+                elif not isinstance(to[list_base], list):
                     # This may not have been checked above if merging in a list with an
                     # extension character.
                     raise TypeError(
@@ -2468,11 +2469,8 @@ def SetUpConfigurations(target, target_dict):
         merged_configurations[configuration] = new_configuration_dict
 
     # Put the new configurations back into the target dict as a configuration.
-    for configuration in merged_configurations:
-        target_dict["configurations"][configuration] = merged_configurations[
-            configuration
-        ]
-
+    for configuration, value in merged_configurations.items():
+        target_dict["configurations"][configuration] = value
     # Now drop all the abstract ones.
     configs = target_dict["configurations"]
     target_dict["configurations"] = {
@@ -2542,7 +2540,7 @@ def ProcessListFiltersInDict(name, the_dict):
         if operation not in {"!", "/"}:
             continue
 
-        if type(value) is not list:
+        if not isinstance(value, list):
             raise ValueError(
                 name + " key " + key + " must be list, not " + value.__class__.__name__
             )
@@ -2555,7 +2553,7 @@ def ProcessListFiltersInDict(name, the_dict):
             del_lists.append(key)
             continue
 
-        if type(the_dict[list_key]) is not list:
+        if not isinstance(the_dict[list_key], list):
             value = the_dict[list_key]
             raise ValueError(
                 name
@@ -2668,17 +2666,17 @@ def ProcessListFiltersInDict(name, the_dict):
 
     # Now recurse into subdicts and lists that may contain dicts.
     for key, value in the_dict.items():
-        if type(value) is dict:
+        if isinstance(value, dict):
             ProcessListFiltersInDict(key, value)
-        elif type(value) is list:
+        elif isinstance(value, list):
             ProcessListFiltersInList(key, value)
 
 
 def ProcessListFiltersInList(name, the_list):
     for item in the_list:
-        if type(item) is dict:
+        if isinstance(item, dict):
             ProcessListFiltersInDict(name, item)
-        elif type(item) is list:
+        elif isinstance(item, list):
             ProcessListFiltersInList(name, item)
 
 
@@ -2788,7 +2786,7 @@ def ValidateRunAsInTarget(target, target_dict, build_file):
     run_as = target_dict.get("run_as")
     if not run_as:
         return
-    if type(run_as) is not dict:
+    if not isinstance(run_as, dict):
         raise GypError(
             "The 'run_as' in target %s from file %s should be a "
             "dictionary." % (target_name, build_file)
@@ -2799,19 +2797,19 @@ def ValidateRunAsInTarget(target, target_dict, build_file):
             "The 'run_as' in target %s from file %s must have an "
             "'action' section." % (target_name, build_file)
         )
-    if type(action) is not list:
+    if not isinstance(action, list):
         raise GypError(
             "The 'action' for 'run_as' in target %s from file %s "
             "must be a list." % (target_name, build_file)
         )
     working_directory = run_as.get("working_directory")
-    if working_directory and type(working_directory) is not str:
+    if working_directory and not isinstance(working_directory, str):
         raise GypError(
             "The 'working_directory' for 'run_as' in target %s "
             "in file %s should be a string." % (target_name, build_file)
         )
     environment = run_as.get("environment")
-    if environment and type(environment) is not dict:
+    if environment and not isinstance(environment, dict):
         raise GypError(
             "The 'environment' for 'run_as' in target %s "
             "in file %s should be a dictionary." % (target_name, build_file)
@@ -2843,15 +2841,15 @@ def TurnIntIntoStrInDict(the_dict):
     # Use items instead of iteritems because there's no need to try to look at
     # reinserted keys and their associated values.
     for k, v in the_dict.items():
-        if type(v) is int:
+        if isinstance(v, int):
             v = str(v)
             the_dict[k] = v
-        elif type(v) is dict:
+        elif isinstance(v, dict):
             TurnIntIntoStrInDict(v)
-        elif type(v) is list:
+        elif isinstance(v, list):
             TurnIntIntoStrInList(v)
 
-        if type(k) is int:
+        if isinstance(k, int):
             del the_dict[k]
             the_dict[str(k)] = v
 
@@ -2860,11 +2858,11 @@ def TurnIntIntoStrInList(the_list):
     """Given list the_list, recursively converts all integers into strings.
   """
     for index, item in enumerate(the_list):
-        if type(item) is int:
+        if isinstance(item, int):
             the_list[index] = str(item)
-        elif type(item) is dict:
+        elif isinstance(item, dict):
             TurnIntIntoStrInDict(item)
-        elif type(item) is list:
+        elif isinstance(item, list):
             TurnIntIntoStrInList(item)
 
 
@@ -3019,8 +3017,8 @@ def Load(
                     del target_dict[key]
         ProcessListFiltersInDict(target_name, tmp_dict)
         # Write the results back to |target_dict|.
-        for key in tmp_dict:
-            target_dict[key] = tmp_dict[key]
+        for key, value in tmp_dict.items():
+            target_dict[key] = value
 
     # Make sure every dependency appears at most once.
     RemoveDuplicateDependencies(targets)
diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py
index 5f2c097f63..aee1a542da 100644
--- a/gyp/pylib/gyp/xcode_emulation.py
+++ b/gyp/pylib/gyp/xcode_emulation.py
@@ -1127,8 +1127,8 @@ def _GetIOSPostbuilds(self, configname, output_binary):
     be deployed to a device.  This should be run as the very last step of the
     build."""
         if not (
-            self.isIOS
-            and (self.spec["type"] == "executable" or self._IsXCTest())
+            (self.isIOS
+            and (self.spec["type"] == "executable" or self._IsXCTest()))
             or self.IsIosFramework()
         ):
             return []
@@ -1174,8 +1174,9 @@ def _GetIOSPostbuilds(self, configname, output_binary):
                 # Then re-sign everything with 'preserve=True'
                 postbuilds.extend(
                     [
-                        '%s code-sign-bundle "%s" "%s" "%s" "%s" %s'
+                        '%s %s code-sign-bundle "%s" "%s" "%s" "%s" %s'
                         % (
+                            sys.executable,
                             os.path.join("${TARGET_BUILD_DIR}", "gyp-mac-tool"),
                             key,
                             settings.get("CODE_SIGN_ENTITLEMENTS", ""),
@@ -1190,8 +1191,9 @@ def _GetIOSPostbuilds(self, configname, output_binary):
             for target in targets:
                 postbuilds.extend(
                     [
-                        '%s code-sign-bundle "%s" "%s" "%s" "%s" %s'
+                        '%s %s code-sign-bundle "%s" "%s" "%s" "%s" %s'
                         % (
+                            sys.executable,
                             os.path.join("${TARGET_BUILD_DIR}", "gyp-mac-tool"),
                             key,
                             settings.get("CODE_SIGN_ENTITLEMENTS", ""),
@@ -1204,8 +1206,9 @@ def _GetIOSPostbuilds(self, configname, output_binary):
 
         postbuilds.extend(
             [
-                '%s code-sign-bundle "%s" "%s" "%s" "%s" %s'
+                '%s %s code-sign-bundle "%s" "%s" "%s" "%s" %s'
                 % (
+                    sys.executable,
                     os.path.join("${TARGET_BUILD_DIR}", "gyp-mac-tool"),
                     key,
                     settings.get("CODE_SIGN_ENTITLEMENTS", ""),
@@ -1858,7 +1861,7 @@ def _TopologicallySortedEnvVarKeys(env):
     regex = re.compile(r"\$\{([a-zA-Z0-9\-_]+)\}")
 
     def GetEdges(node):
-        # Use a definition of edges such that user_of_variable -> used_varible.
+        # Use a definition of edges such that user_of_variable -> used_variable.
         # This happens to be easier in this case, since a variable's
         # definition contains all variables it references in a single string.
         # We can then reverse the result of the topological sort at the end.
diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py
index 33c667c266..cd72aa262d 100644
--- a/gyp/pylib/gyp/xcodeproj_file.py
+++ b/gyp/pylib/gyp/xcodeproj_file.py
@@ -74,7 +74,7 @@
 PBXBuildFile appears extraneous, but there's actually one reason for this:
 file-specific compiler flags are added to the PBXBuildFile object so as to
 allow a single file to be a member of multiple targets while having distinct
-compiler flags for each.  These flags can be modified in the Xcode applciation
+compiler flags for each.  These flags can be modified in the Xcode application
 in the "Build" tab of a File Info window.
 
 When a project is open in the Xcode application, Xcode will rewrite it.  As
@@ -662,7 +662,7 @@ def _XCKVPrint(self, file, tabs, key, value):
 
     tabs is an int identifying the indentation level.  If the class'
     _should_print_single_line variable is True, tabs is ignored and the
-    key-value pair will be followed by a space insead of a newline.
+    key-value pair will be followed by a space instead of a newline.
     """
 
         if self._should_print_single_line:
@@ -781,7 +781,7 @@ def UpdateProperties(self, properties, do_copy=False):
             # Make sure the property conforms to the schema.
             (is_list, property_type, is_strong) = self._schema[property][0:3]
             if is_list:
-                if value.__class__ != list:
+                if not isinstance(value, list):
                     raise TypeError(
                         property
                         + " of "
@@ -791,7 +791,7 @@ def UpdateProperties(self, properties, do_copy=False):
                     )
                 for item in value:
                     if not isinstance(item, property_type) and not (
-                        isinstance(item, str) and property_type == str
+                        isinstance(item, str) and isinstance(property_type, str)
                     ):
                         # Accept unicode where str is specified.  str is treated as
                         # UTF-8-encoded.
@@ -806,7 +806,7 @@ def UpdateProperties(self, properties, do_copy=False):
                             + item.__class__.__name__
                         )
             elif not isinstance(value, property_type) and not (
-                isinstance(value, str) and property_type == str
+                isinstance(value, str) and isinstance(property_type, str)
             ):
                 # Accept unicode where str is specified.  str is treated as
                 # UTF-8-encoded.
@@ -2994,7 +2994,7 @@ def AddOrGetProjectReference(self, other_pbxproject):
                 key=lambda x: x["ProjectRef"].Name().lower()
             )
         else:
-            # The link already exists.  Pull out the relevnt data.
+            # The link already exists.  Pull out the relevant data.
             project_ref_dict = self._other_pbxprojects[other_pbxproject]
             product_group = project_ref_dict["ProductGroup"]
             project_ref = project_ref_dict["ProjectRef"]
@@ -3017,10 +3017,10 @@ def _AllSymrootsUnique(self, target, inherit_unique_symroot):
         symroots = self._DefinedSymroots(target)
         for s in self._DefinedSymroots(target):
             if (
-                s is not None
-                and not self._IsUniqueSymrootForTarget(s)
-                or s is None
-                and not inherit_unique_symroot
+                (s is not None
+                and not self._IsUniqueSymrootForTarget(s))
+                or (s is None
+                and not inherit_unique_symroot)
             ):
                 return False
         return True if symroots else inherit_unique_symroot
diff --git a/gyp/pylib/packaging/metadata.py b/gyp/pylib/packaging/metadata.py
index fb27493079..23bb564f3d 100644
--- a/gyp/pylib/packaging/metadata.py
+++ b/gyp/pylib/packaging/metadata.py
@@ -145,7 +145,7 @@ class RawMetadata(TypedDict, total=False):
 
     # Metadata 2.3 - PEP 685
     # No new fields were added in PEP 685, just some edge case were
-    # tightened up to provide better interoptability.
+    # tightened up to provide better interoperability.
 
 
 _STRING_FIELDS = {
@@ -206,10 +206,10 @@ def _parse_project_urls(data: List[str]) -> Dict[str, str]:
         # be the missing value, then they'd have multiple '' values that
         # overwrite each other in a accumulating dict.
         #
-        # The other potentional issue is that it's possible to have the
+        # The other potential issue is that it's possible to have the
         # same label multiple times in the metadata, with no solid "right"
         # answer with what to do in that case. As such, we'll do the only
-        # thing we can, which is treat the field as unparseable and add it
+        # thing we can, which is treat the field as unparsable and add it
         # to our list of unparsed fields.
         parts = [p.strip() for p in pair.split(",", 1)]
         parts.extend([""] * (max(0, 2 - len(parts))))  # Ensure 2 items
@@ -222,8 +222,8 @@ def _parse_project_urls(data: List[str]) -> Dict[str, str]:
         label, url = parts
         if label in urls:
             # The label already exists in our set of urls, so this field
-            # is unparseable, and we can just add the whole thing to our
-            # unparseable data and stop processing it.
+            # is unparsable, and we can just add the whole thing to our
+            # unparsable data and stop processing it.
             raise KeyError("duplicate labels in project urls")
         urls[label] = url
 
@@ -433,7 +433,7 @@ def parse_email(data: Union[bytes, str]) -> Tuple[RawMetadata, Dict[str, List[st
             except KeyError:
                 unparsed[name] = value
         # Nothing that we've done has managed to parse this, so it'll just
-        # throw it in our unparseable data and move on.
+        # throw it in our unparsable data and move on.
         else:
             unparsed[name] = value
 
@@ -450,7 +450,7 @@ def parse_email(data: Union[bytes, str]) -> Tuple[RawMetadata, Dict[str, List[st
     else:
         if payload:
             # Check to see if we've already got a description, if so then both
-            # it, and this body move to unparseable.
+            # it, and this body move to unparsable.
             if "description" in raw:
                 description_header = cast(str, raw.pop("description"))
                 unparsed.setdefault("description", []).extend(
diff --git a/gyp/pyproject.toml b/gyp/pyproject.toml
index def9858e44..4b0c88c8a2 100644
--- a/gyp/pyproject.toml
+++ b/gyp/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "gyp-next"
-version = "0.18.1"
+version = "0.19.1"
 authors = [
   { name="Node.js contributors", email="ryzokuken@disroot.org" },
 ]
@@ -92,7 +92,6 @@ select = [
   # "TRY",  # tryceratops
 ]
 ignore = [
-  "E721",
   "PLC1901",
   "PLR0402",
   "PLR1714",
diff --git a/gyp/tools/pretty_sln.py b/gyp/tools/pretty_sln.py
index cf0638a23d..8026699d39 100755
--- a/gyp/tools/pretty_sln.py
+++ b/gyp/tools/pretty_sln.py
@@ -93,10 +93,10 @@ def ParseSolution(solution_file):
             continue
 
     # Change all dependencies clsid to name instead.
-    for project in dependencies:
+    for project, deps in dependencies.items():
         # For each dependencies in this project
         new_dep_array = []
-        for dep in dependencies[project]:
+        for dep in deps:
             # Look for the project name matching this cldis
             for project_info in projects:
                 if projects[project_info][1] == dep:
diff --git a/gyp/tools/pretty_vcproj.py b/gyp/tools/pretty_vcproj.py
index 72c65d7fc4..862400358a 100755
--- a/gyp/tools/pretty_vcproj.py
+++ b/gyp/tools/pretty_vcproj.py
@@ -118,8 +118,8 @@ def FixFilenames(filenames, current_directory):
     new_list = []
     for filename in filenames:
         if filename:
-            for key in REPLACEMENTS:
-                filename = filename.replace(key, REPLACEMENTS[key])
+            for key, value in REPLACEMENTS.items():
+                filename = filename.replace(key, value)
             os.chdir(current_directory)
             filename = filename.strip("\"' ")
             if filename.startswith("$"):
@@ -207,7 +207,7 @@ def CleanupVcproj(node):
         node.appendChild(new_node)
 
 
-def GetConfiguationNodes(vcproj):
+def GetConfigurationNodes(vcproj):
     # TODO(nsylvain): Find a better way to navigate the xml.
     nodes = []
     for node in vcproj.childNodes:
@@ -307,7 +307,7 @@ def main(argv):
 
     # First thing we need to do is find the Configuration Node and merge them
     # with the vsprops they include.
-    for configuration_node in GetConfiguationNodes(dom.documentElement):
+    for configuration_node in GetConfigurationNodes(dom.documentElement):
         # Get the property sheets associated with this configuration.
         vsprops = configuration_node.getAttribute("InheritedPropertySheets")
 

From b5f34c0646626cf5133ff362c31a30880ef2fc75 Mon Sep 17 00:00:00 2001
From: "github-actions[bot]"
 <41898282+github-actions[bot]@users.noreply.github.com>
Date: Mon, 10 Feb 2025 11:34:54 +0100
Subject: [PATCH 458/551] chore(main): release 11.1.0 (#3110)

Co-authored-by: Node.js GitHub Bot 
---
 .release-please-manifest.json |  2 +-
 CHANGELOG.md                  | 20 ++++++++++++++++++++
 package.json                  |  2 +-
 3 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
index 26a3463a2e..b913264022 100644
--- a/.release-please-manifest.json
+++ b/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "11.0.0"
+    ".": "11.1.0"
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8374a920b7..3bb923e6a0 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,25 @@
 # Changelog
 
+## [11.1.0](https://github.com/nodejs/node-gyp/compare/v11.0.0...v11.1.0) (2025-02-10)
+
+
+### Features
+
+* update gyp-next to v0.19.1 ([#3122](https://github.com/nodejs/node-gyp/issues/3122)) ([504250e](https://github.com/nodejs/node-gyp/commit/504250e5e3e27c6ef6dcfcaa744b36e1a99c1be8))
+
+
+### Bug Fixes
+
+* Find VC.Tools.ARM64 on arm64 machine ([#3075](https://github.com/nodejs/node-gyp/issues/3075)) ([b899fae](https://github.com/nodejs/node-gyp/commit/b899faed56270d3d8496da7576b5750b264c2c21))
+* try libnode.dll first in load_exe_hook ([#2834](https://github.com/nodejs/node-gyp/issues/2834)) ([b9d10a5](https://github.com/nodejs/node-gyp/commit/b9d10a5a37081e2a731937e43eca52c83609e7f5))
+
+
+### Miscellaneous
+
+* add gyp-next updater ([#3105](https://github.com/nodejs/node-gyp/issues/3105)) ([e3f9a77](https://github.com/nodejs/node-gyp/commit/e3f9a7756f65a7f4e50799017b3dc51d5bc195b2))
+* Test on Ubuntu-24.04-arm and Node.js v23 ([#3121](https://github.com/nodejs/node-gyp/issues/3121)) ([2530f51](https://github.com/nodejs/node-gyp/commit/2530f51cec3ba595184e5bcb7fe1245e240beb59))
+* Use astral-sh/ruff-action@v3 to run the Python linter ([#3114](https://github.com/nodejs/node-gyp/issues/3114)) ([94448fc](https://github.com/nodejs/node-gyp/commit/94448fcd9f090814bce1c4361471dae199dc2e82))
+
 ## [11.0.0](https://github.com/nodejs/node-gyp/compare/v10.3.1...v11.0.0) (2024-12-03)
 
 
diff --git a/package.json b/package.json
index 4a1cfb0eb1..2bc123c87e 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "11.0.0",
+  "version": "11.1.0",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From c3b3ab06ee0f092cd5c0646120d57e56d41b79fc Mon Sep 17 00:00:00 2001
From: Ben McCann <322311+benmccann@users.noreply.github.com>
Date: Wed, 19 Feb 2025 09:33:05 -0800
Subject: [PATCH 459/551] chore: switch to tinyglobby (#3133)

---
 lib/build.js | 2 +-
 package.json | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/build.js b/lib/build.js
index e1f49bb6ff..b66eb9698d 100644
--- a/lib/build.js
+++ b/lib/build.js
@@ -3,7 +3,7 @@
 const gracefulFs = require('graceful-fs')
 const fs = gracefulFs.promises
 const path = require('path')
-const { glob } = require('glob')
+const { glob } = require('tinyglobby')
 const log = require('./log')
 const which = require('which')
 const win = process.platform === 'win32'
diff --git a/package.json b/package.json
index 2bc123c87e..406c67cf51 100644
--- a/package.json
+++ b/package.json
@@ -24,13 +24,13 @@
   "dependencies": {
     "env-paths": "^2.2.0",
     "exponential-backoff": "^3.1.1",
-    "glob": "^10.3.10",
     "graceful-fs": "^4.2.6",
     "make-fetch-happen": "^14.0.3",
     "nopt": "^8.0.0",
     "proc-log": "^5.0.0",
     "semver": "^7.3.5",
     "tar": "^7.4.3",
+    "tinyglobby": "^0.2.11",
     "which": "^5.0.0"
   },
   "engines": {

From b21cf874f58883f3fd4dd07bec3b584fb07e831d Mon Sep 17 00:00:00 2001
From: Ben McCann <322311+benmccann@users.noreply.github.com>
Date: Wed, 19 Feb 2025 20:17:38 -0800
Subject: [PATCH 460/551] chore: update tinyglobby (#3136)

---
 lib/build.js | 5 +++--
 package.json | 2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/lib/build.js b/lib/build.js
index b66eb9698d..ae38d22e0b 100644
--- a/lib/build.js
+++ b/lib/build.js
@@ -84,9 +84,10 @@ async function build (gyp, argv) {
    */
 
   async function findSolutionFile () {
-    const files = await glob('build/*.sln')
+    const files = await glob('build/*.sln', { expandDirectories: false })
     if (files.length === 0) {
-      if (gracefulFs.existsSync('build/Makefile') || (await glob('build/*.mk')).length !== 0) {
+      if (gracefulFs.existsSync('build/Makefile') ||
+          (await glob('build/*.mk', { expandDirectories: false })).length !== 0) {
         command = makeCommand
         await doWhich(false)
         return
diff --git a/package.json b/package.json
index 406c67cf51..b5d81d463e 100644
--- a/package.json
+++ b/package.json
@@ -30,7 +30,7 @@
     "proc-log": "^5.0.0",
     "semver": "^7.3.5",
     "tar": "^7.4.3",
-    "tinyglobby": "^0.2.11",
+    "tinyglobby": "^0.2.12",
     "which": "^5.0.0"
   },
   "engines": {

From ee1d6fd8d83c9dd3eae7df7ec533bb6b39e1a812 Mon Sep 17 00:00:00 2001
From: Toyo Li 
Date: Thu, 20 Mar 2025 01:05:14 +0800
Subject: [PATCH 461/551] test: fix wasm test on Windows (#3145)

* fix wasm test on windows

* test not rely lint python
---
 .github/workflows/tests.yml | 15 ++++++++++++---
 test/test-windows-make.js   |  9 ++++++---
 2 files changed, 18 insertions(+), 6 deletions(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index f68abd41ef..e9585af26f 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -88,8 +88,6 @@ jobs:
         npm test
 
   tests:
-    # lint-python takes ~5 seconds, so wait for it to pass before running the full matrix of tests.
-    needs: [lint-python]
     strategy:
       fail-fast: false
       max-parallel: 11
@@ -101,7 +99,7 @@ jobs:
           - os: macos-13
             python: "3.13"
             node: 23.x
-          - os: ubuntu-24.04-arm 
+          - os: ubuntu-24.04-arm
             python: "3.13"
             node: 23.x
           - os: windows-2025
@@ -109,6 +107,10 @@ jobs:
             node: 23.x
     name: ${{ matrix.os }} - ${{ matrix.python }} - ${{ matrix.node }}
     runs-on: ${{ matrix.os }}
+    env:
+      WASI_VERSION: '25'
+      WASI_VERSION_FULL: '25.0'
+      WASI_SDK_PATH: 'wasi-sdk-25.0'
     steps:
       - name: Checkout Repository
         uses: actions/checkout@v4
@@ -123,6 +125,13 @@ jobs:
         env:
           PYTHON_VERSION: ${{ matrix.python }}  # Why do this?
       - uses: seanmiddleditch/gha-setup-ninja@v6
+      - name: Install wasi-sdk (Windows)
+        shell: pwsh
+        if: runner.os == 'Windows'
+        run: |
+          Start-BitsTransfer -Source https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${env:WASI_VERSION}/wasi-sdk-${env:WASI_VERSION_FULL}-x86_64-windows.tar.gz
+          New-Item -ItemType Directory -Path ${env:WASI_SDK_PATH}
+          tar -zxvf wasi-sdk-${env:WASI_VERSION_FULL}-x86_64-windows.tar.gz -C ${env:WASI_SDK_PATH} --strip 1
       - name: Install Dependencies
         run: |
           npm install
diff --git a/test/test-windows-make.js b/test/test-windows-make.js
index 41cefc3035..edd9d36412 100644
--- a/test/test-windows-make.js
+++ b/test/test-windows-make.js
@@ -46,10 +46,12 @@ function getEnv (target) {
     env.CC_target = 'emcc'
     env.CXX_target = 'em++'
   } else if (target === 'wasi') {
+    if (!process.env.WASI_SDK_PATH) return env
     env.AR_target = path.resolve(__dirname, '..', process.env.WASI_SDK_PATH, 'bin', executable('ar'))
     env.CC_target = path.resolve(__dirname, '..', process.env.WASI_SDK_PATH, 'bin', executable('clang'))
     env.CXX_target = path.resolve(__dirname, '..', process.env.WASI_SDK_PATH, 'bin', executable('clang++'))
   } else if (target === 'wasm') {
+    if (!process.env.WASI_SDK_PATH) return env
     env.AR_target = path.resolve(__dirname, '..', process.env.WASI_SDK_PATH, 'bin', executable('ar'))
     env.CC_target = path.resolve(__dirname, '..', process.env.WASI_SDK_PATH, 'bin', executable('clang'))
     env.CXX_target = path.resolve(__dirname, '..', process.env.WASI_SDK_PATH, 'bin', executable('clang++'))
@@ -72,16 +74,17 @@ function quote (path) {
   if (path.includes(' ')) {
     return `"${path}"`
   }
+  return path
 }
 
 describe('windows-cross-compile', function () {
   it('build simple node-api addon', async function () {
     if (process.platform !== 'win32') {
-      return this.skip('This test is only for windows')
+      return this.skip('This test is only for Windows')
     }
-    const env = getEnv('win-clang')
+    const env = getEnv('wasm')
     if (!gracefulFs.existsSync(env.CC_target)) {
-      return this.skip('Visual Studio Clang is not installed')
+      return this.skip('CC_target does not exist')
     }
 
     // handle bash whitespace

From 3e1cdd4630f6b32feb8ef704a7b1e8cbf8d8ff2f Mon Sep 17 00:00:00 2001
From: Stefan Stojanovic 
Date: Tue, 1 Apr 2025 11:45:00 +0200
Subject: [PATCH 462/551] win,src: fix GetModuleHandle for libnode.dll (#3139)

Fixes: https://github.com/nodejs/node-gyp/issues/3126
---
 src/win_delay_load_hook.cc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/win_delay_load_hook.cc b/src/win_delay_load_hook.cc
index c6e80aa313..63e197706d 100644
--- a/src/win_delay_load_hook.cc
+++ b/src/win_delay_load_hook.cc
@@ -29,7 +29,7 @@ static FARPROC WINAPI load_exe_hook(unsigned int event, DelayLoadInfo* info) {
     return NULL;
 
   // try for libnode.dll to compat node.js that using 'vcbuild.bat dll'
-  m = GetModuleHandle("libnode.dll");
+  m = GetModuleHandle(TEXT("libnode.dll"));
   if (m == NULL) m = GetModuleHandle(NULL);
   return (FARPROC) m;
 }

From 48851107ad8c5d2cf18a55e8bd2764f5938e7102 Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Tue, 1 Apr 2025 11:57:32 +0200
Subject: [PATCH 463/551] docs: add ffi-napi to docs/README.md (#3138)

---
 docs/README.md | 19 ++++++-------------
 1 file changed, 6 insertions(+), 13 deletions(-)

diff --git a/docs/README.md b/docs/README.md
index 90cbedc491..c92746c5ad 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -1,4 +1,4 @@
-## Versions of `node-gyp` that are earlier than v10.x.x
+## Versions of `node-gyp` that are earlier than v11.x.x
 
 Please look thru your error log for the string `gyp info using node-gyp@` and if that version number is less than the [current release of node-gyp](https://github.com/nodejs/node-gyp/releases) then __please upgrade__ using [these instructions](https://github.com/nodejs/node-gyp/blob/main/docs/Updating-npm-bundled-node-gyp.md) and then try your command again.
 
@@ -13,19 +13,12 @@ npm install --global node-sass@latest
 ```
 `node-sass` projects _may_ work by downgrading to Node.js v14 but [that release is end-of-life](https://github.com/nodejs/release#release-schedule).
 
-In any case, please avoid opening new `node-sass` issues on this repo because we [cannot help much](https://github.com/nodejs/node-gyp/issues?q=is%3Aissue+label%3A%22Node+Sass+--%3E+Dart+Sass%22+).
+In any case, please avoid opening new `node-sass` issues on this repo because we [cannot help much](https://github.com/nodejs/node-gyp/issues?q=is%3Aissue+label%3A%22Node+Sass+--%3E+Dart+Sass%22).
 
-## `node-pre-gyp` is no longer maintained
+## `ffi-napi` is no longer maintained
 
-* mapbox/node-pre-gyp#657
+* node-ffi-napi/node-ffi-napi#269
 
-Support in the `abi_crosswalk.json` file ends at Node.js v17 but [that release is end-of-life](https://github.com/nodejs/release#release-schedule).
+There are a couple of workarounds (https://koffi.dev or `node-ffi-rs`) on that issue but using `ffi-napi` or its forks has proven problematic on modern versions of operating systems, Node.js, node-gyp, and Python.
 
-In any case, please avoid opening new `node-pre-gyp` issues on this repo because we [cannot help much](https://github.com/nodejs/node-gyp/issues?q=is%3Aissue+label%3A%22node-pre-gyp+is+unmaintained%22).
-
-Unsupported __WORKAROUND__ for versions of Node.js > v17
-```
-npm ci  # mapbox/node-pre-gyp
-npm run update-crosswalk
-# npm audit  # Currently fails on a `Severity: critical` issue
-```
+In any case, please avoid opening new `ffi-napi` issues on this repo because we [cannot help much](https://github.com/nodejs/node-gyp/issues?q=label%3Affi-napi).

From 0cf16d29fe604266fb47325496287a63075ea532 Mon Sep 17 00:00:00 2001
From: Aman Karmani 
Date: Tue, 1 Apr 2025 04:59:19 -0500
Subject: [PATCH 464/551] fix: disable msbuild.exe nodeReuse (#3112)

On windows, one may experience errors due to concurrent node-gyp calls (see #3095)

When retrying after such a failure, retries are observed to fail repeatedly with strange
EBUSY and EPERM errors. This turns out to be due to a caching feature where msbuild.exe
will continue to run in the background (for either 15s or 15m depending on VS version),
and keep open file handles to files and directories that the user may be trying to delete
(for example with a new call to `npm ci`).

This behavior is well documented, as is the recommended workaround implemented here.
---
 lib/build.js | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/build.js b/lib/build.js
index ae38d22e0b..cae506e842 100644
--- a/lib/build.js
+++ b/lib/build.js
@@ -142,6 +142,8 @@ async function build (gyp, argv) {
     if (msvs) {
       // Turn off the Microsoft logo on Windows
       argv.push('/nologo')
+      // No lingering msbuild processes and open file handles
+      argv.push('/nodeReuse:false')
     }
 
     // Specify the build type, Release by default

From 80e9c795a739c490cfbc85633e63022b36a7c70d Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Tue, 1 Apr 2025 07:51:19 -0400
Subject: [PATCH 465/551] feat: update gyp-next to v0.20.0 (#3149)

---
 gyp/.github/workflows/node-gyp.yml            |  14 +-
 gyp/.github/workflows/nodejs.yml              |   9 +-
 gyp/.github/workflows/python_tests.yml        |   8 +-
 gyp/.github/workflows/release-please.yml      |   9 +-
 gyp/.gitignore                                |   3 +
 gyp/.release-please-manifest.json             |   2 +-
 gyp/CHANGELOG.md                              |  13 ++
 gyp/docs/Hacking.md                           |   2 +-
 gyp/docs/InputFormatReference.md              |   5 +-
 gyp/gyp_main.py                               |   2 +-
 gyp/pylib/gyp/MSVSProject.py                  |   2 +-
 gyp/pylib/gyp/MSVSSettings_test.py            |   4 +-
 gyp/pylib/gyp/MSVSToolFile.py                 |   2 +-
 gyp/pylib/gyp/MSVSUserFile.py                 |   3 +-
 gyp/pylib/gyp/MSVSUtil.py                     |   1 -
 gyp/pylib/gyp/MSVSVersion.py                  |   2 +-
 gyp/pylib/gyp/__init__.py                     |  29 ++-
 gyp/pylib/gyp/common.py                       |   8 +-
 gyp/pylib/gyp/common_test.py                  |  24 +--
 gyp/pylib/gyp/easy_xml.py                     |   6 +-
 gyp/pylib/gyp/easy_xml_test.py                |   4 +-
 gyp/pylib/gyp/generator/analyzer.py           |   7 +-
 gyp/pylib/gyp/generator/android.py            |  41 ++---
 gyp/pylib/gyp/generator/cmake.py              |   1 +
 .../gyp/generator/compile_commands_json.py    |   5 +-
 .../gyp/generator/dump_dependency_json.py     |   3 +-
 gyp/pylib/gyp/generator/eclipse.py            |   7 +-
 gyp/pylib/gyp/generator/gypd.py               |   2 +-
 gyp/pylib/gyp/generator/gypsh.py              |   1 -
 gyp/pylib/gyp/generator/make.py               |  44 ++---
 gyp/pylib/gyp/generator/msvs.py               | 173 +++++++++---------
 gyp/pylib/gyp/generator/msvs_test.py          |   4 +-
 gyp/pylib/gyp/generator/ninja.py              |  22 +--
 gyp/pylib/gyp/generator/ninja_test.py         |   4 +-
 gyp/pylib/gyp/generator/xcode.py              |  12 +-
 gyp/pylib/gyp/generator/xcode_test.py         |   5 +-
 gyp/pylib/gyp/input.py                        |  47 +++--
 gyp/pylib/gyp/input_test.py                   |   3 +-
 gyp/pylib/gyp/mac_tool.py                     |  12 +-
 gyp/pylib/gyp/msvs_emulation.py               |   6 +-
 gyp/pylib/gyp/win_tool.py                     |   2 +-
 gyp/pylib/gyp/xcode_emulation.py              |  12 +-
 gyp/pylib/gyp/xcode_emulation_test.py         |   3 +-
 gyp/pylib/gyp/xcode_ninja.py                  |   3 +-
 gyp/pylib/gyp/xcodeproj_file.py               |  28 ++-
 gyp/pyproject.toml                            |   8 +-
 gyp/tools/pretty_gyp.py                       |   5 +-
 gyp/tools/pretty_sln.py                       |   3 +-
 gyp/tools/pretty_vcproj.py                    |   4 +-
 49 files changed, 312 insertions(+), 307 deletions(-)

diff --git a/gyp/.github/workflows/node-gyp.yml b/gyp/.github/workflows/node-gyp.yml
index 4382979fc3..56cdb4086c 100644
--- a/gyp/.github/workflows/node-gyp.yml
+++ b/gyp/.github/workflows/node-gyp.yml
@@ -10,8 +10,15 @@ jobs:
       fail-fast: false
       matrix:
         node-version: ["22"]
-        os: [macos-13, macos-14, ubuntu-latest, windows-latest]
-        python-version: ["3.8", "3.10", "3.12", "3.13"]
+        os: [macos-latest, ubuntu-latest, windows-latest]
+        python-version: ["3.9", "3.11", "3.13"]
+        include:
+          - node-version: "22"
+            os: macos-13
+            python-version: "3.13"
+          - node-version: "22"
+            os: windows-2025
+            python-version: "3.13"
     runs-on: ${{ matrix.os }}
     steps:
       - name: Clone gyp-next
@@ -33,7 +40,7 @@ jobs:
       - name: Install Python dependencies
         run: |
           cd gyp-next
-          python -m pip install --upgrade pip setuptools
+          python -m pip install --upgrade pip
           pip install --editable .
           pip uninstall -y gyp-next
       - name: Install Node.js dependencies
@@ -47,7 +54,6 @@ jobs:
           cp -r gyp-next node-gyp/gyp
       - name: Run tests (macOS or Linux)
         if: runner.os != 'Windows'
-        shell: bash
         run: |
           cd node-gyp
           npm test --python="${pythonLocation}/python"
diff --git a/gyp/.github/workflows/nodejs.yml b/gyp/.github/workflows/nodejs.yml
index 53935d4069..213fcbd0c1 100644
--- a/gyp/.github/workflows/nodejs.yml
+++ b/gyp/.github/workflows/nodejs.yml
@@ -9,8 +9,13 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: [macos-13, macos-latest, ubuntu-latest, windows-latest]
-        python: ["3.8", "3.10", "3.12", "3.13"]
+        os: [macos-latest, ubuntu-latest, windows-latest]
+        python: ["3.9", "3.11", "3.13"]
+        include:
+          - os: macos-13
+            python-version: "3.13"
+          - os: windows-2025
+            python-version: "3.13"
 
     runs-on: ${{ matrix.os }}
     steps:
diff --git a/gyp/.github/workflows/python_tests.yml b/gyp/.github/workflows/python_tests.yml
index 507d4d2d75..b076442051 100644
--- a/gyp/.github/workflows/python_tests.yml
+++ b/gyp/.github/workflows/python_tests.yml
@@ -14,7 +14,7 @@ jobs:
       fail-fast: false
       max-parallel: 5
       matrix:
-        os: [macos-13, macos-14, ubuntu-latest] # , windows-latest]
+        os: [macos-13, macos-latest, ubuntu-latest] # , windows-latest]
         python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"]
     steps:
       - uses: actions/checkout@v4
@@ -23,14 +23,14 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
           allow-prereleases: true
-      - uses: seanmiddleditch/gha-setup-ninja@v5
+      - uses: seanmiddleditch/gha-setup-ninja@v6
       - name: Install dependencies
         run: |
-          python -m pip install --upgrade pip setuptools
+          python -m pip install --upgrade pip
           pip install --editable ".[dev]"
       - run: ./gyp -V && ./gyp --version && gyp -V && gyp --version
       - name: Lint with ruff  # See pyproject.toml for settings
-        run: ruff check --output-format=github .
+        uses: astral-sh/ruff-action@v3
       - name: Test with pytest  # See pyproject.toml for settings
         run: pytest
       # - name: Run doctests with pytest
diff --git a/gyp/.github/workflows/release-please.yml b/gyp/.github/workflows/release-please.yml
index a96a88fe68..a843939215 100644
--- a/gyp/.github/workflows/release-please.yml
+++ b/gyp/.github/workflows/release-please.yml
@@ -25,15 +25,8 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v4
-    - name: Set up Python
-      uses: actions/setup-python@v5
-      with:
-        python-version: "3.12"
-    - name: Install pypa/build
-      run: >-
-        python3 -m pip install build --user
     - name: Build a binary wheel and a source tarball
-      run: python3 -m build
+      run: pipx run build
     - name: Store the distribution packages
       uses: actions/upload-artifact@v4
       with:
diff --git a/gyp/.gitignore b/gyp/.gitignore
index d82fa7a96c..5f71dbd435 100644
--- a/gyp/.gitignore
+++ b/gyp/.gitignore
@@ -141,3 +141,6 @@ cython_debug/
 # static files generated from Django application using `collectstatic`
 media
 static
+
+test/fixtures/out
+*.actual
diff --git a/gyp/.release-please-manifest.json b/gyp/.release-please-manifest.json
index 1f9113816b..589cd4553e 100644
--- a/gyp/.release-please-manifest.json
+++ b/gyp/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "0.19.1"
+    ".": "0.20.0"
 }
diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md
index 22257ab93d..3f676055a6 100644
--- a/gyp/CHANGELOG.md
+++ b/gyp/CHANGELOG.md
@@ -1,5 +1,18 @@
 # Changelog
 
+## [0.20.0](https://github.com/nodejs/gyp-next/compare/v0.19.1...v0.20.0) (2025-03-27)
+
+
+### ⚠ BREAKING CHANGES
+
+* resolve issue with relative paths during linking ([#284](https://github.com/nodejs/gyp-next/issues/284))
+
+### Bug Fixes
+
+* python lint  more ruff rules ([#291](https://github.com/nodejs/gyp-next/issues/291)) ([fabc78c](https://github.com/nodejs/gyp-next/commit/fabc78caffcf988365d970ced5a151f40525077e))
+* remove explicit installation of setuptools ([#278](https://github.com/nodejs/gyp-next/issues/278)) ([e476778](https://github.com/nodejs/gyp-next/commit/e4767782c70ca8427184694589d9f0ded5eeed22))
+* resolve issue with relative paths during linking ([#284](https://github.com/nodejs/gyp-next/issues/284)) ([a2d7439](https://github.com/nodejs/gyp-next/commit/a2d7439fbd3c03f01e1149fdbe682f754bc6cc7f))
+
 ## [0.19.1](https://github.com/nodejs/gyp-next/compare/v0.19.0...v0.19.1) (2024-12-09)
 
 
diff --git a/gyp/docs/Hacking.md b/gyp/docs/Hacking.md
index b00783bd36..156d485b5b 100644
--- a/gyp/docs/Hacking.md
+++ b/gyp/docs/Hacking.md
@@ -24,7 +24,7 @@ to make sure your changes aren't breaking anything important.
 You run the test driver with e.g.
 
 ``` sh
-$ python -m pip install --upgrade pip setuptools
+$ python -m pip install --upgrade pip
 $ pip install --editable ".[dev]"
 $ python -m pytest
 ```
diff --git a/gyp/docs/InputFormatReference.md b/gyp/docs/InputFormatReference.md
index 2b2c180f44..4b114f2deb 100644
--- a/gyp/docs/InputFormatReference.md
+++ b/gyp/docs/InputFormatReference.md
@@ -194,6 +194,7 @@ lists associated with the following keys, are treated as pathnames:
   * include\_dirs
   * inputs
   * libraries
+  * library\_dirs
   * outputs
   * sources
   * mac\_bundle\_resources
@@ -231,7 +232,8 @@ Source dictionary from `../build/common.gypi`:
 ```
 {
   'include_dirs': ['include'],  # Treated as relative to ../build
-  'libraries': ['-lz'],  # Not treated as a pathname, begins with a dash
+  'library_dirs': ['lib'],      # Treated as relative to ../build
+  'libraries': ['-lz'],   # Not treated as a pathname, begins with a dash
   'defines': ['NDEBUG'],  # defines does not contain pathnames
 }
 ```
@@ -250,6 +252,7 @@ Merged dictionary:
 {
   'sources': ['string_util.cc'],
   'include_dirs': ['../build/include'],
+  'library_dirs': ['../build/lib'],
   'libraries': ['-lz'],
   'defines': ['NDEBUG'],
 }
diff --git a/gyp/gyp_main.py b/gyp/gyp_main.py
index f23dcdf882..bf16987485 100755
--- a/gyp/gyp_main.py
+++ b/gyp/gyp_main.py
@@ -5,8 +5,8 @@
 # found in the LICENSE file.
 
 import os
-import sys
 import subprocess
+import sys
 
 
 def IsCygwin():
diff --git a/gyp/pylib/gyp/MSVSProject.py b/gyp/pylib/gyp/MSVSProject.py
index 629f3f61b4..339d27d402 100644
--- a/gyp/pylib/gyp/MSVSProject.py
+++ b/gyp/pylib/gyp/MSVSProject.py
@@ -4,7 +4,7 @@
 
 """Visual Studio project reader/writer."""
 
-import gyp.easy_xml as easy_xml
+from gyp import easy_xml
 
 # ------------------------------------------------------------------------------
 
diff --git a/gyp/pylib/gyp/MSVSSettings_test.py b/gyp/pylib/gyp/MSVSSettings_test.py
index 6ca09687ad..0504728d99 100755
--- a/gyp/pylib/gyp/MSVSSettings_test.py
+++ b/gyp/pylib/gyp/MSVSSettings_test.py
@@ -7,10 +7,10 @@
 """Unit tests for the MSVSSettings.py file."""
 
 import unittest
-import gyp.MSVSSettings as MSVSSettings
-
 from io import StringIO
 
+from gyp import MSVSSettings
+
 
 class TestSequenceFunctions(unittest.TestCase):
     def setUp(self):
diff --git a/gyp/pylib/gyp/MSVSToolFile.py b/gyp/pylib/gyp/MSVSToolFile.py
index 2e5c811bdd..901ba84588 100644
--- a/gyp/pylib/gyp/MSVSToolFile.py
+++ b/gyp/pylib/gyp/MSVSToolFile.py
@@ -4,7 +4,7 @@
 
 """Visual Studio project reader/writer."""
 
-import gyp.easy_xml as easy_xml
+from gyp import easy_xml
 
 
 class Writer:
diff --git a/gyp/pylib/gyp/MSVSUserFile.py b/gyp/pylib/gyp/MSVSUserFile.py
index e580c00fb7..23d3e16953 100644
--- a/gyp/pylib/gyp/MSVSUserFile.py
+++ b/gyp/pylib/gyp/MSVSUserFile.py
@@ -8,8 +8,7 @@
 import re
 import socket  # for gethostname
 
-import gyp.easy_xml as easy_xml
-
+from gyp import easy_xml
 
 # ------------------------------------------------------------------------------
 
diff --git a/gyp/pylib/gyp/MSVSUtil.py b/gyp/pylib/gyp/MSVSUtil.py
index 36bb782bd3..27647f11d0 100644
--- a/gyp/pylib/gyp/MSVSUtil.py
+++ b/gyp/pylib/gyp/MSVSUtil.py
@@ -7,7 +7,6 @@
 import copy
 import os
 
-
 # A dictionary mapping supported target types to extensions.
 TARGET_TYPE_EXT = {
     "executable": "exe",
diff --git a/gyp/pylib/gyp/MSVSVersion.py b/gyp/pylib/gyp/MSVSVersion.py
index 1b35362922..93f48bc05c 100644
--- a/gyp/pylib/gyp/MSVSVersion.py
+++ b/gyp/pylib/gyp/MSVSVersion.py
@@ -5,11 +5,11 @@
 """Handle version information related to Visual Stuio."""
 
 import errno
+import glob
 import os
 import re
 import subprocess
 import sys
-import glob
 
 
 def JoinPath(*args):
diff --git a/gyp/pylib/gyp/__init__.py b/gyp/pylib/gyp/__init__.py
index 8933d0c4f7..77800661a4 100755
--- a/gyp/pylib/gyp/__init__.py
+++ b/gyp/pylib/gyp/__init__.py
@@ -5,16 +5,17 @@
 # found in the LICENSE file.
 
 from __future__ import annotations
-import copy
-import gyp.input
+
 import argparse
+import copy
 import os.path
 import re
 import shlex
 import sys
 import traceback
-from gyp.common import GypError
 
+import gyp.input
+from gyp.common import GypError
 
 # Default debug modes for GYP
 debug = {}
@@ -205,8 +206,7 @@ def NameValueListToDict(name_value_list):
 
 
 def ShlexEnv(env_name):
-    flags = os.environ.get(env_name, [])
-    if flags:
+    if flags := os.environ.get(env_name) or []:
         flags = shlex.split(flags)
     return flags
 
@@ -361,7 +361,7 @@ def gyp_main(args):
         action="store",
         env_name="GYP_CONFIG_DIR",
         default=None,
-        help="The location for configuration files like " "include.gypi.",
+        help="The location for configuration files like include.gypi.",
     )
     parser.add_argument(
         "-d",
@@ -525,19 +525,18 @@ def gyp_main(args):
         # If no format was given on the command line, then check the env variable.
         generate_formats = []
         if options.use_environment:
-            generate_formats = os.environ.get("GYP_GENERATORS", [])
+            generate_formats = os.environ.get("GYP_GENERATORS") or []
         if generate_formats:
             generate_formats = re.split(r"[\s,]", generate_formats)
         if generate_formats:
             options.formats = generate_formats
+        # Nothing in the variable, default based on platform.
+        elif sys.platform == "darwin":
+            options.formats = ["xcode"]
+        elif sys.platform in ("win32", "cygwin"):
+            options.formats = ["msvs"]
         else:
-            # Nothing in the variable, default based on platform.
-            if sys.platform == "darwin":
-                options.formats = ["xcode"]
-            elif sys.platform in ("win32", "cygwin"):
-                options.formats = ["msvs"]
-            else:
-                options.formats = ["make"]
+            options.formats = ["make"]
 
     if not options.generator_output and options.use_environment:
         g_o = os.environ.get("GYP_GENERATOR_OUTPUT")
@@ -696,7 +695,7 @@ def main(args):
         return 1
 
 
-# NOTE: setuptools generated console_scripts calls function with no arguments
+# NOTE: console_scripts calls this function with no arguments
 def script_main():
     return main(sys.argv[1:])
 
diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py
index 762ae02109..fbf1024fc3 100644
--- a/gyp/pylib/gyp/common.py
+++ b/gyp/pylib/gyp/common.py
@@ -6,11 +6,10 @@
 import filecmp
 import os.path
 import re
-import tempfile
-import sys
-import subprocess
 import shlex
-
+import subprocess
+import sys
+import tempfile
 from collections.abc import MutableSet
 
 
@@ -35,7 +34,6 @@ class GypError(Exception):
   to the user.  The main entry point will catch and display this.
   """
 
-    pass
 
 
 def ExceptionAppend(e, msg):
diff --git a/gyp/pylib/gyp/common_test.py b/gyp/pylib/gyp/common_test.py
index b6c4cccc1a..bd7172afaf 100755
--- a/gyp/pylib/gyp/common_test.py
+++ b/gyp/pylib/gyp/common_test.py
@@ -6,11 +6,13 @@
 
 """Unit tests for the common.py file."""
 
-import gyp.common
-import unittest
-import sys
 import os
-from unittest.mock import patch, MagicMock
+import sys
+import unittest
+from unittest.mock import MagicMock, patch
+
+import gyp.common
+
 
 class TestTopologicallySorted(unittest.TestCase):
     def test_Valid(self):
@@ -109,14 +111,14 @@ def mock_run(env, defines_stdout, expected_cmd):
                 return [defines, flavor]
 
         [defines1, _] = mock_run({}, "", [])
-        assert {} == defines1
+        assert defines1 == {}
 
         [defines2, flavor2] = mock_run(
             { "CC_target": "/opt/wasi-sdk/bin/clang" },
             "#define __wasm__ 1\n#define __wasi__ 1\n",
             ["/opt/wasi-sdk/bin/clang"]
         )
-        assert { "__wasm__": "1", "__wasi__": "1" } == defines2
+        assert defines2 == { "__wasm__": "1", "__wasi__": "1" }
         assert flavor2 == "wasi"
 
         [defines3, flavor3] = mock_run(
@@ -124,7 +126,7 @@ def mock_run(env, defines_stdout, expected_cmd):
             "#define __wasm__ 1\n",
             ["/opt/wasi-sdk/bin/clang", "--target=wasm32"]
         )
-        assert { "__wasm__": "1" } == defines3
+        assert defines3 == { "__wasm__": "1" }
         assert flavor3 == "wasm"
 
         [defines4, flavor4] = mock_run(
@@ -132,7 +134,7 @@ def mock_run(env, defines_stdout, expected_cmd):
             "#define __EMSCRIPTEN__ 1\n",
             ["/emsdk/upstream/emscripten/emcc"]
         )
-        assert { "__EMSCRIPTEN__": "1" } == defines4
+        assert defines4 == { "__EMSCRIPTEN__": "1" }
         assert flavor4 == "emscripten"
 
         # Test path which include white space
@@ -149,11 +151,11 @@ def mock_run(env, defines_stdout, expected_cmd):
                 "-pthread"
             ]
         )
-        assert {
+        assert defines5 == {
             "__wasm__": "1",
             "__wasi__": "1",
             "_REENTRANT": "1"
-        } == defines5
+        }
         assert flavor5 == "wasi"
 
         original_sep = os.sep
@@ -164,7 +166,7 @@ def mock_run(env, defines_stdout, expected_cmd):
             ["C:/Program Files/wasi-sdk/clang.exe"]
         )
         os.sep = original_sep
-        assert { "__wasm__": "1", "__wasi__": "1" } == defines6
+        assert defines6 == { "__wasm__": "1", "__wasi__": "1" }
         assert flavor6 == "wasi"
 
 if __name__ == "__main__":
diff --git a/gyp/pylib/gyp/easy_xml.py b/gyp/pylib/gyp/easy_xml.py
index 02567b2514..e4d2f82b68 100644
--- a/gyp/pylib/gyp/easy_xml.py
+++ b/gyp/pylib/gyp/easy_xml.py
@@ -2,10 +2,10 @@
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
-import sys
-import re
-import os
 import locale
+import os
+import re
+import sys
 from functools import reduce
 
 
diff --git a/gyp/pylib/gyp/easy_xml_test.py b/gyp/pylib/gyp/easy_xml_test.py
index 2d9b15210d..bb97b802c5 100755
--- a/gyp/pylib/gyp/easy_xml_test.py
+++ b/gyp/pylib/gyp/easy_xml_test.py
@@ -6,11 +6,11 @@
 
 """ Unit tests for the easy_xml.py file. """
 
-import gyp.easy_xml as easy_xml
 import unittest
-
 from io import StringIO
 
+from gyp import easy_xml
+
 
 class TestSequenceFunctions(unittest.TestCase):
     def setUp(self):
diff --git a/gyp/pylib/gyp/generator/analyzer.py b/gyp/pylib/gyp/generator/analyzer.py
index 64573ad2cc..cb18742cd8 100644
--- a/gyp/pylib/gyp/generator/analyzer.py
+++ b/gyp/pylib/gyp/generator/analyzer.py
@@ -63,11 +63,12 @@
 """
 
 
-import gyp.common
 import json
 import os
 import posixpath
 
+import gyp.common
+
 debug = False
 
 found_dependency_string = "Found dependency"
@@ -157,7 +158,7 @@ def _AddSources(sources, base_path, base_path_components, result):
   and tracked in some other means."""
     # NOTE: gyp paths are always posix style.
     for source in sources:
-        if not len(source) or source.startswith("!!!") or source.startswith("$"):
+        if not len(source) or source.startswith(("!!!", "$")):
             continue
         # variable expansion may lead to //.
         org_source = source
@@ -747,7 +748,7 @@ def GenerateOutput(target_list, target_dicts, data, params):
 
         if not config.files:
             raise Exception(
-                "Must specify files to analyze via config_path generator " "flag"
+                "Must specify files to analyze via config_path generator flag"
             )
 
         toplevel_dir = _ToGypPath(os.path.abspath(params["options"].toplevel_dir))
diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py
index 64da385e6a..5ebe58bb55 100644
--- a/gyp/pylib/gyp/generator/android.py
+++ b/gyp/pylib/gyp/generator/android.py
@@ -15,13 +15,14 @@
 # Try to avoid setting global variables where possible.
 
 
-import gyp
-import gyp.common
-import gyp.generator.make as make  # Reuse global functions from make backend.
 import os
 import re
 import subprocess
 
+import gyp
+import gyp.common
+from gyp.generator import make  # Reuse global functions from make backend.
+
 generator_default_variables = {
     "OS": "android",
     "EXECUTABLE_PREFIX": "",
@@ -177,7 +178,7 @@ def Write(
             self.WriteLn("LOCAL_MULTILIB := $(GYP_HOST_MULTILIB)")
         elif sdk_version > 0:
             self.WriteLn(
-                "LOCAL_MODULE_TARGET_ARCH := " "$(TARGET_$(GYP_VAR_PREFIX)ARCH)"
+                "LOCAL_MODULE_TARGET_ARCH := $(TARGET_$(GYP_VAR_PREFIX)ARCH)"
             )
             self.WriteLn("LOCAL_SDK_VERSION := %s" % sdk_version)
 
@@ -587,11 +588,10 @@ def WriteSources(self, spec, configs, extra_sources):
         local_files = []
         for source in sources:
             (root, ext) = os.path.splitext(source)
-            if "$(gyp_shared_intermediate_dir)" in source:
-                extra_sources.append(source)
-            elif "$(gyp_intermediate_dir)" in source:
-                extra_sources.append(source)
-            elif IsCPPExtension(ext) and ext != local_cpp_extension:
+            if ("$(gyp_shared_intermediate_dir)" in source
+                or "$(gyp_intermediate_dir)" in source
+                or (IsCPPExtension(ext) and ext != local_cpp_extension)
+            ):
                 extra_sources.append(source)
             else:
                 local_files.append(os.path.normpath(os.path.join(self.path, source)))
@@ -730,19 +730,18 @@ def ComputeOutput(self, spec):
                 path = "$($(GYP_HOST_VAR_PREFIX)HOST_OUT_INTERMEDIATE_LIBRARIES)"
             else:
                 path = "$($(GYP_VAR_PREFIX)TARGET_OUT_INTERMEDIATE_LIBRARIES)"
+        # Other targets just get built into their intermediate dir.
+        elif self.toolset == "host":
+            path = (
+                "$(call intermediates-dir-for,%s,%s,true,,"
+                "$(GYP_HOST_VAR_PREFIX))"
+                % (self.android_class, self.android_module)
+            )
         else:
-            # Other targets just get built into their intermediate dir.
-            if self.toolset == "host":
-                path = (
-                    "$(call intermediates-dir-for,%s,%s,true,,"
-                    "$(GYP_HOST_VAR_PREFIX))"
-                    % (self.android_class, self.android_module)
-                )
-            else:
-                path = (
-                    f"$(call intermediates-dir-for,{self.android_class},"
-                    f"{self.android_module},,,$(GYP_VAR_PREFIX))"
-                )
+            path = (
+                f"$(call intermediates-dir-for,{self.android_class},"
+                f"{self.android_module},,,$(GYP_VAR_PREFIX))"
+            )
 
         assert spec.get("product_dir") is None  # TODO: not supported?
         return os.path.join(path, self.ComputeOutputBasename(spec))
diff --git a/gyp/pylib/gyp/generator/cmake.py b/gyp/pylib/gyp/generator/cmake.py
index 8720a3daf3..e69103e1b9 100644
--- a/gyp/pylib/gyp/generator/cmake.py
+++ b/gyp/pylib/gyp/generator/cmake.py
@@ -33,6 +33,7 @@
 import os
 import signal
 import subprocess
+
 import gyp.common
 import gyp.xcode_emulation
 
diff --git a/gyp/pylib/gyp/generator/compile_commands_json.py b/gyp/pylib/gyp/generator/compile_commands_json.py
index 5d7f14da96..bebb130315 100644
--- a/gyp/pylib/gyp/generator/compile_commands_json.py
+++ b/gyp/pylib/gyp/generator/compile_commands_json.py
@@ -2,11 +2,12 @@
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
-import gyp.common
-import gyp.xcode_emulation
 import json
 import os
 
+import gyp.common
+import gyp.xcode_emulation
+
 generator_additional_non_configuration_keys = []
 generator_additional_path_sections = []
 generator_extra_sources_for_rules = []
diff --git a/gyp/pylib/gyp/generator/dump_dependency_json.py b/gyp/pylib/gyp/generator/dump_dependency_json.py
index 99d5c1fd69..e41c72d710 100644
--- a/gyp/pylib/gyp/generator/dump_dependency_json.py
+++ b/gyp/pylib/gyp/generator/dump_dependency_json.py
@@ -3,11 +3,12 @@
 # found in the LICENSE file.
 
 
+import json
 import os
+
 import gyp
 import gyp.common
 import gyp.msvs_emulation
-import json
 
 generator_supports_multiple_toolsets = True
 
diff --git a/gyp/pylib/gyp/generator/eclipse.py b/gyp/pylib/gyp/generator/eclipse.py
index 52aeae6050..ed6daa91ba 100644
--- a/gyp/pylib/gyp/generator/eclipse.py
+++ b/gyp/pylib/gyp/generator/eclipse.py
@@ -17,14 +17,15 @@
 This generator has no automated tests, so expect it to be broken.
 """
 
-from xml.sax.saxutils import escape
 import os.path
+import shlex
 import subprocess
+import xml.etree.ElementTree as ET
+from xml.sax.saxutils import escape
+
 import gyp
 import gyp.common
 import gyp.msvs_emulation
-import shlex
-import xml.etree.ElementTree as ET
 
 generator_wants_static_library_dependencies_adjusted = False
 
diff --git a/gyp/pylib/gyp/generator/gypd.py b/gyp/pylib/gyp/generator/gypd.py
index 4171704c47..a0aa6d9245 100644
--- a/gyp/pylib/gyp/generator/gypd.py
+++ b/gyp/pylib/gyp/generator/gypd.py
@@ -31,9 +31,9 @@
 """
 
 
-import gyp.common
 import pprint
 
+import gyp.common
 
 # These variables should just be spit back out as variable references.
 _generator_identity_variables = [
diff --git a/gyp/pylib/gyp/generator/gypsh.py b/gyp/pylib/gyp/generator/gypsh.py
index 8dfb1f1645..36a05deb7e 100644
--- a/gyp/pylib/gyp/generator/gypsh.py
+++ b/gyp/pylib/gyp/generator/gypsh.py
@@ -17,7 +17,6 @@
 import code
 import sys
 
-
 # All of this stuff about generator variables was lovingly ripped from gypd.py.
 # That module has a much better description of what's going on and why.
 _generator_identity_variables = [
diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py
index 634da8973c..e860479069 100644
--- a/gyp/pylib/gyp/generator/make.py
+++ b/gyp/pylib/gyp/generator/make.py
@@ -22,17 +22,17 @@
 # the side to keep the files readable.
 
 
+import hashlib
 import os
 import re
 import subprocess
 import sys
+
 import gyp
 import gyp.common
 import gyp.xcode_emulation
 from gyp.common import GetEnvironFallback
 
-import hashlib
-
 generator_default_variables = {
     "EXECUTABLE_PREFIX": "",
     "EXECUTABLE_SUFFIX": "",
@@ -1440,7 +1440,7 @@ def WriteSources(
         for obj in objs:
             assert " " not in obj, "Spaces in object filenames not supported (%s)" % obj
         self.WriteLn(
-            "# Add to the list of files we specially track " "dependencies for."
+            "# Add to the list of files we specially track dependencies for."
         )
         self.WriteLn("all_deps += $(OBJS)")
         self.WriteLn()
@@ -1450,7 +1450,7 @@ def WriteSources(
             self.WriteMakeRule(
                 ["$(OBJS)"],
                 deps,
-                comment="Make sure our dependencies are built " "before any of us.",
+                comment="Make sure our dependencies are built before any of us.",
                 order_only=True,
             )
 
@@ -1461,7 +1461,7 @@ def WriteSources(
             self.WriteMakeRule(
                 ["$(OBJS)"],
                 extra_outputs,
-                comment="Make sure our actions/rules run " "before any of us.",
+                comment="Make sure our actions/rules run before any of us.",
                 order_only=True,
             )
 
@@ -1699,7 +1699,7 @@ def WriteTarget(
             self.WriteMakeRule(
                 extra_outputs,
                 deps,
-                comment=("Preserve order dependency of " "special output on deps."),
+                comment=("Preserve order dependency of special output on deps."),
                 order_only=True,
             )
 
@@ -1738,7 +1738,8 @@ def WriteTarget(
                         # into the link command, so we need lots of escaping.
                         ldflags.append(r"-Wl,-rpath=\$$ORIGIN/")
                         ldflags.append(r"-Wl,-rpath-link=\$(builddir)/")
-                library_dirs = config.get("library_dirs", [])
+                if library_dirs := config.get("library_dirs", []):
+                    library_dirs = [Sourceify(self.Absolutify(i)) for i in library_dirs]
                 ldflags += [("-L%s" % library_dir) for library_dir in library_dirs]
                 self.WriteList(ldflags, "LDFLAGS_%s" % configname)
                 if self.flavor == "mac":
@@ -1844,7 +1845,7 @@ def WriteTarget(
                 "on the bundle, not the binary (target '%s')" % self.target
             )
             assert "product_dir" not in spec, (
-                "Postbuilds do not work with " "custom product_dir"
+                "Postbuilds do not work with custom product_dir"
             )
 
         if self.type == "executable":
@@ -1895,21 +1896,20 @@ def WriteTarget(
                         part_of_all,
                         postbuilds=postbuilds,
                     )
+            elif self.flavor in ("linux", "android"):
+                self.WriteMakeRule(
+                    [self.output_binary],
+                    link_deps,
+                    actions=["$(call create_archive,$@,$^)"],
+                )
             else:
-                if self.flavor in ("linux", "android"):
-                    self.WriteMakeRule(
-                        [self.output_binary],
-                        link_deps,
-                        actions=["$(call create_archive,$@,$^)"],
-                    )
-                else:
-                    self.WriteDoCmd(
-                        [self.output_binary],
-                        link_deps,
-                        "alink",
-                        part_of_all,
-                        postbuilds=postbuilds,
-                    )
+                self.WriteDoCmd(
+                    [self.output_binary],
+                    link_deps,
+                    "alink",
+                    part_of_all,
+                    postbuilds=postbuilds,
+                )
         elif self.type == "shared_library":
             self.WriteLn(
                 "%s: LD_INPUTS := %s"
diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py
index bea6e64348..b4aea2e69a 100644
--- a/gyp/pylib/gyp/generator/msvs.py
+++ b/gyp/pylib/gyp/generator/msvs.py
@@ -9,22 +9,21 @@
 import re
 import subprocess
 import sys
-
 from collections import OrderedDict
 
 import gyp.common
-import gyp.easy_xml as easy_xml
 import gyp.generator.ninja as ninja_generator
-import gyp.MSVSNew as MSVSNew
-import gyp.MSVSProject as MSVSProject
-import gyp.MSVSSettings as MSVSSettings
-import gyp.MSVSToolFile as MSVSToolFile
-import gyp.MSVSUserFile as MSVSUserFile
-import gyp.MSVSUtil as MSVSUtil
-import gyp.MSVSVersion as MSVSVersion
-from gyp.common import GypError
-from gyp.common import OrderedSet
-
+from gyp import (
+    MSVSNew,
+    MSVSProject,
+    MSVSSettings,
+    MSVSToolFile,
+    MSVSUserFile,
+    MSVSUtil,
+    MSVSVersion,
+    easy_xml,
+)
+from gyp.common import GypError, OrderedSet
 
 # Regular expression for validating Visual Studio GUIDs.  If the GUID
 # contains lowercase hex letters, MSVS will be fine. However,
@@ -185,7 +184,7 @@ def _IsWindowsAbsPath(path):
   it does not treat those as relative, which results in bad paths like:
   '..\\C:\\\\some_source_code_file.cc'
   """
-    return path.startswith("c:") or path.startswith("C:")
+    return path.startswith(("c:", "C:"))
 
 
 def _FixPaths(paths, separator="\\"):
@@ -2507,7 +2506,7 @@ def _GenerateMSBuildRuleTargetsFile(targets_path, msbuild_rules):
         rule_name = rule.rule_name
         target_outputs = "%%(%s.Outputs)" % rule_name
         target_inputs = (
-            "%%(%s.Identity);%%(%s.AdditionalDependencies);" "$(MSBuildProjectFile)"
+            "%%(%s.Identity);%%(%s.AdditionalDependencies);$(MSBuildProjectFile)"
         ) % (rule_name, rule_name)
         rule_inputs = "%%(%s.Identity)" % rule_name
         extension_condition = (
@@ -3100,9 +3099,7 @@ def _ConvertMSVSBuildAttributes(spec, config, build_file):
             msbuild_attributes[a] = _ConvertMSVSCharacterSet(msvs_attributes[a])
         elif a == "ConfigurationType":
             msbuild_attributes[a] = _ConvertMSVSConfigurationType(msvs_attributes[a])
-        elif a == "SpectreMitigation":
-            msbuild_attributes[a] = msvs_attributes[a]
-        elif a == "VCToolsVersion":
+        elif a == "SpectreMitigation" or a == "VCToolsVersion":
             msbuild_attributes[a] = msvs_attributes[a]
         else:
             print("Warning: Do not know how to convert MSVS attribute " + a)
@@ -3491,11 +3488,10 @@ def _VerifySourcesExist(sources, root_dir):
     for source in sources:
         if isinstance(source, MSVSProject.Filter):
             missing_sources.extend(_VerifySourcesExist(source.contents, root_dir))
-        else:
-            if "$" not in source:
-                full_path = os.path.join(root_dir, source)
-                if not os.path.exists(full_path):
-                    missing_sources.append(full_path)
+        elif "$" not in source:
+            full_path = os.path.join(root_dir, source)
+            if not os.path.exists(full_path):
+                missing_sources.append(full_path)
     return missing_sources
 
 
@@ -3565,75 +3561,74 @@ def _AddSources2(
                 sources_handled_by_action,
                 list_excluded,
             )
-        else:
-            if source not in sources_handled_by_action:
-                detail = []
-                excluded_configurations = exclusions.get(source, [])
-                if len(excluded_configurations) == len(spec["configurations"]):
-                    detail.append(["ExcludedFromBuild", "true"])
-                else:
-                    for config_name, configuration in sorted(excluded_configurations):
-                        condition = _GetConfigurationCondition(
-                            config_name, configuration
-                        )
-                        detail.append(
-                            ["ExcludedFromBuild", {"Condition": condition}, "true"]
-                        )
-                # Add precompile if needed
-                for config_name, configuration in spec["configurations"].items():
-                    precompiled_source = configuration.get(
-                        "msvs_precompiled_source", ""
+        elif source not in sources_handled_by_action:
+            detail = []
+            excluded_configurations = exclusions.get(source, [])
+            if len(excluded_configurations) == len(spec["configurations"]):
+                detail.append(["ExcludedFromBuild", "true"])
+            else:
+                for config_name, configuration in sorted(excluded_configurations):
+                    condition = _GetConfigurationCondition(
+                        config_name, configuration
                     )
-                    if precompiled_source != "":
-                        precompiled_source = _FixPath(precompiled_source)
-                        if not extensions_excluded_from_precompile:
-                            # If the precompiled header is generated by a C source,
-                            # we must not try to use it for C++ sources,
-                            # and vice versa.
-                            basename, extension = os.path.splitext(precompiled_source)
-                            if extension == ".c":
-                                extensions_excluded_from_precompile = [
-                                    ".cc",
-                                    ".cpp",
-                                    ".cxx",
-                                ]
-                            else:
-                                extensions_excluded_from_precompile = [".c"]
-
-                    if precompiled_source == source:
-                        condition = _GetConfigurationCondition(
-                            config_name, configuration, spec
-                        )
-                        detail.append(
-                            ["PrecompiledHeader", {"Condition": condition}, "Create"]
-                        )
-                    else:
-                        # Turn off precompiled header usage for source files of a
-                        # different type than the file that generated the
-                        # precompiled header.
-                        for extension in extensions_excluded_from_precompile:
-                            if source.endswith(extension):
-                                detail.append(["PrecompiledHeader", ""])
-                                detail.append(["ForcedIncludeFiles", ""])
-
-                group, element = _MapFileToMsBuildSourceType(
-                    source,
-                    rule_dependencies,
-                    extension_to_rule_name,
-                    _GetUniquePlatforms(spec),
-                    spec["toolset"],
+                    detail.append(
+                        ["ExcludedFromBuild", {"Condition": condition}, "true"]
+                    )
+            # Add precompile if needed
+            for config_name, configuration in spec["configurations"].items():
+                precompiled_source = configuration.get(
+                    "msvs_precompiled_source", ""
                 )
-                if group == "compile" and not os.path.isabs(source):
-                    # Add an  value to support duplicate source
-                    # file basenames, except for absolute paths to avoid paths
-                    # with more than 260 characters.
-                    file_name = os.path.splitext(source)[0] + ".obj"
-                    if file_name.startswith("..\\"):
-                        file_name = re.sub(r"^(\.\.\\)+", "", file_name)
-                    elif file_name.startswith("$("):
-                        file_name = re.sub(r"^\$\([^)]+\)\\", "", file_name)
-                    detail.append(["ObjectFileName", "$(IntDir)\\" + file_name])
-                grouped_sources[group].append([element, {"Include": source}] + detail)
+                if precompiled_source != "":
+                    precompiled_source = _FixPath(precompiled_source)
+                    if not extensions_excluded_from_precompile:
+                        # If the precompiled header is generated by a C source,
+                        # we must not try to use it for C++ sources,
+                        # and vice versa.
+                        basename, extension = os.path.splitext(precompiled_source)
+                        if extension == ".c":
+                            extensions_excluded_from_precompile = [
+                                ".cc",
+                                ".cpp",
+                                ".cxx",
+                            ]
+                        else:
+                            extensions_excluded_from_precompile = [".c"]
+
+                if precompiled_source == source:
+                    condition = _GetConfigurationCondition(
+                        config_name, configuration, spec
+                    )
+                    detail.append(
+                        ["PrecompiledHeader", {"Condition": condition}, "Create"]
+                    )
+                else:
+                    # Turn off precompiled header usage for source files of a
+                    # different type than the file that generated the
+                    # precompiled header.
+                    for extension in extensions_excluded_from_precompile:
+                        if source.endswith(extension):
+                            detail.append(["PrecompiledHeader", ""])
+                            detail.append(["ForcedIncludeFiles", ""])
+
+            group, element = _MapFileToMsBuildSourceType(
+                source,
+                rule_dependencies,
+                extension_to_rule_name,
+                _GetUniquePlatforms(spec),
+                spec["toolset"],
+            )
+            if group == "compile" and not os.path.isabs(source):
+                # Add an  value to support duplicate source
+                # file basenames, except for absolute paths to avoid paths
+                # with more than 260 characters.
+                file_name = os.path.splitext(source)[0] + ".obj"
+                if file_name.startswith("..\\"):
+                    file_name = re.sub(r"^(\.\.\\)+", "", file_name)
+                elif file_name.startswith("$("):
+                    file_name = re.sub(r"^\$\([^)]+\)\\", "", file_name)
+                detail.append(["ObjectFileName", "$(IntDir)\\" + file_name])
+            grouped_sources[group].append([element, {"Include": source}] + detail)
 
 
 def _GetMSBuildProjectReferences(project):
diff --git a/gyp/pylib/gyp/generator/msvs_test.py b/gyp/pylib/gyp/generator/msvs_test.py
index e80b57f06a..8cea3d1479 100755
--- a/gyp/pylib/gyp/generator/msvs_test.py
+++ b/gyp/pylib/gyp/generator/msvs_test.py
@@ -5,11 +5,11 @@
 
 """ Unit tests for the msvs.py file. """
 
-import gyp.generator.msvs as msvs
 import unittest
-
 from io import StringIO
 
+from gyp.generator import msvs
+
 
 class TestSequenceFunctions(unittest.TestCase):
     def setUp(self):
diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py
index ae3dded9b4..b7ac823d14 100644
--- a/gyp/pylib/gyp/generator/ninja.py
+++ b/gyp/pylib/gyp/generator/ninja.py
@@ -10,20 +10,18 @@
 import multiprocessing
 import os.path
 import re
-import signal
 import shutil
+import signal
 import subprocess
 import sys
+from io import StringIO
+
 import gyp
 import gyp.common
 import gyp.msvs_emulation
-import gyp.MSVSUtil as MSVSUtil
 import gyp.xcode_emulation
-
-from io import StringIO
-
+from gyp import MSVSUtil, ninja_syntax
 from gyp.common import GetEnvironFallback
-import gyp.ninja_syntax as ninja_syntax
 
 generator_default_variables = {
     "EXECUTABLE_PREFIX": "",
@@ -1465,7 +1463,7 @@ def WriteLinkForArch(
             # Respect environment variables related to build, but target-specific
             # flags can still override them.
             ldflags = env_ldflags + config.get("ldflags", [])
-            if is_executable and len(solibs):
+            if is_executable and solibs:
                 rpath = "lib/"
                 if self.toolset != "target":
                     rpath += self.toolset
@@ -1555,7 +1553,7 @@ def WriteLinkForArch(
             if pdbname:
                 output = [output, pdbname]
 
-        if len(solibs):
+        if solibs:
             extra_bindings.append(
                 ("solibs", gyp.common.EncodePOSIXShellList(sorted(solibs)))
             )
@@ -2085,7 +2083,7 @@ def CommandWithWrapper(cmd, wrappers, prog):
 
 def GetDefaultConcurrentLinks():
     """Returns a best-guess for a number of concurrent links."""
-    pool_size = int(os.environ.get("GYP_LINK_CONCURRENCY", 0))
+    pool_size = int(os.environ.get("GYP_LINK_CONCURRENCY") or 0)
     if pool_size:
         return pool_size
 
@@ -2112,7 +2110,7 @@ class MEMORYSTATUSEX(ctypes.Structure):
         # VS 2015 uses 20% more working set than VS 2013 and can consume all RAM
         # on a 64 GiB machine.
         mem_limit = max(1, stat.ullTotalPhys // (5 * (2 ** 30)))  # total / 5GiB
-        hard_cap = max(1, int(os.environ.get("GYP_LINK_CONCURRENCY_MAX", 2 ** 32)))
+        hard_cap = max(1, int(os.environ.get("GYP_LINK_CONCURRENCY_MAX") or 2 ** 32))
         return min(mem_limit, hard_cap)
     elif sys.platform.startswith("linux"):
         if os.path.exists("/proc/meminfo"):
@@ -2535,7 +2533,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name
             % {"suffix": "@$link_file_list"},
             rspfile="$link_file_list",
             rspfile_content=(
-                "-Wl,--whole-archive $in $solibs -Wl," "--no-whole-archive $libs"
+                "-Wl,--whole-archive $in $solibs -Wl,--no-whole-archive $libs"
             ),
             pool="link_pool",
         )
@@ -2684,7 +2682,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name
         master_ninja.rule(
             "link",
             description="LINK $out, POSTBUILDS",
-            command=("$ld $ldflags -o $out " "$in $solibs $libs$postbuilds"),
+            command=("$ld $ldflags -o $out $in $solibs $libs$postbuilds"),
             pool="link_pool",
         )
         master_ninja.rule(
diff --git a/gyp/pylib/gyp/generator/ninja_test.py b/gyp/pylib/gyp/generator/ninja_test.py
index 15cddfdf24..581b14595e 100644
--- a/gyp/pylib/gyp/generator/ninja_test.py
+++ b/gyp/pylib/gyp/generator/ninja_test.py
@@ -6,11 +6,11 @@
 
 """ Unit tests for the ninja.py file. """
 
-from pathlib import Path
 import sys
 import unittest
+from pathlib import Path
 
-import gyp.generator.ninja as ninja
+from gyp.generator import ninja
 
 
 class TestPrefixesAndSuffixes(unittest.TestCase):
diff --git a/gyp/pylib/gyp/generator/xcode.py b/gyp/pylib/gyp/generator/xcode.py
index c3c000c4ef..cdf11c3b27 100644
--- a/gyp/pylib/gyp/generator/xcode.py
+++ b/gyp/pylib/gyp/generator/xcode.py
@@ -3,19 +3,19 @@
 # found in the LICENSE file.
 
 
-import filecmp
-import gyp.common
-import gyp.xcodeproj_file
-import gyp.xcode_ninja
 import errno
+import filecmp
 import os
-import sys
 import posixpath
 import re
 import shutil
 import subprocess
+import sys
 import tempfile
 
+import gyp.common
+import gyp.xcode_ninja
+import gyp.xcodeproj_file
 
 # Project files generated by this module will use _intermediate_var as a
 # custom Xcode setting whose value is a DerivedSources-like directory that's
@@ -793,7 +793,7 @@ def GenerateOutput(target_list, target_dicts, data, params):
             except KeyError as e:
                 gyp.common.ExceptionAppend(
                     e,
-                    "-- unknown product type while " "writing target %s" % target_name,
+                    "-- unknown product type while writing target %s" % target_name,
                 )
                 raise
         else:
diff --git a/gyp/pylib/gyp/generator/xcode_test.py b/gyp/pylib/gyp/generator/xcode_test.py
index 49772d1f4d..b0b51a08a6 100644
--- a/gyp/pylib/gyp/generator/xcode_test.py
+++ b/gyp/pylib/gyp/generator/xcode_test.py
@@ -6,9 +6,10 @@
 
 """ Unit tests for the xcode.py file. """
 
-import gyp.generator.xcode as xcode
-import unittest
 import sys
+import unittest
+
+from gyp.generator import xcode
 
 
 class TestEscapeXcodeDefine(unittest.TestCase):
diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py
index 5e71fdace0..994bf6625f 100644
--- a/gyp/pylib/gyp/input.py
+++ b/gyp/pylib/gyp/input.py
@@ -4,9 +4,6 @@
 
 
 import ast
-
-import gyp.common
-import gyp.simple_copy
 import multiprocessing
 import os.path
 import re
@@ -16,10 +13,13 @@
 import sys
 import threading
 import traceback
-from gyp.common import GypError
-from gyp.common import OrderedSet
+
 from packaging.version import Version
 
+import gyp.common
+import gyp.simple_copy
+from gyp.common import GypError, OrderedSet
+
 # A list of types that are treated as linkable.
 linkable_types = [
     "executable",
@@ -990,25 +990,24 @@ def ExpandVariables(input, phase, variables, build_file):
                 )
                 replacement = cached_value
 
-        else:
-            if contents not in variables:
-                if contents[-1] in ["!", "/"]:
-                    # In order to allow cross-compiles (nacl) to happen more naturally,
-                    # we will allow references to >(sources/) etc. to resolve to
-                    # and empty list if undefined. This allows actions to:
-                    # 'action!': [
-                    #   '>@(_sources!)',
-                    # ],
-                    # 'action/': [
-                    #   '>@(_sources/)',
-                    # ],
-                    replacement = []
-                else:
-                    raise GypError(
-                        "Undefined variable " + contents + " in " + build_file
-                    )
+        elif contents not in variables:
+            if contents[-1] in ["!", "/"]:
+                # In order to allow cross-compiles (nacl) to happen more naturally,
+                # we will allow references to >(sources/) etc. to resolve to
+                # and empty list if undefined. This allows actions to:
+                # 'action!': [
+                #   '>@(_sources!)',
+                # ],
+                # 'action/': [
+                #   '>@(_sources/)',
+                # ],
+                replacement = []
             else:
-                replacement = variables[contents]
+                raise GypError(
+                    "Undefined variable " + contents + " in " + build_file
+                )
+        else:
+            replacement = variables[contents]
 
         if isinstance(replacement, bytes) and not isinstance(replacement, str):
             replacement = replacement.decode("utf-8")  # done on Python 3 only
@@ -1074,7 +1073,7 @@ def ExpandVariables(input, phase, variables, build_file):
     if output == input:
         gyp.DebugOutput(
             gyp.DEBUG_VARIABLES,
-            "Found only identity matches on %r, avoiding infinite " "recursion.",
+            "Found only identity matches on %r, avoiding infinite recursion.",
             output,
         )
     else:
diff --git a/gyp/pylib/gyp/input_test.py b/gyp/pylib/gyp/input_test.py
index a18f72e9eb..ff8c8fbecc 100755
--- a/gyp/pylib/gyp/input_test.py
+++ b/gyp/pylib/gyp/input_test.py
@@ -6,9 +6,10 @@
 
 """Unit tests for the input.py file."""
 
-import gyp.input
 import unittest
 
+import gyp.input
+
 
 class TestFindCycles(unittest.TestCase):
     def setUp(self):
diff --git a/gyp/pylib/gyp/mac_tool.py b/gyp/pylib/gyp/mac_tool.py
index 59647c9a89..70aab4f178 100755
--- a/gyp/pylib/gyp/mac_tool.py
+++ b/gyp/pylib/gyp/mac_tool.py
@@ -59,9 +59,7 @@ def ExecCopyBundleResource(self, source, dest, convert_to_binary):
             if os.path.exists(dest):
                 shutil.rmtree(dest)
             shutil.copytree(source, dest)
-        elif extension == ".xib":
-            return self._CopyXIBFile(source, dest)
-        elif extension == ".storyboard":
+        elif extension in {".xib", ".storyboard"}:
             return self._CopyXIBFile(source, dest)
         elif extension == ".strings" and not convert_to_binary:
             self._CopyStringsFile(source, dest)
@@ -70,7 +68,7 @@ def ExecCopyBundleResource(self, source, dest, convert_to_binary):
                 os.unlink(dest)
             shutil.copy(source, dest)
 
-        if convert_to_binary and extension in (".plist", ".strings"):
+        if convert_to_binary and extension in {".plist", ".strings"}:
             self._ConvertToBinary(dest)
 
     def _CopyXIBFile(self, source, dest):
@@ -164,9 +162,7 @@ def _DetectInputEncoding(self, file_name):
                 header = fp.read(3)
             except Exception:
                 return None
-        if header.startswith(b"\xFE\xFF"):
-            return "UTF-16"
-        elif header.startswith(b"\xFF\xFE"):
+        if header.startswith((b"\xFE\xFF", b"\xFF\xFE")):
             return "UTF-16"
         elif header.startswith(b"\xEF\xBB\xBF"):
             return "UTF-8"
@@ -261,7 +257,7 @@ def ExecFilterLibtool(self, *cmd_list):
         """Calls libtool and filters out '/path/to/libtool: file: foo.o has no
     symbols'."""
         libtool_re = re.compile(
-            r"^.*libtool: (?:for architecture: \S* )?" r"file: .* has no symbols$"
+            r"^.*libtool: (?:for architecture: \S* )?file: .* has no symbols$"
         )
         libtool_re5 = re.compile(
             r"^.*libtool: warning for library: "
diff --git a/gyp/pylib/gyp/msvs_emulation.py b/gyp/pylib/gyp/msvs_emulation.py
index adda5a0273..ace0cae5eb 100644
--- a/gyp/pylib/gyp/msvs_emulation.py
+++ b/gyp/pylib/gyp/msvs_emulation.py
@@ -7,15 +7,15 @@
 build systems, primarily ninja.
 """
 
-import collections
 import os
 import re
 import subprocess
 import sys
+from collections import namedtuple
 
-from gyp.common import OrderedSet
 import gyp.MSVSUtil
 import gyp.MSVSVersion
+from gyp.common import OrderedSet
 
 windows_quoter_regex = re.compile(r'(\\*)"')
 
@@ -933,7 +933,7 @@ def BuildCygwinBashCommandLine(self, args, path_to_base):
         )
         return cmd
 
-    RuleShellFlags = collections.namedtuple("RuleShellFlags", ["cygwin", "quote"])
+    RuleShellFlags = namedtuple("RuleShellFlags", ["cygwin", "quote"])  # noqa: PYI024
 
     def GetRuleShellFlags(self, rule):
         """Return RuleShellFlags about how the given rule should be run. This
diff --git a/gyp/pylib/gyp/win_tool.py b/gyp/pylib/gyp/win_tool.py
index 171d729574..7e647f40a8 100755
--- a/gyp/pylib/gyp/win_tool.py
+++ b/gyp/pylib/gyp/win_tool.py
@@ -13,9 +13,9 @@
 import os
 import re
 import shutil
-import subprocess
 import stat
 import string
+import subprocess
 import sys
 
 BASE_DIR = os.path.dirname(os.path.abspath(__file__))
diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py
index aee1a542da..85a63dfd7a 100644
--- a/gyp/pylib/gyp/xcode_emulation.py
+++ b/gyp/pylib/gyp/xcode_emulation.py
@@ -9,13 +9,14 @@
 
 
 import copy
-import gyp.common
 import os
 import os.path
 import re
 import shlex
 import subprocess
 import sys
+
+import gyp.common
 from gyp.common import GypError
 
 # Populated lazily by XcodeVersion, for efficiency, and to fix an issue when
@@ -471,17 +472,14 @@ def _GetStandaloneBinaryPath(self):
         """Returns the name of the non-bundle binary represented by this target.
     E.g. hello_world. Only valid for non-bundles."""
         assert not self._IsBundle()
-        assert self.spec["type"] in (
+        assert self.spec["type"] in {
             "executable",
             "shared_library",
             "static_library",
             "loadable_module",
-        ), ("Unexpected type %s" % self.spec["type"])
+        }, ("Unexpected type %s" % self.spec["type"])
         target = self.spec["target_name"]
-        if self.spec["type"] == "static_library":
-            if target[:3] == "lib":
-                target = target[3:]
-        elif self.spec["type"] in ("loadable_module", "shared_library"):
+        if self.spec["type"] in {"loadable_module", "shared_library", "static_library"}:
             if target[:3] == "lib":
                 target = target[3:]
 
diff --git a/gyp/pylib/gyp/xcode_emulation_test.py b/gyp/pylib/gyp/xcode_emulation_test.py
index 98b02320d5..03cbbaea84 100644
--- a/gyp/pylib/gyp/xcode_emulation_test.py
+++ b/gyp/pylib/gyp/xcode_emulation_test.py
@@ -2,10 +2,11 @@
 
 """Unit tests for the xcode_emulation.py file."""
 
-from gyp.xcode_emulation import XcodeSettings
 import sys
 import unittest
 
+from gyp.xcode_emulation import XcodeSettings
+
 
 class TestXcodeSettings(unittest.TestCase):
     def setUp(self):
diff --git a/gyp/pylib/gyp/xcode_ninja.py b/gyp/pylib/gyp/xcode_ninja.py
index bb74eacbea..cac1af56f7 100644
--- a/gyp/pylib/gyp/xcode_ninja.py
+++ b/gyp/pylib/gyp/xcode_ninja.py
@@ -13,11 +13,12 @@
 """
 
 import errno
-import gyp.generator.ninja
 import os
 import re
 import xml.sax.saxutils
 
+import gyp.generator.ninja
+
 
 def _WriteWorkspace(main_gyp, sources_gyp, params):
     """ Create a workspace to wrap main and sources gyp paths. """
diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py
index cd72aa262d..be17ef946d 100644
--- a/gyp/pylib/gyp/xcodeproj_file.py
+++ b/gyp/pylib/gyp/xcodeproj_file.py
@@ -137,14 +137,15 @@
 a project file is output.
 """
 
-import gyp.common
-from functools import cmp_to_key
 import hashlib
-from operator import attrgetter
 import posixpath
 import re
 import struct
 import sys
+from functools import cmp_to_key
+from operator import attrgetter
+
+import gyp.common
 
 
 def cmp(x, y):
@@ -460,7 +461,7 @@ def _HashUpdate(hash, data):
             digest_int_count = hash.digest_size // 4
             digest_ints = struct.unpack(">" + "I" * digest_int_count, hash.digest())
             id_ints = [0, 0, 0]
-            for index in range(0, digest_int_count):
+            for index in range(digest_int_count):
                 id_ints[index % 3] ^= digest_ints[index]
             self.id = "%08X%08X%08X" % tuple(id_ints)
 
@@ -1640,7 +1641,6 @@ class PBXVariantGroup(PBXGroup, XCFileLikeElement):
     """PBXVariantGroup is used by Xcode to represent localizations."""
 
     # No additions to the schema relative to PBXGroup.
-    pass
 
 
 # PBXReferenceProxy is also an XCFileLikeElement subclass.  It is defined below
@@ -1766,9 +1766,8 @@ def GetBuildSetting(self, key):
             configuration_value = configuration.GetBuildSetting(key)
             if value is None:
                 value = configuration_value
-            else:
-                if value != configuration_value:
-                    raise ValueError("Variant values for " + key)
+            elif value != configuration_value:
+                raise ValueError("Variant values for " + key)
 
         return value
 
@@ -1924,14 +1923,13 @@ def _AddBuildFileToDicts(self, pbxbuildfile, path=None):
             # It's best when the caller provides the path.
             if isinstance(xcfilelikeelement, PBXVariantGroup):
                 paths.append(path)
+        # If the caller didn't provide a path, there can be either multiple
+        # paths (PBXVariantGroup) or one.
+        elif isinstance(xcfilelikeelement, PBXVariantGroup):
+            for variant in xcfilelikeelement._properties["children"]:
+                paths.append(variant.FullPath())
         else:
-            # If the caller didn't provide a path, there can be either multiple
-            # paths (PBXVariantGroup) or one.
-            if isinstance(xcfilelikeelement, PBXVariantGroup):
-                for variant in xcfilelikeelement._properties["children"]:
-                    paths.append(variant.FullPath())
-            else:
-                paths.append(xcfilelikeelement.FullPath())
+            paths.append(xcfilelikeelement.FullPath())
 
         # Add the paths first, because if something's going to raise, the
         # messages provided by _AddPathToDict are more useful owing to its
diff --git a/gyp/pyproject.toml b/gyp/pyproject.toml
index 4b0c88c8a2..537308731f 100644
--- a/gyp/pyproject.toml
+++ b/gyp/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "gyp-next"
-version = "0.19.1"
+version = "0.20.0"
 authors = [
   { name="Node.js contributors", email="ryzokuken@disroot.org" },
 ]
@@ -92,14 +92,9 @@ select = [
   # "TRY",  # tryceratops
 ]
 ignore = [
-  "PLC1901",
-  "PLR0402",
   "PLR1714",
-  "PLR2004",
-  "PLR5501",
   "PLW0603",
   "PLW2901",
-  "PYI024",
   "RUF005",
   "RUF012",
   "UP031",
@@ -109,6 +104,7 @@ ignore = [
 max-complexity = 101
 
 [tool.ruff.lint.pylint]
+allow-magic-value-types = ["float", "int", "str"]
 max-args = 11
 max-branches = 108
 max-returns = 10
diff --git a/gyp/tools/pretty_gyp.py b/gyp/tools/pretty_gyp.py
index 6eef3a1bbf..a023887205 100755
--- a/gyp/tools/pretty_gyp.py
+++ b/gyp/tools/pretty_gyp.py
@@ -7,9 +7,8 @@
 """Pretty-prints the contents of a GYP file."""
 
 
-import sys
 import re
-
+import sys
 
 # Regex to remove comments when we're counting braces.
 COMMENT_RE = re.compile(r"\s*#.*")
@@ -136,7 +135,7 @@ def prettyprint_input(lines):
                 else:
                     print(" " * (basic_offset * indent) + line)
             else:
-                print("")
+                print()
 
 
 def main():
diff --git a/gyp/tools/pretty_sln.py b/gyp/tools/pretty_sln.py
index 8026699d39..850fb150f4 100755
--- a/gyp/tools/pretty_sln.py
+++ b/gyp/tools/pretty_sln.py
@@ -16,6 +16,7 @@
 import os
 import re
 import sys
+
 import pretty_vcproj
 
 __author__ = "nsylvain (Nicolas Sylvain)"
@@ -118,7 +119,7 @@ def PrintDependencies(projects, deps):
         if dep_list:
             for dep in dep_list:
                 print("  - %s" % dep)
-        print("")
+        print()
 
     print("--                                   --")
 
diff --git a/gyp/tools/pretty_vcproj.py b/gyp/tools/pretty_vcproj.py
index 862400358a..c7427ed2d8 100755
--- a/gyp/tools/pretty_vcproj.py
+++ b/gyp/tools/pretty_vcproj.py
@@ -15,9 +15,7 @@
 
 import os
 import sys
-
-from xml.dom.minidom import parse
-from xml.dom.minidom import Node
+from xml.dom.minidom import Node, parse
 
 __author__ = "nsylvain (Nicolas Sylvain)"
 ARGUMENTS = None

From a2772a76709f939af1e80dd8fe766ca2143aa5bf Mon Sep 17 00:00:00 2001
From: Aman Karmani 
Date: Tue, 1 Apr 2025 07:53:48 -0500
Subject: [PATCH 466/551] fix: use maxRetries on fs.rm calls (#3113)

* fix: remove more forcefully

* add max retries to all fs.rm calls

---------

Co-authored-by: Luke Karrys 
---
 lib/build.js   | 2 +-
 lib/clean.js   | 2 +-
 lib/install.js | 2 +-
 lib/remove.js  | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/lib/build.js b/lib/build.js
index cae506e842..9c0cca8fc2 100644
--- a/lib/build.js
+++ b/lib/build.js
@@ -212,7 +212,7 @@ async function build (gyp, argv) {
     await new Promise((resolve, reject) => proc.on('exit', async (code, signal) => {
       if (buildBinsDir) {
         // Clean up the build-time dependency symlinks:
-        await fs.rm(buildBinsDir, { recursive: true })
+        await fs.rm(buildBinsDir, { recursive: true, maxRetries: 3 })
       }
 
       if (code !== 0) {
diff --git a/lib/clean.js b/lib/clean.js
index 523f8016ca..479c374f10 100644
--- a/lib/clean.js
+++ b/lib/clean.js
@@ -8,7 +8,7 @@ async function clean (gyp, argv) {
   const buildDir = 'build'
 
   log.verbose('clean', 'removing "%s" directory', buildDir)
-  await fs.rm(buildDir, { recursive: true, force: true })
+  await fs.rm(buildDir, { recursive: true, force: true, maxRetries: 3 })
 }
 
 module.exports = clean
diff --git a/lib/install.js b/lib/install.js
index 7196a31629..90be86c822 100644
--- a/lib/install.js
+++ b/lib/install.js
@@ -284,7 +284,7 @@ async function install (gyp, argv) {
       if (tarExtractDir !== devDir) {
         try {
           // try to cleanup temp dir
-          await fs.rm(tarExtractDir, { recursive: true })
+          await fs.rm(tarExtractDir, { recursive: true, maxRetries: 3 })
         } catch {
           log.warn('failed to clean up temp tarball extract directory')
         }
diff --git a/lib/remove.js b/lib/remove.js
index 7efdb01a66..55736f71d9 100644
--- a/lib/remove.js
+++ b/lib/remove.js
@@ -36,7 +36,7 @@ async function remove (gyp, argv) {
     throw err
   }
 
-  await fs.rm(versionPath, { recursive: true, force: true })
+  await fs.rm(versionPath, { recursive: true, force: true, maxRetries: 3 })
 }
 
 module.exports = remove

From 0ccbe7e90afb096b46a7818ba127a4871237952e Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Tue, 1 Apr 2025 15:23:53 +0200
Subject: [PATCH 467/551] test: use maxRetries with tests too (#3150)

---
 test/test-download.js |  2 +-
 test/test-install.js  | 12 +++++++-----
 2 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/test/test-download.js b/test/test-download.js
index 1d0f3ab286..a746c98cc6 100644
--- a/test/test-download.js
+++ b/test/test-download.js
@@ -163,7 +163,7 @@ describe('download', function () {
     this.timeout(platformTimeout(1, { win32: 5 }))
 
     const expectedDir = path.join(devDir, process.version.replace(/^v/, ''))
-    await fs.rm(expectedDir, { recursive: true, force: true })
+    await fs.rm(expectedDir, { recursive: true, force: true, maxRetries: 3 })
 
     const prog = gyp()
     prog.parseArgv([])
diff --git a/test/test-install.js b/test/test-install.js
index de91b26ad8..3343545ba6 100644
--- a/test/test-install.js
+++ b/test/test-install.js
@@ -63,7 +63,7 @@ describe('install', function () {
     })
 
     afterEach(async () => {
-      await rm(prog.devDir, { recursive: true, force: true })
+      await rm(prog.devDir, { recursive: true, force: true, maxRetries: 3 })
       prog = null
     })
 
@@ -74,14 +74,16 @@ describe('install', function () {
       }
 
       return it(name, async function () {
-        this.timeout(platformTimeout(1, { win32: 20 }))
-        const start = Date.now()
+        this.timeout(platformTimeout(2, { win32: 20 }))
         await fn.call(this)
         const expectedDir = path.join(prog.devDir, process.version.replace(/^v/, ''))
-        await rm(expectedDir, { recursive: true, force: true })
+        await rm(expectedDir, { recursive: true, force: true, maxRetries: 3 })
         await Promise.all(new Array(10).fill(0).map(async (_, i) => {
+          const title = `${' '.repeat(8)}${name} ${(i + 1).toString().padEnd(2, ' ')}`
+          console.log(`${title} : Start`)
+          console.time(title)
           await install(prog, [])
-          console.log(`${' '.repeat(8)}${name} ${(i + 1).toString().padEnd(2, ' ')} (${Date.now() - start}ms)`)
+          console.timeEnd(title)
         }))
       })
     }

From fc557e4b2bc1115a5fce6bb5b74739bfeac4aba6 Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Tue, 1 Apr 2025 09:24:24 -0400
Subject: [PATCH 468/551] chore(main): release 11.2.0

---
 .release-please-manifest.json |  2 +-
 CHANGELOG.md                  | 30 ++++++++++++++++++++++++++++++
 package.json                  |  2 +-
 3 files changed, 32 insertions(+), 2 deletions(-)

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
index b913264022..f098464b1f 100644
--- a/.release-please-manifest.json
+++ b/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "11.1.0"
+    ".": "11.2.0"
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3bb923e6a0..e206e5d9f3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,35 @@
 # Changelog
 
+## [11.2.0](https://github.com/nodejs/node-gyp/compare/v11.1.0...v11.2.0) (2025-04-01)
+
+
+### Features
+
+* update gyp-next to v0.20.0 ([#3149](https://github.com/nodejs/node-gyp/issues/3149)) ([80e9c79](https://github.com/nodejs/node-gyp/commit/80e9c795a739c490cfbc85633e63022b36a7c70d))
+
+
+### Bug Fixes
+
+* disable msbuild.exe nodeReuse ([#3112](https://github.com/nodejs/node-gyp/issues/3112)) ([0cf16d2](https://github.com/nodejs/node-gyp/commit/0cf16d29fe604266fb47325496287a63075ea532))
+* use maxRetries on fs.rm calls ([#3113](https://github.com/nodejs/node-gyp/issues/3113)) ([a2772a7](https://github.com/nodejs/node-gyp/commit/a2772a76709f939af1e80dd8fe766ca2143aa5bf))
+
+
+### Tests
+
+* fix wasm test on Windows ([#3145](https://github.com/nodejs/node-gyp/issues/3145)) ([ee1d6fd](https://github.com/nodejs/node-gyp/commit/ee1d6fd8d83c9dd3eae7df7ec533bb6b39e1a812))
+* use maxRetries with tests too ([#3150](https://github.com/nodejs/node-gyp/issues/3150)) ([0ccbe7e](https://github.com/nodejs/node-gyp/commit/0ccbe7e90afb096b46a7818ba127a4871237952e))
+
+
+### Doc
+
+* add ffi-napi to docs/README.md ([#3138](https://github.com/nodejs/node-gyp/issues/3138)) ([4885110](https://github.com/nodejs/node-gyp/commit/48851107ad8c5d2cf18a55e8bd2764f5938e7102))
+
+
+### Miscellaneous
+
+* switch to tinyglobby ([#3133](https://github.com/nodejs/node-gyp/issues/3133)) ([c3b3ab0](https://github.com/nodejs/node-gyp/commit/c3b3ab06ee0f092cd5c0646120d57e56d41b79fc))
+* update tinyglobby ([#3136](https://github.com/nodejs/node-gyp/issues/3136)) ([b21cf87](https://github.com/nodejs/node-gyp/commit/b21cf874f58883f3fd4dd07bec3b584fb07e831d))
+
 ## [11.1.0](https://github.com/nodejs/node-gyp/compare/v11.0.0...v11.1.0) (2025-02-10)
 
 
diff --git a/package.json b/package.json
index b5d81d463e..f69a022ef3 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "11.1.0",
+  "version": "11.2.0",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From 8f3cd8b3a157bccd8d7110e7d46a27c2926625cd Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Wed, 2 Apr 2025 11:09:15 +0200
Subject: [PATCH 469/551] chore: retry wasi-sdk download in CI (#3151)

---
 .github/workflows/tests.yml | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index e9585af26f..6bf975cf9f 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -129,7 +129,27 @@ jobs:
         shell: pwsh
         if: runner.os == 'Windows'
         run: |
-          Start-BitsTransfer -Source https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${env:WASI_VERSION}/wasi-sdk-${env:WASI_VERSION_FULL}-x86_64-windows.tar.gz
+          $MaxRetries = 3
+          $RetryDelay = 10
+          $Attempt = 0
+          while ($Attempt -lt $MaxRetries) {
+            try {
+              Start-BitsTransfer -Source https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${env:WASI_VERSION}/wasi-sdk-${env:WASI_VERSION_FULL}-x86_64-windows.tar.gz
+              break
+            }
+            catch {
+              Write-Host "Error: $($_.Exception.Message)"
+              $Attempt++
+              if ($Attempt -lt $MaxRetries) {
+                Write-Host "Retrying in $RetryDelay seconds"
+                Start-Sleep -Seconds $RetryDelay
+              }
+              else {
+                Write-Host "Max retries reached. Download failed."
+                exit 1
+              }
+            }
+          }
           New-Item -ItemType Directory -Path ${env:WASI_SDK_PATH}
           tar -zxvf wasi-sdk-${env:WASI_VERSION_FULL}-x86_64-windows.tar.gz -C ${env:WASI_SDK_PATH} --strip 1
       - name: Install Dependencies

From 7d883b5cf4c26e76065201f85b0be36d5ebdcc0e Mon Sep 17 00:00:00 2001
From: Liam Mitchell 
Date: Sat, 12 Apr 2025 00:35:13 +0200
Subject: [PATCH 470/551] fix: Correct Visual Studio 2019 test version (#3153)

---
 .github/workflows/visual-studio.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/visual-studio.yml b/.github/workflows/visual-studio.yml
index 3412837422..e36639d7eb 100644
--- a/.github/workflows/visual-studio.yml
+++ b/.github/workflows/visual-studio.yml
@@ -18,7 +18,7 @@ jobs:
       matrix:
         include:
           - os: windows-2019
-            msvs-verison: 2019
+            msvs-version: 2019
           - os: windows-2022
             msvs-version: 2022
           - os: windows-2025

From 0e656322c1e94041331ab3b01bf66c2ef9bd6ead Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Thu, 17 Jul 2025 07:52:21 -0400
Subject: [PATCH 471/551] feat: update gyp-next to v0.20.2 (#3169)

* feat: update gyp-next to v0.20.2
---
 gyp/.github/workflows/node-gyp.yml |  7 ++++--
 gyp/.github/workflows/nodejs.yml   |  6 +++--
 gyp/.release-please-manifest.json  |  2 +-
 gyp/CHANGELOG.md                   | 15 +++++++++++++
 gyp/pylib/gyp/MSVSSettings.py      |  3 +--
 gyp/pylib/gyp/MSVSVersion.py       |  5 ++---
 gyp/pylib/gyp/__init__.py          |  2 +-
 gyp/pylib/gyp/common.py            | 36 ++++++++++++++++++++++--------
 gyp/pylib/gyp/common_test.py       | 32 +++++++++++++++++++-------
 gyp/pylib/gyp/generator/android.py |  3 +--
 gyp/pylib/gyp/generator/cmake.py   |  6 ++---
 gyp/pylib/gyp/generator/eclipse.py |  3 +--
 gyp/pylib/gyp/generator/make.py    | 10 ++++-----
 gyp/pylib/gyp/generator/msvs.py    | 12 ++++------
 gyp/pylib/gyp/generator/ninja.py   | 22 +++++++-----------
 gyp/pylib/gyp/mac_tool.py          |  5 ++---
 gyp/pylib/gyp/msvs_emulation.py    | 16 +++++--------
 gyp/pylib/gyp/win_tool.py          |  3 +--
 gyp/pylib/gyp/xcode_emulation.py   |  9 +++-----
 gyp/pylib/gyp/xcode_ninja.py       |  3 +--
 gyp/pylib/gyp/xcodeproj_file.py    | 12 ++++------
 gyp/pylib/packaging/_elffile.py    |  3 +--
 gyp/pylib/packaging/markers.py     |  3 +--
 gyp/pylib/packaging/metadata.py    |  3 +--
 gyp/pyproject.toml                 |  3 +--
 gyp/test_gyp.py                    |  4 ++--
 26 files changed, 122 insertions(+), 106 deletions(-)

diff --git a/gyp/.github/workflows/node-gyp.yml b/gyp/.github/workflows/node-gyp.yml
index 56cdb4086c..c13b28effb 100644
--- a/gyp/.github/workflows/node-gyp.yml
+++ b/gyp/.github/workflows/node-gyp.yml
@@ -14,10 +14,13 @@ jobs:
         python-version: ["3.9", "3.11", "3.13"]
         include:
           - node-version: "22"
-            os: macos-13
+            os: macos-13  # macOS on Intel
             python-version: "3.13"
           - node-version: "22"
-            os: windows-2025
+            os: ubuntu-24.04-arm  # Ubuntu on ARM
+            python-version: "3.13"
+          - node-version: "22"
+            os: windows-11-arm  # Windows on ARM
             python-version: "3.13"
     runs-on: ${{ matrix.os }}
     steps:
diff --git a/gyp/.github/workflows/nodejs.yml b/gyp/.github/workflows/nodejs.yml
index 213fcbd0c1..416dfe45d1 100644
--- a/gyp/.github/workflows/nodejs.yml
+++ b/gyp/.github/workflows/nodejs.yml
@@ -12,9 +12,11 @@ jobs:
         os: [macos-latest, ubuntu-latest, windows-latest]
         python: ["3.9", "3.11", "3.13"]
         include:
-          - os: macos-13
+          - os: macos-13  # macOS on Intel
             python-version: "3.13"
-          - os: windows-2025
+          - os: ubuntu-24.04-arm  # Ubuntu on ARM
+            python-version: "3.13"
+          - os: windows-11-arm  # Windows on ARM
             python-version: "3.13"
 
     runs-on: ${{ matrix.os }}
diff --git a/gyp/.release-please-manifest.json b/gyp/.release-please-manifest.json
index 589cd4553e..69ae3d039e 100644
--- a/gyp/.release-please-manifest.json
+++ b/gyp/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "0.20.0"
+    ".": "0.20.2"
 }
diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md
index 3f676055a6..c1c1c5909d 100644
--- a/gyp/CHANGELOG.md
+++ b/gyp/CHANGELOG.md
@@ -1,5 +1,20 @@
 # Changelog
 
+## [0.20.2](https://github.com/nodejs/gyp-next/compare/v0.20.1...v0.20.2) (2025-06-22)
+
+
+### Bug Fixes
+
+* Python lint import-outside-top-level ruff rule PLC0415 ([#298](https://github.com/nodejs/gyp-next/issues/298)) ([34f4df6](https://github.com/nodejs/gyp-next/commit/34f4df614936ee6a056e47406ebbe7e3c1cb6540))
+
+## [0.20.1](https://github.com/nodejs/gyp-next/compare/v0.20.0...v0.20.1) (2025-06-06)
+
+
+### Bug Fixes
+
+* Ensure Consistent Order of build_files in WriteAutoRegenerationRule ([#293](https://github.com/nodejs/gyp-next/issues/293)) ([59b5903](https://github.com/nodejs/gyp-next/commit/59b59035f4ae63419343ffdafe0f0ff511ada17d))
+* ignore failure of `GetCompilerPredefines` ([#295](https://github.com/nodejs/gyp-next/issues/295)) ([0eaea29](https://github.com/nodejs/gyp-next/commit/0eaea297f0fbb0869597aa162f66f78eb2468fad))
+
 ## [0.20.0](https://github.com/nodejs/gyp-next/compare/v0.19.1...v0.20.0) (2025-03-27)
 
 
diff --git a/gyp/pylib/gyp/MSVSSettings.py b/gyp/pylib/gyp/MSVSSettings.py
index fea6e67286..8f8f53bb71 100644
--- a/gyp/pylib/gyp/MSVSSettings.py
+++ b/gyp/pylib/gyp/MSVSSettings.py
@@ -396,8 +396,7 @@ def _ValidateExclusionSetting(setting, settings, error_msg, stderr=sys.stderr):
     # This may be unrecognized because it's an exclusion list. If the
     # setting name has the _excluded suffix, then check the root name.
     unrecognized = True
-    m = re.match(_EXCLUDED_SUFFIX_RE, setting)
-    if m:
+    if m := re.match(_EXCLUDED_SUFFIX_RE, setting):
         root_setting = m.group(1)
         unrecognized = root_setting not in settings
 
diff --git a/gyp/pylib/gyp/MSVSVersion.py b/gyp/pylib/gyp/MSVSVersion.py
index 93f48bc05c..a34631b452 100644
--- a/gyp/pylib/gyp/MSVSVersion.py
+++ b/gyp/pylib/gyp/MSVSVersion.py
@@ -219,7 +219,7 @@ def _RegistryGetValueUsingWinReg(key, value):
     contents of the registry key's value, or None on failure.  Throws
     ImportError if winreg is unavailable.
   """
-    from winreg import HKEY_LOCAL_MACHINE, OpenKey, QueryValueEx
+    from winreg import HKEY_LOCAL_MACHINE, OpenKey, QueryValueEx  # noqa: PLC0415
     try:
         root, subkey = key.split("\\", 1)
         assert root == "HKLM"  # Only need HKLM for now.
@@ -552,8 +552,7 @@ def SelectVisualStudioVersion(version="auto", allow_fallback=True):
         "2019": ("16.0",),
         "2022": ("17.0",),
     }
-    override_path = os.environ.get("GYP_MSVS_OVERRIDE_PATH")
-    if override_path:
+    if override_path := os.environ.get("GYP_MSVS_OVERRIDE_PATH"):
         msvs_version = os.environ.get("GYP_MSVS_VERSION")
         if not msvs_version:
             raise ValueError(
diff --git a/gyp/pylib/gyp/__init__.py b/gyp/pylib/gyp/__init__.py
index 77800661a4..efc15fc62f 100755
--- a/gyp/pylib/gyp/__init__.py
+++ b/gyp/pylib/gyp/__init__.py
@@ -489,7 +489,7 @@ def gyp_main(args):
 
     options, build_files_arg = parser.parse_args(args)
     if options.version:
-        import pkg_resources
+        import pkg_resources  # noqa: PLC0415
         print(f"v{pkg_resources.get_distribution('gyp-next').version}")
         return 0
     build_files = build_files_arg
diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py
index fbf1024fc3..9431c432c8 100644
--- a/gyp/pylib/gyp/common.py
+++ b/gyp/pylib/gyp/common.py
@@ -421,8 +421,9 @@ def EnsureDirExists(path):
     except OSError:
         pass
 
-def GetCrossCompilerPredefines():  # -> dict
+def GetCompilerPredefines():  # -> dict
     cmd = []
+    defines = {}
 
     # shlex.split() will eat '\' in posix mode, but
     # setting posix=False will preserve extra '"' cause CreateProcess fail on Windows
@@ -439,7 +440,7 @@ def replace_sep(s):
         if CXXFLAGS := os.environ.get("CXXFLAGS"):
             cmd += shlex.split(replace_sep(CXXFLAGS))
     else:
-        return {}
+        return defines
 
     if sys.platform == "win32":
         fd, input = tempfile.mkstemp(suffix=".c")
@@ -450,17 +451,33 @@ def replace_sep(s):
                 real_cmd, shell=True,
                 capture_output=True, check=True
             ).stdout
+        except subprocess.CalledProcessError as e:
+            print(
+                "Warning: failed to get compiler predefines\n"
+                "cmd: %s\n"
+                "status: %d" % (e.cmd, e.returncode),
+                file=sys.stderr
+            )
+            return defines
         finally:
             os.unlink(input)
     else:
         input = "/dev/null"
         real_cmd = [*cmd, "-dM", "-E", "-x", "c", input]
-        stdout = subprocess.run(
-            real_cmd, shell=False,
-            capture_output=True, check=True
-        ).stdout
+        try:
+            stdout = subprocess.run(
+                real_cmd, shell=False,
+                capture_output=True, check=True
+            ).stdout
+        except subprocess.CalledProcessError as e:
+            print(
+                "Warning: failed to get compiler predefines\n"
+                "cmd: %s\n"
+                "status: %d" % (e.cmd, e.returncode),
+                file=sys.stderr
+            )
+            return defines
 
-    defines = {}
     lines = stdout.decode("utf-8").replace("\r\n", "\n").split("\n")
     for line in lines:
         if (line or "").startswith("#define "):
@@ -499,7 +516,7 @@ def GetFlavor(params):
     if "flavor" in params:
         return params["flavor"]
 
-    defines = GetCrossCompilerPredefines()
+    defines = GetCompilerPredefines()
     if "__EMSCRIPTEN__" in defines:
         return "emscripten"
     if "__wasm__" in defines:
@@ -566,7 +583,8 @@ def uniquer(seq, idfun=lambda x: x):
 
 
 # Based on http://code.activestate.com/recipes/576694/.
-class OrderedSet(MutableSet):
+class OrderedSet(MutableSet):  # noqa: PLW1641
+    # TODO (cclauss): Fix eq-without-hash ruff rule PLW1641
     def __init__(self, iterable=None):
         self.end = end = []
         end += [None, end, end]  # sentinel node for doubly linked list
diff --git a/gyp/pylib/gyp/common_test.py b/gyp/pylib/gyp/common_test.py
index bd7172afaf..d25e621ab5 100755
--- a/gyp/pylib/gyp/common_test.py
+++ b/gyp/pylib/gyp/common_test.py
@@ -7,6 +7,7 @@
 """Unit tests for the common.py file."""
 
 import os
+import subprocess
 import sys
 import unittest
 from unittest.mock import MagicMock, patch
@@ -85,22 +86,34 @@ def decode(self, encoding):
     @patch("os.close")
     @patch("os.unlink")
     @patch("tempfile.mkstemp")
-    def test_GetCrossCompilerPredefines(self, mock_mkstemp, mock_unlink, mock_close):
+    def test_GetCompilerPredefines(self, mock_mkstemp, mock_unlink, mock_close):
         mock_close.return_value = None
         mock_unlink.return_value = None
         mock_mkstemp.return_value = (0, "temp.c")
 
-        def mock_run(env, defines_stdout, expected_cmd):
+        def mock_run(env, defines_stdout, expected_cmd, throws=False):
             with patch("subprocess.run") as mock_run:
-                mock_process = MagicMock()
-                mock_process.returncode = 0
-                mock_process.stdout = TestGetFlavor.MockCommunicate(defines_stdout)
-                mock_run.return_value = mock_process
                 expected_input = "temp.c" if sys.platform == "win32" else "/dev/null"
+                if throws:
+                    mock_run.side_effect = subprocess.CalledProcessError(
+                        returncode=1,
+                        cmd=[
+                            *expected_cmd,
+                            "-dM", "-E", "-x", "c", expected_input
+                        ]
+                    )
+                else:
+                    mock_process = MagicMock()
+                    mock_process.returncode = 0
+                    mock_process.stdout = TestGetFlavor.MockCommunicate(defines_stdout)
+                    mock_run.return_value = mock_process
                 with patch.dict(os.environ, env):
-                    defines = gyp.common.GetCrossCompilerPredefines()
+                    try:
+                        defines = gyp.common.GetCompilerPredefines()
+                    except Exception as e:
+                        self.fail(f"GetCompilerPredefines raised an exception: {e}")
                     flavor = gyp.common.GetFlavor({})
-                if env.get("CC_target"):
+                if env.get("CC_target") or env.get("CC"):
                     mock_run.assert_called_with(
                         [
                             *expected_cmd,
@@ -110,6 +123,9 @@ def mock_run(env, defines_stdout, expected_cmd):
                         capture_output=True, check=True)
                 return [defines, flavor]
 
+        [defines0, _] = mock_run({ "CC": "cl.exe" }, "", ["cl.exe"], True)
+        assert defines0 == {}
+
         [defines1, _] = mock_run({}, "", [])
         assert defines1 == {}
 
diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py
index 5ebe58bb55..cc02298e4e 100644
--- a/gyp/pylib/gyp/generator/android.py
+++ b/gyp/pylib/gyp/generator/android.py
@@ -900,8 +900,7 @@ def WriteTarget(
         if self.type != "none":
             self.WriteTargetFlags(spec, configs, link_deps)
 
-        settings = spec.get("aosp_build_settings", {})
-        if settings:
+        if settings := spec.get("aosp_build_settings", {}):
             self.WriteLn("### Set directly by aosp_build_settings.")
             for k, v in settings.items():
                 if isinstance(v, list):
diff --git a/gyp/pylib/gyp/generator/cmake.py b/gyp/pylib/gyp/generator/cmake.py
index e69103e1b9..db8cc6e026 100644
--- a/gyp/pylib/gyp/generator/cmake.py
+++ b/gyp/pylib/gyp/generator/cmake.py
@@ -810,8 +810,7 @@ def WriteTarget(
     # link directories to targets defined after it is called.
     # As a result, link_directories must come before the target definition.
     # CMake unfortunately has no means of removing entries from LINK_DIRECTORIES.
-    library_dirs = config.get("library_dirs")
-    if library_dirs is not None:
+    if (library_dirs := config.get("library_dirs")) is not None:
         output.write("link_directories(")
         for library_dir in library_dirs:
             output.write(" ")
@@ -1295,8 +1294,7 @@ def CallGenerateOutputForConfig(arglist):
 
 
 def GenerateOutput(target_list, target_dicts, data, params):
-    user_config = params.get("generator_flags", {}).get("config", None)
-    if user_config:
+    if user_config := params.get("generator_flags", {}).get("config", None):
         GenerateOutputForConfig(target_list, target_dicts, data, params, user_config)
     else:
         config_names = target_dicts[target_list[0]]["configurations"]
diff --git a/gyp/pylib/gyp/generator/eclipse.py b/gyp/pylib/gyp/generator/eclipse.py
index ed6daa91ba..be7ecd8b3b 100644
--- a/gyp/pylib/gyp/generator/eclipse.py
+++ b/gyp/pylib/gyp/generator/eclipse.py
@@ -451,8 +451,7 @@ def GenerateOutput(target_list, target_dicts, data, params):
     if params["options"].generator_output:
         raise NotImplementedError("--generator_output not implemented for eclipse")
 
-    user_config = params.get("generator_flags", {}).get("config", None)
-    if user_config:
+    if user_config := params.get("generator_flags", {}).get("config", None):
         GenerateOutputForConfig(target_list, target_dicts, data, params, user_config)
     else:
         config_names = target_dicts[target_list[0]]["configurations"]
diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py
index e860479069..7118492e77 100644
--- a/gyp/pylib/gyp/generator/make.py
+++ b/gyp/pylib/gyp/generator/make.py
@@ -78,7 +78,7 @@ def CalculateVariables(default_variables, params):
 
         # Copy additional generator configuration data from Xcode, which is shared
         # by the Mac Make generator.
-        import gyp.generator.xcode as xcode_generator
+        import gyp.generator.xcode as xcode_generator  # noqa: PLC0415
 
         global generator_additional_non_configuration_keys
         generator_additional_non_configuration_keys = getattr(
@@ -1465,8 +1465,7 @@ def WriteSources(
                 order_only=True,
             )
 
-        pchdeps = precompiled_header.GetObjDependencies(compilable, objs)
-        if pchdeps:
+        if pchdeps := precompiled_header.GetObjDependencies(compilable, objs):
             self.WriteLn("# Dependencies from obj files to their precompiled headers")
             for source, obj, gch in pchdeps:
                 self.WriteLn(f"{obj}: {gch}")
@@ -1600,8 +1599,7 @@ def ComputeOutputBasename(self, spec):
 
         target_prefix = spec.get("product_prefix", target_prefix)
         target = spec.get("product_name", target)
-        product_ext = spec.get("product_extension")
-        if product_ext:
+        if product_ext := spec.get("product_extension"):
             target_ext = "." + product_ext
 
         return target_prefix + target + target_ext
@@ -2383,7 +2381,7 @@ def WriteAutoRegenerationRule(params, root_makefile, makefile_name, build_files)
         % {
             "makefile_name": makefile_name,
             "deps": replace_sep(
-                " ".join(SourceifyAndQuoteSpaces(bf) for bf in build_files)
+                " ".join(sorted(SourceifyAndQuoteSpaces(bf) for bf in build_files))
             ),
             "cmd": replace_sep(gyp.common.EncodePOSIXShellList(
                 [gyp_binary, "-fmake"] + gyp.RegenerateFlags(options) + build_files_args
diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py
index b4aea2e69a..baf5042150 100644
--- a/gyp/pylib/gyp/generator/msvs.py
+++ b/gyp/pylib/gyp/generator/msvs.py
@@ -1364,8 +1364,7 @@ def _GetOutputTargetExt(spec):
   Returns:
     A string with the extension, or None
   """
-    target_extension = spec.get("product_extension")
-    if target_extension:
+    if target_extension := spec.get("product_extension"):
         return "." + target_extension
     return None
 
@@ -3166,8 +3165,7 @@ def _GetMSBuildAttributes(spec, config, build_file):
         "windows_driver": "Link",
         "static_library": "Lib",
     }
-    msbuild_tool = msbuild_tool_map.get(spec["type"])
-    if msbuild_tool:
+    if msbuild_tool := msbuild_tool_map.get(spec["type"]):
         msbuild_settings = config["finalized_msbuild_settings"]
         out_file = msbuild_settings[msbuild_tool].get("OutputFile")
         if out_file:
@@ -3184,8 +3182,7 @@ def _GetMSBuildConfigurationGlobalProperties(spec, configurations, build_file):
     # there are actions.
     # TODO(jeanluc) Handle the equivalent of setting 'CYGWIN=nontsec'.
     new_paths = []
-    cygwin_dirs = spec.get("msvs_cygwin_dirs", ["."])[0]
-    if cygwin_dirs:
+    if cygwin_dirs := spec.get("msvs_cygwin_dirs", ["."])[0]:
         cyg_path = "$(MSBuildProjectDirectory)\\%s\\bin\\" % _FixPath(cygwin_dirs)
         new_paths.append(cyg_path)
         # TODO(jeanluc) Change the convention to have both a cygwin_dir and a
@@ -3370,7 +3367,6 @@ def _FinalizeMSBuildSettings(spec, configuration):
     prebuild = configuration.get("msvs_prebuild")
     postbuild = configuration.get("msvs_postbuild")
     def_file = _GetModuleDefinition(spec)
-    precompiled_header = configuration.get("msvs_precompiled_header")
 
     # Add the information to the appropriate tool
     # TODO(jeanluc) We could optimize and generate these settings only if
@@ -3408,7 +3404,7 @@ def _FinalizeMSBuildSettings(spec, configuration):
         msbuild_settings, "ClCompile", "DisableSpecificWarnings", disabled_warnings
     )
     # Turn on precompiled headers if appropriate.
-    if precompiled_header:
+    if precompiled_header := configuration.get("msvs_precompiled_header"):
         # While MSVC works with just file name eg. "v8_pch.h", ClangCL requires
         # the full path eg. "tools/msvs/pch/v8_pch.h" to find the file.
         # P.S. Only ClangCL defines msbuild_toolset, for MSVC it is None.
diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py
index b7ac823d14..d5dfa1a118 100644
--- a/gyp/pylib/gyp/generator/ninja.py
+++ b/gyp/pylib/gyp/generator/ninja.py
@@ -5,6 +5,7 @@
 
 import collections
 import copy
+import ctypes
 import hashlib
 import json
 import multiprocessing
@@ -263,8 +264,7 @@ def ExpandSpecial(self, path, product_dir=None):
         dir.
         """
 
-        PRODUCT_DIR = "$!PRODUCT_DIR"
-        if PRODUCT_DIR in path:
+        if (PRODUCT_DIR := "$!PRODUCT_DIR") in path:
             if product_dir:
                 path = path.replace(PRODUCT_DIR, product_dir)
             else:
@@ -272,8 +272,7 @@ def ExpandSpecial(self, path, product_dir=None):
                 path = path.replace(PRODUCT_DIR + "\\", "")
                 path = path.replace(PRODUCT_DIR, ".")
 
-        INTERMEDIATE_DIR = "$!INTERMEDIATE_DIR"
-        if INTERMEDIATE_DIR in path:
+        if (INTERMEDIATE_DIR := "$!INTERMEDIATE_DIR") in path:
             int_dir = self.GypPathToUniqueOutput("gen")
             # GypPathToUniqueOutput generates a path relative to the product dir,
             # so insert product_dir in front if it is provided.
@@ -1995,7 +1994,7 @@ def CalculateVariables(default_variables, params):
 
         # Copy additional generator configuration data from Xcode, which is shared
         # by the Mac Ninja generator.
-        import gyp.generator.xcode as xcode_generator
+        import gyp.generator.xcode as xcode_generator  # noqa: PLC0415
 
         generator_additional_non_configuration_keys = getattr(
             xcode_generator, "generator_additional_non_configuration_keys", []
@@ -2018,7 +2017,7 @@ def CalculateVariables(default_variables, params):
 
         # Copy additional generator configuration data from VS, which is shared
         # by the Windows Ninja generator.
-        import gyp.generator.msvs as msvs_generator
+        import gyp.generator.msvs as msvs_generator  # noqa: PLC0415
 
         generator_additional_non_configuration_keys = getattr(
             msvs_generator, "generator_additional_non_configuration_keys", []
@@ -2075,21 +2074,17 @@ def OpenOutput(path, mode="w"):
 
 
 def CommandWithWrapper(cmd, wrappers, prog):
-    wrapper = wrappers.get(cmd, "")
-    if wrapper:
+    if wrapper := wrappers.get(cmd, ""):
         return wrapper + " " + prog
     return prog
 
 
 def GetDefaultConcurrentLinks():
     """Returns a best-guess for a number of concurrent links."""
-    pool_size = int(os.environ.get("GYP_LINK_CONCURRENCY") or 0)
-    if pool_size:
+    if pool_size := int(os.environ.get("GYP_LINK_CONCURRENCY") or 0):
         return pool_size
 
     if sys.platform in ("win32", "cygwin"):
-        import ctypes
-
         class MEMORYSTATUSEX(ctypes.Structure):
             _fields_ = [
                 ("dwLength", ctypes.c_ulong),
@@ -2305,8 +2300,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name
             key_prefix = re.sub(r"\.HOST$", ".host", key_prefix)
             wrappers[key_prefix] = os.path.join(build_to_root, value)
 
-    mac_toolchain_dir = generator_flags.get("mac_toolchain_dir", None)
-    if mac_toolchain_dir:
+    if mac_toolchain_dir := generator_flags.get("mac_toolchain_dir", None):
         wrappers["LINK"] = "export DEVELOPER_DIR='%s' &&" % mac_toolchain_dir
 
     if flavor == "win":
diff --git a/gyp/pylib/gyp/mac_tool.py b/gyp/pylib/gyp/mac_tool.py
index 70aab4f178..58fb9c7398 100755
--- a/gyp/pylib/gyp/mac_tool.py
+++ b/gyp/pylib/gyp/mac_tool.py
@@ -25,8 +25,7 @@
 
 def main(args):
     executor = MacTool()
-    exit_code = executor.Dispatch(args)
-    if exit_code is not None:
+    if (exit_code := executor.Dispatch(args)) is not None:
         sys.exit(exit_code)
 
 
@@ -142,7 +141,7 @@ def _CopyStringsFile(self, source, dest):
         #     CFPropertyListCreateFromXMLData(): Old-style plist parser: missing
         #     semicolon in dictionary.
         # on invalid files. Do the same kind of validation.
-        import CoreFoundation
+        import CoreFoundation  # noqa: PLC0415
 
         with open(source, "rb") as in_file:
             s = in_file.read()
diff --git a/gyp/pylib/gyp/msvs_emulation.py b/gyp/pylib/gyp/msvs_emulation.py
index ace0cae5eb..f8b3b87d94 100644
--- a/gyp/pylib/gyp/msvs_emulation.py
+++ b/gyp/pylib/gyp/msvs_emulation.py
@@ -247,9 +247,7 @@ def GetExtension(self):
         the target type.
         """
         ext = self.spec.get("product_extension", None)
-        if ext:
-            return ext
-        return gyp.MSVSUtil.TARGET_TYPE_EXT.get(self.spec["type"], "")
+        return ext or gyp.MSVSUtil.TARGET_TYPE_EXT.get(self.spec["type"], "")
 
     def GetVSMacroEnv(self, base_to_build=None, config=None):
         """Get a dict of variables mapping internal VS macro names to their gyp
@@ -625,8 +623,7 @@ def GetDefFile(self, gyp_to_build_path):
     def _GetDefFileAsLdflags(self, ldflags, gyp_to_build_path):
         """.def files get implicitly converted to a ModuleDefinitionFile for the
         linker in the VS generator. Emulate that behaviour here."""
-        def_file = self.GetDefFile(gyp_to_build_path)
-        if def_file:
+        if def_file := self.GetDefFile(gyp_to_build_path):
             ldflags.append('/DEF:"%s"' % def_file)
 
     def GetPGDName(self, config, expand_special):
@@ -674,14 +671,11 @@ def GetLdflags(
         )
         ld("DelayLoadDLLs", prefix="/DELAYLOAD:")
         ld("TreatLinkerWarningAsErrors", prefix="/WX", map={"true": "", "false": ":NO"})
-        out = self.GetOutputName(config, expand_special)
-        if out:
+        if out := self.GetOutputName(config, expand_special):
             ldflags.append("/OUT:" + out)
-        pdb = self.GetPDBName(config, expand_special, output_name + ".pdb")
-        if pdb:
+        if pdb := self.GetPDBName(config, expand_special, output_name + ".pdb"):
             ldflags.append("/PDB:" + pdb)
-        pgd = self.GetPGDName(config, expand_special)
-        if pgd:
+        if pgd := self.GetPGDName(config, expand_special):
             ldflags.append("/PGD:" + pgd)
         map_file = self.GetMapFileName(config, expand_special)
         ld("GenerateMapFile", map={"true": "/MAP:" + map_file if map_file else "/MAP"})
diff --git a/gyp/pylib/gyp/win_tool.py b/gyp/pylib/gyp/win_tool.py
index 7e647f40a8..3004f533ca 100755
--- a/gyp/pylib/gyp/win_tool.py
+++ b/gyp/pylib/gyp/win_tool.py
@@ -27,8 +27,7 @@
 
 def main(args):
     executor = WinTool()
-    exit_code = executor.Dispatch(args)
-    if exit_code is not None:
+    if (exit_code := executor.Dispatch(args)) is not None:
         sys.exit(exit_code)
 
 
diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py
index 85a63dfd7a..0746865dc8 100644
--- a/gyp/pylib/gyp/xcode_emulation.py
+++ b/gyp/pylib/gyp/xcode_emulation.py
@@ -1350,8 +1350,7 @@ def _DefaultSdkRoot(self):
         if xcode_version < "0500":
             return ""
         default_sdk_path = self._XcodeSdkPath("")
-        default_sdk_root = XcodeSettings._sdk_root_cache.get(default_sdk_path)
-        if default_sdk_root:
+        if default_sdk_root := XcodeSettings._sdk_root_cache.get(default_sdk_path):
             return default_sdk_root
         try:
             all_sdks = GetStdout(["xcodebuild", "-showsdks"])
@@ -1787,11 +1786,9 @@ def _GetXcodeEnv(
         env["INFOPLIST_PATH"] = xcode_settings.GetBundlePlistPath()
         env["WRAPPER_NAME"] = xcode_settings.GetWrapperName()
 
-    install_name = xcode_settings.GetInstallName()
-    if install_name:
+    if install_name := xcode_settings.GetInstallName():
         env["LD_DYLIB_INSTALL_NAME"] = install_name
-    install_name_base = xcode_settings.GetInstallNameBase()
-    if install_name_base:
+    if install_name_base := xcode_settings.GetInstallNameBase():
         env["DYLIB_INSTALL_NAME_BASE"] = install_name_base
     xcode_version, _ = XcodeVersion()
     if xcode_version >= "0500" and not env.get("SDKROOT"):
diff --git a/gyp/pylib/gyp/xcode_ninja.py b/gyp/pylib/gyp/xcode_ninja.py
index cac1af56f7..ae3079d85a 100644
--- a/gyp/pylib/gyp/xcode_ninja.py
+++ b/gyp/pylib/gyp/xcode_ninja.py
@@ -70,12 +70,11 @@ def _TargetFromSpec(old_spec, params):
 
     target_name = old_spec.get("target_name")
     product_name = old_spec.get("product_name", target_name)
-    product_extension = old_spec.get("product_extension")
 
     ninja_target = {}
     ninja_target["target_name"] = target_name
     ninja_target["product_name"] = product_name
-    if product_extension:
+    if product_extension := old_spec.get("product_extension"):
         ninja_target["product_extension"] = product_extension
     ninja_target["toolset"] = old_spec.get("toolset")
     ninja_target["default_configuration"] = old_spec.get("default_configuration")
diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py
index be17ef946d..0376693d95 100644
--- a/gyp/pylib/gyp/xcodeproj_file.py
+++ b/gyp/pylib/gyp/xcodeproj_file.py
@@ -183,8 +183,7 @@ def SourceTreeAndPathFromPath(input_path):
     'path'         (None, 'path')
   """
 
-    source_group_match = _path_leading_variable.match(input_path)
-    if source_group_match:
+    if source_group_match := _path_leading_variable.match(input_path):
         source_tree = source_group_match.group(1)
         output_path = source_group_match.group(3)  # This may be None.
     else:
@@ -390,8 +389,7 @@ def Comment(self):
     def Hashables(self):
         hashables = [self.__class__.__name__]
 
-        name = self.Name()
-        if name is not None:
+        if (name := self.Name()) is not None:
             hashables.append(name)
 
         hashables.extend(self._hashables)
@@ -1051,8 +1049,7 @@ def Hashables(self):
         # including paths with a sourceTree, they'll still inherit their parents'
         # hashables, even though the paths aren't relative to their parents.  This
         # is not expected to be much of a problem in practice.
-        path = self.PathFromSourceTreeAndPath()
-        if path is not None:
+        if (path := self.PathFromSourceTreeAndPath()) is not None:
             components = path.split(posixpath.sep)
             for component in components:
                 hashables.append(self.__class__.__name__ + ".path")
@@ -2109,8 +2106,7 @@ def SetDestination(self, path):
     specifically, "$(DIR)/path".
     """
 
-        path_tree_match = self.path_tree_re.search(path)
-        if path_tree_match:
+        if path_tree_match := self.path_tree_re.search(path):
             path_tree = path_tree_match.group(1)
             if path_tree in self.path_tree_first_to_subfolder:
                 subfolder = self.path_tree_first_to_subfolder[path_tree]
diff --git a/gyp/pylib/packaging/_elffile.py b/gyp/pylib/packaging/_elffile.py
index 6fb19b30bb..cb33e10556 100644
--- a/gyp/pylib/packaging/_elffile.py
+++ b/gyp/pylib/packaging/_elffile.py
@@ -48,8 +48,7 @@ def __init__(self, f: IO[bytes]) -> None:
             ident = self._read("16B")
         except struct.error:
             raise ELFInvalid("unable to parse identification")
-        magic = bytes(ident[:4])
-        if magic != b"\x7fELF":
+        if (magic := bytes(ident[:4])) != b"\x7fELF":
             raise ELFInvalid(f"invalid magic: {magic!r}")
 
         self.capacity = ident[4]  # Format for program header (bitness).
diff --git a/gyp/pylib/packaging/markers.py b/gyp/pylib/packaging/markers.py
index 8b98fca723..7e4d150208 100644
--- a/gyp/pylib/packaging/markers.py
+++ b/gyp/pylib/packaging/markers.py
@@ -166,8 +166,7 @@ def _evaluate_markers(markers: MarkerList, environment: Dict[str, str]) -> bool:
 
 def format_full_version(info: "sys._version_info") -> str:
     version = "{0.major}.{0.minor}.{0.micro}".format(info)
-    kind = info.releaselevel
-    if kind != "final":
+    if (kind := info.releaselevel) != "final":
         version += kind[0] + str(info.serial)
     return version
 
diff --git a/gyp/pylib/packaging/metadata.py b/gyp/pylib/packaging/metadata.py
index 23bb564f3d..43f5c5b30d 100644
--- a/gyp/pylib/packaging/metadata.py
+++ b/gyp/pylib/packaging/metadata.py
@@ -591,8 +591,7 @@ def _process_description_content_type(self, value: str) -> str:
                 f"{{field}} must be one of {list(content_types)}, not {value!r}"
             )
 
-        charset = parameters.get("charset", "UTF-8")
-        if charset != "UTF-8":
+        if (charset := parameters.get("charset", "UTF-8")) != "UTF-8":
             raise self._invalid_metadata(
                 f"{{field}} can only specify the UTF-8 charset, not {list(charset)}"
             )
diff --git a/gyp/pyproject.toml b/gyp/pyproject.toml
index 537308731f..62fb2bf8ca 100644
--- a/gyp/pyproject.toml
+++ b/gyp/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "gyp-next"
-version = "0.20.0"
+version = "0.20.2"
 authors = [
   { name="Node.js contributors", email="ryzokuken@disroot.org" },
 ]
@@ -39,7 +39,6 @@ gyp = "gyp:script_main"
 [tool.ruff]
 extend-exclude = ["pylib/packaging"]
 line-length = 88
-target-version = "py37"
 
 [tool.ruff.lint]
 select = [
diff --git a/gyp/test_gyp.py b/gyp/test_gyp.py
index b7bb956b8e..8e910a2b76 100755
--- a/gyp/test_gyp.py
+++ b/gyp/test_gyp.py
@@ -148,13 +148,13 @@ def print_configuration_info():
     print("Test configuration:")
     if sys.platform == "darwin":
         sys.path.append(os.path.abspath("test/lib"))
-        import TestMac
+        import TestMac  # noqa: PLC0415
 
         print(f"  Mac {platform.mac_ver()[0]} {platform.mac_ver()[2]}")
         print(f"  Xcode {TestMac.Xcode.Version()}")
     elif sys.platform == "win32":
         sys.path.append(os.path.abspath("pylib"))
-        import gyp.MSVSVersion
+        import gyp.MSVSVersion  # noqa: PLC0415
 
         print("  Win %s %s\n" % platform.win32_ver()[0:2])
         print("  MSVS %s" % gyp.MSVSVersion.SelectVisualStudioVersion().Description())

From 3df8789a9aa73c60707eec8f02f4e926491d6102 Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Thu, 24 Jul 2025 16:38:11 -0400
Subject: [PATCH 472/551] chore: Windows 2019 has been removed from GitHub
 Actions (#3190)

* actions/runner-images#12045
---
 .github/workflows/visual-studio.yml | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/.github/workflows/visual-studio.yml b/.github/workflows/visual-studio.yml
index e36639d7eb..b8a3baa84b 100644
--- a/.github/workflows/visual-studio.yml
+++ b/.github/workflows/visual-studio.yml
@@ -17,8 +17,6 @@ jobs:
       max-parallel: 8
       matrix:
         include:
-          - os: windows-2019
-            msvs-version: 2019
           - os: windows-2022
             msvs-version: 2022
           - os: windows-2025
@@ -27,10 +25,10 @@ jobs:
     steps:
       - name: Checkout Repository
         uses: actions/checkout@v4
-      - name: Use Python 3.12
+      - name: Use Python 3
         uses: actions/setup-python@v5
         with:
-          python-version: "3.12"
+          python-version: "3.x"
       - name: Install Dependencies
         run: npm install
       - name: Run Node tests

From b81a665acfb9d88102e8044a8ec8ca74a3e9eccc Mon Sep 17 00:00:00 2001
From: Aman Karmani 
Date: Tue, 29 Jul 2025 16:50:08 -0700
Subject: [PATCH 473/551] fix: Normalize win32 library names (#3189)

---
 addon.gypi | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/addon.gypi b/addon.gypi
index b4ac369acb..d8d188e2f8 100644
--- a/addon.gypi
+++ b/addon.gypi
@@ -179,7 +179,7 @@
           '-loleaut32.lib',
           '-luuid.lib',
           '-lodbc32.lib',
-          '-lDelayImp.lib',
+          '-ldelayimp.lib',
           '-l"<(node_lib_file)"'
         ],
         'msvs_disabled_warnings': [

From b41864f7c1c60e4a160c1b4dd91558dcaa3f74e4 Mon Sep 17 00:00:00 2001
From: Aman Karmani 
Date: Tue, 29 Jul 2025 16:58:15 -0700
Subject: [PATCH 474/551] fix: use temp dir for tar extraction on all platforms
 (#3170)

fixes #3165
---
 lib/install.js | 20 ++++++++------------
 1 file changed, 8 insertions(+), 12 deletions(-)

diff --git a/lib/install.js b/lib/install.js
index 90be86c822..ee4adb1e67 100644
--- a/lib/install.js
+++ b/lib/install.js
@@ -200,10 +200,10 @@ async function install (gyp, argv) {
     // download the tarball and extract!
     // Ommited on Windows if only new node.lib is required
 
-    // on Windows there can be file errors from tar if parallel installs
+    // there can be file errors from tar if parallel installs
     // are happening (not uncommon with multiple native modules) so
     // extract the tarball to a temp directory first and then copy over
-    const tarExtractDir = win ? await fs.mkdtemp(path.join(os.tmpdir(), 'node-gyp-tmp-')) : devDir
+    const tarExtractDir = await fs.mkdtemp(path.join(os.tmpdir(), 'node-gyp-tmp-'))
 
     try {
       if (shouldDownloadTarball) {
@@ -277,17 +277,13 @@ async function install (gyp, argv) {
       }
 
       // copy over the files from the temp tarball extract directory to devDir
-      if (tarExtractDir !== devDir) {
-        await copyDirectory(tarExtractDir, devDir)
-      }
+      await copyDirectory(tarExtractDir, devDir)
     } finally {
-      if (tarExtractDir !== devDir) {
-        try {
-          // try to cleanup temp dir
-          await fs.rm(tarExtractDir, { recursive: true, maxRetries: 3 })
-        } catch {
-          log.warn('failed to clean up temp tarball extract directory')
-        }
+      try {
+        // try to cleanup temp dir
+        await fs.rm(tarExtractDir, { recursive: true, maxRetries: 3 })
+      } catch {
+        log.warn('failed to clean up temp tarball extract directory')
       }
     }
 

From 02f747f13b8fea73416ca8e302801f4a3f20a5f6 Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Tue, 29 Jul 2025 19:58:38 -0400
Subject: [PATCH 475/551] chore(main): release 11.3.0

---
 .release-please-manifest.json |  2 +-
 CHANGELOG.md                  | 20 ++++++++++++++++++++
 package.json                  |  2 +-
 3 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
index f098464b1f..200e9cc761 100644
--- a/.release-please-manifest.json
+++ b/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "11.2.0"
+    ".": "11.3.0"
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e206e5d9f3..5030eaa973 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,25 @@
 # Changelog
 
+## [11.3.0](https://github.com/nodejs/node-gyp/compare/v11.2.0...v11.3.0) (2025-07-29)
+
+
+### Features
+
+* update gyp-next to v0.20.2 ([#3169](https://github.com/nodejs/node-gyp/issues/3169)) ([0e65632](https://github.com/nodejs/node-gyp/commit/0e656322c1e94041331ab3b01bf66c2ef9bd6ead))
+
+
+### Bug Fixes
+
+* Correct Visual Studio 2019 test version ([#3153](https://github.com/nodejs/node-gyp/issues/3153)) ([7d883b5](https://github.com/nodejs/node-gyp/commit/7d883b5cf4c26e76065201f85b0be36d5ebdcc0e))
+* Normalize win32 library names ([#3189](https://github.com/nodejs/node-gyp/issues/3189)) ([b81a665](https://github.com/nodejs/node-gyp/commit/b81a665acfb9d88102e8044a8ec8ca74a3e9eccc))
+* use temp dir for tar extraction on all platforms ([#3170](https://github.com/nodejs/node-gyp/issues/3170)) ([b41864f](https://github.com/nodejs/node-gyp/commit/b41864f7c1c60e4a160c1b4dd91558dcaa3f74e4)), closes [#3165](https://github.com/nodejs/node-gyp/issues/3165)
+
+
+### Miscellaneous
+
+* retry wasi-sdk download in CI ([#3151](https://github.com/nodejs/node-gyp/issues/3151)) ([8f3cd8b](https://github.com/nodejs/node-gyp/commit/8f3cd8b3a157bccd8d7110e7d46a27c2926625cd))
+* Windows 2019 has been removed from GitHub Actions ([#3190](https://github.com/nodejs/node-gyp/issues/3190)) ([3df8789](https://github.com/nodejs/node-gyp/commit/3df8789a9aa73c60707eec8f02f4e926491d6102))
+
 ## [11.2.0](https://github.com/nodejs/node-gyp/compare/v11.1.0...v11.2.0) (2025-04-01)
 
 
diff --git a/package.json b/package.json
index f69a022ef3..eecdf7b5ec 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "11.2.0",
+  "version": "11.3.0",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From 27f5505ec236551081366bf8a9c13ef5d8e468bf Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Wed, 13 Aug 2025 07:03:45 +0200
Subject: [PATCH 476/551] build(deps): bump actions/checkout from 4 to 5
 (#3193)

Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 5.
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 .github/workflows/release-please.yml  |  2 +-
 .github/workflows/tests.yml           | 10 +++++-----
 .github/workflows/update-gyp-next.yml |  2 +-
 .github/workflows/visual-studio.yml   |  2 +-
 4 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml
index 5c6a1ec615..f7c4889193 100644
--- a/.github/workflows/release-please.yml
+++ b/.github/workflows/release-please.yml
@@ -32,7 +32,7 @@ jobs:
       contents: read
       id-token: write # to generate npm provenance statements
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v5
       - uses: actions/setup-node@v4
         with:
           node-version: lts/*
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 6bf975cf9f..8f16c497ee 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -17,7 +17,7 @@ jobs:
     name: Lint Python
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v5
     - uses: astral-sh/ruff-action@v3
       with:
         args: "check --select=E,F,PLC,PLE,UP,W,YTT --ignore=E721,PLC0206,PLC1901,S101,UP031 --target-version=py39"
@@ -27,7 +27,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Checkout Repository
-      uses: actions/checkout@v4
+      uses: actions/checkout@v5
     - name: Use Node.js 22.x
       uses: actions/setup-node@v4
       with:
@@ -42,7 +42,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Checkout Repository
-      uses: actions/checkout@v4
+      uses: actions/checkout@v5
     - name: Use Node.js 22.x
       uses: actions/setup-node@v4
       with:
@@ -60,7 +60,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Checkout Repository
-      uses: actions/checkout@v4
+      uses: actions/checkout@v5
     - name: Use Node.js 22.x
       uses: actions/setup-node@v4
       with:
@@ -113,7 +113,7 @@ jobs:
       WASI_SDK_PATH: 'wasi-sdk-25.0'
     steps:
       - name: Checkout Repository
-        uses: actions/checkout@v4
+        uses: actions/checkout@v5
       - name: Use Node.js ${{ matrix.node }}
         uses: actions/setup-node@v4
         with:
diff --git a/.github/workflows/update-gyp-next.yml b/.github/workflows/update-gyp-next.yml
index f43be6c70c..94d36c3afd 100644
--- a/.github/workflows/update-gyp-next.yml
+++ b/.github/workflows/update-gyp-next.yml
@@ -17,7 +17,7 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v5
         with:
           persist-credentials: false
 
diff --git a/.github/workflows/visual-studio.yml b/.github/workflows/visual-studio.yml
index b8a3baa84b..a284e28768 100644
--- a/.github/workflows/visual-studio.yml
+++ b/.github/workflows/visual-studio.yml
@@ -24,7 +24,7 @@ jobs:
     runs-on: ${{ matrix.os }}
     steps:
       - name: Checkout Repository
-        uses: actions/checkout@v4
+        uses: actions/checkout@v5
       - name: Use Python 3
         uses: actions/setup-python@v5
         with:

From 71a910e61e21927b3119c218019cee52ea784476 Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Wed, 13 Aug 2025 10:50:57 +0200
Subject: [PATCH 477/551] Lint Python: Ignore ruff rule PLC0415 (#3195)

https://docs.astral.sh/ruff/rules/import-outside-top-level

There are several valid reasons for conditionally importing Python modules, especially for OS-specific code, so let's intentionally ignore this rule.

% `ruff check --select=PLC0415`
```
Error: gyp/pylib/gyp/MSVSVersion.py:222:5: PLC0415 `import` should be at the top-level of a file
Error: gyp/pylib/gyp/__init__.py:492:9: PLC0415 `import` should be at the top-level of a file
Error: gyp/pylib/gyp/generator/make.py:81:9: PLC0415 `import` should be at the top-level of a file
Error: gyp/pylib/gyp/generator/ninja.py:1998:9: PLC0415 `import` should be at the top-level of a file
Error: gyp/pylib/gyp/generator/ninja.py:2021:9: PLC0415 `import` should be at the top-level of a file
Error: gyp/pylib/gyp/generator/ninja.py:2091:9: PLC0415 `import` should be at the top-level of a file
Error: gyp/pylib/gyp/mac_tool.py:145:9: PLC0415 `import` should be at the top-level of a file
Error: gyp/test_gyp.py:151:9: PLC0415 `import` should be at the top-level of a file
Error: gyp/test_gyp.py:157:9: PLC0415 `import` should be at the top-level of a file
```
---
 .github/workflows/tests.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 8f16c497ee..d8720d768a 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -20,7 +20,7 @@ jobs:
     - uses: actions/checkout@v5
     - uses: astral-sh/ruff-action@v3
       with:
-        args: "check --select=E,F,PLC,PLE,UP,W,YTT --ignore=E721,PLC0206,PLC1901,S101,UP031 --target-version=py39"
+        args: "check --select=E,F,PLC,PLE,UP,W,YTT --ignore=E721,PLC0206,PLC0415,PLC1901,S101,UP031 --target-version=py39"
 
   lint-js:
     name: Lint JS

From 077361502933fcb994ca365c3c07c03177503df2 Mon Sep 17 00:00:00 2001
From: Chengzhong Wu 
Date: Mon, 18 Aug 2025 20:38:40 +0100
Subject: [PATCH 478/551] chore: use npm oicd connection for publishing (#3197)

---
 .github/workflows/release-please.yml | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml
index f7c4889193..e4854c8151 100644
--- a/.github/workflows/release-please.yml
+++ b/.github/workflows/release-please.yml
@@ -5,6 +5,10 @@ on:
     branches:
       - main
 
+permissions:
+  id-token: write  # Required for OIDC
+  contents: read
+
 jobs:
   release-please:
     outputs:
@@ -38,5 +42,3 @@ jobs:
           node-version: lts/*
           registry-url: 'https://registry.npmjs.org'
       - run: npm publish --provenance --access public
-        env:
-          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

From 1822dff4f616a30ac3ca72e5946d81389cb8557e Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Tue, 19 Aug 2025 10:52:05 -0700
Subject: [PATCH 479/551] feat: support reading config from package.json
 (#3196)

`npm@11` has deprecated setting arbitrary config values (https://github.com/npm/cli/issues/8153), which was the previously recommended place for users to set node-gyp related config. The new recommendation from npm is to use the `config` object in package.json for this (https://github.com/npm/cli/issues/8153#issuecomment-2718937461). I think it makes sense to follow npm's recommendation here.

When a user sets a value in `package.json#config` it will be set in the environment by npm with the prefix `npm_package_config_`.

I think it makes sense to allow prescribe that they keys be prefixed with `node_gyp_`. So as an example:

**package.json**
```json
{
  "config": {
    "node_gyp_devdir": "/tmp/.gyp"
  }
}
```

Then in `node-gyp` we can read the env var `npm_package_config_node_gyp_devdir`. If a user wants to set a global value, they can set that env var on their system.

Fixes #3156
---
 README.md            | 25 +++++++++++++++++++++---
 lib/node-gyp.js      | 46 +++++++++++++++++++++++++++-----------------
 test/test-options.js | 29 ++++++++++++++++++++++------
 3 files changed, 73 insertions(+), 27 deletions(-)

diff --git a/README.md b/README.md
index 474c59b458..72833b1363 100644
--- a/README.md
+++ b/README.md
@@ -235,9 +235,24 @@ Some additional resources for Node.js native addons and writing `gyp` configurat
 
 ## Configuration
 
+### package.json
+
+Use the `config` object in your package.json with each key in the form `node_gyp_OPTION_NAME`. Any of the command
+options listed above can be set (dashes in option names should be replaced by underscores).
+
+For example, to set `devdir` equal to `/tmp/.gyp`, your package.json would contain this:
+
+```json
+{
+  "config": {
+    "node_gyp_devdir": "/tmp/.gyp"
+  }
+}
+```
+
 ### Environment variables
 
-Use the form `npm_config_OPTION_NAME` for any of the command options listed
+Use the form `npm_package_config_node_gyp_OPTION_NAME` for any of the command options listed
 above (dashes in option names should be replaced by underscores).
 
 For example, to set `devdir` equal to `/tmp/.gyp`, you would:
@@ -245,15 +260,19 @@ For example, to set `devdir` equal to `/tmp/.gyp`, you would:
 Run this on Unix:
 
 ```bash
-export npm_config_devdir=/tmp/.gyp
+export npm_package_config_node_gyp_devdir=/tmp/.gyp
 ```
 
 Or this on Windows:
 
 ```console
-set npm_config_devdir=c:\temp\.gyp
+set npm_package_config_node_gyp_devdir=c:\temp\.gyp
 ```
 
+Note that in versions of npm before v11 it was possible to use the prefix `npm_config_` for
+environement variables. This was deprecated in npm@11 and will be removed in npm@12 so it
+is recommened to convert your environment variables to the above format.
+
 ### `npm` configuration for npm versions before v9
 
 Use the form `OPTION_NAME` for any of the command options listed above.
diff --git a/lib/node-gyp.js b/lib/node-gyp.js
index 5e25bf996f..26fcb2ee49 100644
--- a/lib/node-gyp.js
+++ b/lib/node-gyp.js
@@ -122,31 +122,41 @@ class Gyp extends EventEmitter {
     }
 
     // support for inheriting config env variables from npm
+    // npm will set environment variables in the following forms:
+    // - `npm_config_` for values from npm's own config. Setting arbitrary
+    //   options on npm's config was deprecated in npm v11 but node-gyp still
+    //   supports it for backwards compatibility.
+    //   See https://github.com/nodejs/node-gyp/issues/3156
+    // - `npm_package_config_node_gyp_` for values from the `config` object
+    //   in package.json. This is the preferred way to set options for node-gyp
+    //   since npm v11. The `node_gyp_` prefix is used to avoid conflicts with
+    //   other tools.
+    // The `npm_package_config_node_gyp_` prefix will take precedence over
+    // `npm_config_` keys.
     const npmConfigPrefix = 'npm_config_'
-    Object.keys(process.env).forEach((name) => {
-      if (name.indexOf(npmConfigPrefix) !== 0) {
-        return
-      }
-      const val = process.env[name]
-      if (name === npmConfigPrefix + 'loglevel') {
-        log.logger.level = val
-      } else {
+    const npmPackageConfigPrefix = 'npm_package_config_node_gyp_'
+
+    const configEnvKeys = Object.keys(process.env)
+      .filter((k) => k.startsWith(npmConfigPrefix) || k.startsWith(npmPackageConfigPrefix))
+      // sort so that npm_package_config_node_gyp_ keys come last and will override
+      .sort((a) => a.startsWith(npmConfigPrefix) ? -1 : 1)
+
+    for (const key of configEnvKeys) {
       // add the user-defined options to the config
-        name = name.substring(npmConfigPrefix.length)
-        // gyp@741b7f1 enters an infinite loop when it encounters
-        // zero-length options so ensure those don't get through.
-        if (name) {
+      const name = key.startsWith(npmConfigPrefix)
+        ? key.substring(npmConfigPrefix.length)
+        : key.substring(npmPackageConfigPrefix.length)
+      // gyp@741b7f1 enters an infinite loop when it encounters
+      // zero-length options so ensure those don't get through.
+      if (name) {
         // convert names like force_process_config to force-process-config
-          if (name.includes('_')) {
-            name = name.replace(/_/g, '-')
-          }
-          this.opts[name] = val
-        }
+        this.opts[name.replaceAll('_', '-')] = process.env[key]
       }
-    })
+    }
 
     if (this.opts.loglevel) {
       log.logger.level = this.opts.loglevel
+      delete this.opts.loglevel
     }
     log.resume()
   }
diff --git a/test/test-options.js b/test/test-options.js
index 8d281db8e8..8ea616fd1a 100644
--- a/test/test-options.js
+++ b/test/test-options.js
@@ -3,31 +3,48 @@
 const { describe, it } = require('mocha')
 const assert = require('assert')
 const gyp = require('../lib/node-gyp')
+const log = require('../lib/log')
 
 describe('options', function () {
   it('options in environment', () => {
     // `npm test` dumps a ton of npm_config_* variables in the environment.
     Object.keys(process.env)
-      .filter((key) => /^npm_config_/.test(key))
+      .filter((key) => /^npm_config_/i.test(key) || /^npm_package_config_node_gyp_/i.test(key))
       .forEach((key) => { delete process.env[key] })
 
     // in some platforms, certain keys are stubborn and cannot be removed
     const keys = Object.keys(process.env)
-      .filter((key) => /^npm_config_/.test(key))
+      .filter((key) => /^npm_config_/i.test(key) || /^npm_package_config_node_gyp_/i.test(key))
       .map((key) => key.substring('npm_config_'.length))
-      .concat('argv', 'x')
+
+    // Environment variables with the following prefixes should be added to opts.
+    // - `npm_config_` for npm versions before v11.
+    // - `npm_package_config_node_gyp_` for npm versions 11 and later.
 
     // Zero-length keys should get filtered out.
     process.env.npm_config_ = '42'
+    process.env.npm_package_config_node_gyp_ = '42'
     // Other keys should get added.
+    process.env.npm_package_config_node_gyp_foo = '42'
     process.env.npm_config_x = '42'
-    // Except loglevel.
-    process.env.npm_config_loglevel = 'debug'
+    process.env.npm_config_y = '41'
+    // Package config should take precedence over npm_config_ keys.
+    process.env.npm_package_config_node_gyp_y = '42'
+    // loglevel does not get added to opts but will change the logger's level.
+    process.env.npm_config_loglevel = 'silly'
 
     const g = gyp()
+
+    assert.strictEqual(log.logger.level.id, 'info')
+
     g.parseArgv(['rebuild']) // Also sets opts.argv.
 
-    assert.deepStrictEqual(Object.keys(g.opts).sort(), keys.sort())
+    assert.strictEqual(log.logger.level.id, 'silly')
+
+    assert.deepStrictEqual(Object.keys(g.opts).sort(), [...keys, 'argv', 'x', 'y', 'foo'].sort())
+    assert.strictEqual(g.opts['x'], '42')
+    assert.strictEqual(g.opts['y'], '42')
+    assert.strictEqual(g.opts['foo'], '42')
   })
 
   it('options with spaces in environment', () => {

From 5538e6c5d78dffd41e2a588adfa7ea9022150b9d Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Tue, 19 Aug 2025 12:39:44 -0700
Subject: [PATCH 480/551] feat: read from config case-insensitively (#3198)

---
 lib/node-gyp.js      | 17 +++++++++--------
 test/test-options.js |  7 ++++++-
 2 files changed, 15 insertions(+), 9 deletions(-)

diff --git a/lib/node-gyp.js b/lib/node-gyp.js
index 26fcb2ee49..dafce99d49 100644
--- a/lib/node-gyp.js
+++ b/lib/node-gyp.js
@@ -133,24 +133,25 @@ class Gyp extends EventEmitter {
     //   other tools.
     // The `npm_package_config_node_gyp_` prefix will take precedence over
     // `npm_config_` keys.
-    const npmConfigPrefix = 'npm_config_'
-    const npmPackageConfigPrefix = 'npm_package_config_node_gyp_'
+    const npmConfigPrefix = /^npm_config_/i
+    const npmPackageConfigPrefix = /^npm_package_config_node_gyp_/i
 
     const configEnvKeys = Object.keys(process.env)
-      .filter((k) => k.startsWith(npmConfigPrefix) || k.startsWith(npmPackageConfigPrefix))
+      .filter((k) => npmConfigPrefix.test(k) || npmPackageConfigPrefix.test(k))
       // sort so that npm_package_config_node_gyp_ keys come last and will override
-      .sort((a) => a.startsWith(npmConfigPrefix) ? -1 : 1)
+      .sort((a) => npmConfigPrefix.test(a) ? -1 : 1)
 
     for (const key of configEnvKeys) {
       // add the user-defined options to the config
-      const name = key.startsWith(npmConfigPrefix)
-        ? key.substring(npmConfigPrefix.length)
-        : key.substring(npmPackageConfigPrefix.length)
+      const name = npmConfigPrefix.test(key)
+        ? key.replace(npmConfigPrefix, '')
+        : key.replace(npmPackageConfigPrefix, '')
       // gyp@741b7f1 enters an infinite loop when it encounters
       // zero-length options so ensure those don't get through.
       if (name) {
         // convert names like force_process_config to force-process-config
-        this.opts[name.replaceAll('_', '-')] = process.env[key]
+        // and convert to lowercase
+        this.opts[name.replaceAll('_', '-').toLowerCase()] = process.env[key]
       }
     }
 
diff --git a/test/test-options.js b/test/test-options.js
index 8ea616fd1a..e14f827d56 100644
--- a/test/test-options.js
+++ b/test/test-options.js
@@ -30,6 +30,9 @@ describe('options', function () {
     process.env.npm_config_y = '41'
     // Package config should take precedence over npm_config_ keys.
     process.env.npm_package_config_node_gyp_y = '42'
+    // All configs should be case-insensitive.
+    process.env.NPM_PACKAGE_CONFIG_NODE_GYP_XX = 'value'
+    process.env.NPM_CONFIG_YY = 'value'
     // loglevel does not get added to opts but will change the logger's level.
     process.env.npm_config_loglevel = 'silly'
 
@@ -41,10 +44,12 @@ describe('options', function () {
 
     assert.strictEqual(log.logger.level.id, 'silly')
 
-    assert.deepStrictEqual(Object.keys(g.opts).sort(), [...keys, 'argv', 'x', 'y', 'foo'].sort())
+    assert.deepStrictEqual(Object.keys(g.opts).sort(), [...keys, 'argv', 'x', 'y', 'foo', 'xx', 'yy'].sort())
     assert.strictEqual(g.opts['x'], '42')
     assert.strictEqual(g.opts['y'], '42')
     assert.strictEqual(g.opts['foo'], '42')
+    assert.strictEqual(g.opts['xx'], 'value')
+    assert.strictEqual(g.opts['yy'], 'value')
   })
 
   it('options with spaces in environment', () => {

From af41747502e7d840b88c88e69d0622a7e45e92af Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Tue, 19 Aug 2025 20:40:11 +0100
Subject: [PATCH 481/551] chore(main): release 11.4.0

---
 .release-please-manifest.json |  2 +-
 CHANGELOG.md                  | 18 ++++++++++++++++++
 package.json                  |  2 +-
 3 files changed, 20 insertions(+), 2 deletions(-)

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
index 200e9cc761..f53d4ebc6a 100644
--- a/.release-please-manifest.json
+++ b/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "11.3.0"
+    ".": "11.4.0"
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5030eaa973..1763cc56c6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,23 @@
 # Changelog
 
+## [11.4.0](https://github.com/nodejs/node-gyp/compare/v11.3.0...v11.4.0) (2025-08-19)
+
+
+### Features
+
+* read from config case-insensitively ([#3198](https://github.com/nodejs/node-gyp/issues/3198)) ([5538e6c](https://github.com/nodejs/node-gyp/commit/5538e6c5d78dffd41e2a588adfa7ea9022150b9d))
+* support reading config from package.json ([#3196](https://github.com/nodejs/node-gyp/issues/3196)) ([1822dff](https://github.com/nodejs/node-gyp/commit/1822dff4f616a30ac3ca72e5946d81389cb8557e)), closes [#3156](https://github.com/nodejs/node-gyp/issues/3156)
+
+
+### Core
+
+* **deps:** bump actions/checkout from 4 to 5 ([#3193](https://github.com/nodejs/node-gyp/issues/3193)) ([27f5505](https://github.com/nodejs/node-gyp/commit/27f5505ec236551081366bf8a9c13ef5d8e468bf))
+
+
+### Miscellaneous
+
+* use npm oicd connection for publishing ([#3197](https://github.com/nodejs/node-gyp/issues/3197)) ([0773615](https://github.com/nodejs/node-gyp/commit/077361502933fcb994ca365c3c07c03177503df2))
+
 ## [11.3.0](https://github.com/nodejs/node-gyp/compare/v11.2.0...v11.3.0) (2025-07-29)
 
 
diff --git a/package.json b/package.json
index eecdf7b5ec..ac08c5fd5d 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "11.3.0",
+  "version": "11.4.0",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From 6b9638a0f80352e5bf7c1702e6ef622a6474d44a Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Wed, 20 Aug 2025 12:38:50 -0700
Subject: [PATCH 482/551] chore(release): use npm@11 for OIDC publishing
 (#3202)

OIDC is only supported in npm >=11.5.1.

Fixes #3201
---
 .github/workflows/release-please.yml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml
index e4854c8151..4779f74541 100644
--- a/.github/workflows/release-please.yml
+++ b/.github/workflows/release-please.yml
@@ -41,4 +41,5 @@ jobs:
         with:
           node-version: lts/*
           registry-url: 'https://registry.npmjs.org'
+      - run: npm install npm@11 -g # Use npm@11 to publish with OIDC
       - run: npm publish --provenance --access public

From 1ee15f5d53956ddddb8b025271fe11fbefcd6f41 Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Wed, 20 Aug 2025 21:45:00 +0100
Subject: [PATCH 483/551] chore(main): release 11.4.1 (#3204)

---
 .release-please-manifest.json | 2 +-
 CHANGELOG.md                  | 7 +++++++
 package.json                  | 2 +-
 3 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
index f53d4ebc6a..8205383b6a 100644
--- a/.release-please-manifest.json
+++ b/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "11.4.0"
+    ".": "11.4.1"
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1763cc56c6..054f649b57 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,12 @@
 # Changelog
 
+## [11.4.1](https://github.com/nodejs/node-gyp/compare/v11.4.0...v11.4.1) (2025-08-20)
+
+
+### Miscellaneous
+
+* **release:** use npm@11 for OIDC publishing ([#3202](https://github.com/nodejs/node-gyp/issues/3202)) ([6b9638a](https://github.com/nodejs/node-gyp/commit/6b9638a0f80352e5bf7c1702e6ef622a6474d44a)), closes [#3201](https://github.com/nodejs/node-gyp/issues/3201)
+
 ## [11.4.0](https://github.com/nodejs/node-gyp/compare/v11.3.0...v11.4.0) (2025-08-19)
 
 
diff --git a/package.json b/package.json
index ac08c5fd5d..f0fb3bfa3a 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "11.4.0",
+  "version": "11.4.1",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From b0ee8a04ec2a32282cb9259f142ca622e67a3d36 Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Fri, 22 Aug 2025 21:04:57 +0100
Subject: [PATCH 484/551] feat: update gyp-next to v0.20.3 (#3205)

---
 gyp/.release-please-manifest.json |  2 +-
 gyp/CHANGELOG.md                  |  8 ++++++++
 gyp/pylib/gyp/generator/make.py   |  4 ++--
 gyp/pylib/gyp/xcode_emulation.py  | 11 ++++++-----
 gyp/pyproject.toml                |  2 +-
 5 files changed, 18 insertions(+), 9 deletions(-)

diff --git a/gyp/.release-please-manifest.json b/gyp/.release-please-manifest.json
index 69ae3d039e..f4119499ca 100644
--- a/gyp/.release-please-manifest.json
+++ b/gyp/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "0.20.2"
+    ".": "0.20.3"
 }
diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md
index c1c1c5909d..7af27a435c 100644
--- a/gyp/CHANGELOG.md
+++ b/gyp/CHANGELOG.md
@@ -1,5 +1,13 @@
 # Changelog
 
+## [0.20.3](https://github.com/nodejs/gyp-next/compare/v0.20.2...v0.20.3) (2025-08-20)
+
+
+### Bug Fixes
+
+* compilation failure on the OpenHarmony platform ([#301](https://github.com/nodejs/gyp-next/issues/301)) ([0cf7a14](https://github.com/nodejs/gyp-next/commit/0cf7a142be06f686b8b42849791de902f177cf9f))
+* make xcode_emulation handle `xcodebuild` not in the `PATH` ([#303](https://github.com/nodejs/gyp-next/issues/303)) ([8224dee](https://github.com/nodejs/gyp-next/commit/8224deef984add7e7afe846cfb82c9d3fa6da1fb))
+
 ## [0.20.2](https://github.com/nodejs/gyp-next/compare/v0.20.1...v0.20.2) (2025-06-22)
 
 
diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py
index 7118492e77..0ba1a8c4e1 100644
--- a/gyp/pylib/gyp/generator/make.py
+++ b/gyp/pylib/gyp/generator/make.py
@@ -1880,7 +1880,7 @@ def WriteTarget(
                 self.flavor not in ("mac", "openbsd", "netbsd", "win")
                 and not self.is_standalone_static_library
             ):
-                if self.flavor in ("linux", "android"):
+                if self.flavor in ("linux", "android", "openharmony"):
                     self.WriteMakeRule(
                         [self.output_binary],
                         link_deps,
@@ -1894,7 +1894,7 @@ def WriteTarget(
                         part_of_all,
                         postbuilds=postbuilds,
                     )
-            elif self.flavor in ("linux", "android"):
+            elif self.flavor in ("linux", "android", "openharmony"):
                 self.WriteMakeRule(
                     [self.output_binary],
                     link_deps,
diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py
index 0746865dc8..08e645c57d 100644
--- a/gyp/pylib/gyp/xcode_emulation.py
+++ b/gyp/pylib/gyp/xcode_emulation.py
@@ -521,7 +521,7 @@ def _GetSdkVersionInfoItem(self, sdk, infoitem):
         # most sensible route and should still do the right thing.
         try:
             return GetStdoutQuiet(["xcrun", "--sdk", sdk, infoitem])
-        except GypError:
+        except (GypError, OSError):
             pass
 
     def _SdkRoot(self, configname):
@@ -1354,7 +1354,7 @@ def _DefaultSdkRoot(self):
             return default_sdk_root
         try:
             all_sdks = GetStdout(["xcodebuild", "-showsdks"])
-        except GypError:
+        except (GypError, OSError):
             # If xcodebuild fails, there will be no valid SDKs
             return ""
         for line in all_sdks.splitlines():
@@ -1508,7 +1508,8 @@ def XcodeVersion():
             raise GypError("xcodebuild returned unexpected results")
         version = version_list[0].split()[-1]  # Last word on first line
         build = version_list[-1].split()[-1]  # Last word on last line
-    except GypError:  # Xcode not installed so look for XCode Command Line Tools
+    except (GypError, OSError):
+        # Xcode not installed so look for XCode Command Line Tools
         version = CLTVersion()  # macOS Catalina returns 11.0.0.0.1.1567737322
         if not version:
             raise GypError("No Xcode or CLT version detected!")
@@ -1541,14 +1542,14 @@ def CLTVersion():
         try:
             output = GetStdout(["/usr/sbin/pkgutil", "--pkg-info", key])
             return re.search(regex, output).groupdict()["version"]
-        except GypError:
+        except (GypError, OSError):
             continue
 
     regex = re.compile(r"Command Line Tools for Xcode\s+(?P\S+)")
     try:
         output = GetStdout(["/usr/sbin/softwareupdate", "--history"])
         return re.search(regex, output).groupdict()["version"]
-    except GypError:
+    except (GypError, OSError):
         return None
 
 
diff --git a/gyp/pyproject.toml b/gyp/pyproject.toml
index 62fb2bf8ca..b233d8504d 100644
--- a/gyp/pyproject.toml
+++ b/gyp/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "gyp-next"
-version = "0.20.2"
+version = "0.20.3"
 authors = [
   { name="Node.js contributors", email="ryzokuken@disroot.org" },
 ]

From b406532c77659c441c845708ec3ecdf09f013a3b Mon Sep 17 00:00:00 2001
From: hqzing 
Date: Tue, 26 Aug 2025 19:48:19 +0800
Subject: [PATCH 485/551] fix: add adaptation for OpenHarmony platform (#3207)

---
 addon.gypi | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/addon.gypi b/addon.gypi
index d8d188e2f8..4f112df81c 100644
--- a/addon.gypi
+++ b/addon.gypi
@@ -195,7 +195,7 @@
           '_FILE_OFFSET_BITS=64'
         ],
       }],
-      [ 'OS in "freebsd openbsd netbsd solaris android" or \
+      [ 'OS in "freebsd openbsd netbsd solaris android openharmony" or \
          (OS=="linux" and target_arch!="ia32")', {
         'cflags': [ '-fPIC' ],
       }],

From adc61b1458315d9648591e74bf16bbe39511401e Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Tue, 26 Aug 2025 13:48:27 +0100
Subject: [PATCH 486/551] fix: update gyp-next to v0.20.4 (#3208)

---
 gyp/.github/workflows/node-gyp.yml            |   4 +-
 gyp/.github/workflows/nodejs.yml              |   4 +-
 gyp/.github/workflows/python_tests.yml        |   3 +-
 gyp/.github/workflows/release-please.yml      |   8 +-
 gyp/.release-please-manifest.json             |   2 +-
 gyp/CHANGELOG.md                              |   8 +
 gyp/README.md                                 |  10 +-
 gyp/pylib/gyp/MSVSNew.py                      |  94 +--
 gyp/pylib/gyp/MSVSProject.py                  | 104 +--
 gyp/pylib/gyp/MSVSSettings.py                 | 236 +++---
 gyp/pylib/gyp/MSVSSettings_test.py            |  86 +-
 gyp/pylib/gyp/MSVSToolFile.py                 |  24 +-
 gyp/pylib/gyp/MSVSUserFile.py                 |  38 +-
 gyp/pylib/gyp/MSVSUtil.py                     | 100 +--
 gyp/pylib/gyp/MSVSVersion.py                  | 135 +--
 gyp/pylib/gyp/__init__.py                     |  77 +-
 gyp/pylib/gyp/common.py                       | 108 ++-
 gyp/pylib/gyp/common_test.py                  |  63 +-
 gyp/pylib/gyp/easy_xml.py                     | 111 +--
 gyp/pylib/gyp/easy_xml_test.py                |   2 +-
 gyp/pylib/gyp/generator/analyzer.py           | 128 +--
 gyp/pylib/gyp/generator/android.py            |  28 +-
 gyp/pylib/gyp/generator/cmake.py              | 131 ++-
 .../gyp/generator/dump_dependency_json.py     |   2 +-
 gyp/pylib/gyp/generator/eclipse.py            |  32 +-
 gyp/pylib/gyp/generator/gypd.py               |   1 -
 gyp/pylib/gyp/generator/gypsh.py              |   1 -
 gyp/pylib/gyp/generator/make.py               |  60 +-
 gyp/pylib/gyp/generator/msvs.py               | 794 +++++++++---------
 gyp/pylib/gyp/generator/msvs_test.py          |   2 +-
 gyp/pylib/gyp/generator/ninja.py              |  30 +-
 gyp/pylib/gyp/generator/ninja_test.py         |   2 +-
 gyp/pylib/gyp/generator/xcode.py              |  32 +-
 gyp/pylib/gyp/generator/xcode_test.py         |   2 +-
 gyp/pylib/gyp/input.py                        | 313 ++++---
 gyp/pylib/gyp/mac_tool.py                     | 191 +++--
 gyp/pylib/gyp/msvs_emulation.py               |  17 +-
 gyp/pylib/gyp/simple_copy.py                  |   4 +-
 gyp/pylib/gyp/win_tool.py                     |  33 +-
 gyp/pylib/gyp/xcode_emulation.py              | 345 ++++----
 gyp/pylib/gyp/xcode_ninja.py                  |  41 +-
 gyp/pylib/gyp/xcodeproj_file.py               | 594 +++++++------
 gyp/pylib/gyp/xml_fix.py                      |   1 -
 gyp/pyproject.toml                            |   2 +-
 gyp/test_gyp.py                               |   1 -
 gyp/tools/graphviz.py                         |   9 +-
 gyp/tools/pretty_gyp.py                       |  19 +-
 gyp/tools/pretty_sln.py                       |  12 +-
 gyp/tools/pretty_vcproj.py                    |  21 +-
 49 files changed, 2023 insertions(+), 2042 deletions(-)

diff --git a/gyp/.github/workflows/node-gyp.yml b/gyp/.github/workflows/node-gyp.yml
index c13b28effb..392722b6d8 100644
--- a/gyp/.github/workflows/node-gyp.yml
+++ b/gyp/.github/workflows/node-gyp.yml
@@ -25,11 +25,11 @@ jobs:
     runs-on: ${{ matrix.os }}
     steps:
       - name: Clone gyp-next
-        uses: actions/checkout@v4
+        uses: actions/checkout@v5
         with:
           path: gyp-next
       - name: Clone nodejs/node-gyp
-        uses: actions/checkout@v4
+        uses: actions/checkout@v5
         with:
           repository: nodejs/node-gyp
           path: node-gyp
diff --git a/gyp/.github/workflows/nodejs.yml b/gyp/.github/workflows/nodejs.yml
index 416dfe45d1..be6abf515a 100644
--- a/gyp/.github/workflows/nodejs.yml
+++ b/gyp/.github/workflows/nodejs.yml
@@ -22,11 +22,11 @@ jobs:
     runs-on: ${{ matrix.os }}
     steps:
       - name: Clone gyp-next
-        uses: actions/checkout@v4
+        uses: actions/checkout@v5
         with:
           path: gyp-next
       - name: Clone nodejs/node
-        uses: actions/checkout@v4
+        uses: actions/checkout@v5
         with:
           repository: nodejs/node
           path: node
diff --git a/gyp/.github/workflows/python_tests.yml b/gyp/.github/workflows/python_tests.yml
index b076442051..7a64bb4c7f 100644
--- a/gyp/.github/workflows/python_tests.yml
+++ b/gyp/.github/workflows/python_tests.yml
@@ -17,7 +17,7 @@ jobs:
         os: [macos-13, macos-latest, ubuntu-latest] # , windows-latest]
         python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"]
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v5
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v5
         with:
@@ -31,6 +31,7 @@ jobs:
       - run: ./gyp -V && ./gyp --version && gyp -V && gyp --version
       - name: Lint with ruff  # See pyproject.toml for settings
         uses: astral-sh/ruff-action@v3
+      - run: ruff format --check --diff
       - name: Test with pytest  # See pyproject.toml for settings
         run: pytest
       # - name: Run doctests with pytest
diff --git a/gyp/.github/workflows/release-please.yml b/gyp/.github/workflows/release-please.yml
index a843939215..2db84faf8b 100644
--- a/gyp/.github/workflows/release-please.yml
+++ b/gyp/.github/workflows/release-please.yml
@@ -24,7 +24,7 @@ jobs:
     if: ${{ needs.release-please.outputs.release_created }}  # only publish on release
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v5
     - name: Build a binary wheel and a source tarball
       run: pipx run build
     - name: Store the distribution packages
@@ -48,7 +48,7 @@ jobs:
       id-token: write # IMPORTANT: mandatory for trusted publishing
     steps:
     - name: Download all the dists
-      uses: actions/download-artifact@v4
+      uses: actions/download-artifact@v5
       with:
         name: python-package-distributions
         path: dist/
@@ -68,12 +68,12 @@ jobs:
       id-token: write # IMPORTANT: mandatory for sigstore
     steps:
     - name: Download all the dists
-      uses: actions/download-artifact@v4
+      uses: actions/download-artifact@v5
       with:
         name: python-package-distributions
         path: dist/
     - name: Sign the dists with Sigstore
-      uses: sigstore/gh-action-sigstore-python@v3.0.0
+      uses: sigstore/gh-action-sigstore-python@v3.0.1
       with:
         inputs: >-
           ./dist/*.tar.gz
diff --git a/gyp/.release-please-manifest.json b/gyp/.release-please-manifest.json
index f4119499ca..bdb726346f 100644
--- a/gyp/.release-please-manifest.json
+++ b/gyp/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "0.20.3"
+    ".": "0.20.4"
 }
diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md
index 7af27a435c..69449c0d80 100644
--- a/gyp/CHANGELOG.md
+++ b/gyp/CHANGELOG.md
@@ -1,5 +1,13 @@
 # Changelog
 
+## [0.20.4](https://github.com/nodejs/gyp-next/compare/v0.20.3...v0.20.4) (2025-08-25)
+
+
+### Bug Fixes
+
+* **cli:** remove duplicate usage ([#308](https://github.com/nodejs/gyp-next/issues/308)) ([0996f60](https://github.com/nodejs/gyp-next/commit/0996f60e9bc83ec9d7b31e39bebd23f8dc990130))
+* **docs:** Add running gyp via uv ([#306](https://github.com/nodejs/gyp-next/issues/306)) ([0e43f61](https://github.com/nodejs/gyp-next/commit/0e43f61da8154f9b460ccba9ce4c0a25d2383ac4))
+
 ## [0.20.3](https://github.com/nodejs/gyp-next/compare/v0.20.2...v0.20.3) (2025-08-20)
 
 
diff --git a/gyp/README.md b/gyp/README.md
index 38792e1de4..6360a6c334 100644
--- a/gyp/README.md
+++ b/gyp/README.md
@@ -6,8 +6,9 @@ Documents are available at [`./docs`](./docs).
 __gyp-next__ is [released](https://github.com/nodejs/gyp-next/releases) to the [__Python Packaging Index__](https://pypi.org/project/gyp-next) (PyPI) and can be installed with the command:
 * `python3 -m pip install gyp-next`
 
-When used as a command line utility, __gyp-next__ can also be installed with [pipx](https://pypa.github.io/pipx):
-* `pipx install gyp-next`
+When used as a command line utility, __gyp-next__ can also be installed with [pipx](https://pypa.github.io/pipx) or [uv](https://docs.astral.sh/uv):
+* `pipx install gyp-next ` # --or--
+* `uv tool install gyp-next`
 ```
 Installing to a new venv 'gyp-next'
   installed package gyp-next 0.13.0, installed using Python 3.10.6
@@ -17,10 +18,11 @@ done! ✨ 🌟 ✨
 ```
 
 Or to run __gyp-next__ directly without installing it:
-* `pipx run gyp-next --help`
+* `pipx run gyp-next --help ` # --or--
+* `uvx --from=gyp-next gyp --help`
 ```
 NOTE: running app 'gyp' from 'gyp-next'
-usage: usage: gyp [options ...] [build_file ...]
+usage: gyp [options ...] [build_file ...]
 
 options:
   -h, --help            show this help message and exit
diff --git a/gyp/pylib/gyp/MSVSNew.py b/gyp/pylib/gyp/MSVSNew.py
index bc0e93d07f..f8e4993d94 100644
--- a/gyp/pylib/gyp/MSVSNew.py
+++ b/gyp/pylib/gyp/MSVSNew.py
@@ -32,18 +32,18 @@ def cmp(x, y):
 def MakeGuid(name, seed="msvs_new"):
     """Returns a GUID for the specified target name.
 
-  Args:
-    name: Target name.
-    seed: Seed for MD5 hash.
-  Returns:
-    A GUID-line string calculated from the name and seed.
-
-  This generates something which looks like a GUID, but depends only on the
-  name and seed.  This means the same name/seed will always generate the same
-  GUID, so that projects and solutions which refer to each other can explicitly
-  determine the GUID to refer to explicitly.  It also means that the GUID will
-  not change when the project for a target is rebuilt.
-  """
+    Args:
+      name: Target name.
+      seed: Seed for MD5 hash.
+    Returns:
+      A GUID-line string calculated from the name and seed.
+
+    This generates something which looks like a GUID, but depends only on the
+    name and seed.  This means the same name/seed will always generate the same
+    GUID, so that projects and solutions which refer to each other can explicitly
+    determine the GUID to refer to explicitly.  It also means that the GUID will
+    not change when the project for a target is rebuilt.
+    """
     # Calculate a MD5 signature for the seed and name.
     d = hashlib.md5((str(seed) + str(name)).encode("utf-8")).hexdigest().upper()
     # Convert most of the signature to GUID form (discard the rest)
@@ -78,15 +78,15 @@ class MSVSFolder(MSVSSolutionEntry):
     def __init__(self, path, name=None, entries=None, guid=None, items=None):
         """Initializes the folder.
 
-    Args:
-      path: Full path to the folder.
-      name: Name of the folder.
-      entries: List of folder entries to nest inside this folder.  May contain
-          Folder or Project objects.  May be None, if the folder is empty.
-      guid: GUID to use for folder, if not None.
-      items: List of solution items to include in the folder project.  May be
-          None, if the folder does not directly contain items.
-    """
+        Args:
+          path: Full path to the folder.
+          name: Name of the folder.
+          entries: List of folder entries to nest inside this folder.  May contain
+              Folder or Project objects.  May be None, if the folder is empty.
+          guid: GUID to use for folder, if not None.
+          items: List of solution items to include in the folder project.  May be
+              None, if the folder does not directly contain items.
+        """
         if name:
             self.name = name
         else:
@@ -128,19 +128,19 @@ def __init__(
     ):
         """Initializes the project.
 
-    Args:
-      path: Absolute path to the project file.
-      name: Name of project.  If None, the name will be the same as the base
-          name of the project file.
-      dependencies: List of other Project objects this project is dependent
-          upon, if not None.
-      guid: GUID to use for project, if not None.
-      spec: Dictionary specifying how to build this project.
-      build_file: Filename of the .gyp file that the vcproj file comes from.
-      config_platform_overrides: optional dict of configuration platforms to
-          used in place of the default for this target.
-      fixpath_prefix: the path used to adjust the behavior of _fixpath
-    """
+        Args:
+          path: Absolute path to the project file.
+          name: Name of project.  If None, the name will be the same as the base
+              name of the project file.
+          dependencies: List of other Project objects this project is dependent
+              upon, if not None.
+          guid: GUID to use for project, if not None.
+          spec: Dictionary specifying how to build this project.
+          build_file: Filename of the .gyp file that the vcproj file comes from.
+          config_platform_overrides: optional dict of configuration platforms to
+              used in place of the default for this target.
+          fixpath_prefix: the path used to adjust the behavior of _fixpath
+        """
         self.path = path
         self.guid = guid
         self.spec = spec
@@ -195,16 +195,16 @@ def __init__(
     ):
         """Initializes the solution.
 
-    Args:
-      path: Path to solution file.
-      version: Format version to emit.
-      entries: List of entries in solution.  May contain Folder or Project
-          objects.  May be None, if the folder is empty.
-      variants: List of build variant strings.  If none, a default list will
-          be used.
-      websiteProperties: Flag to decide if the website properties section
-          is generated.
-    """
+        Args:
+          path: Path to solution file.
+          version: Format version to emit.
+          entries: List of entries in solution.  May contain Folder or Project
+              objects.  May be None, if the folder is empty.
+          variants: List of build variant strings.  If none, a default list will
+              be used.
+          websiteProperties: Flag to decide if the website properties section
+              is generated.
+        """
         self.path = path
         self.websiteProperties = websiteProperties
         self.version = version
@@ -230,9 +230,9 @@ def __init__(
     def Write(self, writer=gyp.common.WriteOnDiff):
         """Writes the solution file to disk.
 
-    Raises:
-      IndexError: An entry appears multiple times.
-    """
+        Raises:
+          IndexError: An entry appears multiple times.
+        """
         # Walk the entry tree and collect all the folders and projects.
         all_entries = set()
         entries_to_check = self.entries[:]
diff --git a/gyp/pylib/gyp/MSVSProject.py b/gyp/pylib/gyp/MSVSProject.py
index 339d27d402..17bb2bbdb8 100644
--- a/gyp/pylib/gyp/MSVSProject.py
+++ b/gyp/pylib/gyp/MSVSProject.py
@@ -15,19 +15,19 @@ class Tool:
     def __init__(self, name, attrs=None):
         """Initializes the tool.
 
-    Args:
-      name: Tool name.
-      attrs: Dict of tool attributes; may be None.
-    """
+        Args:
+          name: Tool name.
+          attrs: Dict of tool attributes; may be None.
+        """
         self._attrs = attrs or {}
         self._attrs["Name"] = name
 
     def _GetSpecification(self):
         """Creates an element for the tool.
 
-    Returns:
-      A new xml.dom.Element for the tool.
-    """
+        Returns:
+          A new xml.dom.Element for the tool.
+        """
         return ["Tool", self._attrs]
 
 
@@ -37,10 +37,10 @@ class Filter:
     def __init__(self, name, contents=None):
         """Initializes the folder.
 
-    Args:
-      name: Filter (folder) name.
-      contents: List of filenames and/or Filter objects contained.
-    """
+        Args:
+          name: Filter (folder) name.
+          contents: List of filenames and/or Filter objects contained.
+        """
         self.name = name
         self.contents = list(contents or [])
 
@@ -54,13 +54,13 @@ class Writer:
     def __init__(self, project_path, version, name, guid=None, platforms=None):
         """Initializes the project.
 
-    Args:
-      project_path: Path to the project file.
-      version: Format version to emit.
-      name: Name of the project.
-      guid: GUID to use for project, if not None.
-      platforms: Array of string, the supported platforms.  If null, ['Win32']
-    """
+        Args:
+          project_path: Path to the project file.
+          version: Format version to emit.
+          name: Name of the project.
+          guid: GUID to use for project, if not None.
+          platforms: Array of string, the supported platforms.  If null, ['Win32']
+        """
         self.project_path = project_path
         self.version = version
         self.name = name
@@ -84,21 +84,21 @@ def __init__(self, project_path, version, name, guid=None, platforms=None):
     def AddToolFile(self, path):
         """Adds a tool file to the project.
 
-    Args:
-      path: Relative path from project to tool file.
-    """
+        Args:
+          path: Relative path from project to tool file.
+        """
         self.tool_files_section.append(["ToolFile", {"RelativePath": path}])
 
     def _GetSpecForConfiguration(self, config_type, config_name, attrs, tools):
         """Returns the specification for a configuration.
 
-    Args:
-      config_type: Type of configuration node.
-      config_name: Configuration name.
-      attrs: Dict of configuration attributes; may be None.
-      tools: List of tools (strings or Tool objects); may be None.
-    Returns:
-    """
+        Args:
+          config_type: Type of configuration node.
+          config_name: Configuration name.
+          attrs: Dict of configuration attributes; may be None.
+          tools: List of tools (strings or Tool objects); may be None.
+        Returns:
+        """
         # Handle defaults
         if not attrs:
             attrs = {}
@@ -122,23 +122,23 @@ def _GetSpecForConfiguration(self, config_type, config_name, attrs, tools):
     def AddConfig(self, name, attrs=None, tools=None):
         """Adds a configuration to the project.
 
-    Args:
-      name: Configuration name.
-      attrs: Dict of configuration attributes; may be None.
-      tools: List of tools (strings or Tool objects); may be None.
-    """
+        Args:
+          name: Configuration name.
+          attrs: Dict of configuration attributes; may be None.
+          tools: List of tools (strings or Tool objects); may be None.
+        """
         spec = self._GetSpecForConfiguration("Configuration", name, attrs, tools)
         self.configurations_section.append(spec)
 
     def _AddFilesToNode(self, parent, files):
         """Adds files and/or filters to the parent node.
 
-    Args:
-      parent: Destination node
-      files: A list of Filter objects and/or relative paths to files.
+        Args:
+          parent: Destination node
+          files: A list of Filter objects and/or relative paths to files.
 
-    Will call itself recursively, if the files list contains Filter objects.
-    """
+        Will call itself recursively, if the files list contains Filter objects.
+        """
         for f in files:
             if isinstance(f, Filter):
                 node = ["Filter", {"Name": f.name}]
@@ -151,13 +151,13 @@ def _AddFilesToNode(self, parent, files):
     def AddFiles(self, files):
         """Adds files to the project.
 
-    Args:
-      files: A list of Filter objects and/or relative paths to files.
+        Args:
+          files: A list of Filter objects and/or relative paths to files.
 
-    This makes a copy of the file/filter tree at the time of this call.  If you
-    later add files to a Filter object which was passed into a previous call
-    to AddFiles(), it will not be reflected in this project.
-    """
+        This makes a copy of the file/filter tree at the time of this call.  If you
+        later add files to a Filter object which was passed into a previous call
+        to AddFiles(), it will not be reflected in this project.
+        """
         self._AddFilesToNode(self.files_section, files)
         # TODO(rspangler) This also doesn't handle adding files to an existing
         # filter.  That is, it doesn't merge the trees.
@@ -165,15 +165,15 @@ def AddFiles(self, files):
     def AddFileConfig(self, path, config, attrs=None, tools=None):
         """Adds a configuration to a file.
 
-    Args:
-      path: Relative path to the file.
-      config: Name of configuration to add.
-      attrs: Dict of configuration attributes; may be None.
-      tools: List of tools (strings or Tool objects); may be None.
+        Args:
+          path: Relative path to the file.
+          config: Name of configuration to add.
+          attrs: Dict of configuration attributes; may be None.
+          tools: List of tools (strings or Tool objects); may be None.
 
-    Raises:
-      ValueError: Relative path does not match any file added via AddFiles().
-    """
+        Raises:
+          ValueError: Relative path does not match any file added via AddFiles().
+        """
         # Find the file node with the right relative path
         parent = self.files_dict.get(path)
         if not parent:
diff --git a/gyp/pylib/gyp/MSVSSettings.py b/gyp/pylib/gyp/MSVSSettings.py
index 8f8f53bb71..155fc3a1cb 100644
--- a/gyp/pylib/gyp/MSVSSettings.py
+++ b/gyp/pylib/gyp/MSVSSettings.py
@@ -35,10 +35,10 @@
 class _Tool:
     """Represents a tool used by MSVS or MSBuild.
 
-  Attributes:
-      msvs_name: The name of the tool in MSVS.
-      msbuild_name: The name of the tool in MSBuild.
-  """
+    Attributes:
+        msvs_name: The name of the tool in MSVS.
+        msbuild_name: The name of the tool in MSBuild.
+    """
 
     def __init__(self, msvs_name, msbuild_name):
         self.msvs_name = msvs_name
@@ -48,11 +48,11 @@ def __init__(self, msvs_name, msbuild_name):
 def _AddTool(tool):
     """Adds a tool to the four dictionaries used to process settings.
 
-  This only defines the tool.  Each setting also needs to be added.
+    This only defines the tool.  Each setting also needs to be added.
 
-  Args:
-    tool: The _Tool object to be added.
-  """
+    Args:
+      tool: The _Tool object to be added.
+    """
     _msvs_validators[tool.msvs_name] = {}
     _msbuild_validators[tool.msbuild_name] = {}
     _msvs_to_msbuild_converters[tool.msvs_name] = {}
@@ -70,35 +70,35 @@ class _Type:
     def ValidateMSVS(self, value):
         """Verifies that the value is legal for MSVS.
 
-    Args:
-      value: the value to check for this type.
+        Args:
+          value: the value to check for this type.
 
-    Raises:
-      ValueError if value is not valid for MSVS.
-    """
+        Raises:
+          ValueError if value is not valid for MSVS.
+        """
 
     def ValidateMSBuild(self, value):
         """Verifies that the value is legal for MSBuild.
 
-    Args:
-      value: the value to check for this type.
+        Args:
+          value: the value to check for this type.
 
-    Raises:
-      ValueError if value is not valid for MSBuild.
-    """
+        Raises:
+          ValueError if value is not valid for MSBuild.
+        """
 
     def ConvertToMSBuild(self, value):
         """Returns the MSBuild equivalent of the MSVS value given.
 
-    Args:
-      value: the MSVS value to convert.
+        Args:
+          value: the MSVS value to convert.
 
-    Returns:
-      the MSBuild equivalent.
+        Returns:
+          the MSBuild equivalent.
 
-    Raises:
-      ValueError if value is not valid.
-    """
+        Raises:
+          ValueError if value is not valid.
+        """
         return value
 
 
@@ -178,15 +178,15 @@ def ConvertToMSBuild(self, value):
 class _Enumeration(_Type):
     """Type of settings that is an enumeration.
 
-  In MSVS, the values are indexes like '0', '1', and '2'.
-  MSBuild uses text labels that are more representative, like 'Win32'.
+    In MSVS, the values are indexes like '0', '1', and '2'.
+    MSBuild uses text labels that are more representative, like 'Win32'.
 
-  Constructor args:
-    label_list: an array of MSBuild labels that correspond to the MSVS index.
-        In the rare cases where MSVS has skipped an index value, None is
-        used in the array to indicate the unused spot.
-    new: an array of labels that are new to MSBuild.
-  """
+    Constructor args:
+      label_list: an array of MSBuild labels that correspond to the MSVS index.
+          In the rare cases where MSVS has skipped an index value, None is
+          used in the array to indicate the unused spot.
+      new: an array of labels that are new to MSBuild.
+    """
 
     def __init__(self, label_list, new=None):
         _Type.__init__(self)
@@ -234,23 +234,23 @@ def ConvertToMSBuild(self, value):
 def _Same(tool, name, setting_type):
     """Defines a setting that has the same name in MSVS and MSBuild.
 
-  Args:
-    tool: a dictionary that gives the names of the tool for MSVS and MSBuild.
-    name: the name of the setting.
-    setting_type: the type of this setting.
-  """
+    Args:
+      tool: a dictionary that gives the names of the tool for MSVS and MSBuild.
+      name: the name of the setting.
+      setting_type: the type of this setting.
+    """
     _Renamed(tool, name, name, setting_type)
 
 
 def _Renamed(tool, msvs_name, msbuild_name, setting_type):
     """Defines a setting for which the name has changed.
 
-  Args:
-    tool: a dictionary that gives the names of the tool for MSVS and MSBuild.
-    msvs_name: the name of the MSVS setting.
-    msbuild_name: the name of the MSBuild setting.
-    setting_type: the type of this setting.
-  """
+    Args:
+      tool: a dictionary that gives the names of the tool for MSVS and MSBuild.
+      msvs_name: the name of the MSVS setting.
+      msbuild_name: the name of the MSBuild setting.
+      setting_type: the type of this setting.
+    """
 
     def _Translate(value, msbuild_settings):
         msbuild_tool_settings = _GetMSBuildToolSettings(msbuild_settings, tool)
@@ -272,13 +272,13 @@ def _MovedAndRenamed(
 ):
     """Defines a setting that may have moved to a new section.
 
-  Args:
-    tool: a dictionary that gives the names of the tool for MSVS and MSBuild.
-    msvs_settings_name: the MSVS name of the setting.
-    msbuild_tool_name: the name of the MSBuild tool to place the setting under.
-    msbuild_settings_name: the MSBuild name of the setting.
-    setting_type: the type of this setting.
-  """
+    Args:
+      tool: a dictionary that gives the names of the tool for MSVS and MSBuild.
+      msvs_settings_name: the MSVS name of the setting.
+      msbuild_tool_name: the name of the MSBuild tool to place the setting under.
+      msbuild_settings_name: the MSBuild name of the setting.
+      setting_type: the type of this setting.
+    """
 
     def _Translate(value, msbuild_settings):
         tool_settings = msbuild_settings.setdefault(msbuild_tool_name, {})
@@ -293,11 +293,11 @@ def _Translate(value, msbuild_settings):
 def _MSVSOnly(tool, name, setting_type):
     """Defines a setting that is only found in MSVS.
 
-  Args:
-    tool: a dictionary that gives the names of the tool for MSVS and MSBuild.
-    name: the name of the setting.
-    setting_type: the type of this setting.
-  """
+    Args:
+      tool: a dictionary that gives the names of the tool for MSVS and MSBuild.
+      name: the name of the setting.
+      setting_type: the type of this setting.
+    """
 
     def _Translate(unused_value, unused_msbuild_settings):
         # Since this is for MSVS only settings, no translation will happen.
@@ -310,11 +310,11 @@ def _Translate(unused_value, unused_msbuild_settings):
 def _MSBuildOnly(tool, name, setting_type):
     """Defines a setting that is only found in MSBuild.
 
-  Args:
-    tool: a dictionary that gives the names of the tool for MSVS and MSBuild.
-    name: the name of the setting.
-    setting_type: the type of this setting.
-  """
+    Args:
+      tool: a dictionary that gives the names of the tool for MSVS and MSBuild.
+      name: the name of the setting.
+      setting_type: the type of this setting.
+    """
 
     def _Translate(value, msbuild_settings):
         # Let msbuild-only properties get translated as-is from msvs_settings.
@@ -328,11 +328,11 @@ def _Translate(value, msbuild_settings):
 def _ConvertedToAdditionalOption(tool, msvs_name, flag):
     """Defines a setting that's handled via a command line option in MSBuild.
 
-  Args:
-    tool: a dictionary that gives the names of the tool for MSVS and MSBuild.
-    msvs_name: the name of the MSVS setting that if 'true' becomes a flag
-    flag: the flag to insert at the end of the AdditionalOptions
-  """
+    Args:
+      tool: a dictionary that gives the names of the tool for MSVS and MSBuild.
+      msvs_name: the name of the MSVS setting that if 'true' becomes a flag
+      flag: the flag to insert at the end of the AdditionalOptions
+    """
 
     def _Translate(value, msbuild_settings):
         if value == "true":
@@ -384,15 +384,15 @@ def _Translate(value, msbuild_settings):
 def _ValidateExclusionSetting(setting, settings, error_msg, stderr=sys.stderr):
     """Verify that 'setting' is valid if it is generated from an exclusion list.
 
-  If the setting appears to be generated from an exclusion list, the root name
-  is checked.
+    If the setting appears to be generated from an exclusion list, the root name
+    is checked.
 
-  Args:
-      setting:   A string that is the setting name to validate
-      settings:  A dictionary where the keys are valid settings
-      error_msg: The message to emit in the event of error
-      stderr:    The stream receiving the error messages.
-  """
+    Args:
+        setting:   A string that is the setting name to validate
+        settings:  A dictionary where the keys are valid settings
+        error_msg: The message to emit in the event of error
+        stderr:    The stream receiving the error messages.
+    """
     # This may be unrecognized because it's an exclusion list. If the
     # setting name has the _excluded suffix, then check the root name.
     unrecognized = True
@@ -408,11 +408,11 @@ def _ValidateExclusionSetting(setting, settings, error_msg, stderr=sys.stderr):
 def FixVCMacroSlashes(s):
     """Replace macros which have excessive following slashes.
 
-  These macros are known to have a built-in trailing slash. Furthermore, many
-  scripts hiccup on processing paths with extra slashes in the middle.
+    These macros are known to have a built-in trailing slash. Furthermore, many
+    scripts hiccup on processing paths with extra slashes in the middle.
 
-  This list is probably not exhaustive.  Add as needed.
-  """
+    This list is probably not exhaustive.  Add as needed.
+    """
     if "$" in s:
         s = fix_vc_macro_slashes_regex.sub(r"\1", s)
     return s
@@ -421,8 +421,8 @@ def FixVCMacroSlashes(s):
 def ConvertVCMacrosToMSBuild(s):
     """Convert the MSVS macros found in the string to the MSBuild equivalent.
 
-  This list is probably not exhaustive.  Add as needed.
-  """
+    This list is probably not exhaustive.  Add as needed.
+    """
     if "$" in s:
         replace_map = {
             "$(ConfigurationName)": "$(Configuration)",
@@ -444,16 +444,16 @@ def ConvertVCMacrosToMSBuild(s):
 def ConvertToMSBuildSettings(msvs_settings, stderr=sys.stderr):
     """Converts MSVS settings (VS2008 and earlier) to MSBuild settings (VS2010+).
 
-  Args:
-      msvs_settings: A dictionary.  The key is the tool name.  The values are
-          themselves dictionaries of settings and their values.
-      stderr: The stream receiving the error messages.
+    Args:
+        msvs_settings: A dictionary.  The key is the tool name.  The values are
+            themselves dictionaries of settings and their values.
+        stderr: The stream receiving the error messages.
 
-  Returns:
-      A dictionary of MSBuild settings.  The key is either the MSBuild tool name
-      or the empty string (for the global settings).  The values are themselves
-      dictionaries of settings and their values.
-  """
+    Returns:
+        A dictionary of MSBuild settings.  The key is either the MSBuild tool name
+        or the empty string (for the global settings).  The values are themselves
+        dictionaries of settings and their values.
+    """
     msbuild_settings = {}
     for msvs_tool_name, msvs_tool_settings in msvs_settings.items():
         if msvs_tool_name in _msvs_to_msbuild_converters:
@@ -492,36 +492,36 @@ def ConvertToMSBuildSettings(msvs_settings, stderr=sys.stderr):
 def ValidateMSVSSettings(settings, stderr=sys.stderr):
     """Validates that the names of the settings are valid for MSVS.
 
-  Args:
-      settings: A dictionary.  The key is the tool name.  The values are
-          themselves dictionaries of settings and their values.
-      stderr: The stream receiving the error messages.
-  """
+    Args:
+        settings: A dictionary.  The key is the tool name.  The values are
+            themselves dictionaries of settings and their values.
+        stderr: The stream receiving the error messages.
+    """
     _ValidateSettings(_msvs_validators, settings, stderr)
 
 
 def ValidateMSBuildSettings(settings, stderr=sys.stderr):
     """Validates that the names of the settings are valid for MSBuild.
 
-  Args:
-      settings: A dictionary.  The key is the tool name.  The values are
-          themselves dictionaries of settings and their values.
-      stderr: The stream receiving the error messages.
-  """
+    Args:
+        settings: A dictionary.  The key is the tool name.  The values are
+            themselves dictionaries of settings and their values.
+        stderr: The stream receiving the error messages.
+    """
     _ValidateSettings(_msbuild_validators, settings, stderr)
 
 
 def _ValidateSettings(validators, settings, stderr):
     """Validates that the settings are valid for MSBuild or MSVS.
 
-  We currently only validate the names of the settings, not their values.
+    We currently only validate the names of the settings, not their values.
 
-  Args:
-      validators: A dictionary of tools and their validators.
-      settings: A dictionary.  The key is the tool name.  The values are
-          themselves dictionaries of settings and their values.
-      stderr: The stream receiving the error messages.
-  """
+    Args:
+        validators: A dictionary of tools and their validators.
+        settings: A dictionary.  The key is the tool name.  The values are
+            themselves dictionaries of settings and their values.
+        stderr: The stream receiving the error messages.
+    """
     for tool_name in settings:
         if tool_name in validators:
             tool_validators = validators[tool_name]
@@ -637,7 +637,9 @@ def _ValidateSettings(validators, settings, stderr):
     ),
 )  # /RTC1
 _Same(
-    _compile, "BrowseInformation", _Enumeration(["false", "true", "true"])  # /FR
+    _compile,
+    "BrowseInformation",
+    _Enumeration(["false", "true", "true"]),  # /FR
 )  # /Fr
 _Same(
     _compile,
@@ -695,7 +697,9 @@ def _ValidateSettings(validators, settings, stderr):
     _Enumeration(["false", "Sync", "Async"], new=["SyncCThrow"]),  # /EHsc  # /EHa
 )  # /EHs
 _Same(
-    _compile, "FavorSizeOrSpeed", _Enumeration(["Neither", "Speed", "Size"])  # /Ot
+    _compile,
+    "FavorSizeOrSpeed",
+    _Enumeration(["Neither", "Speed", "Size"]),  # /Ot
 )  # /Os
 _Same(
     _compile,
@@ -908,7 +912,9 @@ def _ValidateSettings(validators, settings, stderr):
 )  # /MACHINE:X64
 
 _Same(
-    _link, "AssemblyDebug", _Enumeration(["", "true", "false"])  # /ASSEMBLYDEBUG
+    _link,
+    "AssemblyDebug",
+    _Enumeration(["", "true", "false"]),  # /ASSEMBLYDEBUG
 )  # /ASSEMBLYDEBUG:DISABLE
 _Same(
     _link,
@@ -1158,17 +1164,23 @@ def _ValidateSettings(validators, settings, stderr):
 _MSBuildOnly(_midl, "ApplicationConfigurationMode", _boolean)  # /app_config
 _MSBuildOnly(_midl, "ClientStubFile", _file_name)  # /cstub
 _MSBuildOnly(
-    _midl, "GenerateClientFiles", _Enumeration([], new=["Stub", "None"])  # /client stub
+    _midl,
+    "GenerateClientFiles",
+    _Enumeration([], new=["Stub", "None"]),  # /client stub
 )  # /client none
 _MSBuildOnly(
-    _midl, "GenerateServerFiles", _Enumeration([], new=["Stub", "None"])  # /client stub
+    _midl,
+    "GenerateServerFiles",
+    _Enumeration([], new=["Stub", "None"]),  # /client stub
 )  # /client none
 _MSBuildOnly(_midl, "LocaleID", _integer)  # /lcid DECIMAL
 _MSBuildOnly(_midl, "ServerStubFile", _file_name)  # /sstub
 _MSBuildOnly(_midl, "SuppressCompilerWarnings", _boolean)  # /no_warn
 _MSBuildOnly(_midl, "TrackerLogDirectory", _folder_name)
 _MSBuildOnly(
-    _midl, "TypeLibFormat", _Enumeration([], new=["NewFormat", "OldFormat"])  # /newtlb
+    _midl,
+    "TypeLibFormat",
+    _Enumeration([], new=["NewFormat", "OldFormat"]),  # /newtlb
 )  # /oldtlb
 
 
diff --git a/gyp/pylib/gyp/MSVSSettings_test.py b/gyp/pylib/gyp/MSVSSettings_test.py
index 0504728d99..0e661995fb 100755
--- a/gyp/pylib/gyp/MSVSSettings_test.py
+++ b/gyp/pylib/gyp/MSVSSettings_test.py
@@ -1143,47 +1143,47 @@ def testConvertToMSBuildSettings_full_synthetic(self):
     def testConvertToMSBuildSettings_actual(self):
         """Tests the conversion of an actual project.
 
-    A VS2008 project with most of the options defined was created through the
-    VS2008 IDE.  It was then converted to VS2010.  The tool settings found in
-    the .vcproj and .vcxproj files were converted to the two dictionaries
-    msvs_settings and expected_msbuild_settings.
+        A VS2008 project with most of the options defined was created through the
+        VS2008 IDE.  It was then converted to VS2010.  The tool settings found in
+        the .vcproj and .vcxproj files were converted to the two dictionaries
+        msvs_settings and expected_msbuild_settings.
 
-    Note that for many settings, the VS2010 converter adds macros like
-    %(AdditionalIncludeDirectories) to make sure than inherited values are
-    included.  Since the Gyp projects we generate do not use inheritance,
-    we removed these macros.  They were:
-        ClCompile:
-            AdditionalIncludeDirectories:  ';%(AdditionalIncludeDirectories)'
-            AdditionalOptions:  ' %(AdditionalOptions)'
-            AdditionalUsingDirectories:  ';%(AdditionalUsingDirectories)'
-            DisableSpecificWarnings: ';%(DisableSpecificWarnings)',
-            ForcedIncludeFiles:  ';%(ForcedIncludeFiles)',
-            ForcedUsingFiles:  ';%(ForcedUsingFiles)',
-            PreprocessorDefinitions:  ';%(PreprocessorDefinitions)',
-            UndefinePreprocessorDefinitions:
-                ';%(UndefinePreprocessorDefinitions)',
-        Link:
-            AdditionalDependencies:  ';%(AdditionalDependencies)',
-            AdditionalLibraryDirectories:  ';%(AdditionalLibraryDirectories)',
-            AdditionalManifestDependencies:
-                ';%(AdditionalManifestDependencies)',
-            AdditionalOptions:  ' %(AdditionalOptions)',
-            AddModuleNamesToAssembly:  ';%(AddModuleNamesToAssembly)',
-            AssemblyLinkResource:  ';%(AssemblyLinkResource)',
-            DelayLoadDLLs:  ';%(DelayLoadDLLs)',
-            EmbedManagedResourceFile:  ';%(EmbedManagedResourceFile)',
-            ForceSymbolReferences:  ';%(ForceSymbolReferences)',
-            IgnoreSpecificDefaultLibraries:
-                ';%(IgnoreSpecificDefaultLibraries)',
-        ResourceCompile:
-            AdditionalIncludeDirectories:  ';%(AdditionalIncludeDirectories)',
-            AdditionalOptions:  ' %(AdditionalOptions)',
-            PreprocessorDefinitions:  ';%(PreprocessorDefinitions)',
-        Manifest:
-            AdditionalManifestFiles:  ';%(AdditionalManifestFiles)',
-            AdditionalOptions:  ' %(AdditionalOptions)',
-            InputResourceManifests:  ';%(InputResourceManifests)',
-    """
+        Note that for many settings, the VS2010 converter adds macros like
+        %(AdditionalIncludeDirectories) to make sure than inherited values are
+        included.  Since the Gyp projects we generate do not use inheritance,
+        we removed these macros.  They were:
+            ClCompile:
+                AdditionalIncludeDirectories:  ';%(AdditionalIncludeDirectories)'
+                AdditionalOptions:  ' %(AdditionalOptions)'
+                AdditionalUsingDirectories:  ';%(AdditionalUsingDirectories)'
+                DisableSpecificWarnings: ';%(DisableSpecificWarnings)',
+                ForcedIncludeFiles:  ';%(ForcedIncludeFiles)',
+                ForcedUsingFiles:  ';%(ForcedUsingFiles)',
+                PreprocessorDefinitions:  ';%(PreprocessorDefinitions)',
+                UndefinePreprocessorDefinitions:
+                    ';%(UndefinePreprocessorDefinitions)',
+            Link:
+                AdditionalDependencies:  ';%(AdditionalDependencies)',
+                AdditionalLibraryDirectories:  ';%(AdditionalLibraryDirectories)',
+                AdditionalManifestDependencies:
+                    ';%(AdditionalManifestDependencies)',
+                AdditionalOptions:  ' %(AdditionalOptions)',
+                AddModuleNamesToAssembly:  ';%(AddModuleNamesToAssembly)',
+                AssemblyLinkResource:  ';%(AssemblyLinkResource)',
+                DelayLoadDLLs:  ';%(DelayLoadDLLs)',
+                EmbedManagedResourceFile:  ';%(EmbedManagedResourceFile)',
+                ForceSymbolReferences:  ';%(ForceSymbolReferences)',
+                IgnoreSpecificDefaultLibraries:
+                    ';%(IgnoreSpecificDefaultLibraries)',
+            ResourceCompile:
+                AdditionalIncludeDirectories:  ';%(AdditionalIncludeDirectories)',
+                AdditionalOptions:  ' %(AdditionalOptions)',
+                PreprocessorDefinitions:  ';%(PreprocessorDefinitions)',
+            Manifest:
+                AdditionalManifestFiles:  ';%(AdditionalManifestFiles)',
+                AdditionalOptions:  ' %(AdditionalOptions)',
+                InputResourceManifests:  ';%(InputResourceManifests)',
+        """
         msvs_settings = {
             "VCCLCompilerTool": {
                 "AdditionalIncludeDirectories": "dir1",
@@ -1346,8 +1346,7 @@ def testConvertToMSBuildSettings_actual(self):
                 "EmbedManifest": "false",
                 "GenerateCatalogFiles": "true",
                 "InputResourceManifests": "asfsfdafs",
-                "ManifestResourceFile":
-                    "$(IntDir)\\$(TargetFileName).embed.manifest.resfdsf",
+                "ManifestResourceFile": "$(IntDir)\\$(TargetFileName).embed.manifest.resfdsf",  # noqa: E501
                 "OutputManifestFile": "$(TargetPath).manifestdfs",
                 "RegistrarScriptFile": "sdfsfd",
                 "ReplacementsFile": "sdffsd",
@@ -1531,8 +1530,7 @@ def testConvertToMSBuildSettings_actual(self):
                 "LinkIncremental": "",
             },
             "ManifestResourceCompile": {
-                "ResourceOutputFileName":
-                    "$(IntDir)$(TargetFileName).embed.manifest.resfdsf"
+                "ResourceOutputFileName": "$(IntDir)$(TargetFileName).embed.manifest.resfdsf"  # noqa: E501
             },
         }
         self.maxDiff = 9999  # on failure display a long diff
diff --git a/gyp/pylib/gyp/MSVSToolFile.py b/gyp/pylib/gyp/MSVSToolFile.py
index 901ba84588..61ca37c12d 100644
--- a/gyp/pylib/gyp/MSVSToolFile.py
+++ b/gyp/pylib/gyp/MSVSToolFile.py
@@ -13,10 +13,10 @@ class Writer:
     def __init__(self, tool_file_path, name):
         """Initializes the tool file.
 
-    Args:
-      tool_file_path: Path to the tool file.
-      name: Name of the tool file.
-    """
+        Args:
+          tool_file_path: Path to the tool file.
+          name: Name of the tool file.
+        """
         self.tool_file_path = tool_file_path
         self.name = name
         self.rules_section = ["Rules"]
@@ -26,14 +26,14 @@ def AddCustomBuildRule(
     ):
         """Adds a rule to the tool file.
 
-    Args:
-      name: Name of the rule.
-      description: Description of the rule.
-      cmd: Command line of the rule.
-      additional_dependencies: other files which may trigger the rule.
-      outputs: outputs of the rule.
-      extensions: extensions handled by the rule.
-    """
+        Args:
+          name: Name of the rule.
+          description: Description of the rule.
+          cmd: Command line of the rule.
+          additional_dependencies: other files which may trigger the rule.
+          outputs: outputs of the rule.
+          extensions: extensions handled by the rule.
+        """
         rule = [
             "CustomBuildRule",
             {
diff --git a/gyp/pylib/gyp/MSVSUserFile.py b/gyp/pylib/gyp/MSVSUserFile.py
index 23d3e16953..b93613bd1d 100644
--- a/gyp/pylib/gyp/MSVSUserFile.py
+++ b/gyp/pylib/gyp/MSVSUserFile.py
@@ -15,11 +15,11 @@
 
 def _FindCommandInPath(command):
     """If there are no slashes in the command given, this function
-     searches the PATH env to find the given command, and converts it
-     to an absolute path.  We have to do this because MSVS is looking
-     for an actual file to launch a debugger on, not just a command
-     line.  Note that this happens at GYP time, so anything needing to
-     be built needs to have a full path."""
+    searches the PATH env to find the given command, and converts it
+    to an absolute path.  We have to do this because MSVS is looking
+    for an actual file to launch a debugger on, not just a command
+    line.  Note that this happens at GYP time, so anything needing to
+    be built needs to have a full path."""
     if "/" in command or "\\" in command:
         # If the command already has path elements (either relative or
         # absolute), then assume it is constructed properly.
@@ -58,11 +58,11 @@ class Writer:
     def __init__(self, user_file_path, version, name):
         """Initializes the user file.
 
-    Args:
-      user_file_path: Path to the user file.
-      version: Version info.
-      name: Name of the user file.
-    """
+        Args:
+          user_file_path: Path to the user file.
+          version: Version info.
+          name: Name of the user file.
+        """
         self.user_file_path = user_file_path
         self.version = version
         self.name = name
@@ -71,9 +71,9 @@ def __init__(self, user_file_path, version, name):
     def AddConfig(self, name):
         """Adds a configuration to the project.
 
-    Args:
-      name: Configuration name.
-    """
+        Args:
+          name: Configuration name.
+        """
         self.configurations[name] = ["Configuration", {"Name": name}]
 
     def AddDebugSettings(
@@ -81,12 +81,12 @@ def AddDebugSettings(
     ):
         """Adds a DebugSettings node to the user file for a particular config.
 
-    Args:
-      command: command line to run.  First element in the list is the
-        executable.  All elements of the command will be quoted if
-        necessary.
-      working_directory: other files which may trigger the rule. (optional)
-    """
+        Args:
+          command: command line to run.  First element in the list is the
+            executable.  All elements of the command will be quoted if
+            necessary.
+          working_directory: other files which may trigger the rule. (optional)
+        """
         command = _QuoteWin32CommandLineArgs(command)
 
         abs_command = _FindCommandInPath(command[0])
diff --git a/gyp/pylib/gyp/MSVSUtil.py b/gyp/pylib/gyp/MSVSUtil.py
index 27647f11d0..5a1b4ae319 100644
--- a/gyp/pylib/gyp/MSVSUtil.py
+++ b/gyp/pylib/gyp/MSVSUtil.py
@@ -29,13 +29,13 @@ def _GetLargePdbShimCcPath():
 def _DeepCopySomeKeys(in_dict, keys):
     """Performs a partial deep-copy on |in_dict|, only copying the keys in |keys|.
 
-  Arguments:
-    in_dict: The dictionary to copy.
-    keys: The keys to be copied. If a key is in this list and doesn't exist in
-        |in_dict| this is not an error.
-  Returns:
-    The partially deep-copied dictionary.
-  """
+    Arguments:
+      in_dict: The dictionary to copy.
+      keys: The keys to be copied. If a key is in this list and doesn't exist in
+          |in_dict| this is not an error.
+    Returns:
+      The partially deep-copied dictionary.
+    """
     d = {}
     for key in keys:
         if key not in in_dict:
@@ -47,12 +47,12 @@ def _DeepCopySomeKeys(in_dict, keys):
 def _SuffixName(name, suffix):
     """Add a suffix to the end of a target.
 
-  Arguments:
-    name: name of the target (foo#target)
-    suffix: the suffix to be added
-  Returns:
-    Target name with suffix added (foo_suffix#target)
-  """
+    Arguments:
+      name: name of the target (foo#target)
+      suffix: the suffix to be added
+    Returns:
+      Target name with suffix added (foo_suffix#target)
+    """
     parts = name.rsplit("#", 1)
     parts[0] = f"{parts[0]}_{suffix}"
     return "#".join(parts)
@@ -61,24 +61,24 @@ def _SuffixName(name, suffix):
 def _ShardName(name, number):
     """Add a shard number to the end of a target.
 
-  Arguments:
-    name: name of the target (foo#target)
-    number: shard number
-  Returns:
-    Target name with shard added (foo_1#target)
-  """
+    Arguments:
+      name: name of the target (foo#target)
+      number: shard number
+    Returns:
+      Target name with shard added (foo_1#target)
+    """
     return _SuffixName(name, str(number))
 
 
 def ShardTargets(target_list, target_dicts):
     """Shard some targets apart to work around the linkers limits.
 
-  Arguments:
-    target_list: List of target pairs: 'base/base.gyp:base'.
-    target_dicts: Dict of target properties keyed on target pair.
-  Returns:
-    Tuple of the new sharded versions of the inputs.
-  """
+    Arguments:
+      target_list: List of target pairs: 'base/base.gyp:base'.
+      target_dicts: Dict of target properties keyed on target pair.
+    Returns:
+      Tuple of the new sharded versions of the inputs.
+    """
     # Gather the targets to shard, and how many pieces.
     targets_to_shard = {}
     for t in target_dicts:
@@ -128,22 +128,22 @@ def ShardTargets(target_list, target_dicts):
 
 def _GetPdbPath(target_dict, config_name, vars):
     """Returns the path to the PDB file that will be generated by a given
-  configuration.
-
-  The lookup proceeds as follows:
-    - Look for an explicit path in the VCLinkerTool configuration block.
-    - Look for an 'msvs_large_pdb_path' variable.
-    - Use '<(PRODUCT_DIR)/<(product_name).(exe|dll).pdb' if 'product_name' is
-      specified.
-    - Use '<(PRODUCT_DIR)/<(target_name).(exe|dll).pdb'.
-
-  Arguments:
-    target_dict: The target dictionary to be searched.
-    config_name: The name of the configuration of interest.
-    vars: A dictionary of common GYP variables with generator-specific values.
-  Returns:
-    The path of the corresponding PDB file.
-  """
+    configuration.
+
+    The lookup proceeds as follows:
+      - Look for an explicit path in the VCLinkerTool configuration block.
+      - Look for an 'msvs_large_pdb_path' variable.
+      - Use '<(PRODUCT_DIR)/<(product_name).(exe|dll).pdb' if 'product_name' is
+        specified.
+      - Use '<(PRODUCT_DIR)/<(target_name).(exe|dll).pdb'.
+
+    Arguments:
+      target_dict: The target dictionary to be searched.
+      config_name: The name of the configuration of interest.
+      vars: A dictionary of common GYP variables with generator-specific values.
+    Returns:
+      The path of the corresponding PDB file.
+    """
     config = target_dict["configurations"][config_name]
     msvs = config.setdefault("msvs_settings", {})
 
@@ -168,16 +168,16 @@ def _GetPdbPath(target_dict, config_name, vars):
 def InsertLargePdbShims(target_list, target_dicts, vars):
     """Insert a shim target that forces the linker to use 4KB pagesize PDBs.
 
-  This is a workaround for targets with PDBs greater than 1GB in size, the
-  limit for the 1KB pagesize PDBs created by the linker by default.
+    This is a workaround for targets with PDBs greater than 1GB in size, the
+    limit for the 1KB pagesize PDBs created by the linker by default.
 
-  Arguments:
-    target_list: List of target pairs: 'base/base.gyp:base'.
-    target_dicts: Dict of target properties keyed on target pair.
-    vars: A dictionary of common GYP variables with generator-specific values.
-  Returns:
-    Tuple of the shimmed version of the inputs.
-  """
+    Arguments:
+      target_list: List of target pairs: 'base/base.gyp:base'.
+      target_dicts: Dict of target properties keyed on target pair.
+      vars: A dictionary of common GYP variables with generator-specific values.
+    Returns:
+      Tuple of the shimmed version of the inputs.
+    """
     # Determine which targets need shimming.
     targets_to_shim = []
     for t in target_dicts:
diff --git a/gyp/pylib/gyp/MSVSVersion.py b/gyp/pylib/gyp/MSVSVersion.py
index a34631b452..09baf44b2b 100644
--- a/gyp/pylib/gyp/MSVSVersion.py
+++ b/gyp/pylib/gyp/MSVSVersion.py
@@ -76,17 +76,17 @@ def Path(self):
         return self.path
 
     def ToolPath(self, tool):
-        """Returns the path to a given compiler tool. """
+        """Returns the path to a given compiler tool."""
         return os.path.normpath(os.path.join(self.path, "VC/bin", tool))
 
     def DefaultToolset(self):
         """Returns the msbuild toolset version that will be used in the absence
-    of a user override."""
+        of a user override."""
         return self.default_toolset
 
     def _SetupScriptInternal(self, target_arch):
         """Returns a command (with arguments) to be used to set up the
-    environment."""
+        environment."""
         assert target_arch in ("x86", "x64"), "target_arch not supported"
         # If WindowsSDKDir is set and SetEnv.Cmd exists then we are using the
         # depot_tools build tools and should run SetEnv.Cmd to set up the
@@ -154,16 +154,16 @@ def SetupScript(self, target_arch):
 def _RegistryQueryBase(sysdir, key, value):
     """Use reg.exe to read a particular key.
 
-  While ideally we might use the win32 module, we would like gyp to be
-  python neutral, so for instance cygwin python lacks this module.
+    While ideally we might use the win32 module, we would like gyp to be
+    python neutral, so for instance cygwin python lacks this module.
 
-  Arguments:
-    sysdir: The system subdirectory to attempt to launch reg.exe from.
-    key: The registry key to read from.
-    value: The particular value to read.
-  Return:
-    stdout from reg.exe, or None for failure.
-  """
+    Arguments:
+      sysdir: The system subdirectory to attempt to launch reg.exe from.
+      key: The registry key to read from.
+      value: The particular value to read.
+    Return:
+      stdout from reg.exe, or None for failure.
+    """
     # Skip if not on Windows or Python Win32 setup issue
     if sys.platform not in ("win32", "cygwin"):
         return None
@@ -184,20 +184,20 @@ def _RegistryQueryBase(sysdir, key, value):
 def _RegistryQuery(key, value=None):
     r"""Use reg.exe to read a particular key through _RegistryQueryBase.
 
-  First tries to launch from %WinDir%\Sysnative to avoid WoW64 redirection. If
-  that fails, it falls back to System32.  Sysnative is available on Vista and
-  up and available on Windows Server 2003 and XP through KB patch 942589. Note
-  that Sysnative will always fail if using 64-bit python due to it being a
-  virtual directory and System32 will work correctly in the first place.
+    First tries to launch from %WinDir%\Sysnative to avoid WoW64 redirection. If
+    that fails, it falls back to System32.  Sysnative is available on Vista and
+    up and available on Windows Server 2003 and XP through KB patch 942589. Note
+    that Sysnative will always fail if using 64-bit python due to it being a
+    virtual directory and System32 will work correctly in the first place.
 
-  KB 942589 - http://support.microsoft.com/kb/942589/en-us.
+    KB 942589 - http://support.microsoft.com/kb/942589/en-us.
 
-  Arguments:
-    key: The registry key.
-    value: The particular registry value to read (optional).
-  Return:
-    stdout from reg.exe, or None for failure.
-  """
+    Arguments:
+      key: The registry key.
+      value: The particular registry value to read (optional).
+    Return:
+      stdout from reg.exe, or None for failure.
+    """
     text = None
     try:
         text = _RegistryQueryBase("Sysnative", key, value)
@@ -212,14 +212,15 @@ def _RegistryQuery(key, value=None):
 def _RegistryGetValueUsingWinReg(key, value):
     """Use the _winreg module to obtain the value of a registry key.
 
-  Args:
-    key: The registry key.
-    value: The particular registry value to read.
-  Return:
-    contents of the registry key's value, or None on failure.  Throws
-    ImportError if winreg is unavailable.
-  """
+    Args:
+      key: The registry key.
+      value: The particular registry value to read.
+    Return:
+      contents of the registry key's value, or None on failure.  Throws
+      ImportError if winreg is unavailable.
+    """
     from winreg import HKEY_LOCAL_MACHINE, OpenKey, QueryValueEx  # noqa: PLC0415
+
     try:
         root, subkey = key.split("\\", 1)
         assert root == "HKLM"  # Only need HKLM for now.
@@ -232,17 +233,17 @@ def _RegistryGetValueUsingWinReg(key, value):
 def _RegistryGetValue(key, value):
     """Use _winreg or reg.exe to obtain the value of a registry key.
 
-  Using _winreg is preferable because it solves an issue on some corporate
-  environments where access to reg.exe is locked down. However, we still need
-  to fallback to reg.exe for the case where the _winreg module is not available
-  (for example in cygwin python).
-
-  Args:
-    key: The registry key.
-    value: The particular registry value to read.
-  Return:
-    contents of the registry key's value, or None on failure.
-  """
+    Using _winreg is preferable because it solves an issue on some corporate
+    environments where access to reg.exe is locked down. However, we still need
+    to fallback to reg.exe for the case where the _winreg module is not available
+    (for example in cygwin python).
+
+    Args:
+      key: The registry key.
+      value: The particular registry value to read.
+    Return:
+      contents of the registry key's value, or None on failure.
+    """
     try:
         return _RegistryGetValueUsingWinReg(key, value)
     except ImportError:
@@ -262,10 +263,10 @@ def _RegistryGetValue(key, value):
 def _CreateVersion(name, path, sdk_based=False):
     """Sets up MSVS project generation.
 
-  Setup is based off the GYP_MSVS_VERSION environment variable or whatever is
-  autodetected if GYP_MSVS_VERSION is not explicitly specified. If a version is
-  passed in that doesn't match a value in versions python will throw a error.
-  """
+    Setup is based off the GYP_MSVS_VERSION environment variable or whatever is
+    autodetected if GYP_MSVS_VERSION is not explicitly specified. If a version is
+    passed in that doesn't match a value in versions python will throw a error.
+    """
     if path:
         path = os.path.normpath(path)
     versions = {
@@ -435,22 +436,22 @@ def _ConvertToCygpath(path):
 def _DetectVisualStudioVersions(versions_to_check, force_express):
     """Collect the list of installed visual studio versions.
 
-  Returns:
-    A list of visual studio versions installed in descending order of
-    usage preference.
-    Base this on the registry and a quick check if devenv.exe exists.
-    Possibilities are:
-      2005(e) - Visual Studio 2005 (8)
-      2008(e) - Visual Studio 2008 (9)
-      2010(e) - Visual Studio 2010 (10)
-      2012(e) - Visual Studio 2012 (11)
-      2013(e) - Visual Studio 2013 (12)
-      2015    - Visual Studio 2015 (14)
-      2017    - Visual Studio 2017 (15)
-      2019    - Visual Studio 2019 (16)
-      2022    - Visual Studio 2022 (17)
-    Where (e) is e for express editions of MSVS and blank otherwise.
-  """
+    Returns:
+      A list of visual studio versions installed in descending order of
+      usage preference.
+      Base this on the registry and a quick check if devenv.exe exists.
+      Possibilities are:
+        2005(e) - Visual Studio 2005 (8)
+        2008(e) - Visual Studio 2008 (9)
+        2010(e) - Visual Studio 2010 (10)
+        2012(e) - Visual Studio 2012 (11)
+        2013(e) - Visual Studio 2013 (12)
+        2015    - Visual Studio 2015 (14)
+        2017    - Visual Studio 2017 (15)
+        2019    - Visual Studio 2019 (16)
+        2022    - Visual Studio 2022 (17)
+      Where (e) is e for express editions of MSVS and blank otherwise.
+    """
     version_to_year = {
         "8.0": "2005",
         "9.0": "2008",
@@ -527,11 +528,11 @@ def _DetectVisualStudioVersions(versions_to_check, force_express):
 def SelectVisualStudioVersion(version="auto", allow_fallback=True):
     """Select which version of Visual Studio projects to generate.
 
-  Arguments:
-    version: Hook to allow caller to force a particular version (vs auto).
-  Returns:
-    An object representing a visual studio project format version.
-  """
+    Arguments:
+      version: Hook to allow caller to force a particular version (vs auto).
+    Returns:
+      An object representing a visual studio project format version.
+    """
     # In auto mode, check environment variable for override.
     if version == "auto":
         version = os.environ.get("GYP_MSVS_VERSION", "auto")
diff --git a/gyp/pylib/gyp/__init__.py b/gyp/pylib/gyp/__init__.py
index efc15fc62f..3a70cf076c 100755
--- a/gyp/pylib/gyp/__init__.py
+++ b/gyp/pylib/gyp/__init__.py
@@ -25,19 +25,21 @@
 DEBUG_VARIABLES = "variables"
 DEBUG_INCLUDES = "includes"
 
+
 def EscapeForCString(string: bytes | str) -> str:
     if isinstance(string, str):
-        string = string.encode(encoding='utf8')
+        string = string.encode(encoding="utf8")
 
-    backslash_or_double_quote = {ord('\\'), ord('"')}
-    result = ''
+    backslash_or_double_quote = {ord("\\"), ord('"')}
+    result = ""
     for char in string:
         if char in backslash_or_double_quote or not 32 <= char < 127:
-            result += '\\%03o' % char
+            result += "\\%03o" % char
         else:
             result += chr(char)
     return result
 
+
 def DebugOutput(mode, message, *args):
     if "all" in gyp.debug or mode in gyp.debug:
         ctx = ("unknown", 0, "unknown")
@@ -76,11 +78,11 @@ def Load(
     circular_check=True,
 ):
     """
-  Loads one or more specified build files.
-  default_variables and includes will be copied before use.
-  Returns the generator for the specified format and the
-  data returned by loading the specified build files.
-  """
+    Loads one or more specified build files.
+    default_variables and includes will be copied before use.
+    Returns the generator for the specified format and the
+    data returned by loading the specified build files.
+    """
     if params is None:
         params = {}
 
@@ -114,7 +116,7 @@ def Load(
     # These parameters are passed in order (as opposed to by key)
     # because ActivePython cannot handle key parameters to __import__.
     generator = __import__(generator_name, globals(), locals(), generator_name)
-    for (key, val) in generator.generator_default_variables.items():
+    for key, val in generator.generator_default_variables.items():
         default_variables.setdefault(key, val)
 
     output_dir = params["options"].generator_output or params["options"].toplevel_dir
@@ -184,10 +186,10 @@ def Load(
 
 def NameValueListToDict(name_value_list):
     """
-  Takes an array of strings of the form 'NAME=VALUE' and creates a dictionary
-  of the pairs.  If a string is simply NAME, then the value in the dictionary
-  is set to True.  If VALUE can be converted to an integer, it is.
-  """
+    Takes an array of strings of the form 'NAME=VALUE' and creates a dictionary
+    of the pairs.  If a string is simply NAME, then the value in the dictionary
+    is set to True.  If VALUE can be converted to an integer, it is.
+    """
     result = {}
     for item in name_value_list:
         tokens = item.split("=", 1)
@@ -220,13 +222,13 @@ def FormatOpt(opt, value):
 def RegenerateAppendFlag(flag, values, predicate, env_name, options):
     """Regenerate a list of command line flags, for an option of action='append'.
 
-  The |env_name|, if given, is checked in the environment and used to generate
-  an initial list of options, then the options that were specified on the
-  command line (given in |values|) are appended.  This matches the handling of
-  environment variables and command line flags where command line flags override
-  the environment, while not requiring the environment to be set when the flags
-  are used again.
-  """
+    The |env_name|, if given, is checked in the environment and used to generate
+    an initial list of options, then the options that were specified on the
+    command line (given in |values|) are appended.  This matches the handling of
+    environment variables and command line flags where command line flags override
+    the environment, while not requiring the environment to be set when the flags
+    are used again.
+    """
     flags = []
     if options.use_environment and env_name:
         for flag_value in ShlexEnv(env_name):
@@ -242,14 +244,14 @@ def RegenerateAppendFlag(flag, values, predicate, env_name, options):
 
 def RegenerateFlags(options):
     """Given a parsed options object, and taking the environment variables into
-  account, returns a list of flags that should regenerate an equivalent options
-  object (even in the absence of the environment variables.)
+    account, returns a list of flags that should regenerate an equivalent options
+    object (even in the absence of the environment variables.)
 
-  Any path options will be normalized relative to depth.
+    Any path options will be normalized relative to depth.
 
-  The format flag is not included, as it is assumed the calling generator will
-  set that as appropriate.
-  """
+    The format flag is not included, as it is assumed the calling generator will
+    set that as appropriate.
+    """
 
     def FixPath(path):
         path = gyp.common.FixIfRelativePath(path, options.depth)
@@ -307,15 +309,15 @@ def __init__(self, usage):
     def add_argument(self, *args, **kw):
         """Add an option to the parser.
 
-    This accepts the same arguments as ArgumentParser.add_argument, plus the
-    following:
-      regenerate: can be set to False to prevent this option from being included
-                  in regeneration.
-      env_name: name of environment variable that additional values for this
-                option come from.
-      type: adds type='path', to tell the regenerator that the values of
-            this option need to be made relative to options.depth
-    """
+        This accepts the same arguments as ArgumentParser.add_argument, plus the
+        following:
+          regenerate: can be set to False to prevent this option from being included
+                      in regeneration.
+          env_name: name of environment variable that additional values for this
+                    option come from.
+          type: adds type='path', to tell the regenerator that the values of
+                this option need to be made relative to options.depth
+        """
         env_name = kw.pop("env_name", None)
         if "dest" in kw and kw.pop("regenerate", True):
             dest = kw["dest"]
@@ -343,7 +345,7 @@ def parse_args(self, *args):
 
 def gyp_main(args):
     my_name = os.path.basename(sys.argv[0])
-    usage = "usage: %(prog)s [options ...] [build_file ...]"
+    usage = "%(prog)s [options ...] [build_file ...]"
 
     parser = RegeneratableOptionParser(usage=usage.replace("%s", "%(prog)s"))
     parser.add_argument(
@@ -490,6 +492,7 @@ def gyp_main(args):
     options, build_files_arg = parser.parse_args(args)
     if options.version:
         import pkg_resources  # noqa: PLC0415
+
         print(f"v{pkg_resources.get_distribution('gyp-next').version}")
         return 0
     build_files = build_files_arg
diff --git a/gyp/pylib/gyp/common.py b/gyp/pylib/gyp/common.py
index 9431c432c8..223ce47b00 100644
--- a/gyp/pylib/gyp/common.py
+++ b/gyp/pylib/gyp/common.py
@@ -31,9 +31,8 @@ def __call__(self, *args):
 
 class GypError(Exception):
     """Error class representing an error, which is to be presented
-  to the user.  The main entry point will catch and display this.
-  """
-
+    to the user.  The main entry point will catch and display this.
+    """
 
 
 def ExceptionAppend(e, msg):
@@ -48,9 +47,9 @@ def ExceptionAppend(e, msg):
 
 def FindQualifiedTargets(target, qualified_list):
     """
-  Given a list of qualified targets, return the qualified targets for the
-  specified |target|.
-  """
+    Given a list of qualified targets, return the qualified targets for the
+    specified |target|.
+    """
     return [t for t in qualified_list if ParseQualifiedTarget(t)[1] == target]
 
 
@@ -115,7 +114,7 @@ def BuildFile(fully_qualified_target):
 
 def GetEnvironFallback(var_list, default):
     """Look up a key in the environment, with fallback to secondary keys
-  and finally falling back to a default value."""
+    and finally falling back to a default value."""
     for var in var_list:
         if var in os.environ:
             return os.environ[var]
@@ -178,11 +177,11 @@ def RelativePath(path, relative_to, follow_path_symlink=True):
 @memoize
 def InvertRelativePath(path, toplevel_dir=None):
     """Given a path like foo/bar that is relative to toplevel_dir, return
-  the inverse relative path back to the toplevel_dir.
+    the inverse relative path back to the toplevel_dir.
 
-  E.g. os.path.normpath(os.path.join(path, InvertRelativePath(path)))
-  should always produce the empty string, unless the path contains symlinks.
-  """
+    E.g. os.path.normpath(os.path.join(path, InvertRelativePath(path)))
+    should always produce the empty string, unless the path contains symlinks.
+    """
     if not path:
         return path
     toplevel_dir = "." if toplevel_dir is None else toplevel_dir
@@ -262,12 +261,12 @@ def UnrelativePath(path, relative_to):
 def EncodePOSIXShellArgument(argument):
     """Encodes |argument| suitably for consumption by POSIX shells.
 
-  argument may be quoted and escaped as necessary to ensure that POSIX shells
-  treat the returned value as a literal representing the argument passed to
-  this function.  Parameter (variable) expansions beginning with $ are allowed
-  to remain intact without escaping the $, to allow the argument to contain
-  references to variables to be expanded by the shell.
-  """
+    argument may be quoted and escaped as necessary to ensure that POSIX shells
+    treat the returned value as a literal representing the argument passed to
+    this function.  Parameter (variable) expansions beginning with $ are allowed
+    to remain intact without escaping the $, to allow the argument to contain
+    references to variables to be expanded by the shell.
+    """
 
     if not isinstance(argument, str):
         argument = str(argument)
@@ -282,9 +281,9 @@ def EncodePOSIXShellArgument(argument):
 def EncodePOSIXShellList(list):
     """Encodes |list| suitably for consumption by POSIX shells.
 
-  Returns EncodePOSIXShellArgument for each item in list, and joins them
-  together using the space character as an argument separator.
-  """
+    Returns EncodePOSIXShellArgument for each item in list, and joins them
+    together using the space character as an argument separator.
+    """
 
     encoded_arguments = []
     for argument in list:
@@ -312,14 +311,12 @@ def DeepDependencyTargets(target_dicts, roots):
 
 
 def BuildFileTargets(target_list, build_file):
-    """From a target_list, returns the subset from the specified build_file.
-  """
+    """From a target_list, returns the subset from the specified build_file."""
     return [p for p in target_list if BuildFile(p) == build_file]
 
 
 def AllTargets(target_list, target_dicts, build_file):
-    """Returns all targets (direct and dependencies) for the specified build_file.
-  """
+    """Returns all targets (direct and dependencies) for the specified build_file."""
     bftargets = BuildFileTargets(target_list, build_file)
     deptargets = DeepDependencyTargets(target_dicts, bftargets)
     return bftargets + deptargets
@@ -328,12 +325,12 @@ def AllTargets(target_list, target_dicts, build_file):
 def WriteOnDiff(filename):
     """Write to a file only if the new contents differ.
 
-  Arguments:
-    filename: name of the file to potentially write to.
-  Returns:
-    A file like object which will write to temporary file and only overwrite
-    the target if it differs (on close).
-  """
+    Arguments:
+      filename: name of the file to potentially write to.
+    Returns:
+      A file like object which will write to temporary file and only overwrite
+      the target if it differs (on close).
+    """
 
     class Writer:
         """Wrapper around file which only covers the target if it differs."""
@@ -421,6 +418,7 @@ def EnsureDirExists(path):
     except OSError:
         pass
 
+
 def GetCompilerPredefines():  # -> dict
     cmd = []
     defines = {}
@@ -448,15 +446,14 @@ def replace_sep(s):
         try:
             os.close(fd)
             stdout = subprocess.run(
-                real_cmd, shell=True,
-                capture_output=True, check=True
+                real_cmd, shell=True, capture_output=True, check=True
             ).stdout
         except subprocess.CalledProcessError as e:
             print(
                 "Warning: failed to get compiler predefines\n"
                 "cmd: %s\n"
                 "status: %d" % (e.cmd, e.returncode),
-                file=sys.stderr
+                file=sys.stderr,
             )
             return defines
         finally:
@@ -466,15 +463,14 @@ def replace_sep(s):
         real_cmd = [*cmd, "-dM", "-E", "-x", "c", input]
         try:
             stdout = subprocess.run(
-                real_cmd, shell=False,
-                capture_output=True, check=True
+                real_cmd, shell=False, capture_output=True, check=True
             ).stdout
         except subprocess.CalledProcessError as e:
             print(
                 "Warning: failed to get compiler predefines\n"
                 "cmd: %s\n"
                 "status: %d" % (e.cmd, e.returncode),
-                file=sys.stderr
+                file=sys.stderr,
             )
             return defines
 
@@ -485,6 +481,7 @@ def replace_sep(s):
             defines[key] = " ".join(value)
     return defines
 
+
 def GetFlavorByPlatform():
     """Returns |params.flavor| if it's set, the system's default flavor else."""
     flavors = {
@@ -512,6 +509,7 @@ def GetFlavorByPlatform():
 
     return "linux"
 
+
 def GetFlavor(params):
     if "flavor" in params:
         return params["flavor"]
@@ -527,7 +525,7 @@ def GetFlavor(params):
 
 def CopyTool(flavor, out_path, generator_flags={}):
     """Finds (flock|mac|win)_tool.gyp in the gyp directory and copies it
-  to |out_path|."""
+    to |out_path|."""
     # aix and solaris just need flock emulation. mac and win use more complicated
     # support scripts.
     prefix = {
@@ -662,24 +660,24 @@ def __str__(self):
 def TopologicallySorted(graph, get_edges):
     r"""Topologically sort based on a user provided edge definition.
 
-  Args:
-    graph: A list of node names.
-    get_edges: A function mapping from node name to a hashable collection
-               of node names which this node has outgoing edges to.
-  Returns:
-    A list containing all of the node in graph in topological order.
-    It is assumed that calling get_edges once for each node and caching is
-    cheaper than repeatedly calling get_edges.
-  Raises:
-    CycleError in the event of a cycle.
-  Example:
-    graph = {'a': '$(b) $(c)', 'b': 'hi', 'c': '$(b)'}
-    def GetEdges(node):
-      return re.findall(r'\$\(([^))]\)', graph[node])
-    print TopologicallySorted(graph.keys(), GetEdges)
-    ==>
-    ['a', 'c', b']
-  """
+    Args:
+      graph: A list of node names.
+      get_edges: A function mapping from node name to a hashable collection
+                 of node names which this node has outgoing edges to.
+    Returns:
+      A list containing all of the node in graph in topological order.
+      It is assumed that calling get_edges once for each node and caching is
+      cheaper than repeatedly calling get_edges.
+    Raises:
+      CycleError in the event of a cycle.
+    Example:
+      graph = {'a': '$(b) $(c)', 'b': 'hi', 'c': '$(b)'}
+      def GetEdges(node):
+        return re.findall(r'\$\(([^))]\)', graph[node])
+      print TopologicallySorted(graph.keys(), GetEdges)
+      ==>
+      ['a', 'c', b']
+    """
     get_edges = memoize(get_edges)
     visited = set()
     visiting = set()
diff --git a/gyp/pylib/gyp/common_test.py b/gyp/pylib/gyp/common_test.py
index d25e621ab5..b5988816c0 100755
--- a/gyp/pylib/gyp/common_test.py
+++ b/gyp/pylib/gyp/common_test.py
@@ -28,8 +28,12 @@ def test_Valid(self):
         def GetEdge(node):
             return tuple(graph[node])
 
-        assert gyp.common.TopologicallySorted(
-            graph.keys(), GetEdge) == ["a", "c", "d", "b"]
+        assert gyp.common.TopologicallySorted(graph.keys(), GetEdge) == [
+            "a",
+            "c",
+            "d",
+            "b",
+        ]
 
     def test_Cycle(self):
         """Test that an exception is thrown on a cyclic graph."""
@@ -97,10 +101,7 @@ def mock_run(env, defines_stdout, expected_cmd, throws=False):
                 if throws:
                     mock_run.side_effect = subprocess.CalledProcessError(
                         returncode=1,
-                        cmd=[
-                            *expected_cmd,
-                            "-dM", "-E", "-x", "c", expected_input
-                        ]
+                        cmd=[*expected_cmd, "-dM", "-E", "-x", "c", expected_input],
                     )
                 else:
                     mock_process = MagicMock()
@@ -115,75 +116,71 @@ def mock_run(env, defines_stdout, expected_cmd, throws=False):
                     flavor = gyp.common.GetFlavor({})
                 if env.get("CC_target") or env.get("CC"):
                     mock_run.assert_called_with(
-                        [
-                            *expected_cmd,
-                            "-dM", "-E", "-x", "c", expected_input
-                        ],
+                        [*expected_cmd, "-dM", "-E", "-x", "c", expected_input],
                         shell=sys.platform == "win32",
-                        capture_output=True, check=True)
+                        capture_output=True,
+                        check=True,
+                    )
                 return [defines, flavor]
 
-        [defines0, _] = mock_run({ "CC": "cl.exe" }, "", ["cl.exe"], True)
+        [defines0, _] = mock_run({"CC": "cl.exe"}, "", ["cl.exe"], True)
         assert defines0 == {}
 
         [defines1, _] = mock_run({}, "", [])
         assert defines1 == {}
 
         [defines2, flavor2] = mock_run(
-            { "CC_target": "/opt/wasi-sdk/bin/clang" },
+            {"CC_target": "/opt/wasi-sdk/bin/clang"},
             "#define __wasm__ 1\n#define __wasi__ 1\n",
-            ["/opt/wasi-sdk/bin/clang"]
+            ["/opt/wasi-sdk/bin/clang"],
         )
-        assert defines2 == { "__wasm__": "1", "__wasi__": "1" }
+        assert defines2 == {"__wasm__": "1", "__wasi__": "1"}
         assert flavor2 == "wasi"
 
         [defines3, flavor3] = mock_run(
-            { "CC_target": "/opt/wasi-sdk/bin/clang --target=wasm32" },
+            {"CC_target": "/opt/wasi-sdk/bin/clang --target=wasm32"},
             "#define __wasm__ 1\n",
-            ["/opt/wasi-sdk/bin/clang", "--target=wasm32"]
+            ["/opt/wasi-sdk/bin/clang", "--target=wasm32"],
         )
-        assert defines3 == { "__wasm__": "1" }
+        assert defines3 == {"__wasm__": "1"}
         assert flavor3 == "wasm"
 
         [defines4, flavor4] = mock_run(
-            { "CC_target": "/emsdk/upstream/emscripten/emcc" },
+            {"CC_target": "/emsdk/upstream/emscripten/emcc"},
             "#define __EMSCRIPTEN__ 1\n",
-            ["/emsdk/upstream/emscripten/emcc"]
+            ["/emsdk/upstream/emscripten/emcc"],
         )
-        assert defines4 == { "__EMSCRIPTEN__": "1" }
+        assert defines4 == {"__EMSCRIPTEN__": "1"}
         assert flavor4 == "emscripten"
 
         # Test path which include white space
         [defines5, flavor5] = mock_run(
             {
-                "CC_target": "\"/Users/Toyo Li/wasi-sdk/bin/clang\" -O3",
-                "CFLAGS": "--target=wasm32-wasi-threads -pthread"
+                "CC_target": '"/Users/Toyo Li/wasi-sdk/bin/clang" -O3',
+                "CFLAGS": "--target=wasm32-wasi-threads -pthread",
             },
             "#define __wasm__ 1\n#define __wasi__ 1\n#define _REENTRANT 1\n",
             [
                 "/Users/Toyo Li/wasi-sdk/bin/clang",
                 "-O3",
                 "--target=wasm32-wasi-threads",
-                "-pthread"
-            ]
+                "-pthread",
+            ],
         )
-        assert defines5 == {
-            "__wasm__": "1",
-            "__wasi__": "1",
-            "_REENTRANT": "1"
-        }
+        assert defines5 == {"__wasm__": "1", "__wasi__": "1", "_REENTRANT": "1"}
         assert flavor5 == "wasi"
 
         original_sep = os.sep
         os.sep = "\\"
         [defines6, flavor6] = mock_run(
-            { "CC_target": "\"C:\\Program Files\\wasi-sdk\\clang.exe\"" },
+            {"CC_target": '"C:\\Program Files\\wasi-sdk\\clang.exe"'},
             "#define __wasm__ 1\n#define __wasi__ 1\n",
-            ["C:/Program Files/wasi-sdk/clang.exe"]
+            ["C:/Program Files/wasi-sdk/clang.exe"],
         )
         os.sep = original_sep
-        assert defines6 == { "__wasm__": "1", "__wasi__": "1" }
+        assert defines6 == {"__wasm__": "1", "__wasi__": "1"}
         assert flavor6 == "wasi"
 
+
 if __name__ == "__main__":
     unittest.main()
diff --git a/gyp/pylib/gyp/easy_xml.py b/gyp/pylib/gyp/easy_xml.py
index e4d2f82b68..a5d95153ec 100644
--- a/gyp/pylib/gyp/easy_xml.py
+++ b/gyp/pylib/gyp/easy_xml.py
@@ -10,43 +10,43 @@
 
 
 def XmlToString(content, encoding="utf-8", pretty=False):
-    """ Writes the XML content to disk, touching the file only if it has changed.
-
-  Visual Studio files have a lot of pre-defined structures.  This function makes
-  it easy to represent these structures as Python data structures, instead of
-  having to create a lot of function calls.
-
-  Each XML element of the content is represented as a list composed of:
-  1. The name of the element, a string,
-  2. The attributes of the element, a dictionary (optional), and
-  3+. The content of the element, if any.  Strings are simple text nodes and
-      lists are child elements.
-
-  Example 1:
-      
-  becomes
-      ['test']
-
-  Example 2:
-      
-         This is
-         it!
-      
-
-  becomes
-      ['myelement', {'a':'value1', 'b':'value2'},
-         ['childtype', 'This is'],
-         ['childtype', 'it!'],
-      ]
-
-  Args:
-    content:  The structured content to be converted.
-    encoding: The encoding to report on the first XML line.
-    pretty: True if we want pretty printing with indents and new lines.
-
-  Returns:
-    The XML content as a string.
-  """
+    """Writes the XML content to disk, touching the file only if it has changed.
+
+    Visual Studio files have a lot of pre-defined structures.  This function makes
+    it easy to represent these structures as Python data structures, instead of
+    having to create a lot of function calls.
+
+    Each XML element of the content is represented as a list composed of:
+    1. The name of the element, a string,
+    2. The attributes of the element, a dictionary (optional), and
+    3+. The content of the element, if any.  Strings are simple text nodes and
+        lists are child elements.
+
+    Example 1:
+        
+    becomes
+        ['test']
+
+    Example 2:
+        
+           This is
+           it!
+        
+
+    becomes
+        ['myelement', {'a':'value1', 'b':'value2'},
+           ['childtype', 'This is'],
+           ['childtype', 'it!'],
+        ]
+
+    Args:
+      content:  The structured content to be converted.
+      encoding: The encoding to report on the first XML line.
+      pretty: True if we want pretty printing with indents and new lines.
+
+    Returns:
+      The XML content as a string.
+    """
     # We create a huge list of all the elements of the file.
     xml_parts = ['' % encoding]
     if pretty:
@@ -58,14 +58,14 @@ def XmlToString(content, encoding="utf-8", pretty=False):
 
 
 def _ConstructContentList(xml_parts, specification, pretty, level=0):
-    """ Appends the XML parts corresponding to the specification.
-
-  Args:
-    xml_parts: A list of XML parts to be appended to.
-    specification:  The specification of the element.  See EasyXml docs.
-    pretty: True if we want pretty printing with indents and new lines.
-    level: Indentation level.
-  """
+    """Appends the XML parts corresponding to the specification.
+
+    Args:
+      xml_parts: A list of XML parts to be appended to.
+      specification:  The specification of the element.  See EasyXml docs.
+      pretty: True if we want pretty printing with indents and new lines.
+      level: Indentation level.
+    """
     # The first item in a specification is the name of the element.
     if pretty:
         indentation = "  " * level
@@ -107,16 +107,17 @@ def _ConstructContentList(xml_parts, specification, pretty, level=0):
         xml_parts.append("/>%s" % new_line)
 
 
-def WriteXmlIfChanged(content, path, encoding="utf-8", pretty=False,
-                      win32=(sys.platform == "win32")):
-    """ Writes the XML content to disk, touching the file only if it has changed.
+def WriteXmlIfChanged(
+    content, path, encoding="utf-8", pretty=False, win32=(sys.platform == "win32")
+):
+    """Writes the XML content to disk, touching the file only if it has changed.
 
-  Args:
-    content:  The structured content to be written.
-    path: Location of the file.
-    encoding: The encoding to report on the first line of the XML file.
-    pretty: True if we want pretty printing with indents and new lines.
-  """
+    Args:
+      content:  The structured content to be written.
+      path: Location of the file.
+      encoding: The encoding to report on the first line of the XML file.
+      pretty: True if we want pretty printing with indents and new lines.
+    """
     xml_string = XmlToString(content, encoding, pretty)
     if win32 and os.linesep != "\r\n":
         xml_string = xml_string.replace("\n", "\r\n")
@@ -157,7 +158,7 @@ def WriteXmlIfChanged(content, path, encoding="utf-8", pretty=False,
 
 
 def _XmlEscape(value, attr=False):
-    """ Escape a string for inclusion in XML."""
+    """Escape a string for inclusion in XML."""
 
     def replace(match):
         m = match.string[match.start() : match.end()]
diff --git a/gyp/pylib/gyp/easy_xml_test.py b/gyp/pylib/gyp/easy_xml_test.py
index bb97b802c5..29f5dad5a6 100755
--- a/gyp/pylib/gyp/easy_xml_test.py
+++ b/gyp/pylib/gyp/easy_xml_test.py
@@ -4,7 +4,7 @@
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
-""" Unit tests for the easy_xml.py file. """
+"""Unit tests for the easy_xml.py file."""
 
 import unittest
 from io import StringIO
diff --git a/gyp/pylib/gyp/generator/analyzer.py b/gyp/pylib/gyp/generator/analyzer.py
index cb18742cd8..420c4e49eb 100644
--- a/gyp/pylib/gyp/generator/analyzer.py
+++ b/gyp/pylib/gyp/generator/analyzer.py
@@ -62,7 +62,6 @@
 then the "all" target includes "b1" and "b2".
 """
 
-
 import json
 import os
 import posixpath
@@ -130,8 +129,8 @@ def _ToGypPath(path):
 
 def _ResolveParent(path, base_path_components):
     """Resolves |path|, which starts with at least one '../'. Returns an empty
-  string if the path shouldn't be considered. See _AddSources() for a
-  description of |base_path_components|."""
+    string if the path shouldn't be considered. See _AddSources() for a
+    description of |base_path_components|."""
     depth = 0
     while path.startswith("../"):
         depth += 1
@@ -151,11 +150,11 @@ def _ResolveParent(path, base_path_components):
 
 def _AddSources(sources, base_path, base_path_components, result):
     """Extracts valid sources from |sources| and adds them to |result|. Each
-  source file is relative to |base_path|, but may contain '..'. To make
-  resolving '..' easier |base_path_components| contains each of the
-  directories in |base_path|. Additionally each source may contain variables.
-  Such sources are ignored as it is assumed dependencies on them are expressed
-  and tracked in some other means."""
+    source file is relative to |base_path|, but may contain '..'. To make
+    resolving '..' easier |base_path_components| contains each of the
+    directories in |base_path|. Additionally each source may contain variables.
+    Such sources are ignored as it is assumed dependencies on them are expressed
+    and tracked in some other means."""
     # NOTE: gyp paths are always posix style.
     for source in sources:
         if not len(source) or source.startswith(("!!!", "$")):
@@ -218,23 +217,23 @@ def _ExtractSources(target, target_dict, toplevel_dir):
 
 class Target:
     """Holds information about a particular target:
-  deps: set of Targets this Target depends upon. This is not recursive, only the
-    direct dependent Targets.
-  match_status: one of the MatchStatus values.
-  back_deps: set of Targets that have a dependency on this Target.
-  visited: used during iteration to indicate whether we've visited this target.
-    This is used for two iterations, once in building the set of Targets and
-    again in _GetBuildTargets().
-  name: fully qualified name of the target.
-  requires_build: True if the target type is such that it needs to be built.
-    See _DoesTargetTypeRequireBuild for details.
-  added_to_compile_targets: used when determining if the target was added to the
-    set of targets that needs to be built.
-  in_roots: true if this target is a descendant of one of the root nodes.
-  is_executable: true if the type of target is executable.
-  is_static_library: true if the type of target is static_library.
-  is_or_has_linked_ancestor: true if the target does a link (eg executable), or
-    if there is a target in back_deps that does a link."""
+    deps: set of Targets this Target depends upon. This is not recursive, only the
+      direct dependent Targets.
+    match_status: one of the MatchStatus values.
+    back_deps: set of Targets that have a dependency on this Target.
+    visited: used during iteration to indicate whether we've visited this target.
+      This is used for two iterations, once in building the set of Targets and
+      again in _GetBuildTargets().
+    name: fully qualified name of the target.
+    requires_build: True if the target type is such that it needs to be built.
+      See _DoesTargetTypeRequireBuild for details.
+    added_to_compile_targets: used when determining if the target was added to the
+      set of targets that needs to be built.
+    in_roots: true if this target is a descendant of one of the root nodes.
+    is_executable: true if the type of target is executable.
+    is_static_library: true if the type of target is static_library.
+    is_or_has_linked_ancestor: true if the target does a link (eg executable), or
+      if there is a target in back_deps that does a link."""
 
     def __init__(self, name):
         self.deps = set()
@@ -254,8 +253,8 @@ def __init__(self, name):
 
 class Config:
     """Details what we're looking for
-  files: set of files to search for
-  targets: see file description for details."""
+    files: set of files to search for
+    targets: see file description for details."""
 
     def __init__(self):
         self.files = []
@@ -265,7 +264,7 @@ def __init__(self):
 
     def Init(self, params):
         """Initializes Config. This is a separate method as it raises an exception
-    if there is a parse error."""
+        if there is a parse error."""
         generator_flags = params.get("generator_flags", {})
         config_path = generator_flags.get("config_path", None)
         if not config_path:
@@ -289,8 +288,8 @@ def Init(self, params):
 
 def _WasBuildFileModified(build_file, data, files, toplevel_dir):
     """Returns true if the build file |build_file| is either in |files| or
-  one of the files included by |build_file| is in |files|. |toplevel_dir| is
-  the root of the source tree."""
+    one of the files included by |build_file| is in |files|. |toplevel_dir| is
+    the root of the source tree."""
     if _ToLocalPath(toplevel_dir, _ToGypPath(build_file)) in files:
         if debug:
             print("gyp file modified", build_file)
@@ -319,8 +318,8 @@ def _WasBuildFileModified(build_file, data, files, toplevel_dir):
 
 def _GetOrCreateTargetByName(targets, target_name):
     """Creates or returns the Target at targets[target_name]. If there is no
-  Target for |target_name| one is created. Returns a tuple of whether a new
-  Target was created and the Target."""
+    Target for |target_name| one is created. Returns a tuple of whether a new
+    Target was created and the Target."""
     if target_name in targets:
         return False, targets[target_name]
     target = Target(target_name)
@@ -340,13 +339,13 @@ def _DoesTargetTypeRequireBuild(target_dict):
 
 def _GenerateTargets(data, target_list, target_dicts, toplevel_dir, files, build_files):
     """Returns a tuple of the following:
-  . A dictionary mapping from fully qualified name to Target.
-  . A list of the targets that have a source file in |files|.
-  . Targets that constitute the 'all' target. See description at top of file
-    for details on the 'all' target.
-  This sets the |match_status| of the targets that contain any of the source
-  files in |files| to MATCH_STATUS_MATCHES.
-  |toplevel_dir| is the root of the source tree."""
+    . A dictionary mapping from fully qualified name to Target.
+    . A list of the targets that have a source file in |files|.
+    . Targets that constitute the 'all' target. See description at top of file
+      for details on the 'all' target.
+    This sets the |match_status| of the targets that contain any of the source
+    files in |files| to MATCH_STATUS_MATCHES.
+    |toplevel_dir| is the root of the source tree."""
     # Maps from target name to Target.
     name_to_target = {}
 
@@ -379,9 +378,10 @@ def _GenerateTargets(data, target_list, target_dicts, toplevel_dir, files, build
         target_type = target_dicts[target_name]["type"]
         target.is_executable = target_type == "executable"
         target.is_static_library = target_type == "static_library"
-        target.is_or_has_linked_ancestor = (
-            target_type in {"executable", "shared_library"}
-        )
+        target.is_or_has_linked_ancestor = target_type in {
+            "executable",
+            "shared_library",
+        }
 
         build_file = gyp.common.ParseQualifiedTarget(target_name)[0]
         if build_file not in build_file_in_files:
@@ -427,9 +427,9 @@ def _GenerateTargets(data, target_list, target_dicts, toplevel_dir, files, build
 
 def _GetUnqualifiedToTargetMapping(all_targets, to_find):
     """Returns a tuple of the following:
-  . mapping (dictionary) from unqualified name to Target for all the
-    Targets in |to_find|.
-  . any target names not found. If this is empty all targets were found."""
+    . mapping (dictionary) from unqualified name to Target for all the
+      Targets in |to_find|.
+    . any target names not found. If this is empty all targets were found."""
     result = {}
     if not to_find:
         return {}, []
@@ -446,15 +446,15 @@ def _GetUnqualifiedToTargetMapping(all_targets, to_find):
 
 def _DoesTargetDependOnMatchingTargets(target):
     """Returns true if |target| or any of its dependencies is one of the
-  targets containing the files supplied as input to analyzer. This updates
-  |matches| of the Targets as it recurses.
-  target: the Target to look for."""
+    targets containing the files supplied as input to analyzer. This updates
+    |matches| of the Targets as it recurses.
+    target: the Target to look for."""
     if target.match_status == MATCH_STATUS_DOESNT_MATCH:
         return False
-    if (
-        target.match_status in {MATCH_STATUS_MATCHES,
-                                MATCH_STATUS_MATCHES_BY_DEPENDENCY}
-    ):
+    if target.match_status in {
+        MATCH_STATUS_MATCHES,
+        MATCH_STATUS_MATCHES_BY_DEPENDENCY,
+    }:
         return True
     for dep in target.deps:
         if _DoesTargetDependOnMatchingTargets(dep):
@@ -467,9 +467,9 @@ def _DoesTargetDependOnMatchingTargets(target):
 
 def _GetTargetsDependingOnMatchingTargets(possible_targets):
     """Returns the list of Targets in |possible_targets| that depend (either
-  directly on indirectly) on at least one of the targets containing the files
-  supplied as input to analyzer.
-  possible_targets: targets to search from."""
+    directly on indirectly) on at least one of the targets containing the files
+    supplied as input to analyzer.
+    possible_targets: targets to search from."""
     found = []
     print("Targets that matched by dependency:")
     for target in possible_targets:
@@ -480,11 +480,11 @@ def _GetTargetsDependingOnMatchingTargets(possible_targets):
 
 def _AddCompileTargets(target, roots, add_if_no_ancestor, result):
     """Recurses through all targets that depend on |target|, adding all targets
-  that need to be built (and are in |roots|) to |result|.
-  roots: set of root targets.
-  add_if_no_ancestor: If true and there are no ancestors of |target| then add
-  |target| to |result|. |target| must still be in |roots|.
-  result: targets that need to be built are added here."""
+    that need to be built (and are in |roots|) to |result|.
+    roots: set of root targets.
+    add_if_no_ancestor: If true and there are no ancestors of |target| then add
+    |target| to |result|. |target| must still be in |roots|.
+    result: targets that need to be built are added here."""
     if target.visited:
         return
 
@@ -537,8 +537,8 @@ def _AddCompileTargets(target, roots, add_if_no_ancestor, result):
 
 def _GetCompileTargets(matching_targets, supplied_targets):
     """Returns the set of Targets that require a build.
-  matching_targets: targets that changed and need to be built.
-  supplied_targets: set of targets supplied to analyzer to search from."""
+    matching_targets: targets that changed and need to be built.
+    supplied_targets: set of targets supplied to analyzer to search from."""
     result = set()
     for target in matching_targets:
         print("finding compile targets for match", target.name)
@@ -592,7 +592,7 @@ def _WriteOutput(params, **values):
 
 def _WasGypIncludeFileModified(params, files):
     """Returns true if one of the files in |files| is in the set of included
-  files."""
+    files."""
     if params["options"].includes:
         for include in params["options"].includes:
             if _ToGypPath(os.path.normpath(include)) in files:
@@ -608,7 +608,7 @@ def _NamesNotIn(names, mapping):
 
 def _LookupTargets(names, mapping):
     """Returns a list of the mapping[name] for each value in |names| that is in
-  |mapping|."""
+    |mapping|."""
     return [mapping[name] for name in names if name in mapping]
 
 
diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py
index cc02298e4e..cfc0681f6b 100644
--- a/gyp/pylib/gyp/generator/android.py
+++ b/gyp/pylib/gyp/generator/android.py
@@ -177,9 +177,7 @@ def Write(
             self.WriteLn("LOCAL_IS_HOST_MODULE := true")
             self.WriteLn("LOCAL_MULTILIB := $(GYP_HOST_MULTILIB)")
         elif sdk_version > 0:
-            self.WriteLn(
-                "LOCAL_MODULE_TARGET_ARCH := $(TARGET_$(GYP_VAR_PREFIX)ARCH)"
-            )
+            self.WriteLn("LOCAL_MODULE_TARGET_ARCH := $(TARGET_$(GYP_VAR_PREFIX)ARCH)")
             self.WriteLn("LOCAL_SDK_VERSION := %s" % sdk_version)
 
         # Grab output directories; needed for Actions and Rules.
@@ -588,7 +586,8 @@ def WriteSources(self, spec, configs, extra_sources):
         local_files = []
         for source in sources:
             (root, ext) = os.path.splitext(source)
-            if ("$(gyp_shared_intermediate_dir)" in source
+            if (
+                "$(gyp_shared_intermediate_dir)" in source
                 or "$(gyp_intermediate_dir)" in source
                 or (IsCPPExtension(ext) and ext != local_cpp_extension)
             ):
@@ -734,8 +733,7 @@ def ComputeOutput(self, spec):
         elif self.toolset == "host":
             path = (
                 "$(call intermediates-dir-for,%s,%s,true,,"
-                "$(GYP_HOST_VAR_PREFIX))"
-                % (self.android_class, self.android_module)
+                "$(GYP_HOST_VAR_PREFIX))" % (self.android_class, self.android_module)
             )
         else:
             path = (
@@ -1001,9 +999,9 @@ def LocalPathify(self, path):
         # - i.e. that the resulting path is still inside the project tree. The
         # path may legitimately have ended up containing just $(LOCAL_PATH), though,
         # so we don't look for a slash.
-        assert local_path.startswith(
-            "$(LOCAL_PATH)"
-        ), f"Path {path} attempts to escape from gyp path {self.path} !)"
+        assert local_path.startswith("$(LOCAL_PATH)"), (
+            f"Path {path} attempts to escape from gyp path {self.path} !)"
+        )
         return local_path
 
     def ExpandInputRoot(self, template, expansion, dirname):
@@ -1045,9 +1043,9 @@ def CalculateMakefilePath(build_file, base_name):
         base_path = gyp.common.RelativePath(os.path.dirname(build_file), options.depth)
         # We write the file in the base_path directory.
         output_file = os.path.join(options.depth, base_path, base_name)
-        assert (
-            not options.generator_output
-        ), "The Android backend does not support options.generator_output."
+        assert not options.generator_output, (
+            "The Android backend does not support options.generator_output."
+        )
         base_path = gyp.common.RelativePath(
             os.path.dirname(build_file), options.toplevel_dir
         )
@@ -1067,9 +1065,9 @@ def CalculateMakefilePath(build_file, base_name):
 
     makefile_name = "GypAndroid" + options.suffix + ".mk"
     makefile_path = os.path.join(options.toplevel_dir, makefile_name)
-    assert (
-        not options.generator_output
-    ), "The Android backend does not support options.generator_output."
+    assert not options.generator_output, (
+        "The Android backend does not support options.generator_output."
+    )
     gyp.common.EnsureDirExists(makefile_path)
     root_makefile = open(makefile_path, "w")
 
diff --git a/gyp/pylib/gyp/generator/cmake.py b/gyp/pylib/gyp/generator/cmake.py
index db8cc6e026..dc9ea39acb 100644
--- a/gyp/pylib/gyp/generator/cmake.py
+++ b/gyp/pylib/gyp/generator/cmake.py
@@ -28,7 +28,6 @@
 CMakeLists.txt file.
 """
 
-
 import multiprocessing
 import os
 import signal
@@ -97,11 +96,11 @@ def Linkable(filename):
 def NormjoinPathForceCMakeSource(base_path, rel_path):
     """Resolves rel_path against base_path and returns the result.
 
-  If rel_path is an absolute path it is returned unchanged.
-  Otherwise it is resolved against base_path and normalized.
-  If the result is a relative path, it is forced to be relative to the
-  CMakeLists.txt.
-  """
+    If rel_path is an absolute path it is returned unchanged.
+    Otherwise it is resolved against base_path and normalized.
+    If the result is a relative path, it is forced to be relative to the
+    CMakeLists.txt.
+    """
     if os.path.isabs(rel_path):
         return rel_path
     if any(rel_path.startswith(var) for var in FULL_PATH_VARS):
@@ -114,10 +113,10 @@ def NormjoinPathForceCMakeSource(base_path, rel_path):
 
 def NormjoinPath(base_path, rel_path):
     """Resolves rel_path against base_path and returns the result.
-  TODO: what is this really used for?
-  If rel_path begins with '$' it is returned unchanged.
-  Otherwise it is resolved against base_path if relative, then normalized.
-  """
+    TODO: what is this really used for?
+    If rel_path begins with '$' it is returned unchanged.
+    Otherwise it is resolved against base_path if relative, then normalized.
+    """
     if rel_path.startswith("$") and not rel_path.startswith("${configuration}"):
         return rel_path
     return os.path.normpath(os.path.join(base_path, rel_path))
@@ -126,19 +125,19 @@ def NormjoinPath(base_path, rel_path):
 def CMakeStringEscape(a):
     """Escapes the string 'a' for use inside a CMake string.
 
-  This means escaping
-  '\' otherwise it may be seen as modifying the next character
-  '"' otherwise it will end the string
-  ';' otherwise the string becomes a list
+    This means escaping
+    '\' otherwise it may be seen as modifying the next character
+    '"' otherwise it will end the string
+    ';' otherwise the string becomes a list
 
-  The following do not need to be escaped
-  '#' when the lexer is in string state, this does not start a comment
+    The following do not need to be escaped
+    '#' when the lexer is in string state, this does not start a comment
 
-  The following are yet unknown
-  '$' generator variables (like ${obj}) must not be escaped,
-      but text $ should be escaped
-      what is wanted is to know which $ come from generator variables
-  """
+    The following are yet unknown
+    '$' generator variables (like ${obj}) must not be escaped,
+        but text $ should be escaped
+        what is wanted is to know which $ come from generator variables
+    """
     return a.replace("\\", "\\\\").replace(";", "\\;").replace('"', '\\"')
 
 
@@ -237,25 +236,25 @@ def __init__(self, command, modifier, property_modifier):
 def StringToCMakeTargetName(a):
     """Converts the given string 'a' to a valid CMake target name.
 
-  All invalid characters are replaced by '_'.
-  Invalid for cmake: ' ', '/', '(', ')', '"'
-  Invalid for make: ':'
-  Invalid for unknown reasons but cause failures: '.'
-  """
+    All invalid characters are replaced by '_'.
+    Invalid for cmake: ' ', '/', '(', ')', '"'
+    Invalid for make: ':'
+    Invalid for unknown reasons but cause failures: '.'
+    """
     return a.translate(_maketrans(' /():."', "_______"))
 
 
 def WriteActions(target_name, actions, extra_sources, extra_deps, path_to_gyp, output):
     """Write CMake for the 'actions' in the target.
 
-  Args:
-    target_name: the name of the CMake target being generated.
-    actions: the Gyp 'actions' dict for this target.
-    extra_sources: [(, )] to append with generated source files.
-    extra_deps: [] to append with generated targets.
-    path_to_gyp: relative path from CMakeLists.txt being generated to
-        the Gyp file in which the target being generated is defined.
-  """
+    Args:
+      target_name: the name of the CMake target being generated.
+      actions: the Gyp 'actions' dict for this target.
+      extra_sources: [(, )] to append with generated source files.
+      extra_deps: [] to append with generated targets.
+      path_to_gyp: relative path from CMakeLists.txt being generated to
+          the Gyp file in which the target being generated is defined.
+    """
     for action in actions:
         action_name = StringToCMakeTargetName(action["action_name"])
         action_target_name = f"{target_name}__{action_name}"
@@ -337,14 +336,14 @@ def NormjoinRulePathForceCMakeSource(base_path, rel_path, rule_source):
 def WriteRules(target_name, rules, extra_sources, extra_deps, path_to_gyp, output):
     """Write CMake for the 'rules' in the target.
 
-  Args:
-    target_name: the name of the CMake target being generated.
-    actions: the Gyp 'actions' dict for this target.
-    extra_sources: [(, )] to append with generated source files.
-    extra_deps: [] to append with generated targets.
-    path_to_gyp: relative path from CMakeLists.txt being generated to
-        the Gyp file in which the target being generated is defined.
-  """
+    Args:
+      target_name: the name of the CMake target being generated.
+      actions: the Gyp 'actions' dict for this target.
+      extra_sources: [(, )] to append with generated source files.
+      extra_deps: [] to append with generated targets.
+      path_to_gyp: relative path from CMakeLists.txt being generated to
+          the Gyp file in which the target being generated is defined.
+    """
     for rule in rules:
         rule_name = StringToCMakeTargetName(target_name + "__" + rule["rule_name"])
 
@@ -455,13 +454,13 @@ def WriteRules(target_name, rules, extra_sources, extra_deps, path_to_gyp, outpu
 def WriteCopies(target_name, copies, extra_deps, path_to_gyp, output):
     """Write CMake for the 'copies' in the target.
 
-  Args:
-    target_name: the name of the CMake target being generated.
-    actions: the Gyp 'actions' dict for this target.
-    extra_deps: [] to append with generated targets.
-    path_to_gyp: relative path from CMakeLists.txt being generated to
-        the Gyp file in which the target being generated is defined.
-  """
+    Args:
+      target_name: the name of the CMake target being generated.
+      actions: the Gyp 'actions' dict for this target.
+      extra_deps: [] to append with generated targets.
+      path_to_gyp: relative path from CMakeLists.txt being generated to
+          the Gyp file in which the target being generated is defined.
+    """
     copy_name = target_name + "__copies"
 
     # CMake gets upset with custom targets with OUTPUT which specify no output.
@@ -585,23 +584,23 @@ def CreateCMakeTargetFullName(qualified_target):
 class CMakeNamer:
     """Converts Gyp target names into CMake target names.
 
-  CMake requires that target names be globally unique. One way to ensure
-  this is to fully qualify the names of the targets. Unfortunately, this
-  ends up with all targets looking like "chrome_chrome_gyp_chrome" instead
-  of just "chrome". If this generator were only interested in building, it
-  would be possible to fully qualify all target names, then create
-  unqualified target names which depend on all qualified targets which
-  should have had that name. This is more or less what the 'make' generator
-  does with aliases. However, one goal of this generator is to create CMake
-  files for use with IDEs, and fully qualified names are not as user
-  friendly.
-
-  Since target name collision is rare, we do the above only when required.
-
-  Toolset variants are always qualified from the base, as this is required for
-  building. However, it also makes sense for an IDE, as it is possible for
-  defines to be different.
-  """
+    CMake requires that target names be globally unique. One way to ensure
+    this is to fully qualify the names of the targets. Unfortunately, this
+    ends up with all targets looking like "chrome_chrome_gyp_chrome" instead
+    of just "chrome". If this generator were only interested in building, it
+    would be possible to fully qualify all target names, then create
+    unqualified target names which depend on all qualified targets which
+    should have had that name. This is more or less what the 'make' generator
+    does with aliases. However, one goal of this generator is to create CMake
+    files for use with IDEs, and fully qualified names are not as user
+    friendly.
+
+    Since target name collision is rare, we do the above only when required.
+
+    Toolset variants are always qualified from the base, as this is required for
+    building. However, it also makes sense for an IDE, as it is possible for
+    defines to be different.
+    """
 
     def __init__(self, target_list):
         self.cmake_target_base_names_conflicting = set()
diff --git a/gyp/pylib/gyp/generator/dump_dependency_json.py b/gyp/pylib/gyp/generator/dump_dependency_json.py
index e41c72d710..c919674024 100644
--- a/gyp/pylib/gyp/generator/dump_dependency_json.py
+++ b/gyp/pylib/gyp/generator/dump_dependency_json.py
@@ -56,7 +56,7 @@ def CalculateVariables(default_variables, params):
 
 def CalculateGeneratorInputInfo(params):
     """Calculate the generator specific info that gets fed to input (called by
-  gyp)."""
+    gyp)."""
     generator_flags = params.get("generator_flags", {})
     if generator_flags.get("adjust_static_libraries", False):
         global generator_wants_static_library_dependencies_adjusted
diff --git a/gyp/pylib/gyp/generator/eclipse.py b/gyp/pylib/gyp/generator/eclipse.py
index be7ecd8b3b..685cd08c96 100644
--- a/gyp/pylib/gyp/generator/eclipse.py
+++ b/gyp/pylib/gyp/generator/eclipse.py
@@ -69,7 +69,7 @@ def CalculateVariables(default_variables, params):
 
 def CalculateGeneratorInputInfo(params):
     """Calculate the generator specific info that gets fed to input (called by
-  gyp)."""
+    gyp)."""
     generator_flags = params.get("generator_flags", {})
     if generator_flags.get("adjust_static_libraries", False):
         global generator_wants_static_library_dependencies_adjusted
@@ -86,10 +86,10 @@ def GetAllIncludeDirectories(
 ):
     """Calculate the set of include directories to be used.
 
-  Returns:
-    A list including all the include_dir's specified for every target followed
-    by any include directories that were added as cflag compiler options.
-  """
+    Returns:
+      A list including all the include_dir's specified for every target followed
+      by any include directories that were added as cflag compiler options.
+    """
 
     gyp_includes_set = set()
     compiler_includes_list = []
@@ -178,11 +178,11 @@ def GetAllIncludeDirectories(
 def GetCompilerPath(target_list, data, options):
     """Determine a command that can be used to invoke the compiler.
 
-  Returns:
-    If this is a gyp project that has explicit make settings, try to determine
-    the compiler from that.  Otherwise, see if a compiler was specified via the
-    CC_target environment variable.
-  """
+    Returns:
+      If this is a gyp project that has explicit make settings, try to determine
+      the compiler from that.  Otherwise, see if a compiler was specified via the
+      CC_target environment variable.
+    """
     # First, see if the compiler is configured in make's settings.
     build_file, _, _ = gyp.common.ParseQualifiedTarget(target_list[0])
     make_global_settings_dict = data[build_file].get("make_global_settings", {})
@@ -202,10 +202,10 @@ def GetCompilerPath(target_list, data, options):
 def GetAllDefines(target_list, target_dicts, data, config_name, params, compiler_path):
     """Calculate the defines for a project.
 
-  Returns:
-    A dict that includes explicit defines declared in gyp files along with all
-    of the default defines that the compiler uses.
-  """
+    Returns:
+      A dict that includes explicit defines declared in gyp files along with all
+      of the default defines that the compiler uses.
+    """
 
     # Get defines declared in the gyp files.
     all_defines = {}
@@ -373,8 +373,8 @@ def GenerateClasspathFile(
     target_list, target_dicts, toplevel_dir, toplevel_build, out_name
 ):
     """Generates a classpath file suitable for symbol navigation and code
-  completion of Java code (such as in Android projects) by finding all
-  .java and .jar files used as action inputs."""
+    completion of Java code (such as in Android projects) by finding all
+    .java and .jar files used as action inputs."""
     gyp.common.EnsureDirExists(out_name)
     result = ET.Element("classpath")
 
diff --git a/gyp/pylib/gyp/generator/gypd.py b/gyp/pylib/gyp/generator/gypd.py
index a0aa6d9245..3c70b81fd2 100644
--- a/gyp/pylib/gyp/generator/gypd.py
+++ b/gyp/pylib/gyp/generator/gypd.py
@@ -30,7 +30,6 @@
 to change.
 """
 
-
 import pprint
 
 import gyp.common
diff --git a/gyp/pylib/gyp/generator/gypsh.py b/gyp/pylib/gyp/generator/gypsh.py
index 36a05deb7e..72d22ff32b 100644
--- a/gyp/pylib/gyp/generator/gypsh.py
+++ b/gyp/pylib/gyp/generator/gypsh.py
@@ -13,7 +13,6 @@
 The expected usage is "gyp -f gypsh -D OS=desired_os".
 """
 
-
 import code
 import sys
 
diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py
index 0ba1a8c4e1..1f0995718b 100644
--- a/gyp/pylib/gyp/generator/make.py
+++ b/gyp/pylib/gyp/generator/make.py
@@ -218,7 +218,7 @@ def CalculateGeneratorInputInfo(params):
 
 quiet_cmd_solink_module = SOLINK_MODULE($(TOOLSET)) $@
 cmd_solink_module = $(LINK.$(TOOLSET)) -bundle $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -o $@ $(filter-out FORCE_DO_CMD, $^) $(LIBS)
-""" % {'python': sys.executable}  # noqa: E501
+""" % {"python": sys.executable}  # noqa: E501
 
 LINK_COMMANDS_ANDROID = """\
 quiet_cmd_alink = AR($(TOOLSET)) $@
@@ -443,21 +443,27 @@ def CalculateGeneratorInputInfo(params):
 define fixup_dep
 # The depfile may not exist if the input file didn't have any #includes.
 touch $(depfile).raw
-# Fixup path as in (1).""" +
-    (r"""
+# Fixup path as in (1)."""
+    + (
+        r"""
 sed -e "s|^$(notdir $@)|$@|" -re 's/\\\\([^$$])/\/\1/g' $(depfile).raw >> $(depfile)"""
-    if sys.platform == 'win32' else r"""
-sed -e "s|^$(notdir $@)|$@|" $(depfile).raw >> $(depfile)""") +
-    r"""
+        if sys.platform == "win32"
+        else r"""
+sed -e "s|^$(notdir $@)|$@|" $(depfile).raw >> $(depfile)"""
+    )
+    + r"""
 # Add extra rules as in (2).
 # We remove slashes and replace spaces with new lines;
 # remove blank lines;
-# delete the first line and append a colon to the remaining lines.""" +
-    ("""
+# delete the first line and append a colon to the remaining lines."""
+    + (
+        """
 sed -e 's/\\\\\\\\$$//' -e 's/\\\\\\\\/\\//g' -e 'y| |\\n|' $(depfile).raw |\\"""
-    if sys.platform == 'win32' else """
-sed -e 's|\\\\||' -e 'y| |\\n|' $(depfile).raw |\\""") +
-    r"""
+        if sys.platform == "win32"
+        else """
+sed -e 's|\\\\||' -e 'y| |\\n|' $(depfile).raw |\\"""
+    )
+    + r"""
   grep -v '^$$'                             |\
   sed -e 1d -e 's|$$|:|'                     \
     >> $(depfile)
@@ -616,7 +622,7 @@ def CalculateGeneratorInputInfo(params):
 
 quiet_cmd_infoplist = INFOPLIST $@
 cmd_infoplist = $(CC.$(TOOLSET)) -E -P -Wno-trigraphs -x c $(INFOPLIST_DEFINES) "$<" -o "$@"
-""" % {'python': sys.executable}  # noqa: E501
+""" % {"python": sys.executable}  # noqa: E501
 
 
 def WriteRootHeaderSuffixRules(writer):
@@ -733,11 +739,13 @@ def QuoteIfNecessary(string):
         string = '"' + string.replace('"', '\\"') + '"'
     return string
 
+
 def replace_sep(string):
-    if sys.platform == 'win32':
-        string = string.replace('\\\\', '/').replace('\\', '/')
+    if sys.platform == "win32":
+        string = string.replace("\\\\", "/").replace("\\", "/")
     return string
 
+
 def StringToMakefileVariable(string):
     """Convert a string to a value that is acceptable as a make variable name."""
     return re.sub("[^a-zA-Z0-9_]", "_", string)
@@ -1439,9 +1447,7 @@ def WriteSources(
 
         for obj in objs:
             assert " " not in obj, "Spaces in object filenames not supported (%s)" % obj
-        self.WriteLn(
-            "# Add to the list of files we specially track dependencies for."
-        )
+        self.WriteLn("# Add to the list of files we specially track dependencies for.")
         self.WriteLn("all_deps += $(OBJS)")
         self.WriteLn()
 
@@ -1498,7 +1504,8 @@ def WriteSources(
                     "$(OBJS): GYP_OBJCFLAGS := "
                     "$(DEFS_$(BUILDTYPE)) "
                     "$(INCS_$(BUILDTYPE)) "
-                    "%s " % precompiled_header.GetInclude("m")
+                    "%s "
+                    % precompiled_header.GetInclude("m")
                     + "$(CFLAGS_$(BUILDTYPE)) "
                     "$(CFLAGS_C_$(BUILDTYPE)) "
                     "$(CFLAGS_OBJC_$(BUILDTYPE))"
@@ -1507,7 +1514,8 @@ def WriteSources(
                     "$(OBJS): GYP_OBJCXXFLAGS := "
                     "$(DEFS_$(BUILDTYPE)) "
                     "$(INCS_$(BUILDTYPE)) "
-                    "%s " % precompiled_header.GetInclude("mm")
+                    "%s "
+                    % precompiled_header.GetInclude("mm")
                     + "$(CFLAGS_$(BUILDTYPE)) "
                     "$(CFLAGS_CC_$(BUILDTYPE)) "
                     "$(CFLAGS_OBJCC_$(BUILDTYPE))"
@@ -2383,9 +2391,13 @@ def WriteAutoRegenerationRule(params, root_makefile, makefile_name, build_files)
             "deps": replace_sep(
                 " ".join(sorted(SourceifyAndQuoteSpaces(bf) for bf in build_files))
             ),
-            "cmd": replace_sep(gyp.common.EncodePOSIXShellList(
-                [gyp_binary, "-fmake"] + gyp.RegenerateFlags(options) + build_files_args
-            )),
+            "cmd": replace_sep(
+                gyp.common.EncodePOSIXShellList(
+                    [gyp_binary, "-fmake"]
+                    + gyp.RegenerateFlags(options)
+                    + build_files_args
+                )
+            ),
         }
     )
 
@@ -2458,8 +2470,8 @@ def CalculateMakefilePath(build_file, base_name):
     # wasm-ld doesn't support --start-group/--end-group
     link_commands = LINK_COMMANDS_LINUX
     if flavor in ["wasi", "wasm"]:
-        link_commands = link_commands.replace(' -Wl,--start-group', '').replace(
-            ' -Wl,--end-group', ''
+        link_commands = link_commands.replace(" -Wl,--start-group", "").replace(
+            " -Wl,--end-group", ""
         )
 
     CC_target = replace_sep(GetEnvironFallback(("CC_target", "CC"), "$(CC)"))
diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py
index baf5042150..3b258ee8f3 100644
--- a/gyp/pylib/gyp/generator/msvs.py
+++ b/gyp/pylib/gyp/generator/msvs.py
@@ -136,15 +136,15 @@ def _GetDomainAndUserName():
 def _NormalizedSource(source):
     """Normalize the path.
 
-  But not if that gets rid of a variable, as this may expand to something
-  larger than one directory.
+    But not if that gets rid of a variable, as this may expand to something
+    larger than one directory.
 
-  Arguments:
-      source: The path to be normalize.d
+    Arguments:
+        source: The path to be normalize.d
 
-  Returns:
-      The normalized path.
-  """
+    Returns:
+        The normalized path.
+    """
     normalized = os.path.normpath(source)
     if source.count("$") == normalized.count("$"):
         source = normalized
@@ -154,11 +154,11 @@ def _NormalizedSource(source):
 def _FixPath(path, separator="\\"):
     """Convert paths to a form that will make sense in a vcproj file.
 
-  Arguments:
-    path: The path to convert, may contain / etc.
-  Returns:
-    The path with all slashes made into backslashes.
-  """
+    Arguments:
+      path: The path to convert, may contain / etc.
+    Returns:
+      The path with all slashes made into backslashes.
+    """
     if (
         fixpath_prefix
         and path
@@ -179,11 +179,11 @@ def _FixPath(path, separator="\\"):
 
 def _IsWindowsAbsPath(path):
     """
-  On Cygwin systems Python needs a little help determining if a path
-  is an absolute Windows path or not, so that
-  it does not treat those as relative, which results in bad paths like:
-  '..\\C:\\\\some_source_code_file.cc'
-  """
+    On Cygwin systems Python needs a little help determining if a path
+    is an absolute Windows path or not, so that
+    it does not treat those as relative, which results in bad paths like:
+    '..\\C:\\\\some_source_code_file.cc'
+    """
     return path.startswith(("c:", "C:"))
 
 
@@ -197,22 +197,22 @@ def _ConvertSourcesToFilterHierarchy(
 ):
     """Converts a list split source file paths into a vcproj folder hierarchy.
 
-  Arguments:
-    sources: A list of source file paths split.
-    prefix: A list of source file path layers meant to apply to each of sources.
-    excluded: A set of excluded files.
-    msvs_version: A MSVSVersion object.
-
-  Returns:
-    A hierarchy of filenames and MSVSProject.Filter objects that matches the
-    layout of the source tree.
-    For example:
-    _ConvertSourcesToFilterHierarchy([['a', 'bob1.c'], ['b', 'bob2.c']],
-                                     prefix=['joe'])
-    -->
-    [MSVSProject.Filter('a', contents=['joe\\a\\bob1.c']),
-     MSVSProject.Filter('b', contents=['joe\\b\\bob2.c'])]
-  """
+    Arguments:
+      sources: A list of source file paths split.
+      prefix: A list of source file path layers meant to apply to each of sources.
+      excluded: A set of excluded files.
+      msvs_version: A MSVSVersion object.
+
+    Returns:
+      A hierarchy of filenames and MSVSProject.Filter objects that matches the
+      layout of the source tree.
+      For example:
+      _ConvertSourcesToFilterHierarchy([['a', 'bob1.c'], ['b', 'bob2.c']],
+                                       prefix=['joe'])
+      -->
+      [MSVSProject.Filter('a', contents=['joe\\a\\bob1.c']),
+       MSVSProject.Filter('b', contents=['joe\\b\\bob2.c'])]
+    """
     if not prefix:
         prefix = []
     result = []
@@ -361,7 +361,6 @@ def _ConfigWindowsTargetPlatformVersion(config_data, version):
 def _BuildCommandLineForRuleRaw(
     spec, cmd, cygwin_shell, has_input_path, quote_cmd, do_setup_env
 ):
-
     if [x for x in cmd if "$(InputDir)" in x]:
         input_dir_preamble = (
             "set INPUTDIR=$(InputDir)\n"
@@ -425,8 +424,7 @@ def _BuildCommandLineForRuleRaw(
         # Return the path with forward slashes because the command using it might
         # not support backslashes.
         arguments = [
-            i if (i[:1] in "/-" or "=" in i) else _FixPath(i, "/")
-            for i in cmd[1:]
+            i if (i[:1] in "/-" or "=" in i) else _FixPath(i, "/") for i in cmd[1:]
         ]
         arguments = [i.replace("$(InputDir)", "%INPUTDIR%") for i in arguments]
         arguments = [MSVSSettings.FixVCMacroSlashes(i) for i in arguments]
@@ -459,17 +457,17 @@ def _BuildCommandLineForRule(spec, rule, has_input_path, do_setup_env):
 def _AddActionStep(actions_dict, inputs, outputs, description, command):
     """Merge action into an existing list of actions.
 
-  Care must be taken so that actions which have overlapping inputs either don't
-  get assigned to the same input, or get collapsed into one.
-
-  Arguments:
-    actions_dict: dictionary keyed on input name, which maps to a list of
-      dicts describing the actions attached to that input file.
-    inputs: list of inputs
-    outputs: list of outputs
-    description: description of the action
-    command: command line to execute
-  """
+    Care must be taken so that actions which have overlapping inputs either don't
+    get assigned to the same input, or get collapsed into one.
+
+    Arguments:
+      actions_dict: dictionary keyed on input name, which maps to a list of
+        dicts describing the actions attached to that input file.
+      inputs: list of inputs
+      outputs: list of outputs
+      description: description of the action
+      command: command line to execute
+    """
     # Require there to be at least one input (call sites will ensure this).
     assert inputs
 
@@ -496,15 +494,15 @@ def _AddCustomBuildToolForMSVS(
 ):
     """Add a custom build tool to execute something.
 
-  Arguments:
-    p: the target project
-    spec: the target project dict
-    primary_input: input file to attach the build tool to
-    inputs: list of inputs
-    outputs: list of outputs
-    description: description of the action
-    cmd: command line to execute
-  """
+    Arguments:
+      p: the target project
+      spec: the target project dict
+      primary_input: input file to attach the build tool to
+      inputs: list of inputs
+      outputs: list of outputs
+      description: description of the action
+      cmd: command line to execute
+    """
     inputs = _FixPaths(inputs)
     outputs = _FixPaths(outputs)
     tool = MSVSProject.Tool(
@@ -526,12 +524,12 @@ def _AddCustomBuildToolForMSVS(
 def _AddAccumulatedActionsToMSVS(p, spec, actions_dict):
     """Add actions accumulated into an actions_dict, merging as needed.
 
-  Arguments:
-    p: the target project
-    spec: the target project dict
-    actions_dict: dictionary keyed on input name, which maps to a list of
-        dicts describing the actions attached to that input file.
-  """
+    Arguments:
+      p: the target project
+      spec: the target project dict
+      actions_dict: dictionary keyed on input name, which maps to a list of
+          dicts describing the actions attached to that input file.
+    """
     for primary_input in actions_dict:
         inputs = OrderedSet()
         outputs = OrderedSet()
@@ -559,12 +557,12 @@ def _AddAccumulatedActionsToMSVS(p, spec, actions_dict):
 def _RuleExpandPath(path, input_file):
     """Given the input file to which a rule applied, string substitute a path.
 
-  Arguments:
-    path: a path to string expand
-    input_file: the file to which the rule applied.
-  Returns:
-    The string substituted path.
-  """
+    Arguments:
+      path: a path to string expand
+      input_file: the file to which the rule applied.
+    Returns:
+      The string substituted path.
+    """
     path = path.replace(
         "$(InputName)", os.path.splitext(os.path.split(input_file)[1])[0]
     )
@@ -580,24 +578,24 @@ def _RuleExpandPath(path, input_file):
 def _FindRuleTriggerFiles(rule, sources):
     """Find the list of files which a particular rule applies to.
 
-  Arguments:
-    rule: the rule in question
-    sources: the set of all known source files for this project
-  Returns:
-    The list of sources that trigger a particular rule.
-  """
+    Arguments:
+      rule: the rule in question
+      sources: the set of all known source files for this project
+    Returns:
+      The list of sources that trigger a particular rule.
+    """
     return rule.get("rule_sources", [])
 
 
 def _RuleInputsAndOutputs(rule, trigger_file):
     """Find the inputs and outputs generated by a rule.
 
-  Arguments:
-    rule: the rule in question.
-    trigger_file: the main trigger for this rule.
-  Returns:
-    The pair of (inputs, outputs) involved in this rule.
-  """
+    Arguments:
+      rule: the rule in question.
+      trigger_file: the main trigger for this rule.
+    Returns:
+      The pair of (inputs, outputs) involved in this rule.
+    """
     raw_inputs = _FixPaths(rule.get("inputs", []))
     raw_outputs = _FixPaths(rule.get("outputs", []))
     inputs = OrderedSet()
@@ -613,13 +611,13 @@ def _RuleInputsAndOutputs(rule, trigger_file):
 def _GenerateNativeRulesForMSVS(p, rules, output_dir, spec, options):
     """Generate a native rules file.
 
-  Arguments:
-    p: the target project
-    rules: the set of rules to include
-    output_dir: the directory in which the project/gyp resides
-    spec: the project dict
-    options: global generator options
-  """
+    Arguments:
+      p: the target project
+      rules: the set of rules to include
+      output_dir: the directory in which the project/gyp resides
+      spec: the project dict
+      options: global generator options
+    """
     rules_filename = "{}{}.rules".format(spec["target_name"], options.suffix)
     rules_file = MSVSToolFile.Writer(
         os.path.join(output_dir, rules_filename), spec["target_name"]
@@ -658,14 +656,14 @@ def _Cygwinify(path):
 def _GenerateExternalRules(rules, output_dir, spec, sources, options, actions_to_add):
     """Generate an external makefile to do a set of rules.
 
-  Arguments:
-    rules: the list of rules to include
-    output_dir: path containing project and gyp files
-    spec: project specification data
-    sources: set of sources known
-    options: global generator options
-    actions_to_add: The list of actions we will add to.
-  """
+    Arguments:
+      rules: the list of rules to include
+      output_dir: path containing project and gyp files
+      spec: project specification data
+      sources: set of sources known
+      options: global generator options
+      actions_to_add: The list of actions we will add to.
+    """
     filename = "{}_rules{}.mk".format(spec["target_name"], options.suffix)
     mk_file = gyp.common.WriteOnDiff(os.path.join(output_dir, filename))
     # Find cygwin style versions of some paths.
@@ -743,17 +741,17 @@ def _GenerateExternalRules(rules, output_dir, spec, sources, options, actions_to
 def _EscapeEnvironmentVariableExpansion(s):
     """Escapes % characters.
 
-  Escapes any % characters so that Windows-style environment variable
-  expansions will leave them alone.
-  See http://connect.microsoft.com/VisualStudio/feedback/details/106127/cl-d-name-text-containing-percentage-characters-doesnt-compile
-  to understand why we have to do this.
+    Escapes any % characters so that Windows-style environment variable
+    expansions will leave them alone.
+    See http://connect.microsoft.com/VisualStudio/feedback/details/106127/cl-d-name-text-containing-percentage-characters-doesnt-compile
+    to understand why we have to do this.
 
-  Args:
-      s: The string to be escaped.
+    Args:
+        s: The string to be escaped.
 
-  Returns:
-      The escaped string.
-  """
+    Returns:
+        The escaped string.
+    """
     s = s.replace("%", "%%")
     return s
 
@@ -764,17 +762,17 @@ def _EscapeEnvironmentVariableExpansion(s):
 def _EscapeCommandLineArgumentForMSVS(s):
     """Escapes a Windows command-line argument.
 
-  So that the Win32 CommandLineToArgv function will turn the escaped result back
-  into the original string.
-  See http://msdn.microsoft.com/en-us/library/17w5ykft.aspx
-  ("Parsing C++ Command-Line Arguments") to understand why we have to do
-  this.
+    So that the Win32 CommandLineToArgv function will turn the escaped result back
+    into the original string.
+    See http://msdn.microsoft.com/en-us/library/17w5ykft.aspx
+    ("Parsing C++ Command-Line Arguments") to understand why we have to do
+    this.
 
-  Args:
-      s: the string to be escaped.
-  Returns:
-      the escaped string.
-  """
+    Args:
+        s: the string to be escaped.
+    Returns:
+        the escaped string.
+    """
 
     def _Replace(match):
         # For a literal quote, CommandLineToArgv requires an odd number of
@@ -795,24 +793,24 @@ def _Replace(match):
 def _EscapeVCProjCommandLineArgListItem(s):
     """Escapes command line arguments for MSVS.
 
-  The VCProj format stores string lists in a single string using commas and
-  semi-colons as separators, which must be quoted if they are to be
-  interpreted literally. However, command-line arguments may already have
-  quotes, and the VCProj parser is ignorant of the backslash escaping
-  convention used by CommandLineToArgv, so the command-line quotes and the
-  VCProj quotes may not be the same quotes. So to store a general
-  command-line argument in a VCProj list, we need to parse the existing
-  quoting according to VCProj's convention and quote any delimiters that are
-  not already quoted by that convention. The quotes that we add will also be
-  seen by CommandLineToArgv, so if backslashes precede them then we also have
-  to escape those backslashes according to the CommandLineToArgv
-  convention.
-
-  Args:
-      s: the string to be escaped.
-  Returns:
-      the escaped string.
-  """
+    The VCProj format stores string lists in a single string using commas and
+    semi-colons as separators, which must be quoted if they are to be
+    interpreted literally. However, command-line arguments may already have
+    quotes, and the VCProj parser is ignorant of the backslash escaping
+    convention used by CommandLineToArgv, so the command-line quotes and the
+    VCProj quotes may not be the same quotes. So to store a general
+    command-line argument in a VCProj list, we need to parse the existing
+    quoting according to VCProj's convention and quote any delimiters that are
+    not already quoted by that convention. The quotes that we add will also be
+    seen by CommandLineToArgv, so if backslashes precede them then we also have
+    to escape those backslashes according to the CommandLineToArgv
+    convention.
+
+    Args:
+        s: the string to be escaped.
+    Returns:
+        the escaped string.
+    """
 
     def _Replace(match):
         # For a non-literal quote, CommandLineToArgv requires an even number of
@@ -896,15 +894,15 @@ def _GenerateRulesForMSVS(
 ):
     """Generate all the rules for a particular project.
 
-  Arguments:
-    p: the project
-    output_dir: directory to emit rules to
-    options: global options passed to the generator
-    spec: the specification for this project
-    sources: the set of all known source files in this project
-    excluded_sources: the set of sources excluded from normal processing
-    actions_to_add: deferred list of actions to add in
-  """
+    Arguments:
+      p: the project
+      output_dir: directory to emit rules to
+      options: global options passed to the generator
+      spec: the specification for this project
+      sources: the set of all known source files in this project
+      excluded_sources: the set of sources excluded from normal processing
+      actions_to_add: deferred list of actions to add in
+    """
     rules = spec.get("rules", [])
     rules_native = [r for r in rules if not int(r.get("msvs_external_rule", 0))]
     rules_external = [r for r in rules if int(r.get("msvs_external_rule", 0))]
@@ -946,12 +944,12 @@ def _AdjustSourcesForRules(rules, sources, excluded_sources, is_msbuild):
 def _FilterActionsFromExcluded(excluded_sources, actions_to_add):
     """Take inputs with actions attached out of the list of exclusions.
 
-  Arguments:
-    excluded_sources: list of source files not to be built.
-    actions_to_add: dict of actions keyed on source file they're attached to.
-  Returns:
-    excluded_sources with files that have actions attached removed.
-  """
+    Arguments:
+      excluded_sources: list of source files not to be built.
+      actions_to_add: dict of actions keyed on source file they're attached to.
+    Returns:
+      excluded_sources with files that have actions attached removed.
+    """
     must_keep = OrderedSet(_FixPaths(actions_to_add.keys()))
     return [s for s in excluded_sources if s not in must_keep]
 
@@ -963,14 +961,14 @@ def _GetDefaultConfiguration(spec):
 def _GetGuidOfProject(proj_path, spec):
     """Get the guid for the project.
 
-  Arguments:
-    proj_path: Path of the vcproj or vcxproj file to generate.
-    spec: The target dictionary containing the properties of the target.
-  Returns:
-    the guid.
-  Raises:
-    ValueError: if the specified GUID is invalid.
-  """
+    Arguments:
+      proj_path: Path of the vcproj or vcxproj file to generate.
+      spec: The target dictionary containing the properties of the target.
+    Returns:
+      the guid.
+    Raises:
+      ValueError: if the specified GUID is invalid.
+    """
     # Pluck out the default configuration.
     default_config = _GetDefaultConfiguration(spec)
     # Decide the guid of the project.
@@ -989,13 +987,13 @@ def _GetGuidOfProject(proj_path, spec):
 def _GetMsbuildToolsetOfProject(proj_path, spec, version):
     """Get the platform toolset for the project.
 
-  Arguments:
-    proj_path: Path of the vcproj or vcxproj file to generate.
-    spec: The target dictionary containing the properties of the target.
-    version: The MSVSVersion object.
-  Returns:
-    the platform toolset string or None.
-  """
+    Arguments:
+      proj_path: Path of the vcproj or vcxproj file to generate.
+      spec: The target dictionary containing the properties of the target.
+      version: The MSVSVersion object.
+    Returns:
+      the platform toolset string or None.
+    """
     # Pluck out the default configuration.
     default_config = _GetDefaultConfiguration(spec)
     toolset = default_config.get("msbuild_toolset")
@@ -1009,14 +1007,14 @@ def _GetMsbuildToolsetOfProject(proj_path, spec, version):
 def _GenerateProject(project, options, version, generator_flags, spec):
     """Generates a vcproj file.
 
-  Arguments:
-    project: the MSVSProject object.
-    options: global generator options.
-    version: the MSVSVersion object.
-    generator_flags: dict of generator-specific flags.
-  Returns:
-    A list of source files that cannot be found on disk.
-  """
+    Arguments:
+      project: the MSVSProject object.
+      options: global generator options.
+      version: the MSVSVersion object.
+      generator_flags: dict of generator-specific flags.
+    Returns:
+      A list of source files that cannot be found on disk.
+    """
     default_config = _GetDefaultConfiguration(project.spec)
 
     # Skip emitting anything if told to with msvs_existing_vcproj option.
@@ -1032,12 +1030,12 @@ def _GenerateProject(project, options, version, generator_flags, spec):
 def _GenerateMSVSProject(project, options, version, generator_flags):
     """Generates a .vcproj file.  It may create .rules and .user files too.
 
-  Arguments:
-    project: The project object we will generate the file for.
-    options: Global options passed to the generator.
-    version: The VisualStudioVersion object.
-    generator_flags: dict of generator-specific flags.
-  """
+    Arguments:
+      project: The project object we will generate the file for.
+      options: Global options passed to the generator.
+      version: The VisualStudioVersion object.
+      generator_flags: dict of generator-specific flags.
+    """
     spec = project.spec
     gyp.common.EnsureDirExists(project.path)
 
@@ -1094,11 +1092,11 @@ def _GenerateMSVSProject(project, options, version, generator_flags):
 def _GetUniquePlatforms(spec):
     """Returns the list of unique platforms for this spec, e.g ['win32', ...].
 
-  Arguments:
-    spec: The target dictionary containing the properties of the target.
-  Returns:
-    The MSVSUserFile object created.
-  """
+    Arguments:
+      spec: The target dictionary containing the properties of the target.
+    Returns:
+      The MSVSUserFile object created.
+    """
     # Gather list of unique platforms.
     platforms = OrderedSet()
     for configuration in spec["configurations"]:
@@ -1110,14 +1108,14 @@ def _GetUniquePlatforms(spec):
 def _CreateMSVSUserFile(proj_path, version, spec):
     """Generates a .user file for the user running this Gyp program.
 
-  Arguments:
-    proj_path: The path of the project file being created.  The .user file
-               shares the same path (with an appropriate suffix).
-    version: The VisualStudioVersion object.
-    spec: The target dictionary containing the properties of the target.
-  Returns:
-    The MSVSUserFile object created.
-  """
+    Arguments:
+      proj_path: The path of the project file being created.  The .user file
+                 shares the same path (with an appropriate suffix).
+      version: The VisualStudioVersion object.
+      spec: The target dictionary containing the properties of the target.
+    Returns:
+      The MSVSUserFile object created.
+    """
     (domain, username) = _GetDomainAndUserName()
     vcuser_filename = ".".join([proj_path, domain, username, "user"])
     user_file = MSVSUserFile.Writer(vcuser_filename, version, spec["target_name"])
@@ -1127,14 +1125,14 @@ def _CreateMSVSUserFile(proj_path, version, spec):
 def _GetMSVSConfigurationType(spec, build_file):
     """Returns the configuration type for this project.
 
-  It's a number defined by Microsoft.  May raise an exception.
+    It's a number defined by Microsoft.  May raise an exception.
 
-  Args:
-      spec: The target dictionary containing the properties of the target.
-      build_file: The path of the gyp file.
-  Returns:
-      An integer, the configuration type.
-  """
+    Args:
+        spec: The target dictionary containing the properties of the target.
+        build_file: The path of the gyp file.
+    Returns:
+        An integer, the configuration type.
+    """
     try:
         config_type = {
             "executable": "1",  # .exe
@@ -1161,17 +1159,17 @@ def _GetMSVSConfigurationType(spec, build_file):
 def _AddConfigurationToMSVSProject(p, spec, config_type, config_name, config):
     """Adds a configuration to the MSVS project.
 
-  Many settings in a vcproj file are specific to a configuration.  This
-  function the main part of the vcproj file that's configuration specific.
-
-  Arguments:
-    p: The target project being generated.
-    spec: The target dictionary containing the properties of the target.
-    config_type: The configuration type, a number as defined by Microsoft.
-    config_name: The name of the configuration.
-    config: The dictionary that defines the special processing to be done
-            for this configuration.
-  """
+    Many settings in a vcproj file are specific to a configuration.  This
+    function the main part of the vcproj file that's configuration specific.
+
+    Arguments:
+      p: The target project being generated.
+      spec: The target dictionary containing the properties of the target.
+      config_type: The configuration type, a number as defined by Microsoft.
+      config_name: The name of the configuration.
+      config: The dictionary that defines the special processing to be done
+              for this configuration.
+    """
     # Get the information for this configuration
     include_dirs, midl_include_dirs, resource_include_dirs = _GetIncludeDirs(config)
     libraries = _GetLibraries(spec)
@@ -1251,12 +1249,12 @@ def _AddConfigurationToMSVSProject(p, spec, config_type, config_name, config):
 def _GetIncludeDirs(config):
     """Returns the list of directories to be used for #include directives.
 
-  Arguments:
-    config: The dictionary that defines the special processing to be done
-            for this configuration.
-  Returns:
-    The list of directory paths.
-  """
+    Arguments:
+      config: The dictionary that defines the special processing to be done
+              for this configuration.
+    Returns:
+      The list of directory paths.
+    """
     # TODO(bradnelson): include_dirs should really be flexible enough not to
     #                   require this sort of thing.
     include_dirs = config.get("include_dirs", []) + config.get(
@@ -1275,12 +1273,12 @@ def _GetIncludeDirs(config):
 def _GetLibraryDirs(config):
     """Returns the list of directories to be used for library search paths.
 
-  Arguments:
-    config: The dictionary that defines the special processing to be done
-            for this configuration.
-  Returns:
-    The list of directory paths.
-  """
+    Arguments:
+      config: The dictionary that defines the special processing to be done
+              for this configuration.
+    Returns:
+      The list of directory paths.
+    """
 
     library_dirs = config.get("library_dirs", [])
     library_dirs = _FixPaths(library_dirs)
@@ -1290,11 +1288,11 @@ def _GetLibraryDirs(config):
 def _GetLibraries(spec):
     """Returns the list of libraries for this configuration.
 
-  Arguments:
-    spec: The target dictionary containing the properties of the target.
-  Returns:
-    The list of directory paths.
-  """
+    Arguments:
+      spec: The target dictionary containing the properties of the target.
+    Returns:
+      The list of directory paths.
+    """
     libraries = spec.get("libraries", [])
     # Strip out -l, as it is not used on windows (but is needed so we can pass
     # in libraries that are assumed to be in the default library path).
@@ -1316,14 +1314,14 @@ def _GetLibraries(spec):
 def _GetOutputFilePathAndTool(spec, msbuild):
     """Returns the path and tool to use for this target.
 
-  Figures out the path of the file this spec will create and the name of
-  the VC tool that will create it.
+    Figures out the path of the file this spec will create and the name of
+    the VC tool that will create it.
 
-  Arguments:
-    spec: The target dictionary containing the properties of the target.
-  Returns:
-    A triple of (file path, name of the vc tool, name of the msbuild tool)
-  """
+    Arguments:
+      spec: The target dictionary containing the properties of the target.
+    Returns:
+      A triple of (file path, name of the vc tool, name of the msbuild tool)
+    """
     # Select a name for the output file.
     out_file = ""
     vc_tool = ""
@@ -1355,15 +1353,15 @@ def _GetOutputFilePathAndTool(spec, msbuild):
 def _GetOutputTargetExt(spec):
     """Returns the extension for this target, including the dot
 
-  If product_extension is specified, set target_extension to this to avoid
-  MSB8012, returns None otherwise. Ignores any target_extension settings in
-  the input files.
+    If product_extension is specified, set target_extension to this to avoid
+    MSB8012, returns None otherwise. Ignores any target_extension settings in
+    the input files.
 
-  Arguments:
-    spec: The target dictionary containing the properties of the target.
-  Returns:
-    A string with the extension, or None
-  """
+    Arguments:
+      spec: The target dictionary containing the properties of the target.
+    Returns:
+      A string with the extension, or None
+    """
     if target_extension := spec.get("product_extension"):
         return "." + target_extension
     return None
@@ -1372,12 +1370,12 @@ def _GetOutputTargetExt(spec):
 def _GetDefines(config):
     """Returns the list of preprocessor definitions for this configuration.
 
-  Arguments:
-    config: The dictionary that defines the special processing to be done
-            for this configuration.
-  Returns:
-    The list of preprocessor definitions.
-  """
+    Arguments:
+      config: The dictionary that defines the special processing to be done
+              for this configuration.
+    Returns:
+      The list of preprocessor definitions.
+    """
     defines = []
     for d in config.get("defines", []):
         fd = "=".join([str(dpart) for dpart in d]) if isinstance(d, list) else str(d)
@@ -1411,11 +1409,11 @@ def _GetModuleDefinition(spec):
 def _ConvertToolsToExpectedForm(tools):
     """Convert tools to a form expected by Visual Studio.
 
-  Arguments:
-    tools: A dictionary of settings; the tool name is the key.
-  Returns:
-    A list of Tool objects.
-  """
+    Arguments:
+      tools: A dictionary of settings; the tool name is the key.
+    Returns:
+      A list of Tool objects.
+    """
     tool_list = []
     for tool, settings in tools.items():
         # Collapse settings with lists.
@@ -1438,15 +1436,15 @@ def _ConvertToolsToExpectedForm(tools):
 def _AddConfigurationToMSVS(p, spec, tools, config, config_type, config_name):
     """Add to the project file the configuration specified by config.
 
-  Arguments:
-    p: The target project being generated.
-    spec: the target project dict.
-    tools: A dictionary of settings; the tool name is the key.
-    config: The dictionary that defines the special processing to be done
-            for this configuration.
-    config_type: The configuration type, a number as defined by Microsoft.
-    config_name: The name of the configuration.
-  """
+    Arguments:
+      p: The target project being generated.
+      spec: the target project dict.
+      tools: A dictionary of settings; the tool name is the key.
+      config: The dictionary that defines the special processing to be done
+              for this configuration.
+      config_type: The configuration type, a number as defined by Microsoft.
+      config_name: The name of the configuration.
+    """
     attributes = _GetMSVSAttributes(spec, config, config_type)
     # Add in this configuration.
     tool_list = _ConvertToolsToExpectedForm(tools)
@@ -1487,18 +1485,18 @@ def _AddNormalizedSources(sources_set, sources_array):
 def _PrepareListOfSources(spec, generator_flags, gyp_file):
     """Prepare list of sources and excluded sources.
 
-  Besides the sources specified directly in the spec, adds the gyp file so
-  that a change to it will cause a re-compile. Also adds appropriate sources
-  for actions and copies. Assumes later stage will un-exclude files which
-  have custom build steps attached.
-
-  Arguments:
-    spec: The target dictionary containing the properties of the target.
-    gyp_file: The name of the gyp file.
-  Returns:
-    A pair of (list of sources, list of excluded sources).
-    The sources will be relative to the gyp file.
-  """
+    Besides the sources specified directly in the spec, adds the gyp file so
+    that a change to it will cause a re-compile. Also adds appropriate sources
+    for actions and copies. Assumes later stage will un-exclude files which
+    have custom build steps attached.
+
+    Arguments:
+      spec: The target dictionary containing the properties of the target.
+      gyp_file: The name of the gyp file.
+    Returns:
+      A pair of (list of sources, list of excluded sources).
+      The sources will be relative to the gyp file.
+    """
     sources = OrderedSet()
     _AddNormalizedSources(sources, spec.get("sources", []))
     excluded_sources = OrderedSet()
@@ -1528,19 +1526,19 @@ def _AdjustSourcesAndConvertToFilterHierarchy(
 ):
     """Adjusts the list of sources and excluded sources.
 
-  Also converts the sets to lists.
-
-  Arguments:
-    spec: The target dictionary containing the properties of the target.
-    options: Global generator options.
-    gyp_dir: The path to the gyp file being processed.
-    sources: A set of sources to be included for this project.
-    excluded_sources: A set of sources to be excluded for this project.
-    version: A MSVSVersion object.
-  Returns:
-    A trio of (list of sources, list of excluded sources,
-               path of excluded IDL file)
-  """
+    Also converts the sets to lists.
+
+    Arguments:
+      spec: The target dictionary containing the properties of the target.
+      options: Global generator options.
+      gyp_dir: The path to the gyp file being processed.
+      sources: A set of sources to be included for this project.
+      excluded_sources: A set of sources to be excluded for this project.
+      version: A MSVSVersion object.
+    Returns:
+      A trio of (list of sources, list of excluded sources,
+                 path of excluded IDL file)
+    """
     # Exclude excluded sources coming into the generator.
     excluded_sources.update(OrderedSet(spec.get("sources_excluded", [])))
     # Add excluded sources into sources for good measure.
@@ -1836,8 +1834,11 @@ def _CollapseSingles(parent, node):
     # Recursively explorer the tree of dicts looking for projects which are
     # the sole item in a folder which has the same name as the project. Bring
     # such projects up one level.
-    if (isinstance(node, dict) and len(node) == 1 and
-        next(iter(node)) == parent + ".vcproj"):
+    if (
+        isinstance(node, dict)
+        and len(node) == 1
+        and next(iter(node)) == parent + ".vcproj"
+    ):
         return node[next(iter(node))]
     if not isinstance(node, dict):
         return node
@@ -1906,14 +1907,14 @@ def _GetPlatformOverridesOfProject(spec):
 def _CreateProjectObjects(target_list, target_dicts, options, msvs_version):
     """Create a MSVSProject object for the targets found in target list.
 
-  Arguments:
-    target_list: the list of targets to generate project objects for.
-    target_dicts: the dictionary of specifications.
-    options: global generator options.
-    msvs_version: the MSVSVersion object.
-  Returns:
-    A set of created projects, keyed by target.
-  """
+    Arguments:
+      target_list: the list of targets to generate project objects for.
+      target_dicts: the dictionary of specifications.
+      options: global generator options.
+      msvs_version: the MSVSVersion object.
+    Returns:
+      A set of created projects, keyed by target.
+    """
     global fixpath_prefix
     # Generate each project.
     projects = {}
@@ -1957,15 +1958,15 @@ def _CreateProjectObjects(target_list, target_dicts, options, msvs_version):
 def _InitNinjaFlavor(params, target_list, target_dicts):
     """Initialize targets for the ninja flavor.
 
-  This sets up the necessary variables in the targets to generate msvs projects
-  that use ninja as an external builder. The variables in the spec are only set
-  if they have not been set. This allows individual specs to override the
-  default values initialized here.
-  Arguments:
-    params: Params provided to the generator.
-    target_list: List of target pairs: 'base/base.gyp:base'.
-    target_dicts: Dict of target properties keyed on target pair.
-  """
+    This sets up the necessary variables in the targets to generate msvs projects
+    that use ninja as an external builder. The variables in the spec are only set
+    if they have not been set. This allows individual specs to override the
+    default values initialized here.
+    Arguments:
+      params: Params provided to the generator.
+      target_list: List of target pairs: 'base/base.gyp:base'.
+      target_dicts: Dict of target properties keyed on target pair.
+    """
     for qualified_target in target_list:
         spec = target_dicts[qualified_target]
         if spec.get("msvs_external_builder"):
@@ -2076,12 +2077,12 @@ def CalculateGeneratorInputInfo(params):
 def GenerateOutput(target_list, target_dicts, data, params):
     """Generate .sln and .vcproj files.
 
-  This is the entry point for this generator.
-  Arguments:
-    target_list: List of target pairs: 'base/base.gyp:base'.
-    target_dicts: Dict of target properties keyed on target pair.
-    data: Dictionary containing per .gyp data.
-  """
+    This is the entry point for this generator.
+    Arguments:
+      target_list: List of target pairs: 'base/base.gyp:base'.
+      target_dicts: Dict of target properties keyed on target pair.
+      data: Dictionary containing per .gyp data.
+    """
     global fixpath_prefix
 
     options = params["options"]
@@ -2175,14 +2176,14 @@ def _GenerateMSBuildFiltersFile(
 ):
     """Generate the filters file.
 
-  This file is used by Visual Studio to organize the presentation of source
-  files into folders.
+    This file is used by Visual Studio to organize the presentation of source
+    files into folders.
 
-  Arguments:
-      filters_path: The path of the file to be created.
-      source_files: The hierarchical structure of all the sources.
-      extension_to_rule_name: A dictionary mapping file extensions to rules.
-  """
+    Arguments:
+        filters_path: The path of the file to be created.
+        source_files: The hierarchical structure of all the sources.
+        extension_to_rule_name: A dictionary mapping file extensions to rules.
+    """
     filter_group = []
     source_group = []
     _AppendFiltersForMSBuild(
@@ -2223,14 +2224,14 @@ def _AppendFiltersForMSBuild(
 ):
     """Creates the list of filters and sources to be added in the filter file.
 
-  Args:
-      parent_filter_name: The name of the filter under which the sources are
-          found.
-      sources: The hierarchy of filters and sources to process.
-      extension_to_rule_name: A dictionary mapping file extensions to rules.
-      filter_group: The list to which filter entries will be appended.
-      source_group: The list to which source entries will be appended.
-  """
+    Args:
+        parent_filter_name: The name of the filter under which the sources are
+            found.
+        sources: The hierarchy of filters and sources to process.
+        extension_to_rule_name: A dictionary mapping file extensions to rules.
+        filter_group: The list to which filter entries will be appended.
+        source_group: The list to which source entries will be appended.
+    """
     for source in sources:
         if isinstance(source, MSVSProject.Filter):
             # We have a sub-filter.  Create the name of that sub-filter.
@@ -2274,13 +2275,13 @@ def _MapFileToMsBuildSourceType(
 ):
     """Returns the group and element type of the source file.
 
-  Arguments:
-      source: The source file name.
-      extension_to_rule_name: A dictionary mapping file extensions to rules.
+    Arguments:
+        source: The source file name.
+        extension_to_rule_name: A dictionary mapping file extensions to rules.
 
-  Returns:
-      A pair of (group this file should be part of, the label of element)
-  """
+    Returns:
+        A pair of (group this file should be part of, the label of element)
+    """
     _, ext = os.path.splitext(source)
     ext = ext.lower()
     if ext in extension_to_rule_name:
@@ -2368,22 +2369,22 @@ def _GenerateRulesForMSBuild(
 class MSBuildRule:
     """Used to store information used to generate an MSBuild rule.
 
-  Attributes:
-    rule_name: The rule name, sanitized to use in XML.
-    target_name: The name of the target.
-    after_targets: The name of the AfterTargets element.
-    before_targets: The name of the BeforeTargets element.
-    depends_on: The name of the DependsOn element.
-    compute_output: The name of the ComputeOutput element.
-    dirs_to_make: The name of the DirsToMake element.
-    inputs: The name of the _inputs element.
-    tlog: The name of the _tlog element.
-    extension: The extension this rule applies to.
-    description: The message displayed when this rule is invoked.
-    additional_dependencies: A string listing additional dependencies.
-    outputs: The outputs of this rule.
-    command: The command used to run the rule.
-  """
+    Attributes:
+      rule_name: The rule name, sanitized to use in XML.
+      target_name: The name of the target.
+      after_targets: The name of the AfterTargets element.
+      before_targets: The name of the BeforeTargets element.
+      depends_on: The name of the DependsOn element.
+      compute_output: The name of the ComputeOutput element.
+      dirs_to_make: The name of the DirsToMake element.
+      inputs: The name of the _inputs element.
+      tlog: The name of the _tlog element.
+      extension: The extension this rule applies to.
+      description: The message displayed when this rule is invoked.
+      additional_dependencies: A string listing additional dependencies.
+      outputs: The outputs of this rule.
+      command: The command used to run the rule.
+    """
 
     def __init__(self, rule, spec):
         self.display_name = rule["rule_name"]
@@ -2908,7 +2909,7 @@ def _GetConfigurationCondition(name, settings, spec):
 
 def _GetMSBuildProjectConfigurations(configurations, spec):
     group = ["ItemGroup", {"Label": "ProjectConfigurations"}]
-    for (name, settings) in sorted(configurations.items()):
+    for name, settings in sorted(configurations.items()):
         configuration, platform = _GetConfigurationAndPlatform(name, settings, spec)
         designation = f"{configuration}|{platform}"
         group.append(
@@ -3002,10 +3003,11 @@ def _GetMSBuildConfigurationDetails(spec, build_file):
         vctools_version = msbuild_attributes.get("VCToolsVersion")
         config_type = msbuild_attributes.get("ConfigurationType")
         _AddConditionalProperty(properties, condition, "ConfigurationType", config_type)
-        spectre_mitigation = msbuild_attributes.get('SpectreMitigation')
+        spectre_mitigation = msbuild_attributes.get("SpectreMitigation")
         if spectre_mitigation:
-            _AddConditionalProperty(properties, condition, "SpectreMitigation",
-                                    spectre_mitigation)
+            _AddConditionalProperty(
+                properties, condition, "SpectreMitigation", spectre_mitigation
+            )
         if config_type == "Driver":
             _AddConditionalProperty(properties, condition, "DriverType", "WDM")
             _AddConditionalProperty(
@@ -3193,7 +3195,7 @@ def _GetMSBuildConfigurationGlobalProperties(spec, configurations, build_file):
             new_paths = "$(ExecutablePath);" + ";".join(new_paths)
 
     properties = {}
-    for (name, configuration) in sorted(configurations.items()):
+    for name, configuration in sorted(configurations.items()):
         condition = _GetConfigurationCondition(name, configuration, spec)
         attributes = _GetMSBuildAttributes(spec, configuration, build_file)
         msbuild_settings = configuration["finalized_msbuild_settings"]
@@ -3232,14 +3234,14 @@ def _GetMSBuildConfigurationGlobalProperties(spec, configurations, build_file):
 def _AddConditionalProperty(properties, condition, name, value):
     """Adds a property / conditional value pair to a dictionary.
 
-  Arguments:
-    properties: The dictionary to be modified.  The key is the name of the
-        property.  The value is itself a dictionary; its key is the value and
-        the value a list of condition for which this value is true.
-    condition: The condition under which the named property has the value.
-    name: The name of the property.
-    value: The value of the property.
-  """
+    Arguments:
+      properties: The dictionary to be modified.  The key is the name of the
+          property.  The value is itself a dictionary; its key is the value and
+          the value a list of condition for which this value is true.
+      condition: The condition under which the named property has the value.
+      name: The name of the property.
+      value: The value of the property.
+    """
     if name not in properties:
         properties[name] = {}
     values = properties[name]
@@ -3256,13 +3258,13 @@ def _AddConditionalProperty(properties, condition, name, value):
 def _GetMSBuildPropertyGroup(spec, label, properties):
     """Returns a PropertyGroup definition for the specified properties.
 
-  Arguments:
-    spec: The target project dict.
-    label: An optional label for the PropertyGroup.
-    properties: The dictionary to be converted.  The key is the name of the
-        property.  The value is itself a dictionary; its key is the value and
-        the value a list of condition for which this value is true.
-  """
+    Arguments:
+      spec: The target project dict.
+      label: An optional label for the PropertyGroup.
+      properties: The dictionary to be converted.  The key is the name of the
+          property.  The value is itself a dictionary; its key is the value and
+          the value a list of condition for which this value is true.
+    """
     group = ["PropertyGroup"]
     if label:
         group.append({"Label": label})
@@ -3311,7 +3313,7 @@ def GetEdges(node):
 
 def _GetMSBuildToolSettingsSections(spec, configurations):
     groups = []
-    for (name, configuration) in sorted(configurations.items()):
+    for name, configuration in sorted(configurations.items()):
         msbuild_settings = configuration["finalized_msbuild_settings"]
         group = [
             "ItemDefinitionGroup",
@@ -3408,7 +3410,7 @@ def _FinalizeMSBuildSettings(spec, configuration):
         # While MSVC works with just file name eg. "v8_pch.h", ClangCL requires
         # the full path eg. "tools/msvs/pch/v8_pch.h" to find the file.
         # P.S. Only ClangCL defines msbuild_toolset, for MSVC it is None.
-        if configuration.get("msbuild_toolset") != 'ClangCL':
+        if configuration.get("msbuild_toolset") != "ClangCL":
             precompiled_header = os.path.split(precompiled_header)[1]
         _ToolAppend(msbuild_settings, "ClCompile", "PrecompiledHeader", "Use")
         _ToolAppend(
@@ -3470,16 +3472,16 @@ def _GetValueFormattedForMSBuild(tool_name, name, value):
 def _VerifySourcesExist(sources, root_dir):
     """Verifies that all source files exist on disk.
 
-  Checks that all regular source files, i.e. not created at run time,
-  exist on disk.  Missing files cause needless recompilation but no otherwise
-  visible errors.
+    Checks that all regular source files, i.e. not created at run time,
+    exist on disk.  Missing files cause needless recompilation but no otherwise
+    visible errors.
 
-  Arguments:
-    sources: A recursive list of Filter/file names.
-    root_dir: The root directory for the relative path names.
-  Returns:
-    A list of source files that cannot be found on disk.
-  """
+    Arguments:
+      sources: A recursive list of Filter/file names.
+      root_dir: The root directory for the relative path names.
+    Returns:
+      A list of source files that cannot be found on disk.
+    """
     missing_sources = []
     for source in sources:
         if isinstance(source, MSVSProject.Filter):
@@ -3564,17 +3566,13 @@ def _AddSources2(
                 detail.append(["ExcludedFromBuild", "true"])
             else:
                 for config_name, configuration in sorted(excluded_configurations):
-                    condition = _GetConfigurationCondition(
-                        config_name, configuration
-                    )
+                    condition = _GetConfigurationCondition(config_name, configuration)
                     detail.append(
                         ["ExcludedFromBuild", {"Condition": condition}, "true"]
                     )
             # Add precompile if needed
             for config_name, configuration in spec["configurations"].items():
-                precompiled_source = configuration.get(
-                    "msvs_precompiled_source", ""
-                )
+                precompiled_source = configuration.get("msvs_precompiled_source", "")
                 if precompiled_source != "":
                     precompiled_source = _FixPath(precompiled_source)
                     if not extensions_excluded_from_precompile:
@@ -3822,15 +3820,15 @@ def _GenerateMSBuildProject(project, options, version, generator_flags, spec):
 def _GetMSBuildExternalBuilderTargets(spec):
     """Return a list of MSBuild targets for external builders.
 
-  The "Build" and "Clean" targets are always generated.  If the spec contains
-  'msvs_external_builder_clcompile_cmd', then the "ClCompile" target will also
-  be generated, to support building selected C/C++ files.
+    The "Build" and "Clean" targets are always generated.  If the spec contains
+    'msvs_external_builder_clcompile_cmd', then the "ClCompile" target will also
+    be generated, to support building selected C/C++ files.
 
-  Arguments:
-    spec: The gyp target spec.
-  Returns:
-    List of MSBuild 'Target' specs.
-  """
+    Arguments:
+      spec: The gyp target spec.
+    Returns:
+      List of MSBuild 'Target' specs.
+    """
     build_cmd = _BuildCommandLineForRuleRaw(
         spec, spec["msvs_external_builder_build_cmd"], False, False, False, False
     )
@@ -3878,14 +3876,14 @@ def _GetMSBuildExtensionTargets(targets_files_of_rules):
 def _GenerateActionsForMSBuild(spec, actions_to_add):
     """Add actions accumulated into an actions_to_add, merging as needed.
 
-  Arguments:
-    spec: the target project dict
-    actions_to_add: dictionary keyed on input name, which maps to a list of
-        dicts describing the actions attached to that input file.
+    Arguments:
+      spec: the target project dict
+      actions_to_add: dictionary keyed on input name, which maps to a list of
+          dicts describing the actions attached to that input file.
 
-  Returns:
-    A pair of (action specification, the sources handled by this action).
-  """
+    Returns:
+      A pair of (action specification, the sources handled by this action).
+    """
     sources_handled_by_action = OrderedSet()
     actions_spec = []
     for primary_input, actions in actions_to_add.items():
diff --git a/gyp/pylib/gyp/generator/msvs_test.py b/gyp/pylib/gyp/generator/msvs_test.py
index 8cea3d1479..e3c4758696 100755
--- a/gyp/pylib/gyp/generator/msvs_test.py
+++ b/gyp/pylib/gyp/generator/msvs_test.py
@@ -3,7 +3,7 @@
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
-""" Unit tests for the msvs.py file. """
+"""Unit tests for the msvs.py file."""
 
 import unittest
 from io import StringIO
diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py
index d5dfa1a118..bc9ddd2654 100644
--- a/gyp/pylib/gyp/generator/ninja.py
+++ b/gyp/pylib/gyp/generator/ninja.py
@@ -1303,7 +1303,7 @@ def WritePchTargets(self, ninja_file, pch_commands):
             ninja_file.build(gch, cmd, input, variables=[(var_name, lang_flag)])
 
     def WriteLink(self, spec, config_name, config, link_deps, compile_deps):
-        """Write out a link step. Fills out target.binary. """
+        """Write out a link step. Fills out target.binary."""
         if self.flavor != "mac" or len(self.archs) == 1:
             return self.WriteLinkForArch(
                 self.ninja, spec, config_name, config, link_deps, compile_deps
@@ -1347,7 +1347,7 @@ def WriteLink(self, spec, config_name, config, link_deps, compile_deps):
     def WriteLinkForArch(
         self, ninja_file, spec, config_name, config, link_deps, compile_deps, arch=None
     ):
-        """Write out a link step. Fills out target.binary. """
+        """Write out a link step. Fills out target.binary."""
         command = {
             "executable": "link",
             "loadable_module": "solink_module",
@@ -1755,11 +1755,9 @@ def GetPostbuildCommand(self, spec, output, output_binary, is_command_start):
             + " && ".join([ninja_syntax.escape(command) for command in postbuilds])
         )
         command_string = (
-            commands
-            + "); G=$$?; "
+            commands + "); G=$$?; "
             # Remove the final output if any postbuild failed.
-            "((exit $$G) || rm -rf %s) " % output
-            + "&& exit $$G)"
+            "((exit $$G) || rm -rf %s) " % output + "&& exit $$G)"
         )
         if is_command_start:
             return "(" + command_string + " && "
@@ -1948,7 +1946,8 @@ def WriteNewNinjaRule(
                 )
             else:
                 rspfile_content = gyp.msvs_emulation.EncodeRspFileList(
-                    args, win_shell_flags.quote)
+                    args, win_shell_flags.quote
+                )
             command = (
                 "%s gyp-win-tool action-wrapper $arch " % sys.executable
                 + rspfile
@@ -2085,6 +2084,7 @@ def GetDefaultConcurrentLinks():
         return pool_size
 
     if sys.platform in ("win32", "cygwin"):
+
         class MEMORYSTATUSEX(ctypes.Structure):
             _fields_ = [
                 ("dwLength", ctypes.c_ulong),
@@ -2104,8 +2104,8 @@ class MEMORYSTATUSEX(ctypes.Structure):
 
         # VS 2015 uses 20% more working set than VS 2013 and can consume all RAM
         # on a 64 GiB machine.
-        mem_limit = max(1, stat.ullTotalPhys // (5 * (2 ** 30)))  # total / 5GiB
-        hard_cap = max(1, int(os.environ.get("GYP_LINK_CONCURRENCY_MAX") or 2 ** 32))
+        mem_limit = max(1, stat.ullTotalPhys // (5 * (2**30)))  # total / 5GiB
+        hard_cap = max(1, int(os.environ.get("GYP_LINK_CONCURRENCY_MAX") or 2**32))
         return min(mem_limit, hard_cap)
     elif sys.platform.startswith("linux"):
         if os.path.exists("/proc/meminfo"):
@@ -2116,14 +2116,14 @@ class MEMORYSTATUSEX(ctypes.Structure):
                     if not match:
                         continue
                     # Allow 8Gb per link on Linux because Gold is quite memory hungry
-                    return max(1, int(match.group(1)) // (8 * (2 ** 20)))
+                    return max(1, int(match.group(1)) // (8 * (2**20)))
         return 1
     elif sys.platform == "darwin":
         try:
             avail_bytes = int(subprocess.check_output(["sysctl", "-n", "hw.memsize"]))
             # A static library debug build of Chromium's unit_tests takes ~2.7GB, so
             # 4GB per ld process allows for some more bloat.
-            return max(1, avail_bytes // (4 * (2 ** 30)))  # total / 4GB
+            return max(1, avail_bytes // (4 * (2**30)))  # total / 4GB
         except subprocess.CalledProcessError:
             return 1
     else:
@@ -2411,8 +2411,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name
             "cc_s",
             description="CC $out",
             command=(
-                "$cc $defines $includes $cflags $cflags_c "
-                "$cflags_pch_c -c $in -o $out"
+                "$cc $defines $includes $cflags $cflags_c $cflags_pch_c -c $in -o $out"
             ),
         )
         master_ninja.rule(
@@ -2523,8 +2522,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name
             "solink",
             description="SOLINK $lib",
             restat=True,
-            command=mtime_preserving_solink_base
-            % {"suffix": "@$link_file_list"},
+            command=mtime_preserving_solink_base % {"suffix": "@$link_file_list"},
             rspfile="$link_file_list",
             rspfile_content=(
                 "-Wl,--whole-archive $in $solibs -Wl,--no-whole-archive $libs"
@@ -2709,7 +2707,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name
             command="$env %(python)s gyp-mac-tool compile-ios-framework-header-map "
             "$out $framework $in && $env %(python)s gyp-mac-tool "
             "copy-ios-framework-headers $framework $copy_headers"
-            % {'python': sys.executable},
+            % {"python": sys.executable},
         )
         master_ninja.rule(
             "mac_tool",
diff --git a/gyp/pylib/gyp/generator/ninja_test.py b/gyp/pylib/gyp/generator/ninja_test.py
index 581b14595e..616bc7aaf0 100644
--- a/gyp/pylib/gyp/generator/ninja_test.py
+++ b/gyp/pylib/gyp/generator/ninja_test.py
@@ -4,7 +4,7 @@
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
-""" Unit tests for the ninja.py file. """
+"""Unit tests for the ninja.py file."""
 
 import sys
 import unittest
diff --git a/gyp/pylib/gyp/generator/xcode.py b/gyp/pylib/gyp/generator/xcode.py
index cdf11c3b27..8e05657961 100644
--- a/gyp/pylib/gyp/generator/xcode.py
+++ b/gyp/pylib/gyp/generator/xcode.py
@@ -564,12 +564,12 @@ def AddHeaderToTarget(header, pbxp, xct, is_public):
 def ExpandXcodeVariables(string, expansions):
     """Expands Xcode-style $(VARIABLES) in string per the expansions dict.
 
-  In some rare cases, it is appropriate to expand Xcode variables when a
-  project file is generated.  For any substring $(VAR) in string, if VAR is a
-  key in the expansions dict, $(VAR) will be replaced with expansions[VAR].
-  Any $(VAR) substring in string for which VAR is not a key in the expansions
-  dict will remain in the returned string.
-  """
+    In some rare cases, it is appropriate to expand Xcode variables when a
+    project file is generated.  For any substring $(VAR) in string, if VAR is a
+    key in the expansions dict, $(VAR) will be replaced with expansions[VAR].
+    Any $(VAR) substring in string for which VAR is not a key in the expansions
+    dict will remain in the returned string.
+    """
 
     matches = _xcode_variable_re.findall(string)
     if matches is None:
@@ -592,9 +592,9 @@ def ExpandXcodeVariables(string, expansions):
 
 def EscapeXcodeDefine(s):
     """We must escape the defines that we give to XCode so that it knows not to
-     split on spaces and to respect backslash and quote literals. However, we
-     must not quote the define, or Xcode will incorrectly interpret variables
-     especially $(inherited)."""
+    split on spaces and to respect backslash and quote literals. However, we
+    must not quote the define, or Xcode will incorrectly interpret variables
+    especially $(inherited)."""
     return re.sub(_xcode_define_re, r"\\\1", s)
 
 
@@ -679,9 +679,9 @@ def GenerateOutput(target_list, target_dicts, data, params):
             project_attributes["BuildIndependentTargetsInParallel"] = "YES"
         if upgrade_check_project_version:
             project_attributes["LastUpgradeCheck"] = upgrade_check_project_version
-            project_attributes[
-                "LastTestingUpgradeCheck"
-            ] = upgrade_check_project_version
+            project_attributes["LastTestingUpgradeCheck"] = (
+                upgrade_check_project_version
+            )
             project_attributes["LastSwiftUpdateCheck"] = upgrade_check_project_version
         pbxp.SetProperty("attributes", project_attributes)
 
@@ -734,8 +734,7 @@ def GenerateOutput(target_list, target_dicts, data, params):
             "loadable_module+xcuitest": "com.apple.product-type.bundle.ui-testing",
             "shared_library+bundle": "com.apple.product-type.framework",
             "executable+extension+bundle": "com.apple.product-type.app-extension",
-            "executable+watch+extension+bundle":
-                "com.apple.product-type.watchkit-extension",
+            "executable+watch+extension+bundle": "com.apple.product-type.watchkit-extension",  # noqa: E501
             "executable+watch+bundle": "com.apple.product-type.application.watchapp",
             "mac_kernel_extension+bundle": "com.apple.product-type.kernel-extension",
         }
@@ -780,8 +779,7 @@ def GenerateOutput(target_list, target_dicts, data, params):
                 type_bundle_key += "+watch+extension+bundle"
             elif is_watch_app:
                 assert is_bundle, (
-                    "ios_watch_app flag requires mac_bundle "
-                    "(target %s)" % target_name
+                    "ios_watch_app flag requires mac_bundle (target %s)" % target_name
                 )
                 type_bundle_key += "+watch+bundle"
             elif is_bundle:
@@ -1103,7 +1101,7 @@ def GenerateOutput(target_list, target_dicts, data, params):
                         eol = " \\"
                     makefile.write(f"    {concrete_output}{eol}\n")
 
-                for (rule_source, concrete_outputs, message, action) in zip(
+                for rule_source, concrete_outputs, message, action in zip(
                     rule["rule_sources"],
                     concrete_outputs_by_rule_source,
                     messages,
diff --git a/gyp/pylib/gyp/generator/xcode_test.py b/gyp/pylib/gyp/generator/xcode_test.py
index b0b51a08a6..bfd8c587a3 100644
--- a/gyp/pylib/gyp/generator/xcode_test.py
+++ b/gyp/pylib/gyp/generator/xcode_test.py
@@ -4,7 +4,7 @@
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
-""" Unit tests for the xcode.py file. """
+"""Unit tests for the xcode.py file."""
 
 import sys
 import unittest
diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py
index 994bf6625f..4965ff1571 100644
--- a/gyp/pylib/gyp/input.py
+++ b/gyp/pylib/gyp/input.py
@@ -139,21 +139,21 @@ def IsPathSection(section):
 def GetIncludedBuildFiles(build_file_path, aux_data, included=None):
     """Return a list of all build files included into build_file_path.
 
-  The returned list will contain build_file_path as well as all other files
-  that it included, either directly or indirectly.  Note that the list may
-  contain files that were included into a conditional section that evaluated
-  to false and was not merged into build_file_path's dict.
+    The returned list will contain build_file_path as well as all other files
+    that it included, either directly or indirectly.  Note that the list may
+    contain files that were included into a conditional section that evaluated
+    to false and was not merged into build_file_path's dict.
 
-  aux_data is a dict containing a key for each build file or included build
-  file.  Those keys provide access to dicts whose "included" keys contain
-  lists of all other files included by the build file.
+    aux_data is a dict containing a key for each build file or included build
+    file.  Those keys provide access to dicts whose "included" keys contain
+    lists of all other files included by the build file.
 
-  included should be left at its default None value by external callers.  It
-  is used for recursion.
+    included should be left at its default None value by external callers.  It
+    is used for recursion.
 
-  The returned list will not contain any duplicate entries.  Each build file
-  in the list will be relative to the current directory.
-  """
+    The returned list will not contain any duplicate entries.  Each build file
+    in the list will be relative to the current directory.
+    """
 
     if included is None:
         included = []
@@ -171,10 +171,10 @@ def GetIncludedBuildFiles(build_file_path, aux_data, included=None):
 
 def CheckedEval(file_contents):
     """Return the eval of a gyp file.
-  The gyp file is restricted to dictionaries and lists only, and
-  repeated keys are not allowed.
-  Note that this is slower than eval() is.
-  """
+    The gyp file is restricted to dictionaries and lists only, and
+    repeated keys are not allowed.
+    Note that this is slower than eval() is.
+    """
 
     syntax_tree = ast.parse(file_contents)
     assert isinstance(syntax_tree, ast.Module)
@@ -508,9 +508,9 @@ def CallLoadTargetBuildFile(
 ):
     """Wrapper around LoadTargetBuildFile for parallel processing.
 
-     This wrapper is used when LoadTargetBuildFile is executed in
-     a worker process.
-  """
+    This wrapper is used when LoadTargetBuildFile is executed in
+    a worker process.
+    """
 
     try:
         signal.signal(signal.SIGINT, signal.SIG_IGN)
@@ -559,10 +559,10 @@ class ParallelProcessingError(Exception):
 class ParallelState:
     """Class to keep track of state when processing input files in parallel.
 
-  If build files are loaded in parallel, use this to keep track of
-  state during farming out and processing parallel jobs. It's stored
-  in a global so that the callback function can have access to it.
-  """
+    If build files are loaded in parallel, use this to keep track of
+    state during farming out and processing parallel jobs. It's stored
+    in a global so that the callback function can have access to it.
+    """
 
     def __init__(self):
         # The multiprocessing pool.
@@ -584,8 +584,7 @@ def __init__(self):
         self.error = False
 
     def LoadTargetBuildFileCallback(self, result):
-        """Handle the results of running LoadTargetBuildFile in another process.
-    """
+        """Handle the results of running LoadTargetBuildFile in another process."""
         self.condition.acquire()
         if not result:
             self.error = True
@@ -692,8 +691,8 @@ def FindEnclosingBracketGroup(input_str):
 def IsStrCanonicalInt(string):
     """Returns True if |string| is in its canonical integer form.
 
-  The canonical form is such that str(int(string)) == string.
-  """
+    The canonical form is such that str(int(string)) == string.
+    """
     if isinstance(string, str):
         # This function is called a lot so for maximum performance, avoid
         # involving regexps which would otherwise make the code much
@@ -870,8 +869,9 @@ def ExpandVariables(input, phase, variables, build_file):
         # This works around actions/rules which have more inputs than will
         # fit on the command line.
         if file_list:
-            contents_list = (contents if isinstance(contents, list)
-                             else contents.split(" "))
+            contents_list = (
+                contents if isinstance(contents, list) else contents.split(" ")
+            )
             replacement = contents_list[0]
             if os.path.isabs(replacement):
                 raise GypError('| cannot handle absolute paths, got "%s"' % replacement)
@@ -934,7 +934,6 @@ def ExpandVariables(input, phase, variables, build_file):
                         os.chdir(build_file_dir)
                     sys.path.append(os.getcwd())
                     try:
-
                         parsed_contents = shlex.split(contents)
                         try:
                             py_module = __import__(parsed_contents[0])
@@ -965,7 +964,7 @@ def ExpandVariables(input, phase, variables, build_file):
                             stdout=subprocess.PIPE,
                             shell=use_shell,
                             cwd=build_file_dir,
-                            check=False
+                            check=False,
                         )
                     except Exception as e:
                         raise GypError(
@@ -1003,9 +1002,7 @@ def ExpandVariables(input, phase, variables, build_file):
                 # ],
                 replacement = []
             else:
-                raise GypError(
-                    "Undefined variable " + contents + " in " + build_file
-                )
+                raise GypError("Undefined variable " + contents + " in " + build_file)
         else:
             replacement = variables[contents]
 
@@ -1114,7 +1111,7 @@ def ExpandVariables(input, phase, variables, build_file):
 
 def EvalCondition(condition, conditions_key, phase, variables, build_file):
     """Returns the dict that should be used or None if the result was
-  that nothing should be used."""
+    that nothing should be used."""
     if not isinstance(condition, list):
         raise GypError(conditions_key + " must be a list")
     if len(condition) < 2:
@@ -1159,7 +1156,7 @@ def EvalCondition(condition, conditions_key, phase, variables, build_file):
 
 def EvalSingleCondition(cond_expr, true_dict, false_dict, phase, variables, build_file):
     """Returns true_dict if cond_expr evaluates to true, and false_dict
-  otherwise."""
+    otherwise."""
     # Do expansions on the condition itself.  Since the condition can naturally
     # contain variable references without needing to resort to GYP expansion
     # syntax, this is of dubious value for variables, but someone might want to
@@ -1289,10 +1286,10 @@ def ProcessVariablesAndConditionsInDict(
 ):
     """Handle all variable and command expansion and conditional evaluation.
 
-  This function is the public entry point for all variable expansions and
-  conditional evaluations.  The variables_in dictionary will not be modified
-  by this function.
-  """
+    This function is the public entry point for all variable expansions and
+    conditional evaluations.  The variables_in dictionary will not be modified
+    by this function.
+    """
 
     # Make a copy of the variables_in dict that can be modified during the
     # loading of automatics and the loading of the variables dict.
@@ -1441,15 +1438,15 @@ def ProcessVariablesAndConditionsInList(the_list, phase, variables, build_file):
 def BuildTargetsDict(data):
     """Builds a dict mapping fully-qualified target names to their target dicts.
 
-  |data| is a dict mapping loaded build files by pathname relative to the
-  current directory.  Values in |data| are build file contents.  For each
-  |data| value with a "targets" key, the value of the "targets" key is taken
-  as a list containing target dicts.  Each target's fully-qualified name is
-  constructed from the pathname of the build file (|data| key) and its
-  "target_name" property.  These fully-qualified names are used as the keys
-  in the returned dict.  These keys provide access to the target dicts,
-  the dicts in the "targets" lists.
-  """
+    |data| is a dict mapping loaded build files by pathname relative to the
+    current directory.  Values in |data| are build file contents.  For each
+    |data| value with a "targets" key, the value of the "targets" key is taken
+    as a list containing target dicts.  Each target's fully-qualified name is
+    constructed from the pathname of the build file (|data| key) and its
+    "target_name" property.  These fully-qualified names are used as the keys
+    in the returned dict.  These keys provide access to the target dicts,
+    the dicts in the "targets" lists.
+    """
 
     targets = {}
     for build_file in data["target_build_files"]:
@@ -1467,13 +1464,13 @@ def BuildTargetsDict(data):
 def QualifyDependencies(targets):
     """Make dependency links fully-qualified relative to the current directory.
 
-  |targets| is a dict mapping fully-qualified target names to their target
-  dicts.  For each target in this dict, keys known to contain dependency
-  links are examined, and any dependencies referenced will be rewritten
-  so that they are fully-qualified and relative to the current directory.
-  All rewritten dependencies are suitable for use as keys to |targets| or a
-  similar dict.
-  """
+    |targets| is a dict mapping fully-qualified target names to their target
+    dicts.  For each target in this dict, keys known to contain dependency
+    links are examined, and any dependencies referenced will be rewritten
+    so that they are fully-qualified and relative to the current directory.
+    All rewritten dependencies are suitable for use as keys to |targets| or a
+    similar dict.
+    """
 
     all_dependency_sections = [
         dep + op for dep in dependency_sections for op in ("", "!", "/")
@@ -1516,18 +1513,18 @@ def QualifyDependencies(targets):
 def ExpandWildcardDependencies(targets, data):
     """Expands dependencies specified as build_file:*.
 
-  For each target in |targets|, examines sections containing links to other
-  targets.  If any such section contains a link of the form build_file:*, it
-  is taken as a wildcard link, and is expanded to list each target in
-  build_file.  The |data| dict provides access to build file dicts.
+    For each target in |targets|, examines sections containing links to other
+    targets.  If any such section contains a link of the form build_file:*, it
+    is taken as a wildcard link, and is expanded to list each target in
+    build_file.  The |data| dict provides access to build file dicts.
 
-  Any target that does not wish to be included by wildcard can provide an
-  optional "suppress_wildcard" key in its target dict.  When present and
-  true, a wildcard dependency link will not include such targets.
+    Any target that does not wish to be included by wildcard can provide an
+    optional "suppress_wildcard" key in its target dict.  When present and
+    true, a wildcard dependency link will not include such targets.
 
-  All dependency names, including the keys to |targets| and the values in each
-  dependency list, must be qualified when this function is called.
-  """
+    All dependency names, including the keys to |targets| and the values in each
+    dependency list, must be qualified when this function is called.
+    """
 
     for target, target_dict in targets.items():
         target_build_file = gyp.common.BuildFile(target)
@@ -1573,14 +1570,10 @@ def ExpandWildcardDependencies(targets, data):
                     if int(dependency_target_dict.get("suppress_wildcard", False)):
                         continue
                     dependency_target_name = dependency_target_dict["target_name"]
-                    if (
-                        dependency_target not in {"*", dependency_target_name}
-                    ):
+                    if dependency_target not in {"*", dependency_target_name}:
                         continue
                     dependency_target_toolset = dependency_target_dict["toolset"]
-                    if (
-                        dependency_toolset not in {"*", dependency_target_toolset}
-                    ):
+                    if dependency_toolset not in {"*", dependency_target_toolset}:
                         continue
                     dependency = gyp.common.QualifiedTarget(
                         dependency_build_file,
@@ -1601,7 +1594,7 @@ def Unify(items):
 
 def RemoveDuplicateDependencies(targets):
     """Makes sure every dependency appears only once in all targets's dependency
-  lists."""
+    lists."""
     for target_name, target_dict in targets.items():
         for dependency_key in dependency_sections:
             dependencies = target_dict.get(dependency_key, [])
@@ -1617,25 +1610,21 @@ def Filter(items, item):
 
 def RemoveSelfDependencies(targets):
     """Remove self dependencies from targets that have the prune_self_dependency
-  variable set."""
+    variable set."""
     for target_name, target_dict in targets.items():
         for dependency_key in dependency_sections:
             dependencies = target_dict.get(dependency_key, [])
             if dependencies:
                 for t in dependencies:
                     if t == target_name and (
-                        targets[t]
-                        .get("variables", {})
-                        .get("prune_self_dependency", 0)
+                        targets[t].get("variables", {}).get("prune_self_dependency", 0)
                     ):
-                        target_dict[dependency_key] = Filter(
-                            dependencies, target_name
-                        )
+                        target_dict[dependency_key] = Filter(dependencies, target_name)
 
 
 def RemoveLinkDependenciesFromNoneTargets(targets):
     """Remove dependencies having the 'link_dependency' attribute from the 'none'
-  targets."""
+    targets."""
     for target_name, target_dict in targets.items():
         for dependency_key in dependency_sections:
             dependencies = target_dict.get(dependency_key, [])
@@ -1651,11 +1640,11 @@ def RemoveLinkDependenciesFromNoneTargets(targets):
 class DependencyGraphNode:
     """
 
-  Attributes:
-    ref: A reference to an object that this DependencyGraphNode represents.
-    dependencies: List of DependencyGraphNodes on which this one depends.
-    dependents: List of DependencyGraphNodes that depend on this one.
-  """
+    Attributes:
+      ref: A reference to an object that this DependencyGraphNode represents.
+      dependencies: List of DependencyGraphNodes on which this one depends.
+      dependents: List of DependencyGraphNodes that depend on this one.
+    """
 
     class CircularException(GypError):
         pass
@@ -1721,8 +1710,8 @@ def ExtractNodeRef(node):
 
     def FindCycles(self):
         """
-    Returns a list of cycles in the graph, where each cycle is its own list.
-    """
+        Returns a list of cycles in the graph, where each cycle is its own list.
+        """
         results = []
         visited = set()
 
@@ -1753,21 +1742,21 @@ def DirectDependencies(self, dependencies=None):
 
     def _AddImportedDependencies(self, targets, dependencies=None):
         """Given a list of direct dependencies, adds indirect dependencies that
-    other dependencies have declared to export their settings.
-
-    This method does not operate on self.  Rather, it operates on the list
-    of dependencies in the |dependencies| argument.  For each dependency in
-    that list, if any declares that it exports the settings of one of its
-    own dependencies, those dependencies whose settings are "passed through"
-    are added to the list.  As new items are added to the list, they too will
-    be processed, so it is possible to import settings through multiple levels
-    of dependencies.
-
-    This method is not terribly useful on its own, it depends on being
-    "primed" with a list of direct dependencies such as one provided by
-    DirectDependencies.  DirectAndImportedDependencies is intended to be the
-    public entry point.
-    """
+        other dependencies have declared to export their settings.
+
+        This method does not operate on self.  Rather, it operates on the list
+        of dependencies in the |dependencies| argument.  For each dependency in
+        that list, if any declares that it exports the settings of one of its
+        own dependencies, those dependencies whose settings are "passed through"
+        are added to the list.  As new items are added to the list, they too will
+        be processed, so it is possible to import settings through multiple levels
+        of dependencies.
+
+        This method is not terribly useful on its own, it depends on being
+        "primed" with a list of direct dependencies such as one provided by
+        DirectDependencies.  DirectAndImportedDependencies is intended to be the
+        public entry point.
+        """
 
         if dependencies is None:
             dependencies = []
@@ -1795,9 +1784,9 @@ def _AddImportedDependencies(self, targets, dependencies=None):
 
     def DirectAndImportedDependencies(self, targets, dependencies=None):
         """Returns a list of a target's direct dependencies and all indirect
-    dependencies that a dependency has advertised settings should be exported
-    through the dependency for.
-    """
+        dependencies that a dependency has advertised settings should be exported
+        through the dependency for.
+        """
 
         dependencies = self.DirectDependencies(dependencies)
         return self._AddImportedDependencies(targets, dependencies)
@@ -1823,19 +1812,19 @@ def _LinkDependenciesInternal(
         self, targets, include_shared_libraries, dependencies=None, initial=True
     ):
         """Returns an OrderedSet of dependency targets that are linked
-    into this target.
+        into this target.
 
-    This function has a split personality, depending on the setting of
-    |initial|.  Outside callers should always leave |initial| at its default
-    setting.
+        This function has a split personality, depending on the setting of
+        |initial|.  Outside callers should always leave |initial| at its default
+        setting.
 
-    When adding a target to the list of dependencies, this function will
-    recurse into itself with |initial| set to False, to collect dependencies
-    that are linked into the linkable target for which the list is being built.
+        When adding a target to the list of dependencies, this function will
+        recurse into itself with |initial| set to False, to collect dependencies
+        that are linked into the linkable target for which the list is being built.
 
-    If |include_shared_libraries| is False, the resulting dependencies will not
-    include shared_library targets that are linked into this target.
-    """
+        If |include_shared_libraries| is False, the resulting dependencies will not
+        include shared_library targets that are linked into this target.
+        """
         if dependencies is None:
             # Using a list to get ordered output and a set to do fast "is it
             # already added" checks.
@@ -1917,9 +1906,9 @@ def _LinkDependenciesInternal(
 
     def DependenciesForLinkSettings(self, targets):
         """
-    Returns a list of dependency targets whose link_settings should be merged
-    into this target.
-    """
+        Returns a list of dependency targets whose link_settings should be merged
+        into this target.
+        """
 
         # TODO(sbaig) Currently, chrome depends on the bug that shared libraries'
         # link_settings are propagated.  So for now, we will allow it, unless the
@@ -1932,8 +1921,8 @@ def DependenciesForLinkSettings(self, targets):
 
     def DependenciesToLinkAgainst(self, targets):
         """
-    Returns a list of dependency targets that are linked into this target.
-    """
+        Returns a list of dependency targets that are linked into this target.
+        """
         return self._LinkDependenciesInternal(targets, True)
 
 
@@ -2446,7 +2435,7 @@ def SetUpConfigurations(target, target_dict):
 
     merged_configurations = {}
     configs = target_dict["configurations"]
-    for (configuration, old_configuration_dict) in configs.items():
+    for configuration, old_configuration_dict in configs.items():
         # Skip abstract configurations (saves work only).
         if old_configuration_dict.get("abstract"):
             continue
@@ -2454,7 +2443,7 @@ def SetUpConfigurations(target, target_dict):
         # Get the inheritance relationship right by making a copy of the target
         # dict.
         new_configuration_dict = {}
-        for (key, target_val) in target_dict.items():
+        for key, target_val in target_dict.items():
             key_ext = key[-1:]
             key_base = key[:-1] if key_ext in key_suffixes else key
             if key_base not in non_configuration_keys:
@@ -2502,25 +2491,25 @@ def SetUpConfigurations(target, target_dict):
 def ProcessListFiltersInDict(name, the_dict):
     """Process regular expression and exclusion-based filters on lists.
 
-  An exclusion list is in a dict key named with a trailing "!", like
-  "sources!".  Every item in such a list is removed from the associated
-  main list, which in this example, would be "sources".  Removed items are
-  placed into a "sources_excluded" list in the dict.
-
-  Regular expression (regex) filters are contained in dict keys named with a
-  trailing "/", such as "sources/" to operate on the "sources" list.  Regex
-  filters in a dict take the form:
-    'sources/': [ ['exclude', '_(linux|mac|win)\\.cc$'],
-                  ['include', '_mac\\.cc$'] ],
-  The first filter says to exclude all files ending in _linux.cc, _mac.cc, and
-  _win.cc.  The second filter then includes all files ending in _mac.cc that
-  are now or were once in the "sources" list.  Items matching an "exclude"
-  filter are subject to the same processing as would occur if they were listed
-  by name in an exclusion list (ending in "!").  Items matching an "include"
-  filter are brought back into the main list if previously excluded by an
-  exclusion list or exclusion regex filter.  Subsequent matching "exclude"
-  patterns can still cause items to be excluded after matching an "include".
-  """
+    An exclusion list is in a dict key named with a trailing "!", like
+    "sources!".  Every item in such a list is removed from the associated
+    main list, which in this example, would be "sources".  Removed items are
+    placed into a "sources_excluded" list in the dict.
+
+    Regular expression (regex) filters are contained in dict keys named with a
+    trailing "/", such as "sources/" to operate on the "sources" list.  Regex
+    filters in a dict take the form:
+      'sources/': [ ['exclude', '_(linux|mac|win)\\.cc$'],
+                    ['include', '_mac\\.cc$'] ],
+    The first filter says to exclude all files ending in _linux.cc, _mac.cc, and
+    _win.cc.  The second filter then includes all files ending in _mac.cc that
+    are now or were once in the "sources" list.  Items matching an "exclude"
+    filter are subject to the same processing as would occur if they were listed
+    by name in an exclusion list (ending in "!").  Items matching an "include"
+    filter are brought back into the main list if previously excluded by an
+    exclusion list or exclusion regex filter.  Subsequent matching "exclude"
+    patterns can still cause items to be excluded after matching an "include".
+    """
 
     # Look through the dictionary for any lists whose keys end in "!" or "/".
     # These are lists that will be treated as exclude lists and regular
@@ -2682,12 +2671,12 @@ def ProcessListFiltersInList(name, the_list):
 def ValidateTargetType(target, target_dict):
     """Ensures the 'type' field on the target is one of the known types.
 
-  Arguments:
-    target: string, name of target.
-    target_dict: dict, target spec.
+    Arguments:
+      target: string, name of target.
+      target_dict: dict, target spec.
 
-  Raises an exception on error.
-  """
+    Raises an exception on error.
+    """
     VALID_TARGET_TYPES = (
         "executable",
         "loadable_module",
@@ -2715,14 +2704,14 @@ def ValidateTargetType(target, target_dict):
 
 def ValidateRulesInTarget(target, target_dict, extra_sources_for_rules):
     """Ensures that the rules sections in target_dict are valid and consistent,
-  and determines which sources they apply to.
+    and determines which sources they apply to.
 
-  Arguments:
-    target: string, name of target.
-    target_dict: dict, target spec containing "rules" and "sources" lists.
-    extra_sources_for_rules: a list of keys to scan for rule matches in
-        addition to 'sources'.
-  """
+    Arguments:
+      target: string, name of target.
+      target_dict: dict, target spec containing "rules" and "sources" lists.
+      extra_sources_for_rules: a list of keys to scan for rule matches in
+          addition to 'sources'.
+    """
 
     # Dicts to map between values found in rules' 'rule_name' and 'extension'
     # keys and the rule dicts themselves.
@@ -2734,9 +2723,7 @@ def ValidateRulesInTarget(target, target_dict, extra_sources_for_rules):
         # Make sure that there's no conflict among rule names and extensions.
         rule_name = rule["rule_name"]
         if rule_name in rule_names:
-            raise GypError(
-                f"rule {rule_name} exists in duplicate, target {target}"
-            )
+            raise GypError(f"rule {rule_name} exists in duplicate, target {target}")
         rule_names[rule_name] = rule
 
         rule_extension = rule["extension"]
@@ -2835,8 +2822,7 @@ def ValidateActionsInTarget(target, target_dict, build_file):
 
 
 def TurnIntIntoStrInDict(the_dict):
-    """Given dict the_dict, recursively converts all integers into strings.
-  """
+    """Given dict the_dict, recursively converts all integers into strings."""
     # Use items instead of iteritems because there's no need to try to look at
     # reinserted keys and their associated values.
     for k, v in the_dict.items():
@@ -2854,8 +2840,7 @@ def TurnIntIntoStrInDict(the_dict):
 
 
 def TurnIntIntoStrInList(the_list):
-    """Given list the_list, recursively converts all integers into strings.
-  """
+    """Given list the_list, recursively converts all integers into strings."""
     for index, item in enumerate(the_list):
         if isinstance(item, int):
             the_list[index] = str(item)
@@ -2902,9 +2887,9 @@ def PruneUnwantedTargets(targets, flat_list, dependency_nodes, root_targets, dat
 def VerifyNoCollidingTargets(targets):
     """Verify that no two targets in the same directory share the same name.
 
-  Arguments:
-    targets: A list of targets in the form 'path/to/file.gyp:target_name'.
-  """
+    Arguments:
+      targets: A list of targets in the form 'path/to/file.gyp:target_name'.
+    """
     # Keep a dict going from 'subdirectory:target_name' to 'foo.gyp'.
     used = {}
     for target in targets:
diff --git a/gyp/pylib/gyp/mac_tool.py b/gyp/pylib/gyp/mac_tool.py
index 58fb9c7398..3710178e11 100755
--- a/gyp/pylib/gyp/mac_tool.py
+++ b/gyp/pylib/gyp/mac_tool.py
@@ -8,7 +8,6 @@
 These functions are executed via gyp-mac-tool when using the Makefile generator.
 """
 
-
 import fcntl
 import fnmatch
 import glob
@@ -31,7 +30,7 @@ def main(args):
 
 class MacTool:
     """This class performs all the Mac tooling steps. The methods can either be
-  executed directly, or dispatched from an argument list."""
+    executed directly, or dispatched from an argument list."""
 
     def Dispatch(self, args):
         """Dispatches a string command to a method."""
@@ -47,7 +46,7 @@ def _CommandifyName(self, name_string):
 
     def ExecCopyBundleResource(self, source, dest, convert_to_binary):
         """Copies a resource file to the bundle/Resources directory, performing any
-    necessary compilation on each resource."""
+        necessary compilation on each resource."""
         convert_to_binary = convert_to_binary == "True"
         extension = os.path.splitext(source)[1].lower()
         if os.path.isdir(source):
@@ -155,15 +154,15 @@ def _CopyStringsFile(self, source, dest):
 
     def _DetectInputEncoding(self, file_name):
         """Reads the first few bytes from file_name and tries to guess the text
-    encoding. Returns None as a guess if it can't detect it."""
+        encoding. Returns None as a guess if it can't detect it."""
         with open(file_name, "rb") as fp:
             try:
                 header = fp.read(3)
             except Exception:
                 return None
-        if header.startswith((b"\xFE\xFF", b"\xFF\xFE")):
+        if header.startswith((b"\xfe\xff", b"\xff\xfe")):
             return "UTF-16"
-        elif header.startswith(b"\xEF\xBB\xBF"):
+        elif header.startswith(b"\xef\xbb\xbf"):
             return "UTF-8"
         else:
             return None
@@ -254,7 +253,7 @@ def ExecFlock(self, lockfile, *cmd_list):
 
     def ExecFilterLibtool(self, *cmd_list):
         """Calls libtool and filters out '/path/to/libtool: file: foo.o has no
-    symbols'."""
+        symbols'."""
         libtool_re = re.compile(
             r"^.*libtool: (?:for architecture: \S* )?file: .* has no symbols$"
         )
@@ -303,7 +302,7 @@ def ExecPackageIosFramework(self, framework):
 
     def ExecPackageFramework(self, framework, version):
         """Takes a path to Something.framework and the Current version of that and
-    sets up all the symlinks."""
+        sets up all the symlinks."""
         # Find the name of the binary based on the part before the ".framework".
         binary = os.path.basename(framework).split(".")[0]
 
@@ -332,7 +331,7 @@ def ExecPackageFramework(self, framework, version):
 
     def _Relink(self, dest, link):
         """Creates a symlink to |dest| named |link|. If |link| already exists,
-    it is overwritten."""
+        it is overwritten."""
         if os.path.lexists(link):
             os.remove(link)
         os.symlink(dest, link)
@@ -357,14 +356,14 @@ def ExecCopyIosFrameworkHeaders(self, framework, *copy_headers):
     def ExecCompileXcassets(self, keys, *inputs):
         """Compiles multiple .xcassets files into a single .car file.
 
-    This invokes 'actool' to compile all the inputs .xcassets files. The
-    |keys| arguments is a json-encoded dictionary of extra arguments to
-    pass to 'actool' when the asset catalogs contains an application icon
-    or a launch image.
+        This invokes 'actool' to compile all the inputs .xcassets files. The
+        |keys| arguments is a json-encoded dictionary of extra arguments to
+        pass to 'actool' when the asset catalogs contains an application icon
+        or a launch image.
 
-    Note that 'actool' does not create the Assets.car file if the asset
-    catalogs does not contains imageset.
-    """
+        Note that 'actool' does not create the Assets.car file if the asset
+        catalogs does not contains imageset.
+        """
         command_line = [
             "xcrun",
             "actool",
@@ -437,13 +436,13 @@ def ExecMergeInfoPlist(self, output, *inputs):
     def ExecCodeSignBundle(self, key, entitlements, provisioning, path, preserve):
         """Code sign a bundle.
 
-    This function tries to code sign an iOS bundle, following the same
-    algorithm as Xcode:
-      1. pick the provisioning profile that best match the bundle identifier,
-         and copy it into the bundle as embedded.mobileprovision,
-      2. copy Entitlements.plist from user or SDK next to the bundle,
-      3. code sign the bundle.
-    """
+        This function tries to code sign an iOS bundle, following the same
+        algorithm as Xcode:
+          1. pick the provisioning profile that best match the bundle identifier,
+             and copy it into the bundle as embedded.mobileprovision,
+          2. copy Entitlements.plist from user or SDK next to the bundle,
+          3. code sign the bundle.
+        """
         substitutions, overrides = self._InstallProvisioningProfile(
             provisioning, self._GetCFBundleIdentifier()
         )
@@ -462,16 +461,16 @@ def ExecCodeSignBundle(self, key, entitlements, provisioning, path, preserve):
     def _InstallProvisioningProfile(self, profile, bundle_identifier):
         """Installs embedded.mobileprovision into the bundle.
 
-    Args:
-      profile: string, optional, short name of the .mobileprovision file
-        to use, if empty or the file is missing, the best file installed
-        will be used
-      bundle_identifier: string, value of CFBundleIdentifier from Info.plist
+        Args:
+          profile: string, optional, short name of the .mobileprovision file
+            to use, if empty or the file is missing, the best file installed
+            will be used
+          bundle_identifier: string, value of CFBundleIdentifier from Info.plist
 
-    Returns:
-      A tuple containing two dictionary: variables substitutions and values
-      to overrides when generating the entitlements file.
-    """
+        Returns:
+          A tuple containing two dictionary: variables substitutions and values
+          to overrides when generating the entitlements file.
+        """
         source_path, provisioning_data, team_id = self._FindProvisioningProfile(
             profile, bundle_identifier
         )
@@ -487,24 +486,24 @@ def _InstallProvisioningProfile(self, profile, bundle_identifier):
     def _FindProvisioningProfile(self, profile, bundle_identifier):
         """Finds the .mobileprovision file to use for signing the bundle.
 
-    Checks all the installed provisioning profiles (or if the user specified
-    the PROVISIONING_PROFILE variable, only consult it) and select the most
-    specific that correspond to the bundle identifier.
+        Checks all the installed provisioning profiles (or if the user specified
+        the PROVISIONING_PROFILE variable, only consult it) and select the most
+        specific that correspond to the bundle identifier.
 
-    Args:
-      profile: string, optional, short name of the .mobileprovision file
-        to use, if empty or the file is missing, the best file installed
-        will be used
-      bundle_identifier: string, value of CFBundleIdentifier from Info.plist
+        Args:
+          profile: string, optional, short name of the .mobileprovision file
+            to use, if empty or the file is missing, the best file installed
+            will be used
+          bundle_identifier: string, value of CFBundleIdentifier from Info.plist
 
-    Returns:
-      A tuple of the path to the selected provisioning profile, the data of
-      the embedded plist in the provisioning profile and the team identifier
-      to use for code signing.
+        Returns:
+          A tuple of the path to the selected provisioning profile, the data of
+          the embedded plist in the provisioning profile and the team identifier
+          to use for code signing.
 
-    Raises:
-      SystemExit: if no .mobileprovision can be used to sign the bundle.
-    """
+        Raises:
+          SystemExit: if no .mobileprovision can be used to sign the bundle.
+        """
         profiles_dir = os.path.join(
             os.environ["HOME"], "Library", "MobileDevice", "Provisioning Profiles"
         )
@@ -552,12 +551,12 @@ def _FindProvisioningProfile(self, profile, bundle_identifier):
     def _LoadProvisioningProfile(self, profile_path):
         """Extracts the plist embedded in a provisioning profile.
 
-    Args:
-      profile_path: string, path to the .mobileprovision file
+        Args:
+          profile_path: string, path to the .mobileprovision file
 
-    Returns:
-      Content of the plist embedded in the provisioning profile as a dictionary.
-    """
+        Returns:
+          Content of the plist embedded in the provisioning profile as a dictionary.
+        """
         with tempfile.NamedTemporaryFile() as temp:
             subprocess.check_call(
                 ["security", "cms", "-D", "-i", profile_path, "-o", temp.name]
@@ -580,16 +579,16 @@ def _MergePlist(self, merged_plist, plist):
     def _LoadPlistMaybeBinary(self, plist_path):
         """Loads into a memory a plist possibly encoded in binary format.
 
-    This is a wrapper around plistlib.readPlist that tries to convert the
-    plist to the XML format if it can't be parsed (assuming that it is in
-    the binary format).
+        This is a wrapper around plistlib.readPlist that tries to convert the
+        plist to the XML format if it can't be parsed (assuming that it is in
+        the binary format).
 
-    Args:
-      plist_path: string, path to a plist file, in XML or binary format
+        Args:
+          plist_path: string, path to a plist file, in XML or binary format
 
-    Returns:
-      Content of the plist as a dictionary.
-    """
+        Returns:
+          Content of the plist as a dictionary.
+        """
         try:
             # First, try to read the file using plistlib that only supports XML,
             # and if an exception is raised, convert a temporary copy to XML and
@@ -605,13 +604,13 @@ def _LoadPlistMaybeBinary(self, plist_path):
     def _GetSubstitutions(self, bundle_identifier, app_identifier_prefix):
         """Constructs a dictionary of variable substitutions for Entitlements.plist.
 
-    Args:
-      bundle_identifier: string, value of CFBundleIdentifier from Info.plist
-      app_identifier_prefix: string, value for AppIdentifierPrefix
+        Args:
+          bundle_identifier: string, value of CFBundleIdentifier from Info.plist
+          app_identifier_prefix: string, value for AppIdentifierPrefix
 
-    Returns:
-      Dictionary of substitutions to apply when generating Entitlements.plist.
-    """
+        Returns:
+          Dictionary of substitutions to apply when generating Entitlements.plist.
+        """
         return {
             "CFBundleIdentifier": bundle_identifier,
             "AppIdentifierPrefix": app_identifier_prefix,
@@ -620,9 +619,9 @@ def _GetSubstitutions(self, bundle_identifier, app_identifier_prefix):
     def _GetCFBundleIdentifier(self):
         """Extracts CFBundleIdentifier value from Info.plist in the bundle.
 
-    Returns:
-      Value of CFBundleIdentifier in the Info.plist located in the bundle.
-    """
+        Returns:
+          Value of CFBundleIdentifier in the Info.plist located in the bundle.
+        """
         info_plist_path = os.path.join(
             os.environ["TARGET_BUILD_DIR"], os.environ["INFOPLIST_PATH"]
         )
@@ -632,19 +631,19 @@ def _GetCFBundleIdentifier(self):
     def _InstallEntitlements(self, entitlements, substitutions, overrides):
         """Generates and install the ${BundleName}.xcent entitlements file.
 
-    Expands variables "$(variable)" pattern in the source entitlements file,
-    add extra entitlements defined in the .mobileprovision file and the copy
-    the generated plist to "${BundlePath}.xcent".
+        Expands variables "$(variable)" pattern in the source entitlements file,
+        add extra entitlements defined in the .mobileprovision file and the copy
+        the generated plist to "${BundlePath}.xcent".
 
-    Args:
-      entitlements: string, optional, path to the Entitlements.plist template
-        to use, defaults to "${SDKROOT}/Entitlements.plist"
-      substitutions: dictionary, variable substitutions
-      overrides: dictionary, values to add to the entitlements
+        Args:
+          entitlements: string, optional, path to the Entitlements.plist template
+            to use, defaults to "${SDKROOT}/Entitlements.plist"
+          substitutions: dictionary, variable substitutions
+          overrides: dictionary, values to add to the entitlements
 
-    Returns:
-      Path to the generated entitlements file.
-    """
+        Returns:
+          Path to the generated entitlements file.
+        """
         source_path = entitlements
         target_path = os.path.join(
             os.environ["BUILT_PRODUCTS_DIR"], os.environ["PRODUCT_NAME"] + ".xcent"
@@ -664,15 +663,15 @@ def _InstallEntitlements(self, entitlements, substitutions, overrides):
     def _ExpandVariables(self, data, substitutions):
         """Expands variables "$(variable)" in data.
 
-    Args:
-      data: object, can be either string, list or dictionary
-      substitutions: dictionary, variable substitutions to perform
+        Args:
+          data: object, can be either string, list or dictionary
+          substitutions: dictionary, variable substitutions to perform
 
-    Returns:
-      Copy of data where each references to "$(variable)" has been replaced
-      by the corresponding value found in substitutions, or left intact if
-      the key was not found.
-    """
+        Returns:
+          Copy of data where each references to "$(variable)" has been replaced
+          by the corresponding value found in substitutions, or left intact if
+          the key was not found.
+        """
         if isinstance(data, str):
             for key, value in substitutions.items():
                 data = data.replace("$(%s)" % key, value)
@@ -691,15 +690,15 @@ def NextGreaterPowerOf2(x):
 def WriteHmap(output_name, filelist):
     """Generates a header map based on |filelist|.
 
-  Per Mark Mentovai:
-    A header map is structured essentially as a hash table, keyed by names used
-    in #includes, and providing pathnames to the actual files.
+    Per Mark Mentovai:
+      A header map is structured essentially as a hash table, keyed by names used
+      in #includes, and providing pathnames to the actual files.
 
-  The implementation below and the comment above comes from inspecting:
-    http://www.opensource.apple.com/source/distcc/distcc-2503/distcc_dist/include_server/headermap.py?txt
-  while also looking at the implementation in clang in:
-    https://llvm.org/svn/llvm-project/cfe/trunk/lib/Lex/HeaderMap.cpp
-  """
+    The implementation below and the comment above comes from inspecting:
+      http://www.opensource.apple.com/source/distcc/distcc-2503/distcc_dist/include_server/headermap.py?txt
+    while also looking at the implementation in clang in:
+      https://llvm.org/svn/llvm-project/cfe/trunk/lib/Lex/HeaderMap.cpp
+    """
     magic = 1751998832
     version = 1
     _reserved = 0
diff --git a/gyp/pylib/gyp/msvs_emulation.py b/gyp/pylib/gyp/msvs_emulation.py
index f8b3b87d94..7c461a8fdf 100644
--- a/gyp/pylib/gyp/msvs_emulation.py
+++ b/gyp/pylib/gyp/msvs_emulation.py
@@ -74,8 +74,7 @@ def EncodeRspFileList(args, quote_cmd):
         program = call + " " + os.path.normpath(program)
     else:
         program = os.path.normpath(args[0])
-    return (program + " "
-            + " ".join(QuoteForRspFile(arg, quote_cmd) for arg in args[1:]))
+    return program + " " + " ".join(QuoteForRspFile(arg, quote_cmd) for arg in args[1:])
 
 
 def _GenericRetrieve(root, default, path):
@@ -934,14 +933,17 @@ def GetRuleShellFlags(self, rule):
         includes whether it should run under cygwin (msvs_cygwin_shell), and
         whether the commands should be quoted (msvs_quote_cmd)."""
         # If the variable is unset, or set to 1 we use cygwin
-        cygwin = int(rule.get("msvs_cygwin_shell",
-                              self.spec.get("msvs_cygwin_shell", 1))) != 0
+        cygwin = (
+            int(rule.get("msvs_cygwin_shell", self.spec.get("msvs_cygwin_shell", 1)))
+            != 0
+        )
         # Default to quoting. There's only a few special instances where the
         # target command uses non-standard command line parsing and handle quotes
         # and quote escaping differently.
         quote_cmd = int(rule.get("msvs_quote_cmd", 1))
-        assert quote_cmd != 0 or cygwin != 1, \
-               "msvs_quote_cmd=0 only applicable for msvs_cygwin_shell=0"
+        assert quote_cmd != 0 or cygwin != 1, (
+            "msvs_quote_cmd=0 only applicable for msvs_cygwin_shell=0"
+        )
         return MsvsSettings.RuleShellFlags(cygwin, quote_cmd)
 
     def _HasExplicitRuleForExtension(self, spec, extension):
@@ -1129,8 +1131,7 @@ def _ExtractImportantEnvironment(output_of_set):
     for required in ("SYSTEMROOT", "TEMP", "TMP"):
         if required not in env:
             raise Exception(
-                'Environment variable "%s" '
-                "required to be set to valid path" % required
+                'Environment variable "%s" required to be set to valid path' % required
             )
     return env
 
diff --git a/gyp/pylib/gyp/simple_copy.py b/gyp/pylib/gyp/simple_copy.py
index 729cec0636..8b026642fc 100644
--- a/gyp/pylib/gyp/simple_copy.py
+++ b/gyp/pylib/gyp/simple_copy.py
@@ -17,8 +17,8 @@ class Error(Exception):
 
 def deepcopy(x):
     """Deep copy operation on gyp objects such as strings, ints, dicts
-  and lists. More than twice as fast as copy.deepcopy but much less
-  generic."""
+    and lists. More than twice as fast as copy.deepcopy but much less
+    generic."""
 
     try:
         return _deepcopy_dispatch[type(x)](x)
diff --git a/gyp/pylib/gyp/win_tool.py b/gyp/pylib/gyp/win_tool.py
index 3004f533ca..43665577bd 100755
--- a/gyp/pylib/gyp/win_tool.py
+++ b/gyp/pylib/gyp/win_tool.py
@@ -9,7 +9,6 @@
 These functions are executed via gyp-win-tool when using the ninja generator.
 """
 
-
 import os
 import re
 import shutil
@@ -33,11 +32,11 @@ def main(args):
 
 class WinTool:
     """This class performs all the Windows tooling steps. The methods can either
-  be executed directly, or dispatched from an argument list."""
+    be executed directly, or dispatched from an argument list."""
 
     def _UseSeparateMspdbsrv(self, env, args):
         """Allows to use a unique instance of mspdbsrv.exe per linker instead of a
-    shared one."""
+        shared one."""
         if len(args) < 1:
             raise Exception("Not enough arguments")
 
@@ -114,9 +113,9 @@ def _on_error(fn, path, excinfo):
 
     def ExecLinkWrapper(self, arch, use_separate_mspdbsrv, *args):
         """Filter diagnostic output from link that looks like:
-    '   Creating library ui.dll.lib and object ui.dll.exp'
-    This happens when there are exports from the dll or exe.
-    """
+        '   Creating library ui.dll.lib and object ui.dll.exp'
+        This happens when there are exports from the dll or exe.
+        """
         env = self._GetEnv(arch)
         if use_separate_mspdbsrv == "True":
             self._UseSeparateMspdbsrv(env, args)
@@ -158,10 +157,10 @@ def ExecLinkWithManifests(
         mt,
         rc,
         intermediate_manifest,
-        *manifests
+        *manifests,
     ):
         """A wrapper for handling creating a manifest resource and then executing
-    a link command."""
+        a link command."""
         # The 'normal' way to do manifests is to have link generate a manifest
         # based on gathering dependencies from the object files, then merge that
         # manifest with other manifests supplied as sources, convert the merged
@@ -245,8 +244,8 @@ def dump(filename):
 
     def ExecManifestWrapper(self, arch, *args):
         """Run manifest tool with environment set. Strip out undesirable warning
-    (some XML blocks are recognized by the OS loader, but not the manifest
-    tool)."""
+        (some XML blocks are recognized by the OS loader, but not the manifest
+        tool)."""
         env = self._GetEnv(arch)
         popen = subprocess.Popen(
             args, shell=True, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
@@ -259,8 +258,8 @@ def ExecManifestWrapper(self, arch, *args):
 
     def ExecManifestToRc(self, arch, *args):
         """Creates a resource file pointing a SxS assembly manifest.
-    |args| is tuple containing path to resource file, path to manifest file
-    and resource name which can be "1" (for executables) or "2" (for DLLs)."""
+        |args| is tuple containing path to resource file, path to manifest file
+        and resource name which can be "1" (for executables) or "2" (for DLLs)."""
         manifest_path, resource_path, resource_name = args
         with open(resource_path, "w") as output:
             output.write(
@@ -270,8 +269,8 @@ def ExecManifestToRc(self, arch, *args):
 
     def ExecMidlWrapper(self, arch, outdir, tlb, h, dlldata, iid, proxy, idl, *flags):
         """Filter noisy filenames output from MIDL compile step that isn't
-    quietable via command line flags.
-    """
+        quietable via command line flags.
+        """
         args = (
             ["midl", "/nologo"]
             + list(flags)
@@ -327,7 +326,7 @@ def ExecAsmWrapper(self, arch, *args):
 
     def ExecRcWrapper(self, arch, *args):
         """Filter logo banner from invocations of rc.exe. Older versions of RC
-    don't support the /nologo flag."""
+        don't support the /nologo flag."""
         env = self._GetEnv(arch)
         popen = subprocess.Popen(
             args, shell=True, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
@@ -344,7 +343,7 @@ def ExecRcWrapper(self, arch, *args):
 
     def ExecActionWrapper(self, arch, rspfile, *dir):
         """Runs an action command line from a response file using the environment
-    for |arch|. If |dir| is supplied, use that as the working directory."""
+        for |arch|. If |dir| is supplied, use that as the working directory."""
         env = self._GetEnv(arch)
         # TODO(scottmg): This is a temporary hack to get some specific variables
         # through to actions that are set after gyp-time. http://crbug.com/333738.
@@ -357,7 +356,7 @@ def ExecActionWrapper(self, arch, rspfile, *dir):
 
     def ExecClCompile(self, project_dir, selected_files):
         """Executed by msvs-ninja projects when the 'ClCompile' target is used to
-    build selected C/C++ files."""
+        build selected C/C++ files."""
         project_dir = os.path.relpath(project_dir, BASE_DIR)
         selected_files = selected_files.split(";")
         ninja_targets = [
diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py
index 08e645c57d..192a523529 100644
--- a/gyp/pylib/gyp/xcode_emulation.py
+++ b/gyp/pylib/gyp/xcode_emulation.py
@@ -7,7 +7,6 @@
 other build systems, such as make and ninja.
 """
 
-
 import copy
 import os
 import os.path
@@ -31,7 +30,7 @@
 
 def XcodeArchsVariableMapping(archs, archs_including_64_bit=None):
     """Constructs a dictionary with expansion for $(ARCHS_STANDARD) variable,
-  and optionally for $(ARCHS_STANDARD_INCLUDING_64_BIT)."""
+    and optionally for $(ARCHS_STANDARD_INCLUDING_64_BIT)."""
     mapping = {"$(ARCHS_STANDARD)": archs}
     if archs_including_64_bit:
         mapping["$(ARCHS_STANDARD_INCLUDING_64_BIT)"] = archs_including_64_bit
@@ -40,10 +39,10 @@ def XcodeArchsVariableMapping(archs, archs_including_64_bit=None):
 
 class XcodeArchsDefault:
     """A class to resolve ARCHS variable from xcode_settings, resolving Xcode
-  macros and implementing filtering by VALID_ARCHS. The expansion of macros
-  depends on the SDKROOT used ("macosx", "iphoneos", "iphonesimulator") and
-  on the version of Xcode.
-  """
+    macros and implementing filtering by VALID_ARCHS. The expansion of macros
+    depends on the SDKROOT used ("macosx", "iphoneos", "iphonesimulator") and
+    on the version of Xcode.
+    """
 
     # Match variable like $(ARCHS_STANDARD).
     variable_pattern = re.compile(r"\$\([a-zA-Z_][a-zA-Z0-9_]*\)$")
@@ -82,8 +81,8 @@ def _ExpandArchs(self, archs, sdkroot):
 
     def ActiveArchs(self, archs, valid_archs, sdkroot):
         """Expands variables references in ARCHS, and filter by VALID_ARCHS if it
-    is defined (if not set, Xcode accept any value in ARCHS, otherwise, only
-    values present in VALID_ARCHS are kept)."""
+        is defined (if not set, Xcode accept any value in ARCHS, otherwise, only
+        values present in VALID_ARCHS are kept)."""
         expanded_archs = self._ExpandArchs(archs or self._default, sdkroot or "")
         if valid_archs:
             filtered_archs = []
@@ -96,24 +95,24 @@ def ActiveArchs(self, archs, valid_archs, sdkroot):
 
 def GetXcodeArchsDefault():
     """Returns the |XcodeArchsDefault| object to use to expand ARCHS for the
-  installed version of Xcode. The default values used by Xcode for ARCHS
-  and the expansion of the variables depends on the version of Xcode used.
+    installed version of Xcode. The default values used by Xcode for ARCHS
+    and the expansion of the variables depends on the version of Xcode used.
 
-  For all version anterior to Xcode 5.0 or posterior to Xcode 5.1 included
-  uses $(ARCHS_STANDARD) if ARCHS is unset, while Xcode 5.0 to 5.0.2 uses
-  $(ARCHS_STANDARD_INCLUDING_64_BIT). This variable was added to Xcode 5.0
-  and deprecated with Xcode 5.1.
+    For all version anterior to Xcode 5.0 or posterior to Xcode 5.1 included
+    uses $(ARCHS_STANDARD) if ARCHS is unset, while Xcode 5.0 to 5.0.2 uses
+    $(ARCHS_STANDARD_INCLUDING_64_BIT). This variable was added to Xcode 5.0
+    and deprecated with Xcode 5.1.
 
-  For "macosx" SDKROOT, all version starting with Xcode 5.0 includes 64-bit
-  architecture as part of $(ARCHS_STANDARD) and default to only building it.
+    For "macosx" SDKROOT, all version starting with Xcode 5.0 includes 64-bit
+    architecture as part of $(ARCHS_STANDARD) and default to only building it.
 
-  For "iphoneos" and "iphonesimulator" SDKROOT, 64-bit architectures are part
-  of $(ARCHS_STANDARD_INCLUDING_64_BIT) from Xcode 5.0. From Xcode 5.1, they
-  are also part of $(ARCHS_STANDARD).
+    For "iphoneos" and "iphonesimulator" SDKROOT, 64-bit architectures are part
+    of $(ARCHS_STANDARD_INCLUDING_64_BIT) from Xcode 5.0. From Xcode 5.1, they
+    are also part of $(ARCHS_STANDARD).
 
-  All these rules are coded in the construction of the |XcodeArchsDefault|
-  object to use depending on the version of Xcode detected. The object is
-  for performance reason."""
+    All these rules are coded in the construction of the |XcodeArchsDefault|
+    object to use depending on the version of Xcode detected. The object is
+    for performance reason."""
     global XCODE_ARCHS_DEFAULT_CACHE
     if XCODE_ARCHS_DEFAULT_CACHE:
         return XCODE_ARCHS_DEFAULT_CACHE
@@ -190,8 +189,8 @@ def __init__(self, spec):
 
     def _ConvertConditionalKeys(self, configname):
         """Converts or warns on conditional keys.  Xcode supports conditional keys,
-    such as CODE_SIGN_IDENTITY[sdk=iphoneos*].  This is a partial implementation
-    with some keys converted while the rest force a warning."""
+        such as CODE_SIGN_IDENTITY[sdk=iphoneos*].  This is a partial implementation
+        with some keys converted while the rest force a warning."""
         settings = self.xcode_settings[configname]
         conditional_keys = [key for key in settings if key.endswith("]")]
         for key in conditional_keys:
@@ -256,13 +255,13 @@ def _IsIosWatchApp(self):
 
     def GetFrameworkVersion(self):
         """Returns the framework version of the current target. Only valid for
-    bundles."""
+        bundles."""
         assert self._IsBundle()
         return self.GetPerTargetSetting("FRAMEWORK_VERSION", default="A")
 
     def GetWrapperExtension(self):
         """Returns the bundle extension (.app, .framework, .plugin, etc).  Only
-    valid for bundles."""
+        valid for bundles."""
         assert self._IsBundle()
         if self.spec["type"] in ("loadable_module", "shared_library"):
             default_wrapper_extension = {
@@ -297,13 +296,13 @@ def GetFullProductName(self):
 
     def GetWrapperName(self):
         """Returns the directory name of the bundle represented by this target.
-    Only valid for bundles."""
+        Only valid for bundles."""
         assert self._IsBundle()
         return self.GetProductName() + self.GetWrapperExtension()
 
     def GetBundleContentsFolderPath(self):
         """Returns the qualified path to the bundle's contents folder. E.g.
-    Chromium.app/Contents or Foo.bundle/Versions/A. Only valid for bundles."""
+        Chromium.app/Contents or Foo.bundle/Versions/A. Only valid for bundles."""
         if self.isIOS:
             return self.GetWrapperName()
         assert self._IsBundle()
@@ -317,7 +316,7 @@ def GetBundleContentsFolderPath(self):
 
     def GetBundleResourceFolder(self):
         """Returns the qualified path to the bundle's resource folder. E.g.
-    Chromium.app/Contents/Resources. Only valid for bundles."""
+        Chromium.app/Contents/Resources. Only valid for bundles."""
         assert self._IsBundle()
         if self.isIOS:
             return self.GetBundleContentsFolderPath()
@@ -325,7 +324,7 @@ def GetBundleResourceFolder(self):
 
     def GetBundleExecutableFolderPath(self):
         """Returns the qualified path to the bundle's executables folder. E.g.
-    Chromium.app/Contents/MacOS. Only valid for bundles."""
+        Chromium.app/Contents/MacOS. Only valid for bundles."""
         assert self._IsBundle()
         if self.spec["type"] in ("shared_library") or self.isIOS:
             return self.GetBundleContentsFolderPath()
@@ -334,25 +333,25 @@ def GetBundleExecutableFolderPath(self):
 
     def GetBundleJavaFolderPath(self):
         """Returns the qualified path to the bundle's Java resource folder.
-    E.g. Chromium.app/Contents/Resources/Java. Only valid for bundles."""
+        E.g. Chromium.app/Contents/Resources/Java. Only valid for bundles."""
         assert self._IsBundle()
         return os.path.join(self.GetBundleResourceFolder(), "Java")
 
     def GetBundleFrameworksFolderPath(self):
         """Returns the qualified path to the bundle's frameworks folder. E.g,
-    Chromium.app/Contents/Frameworks. Only valid for bundles."""
+        Chromium.app/Contents/Frameworks. Only valid for bundles."""
         assert self._IsBundle()
         return os.path.join(self.GetBundleContentsFolderPath(), "Frameworks")
 
     def GetBundleSharedFrameworksFolderPath(self):
         """Returns the qualified path to the bundle's frameworks folder. E.g,
-    Chromium.app/Contents/SharedFrameworks. Only valid for bundles."""
+        Chromium.app/Contents/SharedFrameworks. Only valid for bundles."""
         assert self._IsBundle()
         return os.path.join(self.GetBundleContentsFolderPath(), "SharedFrameworks")
 
     def GetBundleSharedSupportFolderPath(self):
         """Returns the qualified path to the bundle's shared support folder. E.g,
-    Chromium.app/Contents/SharedSupport. Only valid for bundles."""
+        Chromium.app/Contents/SharedSupport. Only valid for bundles."""
         assert self._IsBundle()
         if self.spec["type"] == "shared_library":
             return self.GetBundleResourceFolder()
@@ -361,19 +360,19 @@ def GetBundleSharedSupportFolderPath(self):
 
     def GetBundlePlugInsFolderPath(self):
         """Returns the qualified path to the bundle's plugins folder. E.g,
-    Chromium.app/Contents/PlugIns. Only valid for bundles."""
+        Chromium.app/Contents/PlugIns. Only valid for bundles."""
         assert self._IsBundle()
         return os.path.join(self.GetBundleContentsFolderPath(), "PlugIns")
 
     def GetBundleXPCServicesFolderPath(self):
         """Returns the qualified path to the bundle's XPC services folder. E.g,
-    Chromium.app/Contents/XPCServices. Only valid for bundles."""
+        Chromium.app/Contents/XPCServices. Only valid for bundles."""
         assert self._IsBundle()
         return os.path.join(self.GetBundleContentsFolderPath(), "XPCServices")
 
     def GetBundlePlistPath(self):
         """Returns the qualified path to the bundle's plist file. E.g.
-    Chromium.app/Contents/Info.plist. Only valid for bundles."""
+        Chromium.app/Contents/Info.plist. Only valid for bundles."""
         assert self._IsBundle()
         if (
             self.spec["type"] in ("executable", "loadable_module")
@@ -439,7 +438,7 @@ def GetMachOType(self):
 
     def _GetBundleBinaryPath(self):
         """Returns the name of the bundle binary of by this target.
-    E.g. Chromium.app/Contents/MacOS/Chromium. Only valid for bundles."""
+        E.g. Chromium.app/Contents/MacOS/Chromium. Only valid for bundles."""
         assert self._IsBundle()
         return os.path.join(
             self.GetBundleExecutableFolderPath(), self.GetExecutableName()
@@ -470,14 +469,14 @@ def _GetStandaloneExecutablePrefix(self):
 
     def _GetStandaloneBinaryPath(self):
         """Returns the name of the non-bundle binary represented by this target.
-    E.g. hello_world. Only valid for non-bundles."""
+        E.g. hello_world. Only valid for non-bundles."""
         assert not self._IsBundle()
         assert self.spec["type"] in {
             "executable",
             "shared_library",
             "static_library",
             "loadable_module",
-        }, ("Unexpected type %s" % self.spec["type"])
+        }, "Unexpected type %s" % self.spec["type"]
         target = self.spec["target_name"]
         if self.spec["type"] in {"loadable_module", "shared_library", "static_library"}:
             if target[:3] == "lib":
@@ -490,7 +489,7 @@ def _GetStandaloneBinaryPath(self):
 
     def GetExecutableName(self):
         """Returns the executable name of the bundle represented by this target.
-    E.g. Chromium."""
+        E.g. Chromium."""
         if self._IsBundle():
             return self.spec.get("product_name", self.spec["target_name"])
         else:
@@ -498,7 +497,7 @@ def GetExecutableName(self):
 
     def GetExecutablePath(self):
         """Returns the qualified path to the primary executable of the bundle
-    represented by this target. E.g. Chromium.app/Contents/MacOS/Chromium."""
+        represented by this target. E.g. Chromium.app/Contents/MacOS/Chromium."""
         if self._IsBundle():
             return self._GetBundleBinaryPath()
         else:
@@ -568,7 +567,7 @@ def _AppendPlatformVersionMinFlags(self, lst):
 
     def GetCflags(self, configname, arch=None):
         """Returns flags that need to be added to .c, .cc, .m, and .mm
-    compilations."""
+        compilations."""
         # This functions (and the similar ones below) do not offer complete
         # emulation of all xcode_settings keys. They're implemented on demand.
 
@@ -863,7 +862,7 @@ def GetInstallName(self):
 
     def _MapLinkerFlagFilename(self, ldflag, gyp_to_build_path):
         """Checks if ldflag contains a filename and if so remaps it from
-    gyp-directory-relative to build-directory-relative."""
+        gyp-directory-relative to build-directory-relative."""
         # This list is expanded on demand.
         # They get matched as:
         #   -exported_symbols_list file
@@ -895,13 +894,13 @@ def _MapLinkerFlagFilename(self, ldflag, gyp_to_build_path):
     def GetLdflags(self, configname, product_dir, gyp_to_build_path, arch=None):
         """Returns flags that need to be passed to the linker.
 
-    Args:
-        configname: The name of the configuration to get ld flags for.
-        product_dir: The directory where products such static and dynamic
-            libraries are placed. This is added to the library search path.
-        gyp_to_build_path: A function that converts paths relative to the
-            current gyp file to paths relative to the build directory.
-    """
+        Args:
+            configname: The name of the configuration to get ld flags for.
+            product_dir: The directory where products such static and dynamic
+                libraries are placed. This is added to the library search path.
+            gyp_to_build_path: A function that converts paths relative to the
+                current gyp file to paths relative to the build directory.
+        """
         self.configname = configname
         ldflags = []
 
@@ -1001,9 +1000,9 @@ def GetLdflags(self, configname, product_dir, gyp_to_build_path, arch=None):
     def GetLibtoolflags(self, configname):
         """Returns flags that need to be passed to the static linker.
 
-    Args:
-        configname: The name of the configuration to get ld flags for.
-    """
+        Args:
+            configname: The name of the configuration to get ld flags for.
+        """
         self.configname = configname
         libtoolflags = []
 
@@ -1016,7 +1015,7 @@ def GetLibtoolflags(self, configname):
 
     def GetPerTargetSettings(self):
         """Gets a list of all the per-target settings. This will only fetch keys
-    whose values are the same across all configurations."""
+        whose values are the same across all configurations."""
         first_pass = True
         result = {}
         for configname in sorted(self.xcode_settings.keys()):
@@ -1039,7 +1038,7 @@ def GetPerConfigSetting(self, setting, configname, default=None):
 
     def GetPerTargetSetting(self, setting, default=None):
         """Tries to get xcode_settings.setting from spec. Assumes that the setting
-       has the same value in all configurations and throws otherwise."""
+        has the same value in all configurations and throws otherwise."""
         is_first_pass = True
         result = None
         for configname in sorted(self.xcode_settings.keys()):
@@ -1057,15 +1056,14 @@ def GetPerTargetSetting(self, setting, default=None):
 
     def _GetStripPostbuilds(self, configname, output_binary, quiet):
         """Returns a list of shell commands that contain the shell commands
-    necessary to strip this target's binary. These should be run as postbuilds
-    before the actual postbuilds run."""
+        necessary to strip this target's binary. These should be run as postbuilds
+        before the actual postbuilds run."""
         self.configname = configname
 
         result = []
         if self._Test("DEPLOYMENT_POSTPROCESSING", "YES", default="NO") and self._Test(
             "STRIP_INSTALLED_PRODUCT", "YES", default="NO"
         ):
-
             default_strip_style = "debugging"
             if (
                 self.spec["type"] == "loadable_module" or self._IsIosAppExtension()
@@ -1092,8 +1090,8 @@ def _GetStripPostbuilds(self, configname, output_binary, quiet):
 
     def _GetDebugInfoPostbuilds(self, configname, output, output_binary, quiet):
         """Returns a list of shell commands that contain the shell commands
-    necessary to massage this target's debug information. These should be run
-    as postbuilds before the actual postbuilds run."""
+        necessary to massage this target's debug information. These should be run
+        as postbuilds before the actual postbuilds run."""
         self.configname = configname
 
         # For static libraries, no dSYMs are created.
@@ -1114,7 +1112,7 @@ def _GetDebugInfoPostbuilds(self, configname, output, output_binary, quiet):
 
     def _GetTargetPostbuilds(self, configname, output, output_binary, quiet=False):
         """Returns a list of shell commands that contain the shell commands
-    to run as postbuilds for this target, before the actual postbuilds."""
+        to run as postbuilds for this target, before the actual postbuilds."""
         # dSYMs need to build before stripping happens.
         return self._GetDebugInfoPostbuilds(
             configname, output, output_binary, quiet
@@ -1122,11 +1120,10 @@ def _GetTargetPostbuilds(self, configname, output, output_binary, quiet=False):
 
     def _GetIOSPostbuilds(self, configname, output_binary):
         """Return a shell command to codesign the iOS output binary so it can
-    be deployed to a device.  This should be run as the very last step of the
-    build."""
+        be deployed to a device.  This should be run as the very last step of the
+        build."""
         if not (
-            (self.isIOS
-            and (self.spec["type"] == "executable" or self._IsXCTest()))
+            (self.isIOS and (self.spec["type"] == "executable" or self._IsXCTest()))
             or self.IsIosFramework()
         ):
             return []
@@ -1240,7 +1237,7 @@ def AddImplicitPostbuilds(
         self, configname, output, output_binary, postbuilds=[], quiet=False
     ):
         """Returns a list of shell commands that should run before and after
-    |postbuilds|."""
+        |postbuilds|."""
         assert output_binary is not None
         pre = self._GetTargetPostbuilds(configname, output, output_binary, quiet)
         post = self._GetIOSPostbuilds(configname, output_binary)
@@ -1276,8 +1273,8 @@ def _AdjustLibrary(self, library, config_name=None):
 
     def AdjustLibraries(self, libraries, config_name=None):
         """Transforms entries like 'Cocoa.framework' in libraries into entries like
-    '-framework Cocoa', 'libcrypto.dylib' into '-lcrypto', etc.
-    """
+        '-framework Cocoa', 'libcrypto.dylib' into '-lcrypto', etc.
+        """
         libraries = [self._AdjustLibrary(library, config_name) for library in libraries]
         return libraries
 
@@ -1342,10 +1339,10 @@ def GetExtraPlistItems(self, configname=None):
     def _DefaultSdkRoot(self):
         """Returns the default SDKROOT to use.
 
-    Prior to version 5.0.0, if SDKROOT was not explicitly set in the Xcode
-    project, then the environment variable was empty. Starting with this
-    version, Xcode uses the name of the newest SDK installed.
-    """
+        Prior to version 5.0.0, if SDKROOT was not explicitly set in the Xcode
+        project, then the environment variable was empty. Starting with this
+        version, Xcode uses the name of the newest SDK installed.
+        """
         xcode_version, _ = XcodeVersion()
         if xcode_version < "0500":
             return ""
@@ -1370,39 +1367,39 @@ def _DefaultSdkRoot(self):
 class MacPrefixHeader:
     """A class that helps with emulating Xcode's GCC_PREFIX_HEADER feature.
 
-  This feature consists of several pieces:
-  * If GCC_PREFIX_HEADER is present, all compilations in that project get an
-    additional |-include path_to_prefix_header| cflag.
-  * If GCC_PRECOMPILE_PREFIX_HEADER is present too, then the prefix header is
-    instead compiled, and all other compilations in the project get an
-    additional |-include path_to_compiled_header| instead.
-    + Compiled prefix headers have the extension gch. There is one gch file for
-      every language used in the project (c, cc, m, mm), since gch files for
-      different languages aren't compatible.
-    + gch files themselves are built with the target's normal cflags, but they
-      obviously don't get the |-include| flag. Instead, they need a -x flag that
-      describes their language.
-    + All o files in the target need to depend on the gch file, to make sure
-      it's built before any o file is built.
-
-  This class helps with some of these tasks, but it needs help from the build
-  system for writing dependencies to the gch files, for writing build commands
-  for the gch files, and for figuring out the location of the gch files.
-  """
+    This feature consists of several pieces:
+    * If GCC_PREFIX_HEADER is present, all compilations in that project get an
+      additional |-include path_to_prefix_header| cflag.
+    * If GCC_PRECOMPILE_PREFIX_HEADER is present too, then the prefix header is
+      instead compiled, and all other compilations in the project get an
+      additional |-include path_to_compiled_header| instead.
+      + Compiled prefix headers have the extension gch. There is one gch file for
+        every language used in the project (c, cc, m, mm), since gch files for
+        different languages aren't compatible.
+      + gch files themselves are built with the target's normal cflags, but they
+        obviously don't get the |-include| flag. Instead, they need a -x flag that
+        describes their language.
+      + All o files in the target need to depend on the gch file, to make sure
+        it's built before any o file is built.
+
+    This class helps with some of these tasks, but it needs help from the build
+    system for writing dependencies to the gch files, for writing build commands
+    for the gch files, and for figuring out the location of the gch files.
+    """
 
     def __init__(
         self, xcode_settings, gyp_path_to_build_path, gyp_path_to_build_output
     ):
         """If xcode_settings is None, all methods on this class are no-ops.
 
-    Args:
-        gyp_path_to_build_path: A function that takes a gyp-relative path,
-            and returns a path relative to the build directory.
-        gyp_path_to_build_output: A function that takes a gyp-relative path and
-            a language code ('c', 'cc', 'm', or 'mm'), and that returns a path
-            to where the output of precompiling that path for that language
-            should be placed (without the trailing '.gch').
-    """
+        Args:
+            gyp_path_to_build_path: A function that takes a gyp-relative path,
+                and returns a path relative to the build directory.
+            gyp_path_to_build_output: A function that takes a gyp-relative path and
+                a language code ('c', 'cc', 'm', or 'mm'), and that returns a path
+                to where the output of precompiling that path for that language
+                should be placed (without the trailing '.gch').
+        """
         # This doesn't support per-configuration prefix headers. Good enough
         # for now.
         self.header = None
@@ -1447,9 +1444,9 @@ def _Gch(self, lang, arch):
 
     def GetObjDependencies(self, sources, objs, arch=None):
         """Given a list of source files and the corresponding object files, returns
-    a list of (source, object, gch) tuples, where |gch| is the build-directory
-    relative path to the gch file each object file depends on.  |compilable[i]|
-    has to be the source file belonging to |objs[i]|."""
+        a list of (source, object, gch) tuples, where |gch| is the build-directory
+        relative path to the gch file each object file depends on.  |compilable[i]|
+        has to be the source file belonging to |objs[i]|."""
         if not self.header or not self.compile_headers:
             return []
 
@@ -1470,8 +1467,8 @@ def GetObjDependencies(self, sources, objs, arch=None):
 
     def GetPchBuildCommands(self, arch=None):
         """Returns [(path_to_gch, language_flag, language, header)].
-    |path_to_gch| and |header| are relative to the build directory.
-    """
+        |path_to_gch| and |header| are relative to the build directory.
+        """
         if not self.header or not self.compile_headers:
             return []
         return [
@@ -1555,8 +1552,8 @@ def CLTVersion():
 
 def GetStdoutQuiet(cmdlist):
     """Returns the content of standard output returned by invoking |cmdlist|.
-  Ignores the stderr.
-  Raises |GypError| if the command return with a non-zero return code."""
+    Ignores the stderr.
+    Raises |GypError| if the command return with a non-zero return code."""
     job = subprocess.Popen(cmdlist, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
     out = job.communicate()[0].decode("utf-8")
     if job.returncode != 0:
@@ -1566,7 +1563,7 @@ def GetStdoutQuiet(cmdlist):
 
 def GetStdout(cmdlist):
     """Returns the content of standard output returned by invoking |cmdlist|.
-  Raises |GypError| if the command return with a non-zero return code."""
+    Raises |GypError| if the command return with a non-zero return code."""
     job = subprocess.Popen(cmdlist, stdout=subprocess.PIPE)
     out = job.communicate()[0].decode("utf-8")
     if job.returncode != 0:
@@ -1577,9 +1574,9 @@ def GetStdout(cmdlist):
 
 def MergeGlobalXcodeSettingsToSpec(global_dict, spec):
     """Merges the global xcode_settings dictionary into each configuration of the
-  target represented by spec. For keys that are both in the global and the local
-  xcode_settings dict, the local key gets precedence.
-  """
+    target represented by spec. For keys that are both in the global and the local
+    xcode_settings dict, the local key gets precedence.
+    """
     # The xcode generator special-cases global xcode_settings and does something
     # that amounts to merging in the global xcode_settings into each local
     # xcode_settings dict.
@@ -1594,9 +1591,9 @@ def MergeGlobalXcodeSettingsToSpec(global_dict, spec):
 def IsMacBundle(flavor, spec):
     """Returns if |spec| should be treated as a bundle.
 
-  Bundles are directories with a certain subdirectory structure, instead of
-  just a single file. Bundle rules do not produce a binary but also package
-  resources into that directory."""
+    Bundles are directories with a certain subdirectory structure, instead of
+    just a single file. Bundle rules do not produce a binary but also package
+    resources into that directory."""
     is_mac_bundle = (
         int(spec.get("mac_xctest_bundle", 0)) != 0
         or int(spec.get("mac_xcuitest_bundle", 0)) != 0
@@ -1613,14 +1610,14 @@ def IsMacBundle(flavor, spec):
 
 def GetMacBundleResources(product_dir, xcode_settings, resources):
     """Yields (output, resource) pairs for every resource in |resources|.
-  Only call this for mac bundle targets.
-
-  Args:
-      product_dir: Path to the directory containing the output bundle,
-          relative to the build directory.
-      xcode_settings: The XcodeSettings of the current target.
-      resources: A list of bundle resources, relative to the build directory.
-  """
+    Only call this for mac bundle targets.
+
+    Args:
+        product_dir: Path to the directory containing the output bundle,
+            relative to the build directory.
+        xcode_settings: The XcodeSettings of the current target.
+        resources: A list of bundle resources, relative to the build directory.
+    """
     dest = os.path.join(product_dir, xcode_settings.GetBundleResourceFolder())
     for res in resources:
         output = dest
@@ -1651,24 +1648,24 @@ def GetMacBundleResources(product_dir, xcode_settings, resources):
 
 def GetMacInfoPlist(product_dir, xcode_settings, gyp_path_to_build_path):
     """Returns (info_plist, dest_plist, defines, extra_env), where:
-  * |info_plist| is the source plist path, relative to the
-    build directory,
-  * |dest_plist| is the destination plist path, relative to the
-    build directory,
-  * |defines| is a list of preprocessor defines (empty if the plist
-    shouldn't be preprocessed,
-  * |extra_env| is a dict of env variables that should be exported when
-    invoking |mac_tool copy-info-plist|.
-
-  Only call this for mac bundle targets.
-
-  Args:
-      product_dir: Path to the directory containing the output bundle,
-          relative to the build directory.
-      xcode_settings: The XcodeSettings of the current target.
-      gyp_to_build_path: A function that converts paths relative to the
-          current gyp file to paths relative to the build directory.
-  """
+    * |info_plist| is the source plist path, relative to the
+      build directory,
+    * |dest_plist| is the destination plist path, relative to the
+      build directory,
+    * |defines| is a list of preprocessor defines (empty if the plist
+      shouldn't be preprocessed,
+    * |extra_env| is a dict of env variables that should be exported when
+      invoking |mac_tool copy-info-plist|.
+
+    Only call this for mac bundle targets.
+
+    Args:
+        product_dir: Path to the directory containing the output bundle,
+            relative to the build directory.
+        xcode_settings: The XcodeSettings of the current target.
+        gyp_to_build_path: A function that converts paths relative to the
+            current gyp file to paths relative to the build directory.
+    """
     info_plist = xcode_settings.GetPerTargetSetting("INFOPLIST_FILE")
     if not info_plist:
         return None, None, [], {}
@@ -1706,18 +1703,18 @@ def _GetXcodeEnv(
     xcode_settings, built_products_dir, srcroot, configuration, additional_settings=None
 ):
     """Return the environment variables that Xcode would set. See
-  http://developer.apple.com/library/mac/#documentation/DeveloperTools/Reference/XcodeBuildSettingRef/1-Build_Setting_Reference/build_setting_ref.html#//apple_ref/doc/uid/TP40003931-CH3-SW153
-  for a full list.
-
-  Args:
-      xcode_settings: An XcodeSettings object. If this is None, this function
-          returns an empty dict.
-      built_products_dir: Absolute path to the built products dir.
-      srcroot: Absolute path to the source root.
-      configuration: The build configuration name.
-      additional_settings: An optional dict with more values to add to the
-          result.
-  """
+    http://developer.apple.com/library/mac/#documentation/DeveloperTools/Reference/XcodeBuildSettingRef/1-Build_Setting_Reference/build_setting_ref.html#//apple_ref/doc/uid/TP40003931-CH3-SW153
+    for a full list.
+
+    Args:
+        xcode_settings: An XcodeSettings object. If this is None, this function
+            returns an empty dict.
+        built_products_dir: Absolute path to the built products dir.
+        srcroot: Absolute path to the source root.
+        configuration: The build configuration name.
+        additional_settings: An optional dict with more values to add to the
+            result.
+    """
 
     if not xcode_settings:
         return {}
@@ -1771,17 +1768,17 @@ def _GetXcodeEnv(
         )
         env["CONTENTS_FOLDER_PATH"] = xcode_settings.GetBundleContentsFolderPath()
         env["EXECUTABLE_FOLDER_PATH"] = xcode_settings.GetBundleExecutableFolderPath()
-        env[
-            "UNLOCALIZED_RESOURCES_FOLDER_PATH"
-        ] = xcode_settings.GetBundleResourceFolder()
+        env["UNLOCALIZED_RESOURCES_FOLDER_PATH"] = (
+            xcode_settings.GetBundleResourceFolder()
+        )
         env["JAVA_FOLDER_PATH"] = xcode_settings.GetBundleJavaFolderPath()
         env["FRAMEWORKS_FOLDER_PATH"] = xcode_settings.GetBundleFrameworksFolderPath()
-        env[
-            "SHARED_FRAMEWORKS_FOLDER_PATH"
-        ] = xcode_settings.GetBundleSharedFrameworksFolderPath()
-        env[
-            "SHARED_SUPPORT_FOLDER_PATH"
-        ] = xcode_settings.GetBundleSharedSupportFolderPath()
+        env["SHARED_FRAMEWORKS_FOLDER_PATH"] = (
+            xcode_settings.GetBundleSharedFrameworksFolderPath()
+        )
+        env["SHARED_SUPPORT_FOLDER_PATH"] = (
+            xcode_settings.GetBundleSharedSupportFolderPath()
+        )
         env["PLUGINS_FOLDER_PATH"] = xcode_settings.GetBundlePlugInsFolderPath()
         env["XPCSERVICES_FOLDER_PATH"] = xcode_settings.GetBundleXPCServicesFolderPath()
         env["INFOPLIST_PATH"] = xcode_settings.GetBundlePlistPath()
@@ -1817,8 +1814,8 @@ def _GetXcodeEnv(
 
 def _NormalizeEnvVarReferences(str):
     """Takes a string containing variable references in the form ${FOO}, $(FOO),
-  or $FOO, and returns a string with all variable references in the form ${FOO}.
-  """
+    or $FOO, and returns a string with all variable references in the form ${FOO}.
+    """
     # $FOO -> ${FOO}
     str = re.sub(r"\$([a-zA-Z_][a-zA-Z0-9_]*)", r"${\1}", str)
 
@@ -1834,9 +1831,9 @@ def _NormalizeEnvVarReferences(str):
 
 def ExpandEnvVars(string, expansions):
     """Expands ${VARIABLES}, $(VARIABLES), and $VARIABLES in string per the
-  expansions list. If the variable expands to something that references
-  another variable, this variable is expanded as well if it's in env --
-  until no variables present in env are left."""
+    expansions list. If the variable expands to something that references
+    another variable, this variable is expanded as well if it's in env --
+    until no variables present in env are left."""
     for k, v in reversed(expansions):
         string = string.replace("${" + k + "}", v)
         string = string.replace("$(" + k + ")", v)
@@ -1846,11 +1843,11 @@ def ExpandEnvVars(string, expansions):
 
 def _TopologicallySortedEnvVarKeys(env):
     """Takes a dict |env| whose values are strings that can refer to other keys,
-  for example env['foo'] = '$(bar) and $(baz)'. Returns a list L of all keys of
-  env such that key2 is after key1 in L if env[key2] refers to env[key1].
+    for example env['foo'] = '$(bar) and $(baz)'. Returns a list L of all keys of
+    env such that key2 is after key1 in L if env[key2] refers to env[key1].
 
-  Throws an Exception in case of dependency cycles.
-  """
+    Throws an Exception in case of dependency cycles.
+    """
     # Since environment variables can refer to other variables, the evaluation
     # order is important. Below is the logic to compute the dependency graph
     # and sort it.
@@ -1891,7 +1888,7 @@ def GetSortedXcodeEnv(
 
 def GetSpecPostbuildCommands(spec, quiet=False):
     """Returns the list of postbuilds explicitly defined on |spec|, in a form
-  executable by a shell."""
+    executable by a shell."""
     postbuilds = []
     for postbuild in spec.get("postbuilds", []):
         if not quiet:
@@ -1905,7 +1902,7 @@ def GetSpecPostbuildCommands(spec, quiet=False):
 
 def _HasIOSTarget(targets):
     """Returns true if any target contains the iOS specific key
-  IPHONEOS_DEPLOYMENT_TARGET."""
+    IPHONEOS_DEPLOYMENT_TARGET."""
     for target_dict in targets.values():
         for config in target_dict["configurations"].values():
             if config.get("xcode_settings", {}).get("IPHONEOS_DEPLOYMENT_TARGET"):
@@ -1915,7 +1912,7 @@ def _HasIOSTarget(targets):
 
 def _AddIOSDeviceConfigurations(targets):
     """Clone all targets and append -iphoneos to the name. Configure these targets
-  to build for iOS devices and use correct architectures for those builds."""
+    to build for iOS devices and use correct architectures for those builds."""
     for target_dict in targets.values():
         toolset = target_dict["toolset"]
         configs = target_dict["configurations"]
@@ -1931,7 +1928,7 @@ def _AddIOSDeviceConfigurations(targets):
 
 def CloneConfigurationForDeviceAndEmulator(target_dicts):
     """If |target_dicts| contains any iOS targets, automatically create -iphoneos
-  targets for iOS device builds."""
+    targets for iOS device builds."""
     if _HasIOSTarget(target_dicts):
         return _AddIOSDeviceConfigurations(target_dicts)
     return target_dicts
diff --git a/gyp/pylib/gyp/xcode_ninja.py b/gyp/pylib/gyp/xcode_ninja.py
index ae3079d85a..1a97a06c51 100644
--- a/gyp/pylib/gyp/xcode_ninja.py
+++ b/gyp/pylib/gyp/xcode_ninja.py
@@ -21,7 +21,7 @@
 
 
 def _WriteWorkspace(main_gyp, sources_gyp, params):
-    """ Create a workspace to wrap main and sources gyp paths. """
+    """Create a workspace to wrap main and sources gyp paths."""
     (build_file_root, build_file_ext) = os.path.splitext(main_gyp)
     workspace_path = build_file_root + ".xcworkspace"
     options = params["options"]
@@ -57,7 +57,7 @@ def _WriteWorkspace(main_gyp, sources_gyp, params):
 
 
 def _TargetFromSpec(old_spec, params):
-    """ Create fake target for xcode-ninja wrapper. """
+    """Create fake target for xcode-ninja wrapper."""
     # Determine ninja top level build dir (e.g. /path/to/out).
     ninja_toplevel = None
     jobs = 0
@@ -102,9 +102,9 @@ def _TargetFromSpec(old_spec, params):
                     new_xcode_settings[key] = old_xcode_settings[key]
 
             ninja_target["configurations"][config] = {}
-            ninja_target["configurations"][config][
-                "xcode_settings"
-            ] = new_xcode_settings
+            ninja_target["configurations"][config]["xcode_settings"] = (
+                new_xcode_settings
+            )
 
     ninja_target["mac_bundle"] = old_spec.get("mac_bundle", 0)
     ninja_target["mac_xctest_bundle"] = old_spec.get("mac_xctest_bundle", 0)
@@ -137,13 +137,13 @@ def _TargetFromSpec(old_spec, params):
 def IsValidTargetForWrapper(target_extras, executable_target_pattern, spec):
     """Limit targets for Xcode wrapper.
 
-  Xcode sometimes performs poorly with too many targets, so only include
-  proper executable targets, with filters to customize.
-  Arguments:
-    target_extras: Regular expression to always add, matching any target.
-    executable_target_pattern: Regular expression limiting executable targets.
-    spec: Specifications for target.
-  """
+    Xcode sometimes performs poorly with too many targets, so only include
+    proper executable targets, with filters to customize.
+    Arguments:
+      target_extras: Regular expression to always add, matching any target.
+      executable_target_pattern: Regular expression limiting executable targets.
+      spec: Specifications for target.
+    """
     target_name = spec.get("target_name")
     # Always include targets matching target_extras.
     if target_extras is not None and re.search(target_extras, target_name):
@@ -154,7 +154,6 @@ def IsValidTargetForWrapper(target_extras, executable_target_pattern, spec):
         spec.get("type", "") == "executable"
         and spec.get("product_extension", "") != "bundle"
     ):
-
         # If there is a filter and the target does not match, exclude the target.
         if executable_target_pattern is not None:
             if not re.search(executable_target_pattern, target_name):
@@ -166,14 +165,14 @@ def IsValidTargetForWrapper(target_extras, executable_target_pattern, spec):
 def CreateWrapper(target_list, target_dicts, data, params):
     """Initialize targets for the ninja wrapper.
 
-  This sets up the necessary variables in the targets to generate Xcode projects
-  that use ninja as an external builder.
-  Arguments:
-    target_list: List of target pairs: 'base/base.gyp:base'.
-    target_dicts: Dict of target properties keyed on target pair.
-    data: Dict of flattened build files keyed on gyp path.
-    params: Dict of global options for gyp.
-  """
+    This sets up the necessary variables in the targets to generate Xcode projects
+    that use ninja as an external builder.
+    Arguments:
+      target_list: List of target pairs: 'base/base.gyp:base'.
+      target_dicts: Dict of target properties keyed on target pair.
+      data: Dict of flattened build files keyed on gyp path.
+      params: Dict of global options for gyp.
+    """
     orig_gyp = params["build_files"][0]
     for gyp_name, gyp_dict in data.items():
         if gyp_name == orig_gyp:
diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py
index 0376693d95..11e2be0737 100644
--- a/gyp/pylib/gyp/xcodeproj_file.py
+++ b/gyp/pylib/gyp/xcodeproj_file.py
@@ -176,12 +176,12 @@ def cmp(x, y):
 def SourceTreeAndPathFromPath(input_path):
     """Given input_path, returns a tuple with sourceTree and path values.
 
-  Examples:
-    input_path     (source_tree, output_path)
-    '$(VAR)/path'  ('VAR', 'path')
-    '$(VAR)'       ('VAR', None)
-    'path'         (None, 'path')
-  """
+    Examples:
+      input_path     (source_tree, output_path)
+      '$(VAR)/path'  ('VAR', 'path')
+      '$(VAR)'       ('VAR', None)
+      'path'         (None, 'path')
+    """
 
     if source_group_match := _path_leading_variable.match(input_path):
         source_tree = source_group_match.group(1)
@@ -200,70 +200,70 @@ def ConvertVariablesToShellSyntax(input_string):
 class XCObject:
     """The abstract base of all class types used in Xcode project files.
 
-  Class variables:
-    _schema: A dictionary defining the properties of this class.  The keys to
-             _schema are string property keys as used in project files.  Values
-             are a list of four or five elements:
-             [ is_list, property_type, is_strong, is_required, default ]
-             is_list: True if the property described is a list, as opposed
-                      to a single element.
-             property_type: The type to use as the value of the property,
-                            or if is_list is True, the type to use for each
-                            element of the value's list.  property_type must
-                            be an XCObject subclass, or one of the built-in
-                            types str, int, or dict.
-             is_strong: If property_type is an XCObject subclass, is_strong
-                        is True to assert that this class "owns," or serves
-                        as parent, to the property value (or, if is_list is
-                        True, values).  is_strong must be False if
-                        property_type is not an XCObject subclass.
-             is_required: True if the property is required for the class.
-                          Note that is_required being True does not preclude
-                          an empty string ("", in the case of property_type
-                          str) or list ([], in the case of is_list True) from
-                          being set for the property.
-             default: Optional.  If is_required is True, default may be set
-                      to provide a default value for objects that do not supply
-                      their own value.  If is_required is True and default
-                      is not provided, users of the class must supply their own
-                      value for the property.
-             Note that although the values of the array are expressed in
-             boolean terms, subclasses provide values as integers to conserve
-             horizontal space.
-    _should_print_single_line: False in XCObject.  Subclasses whose objects
-                               should be written to the project file in the
-                               alternate single-line format, such as
-                               PBXFileReference and PBXBuildFile, should
-                               set this to True.
-    _encode_transforms: Used by _EncodeString to encode unprintable characters.
-                        The index into this list is the ordinal of the
-                        character to transform; each value is a string
-                        used to represent the character in the output.  XCObject
-                        provides an _encode_transforms list suitable for most
-                        XCObject subclasses.
-    _alternate_encode_transforms: Provided for subclasses that wish to use
-                                  the alternate encoding rules.  Xcode seems
-                                  to use these rules when printing objects in
-                                  single-line format.  Subclasses that desire
-                                  this behavior should set _encode_transforms
-                                  to _alternate_encode_transforms.
-    _hashables: A list of XCObject subclasses that can be hashed by ComputeIDs
-                to construct this object's ID.  Most classes that need custom
-                hashing behavior should do it by overriding Hashables,
-                but in some cases an object's parent may wish to push a
-                hashable value into its child, and it can do so by appending
-                to _hashables.
-  Attributes:
-    id: The object's identifier, a 24-character uppercase hexadecimal string.
-        Usually, objects being created should not set id until the entire
-        project file structure is built.  At that point, UpdateIDs() should
-        be called on the root object to assign deterministic values for id to
-        each object in the tree.
-    parent: The object's parent.  This is set by a parent XCObject when a child
-            object is added to it.
-    _properties: The object's property dictionary.  An object's properties are
-                 described by its class' _schema variable.
-  """
+    Class variables:
+      _schema: A dictionary defining the properties of this class.  The keys to
+               _schema are string property keys as used in project files.  Values
+               are a list of four or five elements:
+               [ is_list, property_type, is_strong, is_required, default ]
+               is_list: True if the property described is a list, as opposed
+                        to a single element.
+               property_type: The type to use as the value of the property,
+                              or if is_list is True, the type to use for each
+                              element of the value's list.  property_type must
+                              be an XCObject subclass, or one of the built-in
+                              types str, int, or dict.
+               is_strong: If property_type is an XCObject subclass, is_strong
+                          is True to assert that this class "owns," or serves
+                          as parent, to the property value (or, if is_list is
+                          True, values).  is_strong must be False if
+                          property_type is not an XCObject subclass.
+               is_required: True if the property is required for the class.
+                            Note that is_required being True does not preclude
+                            an empty string ("", in the case of property_type
+                            str) or list ([], in the case of is_list True) from
+                            being set for the property.
+               default: Optional.  If is_required is True, default may be set
+                        to provide a default value for objects that do not supply
+                        their own value.  If is_required is True and default
+                        is not provided, users of the class must supply their own
+                        value for the property.
+               Note that although the values of the array are expressed in
+               boolean terms, subclasses provide values as integers to conserve
+               horizontal space.
+      _should_print_single_line: False in XCObject.  Subclasses whose objects
+                                 should be written to the project file in the
+                                 alternate single-line format, such as
+                                 PBXFileReference and PBXBuildFile, should
+                                 set this to True.
+      _encode_transforms: Used by _EncodeString to encode unprintable characters.
+                          The index into this list is the ordinal of the
+                          character to transform; each value is a string
+                          used to represent the character in the output.  XCObject
+                          provides an _encode_transforms list suitable for most
+                          XCObject subclasses.
+      _alternate_encode_transforms: Provided for subclasses that wish to use
+                                    the alternate encoding rules.  Xcode seems
+                                    to use these rules when printing objects in
+                                    single-line format.  Subclasses that desire
+                                    this behavior should set _encode_transforms
+                                    to _alternate_encode_transforms.
+      _hashables: A list of XCObject subclasses that can be hashed by ComputeIDs
+                  to construct this object's ID.  Most classes that need custom
+                  hashing behavior should do it by overriding Hashables,
+                  but in some cases an object's parent may wish to push a
+                  hashable value into its child, and it can do so by appending
+                  to _hashables.
+    Attributes:
+      id: The object's identifier, a 24-character uppercase hexadecimal string.
+          Usually, objects being created should not set id until the entire
+          project file structure is built.  At that point, UpdateIDs() should
+          be called on the root object to assign deterministic values for id to
+          each object in the tree.
+      parent: The object's parent.  This is set by a parent XCObject when a child
+              object is added to it.
+      _properties: The object's property dictionary.  An object's properties are
+                   described by its class' _schema variable.
+    """
 
     _schema = {}
     _should_print_single_line = False
@@ -305,12 +305,12 @@ def __repr__(self):
     def Copy(self):
         """Make a copy of this object.
 
-    The new object will have its own copy of lists and dicts.  Any XCObject
-    objects owned by this object (marked "strong") will be copied in the
-    new object, even those found in lists.  If this object has any weak
-    references to other XCObjects, the same references are added to the new
-    object without making a copy.
-    """
+        The new object will have its own copy of lists and dicts.  Any XCObject
+        objects owned by this object (marked "strong") will be copied in the
+        new object, even those found in lists.  If this object has any weak
+        references to other XCObjects, the same references are added to the new
+        object without making a copy.
+        """
 
         that = self.__class__(id=self.id, parent=self.parent)
         for key, value in self._properties.items():
@@ -359,9 +359,9 @@ def Copy(self):
     def Name(self):
         """Return the name corresponding to an object.
 
-    Not all objects necessarily need to be nameable, and not all that do have
-    a "name" property.  Override as needed.
-    """
+        Not all objects necessarily need to be nameable, and not all that do have
+        a "name" property.  Override as needed.
+        """
 
         # If the schema indicates that "name" is required, try to access the
         # property even if it doesn't exist.  This will result in a KeyError
@@ -377,12 +377,12 @@ def Name(self):
     def Comment(self):
         """Return a comment string for the object.
 
-    Most objects just use their name as the comment, but PBXProject uses
-    different values.
+        Most objects just use their name as the comment, but PBXProject uses
+        different values.
 
-    The returned comment is not escaped and does not have any comment marker
-    strings applied to it.
-    """
+        The returned comment is not escaped and does not have any comment marker
+        strings applied to it.
+        """
 
         return self.Name()
 
@@ -402,26 +402,26 @@ def HashablesForChild(self):
     def ComputeIDs(self, recursive=True, overwrite=True, seed_hash=None):
         """Set "id" properties deterministically.
 
-    An object's "id" property is set based on a hash of its class type and
-    name, as well as the class type and name of all ancestor objects.  As
-    such, it is only advisable to call ComputeIDs once an entire project file
-    tree is built.
+        An object's "id" property is set based on a hash of its class type and
+        name, as well as the class type and name of all ancestor objects.  As
+        such, it is only advisable to call ComputeIDs once an entire project file
+        tree is built.
 
-    If recursive is True, recurse into all descendant objects and update their
-    hashes.
+        If recursive is True, recurse into all descendant objects and update their
+        hashes.
 
-    If overwrite is True, any existing value set in the "id" property will be
-    replaced.
-    """
+        If overwrite is True, any existing value set in the "id" property will be
+        replaced.
+        """
 
         def _HashUpdate(hash, data):
             """Update hash with data's length and contents.
 
-      If the hash were updated only with the value of data, it would be
-      possible for clowns to induce collisions by manipulating the names of
-      their objects.  By adding the length, it's exceedingly less likely that
-      ID collisions will be encountered, intentionally or not.
-      """
+            If the hash were updated only with the value of data, it would be
+            possible for clowns to induce collisions by manipulating the names of
+            their objects.  By adding the length, it's exceedingly less likely that
+            ID collisions will be encountered, intentionally or not.
+            """
 
             hash.update(struct.pack(">i", len(data)))
             if isinstance(data, str):
@@ -464,8 +464,7 @@ def _HashUpdate(hash, data):
             self.id = "%08X%08X%08X" % tuple(id_ints)
 
     def EnsureNoIDCollisions(self):
-        """Verifies that no two objects have the same ID.  Checks all descendants.
-    """
+        """Verifies that no two objects have the same ID.  Checks all descendants."""
 
         ids = {}
         descendants = self.Descendants()
@@ -498,8 +497,8 @@ def Children(self):
 
     def Descendants(self):
         """Returns a list of all of this object's descendants, including this
-    object.
-    """
+        object.
+        """
 
         children = self.Children()
         descendants = [self]
@@ -515,8 +514,8 @@ def PBXProjectAncestor(self):
 
     def _EncodeComment(self, comment):
         """Encodes a comment to be placed in the project file output, mimicking
-    Xcode behavior.
-    """
+        Xcode behavior.
+        """
 
         # This mimics Xcode behavior by wrapping the comment in "/*" and "*/".  If
         # the string already contains a "*/", it is turned into "(*)/".  This keeps
@@ -543,8 +542,8 @@ def _EncodeTransform(self, match):
 
     def _EncodeString(self, value):
         """Encodes a string to be placed in the project file output, mimicking
-    Xcode behavior.
-    """
+        Xcode behavior.
+        """
 
         # Use quotation marks when any character outside of the range A-Z, a-z, 0-9,
         # $ (dollar sign), . (period), and _ (underscore) is present.  Also use
@@ -585,18 +584,18 @@ def _XCPrint(self, file, tabs, line):
 
     def _XCPrintableValue(self, tabs, value, flatten_list=False):
         """Returns a representation of value that may be printed in a project file,
-    mimicking Xcode's behavior.
+        mimicking Xcode's behavior.
 
-    _XCPrintableValue can handle str and int values, XCObjects (which are
-    made printable by returning their id property), and list and dict objects
-    composed of any of the above types.  When printing a list or dict, and
-    _should_print_single_line is False, the tabs parameter is used to determine
-    how much to indent the lines corresponding to the items in the list or
-    dict.
+        _XCPrintableValue can handle str and int values, XCObjects (which are
+        made printable by returning their id property), and list and dict objects
+        composed of any of the above types.  When printing a list or dict, and
+        _should_print_single_line is False, the tabs parameter is used to determine
+        how much to indent the lines corresponding to the items in the list or
+        dict.
 
-    If flatten_list is True, single-element lists will be transformed into
-    strings.
-    """
+        If flatten_list is True, single-element lists will be transformed into
+        strings.
+        """
 
         printable = ""
         comment = None
@@ -657,12 +656,12 @@ def _XCPrintableValue(self, tabs, value, flatten_list=False):
 
     def _XCKVPrint(self, file, tabs, key, value):
         """Prints a key and value, members of an XCObject's _properties dictionary,
-    to file.
+        to file.
 
-    tabs is an int identifying the indentation level.  If the class'
-    _should_print_single_line variable is True, tabs is ignored and the
-    key-value pair will be followed by a space instead of a newline.
-    """
+        tabs is an int identifying the indentation level.  If the class'
+        _should_print_single_line variable is True, tabs is ignored and the
+        key-value pair will be followed by a space instead of a newline.
+        """
 
         if self._should_print_single_line:
             printable = ""
@@ -720,8 +719,8 @@ def _XCKVPrint(self, file, tabs, key, value):
 
     def Print(self, file=sys.stdout):
         """Prints a reprentation of this object to file, adhering to Xcode output
-    formatting.
-    """
+        formatting.
+        """
 
         self.VerifyHasRequiredProperties()
 
@@ -759,15 +758,15 @@ def Print(self, file=sys.stdout):
     def UpdateProperties(self, properties, do_copy=False):
         """Merge the supplied properties into the _properties dictionary.
 
-    The input properties must adhere to the class schema or a KeyError or
-    TypeError exception will be raised.  If adding an object of an XCObject
-    subclass and the schema indicates a strong relationship, the object's
-    parent will be set to this object.
+        The input properties must adhere to the class schema or a KeyError or
+        TypeError exception will be raised.  If adding an object of an XCObject
+        subclass and the schema indicates a strong relationship, the object's
+        parent will be set to this object.
 
-    If do_copy is True, then lists, dicts, strong-owned XCObjects, and
-    strong-owned XCObjects in lists will be copied instead of having their
-    references added.
-    """
+        If do_copy is True, then lists, dicts, strong-owned XCObjects, and
+        strong-owned XCObjects in lists will be copied instead of having their
+        references added.
+        """
 
         if properties is None:
             return
@@ -908,8 +907,8 @@ def AppendProperty(self, key, value):
 
     def VerifyHasRequiredProperties(self):
         """Ensure that all properties identified as required by the schema are
-    set.
-    """
+        set.
+        """
 
         # TODO(mark): A stronger verification mechanism is needed.  Some
         # subclasses need to perform validation beyond what the schema can enforce.
@@ -920,7 +919,7 @@ def VerifyHasRequiredProperties(self):
 
     def _SetDefaultsFromSchema(self):
         """Assign object default values according to the schema.  This will not
-    overwrite properties that have already been set."""
+        overwrite properties that have already been set."""
 
         defaults = {}
         for property, attributes in self._schema.items():
@@ -942,7 +941,7 @@ def _SetDefaultsFromSchema(self):
 
 class XCHierarchicalElement(XCObject):
     """Abstract base for PBXGroup and PBXFileReference.  Not represented in a
-  project file."""
+    project file."""
 
     # TODO(mark): Do name and path belong here?  Probably so.
     # If path is set and name is not, name may have a default value.  Name will
@@ -1008,27 +1007,27 @@ def Name(self):
     def Hashables(self):
         """Custom hashables for XCHierarchicalElements.
 
-    XCHierarchicalElements are special.  Generally, their hashes shouldn't
-    change if the paths don't change.  The normal XCObject implementation of
-    Hashables adds a hashable for each object, which means that if
-    the hierarchical structure changes (possibly due to changes caused when
-    TakeOverOnlyChild runs and encounters slight changes in the hierarchy),
-    the hashes will change.  For example, if a project file initially contains
-    a/b/f1 and a/b becomes collapsed into a/b, f1 will have a single parent
-    a/b.  If someone later adds a/f2 to the project file, a/b can no longer be
-    collapsed, and f1 winds up with parent b and grandparent a.  That would
-    be sufficient to change f1's hash.
-
-    To counteract this problem, hashables for all XCHierarchicalElements except
-    for the main group (which has neither a name nor a path) are taken to be
-    just the set of path components.  Because hashables are inherited from
-    parents, this provides assurance that a/b/f1 has the same set of hashables
-    whether its parent is b or a/b.
-
-    The main group is a special case.  As it is permitted to have no name or
-    path, it is permitted to use the standard XCObject hash mechanism.  This
-    is not considered a problem because there can be only one main group.
-    """
+        XCHierarchicalElements are special.  Generally, their hashes shouldn't
+        change if the paths don't change.  The normal XCObject implementation of
+        Hashables adds a hashable for each object, which means that if
+        the hierarchical structure changes (possibly due to changes caused when
+        TakeOverOnlyChild runs and encounters slight changes in the hierarchy),
+        the hashes will change.  For example, if a project file initially contains
+        a/b/f1 and a/b becomes collapsed into a/b, f1 will have a single parent
+        a/b.  If someone later adds a/f2 to the project file, a/b can no longer be
+        collapsed, and f1 winds up with parent b and grandparent a.  That would
+        be sufficient to change f1's hash.
+
+        To counteract this problem, hashables for all XCHierarchicalElements except
+        for the main group (which has neither a name nor a path) are taken to be
+        just the set of path components.  Because hashables are inherited from
+        parents, this provides assurance that a/b/f1 has the same set of hashables
+        whether its parent is b or a/b.
+
+        The main group is a special case.  As it is permitted to have no name or
+        path, it is permitted to use the standard XCObject hash mechanism.  This
+        is not considered a problem because there can be only one main group.
+        """
 
         if self == self.PBXProjectAncestor()._properties["mainGroup"]:
             # super
@@ -1157,12 +1156,12 @@ def FullPath(self):
 
 class PBXGroup(XCHierarchicalElement):
     """
-  Attributes:
-    _children_by_path: Maps pathnames of children of this PBXGroup to the
-      actual child XCHierarchicalElement objects.
-    _variant_children_by_name_and_path: Maps (name, path) tuples of
-      PBXVariantGroup children to the actual child PBXVariantGroup objects.
-  """
+    Attributes:
+      _children_by_path: Maps pathnames of children of this PBXGroup to the
+        actual child XCHierarchicalElement objects.
+      _variant_children_by_name_and_path: Maps (name, path) tuples of
+        PBXVariantGroup children to the actual child PBXVariantGroup objects.
+    """
 
     _schema = XCHierarchicalElement._schema.copy()
     _schema.update(
@@ -1281,20 +1280,20 @@ def GetChildByRemoteObject(self, remote_object):
     def AddOrGetFileByPath(self, path, hierarchical):
         """Returns an existing or new file reference corresponding to path.
 
-    If hierarchical is True, this method will create or use the necessary
-    hierarchical group structure corresponding to path.  Otherwise, it will
-    look in and create an item in the current group only.
+        If hierarchical is True, this method will create or use the necessary
+        hierarchical group structure corresponding to path.  Otherwise, it will
+        look in and create an item in the current group only.
 
-    If an existing matching reference is found, it is returned, otherwise, a
-    new one will be created, added to the correct group, and returned.
+        If an existing matching reference is found, it is returned, otherwise, a
+        new one will be created, added to the correct group, and returned.
 
-    If path identifies a directory by virtue of carrying a trailing slash,
-    this method returns a PBXFileReference of "folder" type.  If path
-    identifies a variant, by virtue of it identifying a file inside a directory
-    with an ".lproj" extension, this method returns a PBXVariantGroup
-    containing the variant named by path, and possibly other variants.  For
-    all other paths, a "normal" PBXFileReference will be returned.
-    """
+        If path identifies a directory by virtue of carrying a trailing slash,
+        this method returns a PBXFileReference of "folder" type.  If path
+        identifies a variant, by virtue of it identifying a file inside a directory
+        with an ".lproj" extension, this method returns a PBXVariantGroup
+        containing the variant named by path, and possibly other variants.  For
+        all other paths, a "normal" PBXFileReference will be returned.
+        """
 
         # Adding or getting a directory?  Directories end with a trailing slash.
         is_dir = False
@@ -1379,15 +1378,15 @@ def AddOrGetFileByPath(self, path, hierarchical):
     def AddOrGetVariantGroupByNameAndPath(self, name, path):
         """Returns an existing or new PBXVariantGroup for name and path.
 
-    If a PBXVariantGroup identified by the name and path arguments is already
-    present as a child of this object, it is returned.  Otherwise, a new
-    PBXVariantGroup with the correct properties is created, added as a child,
-    and returned.
+        If a PBXVariantGroup identified by the name and path arguments is already
+        present as a child of this object, it is returned.  Otherwise, a new
+        PBXVariantGroup with the correct properties is created, added as a child,
+        and returned.
 
-    This method will generally be called by AddOrGetFileByPath, which knows
-    when to create a variant group based on the structure of the pathnames
-    passed to it.
-    """
+        This method will generally be called by AddOrGetFileByPath, which knows
+        when to create a variant group based on the structure of the pathnames
+        passed to it.
+        """
 
         key = (name, path)
         if key in self._variant_children_by_name_and_path:
@@ -1405,19 +1404,19 @@ def AddOrGetVariantGroupByNameAndPath(self, name, path):
 
     def TakeOverOnlyChild(self, recurse=False):
         """If this PBXGroup has only one child and it's also a PBXGroup, take
-    it over by making all of its children this object's children.
-
-    This function will continue to take over only children when those children
-    are groups.  If there are three PBXGroups representing a, b, and c, with
-    c inside b and b inside a, and a and b have no other children, this will
-    result in a taking over both b and c, forming a PBXGroup for a/b/c.
-
-    If recurse is True, this function will recurse into children and ask them
-    to collapse themselves by taking over only children as well.  Assuming
-    an example hierarchy with files at a/b/c/d1, a/b/c/d2, and a/b/c/d3/e/f
-    (d1, d2, and f are files, the rest are groups), recursion will result in
-    a group for a/b/c containing a group for d3/e.
-    """
+        it over by making all of its children this object's children.
+
+        This function will continue to take over only children when those children
+        are groups.  If there are three PBXGroups representing a, b, and c, with
+        c inside b and b inside a, and a and b have no other children, this will
+        result in a taking over both b and c, forming a PBXGroup for a/b/c.
+
+        If recurse is True, this function will recurse into children and ask them
+        to collapse themselves by taking over only children as well.  Assuming
+        an example hierarchy with files at a/b/c/d1, a/b/c/d2, and a/b/c/d3/e/f
+        (d1, d2, and f are files, the rest are groups), recursion will result in
+        a group for a/b/c containing a group for d3/e.
+        """
 
         # At this stage, check that child class types are PBXGroup exactly,
         # instead of using isinstance.  The only subclass of PBXGroup,
@@ -1716,16 +1715,16 @@ def DefaultConfiguration(self):
 
     def HasBuildSetting(self, key):
         """Determines the state of a build setting in all XCBuildConfiguration
-    child objects.
+        child objects.
 
-    If all child objects have key in their build settings, and the value is the
-    same in all child objects, returns 1.
+        If all child objects have key in their build settings, and the value is the
+        same in all child objects, returns 1.
 
-    If no child objects have the key in their build settings, returns 0.
+        If no child objects have the key in their build settings, returns 0.
 
-    If some, but not all, child objects have the key in their build settings,
-    or if any children have different values for the key, returns -1.
-    """
+        If some, but not all, child objects have the key in their build settings,
+        or if any children have different values for the key, returns -1.
+        """
 
         has = None
         value = None
@@ -1751,9 +1750,9 @@ def HasBuildSetting(self, key):
     def GetBuildSetting(self, key):
         """Gets the build setting for key.
 
-    All child XCConfiguration objects must have the same value set for the
-    setting, or a ValueError will be raised.
-    """
+        All child XCConfiguration objects must have the same value set for the
+        setting, or a ValueError will be raised.
+        """
 
         # TODO(mark): This is wrong for build settings that are lists.  The list
         # contents should be compared (and a list copy returned?)
@@ -1770,31 +1769,30 @@ def GetBuildSetting(self, key):
 
     def SetBuildSetting(self, key, value):
         """Sets the build setting for key to value in all child
-    XCBuildConfiguration objects.
-    """
+        XCBuildConfiguration objects.
+        """
 
         for configuration in self._properties["buildConfigurations"]:
             configuration.SetBuildSetting(key, value)
 
     def AppendBuildSetting(self, key, value):
         """Appends value to the build setting for key, which is treated as a list,
-    in all child XCBuildConfiguration objects.
-    """
+        in all child XCBuildConfiguration objects.
+        """
 
         for configuration in self._properties["buildConfigurations"]:
             configuration.AppendBuildSetting(key, value)
 
     def DelBuildSetting(self, key):
         """Deletes the build setting key from all child XCBuildConfiguration
-    objects.
-    """
+        objects.
+        """
 
         for configuration in self._properties["buildConfigurations"]:
             configuration.DelBuildSetting(key)
 
     def SetBaseConfiguration(self, value):
-        """Sets the build configuration in all child XCBuildConfiguration objects.
-    """
+        """Sets the build configuration in all child XCBuildConfiguration objects."""
 
         for configuration in self._properties["buildConfigurations"]:
             configuration.SetBaseConfiguration(value)
@@ -1834,14 +1832,14 @@ def Hashables(self):
 
 class XCBuildPhase(XCObject):
     """Abstract base for build phase classes.  Not represented in a project
-  file.
+    file.
 
-  Attributes:
-    _files_by_path: A dict mapping each path of a child in the files list by
-      path (keys) to the corresponding PBXBuildFile children (values).
-    _files_by_xcfilelikeelement: A dict mapping each XCFileLikeElement (keys)
-      to the corresponding PBXBuildFile children (values).
-  """
+    Attributes:
+      _files_by_path: A dict mapping each path of a child in the files list by
+        path (keys) to the corresponding PBXBuildFile children (values).
+      _files_by_xcfilelikeelement: A dict mapping each XCFileLikeElement (keys)
+        to the corresponding PBXBuildFile children (values).
+    """
 
     # TODO(mark): Some build phase types, like PBXShellScriptBuildPhase, don't
     # actually have a "files" list.  XCBuildPhase should not have "files" but
@@ -1880,8 +1878,8 @@ def FileGroup(self, path):
     def _AddPathToDict(self, pbxbuildfile, path):
         """Adds path to the dict tracking paths belonging to this build phase.
 
-    If the path is already a member of this build phase, raises an exception.
-    """
+        If the path is already a member of this build phase, raises an exception.
+        """
 
         if path in self._files_by_path:
             raise ValueError("Found multiple build files with path " + path)
@@ -1890,28 +1888,28 @@ def _AddPathToDict(self, pbxbuildfile, path):
     def _AddBuildFileToDicts(self, pbxbuildfile, path=None):
         """Maintains the _files_by_path and _files_by_xcfilelikeelement dicts.
 
-    If path is specified, then it is the path that is being added to the
-    phase, and pbxbuildfile must contain either a PBXFileReference directly
-    referencing that path, or it must contain a PBXVariantGroup that itself
-    contains a PBXFileReference referencing the path.
-
-    If path is not specified, either the PBXFileReference's path or the paths
-    of all children of the PBXVariantGroup are taken as being added to the
-    phase.
-
-    If the path is already present in the phase, raises an exception.
-
-    If the PBXFileReference or PBXVariantGroup referenced by pbxbuildfile
-    are already present in the phase, referenced by a different PBXBuildFile
-    object, raises an exception.  This does not raise an exception when
-    a PBXFileReference or PBXVariantGroup reappear and are referenced by the
-    same PBXBuildFile that has already introduced them, because in the case
-    of PBXVariantGroup objects, they may correspond to multiple paths that are
-    not all added simultaneously.  When this situation occurs, the path needs
-    to be added to _files_by_path, but nothing needs to change in
-    _files_by_xcfilelikeelement, and the caller should have avoided adding
-    the PBXBuildFile if it is already present in the list of children.
-    """
+        If path is specified, then it is the path that is being added to the
+        phase, and pbxbuildfile must contain either a PBXFileReference directly
+        referencing that path, or it must contain a PBXVariantGroup that itself
+        contains a PBXFileReference referencing the path.
+
+        If path is not specified, either the PBXFileReference's path or the paths
+        of all children of the PBXVariantGroup are taken as being added to the
+        phase.
+
+        If the path is already present in the phase, raises an exception.
+
+        If the PBXFileReference or PBXVariantGroup referenced by pbxbuildfile
+        are already present in the phase, referenced by a different PBXBuildFile
+        object, raises an exception.  This does not raise an exception when
+        a PBXFileReference or PBXVariantGroup reappear and are referenced by the
+        same PBXBuildFile that has already introduced them, because in the case
+        of PBXVariantGroup objects, they may correspond to multiple paths that are
+        not all added simultaneously.  When this situation occurs, the path needs
+        to be added to _files_by_path, but nothing needs to change in
+        _files_by_xcfilelikeelement, and the caller should have avoided adding
+        the PBXBuildFile if it is already present in the list of children.
+        """
 
         xcfilelikeelement = pbxbuildfile._properties["fileRef"]
 
@@ -2102,9 +2100,9 @@ def FileGroup(self, path):
     def SetDestination(self, path):
         """Set the dstSubfolderSpec and dstPath properties from path.
 
-    path may be specified in the same notation used for XCHierarchicalElements,
-    specifically, "$(DIR)/path".
-    """
+        path may be specified in the same notation used for XCHierarchicalElements,
+        specifically, "$(DIR)/path".
+        """
 
         if path_tree_match := self.path_tree_re.search(path):
             path_tree = path_tree_match.group(1)
@@ -2178,9 +2176,7 @@ def SetDestination(self, path):
             subfolder = 0
             relative_path = path[1:]
         else:
-            raise ValueError(
-                f"Can't use path {path} in a {self.__class__.__name__}"
-            )
+            raise ValueError(f"Can't use path {path} in a {self.__class__.__name__}")
 
         self._properties["dstPath"] = relative_path
         self._properties["dstSubfolderSpec"] = subfolder
@@ -2530,9 +2526,9 @@ def __init__(
                 # loadable modules, but there's precedent: Python loadable modules on
                 # Mac OS X use an .so extension.
                 if self._properties["productType"] == "com.googlecode.gyp.xcode.bundle":
-                    self._properties[
-                        "productType"
-                    ] = "com.apple.product-type.library.dynamic"
+                    self._properties["productType"] = (
+                        "com.apple.product-type.library.dynamic"
+                    )
                     self.SetBuildSetting("MACH_O_TYPE", "mh_bundle")
                     self.SetBuildSetting("DYLIB_CURRENT_VERSION", "")
                     self.SetBuildSetting("DYLIB_COMPATIBILITY_VERSION", "")
@@ -2540,9 +2536,10 @@ def __init__(
                         force_extension = suffix[1:]
 
                 if (
-                    self._properties["productType"] in {
+                    self._properties["productType"]
+                    in {
                         "com.apple.product-type-bundle.unit.test",
-                        "com.apple.product-type-bundle.ui-testing"
+                        "com.apple.product-type-bundle.ui-testing",
                     }
                 ) and force_extension is None:
                     force_extension = suffix[1:]
@@ -2694,10 +2691,8 @@ def AddDependency(self, other):
                 other._properties["productType"] == static_library_type
                 or (
                     (
-                        other._properties["productType"] in {
-                            shared_library_type,
-                            framework_type
-                        }
+                        other._properties["productType"]
+                        in {shared_library_type, framework_type}
                     )
                     and (
                         (not other.HasBuildSetting("MACH_O_TYPE"))
@@ -2706,7 +2701,6 @@ def AddDependency(self, other):
                 )
             )
         ):
-
             file_ref = other.GetProperty("productReference")
 
             pbxproject = self.PBXProjectAncestor()
@@ -2732,13 +2726,13 @@ class PBXProject(XCContainerPortal):
     # PBXContainerItemProxy.
     """
 
-  Attributes:
-    path: "sample.xcodeproj".  TODO(mark) Document me!
-    _other_pbxprojects: A dictionary, keyed by other PBXProject objects.  Each
-                        value is a reference to the dict in the
-                        projectReferences list associated with the keyed
-                        PBXProject.
-  """
+    Attributes:
+      path: "sample.xcodeproj".  TODO(mark) Document me!
+      _other_pbxprojects: A dictionary, keyed by other PBXProject objects.  Each
+                          value is a reference to the dict in the
+                          projectReferences list associated with the keyed
+                          PBXProject.
+    """
 
     _schema = XCContainerPortal._schema.copy()
     _schema.update(
@@ -2833,17 +2827,17 @@ def ProjectsGroup(self):
     def RootGroupForPath(self, path):
         """Returns a PBXGroup child of this object to which path should be added.
 
-    This method is intended to choose between SourceGroup and
-    IntermediatesGroup on the basis of whether path is present in a source
-    directory or an intermediates directory.  For the purposes of this
-    determination, any path located within a derived file directory such as
-    PROJECT_DERIVED_FILE_DIR is treated as being in an intermediates
-    directory.
+        This method is intended to choose between SourceGroup and
+        IntermediatesGroup on the basis of whether path is present in a source
+        directory or an intermediates directory.  For the purposes of this
+        determination, any path located within a derived file directory such as
+        PROJECT_DERIVED_FILE_DIR is treated as being in an intermediates
+        directory.
 
-    The returned value is a two-element tuple.  The first element is the
-    PBXGroup, and the second element specifies whether that group should be
-    organized hierarchically (True) or as a single flat list (False).
-    """
+        The returned value is a two-element tuple.  The first element is the
+        PBXGroup, and the second element specifies whether that group should be
+        organized hierarchically (True) or as a single flat list (False).
+        """
 
         # TODO(mark): make this a class variable and bind to self on call?
         # Also, this list is nowhere near exhaustive.
@@ -2869,11 +2863,11 @@ def RootGroupForPath(self, path):
 
     def AddOrGetFileInRootGroup(self, path):
         """Returns a PBXFileReference corresponding to path in the correct group
-    according to RootGroupForPath's heuristics.
+        according to RootGroupForPath's heuristics.
 
-    If an existing PBXFileReference for path exists, it will be returned.
-    Otherwise, one will be created and returned.
-    """
+        If an existing PBXFileReference for path exists, it will be returned.
+        Otherwise, one will be created and returned.
+        """
 
         (group, hierarchical) = self.RootGroupForPath(path)
         return group.AddOrGetFileByPath(path, hierarchical)
@@ -2923,17 +2917,17 @@ def SortGroups(self):
 
     def AddOrGetProjectReference(self, other_pbxproject):
         """Add a reference to another project file (via PBXProject object) to this
-    one.
+        one.
 
-    Returns [ProductGroup, ProjectRef].  ProductGroup is a PBXGroup object in
-    this project file that contains a PBXReferenceProxy object for each
-    product of each PBXNativeTarget in the other project file.  ProjectRef is
-    a PBXFileReference to the other project file.
+        Returns [ProductGroup, ProjectRef].  ProductGroup is a PBXGroup object in
+        this project file that contains a PBXReferenceProxy object for each
+        product of each PBXNativeTarget in the other project file.  ProjectRef is
+        a PBXFileReference to the other project file.
 
-    If this project file already references the other project file, the
-    existing ProductGroup and ProjectRef are returned.  The ProductGroup will
-    still be updated if necessary.
-    """
+        If this project file already references the other project file, the
+        existing ProductGroup and ProjectRef are returned.  The ProductGroup will
+        still be updated if necessary.
+        """
 
         if "projectReferences" not in self._properties:
             self._properties["projectReferences"] = []
@@ -2985,7 +2979,7 @@ def AddOrGetProjectReference(self, other_pbxproject):
             # Xcode seems to sort this list case-insensitively
             self._properties["projectReferences"] = sorted(
                 self._properties["projectReferences"],
-                key=lambda x: x["ProjectRef"].Name().lower()
+                key=lambda x: x["ProjectRef"].Name().lower(),
             )
         else:
             # The link already exists.  Pull out the relevant data.
@@ -3010,11 +3004,8 @@ def _AllSymrootsUnique(self, target, inherit_unique_symroot):
         # define an explicit value for 'SYMROOT'.
         symroots = self._DefinedSymroots(target)
         for s in self._DefinedSymroots(target):
-            if (
-                (s is not None
-                and not self._IsUniqueSymrootForTarget(s))
-                or (s is None
-                and not inherit_unique_symroot)
+            if (s is not None and not self._IsUniqueSymrootForTarget(s)) or (
+                s is None and not inherit_unique_symroot
             ):
                 return False
         return True if symroots else inherit_unique_symroot
@@ -3118,7 +3109,8 @@ def CompareProducts(x, y, remote_products):
             product_group._properties["children"] = sorted(
                 product_group._properties["children"],
                 key=cmp_to_key(
-                    lambda x, y, rp=remote_products: CompareProducts(x, y, rp)),
+                    lambda x, y, rp=remote_products: CompareProducts(x, y, rp)
+                ),
             )
 
 
@@ -3152,9 +3144,7 @@ def Print(self, file=sys.stdout):
             self._XCPrint(file, 0, "{ ")
         else:
             self._XCPrint(file, 0, "{\n")
-        for property, value in sorted(
-            self._properties.items()
-        ):
+        for property, value in sorted(self._properties.items()):
             if property == "objects":
                 self._PrintObjects(file)
             else:
@@ -3180,9 +3170,7 @@ def _PrintObjects(self, file):
         for class_name in sorted(objects_by_class):
             self._XCPrint(file, 0, "\n")
             self._XCPrint(file, 0, "/* Begin " + class_name + " section */\n")
-            for object in sorted(
-                objects_by_class[class_name], key=attrgetter("id")
-            ):
+            for object in sorted(objects_by_class[class_name], key=attrgetter("id")):
                 object.Print(file)
             self._XCPrint(file, 0, "/* End " + class_name + " section */\n")
 
diff --git a/gyp/pylib/gyp/xml_fix.py b/gyp/pylib/gyp/xml_fix.py
index 5301963669..d7e3b5a956 100644
--- a/gyp/pylib/gyp/xml_fix.py
+++ b/gyp/pylib/gyp/xml_fix.py
@@ -9,7 +9,6 @@
 TODO(bradnelson): Consider dropping this when we drop XP support.
 """
 
-
 import xml.dom.minidom
 
 
diff --git a/gyp/pyproject.toml b/gyp/pyproject.toml
index b233d8504d..3a029c4fc5 100644
--- a/gyp/pyproject.toml
+++ b/gyp/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "gyp-next"
-version = "0.20.3"
+version = "0.20.4"
 authors = [
   { name="Node.js contributors", email="ryzokuken@disroot.org" },
 ]
diff --git a/gyp/test_gyp.py b/gyp/test_gyp.py
index 8e910a2b76..70c81ae8ca 100755
--- a/gyp/test_gyp.py
+++ b/gyp/test_gyp.py
@@ -5,7 +5,6 @@
 
 """gyptest.py -- test runner for GYP tests."""
 
-
 import argparse
 import os
 import platform
diff --git a/gyp/tools/graphviz.py b/gyp/tools/graphviz.py
index f19426b69f..ed1c7ab3cd 100755
--- a/gyp/tools/graphviz.py
+++ b/gyp/tools/graphviz.py
@@ -8,7 +8,6 @@
 generate input suitable for graphviz to render a dependency graph of
 targets."""
 
-
 import collections
 import json
 import sys
@@ -22,7 +21,7 @@ def ParseTarget(target):
 
 def LoadEdges(filename, targets):
     """Load the edges map from the dump file, and filter it to only
-  show targets in |targets| and their depedendents."""
+    show targets in |targets| and their depedendents."""
 
     file = open("dump.json")
     edges = json.load(file)
@@ -43,7 +42,7 @@ def LoadEdges(filename, targets):
 
 def WriteGraph(edges):
     """Print a graphviz graph to stdout.
-  |edges| is a map of target to a list of other targets it depends on."""
+    |edges| is a map of target to a list of other targets it depends on."""
 
     # Bucket targets by file.
     files = collections.defaultdict(list)
@@ -64,9 +63,7 @@ def WriteGraph(edges):
             # the display by making it a box without an internal node.
             target = targets[0]
             build_file, target_name, toolset = ParseTarget(target)
-            print(
-                f'  "{target}" [shape=box, label="{filename}\\n{target_name}"]'
-            )
+            print(f'  "{target}" [shape=box, label="{filename}\\n{target_name}"]')
         else:
             # Group multiple nodes together in a subgraph.
             print('  subgraph "cluster_%s" {' % filename)
diff --git a/gyp/tools/pretty_gyp.py b/gyp/tools/pretty_gyp.py
index a023887205..562a73ee67 100755
--- a/gyp/tools/pretty_gyp.py
+++ b/gyp/tools/pretty_gyp.py
@@ -6,7 +6,6 @@
 
 """Pretty-prints the contents of a GYP file."""
 
-
 import re
 import sys
 
@@ -49,7 +48,7 @@ def mask_quotes(input):
 def do_split(input, masked_input, search_re):
     output = []
     mask_output = []
-    for (line, masked_line) in zip(input, masked_input):
+    for line, masked_line in zip(input, masked_input):
         m = search_re.match(masked_line)
         while m:
             split = len(m.group(1))
@@ -63,13 +62,13 @@ def do_split(input, masked_input, search_re):
 
 def split_double_braces(input):
     """Masks out the quotes and comments, and then splits appropriate
-  lines (lines that matche the double_*_brace re's above) before
-  indenting them below.
+    lines (lines that matche the double_*_brace re's above) before
+    indenting them below.
 
-  These are used to split lines which have multiple braces on them, so
-  that the indentation looks prettier when all laid out (e.g. closing
-  braces make a nice diagonal line).
-  """
+    These are used to split lines which have multiple braces on them, so
+    that the indentation looks prettier when all laid out (e.g. closing
+    braces make a nice diagonal line).
+    """
     double_open_brace_re = re.compile(r"(.*?[\[\{\(,])(\s*)([\[\{\(])")
     double_close_brace_re = re.compile(r"(.*?[\]\}\)],?)(\s*)([\]\}\)])")
 
@@ -85,8 +84,8 @@ def split_double_braces(input):
 def count_braces(line):
     """keeps track of the number of braces on a given line and returns the result.
 
-  It starts at zero and subtracts for closed braces, and adds for open braces.
-  """
+    It starts at zero and subtracts for closed braces, and adds for open braces.
+    """
     open_braces = ["[", "(", "{"]
     close_braces = ["]", ")", "}"]
     closing_prefix_re = re.compile(r"[^\s\]\}\)]\s*[\]\}\)]+,?\s*$")
diff --git a/gyp/tools/pretty_sln.py b/gyp/tools/pretty_sln.py
index 850fb150f4..70c91aefad 100755
--- a/gyp/tools/pretty_sln.py
+++ b/gyp/tools/pretty_sln.py
@@ -6,13 +6,12 @@
 
 """Prints the information in a sln file in a diffable way.
 
-   It first outputs each projects in alphabetical order with their
-   dependencies.
+It first outputs each projects in alphabetical order with their
+dependencies.
 
-   Then it outputs a possible build order.
+Then it outputs a possible build order.
 """
 
-
 import os
 import re
 import sys
@@ -113,7 +112,7 @@ def PrintDependencies(projects, deps):
     print("---------------------------------------")
     print("--                                   --")
 
-    for (project, dep_list) in sorted(deps.items()):
+    for project, dep_list in sorted(deps.items()):
         print("Project : %s" % project)
         print("Path : %s" % projects[project][0])
         if dep_list:
@@ -131,7 +130,7 @@ def PrintBuildOrder(projects, deps):
     print("--                                   --")
 
     built = []
-    for (project, _) in sorted(deps.items()):
+    for project, _ in sorted(deps.items()):
         if project not in built:
             BuildProject(project, built, projects, deps)
 
@@ -139,7 +138,6 @@ def PrintBuildOrder(projects, deps):
 
 
 def PrintVCProj(projects):
-
     for project in projects:
         print("-------------------------------------")
         print("-------------------------------------")
diff --git a/gyp/tools/pretty_vcproj.py b/gyp/tools/pretty_vcproj.py
index c7427ed2d8..82d47a0bdd 100755
--- a/gyp/tools/pretty_vcproj.py
+++ b/gyp/tools/pretty_vcproj.py
@@ -6,13 +6,12 @@
 
 """Make the format of a vcproj really pretty.
 
-   This script normalize and sort an xml. It also fetches all the properties
-   inside linked vsprops and include them explicitly in the vcproj.
+This script normalize and sort an xml. It also fetches all the properties
+inside linked vsprops and include them explicitly in the vcproj.
 
-   It outputs the resulting xml to stdout.
+It outputs the resulting xml to stdout.
 """
 
-
 import os
 import sys
 from xml.dom.minidom import Node, parse
@@ -48,11 +47,11 @@ def get_string(node):
                 node_string += node.getAttribute("Name")
 
                 all_nodes = []
-                for (name, value) in node.attributes.items():
+                for name, value in node.attributes.items():
                     all_nodes.append((name, value))
 
                 all_nodes.sort(CmpTuple())
-                for (name, value) in all_nodes:
+                for name, value in all_nodes:
                     node_string += name
                     node_string += value
 
@@ -81,10 +80,10 @@ def PrettyPrintNode(node, indent=0):
         print("{}<{}".format(" " * indent, node.nodeName))
 
         all_attributes = []
-        for (name, value) in node.attributes.items():
+        for name, value in node.attributes.items():
             all_attributes.append((name, value))
             all_attributes.sort(CmpTuple())
-        for (name, value) in all_attributes:
+        for name, value in all_attributes:
             print('{}  {}="{}"'.format(" " * indent, name, value))
         print("%s>" % (" " * indent))
     if node.nodeValue:
@@ -130,7 +129,7 @@ def FixFilenames(filenames, current_directory):
 def AbsoluteNode(node):
     """Makes all the properties we know about in this node absolute."""
     if node.attributes:
-        for (name, value) in node.attributes.items():
+        for name, value in node.attributes.items():
             if name in [
                 "InheritedPropertySheets",
                 "RelativePath",
@@ -163,7 +162,7 @@ def CleanupVcproj(node):
     # Fix all the semicolon separated attributes to be sorted, and we also
     # remove the dups.
     if node.attributes:
-        for (name, value) in node.attributes.items():
+        for name, value in node.attributes.items():
             sorted_list = sorted(value.split(";"))
             unique_list = []
             for i in sorted_list:
@@ -252,7 +251,7 @@ def MergeAttributes(node1, node2):
     if not node2.attributes:
         return
 
-    for (name, value2) in node2.attributes.items():
+    for name, value2 in node2.attributes.items():
         # Don't merge the 'Name' attribute.
         if name == "Name":
             continue

From cb30a538eadf49ca0310980ffb0bfdb8fcebf0a4 Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Tue, 26 Aug 2025 15:28:06 +0200
Subject: [PATCH 487/551] chore: ruff format Python code (#3203)

---
 .github/workflows/tests.yml   |  1 +
 test/fixtures/test-charmap.py |  2 +-
 update-gyp.py                 | 18 ++++++++----------
 3 files changed, 10 insertions(+), 11 deletions(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index d8720d768a..16762765c2 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -21,6 +21,7 @@ jobs:
     - uses: astral-sh/ruff-action@v3
       with:
         args: "check --select=E,F,PLC,PLE,UP,W,YTT --ignore=E721,PLC0206,PLC0415,PLC1901,S101,UP031 --target-version=py39"
+    - run: ruff format --check --diff
 
   lint-js:
     name: Lint JS
diff --git a/test/fixtures/test-charmap.py b/test/fixtures/test-charmap.py
index 63aa77bb48..9f601af4ab 100644
--- a/test/fixtures/test-charmap.py
+++ b/test/fixtures/test-charmap.py
@@ -19,7 +19,7 @@ def main():
 
     textmap = {
         "cp936": "\u4e2d\u6587",
-        "cp1252": "Lat\u012Bna",
+        "cp1252": "Lat\u012bna",
         "cp932": "\u306b\u307b\u3093\u3054",
     }
     if encoding in textmap:
diff --git a/update-gyp.py b/update-gyp.py
index c65da41472..a822c0485f 100755
--- a/update-gyp.py
+++ b/update-gyp.py
@@ -13,10 +13,9 @@
 CHECKOUT_GYP_PATH = os.path.join(CHECKOUT_PATH, "gyp")
 
 parser = argparse.ArgumentParser()
-parser.add_argument("--no-commit",
-    action="store_true",
-    dest="no_commit",
-    help="do not run git-commit")
+parser.add_argument(
+    "--no-commit", action="store_true", dest="no_commit", help="do not run git-commit"
+)
 parser.add_argument("tag", help="gyp tag to update to")
 args = parser.parse_args()
 
@@ -37,8 +36,8 @@
 
         print("Unzipping...")
         with tarfile.open(tar_file, "r:gz") as tar_ref:
-            def is_within_directory(directory, target):
 
+            def is_within_directory(directory, target):
                 abs_directory = os.path.abspath(directory)
                 abs_target = os.path.abspath(target)
 
@@ -47,7 +46,6 @@ def is_within_directory(directory, target):
                 return prefix == abs_directory
 
             def safe_extract(tar, path=".", members=None, *, numeric_owner=False):
-
                 for member in tar.getmembers():
                     member_path = os.path.join(path, member.name)
                     if not is_within_directory(path, member_path):
@@ -65,7 +63,7 @@ def safe_extract(tar, path=".", members=None, *, numeric_owner=False):
         )
 
 if not args.no_commit:
-  subprocess.check_output(["git", "add", "gyp"], cwd=CHECKOUT_PATH)
-  subprocess.check_output([
-    "git", "commit", "-m", f"feat(gyp): update gyp to {args.tag}"
-  ])
+    subprocess.check_output(["git", "add", "gyp"], cwd=CHECKOUT_PATH)
+    subprocess.check_output(
+        ["git", "commit", "-m", f"feat(gyp): update gyp to {args.tag}"]
+    )

From a4e1da6683a37fde565e1ea50f1fa86fa99a83c7 Mon Sep 17 00:00:00 2001
From: Antoine du Hamel 
Date: Tue, 26 Aug 2025 15:34:39 +0200
Subject: [PATCH 488/551] chore(ci): Update Node.js version matrix in
 `tests.yml` (#3209)

* chore(ci): Update Node.js version to 24.x in `tests.yml`

* Remove EOL Node.js 18.x
---
 .github/workflows/tests.yml | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 16762765c2..c46a4842ee 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -95,17 +95,17 @@ jobs:
       matrix:
         os: [windows-latest, macos-latest, ubuntu-latest]
         python: ["3.9", "3.11", "3.13"]
-        node: [18.x, 20.x, 22.x, 23.x]
+        node: [20.x, 22.x, 24.x]
         include:
           - os: macos-13
             python: "3.13"
-            node: 23.x
+            node: 24.x
           - os: ubuntu-24.04-arm
             python: "3.13"
-            node: 23.x
+            node: 24.x
           - os: windows-2025
             python: "3.13"
-            node: 23.x
+            node: 24.x
     name: ${{ matrix.os }} - ${{ matrix.python }} - ${{ matrix.node }}
     runs-on: ${{ matrix.os }}
     env:

From 289fd574b2c9da120455a158b54ade326519e071 Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Tue, 26 Aug 2025 15:17:22 +0100
Subject: [PATCH 489/551] chore(main): release 11.4.2 (#3206)

---
 .release-please-manifest.json |  2 +-
 CHANGELOG.md                  | 13 +++++++++++++
 package.json                  |  2 +-
 3 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
index 8205383b6a..a94451c9e1 100644
--- a/.release-please-manifest.json
+++ b/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "11.4.1"
+    ".": "11.4.2"
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 054f649b57..952284d697 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,18 @@
 # Changelog
 
+## [11.4.2](https://github.com/nodejs/node-gyp/compare/v11.4.1...v11.4.2) (2025-08-26)
+
+
+### Bug Fixes
+
+* add adaptation for OpenHarmony platform ([#3207](https://github.com/nodejs/node-gyp/issues/3207)) ([b406532](https://github.com/nodejs/node-gyp/commit/b406532c77659c441c845708ec3ecdf09f013a3b))
+
+### Miscellaneous
+
+* update gyp-next to v0.20.4 ([#3208](https://github.com/nodejs/node-gyp/issues/3208)) ([adc61b1](https://github.com/nodejs/node-gyp/commit/adc61b1458315d9648591e74bf16bbe39511401e))
+* **ci:** Update Node.js version matrix in `tests.yml` ([#3209](https://github.com/nodejs/node-gyp/issues/3209)) ([a4e1da6](https://github.com/nodejs/node-gyp/commit/a4e1da6683a37fde565e1ea50f1fa86fa99a83c7))
+* ruff format Python code ([#3203](https://github.com/nodejs/node-gyp/issues/3203)) ([cb30a53](https://github.com/nodejs/node-gyp/commit/cb30a538eadf49ca0310980ffb0bfdb8fcebf0a4))
+
 ## [11.4.1](https://github.com/nodejs/node-gyp/compare/v11.4.0...v11.4.1) (2025-08-20)
 
 
diff --git a/package.json b/package.json
index f0fb3bfa3a..018391bd38 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "11.4.1",
+  "version": "11.4.2",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From 921c04d142549f172d3aeae4097c9e0af05599dd Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Thu, 4 Sep 2025 18:09:19 +0200
Subject: [PATCH 490/551] build(deps): bump actions/setup-node from 4 to 5
 (#3211)

Bumps [actions/setup-node](https://github.com/actions/setup-node) from 4 to 5.
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/setup-node
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 .github/workflows/release-please.yml | 2 +-
 .github/workflows/tests.yml          | 8 ++++----
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml
index 4779f74541..229e12d988 100644
--- a/.github/workflows/release-please.yml
+++ b/.github/workflows/release-please.yml
@@ -37,7 +37,7 @@ jobs:
       id-token: write # to generate npm provenance statements
     steps:
       - uses: actions/checkout@v5
-      - uses: actions/setup-node@v4
+      - uses: actions/setup-node@v5
         with:
           node-version: lts/*
           registry-url: 'https://registry.npmjs.org'
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index c46a4842ee..e6d444e30e 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -30,7 +30,7 @@ jobs:
     - name: Checkout Repository
       uses: actions/checkout@v5
     - name: Use Node.js 22.x
-      uses: actions/setup-node@v4
+      uses: actions/setup-node@v5
       with:
         node-version: 22.x
     - name: Install Dependencies
@@ -45,7 +45,7 @@ jobs:
     - name: Checkout Repository
       uses: actions/checkout@v5
     - name: Use Node.js 22.x
-      uses: actions/setup-node@v4
+      uses: actions/setup-node@v5
       with:
         node-version: 22.x
     - name: Install Dependencies
@@ -63,7 +63,7 @@ jobs:
     - name: Checkout Repository
       uses: actions/checkout@v5
     - name: Use Node.js 22.x
-      uses: actions/setup-node@v4
+      uses: actions/setup-node@v5
       with:
         node-version: 22.x
     - name: Update npm
@@ -116,7 +116,7 @@ jobs:
       - name: Checkout Repository
         uses: actions/checkout@v5
       - name: Use Node.js ${{ matrix.node }}
-        uses: actions/setup-node@v4
+        uses: actions/setup-node@v5
         with:
           node-version: ${{ matrix.node }}
       - name: Use Python ${{ matrix.python }}

From 6b70b05ed21cb977214348c97c2b97515c0d08f3 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Thu, 4 Sep 2025 18:48:59 +0200
Subject: [PATCH 491/551] build(deps): bump actions/setup-python from 5 to 6
 (#3210)

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5 to 6.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 .github/workflows/tests.yml         | 2 +-
 .github/workflows/visual-studio.yml | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index e6d444e30e..203234c06d 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -120,7 +120,7 @@ jobs:
         with:
           node-version: ${{ matrix.node }}
       - name: Use Python ${{ matrix.python }}
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@v6
         with:
           python-version: ${{ matrix.python }}
         env:
diff --git a/.github/workflows/visual-studio.yml b/.github/workflows/visual-studio.yml
index a284e28768..e4a90a3057 100644
--- a/.github/workflows/visual-studio.yml
+++ b/.github/workflows/visual-studio.yml
@@ -26,7 +26,7 @@ jobs:
       - name: Checkout Repository
         uses: actions/checkout@v5
       - name: Use Python 3
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@v6
         with:
           python-version: "3.x"
       - name: Install Dependencies

From c6b968caf7f4e22687fc10716162675b1411f713 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Mon, 8 Sep 2025 15:24:59 +0200
Subject: [PATCH 492/551] build(deps): bump actions/github-script from 7 to 8
 (#3213)

Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 8.
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](https://github.com/actions/github-script/compare/v7...v8)

---
updated-dependencies:
- dependency-name: actions/github-script
  dependency-version: '8'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 .github/workflows/update-gyp-next.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/update-gyp-next.yml b/.github/workflows/update-gyp-next.yml
index 94d36c3afd..ef8b4a807b 100644
--- a/.github/workflows/update-gyp-next.yml
+++ b/.github/workflows/update-gyp-next.yml
@@ -21,7 +21,7 @@ jobs:
         with:
           persist-credentials: false
 
-      - uses: actions/github-script@v7
+      - uses: actions/github-script@v8
         id: get-gyp-next-version
         with:
           script: |

From 8bd3f6354b8bd43262a4d99d58a568beab0459e8 Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Thu, 25 Sep 2025 15:36:12 +0200
Subject: [PATCH 493/551] fix(ci): Run Visual Studio test on Windows 11 on ARM
 (#3217)

---
 .github/workflows/visual-studio.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.github/workflows/visual-studio.yml b/.github/workflows/visual-studio.yml
index e4a90a3057..4608df08b8 100644
--- a/.github/workflows/visual-studio.yml
+++ b/.github/workflows/visual-studio.yml
@@ -21,6 +21,8 @@ jobs:
             msvs-version: 2022
           - os: windows-2025
             msvs-version: 2022  # Fix this when Visual Studio 2025 is released
+          - os: windows-11-arm
+            msvs-version: 2022  # Fix this when Visual Studio 2025 is released
     runs-on: ${{ matrix.os }}
     steps:
       - name: Checkout Repository

From 085b445d1c00f8f1fc6a6ff80d8a93c6643f11ee Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Fri, 26 Sep 2025 13:25:51 +0200
Subject: [PATCH 494/551] fix(ci): Test on Python 3.14 release candidate 3 on
 Linux and macOS (#3216)

---
 .github/workflows/tests.yml | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 203234c06d..0325652d69 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -93,18 +93,22 @@ jobs:
       fail-fast: false
       max-parallel: 11
       matrix:
-        os: [windows-latest, macos-latest, ubuntu-latest]
-        python: ["3.9", "3.11", "3.13"]
+        # Windows on Python 3.14 is blocked by nodejs/node#59983
+        os: [macos-latest, ubuntu-latest]
+        python: ["3.10", "3.12", "3.14"]
         node: [20.x, 22.x, 24.x]
         include:
-          - os: macos-13
-            python: "3.13"
+          - os: windows-latest
+            python: "3.13"  # Windows on Python 3.13 instead of 3.14
             node: 24.x
-          - os: ubuntu-24.04-arm
-            python: "3.13"
+          - os: macos-15-intel  # macOS on Intel
+            python: "3.14"
             node: 24.x
-          - os: windows-2025
-            python: "3.13"
+          - os: ubuntu-24.04-arm  # Ubuntu on ARM
+            python: "3.14"
+            node: 24.x
+          - os: windows-11-arm  # Windows on ARM
+            python: "3.13"  # Windows on Python 3.13 instead of 3.14
             node: 24.x
     name: ${{ matrix.os }} - ${{ matrix.python }} - ${{ matrix.node }}
     runs-on: ${{ matrix.os }}
@@ -123,6 +127,7 @@ jobs:
         uses: actions/setup-python@v6
         with:
           python-version: ${{ matrix.python }}
+          allow-prereleases: true
         env:
           PYTHON_VERSION: ${{ matrix.python }}  # Why do this?
       - uses: seanmiddleditch/gha-setup-ninja@v6
@@ -169,10 +174,10 @@ jobs:
         shell: bash
         run: npm test --python="${pythonLocation}/python"
         env:
-          FULL_TEST: ${{ (matrix.node == '22.x' && matrix.python == '3.13') && '1' || '0' }}
+          FULL_TEST: ${{ (matrix.node == '24.x' && matrix.python == '3.14') && '1' || '0' }}
       - name: Run Tests (Windows)
         if: runner.os == 'Windows'
         shell: bash # Building wasm on Windows requires using make generator, it only works in bash
         run: npm run test --python="${pythonLocation}\\python.exe"
         env:
-          FULL_TEST: ${{ (matrix.node == '22.x' && matrix.python == '3.13') && '1' || '0' }}
+          FULL_TEST: ${{ (matrix.node == '24.x' && matrix.python == '3.13') && '1' || '0' }}

From 848e950833b90f0b25f346710ee42e9be4797604 Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Wed, 15 Oct 2025 11:50:35 +0100
Subject: [PATCH 495/551] feat: update gyp-next to v0.20.5 (#3222)

---
 gyp/.github/workflows/node-gyp.yml            | 28 ++++++++++---------
 gyp/.github/workflows/nodejs.yml              | 18 ++++++++----
 gyp/.github/workflows/python_tests.yml        |  9 ++++--
 gyp/.release-please-manifest.json             |  2 +-
 gyp/CHANGELOG.md                              |  8 ++++++
 gyp/pylib/gyp/generator/android.py            |  2 +-
 .../gyp/generator/compile_commands_json.py    |  2 +-
 gyp/pylib/gyp/generator/gypd.py               |  2 +-
 gyp/pylib/gyp/generator/make.py               |  2 +-
 gyp/pylib/gyp/generator/msvs.py               |  6 ++--
 gyp/pylib/gyp/generator/xcode.py              |  8 +++---
 gyp/pylib/gyp/input.py                        |  2 +-
 gyp/pylib/gyp/xcode_emulation.py              |  8 ++++--
 gyp/pylib/gyp/xcode_ninja.py                  |  2 +-
 gyp/pylib/gyp/xcodeproj_file.py               | 10 +++----
 gyp/pyproject.toml                            |  2 +-
 gyp/tools/graphviz.py                         |  6 ++--
 17 files changed, 69 insertions(+), 48 deletions(-)

diff --git a/gyp/.github/workflows/node-gyp.yml b/gyp/.github/workflows/node-gyp.yml
index 392722b6d8..14976925c1 100644
--- a/gyp/.github/workflows/node-gyp.yml
+++ b/gyp/.github/workflows/node-gyp.yml
@@ -9,19 +9,21 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        node-version: ["22"]
         os: [macos-latest, ubuntu-latest, windows-latest]
-        python-version: ["3.9", "3.11", "3.13"]
+        python-version: ["3.10", "3.12", "3.14"]
+        exclude:
+          # Windows on Python 3.14 is blocked by nodejs/node#59983
+          - os: windows-latest
+            python-version: "3.14"
         include:
-          - node-version: "22"
-            os: macos-13  # macOS on Intel
-            python-version: "3.13"
-          - node-version: "22"
-            os: ubuntu-24.04-arm  # Ubuntu on ARM
-            python-version: "3.13"
-          - node-version: "22"
-            os: windows-11-arm  # Windows on ARM
+          - os: windows-latest  # Windows on Python 3.13 instead of 3.14
             python-version: "3.13"
+          - os: macos-15-intel  # macOS on Intel
+            python-version: "3.14"
+          - os: ubuntu-24.04-arm  # Ubuntu on ARM
+            python-version: "3.14"
+          - os: windows-11-arm  # Windows on ARM
+            python-version: "3.13"  # Windows on Python 3.13 instead of 3.14
     runs-on: ${{ matrix.os }}
     steps:
       - name: Clone gyp-next
@@ -33,10 +35,10 @@ jobs:
         with:
           repository: nodejs/node-gyp
           path: node-gyp
-      - uses: actions/setup-node@v4
+      - uses: actions/setup-node@v5
         with:
-          node-version: ${{ matrix.node-version }}
-      - uses: actions/setup-python@v5
+          node-version: "lts/*"
+      - uses: actions/setup-python@v6
         with:
           python-version: ${{ matrix.python-version }}
           allow-prereleases: true
diff --git a/gyp/.github/workflows/nodejs.yml b/gyp/.github/workflows/nodejs.yml
index be6abf515a..e73e5bd06b 100644
--- a/gyp/.github/workflows/nodejs.yml
+++ b/gyp/.github/workflows/nodejs.yml
@@ -10,14 +10,20 @@ jobs:
       fail-fast: false
       matrix:
         os: [macos-latest, ubuntu-latest, windows-latest]
-        python: ["3.9", "3.11", "3.13"]
+        python-version: ["3.10", "3.12", "3.14"]
+        exclude:
+          # Windows on Python 3.14 is blocked by nodejs/node#59983
+          - os: windows-latest
+            python-version: "3.14"
         include:
-          - os: macos-13  # macOS on Intel
+          - os: windows-latest  # Windows on Python 3.13 instead of 3.14
             python-version: "3.13"
+          - os: macos-15-intel  # macOS on Intel
+            python-version: "3.14"
           - os: ubuntu-24.04-arm  # Ubuntu on ARM
-            python-version: "3.13"
+            python-version: "3.14"
           - os: windows-11-arm  # Windows on ARM
-            python-version: "3.13"
+            python-version: "3.13"  # Windows on Python 3.13 instead of 3.14
 
     runs-on: ${{ matrix.os }}
     steps:
@@ -30,9 +36,9 @@ jobs:
         with:
           repository: nodejs/node
           path: node
-      - uses: actions/setup-python@v5
+      - uses: actions/setup-python@v6
         with:
-          python-version: ${{ matrix.python }}
+          python-version: ${{ matrix.python-version }}
           allow-prereleases: true
       - name: Replace gyp in Node.js
         shell: bash
diff --git a/gyp/.github/workflows/python_tests.yml b/gyp/.github/workflows/python_tests.yml
index 7a64bb4c7f..72dfd58536 100644
--- a/gyp/.github/workflows/python_tests.yml
+++ b/gyp/.github/workflows/python_tests.yml
@@ -14,12 +14,15 @@ jobs:
       fail-fast: false
       max-parallel: 5
       matrix:
-        os: [macos-13, macos-latest, ubuntu-latest] # , windows-latest]
-        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"]
+        os: [macos-15-intel, macos-latest, ubuntu-latest] # , windows-latest]
+        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
+        include:
+          - os: macos-26
+            python-version: 3.x
     steps:
       - uses: actions/checkout@v5
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@v6
         with:
           python-version: ${{ matrix.python-version }}
           allow-prereleases: true
diff --git a/gyp/.release-please-manifest.json b/gyp/.release-please-manifest.json
index bdb726346f..dfc532112e 100644
--- a/gyp/.release-please-manifest.json
+++ b/gyp/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "0.20.4"
+    ".": "0.20.5"
 }
diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md
index 69449c0d80..74dbc82832 100644
--- a/gyp/CHANGELOG.md
+++ b/gyp/CHANGELOG.md
@@ -1,5 +1,13 @@
 # Changelog
 
+## [0.20.5](https://github.com/nodejs/gyp-next/compare/v0.20.4...v0.20.5) (2025-10-13)
+
+
+### Bug Fixes
+
+* Fix ruff v0.13.0 adds ruff rule RUF059 ([bd4491a](https://github.com/nodejs/gyp-next/commit/bd4491a3ba641eeb040b785bbce367f72c3baf19))
+* handle `None` case in xcode_emulation regexes ([#311](https://github.com/nodejs/gyp-next/issues/311)) ([b21ee31](https://github.com/nodejs/gyp-next/commit/b21ee3150eea9fc1a8811e910e5ba64f42e1fb77))
+
 ## [0.20.4](https://github.com/nodejs/gyp-next/compare/v0.20.3...v0.20.4) (2025-08-25)
 
 
diff --git a/gyp/pylib/gyp/generator/android.py b/gyp/pylib/gyp/generator/android.py
index cfc0681f6b..5d5cae2afb 100644
--- a/gyp/pylib/gyp/generator/android.py
+++ b/gyp/pylib/gyp/generator/android.py
@@ -378,7 +378,7 @@ def WriteRules(self, rules, extra_sources, extra_outputs):
             inputs = rule.get("inputs")
             for rule_source in rule.get("rule_sources", []):
                 (rule_source_dirname, rule_source_basename) = os.path.split(rule_source)
-                (rule_source_root, rule_source_ext) = os.path.splitext(
+                (rule_source_root, _rule_source_ext) = os.path.splitext(
                     rule_source_basename
                 )
 
diff --git a/gyp/pylib/gyp/generator/compile_commands_json.py b/gyp/pylib/gyp/generator/compile_commands_json.py
index bebb130315..1361aeca48 100644
--- a/gyp/pylib/gyp/generator/compile_commands_json.py
+++ b/gyp/pylib/gyp/generator/compile_commands_json.py
@@ -100,7 +100,7 @@ def resolve(filename):
 def GenerateOutput(target_list, target_dicts, data, params):
     per_config_commands = {}
     for qualified_target, target in target_dicts.items():
-        build_file, target_name, toolset = gyp.common.ParseQualifiedTarget(
+        build_file, _target_name, _toolset = gyp.common.ParseQualifiedTarget(
             qualified_target
         )
         if IsMac(params):
diff --git a/gyp/pylib/gyp/generator/gypd.py b/gyp/pylib/gyp/generator/gypd.py
index 3c70b81fd2..89af24a201 100644
--- a/gyp/pylib/gyp/generator/gypd.py
+++ b/gyp/pylib/gyp/generator/gypd.py
@@ -73,7 +73,7 @@
 def GenerateOutput(target_list, target_dicts, data, params):
     output_files = {}
     for qualified_target in target_list:
-        [input_file, target] = gyp.common.ParseQualifiedTarget(qualified_target)[0:2]
+        [input_file, _target] = gyp.common.ParseQualifiedTarget(qualified_target)[0:2]
 
         if input_file[-4:] != ".gyp":
             continue
diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py
index 1f0995718b..5f30f39fc5 100644
--- a/gyp/pylib/gyp/generator/make.py
+++ b/gyp/pylib/gyp/generator/make.py
@@ -1169,7 +1169,7 @@ def WriteRules(
             for rule_source in rule.get("rule_sources", []):
                 dirs = set()
                 (rule_source_dirname, rule_source_basename) = os.path.split(rule_source)
-                (rule_source_root, rule_source_ext) = os.path.splitext(
+                (rule_source_root, _rule_source_ext) = os.path.splitext(
                     rule_source_basename
                 )
 
diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py
index 3b258ee8f3..0f14c05504 100644
--- a/gyp/pylib/gyp/generator/msvs.py
+++ b/gyp/pylib/gyp/generator/msvs.py
@@ -1666,7 +1666,7 @@ def _HandlePreCompiledHeaders(p, sources, spec):
             p.AddFileConfig(
                 source, _ConfigFullName(config_name, config), {}, tools=[tool]
             )
-            basename, extension = os.path.splitext(source)
+            _basename, extension = os.path.splitext(source)
             if extension == ".c":
                 extensions_excluded_from_precompile = [".cc", ".cpp", ".cxx"]
             else:
@@ -1677,7 +1677,7 @@ def DisableForSourceTree(source_tree):
             if isinstance(source, MSVSProject.Filter):
                 DisableForSourceTree(source.contents)
             else:
-                basename, extension = os.path.splitext(source)
+                _basename, extension = os.path.splitext(source)
                 if extension in extensions_excluded_from_precompile:
                     for config_name, config in spec["configurations"].items():
                         tool = MSVSProject.Tool(
@@ -3579,7 +3579,7 @@ def _AddSources2(
                         # If the precompiled header is generated by a C source,
                         # we must not try to use it for C++ sources,
                         # and vice versa.
-                        basename, extension = os.path.splitext(precompiled_source)
+                        _basename, extension = os.path.splitext(precompiled_source)
                         if extension == ".c":
                             extensions_excluded_from_precompile = [
                                 ".cc",
diff --git a/gyp/pylib/gyp/generator/xcode.py b/gyp/pylib/gyp/generator/xcode.py
index 8e05657961..db4b45d1a0 100644
--- a/gyp/pylib/gyp/generator/xcode.py
+++ b/gyp/pylib/gyp/generator/xcode.py
@@ -531,7 +531,7 @@ def AddSourceToTarget(source, type, pbxp, xct):
     library_extensions = ["a", "dylib", "framework", "o"]
 
     basename = posixpath.basename(source)
-    (root, ext) = posixpath.splitext(basename)
+    (_root, ext) = posixpath.splitext(basename)
     if ext:
         ext = ext[1:].lower()
 
@@ -696,7 +696,7 @@ def GenerateOutput(target_list, target_dicts, data, params):
     xcode_targets = {}
     xcode_target_to_target_dict = {}
     for qualified_target in target_list:
-        [build_file, target_name, toolset] = gyp.common.ParseQualifiedTarget(
+        [build_file, target_name, _toolset] = gyp.common.ParseQualifiedTarget(
             qualified_target
         )
 
@@ -1215,7 +1215,7 @@ def GenerateOutput(target_list, target_dicts, data, params):
 
         # Add "sources".
         for source in spec.get("sources", []):
-            (source_root, source_extension) = posixpath.splitext(source)
+            (_source_root, source_extension) = posixpath.splitext(source)
             if source_extension[1:] not in rules_by_ext:
                 # AddSourceToTarget will add the file to a root group if it's not
                 # already there.
@@ -1227,7 +1227,7 @@ def GenerateOutput(target_list, target_dicts, data, params):
         # it's a bundle of any type.
         if is_bundle:
             for resource in tgt_mac_bundle_resources:
-                (resource_root, resource_extension) = posixpath.splitext(resource)
+                (_resource_root, resource_extension) = posixpath.splitext(resource)
                 if resource_extension[1:] not in rules_by_ext:
                     AddResourceToTarget(resource, pbxp, xct)
                 else:
diff --git a/gyp/pylib/gyp/input.py b/gyp/pylib/gyp/input.py
index 4965ff1571..f3a5e168f2 100644
--- a/gyp/pylib/gyp/input.py
+++ b/gyp/pylib/gyp/input.py
@@ -2757,7 +2757,7 @@ def ValidateRulesInTarget(target, target_dict, extra_sources_for_rules):
         source_keys.extend(extra_sources_for_rules)
         for source_key in source_keys:
             for source in target_dict.get(source_key, []):
-                (source_root, source_extension) = os.path.splitext(source)
+                (_source_root, source_extension) = os.path.splitext(source)
                 if source_extension.startswith("."):
                     source_extension = source_extension[1:]
                 if source_extension == rule_extension:
diff --git a/gyp/pylib/gyp/xcode_emulation.py b/gyp/pylib/gyp/xcode_emulation.py
index 192a523529..d13eaa9af2 100644
--- a/gyp/pylib/gyp/xcode_emulation.py
+++ b/gyp/pylib/gyp/xcode_emulation.py
@@ -1534,18 +1534,20 @@ def CLTVersion():
     FROM_XCODE_PKG_ID = "com.apple.pkg.DeveloperToolsCLI"
     MAVERICKS_PKG_ID = "com.apple.pkg.CLTools_Executables"
 
-    regex = re.compile("version: (?P.+)")
+    regex = re.compile(r"version: (?P.+)")
     for key in [MAVERICKS_PKG_ID, STANDALONE_PKG_ID, FROM_XCODE_PKG_ID]:
         try:
             output = GetStdout(["/usr/sbin/pkgutil", "--pkg-info", key])
-            return re.search(regex, output).groupdict()["version"]
+            if m := re.search(regex, output):
+                return m.groupdict()["version"]
         except (GypError, OSError):
             continue
 
     regex = re.compile(r"Command Line Tools for Xcode\s+(?P\S+)")
     try:
         output = GetStdout(["/usr/sbin/softwareupdate", "--history"])
-        return re.search(regex, output).groupdict()["version"]
+        if m := re.search(regex, output):
+            return m.groupdict()["version"]
     except (GypError, OSError):
         return None
 
diff --git a/gyp/pylib/gyp/xcode_ninja.py b/gyp/pylib/gyp/xcode_ninja.py
index 1a97a06c51..a133fdbe8b 100644
--- a/gyp/pylib/gyp/xcode_ninja.py
+++ b/gyp/pylib/gyp/xcode_ninja.py
@@ -22,7 +22,7 @@
 
 def _WriteWorkspace(main_gyp, sources_gyp, params):
     """Create a workspace to wrap main and sources gyp paths."""
-    (build_file_root, build_file_ext) = os.path.splitext(main_gyp)
+    (build_file_root, _build_file_ext) = os.path.splitext(main_gyp)
     workspace_path = build_file_root + ".xcworkspace"
     options = params["options"]
     if options.generator_output:
diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py
index 11e2be0737..cb467470d3 100644
--- a/gyp/pylib/gyp/xcodeproj_file.py
+++ b/gyp/pylib/gyp/xcodeproj_file.py
@@ -487,7 +487,7 @@ def Children(self):
 
         children = []
         for property, attributes in self._schema.items():
-            (is_list, property_type, is_strong) = attributes[0:3]
+            (is_list, _property_type, is_strong) = attributes[0:3]
             if is_strong and property in self._properties:
                 if not is_list:
                     children.append(self._properties[property])
@@ -913,7 +913,7 @@ def VerifyHasRequiredProperties(self):
         # TODO(mark): A stronger verification mechanism is needed.  Some
         # subclasses need to perform validation beyond what the schema can enforce.
         for property, attributes in self._schema.items():
-            (is_list, property_type, is_strong, is_required) = attributes[0:4]
+            (_is_list, _property_type, _is_strong, is_required) = attributes[0:4]
             if is_required and property not in self._properties:
                 raise KeyError(self.__class__.__name__ + " requires " + property)
 
@@ -923,7 +923,7 @@ def _SetDefaultsFromSchema(self):
 
         defaults = {}
         for property, attributes in self._schema.items():
-            (is_list, property_type, is_strong, is_required) = attributes[0:4]
+            (_is_list, _property_type, _is_strong, is_required) = attributes[0:4]
             if (
                 is_required
                 and len(attributes) >= 5
@@ -1616,7 +1616,7 @@ def __init__(self, properties=None, id=None, parent=None):
                 prop_name = "lastKnownFileType"
             else:
                 basename = posixpath.basename(self._properties["path"])
-                (root, ext) = posixpath.splitext(basename)
+                (_root, ext) = posixpath.splitext(basename)
                 # Check the map using a lowercase extension.
                 # TODO(mark): Maybe it should try with the original case first and fall
                 # back to lowercase, in case there are any instances where case
@@ -2010,7 +2010,7 @@ def Name(self):
         return "Frameworks"
 
     def FileGroup(self, path):
-        (root, ext) = posixpath.splitext(path)
+        (_root, ext) = posixpath.splitext(path)
         if ext != "":
             ext = ext[1:].lower()
         if ext == "o":
diff --git a/gyp/pyproject.toml b/gyp/pyproject.toml
index 3a029c4fc5..adc82c3350 100644
--- a/gyp/pyproject.toml
+++ b/gyp/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "gyp-next"
-version = "0.20.4"
+version = "0.20.5"
 authors = [
   { name="Node.js contributors", email="ryzokuken@disroot.org" },
 ]
diff --git a/gyp/tools/graphviz.py b/gyp/tools/graphviz.py
index ed1c7ab3cd..65f78011b5 100755
--- a/gyp/tools/graphviz.py
+++ b/gyp/tools/graphviz.py
@@ -47,7 +47,7 @@ def WriteGraph(edges):
     # Bucket targets by file.
     files = collections.defaultdict(list)
     for src, dst in edges.items():
-        build_file, target_name, toolset = ParseTarget(src)
+        build_file, target_name, _toolset = ParseTarget(src)
         files[build_file].append(src)
 
     print("digraph D {")
@@ -62,14 +62,14 @@ def WriteGraph(edges):
             # If there's only one node for this file, simplify
             # the display by making it a box without an internal node.
             target = targets[0]
-            build_file, target_name, toolset = ParseTarget(target)
+            build_file, target_name, _toolset = ParseTarget(target)
             print(f'  "{target}" [shape=box, label="{filename}\\n{target_name}"]')
         else:
             # Group multiple nodes together in a subgraph.
             print('  subgraph "cluster_%s" {' % filename)
             print('    label = "%s"' % filename)
             for target in targets:
-                build_file, target_name, toolset = ParseTarget(target)
+                build_file, target_name, _toolset = ParseTarget(target)
                 print(f'    "{target}" [label="{target_name}"]')
             print("  }")
 

From 3cff85955e6ec8973fc6703ed937313b30bc0093 Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Wed, 15 Oct 2025 13:32:29 +0100
Subject: [PATCH 496/551] chore(main): release 11.5.0 (#3212)

---
 .release-please-manifest.json |  2 +-
 CHANGELOG.md                  | 20 ++++++++++++++++++++
 package.json                  |  2 +-
 3 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
index a94451c9e1..02eef11e2b 100644
--- a/.release-please-manifest.json
+++ b/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "11.4.2"
+    ".": "11.5.0"
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 952284d697..7e2740db65 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,25 @@
 # Changelog
 
+## [11.5.0](https://github.com/nodejs/node-gyp/compare/v11.4.2...v11.5.0) (2025-10-15)
+
+
+### Features
+
+* update gyp-next to v0.20.5 ([#3222](https://github.com/nodejs/node-gyp/issues/3222)) ([848e950](https://github.com/nodejs/node-gyp/commit/848e950833b90f0b25f346710ee42e9be4797604))
+
+
+### Bug Fixes
+
+* **ci:** Run Visual Studio test on Windows 11 on ARM ([#3217](https://github.com/nodejs/node-gyp/issues/3217)) ([8bd3f63](https://github.com/nodejs/node-gyp/commit/8bd3f6354b8bd43262a4d99d58a568beab0459e8))
+* **ci:** Test on Python 3.14 release candidate 3 on Linux and macOS ([#3216](https://github.com/nodejs/node-gyp/issues/3216)) ([085b445](https://github.com/nodejs/node-gyp/commit/085b445d1c00f8f1fc6a6ff80d8a93c6643f11ee))
+
+
+### Core
+
+* **deps:** bump actions/github-script from 7 to 8 ([#3213](https://github.com/nodejs/node-gyp/issues/3213)) ([c6b968c](https://github.com/nodejs/node-gyp/commit/c6b968caf7f4e22687fc10716162675b1411f713))
+* **deps:** bump actions/setup-node from 4 to 5 ([#3211](https://github.com/nodejs/node-gyp/issues/3211)) ([921c04d](https://github.com/nodejs/node-gyp/commit/921c04d142549f172d3aeae4097c9e0af05599dd))
+* **deps:** bump actions/setup-python from 5 to 6 ([#3210](https://github.com/nodejs/node-gyp/issues/3210)) ([6b70b05](https://github.com/nodejs/node-gyp/commit/6b70b05ed21cb977214348c97c2b97515c0d08f3))
+
 ## [11.4.2](https://github.com/nodejs/node-gyp/compare/v11.4.1...v11.4.2) (2025-08-26)
 
 
diff --git a/package.json b/package.json
index 018391bd38..3c9fd0ff31 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "11.4.2",
+  "version": "11.5.0",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From e2b9d21bce27c35d18fcb6f8583e386d15ce395c Mon Sep 17 00:00:00 2001
From: Michael Smith 
Date: Tue, 19 Aug 2025 14:44:47 -0700
Subject: [PATCH 497/551] deps: make-fetch-happen@15.0.0

---
 package.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package.json b/package.json
index 3c9fd0ff31..24d0ebde15 100644
--- a/package.json
+++ b/package.json
@@ -25,7 +25,7 @@
     "env-paths": "^2.2.0",
     "exponential-backoff": "^3.1.1",
     "graceful-fs": "^4.2.6",
-    "make-fetch-happen": "^14.0.3",
+    "make-fetch-happen": "^15.0.0",
     "nopt": "^8.0.0",
     "proc-log": "^5.0.0",
     "semver": "^7.3.5",

From 2f85686bbe745673350a8f9dbb0e86ee0190f213 Mon Sep 17 00:00:00 2001
From: Michael Smith 
Date: Tue, 19 Aug 2025 14:48:43 -0700
Subject: [PATCH 498/551] feat!: align to npm 11 node engine range `node-gyp`
 is now compatible with Node `^20.17.0 || >=22.9.0`

---
 package.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package.json b/package.json
index 24d0ebde15..eecaf4db98 100644
--- a/package.json
+++ b/package.json
@@ -34,7 +34,7 @@
     "which": "^5.0.0"
   },
   "engines": {
-    "node": "^18.17.0 || >=20.5.0"
+    "node": "^20.17.0 || >=22.9.0"
   },
   "devDependencies": {
     "bindings": "^1.5.0",

From ae90e632d9fab85f4cd902dc9205ba9dfafaf3bc Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Tue, 14 Oct 2025 14:26:47 +0000
Subject: [PATCH 499/551] build(deps): bump actions/setup-node from 5 to 6

Bumps [actions/setup-node](https://github.com/actions/setup-node) from 5 to 6.
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/setup-node
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] 
---
 .github/workflows/release-please.yml | 2 +-
 .github/workflows/tests.yml          | 8 ++++----
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml
index 229e12d988..564829cfb8 100644
--- a/.github/workflows/release-please.yml
+++ b/.github/workflows/release-please.yml
@@ -37,7 +37,7 @@ jobs:
       id-token: write # to generate npm provenance statements
     steps:
       - uses: actions/checkout@v5
-      - uses: actions/setup-node@v5
+      - uses: actions/setup-node@v6
         with:
           node-version: lts/*
           registry-url: 'https://registry.npmjs.org'
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 0325652d69..810984e9f8 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -30,7 +30,7 @@ jobs:
     - name: Checkout Repository
       uses: actions/checkout@v5
     - name: Use Node.js 22.x
-      uses: actions/setup-node@v5
+      uses: actions/setup-node@v6
       with:
         node-version: 22.x
     - name: Install Dependencies
@@ -45,7 +45,7 @@ jobs:
     - name: Checkout Repository
       uses: actions/checkout@v5
     - name: Use Node.js 22.x
-      uses: actions/setup-node@v5
+      uses: actions/setup-node@v6
       with:
         node-version: 22.x
     - name: Install Dependencies
@@ -63,7 +63,7 @@ jobs:
     - name: Checkout Repository
       uses: actions/checkout@v5
     - name: Use Node.js 22.x
-      uses: actions/setup-node@v5
+      uses: actions/setup-node@v6
       with:
         node-version: 22.x
     - name: Update npm
@@ -120,7 +120,7 @@ jobs:
       - name: Checkout Repository
         uses: actions/checkout@v5
       - name: Use Node.js ${{ matrix.node }}
-        uses: actions/setup-node@v5
+        uses: actions/setup-node@v6
         with:
           node-version: ${{ matrix.node }}
       - name: Use Python ${{ matrix.python }}

From 4b47bc0ce7ae21976dc9121d4ff34e9750c9a840 Mon Sep 17 00:00:00 2001
From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com>
Date: Wed, 5 Nov 2025 06:26:06 +0000
Subject: [PATCH 500/551] Update tar dependency to ^7.5.2 to address
 CVE-2025-64118

Co-authored-by: cclauss <3709715+cclauss@users.noreply.github.com>
---
 package.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package.json b/package.json
index eecaf4db98..97be30f953 100644
--- a/package.json
+++ b/package.json
@@ -29,7 +29,7 @@
     "nopt": "^8.0.0",
     "proc-log": "^5.0.0",
     "semver": "^7.3.5",
-    "tar": "^7.4.3",
+    "tar": "^7.5.2",
     "tinyglobby": "^0.2.12",
     "which": "^5.0.0"
   },

From 41b0cea2f12342a790580cc8f844f075d49e096c Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Fri, 7 Nov 2025 10:37:40 -0700
Subject: [PATCH 501/551] chore: update devDependencies

---
 package.json | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/package.json b/package.json
index 97be30f953..ef11c0fdfa 100644
--- a/package.json
+++ b/package.json
@@ -38,11 +38,11 @@
   },
   "devDependencies": {
     "bindings": "^1.5.0",
-    "cross-env": "^7.0.3",
-    "eslint": "^9.16.0",
-    "mocha": "^11.0.1",
-    "nan": "^2.14.2",
-    "neostandard": "^0.11.9",
+    "cross-env": "^10.1.0",
+    "eslint": "^9.39.1",
+    "mocha": "^11.7.5",
+    "nan": "^2.23.1",
+    "neostandard": "^0.12.2",
     "require-inject": "^1.4.4"
   },
   "scripts": {

From dfc68dfba3c17deb0bda9a395bb49d8fb9fa5951 Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Fri, 7 Nov 2025 09:52:38 -0700
Subject: [PATCH 502/551] deps: proc-log@6.0.0

---
 package.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package.json b/package.json
index ef11c0fdfa..5973777351 100644
--- a/package.json
+++ b/package.json
@@ -27,7 +27,7 @@
     "graceful-fs": "^4.2.6",
     "make-fetch-happen": "^15.0.0",
     "nopt": "^8.0.0",
-    "proc-log": "^5.0.0",
+    "proc-log": "^6.0.0",
     "semver": "^7.3.5",
     "tar": "^7.5.2",
     "tinyglobby": "^0.2.12",

From 9bdeaf307cd7a254946859d306465989fa39dfb2 Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Fri, 7 Nov 2025 09:53:56 -0700
Subject: [PATCH 503/551] deps: nopt@9.0.0

---
 package.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package.json b/package.json
index 5973777351..04c3939138 100644
--- a/package.json
+++ b/package.json
@@ -26,7 +26,7 @@
     "exponential-backoff": "^3.1.1",
     "graceful-fs": "^4.2.6",
     "make-fetch-happen": "^15.0.0",
-    "nopt": "^8.0.0",
+    "nopt": "^9.0.0",
     "proc-log": "^6.0.0",
     "semver": "^7.3.5",
     "tar": "^7.5.2",

From 86d65c7874eb41eb49c9b8bbf342becac8e57c6f Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Fri, 7 Nov 2025 10:40:40 -0700
Subject: [PATCH 504/551] chore: setup dependabot for npm

---
 .github/dependabot.yml | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/.github/dependabot.yml b/.github/dependabot.yml
index 15e494ec86..6f40ed1c4a 100644
--- a/.github/dependabot.yml
+++ b/.github/dependabot.yml
@@ -6,3 +6,7 @@ updates:
     directory: "/"
     schedule:
       interval: "daily"
+  - package-ecosystem: "npm"
+    directory: "/"
+    schedule:
+      interval: "daily"

From c57cd2e86dc57707475b9f7e676e189f064817de Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Fri, 7 Nov 2025 22:44:18 +0000
Subject: [PATCH 505/551] feat: update gyp-next to v0.21.0

---
 gyp/.github/workflows/node-gyp.yml       |  2 +-
 gyp/.github/workflows/release-please.yml |  8 +++----
 gyp/.release-please-manifest.json        |  2 +-
 gyp/CHANGELOG.md                         |  7 ++++++
 gyp/pylib/gyp/MSVSVersion.py             | 27 +++++++++++++++++++++++-
 gyp/pyproject.toml                       |  2 +-
 6 files changed, 40 insertions(+), 8 deletions(-)

diff --git a/gyp/.github/workflows/node-gyp.yml b/gyp/.github/workflows/node-gyp.yml
index 14976925c1..2e592a6d38 100644
--- a/gyp/.github/workflows/node-gyp.yml
+++ b/gyp/.github/workflows/node-gyp.yml
@@ -35,7 +35,7 @@ jobs:
         with:
           repository: nodejs/node-gyp
           path: node-gyp
-      - uses: actions/setup-node@v5
+      - uses: actions/setup-node@v6
         with:
           node-version: "lts/*"
       - uses: actions/setup-python@v6
diff --git a/gyp/.github/workflows/release-please.yml b/gyp/.github/workflows/release-please.yml
index 2db84faf8b..6a18003c79 100644
--- a/gyp/.github/workflows/release-please.yml
+++ b/gyp/.github/workflows/release-please.yml
@@ -28,7 +28,7 @@ jobs:
     - name: Build a binary wheel and a source tarball
       run: pipx run build
     - name: Store the distribution packages
-      uses: actions/upload-artifact@v4
+      uses: actions/upload-artifact@v5
       with:
         name: python-package-distributions
         path: dist/
@@ -48,7 +48,7 @@ jobs:
       id-token: write # IMPORTANT: mandatory for trusted publishing
     steps:
     - name: Download all the dists
-      uses: actions/download-artifact@v5
+      uses: actions/download-artifact@v6
       with:
         name: python-package-distributions
         path: dist/
@@ -68,12 +68,12 @@ jobs:
       id-token: write # IMPORTANT: mandatory for sigstore
     steps:
     - name: Download all the dists
-      uses: actions/download-artifact@v5
+      uses: actions/download-artifact@v6
       with:
         name: python-package-distributions
         path: dist/
     - name: Sign the dists with Sigstore
-      uses: sigstore/gh-action-sigstore-python@v3.0.1
+      uses: sigstore/gh-action-sigstore-python@v3.1.0
       with:
         inputs: >-
           ./dist/*.tar.gz
diff --git a/gyp/.release-please-manifest.json b/gyp/.release-please-manifest.json
index dfc532112e..ca64307ab8 100644
--- a/gyp/.release-please-manifest.json
+++ b/gyp/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "0.20.5"
+    ".": "0.21.0"
 }
diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md
index 74dbc82832..31f4d25874 100644
--- a/gyp/CHANGELOG.md
+++ b/gyp/CHANGELOG.md
@@ -1,5 +1,12 @@
 # Changelog
 
+## [0.21.0](https://github.com/nodejs/gyp-next/compare/v0.20.5...v0.21.0) (2025-11-04)
+
+
+### Features
+
+* add support for Visual Studio 2026 ([#319](https://github.com/nodejs/gyp-next/issues/319)) ([cb13bbc](https://github.com/nodejs/gyp-next/commit/cb13bbc4ab7a7d86b51221b39ebee4281e1d3e19))
+
 ## [0.20.5](https://github.com/nodejs/gyp-next/compare/v0.20.4...v0.20.5) (2025-10-13)
 
 
diff --git a/gyp/pylib/gyp/MSVSVersion.py b/gyp/pylib/gyp/MSVSVersion.py
index 09baf44b2b..2d8e4ceab9 100644
--- a/gyp/pylib/gyp/MSVSVersion.py
+++ b/gyp/pylib/gyp/MSVSVersion.py
@@ -270,6 +270,18 @@ def _CreateVersion(name, path, sdk_based=False):
     if path:
         path = os.path.normpath(path)
     versions = {
+        "2026": VisualStudioVersion(
+            "2026",
+            "Visual Studio 2026",
+            solution_version="12.00",
+            project_version="18.0",
+            flat_sln=False,
+            uses_vcxproj=True,
+            path=path,
+            sdk_based=sdk_based,
+            default_toolset="v145",
+            compatible_sdks=["v8.1", "v10.0"],
+        ),
         "2022": VisualStudioVersion(
             "2022",
             "Visual Studio 2022",
@@ -462,6 +474,7 @@ def _DetectVisualStudioVersions(versions_to_check, force_express):
         "15.0": "2017",
         "16.0": "2019",
         "17.0": "2022",
+        "18.0": "2026",
     }
     versions = []
     for version in versions_to_check:
@@ -537,7 +550,18 @@ def SelectVisualStudioVersion(version="auto", allow_fallback=True):
     if version == "auto":
         version = os.environ.get("GYP_MSVS_VERSION", "auto")
     version_map = {
-        "auto": ("17.0", "16.0", "15.0", "14.0", "12.0", "10.0", "9.0", "8.0", "11.0"),
+        "auto": (
+            "18.0",
+            "17.0",
+            "16.0",
+            "15.0",
+            "14.0",
+            "12.0",
+            "10.0",
+            "9.0",
+            "8.0",
+            "11.0",
+        ),
         "2005": ("8.0",),
         "2005e": ("8.0",),
         "2008": ("9.0",),
@@ -552,6 +576,7 @@ def SelectVisualStudioVersion(version="auto", allow_fallback=True):
         "2017": ("15.0",),
         "2019": ("16.0",),
         "2022": ("17.0",),
+        "2026": ("18.0",),
     }
     if override_path := os.environ.get("GYP_MSVS_OVERRIDE_PATH"):
         msvs_version = os.environ.get("GYP_MSVS_VERSION")
diff --git a/gyp/pyproject.toml b/gyp/pyproject.toml
index adc82c3350..cd4f0383fd 100644
--- a/gyp/pyproject.toml
+++ b/gyp/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "gyp-next"
-version = "0.20.5"
+version = "0.21.0"
 authors = [
   { name="Node.js contributors", email="ryzokuken@disroot.org" },
 ]

From 5fffb2ffee304cc898fdea7a0cd9e41d54c53839 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Fri, 7 Nov 2025 16:33:38 -0700
Subject: [PATCH 506/551] build(deps): bump env-paths from 2.2.1 to 3.0.0
 (#3235)

* build(deps): bump env-paths from 2.2.1 to 3.0.0

Bumps [env-paths](https://github.com/sindresorhus/env-paths) from 2.2.1 to 3.0.0.
- [Release notes](https://github.com/sindresorhus/env-paths/releases)
- [Commits](https://github.com/sindresorhus/env-paths/compare/v2.2.1...v3.0.0)

---
updated-dependencies:
- dependency-name: env-paths
  dependency-version: 3.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] 

* fixup

---------

Signed-off-by: dependabot[bot] 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Luke Karrys 
---
 bin/node-gyp.js | 2 +-
 package.json    | 2 +-
 test/common.js  | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/bin/node-gyp.js b/bin/node-gyp.js
index f8317b47b3..c08f437a05 100755
--- a/bin/node-gyp.js
+++ b/bin/node-gyp.js
@@ -4,7 +4,7 @@
 
 process.title = 'node-gyp'
 
-const envPaths = require('env-paths')
+const { default: envPaths } = require('env-paths')
 const gyp = require('../')
 const log = require('../lib/log')
 const os = require('os')
diff --git a/package.json b/package.json
index 04c3939138..5ca4342ccf 100644
--- a/package.json
+++ b/package.json
@@ -22,7 +22,7 @@
   "bin": "./bin/node-gyp.js",
   "main": "./lib/node-gyp.js",
   "dependencies": {
-    "env-paths": "^2.2.0",
+    "env-paths": "^3.0.0",
     "exponential-backoff": "^3.1.1",
     "graceful-fs": "^4.2.6",
     "make-fetch-happen": "^15.0.0",
diff --git a/test/common.js b/test/common.js
index 489502d4dd..66d8d0d162 100644
--- a/test/common.js
+++ b/test/common.js
@@ -1,4 +1,4 @@
-const envPaths = require('env-paths')
+const { default: envPaths } = require('env-paths')
 const semver = require('semver')
 
 module.exports.devDir = envPaths('node-gyp', { suffix: '' }).cache

From 3b41971e2f6b90e02b1d7df592d403b8dfc8fa4d Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Fri, 7 Nov 2025 17:05:12 -0700
Subject: [PATCH 507/551] chore: increase test timeouts (#3237)

---
 package.json         | 2 +-
 test/test-install.js | 8 ++++----
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/package.json b/package.json
index 5ca4342ccf..0d0dbaae6e 100644
--- a/package.json
+++ b/package.json
@@ -47,6 +47,6 @@
   },
   "scripts": {
     "lint": "eslint \"*/*.js\" \"test/**/*.js\" \".github/**/*.js\"",
-    "test": "cross-env NODE_GYP_NULL_LOGGER=true mocha --timeout 15000 test/test-download.js test/test-*"
+    "test": "cross-env NODE_GYP_NULL_LOGGER=true mocha --timeout 30000 test/test-download.js test/test-*"
   }
 }
diff --git a/test/test-install.js b/test/test-install.js
index 3343545ba6..8aecf78e11 100644
--- a/test/test-install.js
+++ b/test/test-install.js
@@ -63,7 +63,7 @@ describe('install', function () {
     })
 
     afterEach(async () => {
-      await rm(prog.devDir, { recursive: true, force: true, maxRetries: 3 })
+      await rm(prog.devDir, { recursive: true, force: true, maxRetries: 3, retryDelay: 1000 })
       prog = null
     })
 
@@ -74,11 +74,11 @@ describe('install', function () {
       }
 
       return it(name, async function () {
-        this.timeout(platformTimeout(2, { win32: 20 }))
+        this.timeout(platformTimeout(4, { win32: 20 }))
         await fn.call(this)
         const expectedDir = path.join(prog.devDir, process.version.replace(/^v/, ''))
-        await rm(expectedDir, { recursive: true, force: true, maxRetries: 3 })
-        await Promise.all(new Array(10).fill(0).map(async (_, i) => {
+        await rm(expectedDir, { recursive: true, force: true, maxRetries: 3, retryDelay: 1000 })
+        await Promise.all(new Array(5).fill(0).map(async (_, i) => {
           const title = `${' '.repeat(8)}${name} ${(i + 1).toString().padEnd(2, ' ')}`
           console.log(`${title} : Start`)
           console.time(title)

From eaa8e34cb5a0710bef0602c42e5840b47eb76822 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Mon, 10 Nov 2025 14:45:59 -0700
Subject: [PATCH 508/551] build(deps): bump which from 5.0.0 to 6.0.0 (#3238)

Bumps [which](https://github.com/npm/node-which) from 5.0.0 to 6.0.0.
- [Release notes](https://github.com/npm/node-which/releases)
- [Changelog](https://github.com/npm/node-which/blob/main/CHANGELOG.md)
- [Commits](https://github.com/npm/node-which/compare/v5.0.0...v6.0.0)

---
updated-dependencies:
- dependency-name: which
  dependency-version: 6.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 package.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package.json b/package.json
index 0d0dbaae6e..1a5407af0e 100644
--- a/package.json
+++ b/package.json
@@ -31,7 +31,7 @@
     "semver": "^7.3.5",
     "tar": "^7.5.2",
     "tinyglobby": "^0.2.12",
-    "which": "^5.0.0"
+    "which": "^6.0.0"
   },
   "engines": {
     "node": "^20.17.0 || >=22.9.0"

From 641b220a0aa24218fbdd6a5959245a966c90de8c Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Tue, 11 Nov 2025 04:44:06 +0000
Subject: [PATCH 509/551] chore(main): release 12.0.0 (#3231)

---
 .release-please-manifest.json |  2 +-
 CHANGELOG.md                  | 29 +++++++++++++++++++++++++++++
 package.json                  |  2 +-
 3 files changed, 31 insertions(+), 2 deletions(-)

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
index 02eef11e2b..a3ff6f9b0c 100644
--- a/.release-please-manifest.json
+++ b/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "11.5.0"
+    ".": "12.0.0"
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7e2740db65..65f87f39d8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,34 @@
 # Changelog
 
+## [12.0.0](https://github.com/nodejs/node-gyp/compare/v11.5.0...v12.0.0) (2025-11-10)
+
+
+### ⚠ BREAKING CHANGES
+
+* align to npm 11 node engine range
+
+### Features
+
+* align to npm 11 node engine range ([2f85686](https://github.com/nodejs/node-gyp/commit/2f85686bbe745673350a8f9dbb0e86ee0190f213))
+* update gyp-next to v0.21.0 ([c57cd2e](https://github.com/nodejs/node-gyp/commit/c57cd2e86dc57707475b9f7e676e189f064817de))
+
+
+### Core
+
+* **deps:** bump actions/setup-node from 5 to 6 ([ae90e63](https://github.com/nodejs/node-gyp/commit/ae90e632d9fab85f4cd902dc9205ba9dfafaf3bc))
+* **deps:** bump env-paths from 2.2.1 to 3.0.0 ([#3235](https://github.com/nodejs/node-gyp/issues/3235)) ([5fffb2f](https://github.com/nodejs/node-gyp/commit/5fffb2ffee304cc898fdea7a0cd9e41d54c53839))
+* **deps:** bump which from 5.0.0 to 6.0.0 ([#3238](https://github.com/nodejs/node-gyp/issues/3238)) ([eaa8e34](https://github.com/nodejs/node-gyp/commit/eaa8e34cb5a0710bef0602c42e5840b47eb76822))
+* make-fetch-happen@15.0.0 ([e2b9d21](https://github.com/nodejs/node-gyp/commit/e2b9d21bce27c35d18fcb6f8583e386d15ce395c))
+* nopt@9.0.0 ([9bdeaf3](https://github.com/nodejs/node-gyp/commit/9bdeaf307cd7a254946859d306465989fa39dfb2))
+* proc-log@6.0.0 ([dfc68df](https://github.com/nodejs/node-gyp/commit/dfc68dfba3c17deb0bda9a395bb49d8fb9fa5951))
+
+
+### Miscellaneous
+
+* increase test timeouts ([#3237](https://github.com/nodejs/node-gyp/issues/3237)) ([3b41971](https://github.com/nodejs/node-gyp/commit/3b41971e2f6b90e02b1d7df592d403b8dfc8fa4d))
+* setup dependabot for npm ([86d65c7](https://github.com/nodejs/node-gyp/commit/86d65c7874eb41eb49c9b8bbf342becac8e57c6f))
+* update devDependencies ([41b0cea](https://github.com/nodejs/node-gyp/commit/41b0cea2f12342a790580cc8f844f075d49e096c))
+
 ## [11.5.0](https://github.com/nodejs/node-gyp/compare/v11.4.2...v11.5.0) (2025-10-15)
 
 
diff --git a/package.json b/package.json
index 1a5407af0e..ba483eae04 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "11.5.0",
+  "version": "12.0.0",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From 69e5fd2c98ac83dad5200a47515b301ccd80d2d3 Mon Sep 17 00:00:00 2001
From: xxairsky 
Date: Thu, 13 Nov 2025 04:36:14 +0800
Subject: [PATCH 510/551] feat: Support for Visual Studio 2026 (18.x)

feat: Add support for Visual Studio 2026 (18.x)

- Add versionMajor 18 mapping to versionYear 2026
- Update toolset to v145 for VS2026 (confirmed from MS docs)
- Support both Insiders (18\Insiders) and Release (2026\Community) installation paths
- Add comprehensive test coverage for both VS2026 variants
- Maintain backward compatibility with all existing VS versions

This enables node-gyp to work with Visual Studio 2026 Preview/Insiders and future 18.x releases.

Fixes #3228 #3221 #3214
---
 lib/find-visualstudio.js                     |  12 +-
 test/fixtures/VS_2026_Community_workload.txt | 568 +++++++++++++++++++
 test/fixtures/VS_2026_Insiders_workload.txt  | 568 +++++++++++++++++++
 test/test-find-visualstudio.js               | 108 +++-
 4 files changed, 1243 insertions(+), 13 deletions(-)
 create mode 100644 test/fixtures/VS_2026_Community_workload.txt
 create mode 100644 test/fixtures/VS_2026_Insiders_workload.txt

diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js
index e9aa7fafdc..e0cf383489 100644
--- a/lib/find-visualstudio.js
+++ b/lib/find-visualstudio.js
@@ -119,7 +119,7 @@ class VisualStudioFinder {
   }
 
   async findVisualStudio2019OrNewerFromSpecifiedLocation () {
-    return this.findVSFromSpecifiedLocation([2019, 2022])
+    return this.findVSFromSpecifiedLocation([2019, 2022, 2026])
   }
 
   async findVisualStudio2017FromSpecifiedLocation () {
@@ -162,7 +162,7 @@ class VisualStudioFinder {
   }
 
   async findVisualStudio2019OrNewerUsingSetupModule () {
-    return this.findNewVSUsingSetupModule([2019, 2022])
+    return this.findNewVSUsingSetupModule([2019, 2022, 2026])
   }
 
   async findVisualStudio2017UsingSetupModule () {
@@ -223,7 +223,7 @@ class VisualStudioFinder {
   // Invoke the PowerShell script to get information about Visual Studio 2019
   // or newer installations
   async findVisualStudio2019OrNewer () {
-    return this.findNewVS([2019, 2022])
+    return this.findNewVS([2019, 2022, 2026])
   }
 
   // Invoke the PowerShell script to get information about Visual Studio 2017
@@ -389,6 +389,10 @@ class VisualStudioFinder {
       ret.versionYear = 2022
       return ret
     }
+    if (ret.versionMajor === 18) {
+      ret.versionYear = 2026
+      return ret
+    }
     this.log.silly('- unsupported version:', ret.versionMajor)
     return {}
   }
@@ -456,6 +460,8 @@ class VisualStudioFinder {
       return 'v142'
     } else if (versionYear === 2022) {
       return 'v143'
+    } else if (versionYear === 2026) {
+      return 'v145'
     }
     this.log.silly('- invalid versionYear:', versionYear)
     return null
diff --git a/test/fixtures/VS_2026_Community_workload.txt b/test/fixtures/VS_2026_Community_workload.txt
new file mode 100644
index 0000000000..adfaf08a70
--- /dev/null
+++ b/test/fixtures/VS_2026_Community_workload.txt
@@ -0,0 +1,568 @@
+[
+    {
+        "path": "C:\\Program Files\\Microsoft Visual Studio\\2026\\Community",
+        "version": "18.0.1000.100",
+        "packages": [
+            "Microsoft.VisualStudio.Product.Community",
+            "Microsoft.VisualStudio.PackageGroup.LiveShare.VSCore",
+            "Microsoft.VisualStudio.LiveShare.VSCore",
+            "Microsoft.VisualStudio.Workload.NativeDesktop",
+            "Microsoft.VisualStudio.Component.VC.ASAN",
+            "Microsoft.VisualCpp.ASAN.X86",
+            "Microsoft.VC.14.50.18.0.ASAN.X86.base",
+            "Microsoft.VC.14.50.18.0.ASAN.X64.base",
+            "Microsoft.VC.14.50.18.0.ASAN.Headers.base",
+            "Microsoft.VisualStudio.VC.IDE.Project.Factories",
+            "Microsoft.VisualStudio.Component.VC.TestAdapterForGoogleTest",
+            "Microsoft.VisualStudio.VC.Ide.TestAdapterForGoogleTest",
+            "Microsoft.VisualStudio.Component.VC.TestAdapterForBoostTest",
+            "Microsoft.VisualStudio.VC.Ide.TestAdapterForBoostTest",
+            "Microsoft.VisualStudio.Component.VC.CMake.Project",
+            "Microsoft.VisualStudio.ComponentGroup.WebToolsExtensions.CMake",
+            "Microsoft.VisualStudio.VC.CMake",
+            "Microsoft.VisualStudio.VC.CMake.Project",
+            "Microsoft.VisualStudio.VC.CMake.Client",
+            "Microsoft.VisualStudio.VC.ExternalBuildFramework",
+            "Microsoft.VisualStudio.Component.VC.DiagnosticTools",
+            "Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core",
+            "Microsoft.VisualStudio.PackageGroup.TestTools.Native",
+            "Microsoft.VisualStudio.Component.VC.Redist.14.Latest",
+            "Microsoft.VisualStudio.VC.Templates.UnitTest",
+            "Microsoft.VisualStudio.VC.UnitTest.Desktop.Build.Core",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.V1.CPP",
+            "Microsoft.VisualStudio.VC.Templates.UnitTest.Resources",
+            "Microsoft.VisualStudio.VC.Templates.Desktop",
+            "Microsoft.VisualStudio.Component.Graphics",
+            "Microsoft.VisualStudio.Graphics.Viewers",
+            "Microsoft.VisualStudio.Graphics.Viewers.Resources",
+            "Microsoft.VisualStudio.Component.VC.ATL.ARM64",
+            "Microsoft.VisualCpp.ATL.ARM64",
+            "Microsoft.VC.14.50.18.0.ATL.ARM64.base",
+            "Microsoft.VisualStudio.Component.VC.ATL",
+            "Microsoft.VisualStudio.VC.Ide.ATL",
+            "Microsoft.VisualStudio.VC.Ide.ATL.Resources",
+            "Microsoft.VisualCpp.ATL.X86",
+            "Microsoft.VC.14.50.18.0.ATL.X86.base",
+            "Microsoft.VisualCpp.ATL.X64",
+            "Microsoft.VC.14.50.18.0.ATL.X64.base",
+            "Microsoft.VC.14.50.18.0.Props.ATLMFC",
+            "Microsoft.VisualCpp.ATL.Source",
+            "Microsoft.VC.14.50.18.0.ATL.Source.base",
+            "Microsoft.VisualCpp.ATL.Headers",
+            "Microsoft.VC.14.50.18.0.ATL.Headers.base",
+            "Microsoft.VC.14.50.18.0.Servicing.ATL",
+            "Microsoft.VisualStudio.Component.VC.Tools.ARM64",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.ARM64.v145",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.ARM64",
+            "Microsoft.VS.VC.vcvars.arm64.Shortcuts",
+            "Microsoft.VisualCpp.CA.Ext.Hostx64.TargetARM64",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx64.TargetARM64.base",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx64.TargetARM64.Res.base",
+            "Microsoft.VisualCpp.CA.Ext.Hostx86.TargetARM64",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx86.TargetARM64.base",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx86.TargetARM64.Res.base",
+            "Microsoft.VisualCpp.CA.Ext.HostARM64.TargetARM64",
+            "Microsoft.VC.14.50.18.0.CA.Ext.HostARM64.TargetARM64.base",
+            "Microsoft.VC.14.50.18.0.CA.Ext.HostARM64.TargetARM64.Res.base",
+            "Microsoft.VisualCpp.Tools.Hostx86.Targetarm64",
+            "Microsoft.VC.14.50.18.0.Tools.Hostx86.Targetarm64.base",
+            "Microsoft.VC.14.50.18.0.Tools.HostX86.TargetARM64.Res.base",
+            "Microsoft.VisualCpp.Tools.HostARM64.TargetARM64",
+            "Microsoft.VC.14.50.18.0.Tools.HostARM64.TargetARM64.base",
+            "Microsoft.VC.14.50.18.0.Tools.HostARM64.TargetARM64.Res.base",
+            "Microsoft.VisualCpp.CRT.Redist.ARM64.OneCore.Desktop",
+            "Microsoft.VC.14.50.18.0.CRT.Redist.ARM64.OneCore.Desktop.base",
+            "Microsoft.VisualCpp.CRT.Redist.ARM64",
+            "Microsoft.VC.14.50.18.0.CRT.Redist.ARM64.base",
+            "Microsoft.VisualCpp.CRT.ARM64.OneCore.Desktop",
+            "Microsoft.VC.14.50.18.0.CRT.ARM64.OneCore.Desktop.base",
+            "Microsoft.VC.14.50.18.0.CRT.ARM64.OneCore.Desktop.debug.base",
+            "Microsoft.VisualCpp.CRT.ARM64.Store",
+            "Microsoft.VC.14.50.18.0.CRT.ARM64.Store.base",
+            "Microsoft.VisualCpp.CRT.ARM64.Desktop",
+            "Microsoft.VC.14.50.18.0.CRT.ARM64.Desktop.base",
+            "Microsoft.VC.14.50.18.0.CRT.ARM64.Desktop.debug.base",
+            "Microsoft.VisualStudio.PackageGroup.VC.Tools.x64.ARM64",
+            "Microsoft.VisualCpp.Tools.Core",
+            "Microsoft.VisualCpp.PGO.ARM64",
+            "Microsoft.VC.14.50.18.0.PGO.ARM64.base",
+            "Microsoft.VisualCpp.Premium.Tools.Hostx86.Targetarm64",
+            "Microsoft.VC.14.50.18.0.Premium.Tools.Hostx86.Targetarm64.base",
+            "Microsoft.VC.14.50.18.0.Prem.HostX86.TargetARM64.Res.base",
+            "Microsoft.VisualCpp.Premium.Tools.HostX64.TargetARM64",
+            "Microsoft.VC.14.50.18.0.Premium.Tools.HostX64.TargetARM64.base",
+            "Microsoft.VC.14.50.18.0.Prem.HostX64.TargetARM64.Res.base",
+            "Microsoft.VisualCpp.Premium.Tools.ARM64.Base",
+            "Microsoft.VC.14.50.18.0.Premium.Tools.ARM64.Base.base",
+            "Microsoft.VisualCpp.Tools.HostX64.TargetARM64",
+            "Microsoft.VC.14.50.18.0.Tools.HostX64.TargetARM64.base",
+            "Microsoft.VC.14.50.18.0.Props.ARM64",
+            "Microsoft.VC.14.50.18.0.Tools.HostX64.TargetARM64.Res.base",
+            "Microsoft.VisualStudio.Component.VC.Tools.ARM64EC",
+            "Microsoft.VisualStudio.Component.Windows11SDK.26100",
+            "Win11SDK_10.0.26100",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.ARM64EC.v145",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.ARM64EC",
+            "Microsoft.VisualCpp.CRT.ARM64EC.Store",
+            "Microsoft.VC.14.50.18.0.CRT.ARM64EC.Store.base",
+            "Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
+            "Microsoft.VisualCpp.CodeAnalysis.Extensions",
+            "Microsoft.VisualCpp.CA.Ext.HostARM64.Targetx64",
+            "Microsoft.VC.14.50.18.0.CA.Ext.HostARM64.Targetx64.base",
+            "Microsoft.VC.14.50.18.0.CA.Ext.HostARM64.Targetx64.Res.base",
+            "Microsoft.VisualCpp.CA.Ext.HostARM64.Targetx86",
+            "Microsoft.VC.14.50.18.0.CA.Ext.HostARM64.Targetx86.base",
+            "Microsoft.VC.14.50.18.0.CA.Ext.HostARM64.Targetx86.Res.base",
+            "Microsoft.VisualCpp.CA.Ext.Hostx86.Targetx64",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx86.Targetx64.base",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx86.Targetx64.Res.base",
+            "Microsoft.VisualCpp.CA.Ext.Hostx86.Targetx86",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx86.Targetx86.base",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx86.Targetx86.Res.base",
+            "Microsoft.VisualCpp.CA.Ext.Hostx64.Targetx64",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx64.Targetx64.base",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx64.Targetx64.Res.base",
+            "Microsoft.VisualCpp.CA.Ext.Hostx64.Targetx86",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx64.Targetx86.base",
+            "Microsoft.VC.14.50.18.0.Servicing.CAExtensions",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx64.Targetx86.Res.base",
+            "Microsoft.VisualCpp.Tools.HostX64.TargetX86",
+            "Microsoft.VC.14.50.18.0.Tools.HostX64.TargetX86.base",
+            "Microsoft.VC.14.50.18.0.Tools.HostX64.TargetX86.Res.base",
+            "Microsoft.VisualCpp.Tools.HostX64.TargetX64",
+            "Microsoft.VC.14.50.18.0.Tools.HostX64.TargetX64.base",
+            "Microsoft.VC.14.50.18.0.Tools.HostX64.TargetX64.Res.base",
+            "Microsoft.VisualCpp.Tools.HostARM64.TargetX86",
+            "Microsoft.VC.14.50.18.0.Tools.HostARM64.TargetX86.base",
+            "Microsoft.VisualCpp.RuntimeDebug.14",
+            "Microsoft.VC.14.50.18.0.Tools.HostARM64.TargetX86.Res.base",
+            "Microsoft.VisualCpp.Tools.HostARM64.TargetX64",
+            "Microsoft.VC.14.50.18.0.Tools.HostARM64.TargetX64.base",
+            "Microsoft.VisualCpp.RuntimeDebug.14.ARM64",
+            "Microsoft.VisualCpp.Redist.14.Latest",
+            "Microsoft.VisualCpp.Redist.14.Latest",
+            "Microsoft.VC.14.50.18.0.Tools.HostARM64.Targetx64.Res.base",
+            "Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX64",
+            "Microsoft.VC.14.50.18.0.Premium.Tools.HostX86.TargetX64.base",
+            "Microsoft.VC.14.50.18.0.Prem.Hostx86.Targetx64.Res.base",
+            "Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86",
+            "Microsoft.VC.14.50.18.0.Premium.Tools.HostX86.TargetX86.base",
+            "Microsoft.VC.14.50.18.0.Prem.HostX86.TargetX86.Res.base",
+            "Microsoft.VisualCpp.Premium.Tools.HostARM64.TargetX86",
+            "Microsoft.VC.14.50.18.0.Premium.Tools.HostARM64.TargetX86.base",
+            "Microsoft.VC.14.50.18.0.Prem.HostARM64.TargetX86.Res.base",
+            "Microsoft.VisualCpp.Premium.Tools.HostARM64.TargetX64",
+            "Microsoft.VC.14.50.18.0.Premium.Tools.HostARM64.TargetX64.base",
+            "Microsoft.VC.14.50.18.0.Prem.HostARM64.Targetx64.Res.base",
+            "Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86",
+            "Microsoft.VC.14.50.18.0.Premium.Tools.HostX64.TargetX86.base",
+            "Microsoft.VC.14.50.18.0.Prem.HostX64.TargetX86.Res.base",
+            "Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64",
+            "Microsoft.VC.14.50.18.0.Premium.Tools.HostX64.TargetX64.base",
+            "Microsoft.VC.14.50.18.0.Prem.HostX64.TargetX64.Res.base",
+            "Microsoft.VisualCpp.PGO.X86",
+            "Microsoft.VC.14.50.18.0.PGO.X86.base",
+            "Microsoft.VisualCpp.PGO.X64",
+            "Microsoft.VC.14.50.18.0.PGO.X64.base",
+            "Microsoft.VisualCpp.PGO.Headers",
+            "Microsoft.VC.14.50.18.0.PGO.Headers.base",
+            "Microsoft.VisualCpp.CRT.x86.Store",
+            "Microsoft.VC.14.50.18.0.CRT.x86.Store.base",
+            "Microsoft.VisualCpp.CRT.x86.OneCore.Desktop",
+            "Microsoft.VC.14.50.18.0.CRT.x86.OneCore.Desktop.base",
+            "Microsoft.VisualCpp.CRT.x64.Store",
+            "Microsoft.VC.14.50.18.0.CRT.x64.Store.base",
+            "Microsoft.VisualCpp.CRT.x64.OneCore.Desktop",
+            "Microsoft.VC.14.50.18.0.CRT.x64.OneCore.Desktop.base",
+            "Microsoft.VisualCpp.CRT.Redist.x86.OneCore.Desktop",
+            "Microsoft.VC.14.50.18.0.CRT.Redist.x86.OneCore.Desktop.base",
+            "Microsoft.VisualCpp.CRT.Redist.x64.OneCore.Desktop",
+            "Microsoft.VC.14.50.18.0.CRT.Redist.x64.OneCore.Desktop.base",
+            "Microsoft.VisualStudio.PackageGroup.VC.Tools.x86",
+            "Microsoft.VisualCpp.Tools.Hostx86.Targetx64.Res",
+            "Microsoft.VisualCpp.Tools.HostX86.TargetX64",
+            "Microsoft.VC.14.50.18.0.Tools.HostX86.TargetX64.base",
+            "Microsoft.VC.14.50.18.0.Props.x64",
+            "Microsoft.VC.14.50.18.0.Tools.Hostx86.Targetx64.Res.base",
+            "Microsoft.VisualCpp.Tools.HostX86.TargetX86.Res",
+            "Microsoft.VisualCpp.Tools.HostX86.TargetX86",
+            "Microsoft.VC.14.50.18.0.Tools.HostX86.TargetX86.base",
+            "Microsoft.VC.14.50.18.0.Servicing.Compilers",
+            "Microsoft.VC.14.50.18.0.Props.x86",
+            "Microsoft.VC.14.50.18.0.Props",
+            "Microsoft.VC.14.50.18.0.Tools.HostX86.TargetX86.Res.base",
+            "Microsoft.VisualCpp.Tools.Core.Resources",
+            "Microsoft.VisualCpp.Tools.Core.x86",
+            "Microsoft.VC.14.50.18.0.Tools.Core.Props",
+            "Microsoft.VisualCpp.DIA.SDK",
+            "Microsoft.VisualCpp.Servicing.DIASDK",
+            "Microsoft.VisualCpp.CRT.x86.Desktop",
+            "Microsoft.VC.14.50.18.0.CRT.x86.Desktop.base",
+            "Microsoft.VisualCpp.CRT.x64.Desktop",
+            "Microsoft.VC.14.50.18.0.CRT.x64.Desktop.base",
+            "Microsoft.VisualCpp.CRT.Source",
+            "Microsoft.VC.14.50.18.0.CRT.Source.base",
+            "Microsoft.VisualCpp.CRT.Redist.X86",
+            "Microsoft.VC.14.50.18.0.CRT.Redist.X86.base",
+            "Microsoft.VisualCpp.CRT.Redist.X64",
+            "Microsoft.VisualCpp.CRT.Redist.Resources",
+            "Microsoft.VC.14.50.18.0.CRT.Redist.X64.base",
+            "Microsoft.VisualCpp.CRT.Headers",
+            "Microsoft.VC.14.50.18.0.CRT.Headers.base",
+            "Microsoft.VC.14.50.18.0.Servicing.CrtHeaders",
+            "Microsoft.VC.14.50.18.0.Servicing",
+            "Microsoft.VisualStudio.Component.VC.CoreIde",
+            "Microsoft.VisualStudio.VC.Ide.Pro",
+            "Microsoft.VisualStudio.VC.Ide.Pro.Resources",
+            "Microsoft.VisualStudio.VC.Templates.General",
+            "Microsoft.VisualStudio.VC.Templates.General.Resources",
+            "Microsoft.VisualStudio.VC.Items.Pro",
+            "Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Reduced",
+            "Microsoft.VisualStudio.VC.Ide.x64",
+            "Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Express",
+            "Microsoft.VisualStudio.VC.vcvars",
+            "Microsoft.VS.VC.vcvars.x86.Shortcuts",
+            "Microsoft.VS.VC.vcvars.x64.Shortcuts",
+            "Microsoft.VS.VC.vcvars.arm64_x64.Shortcuts",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.X64.v145",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.X64",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.ARM.v145",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.ARM",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.x86.v145",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.X86",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.Base",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.Base.Resources",
+            "Microsoft.VisualStudio.VC.Ide.WinXPlus",
+            "Microsoft.VisualStudio.VC.Ide.Dskx",
+            "Microsoft.VisualStudio.VC.Ide.Dskx.Resources",
+            "Microsoft.VisualStudio.VC.Ide.Base",
+            "Microsoft.VisualStudio.VC.Ide.LanguageService",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.Scripts",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.PythonDistro",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.10",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.9",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.8",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.7",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.6",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.5",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.4",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.3",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.2",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.1",
+            "Microsoft.VisualStudio.VC.Ide.VCPkgDatabase",
+            "Microsoft.VisualStudio.VC.Ide.Core",
+            "Microsoft.VisualStudio.VC.Ide.ProjectSystem",
+            "Microsoft.VisualStudio.VC.Ide.ProjectSystem.Resources",
+            "Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine",
+            "Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine.Resources",
+            "Microsoft.VisualStudio.VC.Ide.LanguageService.Resources",
+            "Microsoft.VisualStudio.VC.Llvm.Base",
+            "Microsoft.VisualStudio.VC.Ide.Base.Resources",
+            "Microsoft.Net.PackageGroup.4.8.1.Redist",
+            "Microsoft.VisualStudio.Component.IntelliCode",
+            "Microsoft.VisualStudio.IntelliCode.CSharp",
+            "Microsoft.VisualStudio.IntelliCode",
+            "Component.Microsoft.VisualStudio.LiveShare.2026",
+            "Microsoft.VisualStudio.Component.Debugger.JustInTime",
+            "Microsoft.VisualStudio.Debugger.ImmersiveActivateHelper.Msi",
+            "Microsoft.VisualStudio.Debugger.JustInTime",
+            "Microsoft.VisualStudio.Debugger.JustInTime.Msi",
+            "Microsoft.VisualStudio.LiveShare.2026",
+            "Microsoft.Icecap.Analysis",
+            "Microsoft.Icecap.Analysis.Resources",
+            "Microsoft.Icecap.Analysis.Resources.Targeted",
+            "Microsoft.Icecap.Collection.Msi",
+            "Microsoft.Icecap.Collection.Msi.Targeted",
+            "Microsoft.Icecap.Collection.Msi.Resources",
+            "Microsoft.Icecap.Collection.Msi.Resources.Targeted",
+            "Microsoft.DiagnosticsHub.Instrumentation",
+            "Microsoft.DiagnosticsHub.Instrumentation.Targeted",
+            "Microsoft.DiagnosticsHub.CpuSampling",
+            "Microsoft.DiagnosticsHub.CpuSampling.Targeted",
+            "Microsoft.PackageGroup.DiagnosticsHub.Platform",
+            "Microsoft.VisualStudio.InstrumentationEngine.ARM64",
+            "Microsoft.VisualStudio.InstrumentationEngine",
+            "Microsoft.DiagnosticsHub.Runtime.ExternalDependencies",
+            "SQLiteCore",
+            "SQLiteCore.Targeted",
+            "Microsoft.DiagnosticsHub.Runtime.ExternalDependencies.Targeted",
+            "Microsoft.DiagnosticsHub.Runtime",
+            "Microsoft.DiagnosticsHub.Runtime.Targeted",
+            "Microsoft.DiagnosticsHub.Collection.ExternalDependencies.arm64",
+            "Microsoft.DiagnosticsHub.Collection",
+            "Microsoft.DiagnosticsHub.Collection.Service",
+            "Microsoft.VisualStudio.VC.Ide.MDD",
+            "Microsoft.VisualStudio.VC.Ide.Linux.ConnectionManager",
+            "Microsoft.VisualStudio.VisualC.Utilities",
+            "Microsoft.VisualStudio.VisualC.Utilities.Resources",
+            "Microsoft.VisualStudio.VC.Ide.Linux.ConnectionManager.Resources",
+            "Microsoft.VisualStudio.VC.Ide.ResourceEditor",
+            "Microsoft.VisualStudio.VC.Ide.ResourceEditor.Resources",
+            "Microsoft.VisualStudio.PackageGroup.TestTools.Core",
+            "Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.V2.CLI",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.V2.CLI",
+            "Microsoft.VisualStudio.TestTools.Pex.Common",
+            "Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.V1.CLI",
+            "Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.Legacy",
+            "Microsoft.VisualStudio.PackageGroup.MinShell.Interop",
+            "Microsoft.VisualStudio.TestTools.TP.Legacy.Tips.Msi",
+            "Microsoft.VisualStudio.TestTools.TP.Legacy.Tips.Common",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Tips",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Tips.Resources",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.TestSettings",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Professional",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Common",
+            "Microsoft.VisualStudio.TestTools.TP.Legacy.Common.Res",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Core",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Core.Resources",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Agent",
+            "Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.IDE",
+            "Microsoft.VisualStudio.Cache.Service",
+            "Microsoft.VisualStudio.TestTools.TestWIExtension",
+            "Microsoft.VisualStudio.TestTools.TestWIExtension.Res",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.V1.CLI",
+            "Microsoft.VisualStudio.TestTools.TP.V1.CLI.Resources",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.IDE",
+            "Microsoft.VisualStudio.PackageGroup.TestTools.CodeCoverage",
+            "Microsoft.VisualStudio.PackageGroup.TestTools.DataCollectors",
+            "Microsoft.VisualStudio.Component.NuGet",
+            "Microsoft.CredentialProvider",
+            "Microsoft.VisualStudio.NuGet.Licenses",
+            "Microsoft.VisualStudio.Component.TextTemplating",
+            "Microsoft.VisualStudio.TextTemplating.MSBuild",
+            "Microsoft.VisualStudio.TextTemplating.Integration",
+            "Microsoft.VisualStudio.TextTemplating.Core",
+            "Microsoft.VisualStudio.TextTemplating.Integration.Resources",
+            "Microsoft.VisualCpp.CRT.ClickOnce.Msi",
+            "Microsoft.VisualStudio.Component.Roslyn.LanguageServices",
+            "Microsoft.VisualStudio.InteractiveWindow",
+            "Microsoft.DiaSymReader.Native",
+            "Microsoft.VisualCpp.Redist.14",
+            "Microsoft.VisualCpp.Redist.14",
+            "Microsoft.VisualCpp.Servicing.Redist",
+            "Microsoft.VisualStudio.PackageGroup.StaticAnalysis",
+            "Microsoft.VisualStudio.StaticAnalysis.IDE",
+            "Microsoft.VisualStudio.StaticAnalysis.IDE.Resources",
+            "Microsoft.VisualStudio.StaticAnalysis.FxCop.Resources",
+            "Microsoft.VisualStudio.StaticAnalysis.auxil",
+            "Microsoft.VisualStudio.StaticAnalysis.auxil.Resources",
+            "Roslyn.VisualStudio.Setup.ServiceHub",
+            "Microsoft.Component.MSBuild",
+            "Microsoft.NuGet.Build.Tasks.Setup",
+            "Microsoft.VisualStudio.Component.Roslyn.Compiler",
+            "Microsoft.CodeAnalysis.Compilers",
+            "Microsoft.VisualStudio.Component.JavaScript.TypeScript",
+            "Microsoft.VisualStudio.JavaScript.ProjectSystem",
+            "Microsoft.VisualStudio.ComponentGroup.WebToolsExtensions",
+            "Microsoft.VisualStudio.ProTools",
+            "sqlsysclrtypes",
+            "SQLCommon",
+            "Microsoft.VisualStudio.ProTools.Resources",
+            "Microsoft.VisualStudio.Web.Scaffolding",
+            "Microsoft.VisualStudio.WebToolsExtensions",
+            "Microsoft.VisualStudio.ConnectedServices.Core",
+            "Microsoft.VisualStudio.WebTools",
+            "Microsoft.VisualStudio.WebToolsExtensions.MSBuild",
+            "Microsoft.VisualStudio.WebTools.Resources",
+            "Microsoft.VisualStudio.WebTools.WSP.FSA",
+            "Microsoft.VisualStudio.WebTools.WSP.FSA.Resources",
+            "Microsoft.VisualStudio.PackageGroup.Debugger.Script",
+            "Microsoft.VisualStudio.Component.TypeScript.TSServer",
+            "Microsoft.VisualStudio.Package.TypeScript.TSServer",
+            "Microsoft.VisualStudio.PackageGroup.JavaScript.Language",
+            "Microsoft.VisualStudio.Package.NodeJs",
+            "TypeScript.Build",
+            "TypeScript.LanguageService",
+            "TypeScript.Tools",
+            "Microsoft.VisualStudio.PackageGroup.Community",
+            "Microsoft.VisualStudio.Community.VB.x86",
+            "Microsoft.VisualStudio.Community.VB.x64",
+            "Microsoft.VisualStudio.PackageGroup.Core",
+            "Microsoft.VisualStudio.CodeSense.Community",
+            "Microsoft.VisualStudio.TestTools.TeamFoundationClient",
+            "Microsoft.VisualStudio.PackageGroup.Debugger.Core",
+            "Microsoft.VisualStudio.Debugger.BrokeredServices",
+            "Microsoft.VisualStudio.Debugger.VSCodeDebuggerHost",
+            "Microsoft.VisualStudio.Debugger.AzureAttach",
+            "Microsoft.VisualStudio.Web.Azure.Common",
+            "Microsoft.WebTools.Shared",
+            "Microsoft.WebTools.DotNet.Core.ItemTemplates",
+            "Microsoft.VisualStudio.PackageGroup.Debugger.TimeTravel.Replay",
+            "Microsoft.VisualStudio.VC.Ide.Debugger",
+            "Microsoft.VisualStudio.VC.Ide.Debugger.Concord",
+            "Microsoft.VisualStudio.VC.Ide.Debugger.Concord.Resources",
+            "Microsoft.VisualStudio.VC.Ide.Debugger.Resources",
+            "Microsoft.VisualStudio.VC.Ide.Common",
+            "Microsoft.VisualStudio.VC.Ide.Common.Resources",
+            "Microsoft.VisualStudio.Debugger.CollectionAgents",
+            "Microsoft.VisualStudio.Debugger.Parallel",
+            "Microsoft.VisualStudio.Debugger.Parallel.Resources",
+            "Microsoft.VisualStudio.Debugger.Managed",
+            "Microsoft.CodeAnalysis.ExpressionEvaluator",
+            "Microsoft.CodeAnalysis.VisualStudio.Setup",
+            "Microsoft.VisualStudio.Debugger.Concord.Managed",
+            "Microsoft.VisualStudio.Debugger.Concord.Managed.Resources",
+            "Microsoft.VisualStudio.Debugger.Managed.Resources",
+            "Microsoft.VisualStudio.Debugger.TargetComposition",
+            "Microsoft.VisualStudio.Debugger.TargetComposition.Remote.arm64",
+            "Microsoft.VisualStudio.Debugger.TargetComposition.Remote",
+            "Microsoft.VisualStudio.Debugger.TargetComposition.Remote",
+            "Microsoft.VisualStudio.Debugger.Remote",
+            "Microsoft.VisualStudio.Debugger.Concord.Remote",
+            "Microsoft.VisualStudio.Debugger.Concord.Remote.Resources",
+            "Microsoft.VisualStudio.Debugger.Remote",
+            "Microsoft.VisualStudio.Debugger.Remote.ARM64",
+            "Microsoft.VisualStudio.Debugger.Concord.Remote.ARM64",
+            "Microsoft.VisualStudio.Debugger.Concord.Remote.Resources.ARM64",
+            "Microsoft.VisualStudio.Debugger.Remote.Resources.ARM",
+            "Microsoft.VisualStudio.Debugger.Remote.Resources.ARM64",
+            "Microsoft.VisualStudio.Debugger.Concord.Remote",
+            "Microsoft.VisualStudio.Debugger.Concord.Remote.Resources",
+            "Microsoft.VisualStudio.Debugger.Remote.Resources",
+            "Microsoft.VisualStudio.Debugger.Remote.Resources",
+            "Microsoft.VisualStudio.Debugger",
+            "Microsoft.VisualStudio.VC.MSVCDis",
+            "Microsoft.IntelliTrace.DiagnosticsHub",
+            "Microsoft.VisualStudio.Debugger.Concord",
+            "Microsoft.VisualStudio.Debugger.Concord.Resources",
+            "Microsoft.VisualStudio.Debugger.Resources",
+            "Microsoft.VisualStudio.Debugger.Package.DiagHub.Client",
+            "Microsoft.VisualStudio.Debugger.Remote.DiagnosticsHub.Client",
+            "Microsoft.VisualStudio.Debugger.Remote.DiagnosticsHub.Client",
+            "Microsoft.VisualStudio.Debugger.Remote.DiagnosticsHub.Client",
+            "Microsoft.PackageGroup.ClientDiagnostics",
+            "Microsoft.VisualStudio.AppResponsiveness",
+            "Microsoft.VisualStudio.AppResponsiveness.Targeted",
+            "Microsoft.VisualStudio.AppResponsiveness.Resources",
+            "Microsoft.VisualStudio.ClientDiagnostics",
+            "Microsoft.VisualStudio.ClientDiagnostics.Targeted",
+            "Microsoft.VisualStudio.ClientDiagnostics.Resources",
+            "Microsoft.VisualStudio.PackageGroup.CommunityCore",
+            "Microsoft.VisualStudio.ProjectSystem.Full",
+            "Microsoft.VisualStudio.LiveShareApi",
+            "Microsoft.VisualStudio.ProjectSystem.Query",
+            "Microsoft.VisualStudio.ProjectSystem",
+            "Microsoft.VisualStudio.Community.x86",
+            "Microsoft.VisualStudio.Community.x64",
+            "Microsoft.VisualStudio.Community.Msi.Resources",
+            "Microsoft.VisualStudio.Community.Msi",
+            "Microsoft.VisualStudio.Community.Shared.Msi",
+            "Microsoft.VisualStudio.Devenv.Msi",
+            "Microsoft.VisualStudio.Devenv.Shared.Msi",
+            "Microsoft.VisualStudio.MinShell.Interop.Msi",
+            "Microsoft.VisualStudio.MinShell.Interop.Shared.Msi",
+            "Microsoft.VisualStudio.Editors",
+            "Microsoft.VisualStudio.Workload.CoreEditor",
+            "Microsoft.VisualStudio.Component.CoreEditor",
+            "Microsoft.VisualStudio.PackageGroup.CoreEditor",
+            "Microsoft.WebView2",
+            "Microsoft.VisualStudio.ScriptedHost",
+            "Microsoft.VisualStudio.ScriptedHost.Targeted",
+            "Microsoft.VisualCpp.Tools.Common.UtilsPrereq",
+            "Microsoft.VisualCpp.Tools.Common.Utils",
+            "Microsoft.VisualCpp.Tools.Common.Utils.Resources",
+            "Microsoft.VisualStudio.PackageGroup.VsDevCmd",
+            "Microsoft.VisualStudio.VsDevCmd.Ext.NetFxSdk",
+            "Microsoft.VisualStudio.VsDevCmd.Core.WinSdk",
+            "Microsoft.VisualStudio.VsDevCmd.Core.DotNet",
+            "Microsoft.VisualStudio.VC.DevCmd",
+            "Microsoft.VisualStudio.VC.DevCmd.Resources",
+            "Microsoft.VisualStudio.VirtualTree",
+            "Microsoft.DiaSymReader",
+            "Microsoft.Build.Dependencies",
+            "Microsoft.Build.FileTracker.Msi",
+            "Microsoft.Build",
+            "Microsoft.VisualStudio.PackageGroup.NuGet",
+            "Microsoft.DataAI.NuGetRecommender",
+            "Microsoft.VisualStudio.NuGet.Core",
+            "Microsoft.Build.Arm64",
+            "Microsoft.Build.UnGAC",
+            "Microsoft.VisualStudio.TextMateGrammars",
+            "Microsoft.VisualStudio.Platform.Markdown",
+            "Microsoft.VisualStudio.Platform.CrossRepositorySearch",
+            "Microsoft.VisualStudio.PackageGroup.TeamExplorer.Common",
+            "Microsoft.VisualStudio.TeamExplorer",
+            "Microsoft.VisualStudio.PackageGroup.ServiceHub",
+            "Microsoft.ServiceHub.Node",
+            "Microsoft.ServiceHub.Managed",
+            "Microsoft.ServiceHub.arm64",
+            "Microsoft.VisualStudio.ProjectServices",
+            "Microsoft.VisualStudio.OpenFolder.VSIX",
+            "Microsoft.VisualStudio.FileHandler.Msi",
+            "Microsoft.VisualStudio.FileHandler.Msi",
+            "Microsoft.VisualStudio.PackageGroup.MinShell",
+            "Microsoft.VisualStudio.MinShell.Msi",
+            "Microsoft.VisualStudio.MinShell.Shared.Msi",
+            "Microsoft.VisualStudio.MinShell.Msi.Resources",
+            "Microsoft.VisualStudio.MinShell.Interop",
+            "CoreEditorFonts",
+            "Microsoft.VisualStudio.Log",
+            "Microsoft.VisualStudio.Log.Targeted",
+            "Microsoft.VisualStudio.Log.Resources",
+            "Microsoft.VisualStudio.Finalizer",
+            "Microsoft.VisualStudio.Devenv",
+            "Microsoft.VisualStudio.Devenv.Resources",
+            "Microsoft.VisualStudio.CoreEditor",
+            "Microsoft.VisualStudio.Navigation.RichCodeNav",
+            "Microsoft.VisualStudio.Platform.NavigateTo",
+            "Microsoft.VisualStudio.Connected",
+            "SQLitePCLRaw",
+            "SQLitePCLRaw.Targeted",
+            "Microsoft.VisualStudio.Connected.Auto",
+            "Microsoft.VisualStudio.Connected.Auto.Resources",
+            "Microsoft.VisualStudio.AzureSDK",
+            "Microsoft.VisualStudio.PerfLib",
+            "Microsoft.VisualStudio.Connected.Resources",
+            "Microsoft.Net.PackageGroup.4.8.Redist",
+            "Microsoft.VisualStudio.PackageGroup.Progression",
+            "Microsoft.VisualStudio.PerformanceProvider",
+            "Microsoft.VisualStudio.GraphModel",
+            "Microsoft.VisualStudio.GraphProvider",
+            "Microsoft.VisualStudio.Community.VB.Targeted",
+            "Microsoft.VisualStudio.Community.VB.Neutral",
+            "Microsoft.VisualStudio.Community.CSharp.Targeted",
+            "Microsoft.VisualStudio.Community.CSharp.Neutral",
+            "Microsoft.VisualStudio.Community.ProductArch.TargetedExtra",
+            "Microsoft.VisualStudio.Community.ProductArch.Targeted",
+            "Microsoft.VisualStudio.Community.ProductArch.NeutralExtra",
+            "Microsoft.DiaSymReader.PortablePdb",
+            "Microsoft.IntelliTrace.CollectorCab",
+            "Microsoft.VisualStudio.Community.VB.Resources.Targeted",
+            "Microsoft.VisualStudio.Community.VB.Resources.Neutral",
+            "Microsoft.VisualStudio.Community.CSharp.Resources.Targeted",
+            "Microsoft.VisualStudio.Community.CSharp.Resources.Neutral",
+            "Microsoft.VisualStudio.Community.ProductArch.Resources.Targeted",
+            "Microsoft.VisualStudio.Community.ProductArch.Resources.NeutralExtra",
+            "Microsoft.VisualStudio.Net.Eula.Resources",
+            "Microsoft.VisualStudio.Community.ProductArch.Resources.Neutral",
+            "Microsoft.VisualStudio.WebSiteProject.DTE",
+            "Microsoft.VisualStudio.Diagnostics.AspNetHelper",
+            "Microsoft.VisualStudio.Diagnostics.AspNetHelper.Standard",
+            "Microsoft.MSHtml",
+            "Microsoft.VisualStudio.Platform.CallHierarchy",
+            "Microsoft.VisualStudio.Community.ProductArch.Neutral",
+            "Microsoft.VisualStudio.MinShell",
+            "Microsoft.VisualStudio.VsWebProtocolSelector.Msi",
+            "Microsoft.Net.8.0.WindowsDesktop.Runtime",
+            "Microsoft.Net.8.0.Runtime",
+            "Microsoft.VisualStudio.PackageGroup.Setup.Common",
+            "Microsoft.VisualStudio.Setup.WMIProvider",
+            "Microsoft.VisualStudio.Setup.Configuration.Interop",
+            "Microsoft.VisualStudio.Setup.Configuration",
+            "Microsoft.VisualStudio.Extensibility.Container",
+            "Microsoft.VisualStudio.LanguageServer",
+            "Microsoft.VisualStudio.Platform.Terminal",
+            "Microsoft.VisualStudio.MefHosting",
+            "Microsoft.VisualStudio.Initializer",
+            "Microsoft.VisualStudio.ExtensionManager",
+            "Microsoft.VisualStudio.Platform.Editor",
+            "Microsoft.VisualStudio.MinShell.Targeted",
+            "Microsoft.VisualStudio.NativeImageSupport",
+            "Microsoft.VisualStudio.Devenv.Config",
+            "Microsoft.VisualStudio.MinShell.Resources.arm64",
+            "Microsoft.VisualStudio.MinShell.Auto",
+            "Microsoft.VisualStudio.MinShell.Auto.Resources",
+            "Microsoft.VisualStudio.Branding.Community"
+        ]
+    }
+]
diff --git a/test/fixtures/VS_2026_Insiders_workload.txt b/test/fixtures/VS_2026_Insiders_workload.txt
new file mode 100644
index 0000000000..4e20abb101
--- /dev/null
+++ b/test/fixtures/VS_2026_Insiders_workload.txt
@@ -0,0 +1,568 @@
+[
+    {
+        "path": "C:\\Program Files\\Microsoft Visual Studio\\18\\Insiders",
+        "version": "18.3.11206.111",
+        "packages": [
+            "Microsoft.VisualStudio.Product.Community",
+            "Microsoft.VisualStudio.PackageGroup.LiveShare.VSCore",
+            "Microsoft.VisualStudio.LiveShare.VSCore",
+            "Microsoft.VisualStudio.Workload.NativeDesktop",
+            "Microsoft.VisualStudio.Component.VC.ASAN",
+            "Microsoft.VisualCpp.ASAN.X86",
+            "Microsoft.VC.14.50.18.0.ASAN.X86.base",
+            "Microsoft.VC.14.50.18.0.ASAN.X64.base",
+            "Microsoft.VC.14.50.18.0.ASAN.Headers.base",
+            "Microsoft.VisualStudio.VC.IDE.Project.Factories",
+            "Microsoft.VisualStudio.Component.VC.TestAdapterForGoogleTest",
+            "Microsoft.VisualStudio.VC.Ide.TestAdapterForGoogleTest",
+            "Microsoft.VisualStudio.Component.VC.TestAdapterForBoostTest",
+            "Microsoft.VisualStudio.VC.Ide.TestAdapterForBoostTest",
+            "Microsoft.VisualStudio.Component.VC.CMake.Project",
+            "Microsoft.VisualStudio.ComponentGroup.WebToolsExtensions.CMake",
+            "Microsoft.VisualStudio.VC.CMake",
+            "Microsoft.VisualStudio.VC.CMake.Project",
+            "Microsoft.VisualStudio.VC.CMake.Client",
+            "Microsoft.VisualStudio.VC.ExternalBuildFramework",
+            "Microsoft.VisualStudio.Component.VC.DiagnosticTools",
+            "Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core",
+            "Microsoft.VisualStudio.PackageGroup.TestTools.Native",
+            "Microsoft.VisualStudio.Component.VC.Redist.14.Latest",
+            "Microsoft.VisualStudio.VC.Templates.UnitTest",
+            "Microsoft.VisualStudio.VC.UnitTest.Desktop.Build.Core",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.V1.CPP",
+            "Microsoft.VisualStudio.VC.Templates.UnitTest.Resources",
+            "Microsoft.VisualStudio.VC.Templates.Desktop",
+            "Microsoft.VisualStudio.Component.Graphics",
+            "Microsoft.VisualStudio.Graphics.Viewers",
+            "Microsoft.VisualStudio.Graphics.Viewers.Resources",
+            "Microsoft.VisualStudio.Component.VC.ATL.ARM64",
+            "Microsoft.VisualCpp.ATL.ARM64",
+            "Microsoft.VC.14.50.18.0.ATL.ARM64.base",
+            "Microsoft.VisualStudio.Component.VC.ATL",
+            "Microsoft.VisualStudio.VC.Ide.ATL",
+            "Microsoft.VisualStudio.VC.Ide.ATL.Resources",
+            "Microsoft.VisualCpp.ATL.X86",
+            "Microsoft.VC.14.50.18.0.ATL.X86.base",
+            "Microsoft.VisualCpp.ATL.X64",
+            "Microsoft.VC.14.50.18.0.ATL.X64.base",
+            "Microsoft.VC.14.50.18.0.Props.ATLMFC",
+            "Microsoft.VisualCpp.ATL.Source",
+            "Microsoft.VC.14.50.18.0.ATL.Source.base",
+            "Microsoft.VisualCpp.ATL.Headers",
+            "Microsoft.VC.14.50.18.0.ATL.Headers.base",
+            "Microsoft.VC.14.50.18.0.Servicing.ATL",
+            "Microsoft.VisualStudio.Component.VC.Tools.ARM64",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.ARM64.v145",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.ARM64",
+            "Microsoft.VS.VC.vcvars.arm64.Shortcuts",
+            "Microsoft.VisualCpp.CA.Ext.Hostx64.TargetARM64",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx64.TargetARM64.base",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx64.TargetARM64.Res.base",
+            "Microsoft.VisualCpp.CA.Ext.Hostx86.TargetARM64",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx86.TargetARM64.base",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx86.TargetARM64.Res.base",
+            "Microsoft.VisualCpp.CA.Ext.HostARM64.TargetARM64",
+            "Microsoft.VC.14.50.18.0.CA.Ext.HostARM64.TargetARM64.base",
+            "Microsoft.VC.14.50.18.0.CA.Ext.HostARM64.TargetARM64.Res.base",
+            "Microsoft.VisualCpp.Tools.Hostx86.Targetarm64",
+            "Microsoft.VC.14.50.18.0.Tools.Hostx86.Targetarm64.base",
+            "Microsoft.VC.14.50.18.0.Tools.HostX86.TargetARM64.Res.base",
+            "Microsoft.VisualCpp.Tools.HostARM64.TargetARM64",
+            "Microsoft.VC.14.50.18.0.Tools.HostARM64.TargetARM64.base",
+            "Microsoft.VC.14.50.18.0.Tools.HostARM64.TargetARM64.Res.base",
+            "Microsoft.VisualCpp.CRT.Redist.ARM64.OneCore.Desktop",
+            "Microsoft.VC.14.50.18.0.CRT.Redist.ARM64.OneCore.Desktop.base",
+            "Microsoft.VisualCpp.CRT.Redist.ARM64",
+            "Microsoft.VC.14.50.18.0.CRT.Redist.ARM64.base",
+            "Microsoft.VisualCpp.CRT.ARM64.OneCore.Desktop",
+            "Microsoft.VC.14.50.18.0.CRT.ARM64.OneCore.Desktop.base",
+            "Microsoft.VC.14.50.18.0.CRT.ARM64.OneCore.Desktop.debug.base",
+            "Microsoft.VisualCpp.CRT.ARM64.Store",
+            "Microsoft.VC.14.50.18.0.CRT.ARM64.Store.base",
+            "Microsoft.VisualCpp.CRT.ARM64.Desktop",
+            "Microsoft.VC.14.50.18.0.CRT.ARM64.Desktop.base",
+            "Microsoft.VC.14.50.18.0.CRT.ARM64.Desktop.debug.base",
+            "Microsoft.VisualStudio.PackageGroup.VC.Tools.x64.ARM64",
+            "Microsoft.VisualCpp.Tools.Core",
+            "Microsoft.VisualCpp.PGO.ARM64",
+            "Microsoft.VC.14.50.18.0.PGO.ARM64.base",
+            "Microsoft.VisualCpp.Premium.Tools.Hostx86.Targetarm64",
+            "Microsoft.VC.14.50.18.0.Premium.Tools.Hostx86.Targetarm64.base",
+            "Microsoft.VC.14.50.18.0.Prem.HostX86.TargetARM64.Res.base",
+            "Microsoft.VisualCpp.Premium.Tools.HostX64.TargetARM64",
+            "Microsoft.VC.14.50.18.0.Premium.Tools.HostX64.TargetARM64.base",
+            "Microsoft.VC.14.50.18.0.Prem.HostX64.TargetARM64.Res.base",
+            "Microsoft.VisualCpp.Premium.Tools.ARM64.Base",
+            "Microsoft.VC.14.50.18.0.Premium.Tools.ARM64.Base.base",
+            "Microsoft.VisualCpp.Tools.HostX64.TargetARM64",
+            "Microsoft.VC.14.50.18.0.Tools.HostX64.TargetARM64.base",
+            "Microsoft.VC.14.50.18.0.Props.ARM64",
+            "Microsoft.VC.14.50.18.0.Tools.HostX64.TargetARM64.Res.base",
+            "Microsoft.VisualStudio.Component.VC.Tools.ARM64EC",
+            "Microsoft.VisualStudio.Component.Windows11SDK.26100",
+            "Win11SDK_10.0.26100",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.ARM64EC.v145",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.ARM64EC",
+            "Microsoft.VisualCpp.CRT.ARM64EC.Store",
+            "Microsoft.VC.14.50.18.0.CRT.ARM64EC.Store.base",
+            "Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
+            "Microsoft.VisualCpp.CodeAnalysis.Extensions",
+            "Microsoft.VisualCpp.CA.Ext.HostARM64.Targetx64",
+            "Microsoft.VC.14.50.18.0.CA.Ext.HostARM64.Targetx64.base",
+            "Microsoft.VC.14.50.18.0.CA.Ext.HostARM64.Targetx64.Res.base",
+            "Microsoft.VisualCpp.CA.Ext.HostARM64.Targetx86",
+            "Microsoft.VC.14.50.18.0.CA.Ext.HostARM64.Targetx86.base",
+            "Microsoft.VC.14.50.18.0.CA.Ext.HostARM64.Targetx86.Res.base",
+            "Microsoft.VisualCpp.CA.Ext.Hostx86.Targetx64",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx86.Targetx64.base",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx86.Targetx64.Res.base",
+            "Microsoft.VisualCpp.CA.Ext.Hostx86.Targetx86",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx86.Targetx86.base",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx86.Targetx86.Res.base",
+            "Microsoft.VisualCpp.CA.Ext.Hostx64.Targetx64",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx64.Targetx64.base",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx64.Targetx64.Res.base",
+            "Microsoft.VisualCpp.CA.Ext.Hostx64.Targetx86",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx64.Targetx86.base",
+            "Microsoft.VC.14.50.18.0.Servicing.CAExtensions",
+            "Microsoft.VC.14.50.18.0.CA.Ext.Hostx64.Targetx86.Res.base",
+            "Microsoft.VisualCpp.Tools.HostX64.TargetX86",
+            "Microsoft.VC.14.50.18.0.Tools.HostX64.TargetX86.base",
+            "Microsoft.VC.14.50.18.0.Tools.HostX64.TargetX86.Res.base",
+            "Microsoft.VisualCpp.Tools.HostX64.TargetX64",
+            "Microsoft.VC.14.50.18.0.Tools.HostX64.TargetX64.base",
+            "Microsoft.VC.14.50.18.0.Tools.HostX64.TargetX64.Res.base",
+            "Microsoft.VisualCpp.Tools.HostARM64.TargetX86",
+            "Microsoft.VC.14.50.18.0.Tools.HostARM64.TargetX86.base",
+            "Microsoft.VisualCpp.RuntimeDebug.14",
+            "Microsoft.VC.14.50.18.0.Tools.HostARM64.TargetX86.Res.base",
+            "Microsoft.VisualCpp.Tools.HostARM64.TargetX64",
+            "Microsoft.VC.14.50.18.0.Tools.HostARM64.TargetX64.base",
+            "Microsoft.VisualCpp.RuntimeDebug.14.ARM64",
+            "Microsoft.VisualCpp.Redist.14.Latest",
+            "Microsoft.VisualCpp.Redist.14.Latest",
+            "Microsoft.VC.14.50.18.0.Tools.HostARM64.Targetx64.Res.base",
+            "Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX64",
+            "Microsoft.VC.14.50.18.0.Premium.Tools.HostX86.TargetX64.base",
+            "Microsoft.VC.14.50.18.0.Prem.Hostx86.Targetx64.Res.base",
+            "Microsoft.VisualCpp.Premium.Tools.HostX86.TargetX86",
+            "Microsoft.VC.14.50.18.0.Premium.Tools.HostX86.TargetX86.base",
+            "Microsoft.VC.14.50.18.0.Prem.HostX86.TargetX86.Res.base",
+            "Microsoft.VisualCpp.Premium.Tools.HostARM64.TargetX86",
+            "Microsoft.VC.14.50.18.0.Premium.Tools.HostARM64.TargetX86.base",
+            "Microsoft.VC.14.50.18.0.Prem.HostARM64.TargetX86.Res.base",
+            "Microsoft.VisualCpp.Premium.Tools.HostARM64.TargetX64",
+            "Microsoft.VC.14.50.18.0.Premium.Tools.HostARM64.TargetX64.base",
+            "Microsoft.VC.14.50.18.0.Prem.HostARM64.Targetx64.Res.base",
+            "Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX86",
+            "Microsoft.VC.14.50.18.0.Premium.Tools.HostX64.TargetX86.base",
+            "Microsoft.VC.14.50.18.0.Prem.HostX64.TargetX86.Res.base",
+            "Microsoft.VisualCpp.Premium.Tools.HostX64.TargetX64",
+            "Microsoft.VC.14.50.18.0.Premium.Tools.HostX64.TargetX64.base",
+            "Microsoft.VC.14.50.18.0.Prem.HostX64.TargetX64.Res.base",
+            "Microsoft.VisualCpp.PGO.X86",
+            "Microsoft.VC.14.50.18.0.PGO.X86.base",
+            "Microsoft.VisualCpp.PGO.X64",
+            "Microsoft.VC.14.50.18.0.PGO.X64.base",
+            "Microsoft.VisualCpp.PGO.Headers",
+            "Microsoft.VC.14.50.18.0.PGO.Headers.base",
+            "Microsoft.VisualCpp.CRT.x86.Store",
+            "Microsoft.VC.14.50.18.0.CRT.x86.Store.base",
+            "Microsoft.VisualCpp.CRT.x86.OneCore.Desktop",
+            "Microsoft.VC.14.50.18.0.CRT.x86.OneCore.Desktop.base",
+            "Microsoft.VisualCpp.CRT.x64.Store",
+            "Microsoft.VC.14.50.18.0.CRT.x64.Store.base",
+            "Microsoft.VisualCpp.CRT.x64.OneCore.Desktop",
+            "Microsoft.VC.14.50.18.0.CRT.x64.OneCore.Desktop.base",
+            "Microsoft.VisualCpp.CRT.Redist.x86.OneCore.Desktop",
+            "Microsoft.VC.14.50.18.0.CRT.Redist.x86.OneCore.Desktop.base",
+            "Microsoft.VisualCpp.CRT.Redist.x64.OneCore.Desktop",
+            "Microsoft.VC.14.50.18.0.CRT.Redist.x64.OneCore.Desktop.base",
+            "Microsoft.VisualStudio.PackageGroup.VC.Tools.x86",
+            "Microsoft.VisualCpp.Tools.Hostx86.Targetx64.Res",
+            "Microsoft.VisualCpp.Tools.HostX86.TargetX64",
+            "Microsoft.VC.14.50.18.0.Tools.HostX86.TargetX64.base",
+            "Microsoft.VC.14.50.18.0.Props.x64",
+            "Microsoft.VC.14.50.18.0.Tools.Hostx86.Targetx64.Res.base",
+            "Microsoft.VisualCpp.Tools.HostX86.TargetX86.Res",
+            "Microsoft.VisualCpp.Tools.HostX86.TargetX86",
+            "Microsoft.VC.14.50.18.0.Tools.HostX86.TargetX86.base",
+            "Microsoft.VC.14.50.18.0.Servicing.Compilers",
+            "Microsoft.VC.14.50.18.0.Props.x86",
+            "Microsoft.VC.14.50.18.0.Props",
+            "Microsoft.VC.14.50.18.0.Tools.HostX86.TargetX86.Res.base",
+            "Microsoft.VisualCpp.Tools.Core.Resources",
+            "Microsoft.VisualCpp.Tools.Core.x86",
+            "Microsoft.VC.14.50.18.0.Tools.Core.Props",
+            "Microsoft.VisualCpp.DIA.SDK",
+            "Microsoft.VisualCpp.Servicing.DIASDK",
+            "Microsoft.VisualCpp.CRT.x86.Desktop",
+            "Microsoft.VC.14.50.18.0.CRT.x86.Desktop.base",
+            "Microsoft.VisualCpp.CRT.x64.Desktop",
+            "Microsoft.VC.14.50.18.0.CRT.x64.Desktop.base",
+            "Microsoft.VisualCpp.CRT.Source",
+            "Microsoft.VC.14.50.18.0.CRT.Source.base",
+            "Microsoft.VisualCpp.CRT.Redist.X86",
+            "Microsoft.VC.14.50.18.0.CRT.Redist.X86.base",
+            "Microsoft.VisualCpp.CRT.Redist.X64",
+            "Microsoft.VisualCpp.CRT.Redist.Resources",
+            "Microsoft.VC.14.50.18.0.CRT.Redist.X64.base",
+            "Microsoft.VisualCpp.CRT.Headers",
+            "Microsoft.VC.14.50.18.0.CRT.Headers.base",
+            "Microsoft.VC.14.50.18.0.Servicing.CrtHeaders",
+            "Microsoft.VC.14.50.18.0.Servicing",
+            "Microsoft.VisualStudio.Component.VC.CoreIde",
+            "Microsoft.VisualStudio.VC.Ide.Pro",
+            "Microsoft.VisualStudio.VC.Ide.Pro.Resources",
+            "Microsoft.VisualStudio.VC.Templates.General",
+            "Microsoft.VisualStudio.VC.Templates.General.Resources",
+            "Microsoft.VisualStudio.VC.Items.Pro",
+            "Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Reduced",
+            "Microsoft.VisualStudio.VC.Ide.x64",
+            "Microsoft.VisualStudio.PackageGroup.VC.CoreIDE.Express",
+            "Microsoft.VisualStudio.VC.vcvars",
+            "Microsoft.VS.VC.vcvars.x86.Shortcuts",
+            "Microsoft.VS.VC.vcvars.x64.Shortcuts",
+            "Microsoft.VS.VC.vcvars.arm64_x64.Shortcuts",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.X64.v145",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.X64",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.ARM.v145",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.ARM",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.x86.v145",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.X86",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.Base",
+            "Microsoft.VisualStudio.VC.MSBuild.v180.Base.Resources",
+            "Microsoft.VisualStudio.VC.Ide.WinXPlus",
+            "Microsoft.VisualStudio.VC.Ide.Dskx",
+            "Microsoft.VisualStudio.VC.Ide.Dskx.Resources",
+            "Microsoft.VisualStudio.VC.Ide.Base",
+            "Microsoft.VisualStudio.VC.Ide.LanguageService",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.Scripts",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.PythonDistro",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.10",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.9",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.8",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.7",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.6",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.5",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.4",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.3",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.2",
+            "Microsoft.VisualStudio.VC.Ide.SecurityIssueAnalysis.3rdPartyLibs.1",
+            "Microsoft.VisualStudio.VC.Ide.VCPkgDatabase",
+            "Microsoft.VisualStudio.VC.Ide.Core",
+            "Microsoft.VisualStudio.VC.Ide.ProjectSystem",
+            "Microsoft.VisualStudio.VC.Ide.ProjectSystem.Resources",
+            "Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine",
+            "Microsoft.VisualStudio.VC.Ide.Core.VCProjectEngine.Resources",
+            "Microsoft.VisualStudio.VC.Ide.LanguageService.Resources",
+            "Microsoft.VisualStudio.VC.Llvm.Base",
+            "Microsoft.VisualStudio.VC.Ide.Base.Resources",
+            "Microsoft.Net.PackageGroup.4.8.1.Redist",
+            "Microsoft.VisualStudio.Component.IntelliCode",
+            "Microsoft.VisualStudio.IntelliCode.CSharp",
+            "Microsoft.VisualStudio.IntelliCode",
+            "Component.Microsoft.VisualStudio.LiveShare.2026",
+            "Microsoft.VisualStudio.Component.Debugger.JustInTime",
+            "Microsoft.VisualStudio.Debugger.ImmersiveActivateHelper.Msi",
+            "Microsoft.VisualStudio.Debugger.JustInTime",
+            "Microsoft.VisualStudio.Debugger.JustInTime.Msi",
+            "Microsoft.VisualStudio.LiveShare.2026",
+            "Microsoft.Icecap.Analysis",
+            "Microsoft.Icecap.Analysis.Resources",
+            "Microsoft.Icecap.Analysis.Resources.Targeted",
+            "Microsoft.Icecap.Collection.Msi",
+            "Microsoft.Icecap.Collection.Msi.Targeted",
+            "Microsoft.Icecap.Collection.Msi.Resources",
+            "Microsoft.Icecap.Collection.Msi.Resources.Targeted",
+            "Microsoft.DiagnosticsHub.Instrumentation",
+            "Microsoft.DiagnosticsHub.Instrumentation.Targeted",
+            "Microsoft.DiagnosticsHub.CpuSampling",
+            "Microsoft.DiagnosticsHub.CpuSampling.Targeted",
+            "Microsoft.PackageGroup.DiagnosticsHub.Platform",
+            "Microsoft.VisualStudio.InstrumentationEngine.ARM64",
+            "Microsoft.VisualStudio.InstrumentationEngine",
+            "Microsoft.DiagnosticsHub.Runtime.ExternalDependencies",
+            "SQLiteCore",
+            "SQLiteCore.Targeted",
+            "Microsoft.DiagnosticsHub.Runtime.ExternalDependencies.Targeted",
+            "Microsoft.DiagnosticsHub.Runtime",
+            "Microsoft.DiagnosticsHub.Runtime.Targeted",
+            "Microsoft.DiagnosticsHub.Collection.ExternalDependencies.arm64",
+            "Microsoft.DiagnosticsHub.Collection",
+            "Microsoft.DiagnosticsHub.Collection.Service",
+            "Microsoft.VisualStudio.VC.Ide.MDD",
+            "Microsoft.VisualStudio.VC.Ide.Linux.ConnectionManager",
+            "Microsoft.VisualStudio.VisualC.Utilities",
+            "Microsoft.VisualStudio.VisualC.Utilities.Resources",
+            "Microsoft.VisualStudio.VC.Ide.Linux.ConnectionManager.Resources",
+            "Microsoft.VisualStudio.VC.Ide.ResourceEditor",
+            "Microsoft.VisualStudio.VC.Ide.ResourceEditor.Resources",
+            "Microsoft.VisualStudio.PackageGroup.TestTools.Core",
+            "Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.V2.CLI",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.V2.CLI",
+            "Microsoft.VisualStudio.TestTools.Pex.Common",
+            "Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.V1.CLI",
+            "Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.Legacy",
+            "Microsoft.VisualStudio.PackageGroup.MinShell.Interop",
+            "Microsoft.VisualStudio.TestTools.TP.Legacy.Tips.Msi",
+            "Microsoft.VisualStudio.TestTools.TP.Legacy.Tips.Common",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Tips",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Tips.Resources",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.TestSettings",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Professional",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Common",
+            "Microsoft.VisualStudio.TestTools.TP.Legacy.Common.Res",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Core",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Core.Resources",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.Legacy.Agent",
+            "Microsoft.VisualStudio.PackageGroup.TestTools.TestPlatform.IDE",
+            "Microsoft.VisualStudio.Cache.Service",
+            "Microsoft.VisualStudio.TestTools.TestWIExtension",
+            "Microsoft.VisualStudio.TestTools.TestWIExtension.Res",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.V1.CLI",
+            "Microsoft.VisualStudio.TestTools.TP.V1.CLI.Resources",
+            "Microsoft.VisualStudio.TestTools.TestPlatform.IDE",
+            "Microsoft.VisualStudio.PackageGroup.TestTools.CodeCoverage",
+            "Microsoft.VisualStudio.PackageGroup.TestTools.DataCollectors",
+            "Microsoft.VisualStudio.Component.NuGet",
+            "Microsoft.CredentialProvider",
+            "Microsoft.VisualStudio.NuGet.Licenses",
+            "Microsoft.VisualStudio.Component.TextTemplating",
+            "Microsoft.VisualStudio.TextTemplating.MSBuild",
+            "Microsoft.VisualStudio.TextTemplating.Integration",
+            "Microsoft.VisualStudio.TextTemplating.Core",
+            "Microsoft.VisualStudio.TextTemplating.Integration.Resources",
+            "Microsoft.VisualCpp.CRT.ClickOnce.Msi",
+            "Microsoft.VisualStudio.Component.Roslyn.LanguageServices",
+            "Microsoft.VisualStudio.InteractiveWindow",
+            "Microsoft.DiaSymReader.Native",
+            "Microsoft.VisualCpp.Redist.14",
+            "Microsoft.VisualCpp.Redist.14",
+            "Microsoft.VisualCpp.Servicing.Redist",
+            "Microsoft.VisualStudio.PackageGroup.StaticAnalysis",
+            "Microsoft.VisualStudio.StaticAnalysis.IDE",
+            "Microsoft.VisualStudio.StaticAnalysis.IDE.Resources",
+            "Microsoft.VisualStudio.StaticAnalysis.FxCop.Resources",
+            "Microsoft.VisualStudio.StaticAnalysis.auxil",
+            "Microsoft.VisualStudio.StaticAnalysis.auxil.Resources",
+            "Roslyn.VisualStudio.Setup.ServiceHub",
+            "Microsoft.Component.MSBuild",
+            "Microsoft.NuGet.Build.Tasks.Setup",
+            "Microsoft.VisualStudio.Component.Roslyn.Compiler",
+            "Microsoft.CodeAnalysis.Compilers",
+            "Microsoft.VisualStudio.Component.JavaScript.TypeScript",
+            "Microsoft.VisualStudio.JavaScript.ProjectSystem",
+            "Microsoft.VisualStudio.ComponentGroup.WebToolsExtensions",
+            "Microsoft.VisualStudio.ProTools",
+            "sqlsysclrtypes",
+            "SQLCommon",
+            "Microsoft.VisualStudio.ProTools.Resources",
+            "Microsoft.VisualStudio.Web.Scaffolding",
+            "Microsoft.VisualStudio.WebToolsExtensions",
+            "Microsoft.VisualStudio.ConnectedServices.Core",
+            "Microsoft.VisualStudio.WebTools",
+            "Microsoft.VisualStudio.WebToolsExtensions.MSBuild",
+            "Microsoft.VisualStudio.WebTools.Resources",
+            "Microsoft.VisualStudio.WebTools.WSP.FSA",
+            "Microsoft.VisualStudio.WebTools.WSP.FSA.Resources",
+            "Microsoft.VisualStudio.PackageGroup.Debugger.Script",
+            "Microsoft.VisualStudio.Component.TypeScript.TSServer",
+            "Microsoft.VisualStudio.Package.TypeScript.TSServer",
+            "Microsoft.VisualStudio.PackageGroup.JavaScript.Language",
+            "Microsoft.VisualStudio.Package.NodeJs",
+            "TypeScript.Build",
+            "TypeScript.LanguageService",
+            "TypeScript.Tools",
+            "Microsoft.VisualStudio.PackageGroup.Community",
+            "Microsoft.VisualStudio.Community.VB.x86",
+            "Microsoft.VisualStudio.Community.VB.x64",
+            "Microsoft.VisualStudio.PackageGroup.Core",
+            "Microsoft.VisualStudio.CodeSense.Community",
+            "Microsoft.VisualStudio.TestTools.TeamFoundationClient",
+            "Microsoft.VisualStudio.PackageGroup.Debugger.Core",
+            "Microsoft.VisualStudio.Debugger.BrokeredServices",
+            "Microsoft.VisualStudio.Debugger.VSCodeDebuggerHost",
+            "Microsoft.VisualStudio.Debugger.AzureAttach",
+            "Microsoft.VisualStudio.Web.Azure.Common",
+            "Microsoft.WebTools.Shared",
+            "Microsoft.WebTools.DotNet.Core.ItemTemplates",
+            "Microsoft.VisualStudio.PackageGroup.Debugger.TimeTravel.Replay",
+            "Microsoft.VisualStudio.VC.Ide.Debugger",
+            "Microsoft.VisualStudio.VC.Ide.Debugger.Concord",
+            "Microsoft.VisualStudio.VC.Ide.Debugger.Concord.Resources",
+            "Microsoft.VisualStudio.VC.Ide.Debugger.Resources",
+            "Microsoft.VisualStudio.VC.Ide.Common",
+            "Microsoft.VisualStudio.VC.Ide.Common.Resources",
+            "Microsoft.VisualStudio.Debugger.CollectionAgents",
+            "Microsoft.VisualStudio.Debugger.Parallel",
+            "Microsoft.VisualStudio.Debugger.Parallel.Resources",
+            "Microsoft.VisualStudio.Debugger.Managed",
+            "Microsoft.CodeAnalysis.ExpressionEvaluator",
+            "Microsoft.CodeAnalysis.VisualStudio.Setup",
+            "Microsoft.VisualStudio.Debugger.Concord.Managed",
+            "Microsoft.VisualStudio.Debugger.Concord.Managed.Resources",
+            "Microsoft.VisualStudio.Debugger.Managed.Resources",
+            "Microsoft.VisualStudio.Debugger.TargetComposition",
+            "Microsoft.VisualStudio.Debugger.TargetComposition.Remote.arm64",
+            "Microsoft.VisualStudio.Debugger.TargetComposition.Remote",
+            "Microsoft.VisualStudio.Debugger.TargetComposition.Remote",
+            "Microsoft.VisualStudio.Debugger.Remote",
+            "Microsoft.VisualStudio.Debugger.Concord.Remote",
+            "Microsoft.VisualStudio.Debugger.Concord.Remote.Resources",
+            "Microsoft.VisualStudio.Debugger.Remote",
+            "Microsoft.VisualStudio.Debugger.Remote.ARM64",
+            "Microsoft.VisualStudio.Debugger.Concord.Remote.ARM64",
+            "Microsoft.VisualStudio.Debugger.Concord.Remote.Resources.ARM64",
+            "Microsoft.VisualStudio.Debugger.Remote.Resources.ARM",
+            "Microsoft.VisualStudio.Debugger.Remote.Resources.ARM64",
+            "Microsoft.VisualStudio.Debugger.Concord.Remote",
+            "Microsoft.VisualStudio.Debugger.Concord.Remote.Resources",
+            "Microsoft.VisualStudio.Debugger.Remote.Resources",
+            "Microsoft.VisualStudio.Debugger.Remote.Resources",
+            "Microsoft.VisualStudio.Debugger",
+            "Microsoft.VisualStudio.VC.MSVCDis",
+            "Microsoft.IntelliTrace.DiagnosticsHub",
+            "Microsoft.VisualStudio.Debugger.Concord",
+            "Microsoft.VisualStudio.Debugger.Concord.Resources",
+            "Microsoft.VisualStudio.Debugger.Resources",
+            "Microsoft.VisualStudio.Debugger.Package.DiagHub.Client",
+            "Microsoft.VisualStudio.Debugger.Remote.DiagnosticsHub.Client",
+            "Microsoft.VisualStudio.Debugger.Remote.DiagnosticsHub.Client",
+            "Microsoft.VisualStudio.Debugger.Remote.DiagnosticsHub.Client",
+            "Microsoft.PackageGroup.ClientDiagnostics",
+            "Microsoft.VisualStudio.AppResponsiveness",
+            "Microsoft.VisualStudio.AppResponsiveness.Targeted",
+            "Microsoft.VisualStudio.AppResponsiveness.Resources",
+            "Microsoft.VisualStudio.ClientDiagnostics",
+            "Microsoft.VisualStudio.ClientDiagnostics.Targeted",
+            "Microsoft.VisualStudio.ClientDiagnostics.Resources",
+            "Microsoft.VisualStudio.PackageGroup.CommunityCore",
+            "Microsoft.VisualStudio.ProjectSystem.Full",
+            "Microsoft.VisualStudio.LiveShareApi",
+            "Microsoft.VisualStudio.ProjectSystem.Query",
+            "Microsoft.VisualStudio.ProjectSystem",
+            "Microsoft.VisualStudio.Community.x86",
+            "Microsoft.VisualStudio.Community.x64",
+            "Microsoft.VisualStudio.Community.Msi.Resources",
+            "Microsoft.VisualStudio.Community.Msi",
+            "Microsoft.VisualStudio.Community.Shared.Msi",
+            "Microsoft.VisualStudio.Devenv.Msi",
+            "Microsoft.VisualStudio.Devenv.Shared.Msi",
+            "Microsoft.VisualStudio.MinShell.Interop.Msi",
+            "Microsoft.VisualStudio.MinShell.Interop.Shared.Msi",
+            "Microsoft.VisualStudio.Editors",
+            "Microsoft.VisualStudio.Workload.CoreEditor",
+            "Microsoft.VisualStudio.Component.CoreEditor",
+            "Microsoft.VisualStudio.PackageGroup.CoreEditor",
+            "Microsoft.WebView2",
+            "Microsoft.VisualStudio.ScriptedHost",
+            "Microsoft.VisualStudio.ScriptedHost.Targeted",
+            "Microsoft.VisualCpp.Tools.Common.UtilsPrereq",
+            "Microsoft.VisualCpp.Tools.Common.Utils",
+            "Microsoft.VisualCpp.Tools.Common.Utils.Resources",
+            "Microsoft.VisualStudio.PackageGroup.VsDevCmd",
+            "Microsoft.VisualStudio.VsDevCmd.Ext.NetFxSdk",
+            "Microsoft.VisualStudio.VsDevCmd.Core.WinSdk",
+            "Microsoft.VisualStudio.VsDevCmd.Core.DotNet",
+            "Microsoft.VisualStudio.VC.DevCmd",
+            "Microsoft.VisualStudio.VC.DevCmd.Resources",
+            "Microsoft.VisualStudio.VirtualTree",
+            "Microsoft.DiaSymReader",
+            "Microsoft.Build.Dependencies",
+            "Microsoft.Build.FileTracker.Msi",
+            "Microsoft.Build",
+            "Microsoft.VisualStudio.PackageGroup.NuGet",
+            "Microsoft.DataAI.NuGetRecommender",
+            "Microsoft.VisualStudio.NuGet.Core",
+            "Microsoft.Build.Arm64",
+            "Microsoft.Build.UnGAC",
+            "Microsoft.VisualStudio.TextMateGrammars",
+            "Microsoft.VisualStudio.Platform.Markdown",
+            "Microsoft.VisualStudio.Platform.CrossRepositorySearch",
+            "Microsoft.VisualStudio.PackageGroup.TeamExplorer.Common",
+            "Microsoft.VisualStudio.TeamExplorer",
+            "Microsoft.VisualStudio.PackageGroup.ServiceHub",
+            "Microsoft.ServiceHub.Node",
+            "Microsoft.ServiceHub.Managed",
+            "Microsoft.ServiceHub.arm64",
+            "Microsoft.VisualStudio.ProjectServices",
+            "Microsoft.VisualStudio.OpenFolder.VSIX",
+            "Microsoft.VisualStudio.FileHandler.Msi",
+            "Microsoft.VisualStudio.FileHandler.Msi",
+            "Microsoft.VisualStudio.PackageGroup.MinShell",
+            "Microsoft.VisualStudio.MinShell.Msi",
+            "Microsoft.VisualStudio.MinShell.Shared.Msi",
+            "Microsoft.VisualStudio.MinShell.Msi.Resources",
+            "Microsoft.VisualStudio.MinShell.Interop",
+            "CoreEditorFonts",
+            "Microsoft.VisualStudio.Log",
+            "Microsoft.VisualStudio.Log.Targeted",
+            "Microsoft.VisualStudio.Log.Resources",
+            "Microsoft.VisualStudio.Finalizer",
+            "Microsoft.VisualStudio.Devenv",
+            "Microsoft.VisualStudio.Devenv.Resources",
+            "Microsoft.VisualStudio.CoreEditor",
+            "Microsoft.VisualStudio.Navigation.RichCodeNav",
+            "Microsoft.VisualStudio.Platform.NavigateTo",
+            "Microsoft.VisualStudio.Connected",
+            "SQLitePCLRaw",
+            "SQLitePCLRaw.Targeted",
+            "Microsoft.VisualStudio.Connected.Auto",
+            "Microsoft.VisualStudio.Connected.Auto.Resources",
+            "Microsoft.VisualStudio.AzureSDK",
+            "Microsoft.VisualStudio.PerfLib",
+            "Microsoft.VisualStudio.Connected.Resources",
+            "Microsoft.Net.PackageGroup.4.8.Redist",
+            "Microsoft.VisualStudio.PackageGroup.Progression",
+            "Microsoft.VisualStudio.PerformanceProvider",
+            "Microsoft.VisualStudio.GraphModel",
+            "Microsoft.VisualStudio.GraphProvider",
+            "Microsoft.VisualStudio.Community.VB.Targeted",
+            "Microsoft.VisualStudio.Community.VB.Neutral",
+            "Microsoft.VisualStudio.Community.CSharp.Targeted",
+            "Microsoft.VisualStudio.Community.CSharp.Neutral",
+            "Microsoft.VisualStudio.Community.ProductArch.TargetedExtra",
+            "Microsoft.VisualStudio.Community.ProductArch.Targeted",
+            "Microsoft.VisualStudio.Community.ProductArch.NeutralExtra",
+            "Microsoft.DiaSymReader.PortablePdb",
+            "Microsoft.IntelliTrace.CollectorCab",
+            "Microsoft.VisualStudio.Community.VB.Resources.Targeted",
+            "Microsoft.VisualStudio.Community.VB.Resources.Neutral",
+            "Microsoft.VisualStudio.Community.CSharp.Resources.Targeted",
+            "Microsoft.VisualStudio.Community.CSharp.Resources.Neutral",
+            "Microsoft.VisualStudio.Community.ProductArch.Resources.Targeted",
+            "Microsoft.VisualStudio.Community.ProductArch.Resources.NeutralExtra",
+            "Microsoft.VisualStudio.Net.Eula.Resources",
+            "Microsoft.VisualStudio.Community.ProductArch.Resources.Neutral",
+            "Microsoft.VisualStudio.WebSiteProject.DTE",
+            "Microsoft.VisualStudio.Diagnostics.AspNetHelper",
+            "Microsoft.VisualStudio.Diagnostics.AspNetHelper.Standard",
+            "Microsoft.MSHtml",
+            "Microsoft.VisualStudio.Platform.CallHierarchy",
+            "Microsoft.VisualStudio.Community.ProductArch.Neutral",
+            "Microsoft.VisualStudio.MinShell",
+            "Microsoft.VisualStudio.VsWebProtocolSelector.Msi",
+            "Microsoft.Net.8.0.WindowsDesktop.Runtime",
+            "Microsoft.Net.8.0.Runtime",
+            "Microsoft.VisualStudio.PackageGroup.Setup.Common",
+            "Microsoft.VisualStudio.Setup.WMIProvider",
+            "Microsoft.VisualStudio.Setup.Configuration.Interop",
+            "Microsoft.VisualStudio.Setup.Configuration",
+            "Microsoft.VisualStudio.Extensibility.Container",
+            "Microsoft.VisualStudio.LanguageServer",
+            "Microsoft.VisualStudio.Platform.Terminal",
+            "Microsoft.VisualStudio.MefHosting",
+            "Microsoft.VisualStudio.Initializer",
+            "Microsoft.VisualStudio.ExtensionManager",
+            "Microsoft.VisualStudio.Platform.Editor",
+            "Microsoft.VisualStudio.MinShell.Targeted",
+            "Microsoft.VisualStudio.NativeImageSupport",
+            "Microsoft.VisualStudio.Devenv.Config",
+            "Microsoft.VisualStudio.MinShell.Resources.arm64",
+            "Microsoft.VisualStudio.MinShell.Auto",
+            "Microsoft.VisualStudio.MinShell.Auto.Resources",
+            "Microsoft.VisualStudio.Branding.Community"
+        ]
+    }
+]
diff --git a/test/test-find-visualstudio.js b/test/test-find-visualstudio.js
index e53b6678ed..5f1bbe768c 100644
--- a/test/test-find-visualstudio.js
+++ b/test/test-find-visualstudio.js
@@ -79,7 +79,7 @@ describe('find-visualstudio', function () {
       const file = path.join(__dirname, 'fixtures', 'VS_2017_Unusable.txt')
       const data = fs.readFileSync(file)
       const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(parsedData, [2019, 2022])
+      return finder.processData(parsedData, [2019, 2022, 2026])
     }
     finder.findVisualStudio2017 = async () => {
       const file = path.join(__dirname, 'fixtures', 'VS_2017_Unusable.txt')
@@ -221,7 +221,7 @@ describe('find-visualstudio', function () {
         'VS_2017_BuildTools_minimal.txt')
       const data = fs.readFileSync(file)
       const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(parsedData, [2019, 2022])
+      return finder.processData(parsedData, [2019, 2022, 2026])
     }
     finder.findVisualStudio2017 = async () => {
       const file = path.join(__dirname, 'fixtures',
@@ -256,7 +256,7 @@ describe('find-visualstudio', function () {
         'VS_2017_Community_workload.txt')
       const data = fs.readFileSync(file)
       const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(parsedData, [2019, 2022])
+      return finder.processData(parsedData, [2019, 2022, 2026])
     }
     finder.findVisualStudio2017 = async () => {
       const file = path.join(__dirname, 'fixtures',
@@ -290,7 +290,7 @@ describe('find-visualstudio', function () {
       const file = path.join(__dirname, 'fixtures', 'VS_2017_Express.txt')
       const data = fs.readFileSync(file)
       const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(parsedData, [2019, 2022])
+      return finder.processData(parsedData, [2019, 2022, 2026])
     }
     finder.findVisualStudio2017 = async () => {
       const file = path.join(__dirname, 'fixtures', 'VS_2017_Express.txt')
@@ -324,7 +324,7 @@ describe('find-visualstudio', function () {
         'VS_2019_Preview.txt')
       const data = fs.readFileSync(file)
       const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(parsedData, [2019, 2022])
+      return finder.processData(parsedData, [2019, 2022, 2026])
     }
     finder.findVisualStudio2017 = async () => {
       const file = path.join(__dirname, 'fixtures',
@@ -359,7 +359,7 @@ describe('find-visualstudio', function () {
         'VS_2019_BuildTools_minimal.txt')
       const data = fs.readFileSync(file)
       const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(parsedData, [2019, 2022])
+      return finder.processData(parsedData, [2019, 2022, 2026])
     }
     finder.findVisualStudio2017 = async () => {
       const file = path.join(__dirname, 'fixtures',
@@ -394,7 +394,7 @@ describe('find-visualstudio', function () {
         'VS_2019_Community_workload.txt')
       const data = fs.readFileSync(file)
       const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(parsedData, [2019, 2022])
+      return finder.processData(parsedData, [2019, 2022, 2026])
     }
     finder.findVisualStudio2017 = async () => {
       const file = path.join(__dirname, 'fixtures',
@@ -438,7 +438,7 @@ describe('find-visualstudio', function () {
         'VS_2022_Community_workload.txt')
       const data = fs.readFileSync(file)
       const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(parsedData, [2019, 2022])
+      return finder.processData(parsedData, [2019, 2022, 2026])
     }
     finder.findVisualStudio2017 = async () => {
       const file = path.join(__dirname, 'fixtures',
@@ -462,6 +462,94 @@ describe('find-visualstudio', function () {
     })
   })
 
+  it('VS2026 Preview with C++ workload', async function () {
+    const msBuildPath = process.arch === 'arm64'
+      ? 'C:\\Program Files\\Microsoft Visual Studio\\18\\' +
+        'Insiders\\MSBuild\\Current\\Bin\\arm64\\MSBuild.exe'
+      : 'C:\\Program Files\\Microsoft Visual Studio\\18\\' +
+        'Insiders\\MSBuild\\Current\\Bin\\MSBuild.exe'
+
+    const finder = new TestVisualStudioFinder(semverV1, null)
+
+    poison(finder, 'regSearchKeys')
+    finder.msBuildPathExists = (path) => {
+      return true
+    }
+    finder.findNewVSUsingSetupModule = async () => null
+    finder.findVisualStudio2019OrNewer = async () => {
+      const file = path.join(__dirname, 'fixtures',
+        'VS_2026_Insiders_workload.txt')
+      const data = fs.readFileSync(file)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2019, 2022, 2026])
+    }
+    finder.findVisualStudio2017 = async () => {
+      const file = path.join(__dirname, 'fixtures',
+        'VS_2026_Insiders_workload.txt')
+      const data = fs.readFileSync(file)
+      const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
+      return finder.processData(parsedData, [2017])
+    }
+    const { err, info } = await finder.findVisualStudio()
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info, {
+      msBuild: msBuildPath,
+      path:
+        'C:\\Program Files\\Microsoft Visual Studio\\18\\Insiders',
+      sdk: '10.0.26100.0',
+      toolset: 'v145',
+      version: '18.3.11206.111',
+      versionMajor: 18,
+      versionMinor: 3,
+      versionYear: 2026
+    })
+  })
+
+  it('VS2026 Release with C++ workload', async function () {
+    const msBuildPath = process.arch === 'arm64'
+      ? 'C:\\Program Files\\Microsoft Visual Studio\\2026\\' +
+        'Community\\MSBuild\\Current\\Bin\\arm64\\MSBuild.exe'
+      : 'C:\\Program Files\\Microsoft Visual Studio\\2026\\' +
+        'Community\\MSBuild\\Current\\Bin\\MSBuild.exe'
+
+    const finder = new TestVisualStudioFinder(semverV1, null)
+
+    poison(finder, 'regSearchKeys')
+    finder.msBuildPathExists = (path) => {
+      return true
+    }
+    finder.findNewVSUsingSetupModule = async () => null
+    finder.findVisualStudio2019OrNewer = async () => {
+      // Create mock data for release version with 2026 path
+      const releaseData = [{
+        path: 'C:\\Program Files\\Microsoft Visual Studio\\2026\\Community',
+        version: '18.0.1000.100',
+        packages: [
+          'Microsoft.VisualStudio.Product.Community',
+          'Microsoft.VisualStudio.Workload.NativeDesktop',
+          'Microsoft.VisualStudio.Component.VC.Tools.x86.x64',
+          'Microsoft.VisualStudio.Component.Windows11SDK.26100'
+        ]
+      }]
+      return finder.processData(releaseData, [2019, 2022, 2026])
+    }
+    finder.findVisualStudio2017 = async () => null
+
+    const { err, info } = await finder.findVisualStudio()
+    assert.strictEqual(err, null)
+    assert.deepStrictEqual(info, {
+      msBuild: msBuildPath,
+      path:
+        'C:\\Program Files\\Microsoft Visual Studio\\2026\\Community',
+      sdk: '10.0.26100.0',
+      toolset: 'v145',
+      version: '18.0.1000.100',
+      versionMajor: 18,
+      versionMinor: 0,
+      versionYear: 2026
+    })
+  })
+
   it('VS2022 Build Tools with ARM64 MSVC only', async function () {
     if (process.arch !== 'arm64') {
       return
@@ -483,7 +571,7 @@ describe('find-visualstudio', function () {
         'VS_2022_BuildTools_arm64_only.txt')
       const data = fs.readFileSync(file)
       const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(parsedData, [2019, 2022])
+      return finder.processData(parsedData, [2019, 2022, 2026])
     }
     const { err, info } = await finder.findVisualStudio()
     assert.strictEqual(err, null)
@@ -613,7 +701,7 @@ describe('find-visualstudio', function () {
         'VS_2022_Community_workload.txt')))
       const data = JSON.stringify(data0.concat(data1, data2, data3))
       const parsedData = finder.parseData(null, data, '', { checkIsArray: true })
-      return finder.processData(parsedData, [2019, 2022])
+      return finder.processData(parsedData, [2019, 2022, 2026])
     }
     finder.regSearchKeys = async (keys, value, addOpts) => {
       for (let i = 0; i < keys.length; ++i) {

From 4cb46a2bc0a51b1390e6e26199d1d400968a6263 Mon Sep 17 00:00:00 2001
From: Luke Karrys 
Date: Wed, 12 Nov 2025 13:39:27 -0700
Subject: [PATCH 511/551] Revert "build(deps): bump env-paths from 2.2.1 to
 3.0.0 (#3235)" (#3241)

This reverts commit 5fffb2ffee304cc898fdea7a0cd9e41d54c53839.
---
 bin/node-gyp.js | 2 +-
 package.json    | 2 +-
 test/common.js  | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/bin/node-gyp.js b/bin/node-gyp.js
index c08f437a05..f8317b47b3 100755
--- a/bin/node-gyp.js
+++ b/bin/node-gyp.js
@@ -4,7 +4,7 @@
 
 process.title = 'node-gyp'
 
-const { default: envPaths } = require('env-paths')
+const envPaths = require('env-paths')
 const gyp = require('../')
 const log = require('../lib/log')
 const os = require('os')
diff --git a/package.json b/package.json
index ba483eae04..b574758c85 100644
--- a/package.json
+++ b/package.json
@@ -22,7 +22,7 @@
   "bin": "./bin/node-gyp.js",
   "main": "./lib/node-gyp.js",
   "dependencies": {
-    "env-paths": "^3.0.0",
+    "env-paths": "^2.2.0",
     "exponential-backoff": "^3.1.1",
     "graceful-fs": "^4.2.6",
     "make-fetch-happen": "^15.0.0",
diff --git a/test/common.js b/test/common.js
index 66d8d0d162..489502d4dd 100644
--- a/test/common.js
+++ b/test/common.js
@@ -1,4 +1,4 @@
-const { default: envPaths } = require('env-paths')
+const envPaths = require('env-paths')
 const semver = require('semver')
 
 module.exports.devDir = envPaths('node-gyp', { suffix: '' }).cache

From 98307b03a1381cf6697d80081b6cd05a47e412bc Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Wed, 12 Nov 2025 20:36:41 +0000
Subject: [PATCH 512/551] chore(main): release 12.1.0

---
 .release-please-manifest.json | 2 +-
 CHANGELOG.md                  | 8 ++++++++
 package.json                  | 2 +-
 3 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
index a3ff6f9b0c..4899c67643 100644
--- a/.release-please-manifest.json
+++ b/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "12.0.0"
+    ".": "12.1.0"
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 65f87f39d8..2bdec2cc6b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,13 @@
 # Changelog
 
+## [12.1.0](https://github.com/nodejs/node-gyp/compare/v12.0.0...v12.1.0) (2025-11-12)
+
+
+### Features
+
+* Add support for Visual Studio 2026 (18.x) ([69e5fd2](https://github.com/nodejs/node-gyp/commit/69e5fd2c98ac83dad5200a47515b301ccd80d2d3))
+* Support for Visual Studio 2026 (18.x) ([69e5fd2](https://github.com/nodejs/node-gyp/commit/69e5fd2c98ac83dad5200a47515b301ccd80d2d3))
+
 ## [12.0.0](https://github.com/nodejs/node-gyp/compare/v11.5.0...v12.0.0) (2025-11-10)
 
 
diff --git a/package.json b/package.json
index b574758c85..ae60687844 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "12.0.0",
+  "version": "12.1.0",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From 858bb26aead10f25dd6c6da61757572b5e14188e Mon Sep 17 00:00:00 2001
From: Edge-Seven <143301646+Edge-Seven@users.noreply.github.com>
Date: Tue, 18 Nov 2025 15:24:38 +0700
Subject: [PATCH 513/551] Fix typos in some files (#3245)

Co-authored-by: khanhkhanhlele 
---
 lib/install.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/install.js b/lib/install.js
index ee4adb1e67..3580cdd003 100644
--- a/lib/install.js
+++ b/lib/install.js
@@ -198,7 +198,7 @@ async function install (gyp, argv) {
     }
 
     // download the tarball and extract!
-    // Ommited on Windows if only new node.lib is required
+    // Omitted on Windows if only new node.lib is required
 
     // there can be file errors from tar if parallel installs
     // are happening (not uncommon with multiple native modules) so

From db5385c5467e5bfb914b9954f0313c46f1f4e10d Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Fri, 21 Nov 2025 16:07:20 +0100
Subject: [PATCH 514/551] build(deps): bump actions/checkout from 5 to 6
 (#3248)

Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 .github/workflows/release-please.yml  |  2 +-
 .github/workflows/tests.yml           | 10 +++++-----
 .github/workflows/update-gyp-next.yml |  2 +-
 .github/workflows/visual-studio.yml   |  2 +-
 4 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml
index 564829cfb8..70050b9a04 100644
--- a/.github/workflows/release-please.yml
+++ b/.github/workflows/release-please.yml
@@ -36,7 +36,7 @@ jobs:
       contents: read
       id-token: write # to generate npm provenance statements
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v6
       - uses: actions/setup-node@v6
         with:
           node-version: lts/*
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 810984e9f8..e2f34721d2 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -17,7 +17,7 @@ jobs:
     name: Lint Python
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v5
+    - uses: actions/checkout@v6
     - uses: astral-sh/ruff-action@v3
       with:
         args: "check --select=E,F,PLC,PLE,UP,W,YTT --ignore=E721,PLC0206,PLC0415,PLC1901,S101,UP031 --target-version=py39"
@@ -28,7 +28,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Checkout Repository
-      uses: actions/checkout@v5
+      uses: actions/checkout@v6
     - name: Use Node.js 22.x
       uses: actions/setup-node@v6
       with:
@@ -43,7 +43,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Checkout Repository
-      uses: actions/checkout@v5
+      uses: actions/checkout@v6
     - name: Use Node.js 22.x
       uses: actions/setup-node@v6
       with:
@@ -61,7 +61,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Checkout Repository
-      uses: actions/checkout@v5
+      uses: actions/checkout@v6
     - name: Use Node.js 22.x
       uses: actions/setup-node@v6
       with:
@@ -118,7 +118,7 @@ jobs:
       WASI_SDK_PATH: 'wasi-sdk-25.0'
     steps:
       - name: Checkout Repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
       - name: Use Node.js ${{ matrix.node }}
         uses: actions/setup-node@v6
         with:
diff --git a/.github/workflows/update-gyp-next.yml b/.github/workflows/update-gyp-next.yml
index ef8b4a807b..44936014f9 100644
--- a/.github/workflows/update-gyp-next.yml
+++ b/.github/workflows/update-gyp-next.yml
@@ -17,7 +17,7 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v6
         with:
           persist-credentials: false
 
diff --git a/.github/workflows/visual-studio.yml b/.github/workflows/visual-studio.yml
index 4608df08b8..c4d45c7d9f 100644
--- a/.github/workflows/visual-studio.yml
+++ b/.github/workflows/visual-studio.yml
@@ -26,7 +26,7 @@ jobs:
     runs-on: ${{ matrix.os }}
     steps:
       - name: Checkout Repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
       - name: Use Python 3
         uses: actions/setup-python@v6
         with:

From ee9cbdd6e1d40dc7c1cdc5ed6a75432c716eaf3f Mon Sep 17 00:00:00 2001
From: Caleb Everett 
Date: Fri, 19 Dec 2025 10:14:16 -0800
Subject: [PATCH 515/551] feat: include built package version in error logs
 (#3254)

I'm working on upgrading a lot of packages to node 24 and there are many failures in native modules. Looking at logs alone it's hard to spot which package versions are failing to build. node-gyp prints debug info about node, npm, and node-gyp but not the package that is currently being built.

It doesn't look like node-gyp reads package.json so I opted to read npm_package_ env vars, which is similar to the npm_config_ env vars node-gyp already reads.
---
 bin/node-gyp.js | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/bin/node-gyp.js b/bin/node-gyp.js
index f8317b47b3..5393a5c8e1 100755
--- a/bin/node-gyp.js
+++ b/bin/node-gyp.js
@@ -125,6 +125,13 @@ function errorMessage () {
   log.error('cwd', process.cwd())
   log.error('node -v', process.version)
   log.error('node-gyp -v', 'v' + prog.package.version)
+  // print the npm package version
+  for (const env of ['npm_package_name', 'npm_package_version']) {
+    const value = process.env[env]
+    if (value != null) {
+      log.error(`$${env}`, value)
+    }
+  }
 }
 
 function issueMessage () {

From 6c2e3743b3dafe554546f24a1d1de1b23561c68f Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Tue, 30 Dec 2025 13:14:51 +0100
Subject: [PATCH 516/551] msvs_version is no longer a valid npm config setting
 (#3257)

Remove misleading warnings.
% `npm config list -l | grep msvs`
% `npm config set msvs_version=2025`
> npm error `msvs_version` is not a valid npm option
---
 lib/find-visualstudio.js | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js
index e0cf383489..22fc4013f9 100644
--- a/lib/find-visualstudio.js
+++ b/lib/find-visualstudio.js
@@ -30,7 +30,7 @@ class VisualStudioFinder {
     this.configVersionYear = null
     this.configPath = null
     if (this.configMsvsVersion) {
-      this.addLog('msvs_version was set from command line or npm config')
+      this.addLog(`--msvs_version=${this.configMsvsVersion} was set on the command line`)
       if (this.configMsvsVersion.match(/^\d{4}$/)) {
         this.configVersionYear = parseInt(this.configMsvsVersion, 10)
         this.addLog(
@@ -41,7 +41,7 @@ class VisualStudioFinder {
           `- looking for Visual Studio installed in "${this.configPath}"`)
       }
     } else {
-      this.addLog('msvs_version not set from command line or npm config')
+      this.addLog('--msvs_version was not set on the command line')
     }
 
     if (process.env.VCINSTALLDIR) {

From f15b79a03c54cea0f66d940a0d6d839df867a319 Mon Sep 17 00:00:00 2001
From: Yaksh Bariya 
Date: Fri, 2 Jan 2026 21:43:50 +0530
Subject: [PATCH 517/551] fix: cpu concurrency detection on some platforms
 (#3255)

`os.availableParallelism()` should produce more accurate results of how
much parallelism should be used then `os.cpus().length`. The only
breaking change is the newer API requires node versions >= v18.x which
is fine as node-gyp only aims to support latest and LTS releases. And
the oldest LTS release still supported as of this commit is v22.

Fixes #3191
---
 lib/build.js | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/build.js b/lib/build.js
index 9c0cca8fc2..00a0abe691 100644
--- a/lib/build.js
+++ b/lib/build.js
@@ -163,7 +163,7 @@ async function build (gyp, argv) {
         if (!isNaN(j) && j > 0) {
           argv.push('/m:' + j)
         } else if (jobs.toUpperCase() === 'MAX') {
-          argv.push('/m:' + require('os').cpus().length)
+          argv.push('/m:' + require('os').availableParallelism())
         }
       }
     } else {
@@ -178,7 +178,7 @@ async function build (gyp, argv) {
           argv.push(j)
         } else if (jobs.toUpperCase() === 'MAX') {
           argv.push('--jobs')
-          argv.push(require('os').cpus().length)
+          argv.push(require('os').availableParallelism())
         }
       }
     }

From c7c678f89837d956194f326b01c5a8eb1d745db3 Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Fri, 2 Jan 2026 17:46:41 +0100
Subject: [PATCH 518/551] fix: python is no longer a valid npm config setting
 (#3258)

Remove misleading warnings.
% `npm config list -l | grep python ` # no hits
% `npm config set python=3.14`
> npm error `python` is not a valid npm option



##### Checklist


- [ ] `npm install && npm run lint && npm test` passes
- [ ] tests are included 
- [ ] documentation is changed or added
- [x] commit message follows [commit guidelines](https://github.com/googleapis/release-please#how-should-i-write-my-commits)

##### Description of change

---
 lib/find-python.js | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)

diff --git a/lib/find-python.js b/lib/find-python.js
index a71c00c2b6..273b5957b4 100644
--- a/lib/find-python.js
+++ b/lib/find-python.js
@@ -86,14 +86,10 @@ class PythonFinder {
         {
           before: () => {
             if (!this.configPython) {
-              this.addLog(
-                'Python is not set from command line or npm configuration')
+              this.addLog('--python was not set on the command line')
               return SKIP
             }
-            this.addLog('checking Python explicitly set from command line or ' +
-              'npm configuration')
-            this.addLog('- "--python=" or "npm config get python" is ' +
-              `"${this.configPython}"`)
+            this.addLog(`--python=${this.configPython} was set on the command line`)
           },
           check: () => this.checkCommand(this.configPython)
         },
@@ -295,8 +291,6 @@ class PythonFinder {
       `- Use the switch --python="${pathExample}"`,
       '  (accepted by both node-gyp and npm)',
       '- Set the environment variable PYTHON',
-      '- Set the npm configuration variable python:',
-      `  npm config set python "${pathExample}"`,
       'For more information consult the documentation at:',
       'https://github.com/nodejs/node-gyp#installation',
       '**********************************************************'

From a52bc819f44b881854ff798865ad416430e3dce2 Mon Sep 17 00:00:00 2001
From: Chengzhong Wu 
Date: Sun, 4 Jan 2026 13:20:19 -0500
Subject: [PATCH 519/551] doc: add a note about changes in gyp folder (#3259)

---
 CONTRIBUTING.md | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 5b977898f1..618fb6ae9e 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -1,5 +1,13 @@
 # Contributing to node-gyp
 
+## Making changes to gyp-next
+
+Changes in the subfolder `gyp/` should be submitted to the
+[`gyp-next`][] repository first. The `gyp/` folder is regularly
+synced from [`gyp-next`][] with GitHub Actions workflow
+[`update-gyp-next.yml`](.github/workflows/update-gyp-next.yml),
+and any changes in this folder would be overridden by the workflow.
+
 ## Code of Conduct
 
 Please read the
@@ -32,3 +40,5 @@ By making a contribution to this project, I certify that:
   personal information I submit with it, including my sign-off) is
   maintained indefinitely and may be redistributed consistent with
   this project or the open source license(s) involved.
+
+[`gyp-next`]: https://github.com/nodejs/gyp-next

From 7b4f315e4dad880c841d21df641d6dd9b68bf36b Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Sun, 4 Jan 2026 19:21:13 +0100
Subject: [PATCH 520/551] fix: Test Windows on Python 3.14, not 3.13 (#3262)

---
 .github/workflows/tests.yml | 12 +++---------
 1 file changed, 3 insertions(+), 9 deletions(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index e2f34721d2..d530aed20d 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -93,14 +93,10 @@ jobs:
       fail-fast: false
       max-parallel: 11
       matrix:
-        # Windows on Python 3.14 is blocked by nodejs/node#59983
-        os: [macos-latest, ubuntu-latest]
+        os: [macos-latest, ubuntu-latest, windows-latest]
         python: ["3.10", "3.12", "3.14"]
         node: [20.x, 22.x, 24.x]
         include:
-          - os: windows-latest
-            python: "3.13"  # Windows on Python 3.13 instead of 3.14
-            node: 24.x
           - os: macos-15-intel  # macOS on Intel
             python: "3.14"
             node: 24.x
@@ -108,7 +104,7 @@ jobs:
             python: "3.14"
             node: 24.x
           - os: windows-11-arm  # Windows on ARM
-            python: "3.13"  # Windows on Python 3.13 instead of 3.14
+            python: "3.14"
             node: 24.x
     name: ${{ matrix.os }} - ${{ matrix.python }} - ${{ matrix.node }}
     runs-on: ${{ matrix.os }}
@@ -128,8 +124,6 @@ jobs:
         with:
           python-version: ${{ matrix.python }}
           allow-prereleases: true
-        env:
-          PYTHON_VERSION: ${{ matrix.python }}  # Why do this?
       - uses: seanmiddleditch/gha-setup-ninja@v6
       - name: Install wasi-sdk (Windows)
         shell: pwsh
@@ -180,4 +174,4 @@ jobs:
         shell: bash # Building wasm on Windows requires using make generator, it only works in bash
         run: npm run test --python="${pythonLocation}\\python.exe"
         env:
-          FULL_TEST: ${{ (matrix.node == '24.x' && matrix.python == '3.13') && '1' || '0' }}
+          FULL_TEST: ${{ (matrix.node == '24.x' && matrix.python == '3.14') && '1' || '0' }}

From 3f819499d8ce6d46c646466de7b9492bf7bde663 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?J=C3=A9r=C3=A9my=20Lal?= 
Date: Sun, 4 Jan 2026 19:21:43 +0100
Subject: [PATCH 521/551] fix: Switch to URL instead of url.parse (#3256)

---
 lib/process-release.js | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/process-release.js b/lib/process-release.js
index c9a319dfad..b92e8d5b83 100644
--- a/lib/process-release.js
+++ b/lib/process-release.js
@@ -109,15 +109,15 @@ function processRelease (argv, gyp, defaultVersion, defaultRelease) {
     versionDir: (name !== 'node' ? name + '-' : '') + version,
     ia32: {
       libUrl: libUrl32,
-      libPath: normalizePath(path.relative(url.parse(baseUrl).path, url.parse(libUrl32).path))
+      libPath: normalizePath(path.relative(new URL(baseUrl).pathname, new URL(libUrl32).pathname))
     },
     x64: {
       libUrl: libUrl64,
-      libPath: normalizePath(path.relative(url.parse(baseUrl).path, url.parse(libUrl64).path))
+      libPath: normalizePath(path.relative(new URL(baseUrl).pathname, new URL(libUrl64).pathname))
     },
     arm64: {
       libUrl: libUrlArm64,
-      libPath: normalizePath(path.relative(url.parse(baseUrl).path, url.parse(libUrlArm64).path))
+      libPath: normalizePath(path.relative(new URL(baseUrl).pathname, new URL(libUrlArm64).pathname))
     }
   }
 }

From 0407877e3e26d3201f74cf1a9deabbbfc40bdbb7 Mon Sep 17 00:00:00 2001
From: Mike McCready <66998419+MikeMcC399@users.noreply.github.com>
Date: Fri, 16 Jan 2026 17:52:40 +0100
Subject: [PATCH 522/551] readme: update Python manual install instructions for
 Windows (#3265)

---
 README.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/README.md b/README.md
index 72833b1363..f068c7c87c 100644
--- a/README.md
+++ b/README.md
@@ -55,8 +55,8 @@ choco install python visualstudio2022-workload-vctools -y
 
 Or install and configure Python and Visual Studio tools manually:
 
-  * Install the current [version of Python](https://devguide.python.org/versions/) from the
-  [Microsoft Store](https://apps.microsoft.com/store/search?publisher=Python+Software+Foundation).
+  * Follow the instructions in [Using Python on Windows](https://docs.python.org/3/using/windows.html) to install
+    the current [version of Python](https://www.python.org/downloads/).
 
    * Install Visual C++ Build Environment: For Visual Studio 2019 or later, use the `Desktop development with C++` workload from [Visual Studio Community](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=Community).  For a version older than Visual Studio 2019, install [Visual Studio Build Tools](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) with the `Visual C++ build tools` option.
 

From 0f2bc7d2e0665b1c7bb03e1cd8653ea330277a70 Mon Sep 17 00:00:00 2001
From: Mike McCready <66998419+MikeMcC399@users.noreply.github.com>
Date: Sun, 18 Jan 2026 00:10:28 +0100
Subject: [PATCH 523/551] readme: correct typos (#3269)

---
 README.md | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/README.md b/README.md
index f068c7c87c..61c6c09533 100644
--- a/README.md
+++ b/README.md
@@ -66,8 +66,8 @@ Or install and configure Python and Visual Studio tools manually:
 
    To use the native ARM64 C++ compiler on Windows on ARM, ensure that you have Visual Studio 2022 [17.4 or later](https://devblogs.microsoft.com/visualstudio/arm64-visual-studio-is-officially-here/) installed.
 
-It's advised to install following Powershell module: [VSSetup](https://github.com/microsoft/vssetup.powershell) using `Install-Module VSSetup -Scope CurrentUser`.
-This will make Visual Studio detection logic to use more flexible and accessible method, avoiding Powershell's `ConstrainedLanguage` mode.
+It's advised to install the following PowerShell module: [VSSetup](https://github.com/microsoft/vssetup.powershell) using `Install-Module VSSetup -Scope CurrentUser`.
+This will make Visual Studio detection logic use a more flexible and accessible method, avoiding PowerShell's `ConstrainedLanguage` mode.
 
 ### Configuring Python Dependency
 
@@ -270,8 +270,8 @@ set npm_package_config_node_gyp_devdir=c:\temp\.gyp
 ```
 
 Note that in versions of npm before v11 it was possible to use the prefix `npm_config_` for
-environement variables. This was deprecated in npm@11 and will be removed in npm@12 so it
-is recommened to convert your environment variables to the above format.
+environment variables. This was deprecated in npm@11 and will be removed in npm@12 so it
+is recommended to convert your environment variables to the above format.
 
 ### `npm` configuration for npm versions before v9
 

From 30cda268730798dc0f67182c8c568d8b8069964e Mon Sep 17 00:00:00 2001
From: Mike McCready <66998419+MikeMcC399@users.noreply.github.com>
Date: Mon, 19 Jan 2026 10:10:28 +0100
Subject: [PATCH 524/551] readme: remove obsolete Microsoft Node.js Guidelines
 link (#3268)

---
 README.md | 2 --
 1 file changed, 2 deletions(-)

diff --git a/README.md b/README.md
index 61c6c09533..3afa252874 100644
--- a/README.md
+++ b/README.md
@@ -60,8 +60,6 @@ Or install and configure Python and Visual Studio tools manually:
 
    * Install Visual C++ Build Environment: For Visual Studio 2019 or later, use the `Desktop development with C++` workload from [Visual Studio Community](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=Community).  For a version older than Visual Studio 2019, install [Visual Studio Build Tools](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) with the `Visual C++ build tools` option.
 
-   If the above steps didn't work for you, please visit [Microsoft's Node.js Guidelines for Windows](https://github.com/Microsoft/nodejs-guidelines/blob/master/windows-environment.md#compiling-native-addon-modules) for additional tips.
-
    To target native ARM64 Node.js on Windows on ARM, add the components "Visual C++ compilers and libraries for ARM64" and "Visual C++ ATL for ARM64".
 
    To use the native ARM64 C++ compiler on Windows on ARM, ensure that you have Visual Studio 2022 [17.4 or later](https://devblogs.microsoft.com/visualstudio/arm64-visual-studio-is-officially-here/) installed.

From 888ff2c48a4cf5602013b96b52c6670906976f63 Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Mon, 26 Jan 2026 09:06:46 -0500
Subject: [PATCH 525/551] feat: update gyp-next to v0.21.1 (#3273)

---
 gyp/.github/workflows/node-gyp.yml       | 12 +++---------
 gyp/.github/workflows/nodejs.yml         | 12 +++---------
 gyp/.github/workflows/python_tests.yml   |  2 +-
 gyp/.github/workflows/release-please.yml | 10 +++++-----
 gyp/.release-please-manifest.json        |  2 +-
 gyp/CHANGELOG.md                         |  7 +++++++
 gyp/pylib/gyp/MSVSNew.py                 |  6 +++---
 gyp/pylib/gyp/generator/make.py          |  2 +-
 gyp/pylib/gyp/generator/ninja.py         |  7 +++----
 gyp/pylib/gyp/xcodeproj_file.py          |  2 +-
 gyp/pyproject.toml                       |  2 +-
 11 files changed, 29 insertions(+), 35 deletions(-)

diff --git a/gyp/.github/workflows/node-gyp.yml b/gyp/.github/workflows/node-gyp.yml
index 2e592a6d38..016d2e0abc 100644
--- a/gyp/.github/workflows/node-gyp.yml
+++ b/gyp/.github/workflows/node-gyp.yml
@@ -11,27 +11,21 @@ jobs:
       matrix:
         os: [macos-latest, ubuntu-latest, windows-latest]
         python-version: ["3.10", "3.12", "3.14"]
-        exclude:
-          # Windows on Python 3.14 is blocked by nodejs/node#59983
-          - os: windows-latest
-            python-version: "3.14"
         include:
-          - os: windows-latest  # Windows on Python 3.13 instead of 3.14
-            python-version: "3.13"
           - os: macos-15-intel  # macOS on Intel
             python-version: "3.14"
           - os: ubuntu-24.04-arm  # Ubuntu on ARM
             python-version: "3.14"
           - os: windows-11-arm  # Windows on ARM
-            python-version: "3.13"  # Windows on Python 3.13 instead of 3.14
+            python-version: "3.14"
     runs-on: ${{ matrix.os }}
     steps:
       - name: Clone gyp-next
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
         with:
           path: gyp-next
       - name: Clone nodejs/node-gyp
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
         with:
           repository: nodejs/node-gyp
           path: node-gyp
diff --git a/gyp/.github/workflows/nodejs.yml b/gyp/.github/workflows/nodejs.yml
index e73e5bd06b..c88fe7bcf2 100644
--- a/gyp/.github/workflows/nodejs.yml
+++ b/gyp/.github/workflows/nodejs.yml
@@ -11,28 +11,22 @@ jobs:
       matrix:
         os: [macos-latest, ubuntu-latest, windows-latest]
         python-version: ["3.10", "3.12", "3.14"]
-        exclude:
-          # Windows on Python 3.14 is blocked by nodejs/node#59983
-          - os: windows-latest
-            python-version: "3.14"
         include:
-          - os: windows-latest  # Windows on Python 3.13 instead of 3.14
-            python-version: "3.13"
           - os: macos-15-intel  # macOS on Intel
             python-version: "3.14"
           - os: ubuntu-24.04-arm  # Ubuntu on ARM
             python-version: "3.14"
           - os: windows-11-arm  # Windows on ARM
-            python-version: "3.13"  # Windows on Python 3.13 instead of 3.14
+            python-version: "3.14"
 
     runs-on: ${{ matrix.os }}
     steps:
       - name: Clone gyp-next
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
         with:
           path: gyp-next
       - name: Clone nodejs/node
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
         with:
           repository: nodejs/node
           path: node
diff --git a/gyp/.github/workflows/python_tests.yml b/gyp/.github/workflows/python_tests.yml
index 72dfd58536..812d32d7f7 100644
--- a/gyp/.github/workflows/python_tests.yml
+++ b/gyp/.github/workflows/python_tests.yml
@@ -20,7 +20,7 @@ jobs:
           - os: macos-26
             python-version: 3.x
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v6
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v6
         with:
diff --git a/gyp/.github/workflows/release-please.yml b/gyp/.github/workflows/release-please.yml
index 6a18003c79..81f8626c77 100644
--- a/gyp/.github/workflows/release-please.yml
+++ b/gyp/.github/workflows/release-please.yml
@@ -24,11 +24,11 @@ jobs:
     if: ${{ needs.release-please.outputs.release_created }}  # only publish on release
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v5
+    - uses: actions/checkout@v6
     - name: Build a binary wheel and a source tarball
       run: pipx run build
     - name: Store the distribution packages
-      uses: actions/upload-artifact@v5
+      uses: actions/upload-artifact@v6
       with:
         name: python-package-distributions
         path: dist/
@@ -48,7 +48,7 @@ jobs:
       id-token: write # IMPORTANT: mandatory for trusted publishing
     steps:
     - name: Download all the dists
-      uses: actions/download-artifact@v6
+      uses: actions/download-artifact@v7
       with:
         name: python-package-distributions
         path: dist/
@@ -68,12 +68,12 @@ jobs:
       id-token: write # IMPORTANT: mandatory for sigstore
     steps:
     - name: Download all the dists
-      uses: actions/download-artifact@v6
+      uses: actions/download-artifact@v7
       with:
         name: python-package-distributions
         path: dist/
     - name: Sign the dists with Sigstore
-      uses: sigstore/gh-action-sigstore-python@v3.1.0
+      uses: sigstore/gh-action-sigstore-python@v3.2.0
       with:
         inputs: >-
           ./dist/*.tar.gz
diff --git a/gyp/.release-please-manifest.json b/gyp/.release-please-manifest.json
index ca64307ab8..c825abab69 100644
--- a/gyp/.release-please-manifest.json
+++ b/gyp/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "0.21.0"
+    ".": "0.21.1"
 }
diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md
index 31f4d25874..9a14685447 100644
--- a/gyp/CHANGELOG.md
+++ b/gyp/CHANGELOG.md
@@ -1,5 +1,12 @@
 # Changelog
 
+## [0.21.1](https://github.com/nodejs/gyp-next/compare/v0.21.0...v0.21.1) (2026-01-24)
+
+
+### Bug Fixes
+
+* replace weak hash functions with SHA-256 ([#329](https://github.com/nodejs/gyp-next/issues/329)) ([958029e](https://github.com/nodejs/gyp-next/commit/958029e6e4969a871d15e78cd083bb102bebb381))
+
 ## [0.21.0](https://github.com/nodejs/gyp-next/compare/v0.20.5...v0.21.0) (2025-11-04)
 
 
diff --git a/gyp/pylib/gyp/MSVSNew.py b/gyp/pylib/gyp/MSVSNew.py
index f8e4993d94..9149f404a5 100644
--- a/gyp/pylib/gyp/MSVSNew.py
+++ b/gyp/pylib/gyp/MSVSNew.py
@@ -34,7 +34,7 @@ def MakeGuid(name, seed="msvs_new"):
 
     Args:
       name: Target name.
-      seed: Seed for MD5 hash.
+      seed: Seed for SHA-256 hash.
     Returns:
       A GUID-line string calculated from the name and seed.
 
@@ -44,8 +44,8 @@ def MakeGuid(name, seed="msvs_new"):
     determine the GUID to refer to explicitly.  It also means that the GUID will
     not change when the project for a target is rebuilt.
     """
-    # Calculate a MD5 signature for the seed and name.
-    d = hashlib.md5((str(seed) + str(name)).encode("utf-8")).hexdigest().upper()
+    # Calculate a SHA-256 signature for the seed and name.
+    d = hashlib.sha256((str(seed) + str(name)).encode("utf-8")).hexdigest().upper()
     # Convert most of the signature to GUID form (discard the rest)
     guid = (
         "{"
diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py
index 5f30f39fc5..16b6f4e80b 100644
--- a/gyp/pylib/gyp/generator/make.py
+++ b/gyp/pylib/gyp/generator/make.py
@@ -2169,7 +2169,7 @@ def WriteMakeRule(
             # - The multi-output rule will have an do-nothing recipe.
 
             # Hash the target name to avoid generating overlong filenames.
-            cmddigest = hashlib.sha1(
+            cmddigest = hashlib.sha256(
                 (command or self.target).encode("utf-8")
             ).hexdigest()
             intermediate = "%s.intermediate" % cmddigest
diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py
index bc9ddd2654..4eac6cdb27 100644
--- a/gyp/pylib/gyp/generator/ninja.py
+++ b/gyp/pylib/gyp/generator/ninja.py
@@ -809,9 +809,8 @@ def cygwin_munge(path):
                 outputs = [self.GypPathToNinja(o, env) for o in outputs]
                 if self.flavor == "win":
                     # WriteNewNinjaRule uses unique_name to create a rsp file on win.
-                    extra_bindings.append(
-                        ("unique_name", hashlib.md5(outputs[0]).hexdigest())
-                    )
+                    unique_name = hashlib.sha256(outputs[0].encode("utf-8")).hexdigest()
+                    extra_bindings.append(("unique_name", unique_name))
 
                 self.ninja.build(
                     outputs,
@@ -2803,7 +2802,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name
             build_file, name, toolset
         )
         qualified_target_for_hash = qualified_target_for_hash.encode("utf-8")
-        hash_for_rules = hashlib.md5(qualified_target_for_hash).hexdigest()
+        hash_for_rules = hashlib.sha256(qualified_target_for_hash).hexdigest()
 
         base_path = os.path.dirname(build_file)
         obj = "obj"
diff --git a/gyp/pylib/gyp/xcodeproj_file.py b/gyp/pylib/gyp/xcodeproj_file.py
index cb467470d3..2004518dcb 100644
--- a/gyp/pylib/gyp/xcodeproj_file.py
+++ b/gyp/pylib/gyp/xcodeproj_file.py
@@ -429,7 +429,7 @@ def _HashUpdate(hash, data):
             hash.update(data)
 
         if seed_hash is None:
-            seed_hash = hashlib.sha1()
+            seed_hash = hashlib.sha256()
 
         hash = seed_hash.copy()
 
diff --git a/gyp/pyproject.toml b/gyp/pyproject.toml
index cd4f0383fd..fa30c8cf96 100644
--- a/gyp/pyproject.toml
+++ b/gyp/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "gyp-next"
-version = "0.21.0"
+version = "0.21.1"
 authors = [
   { name="Node.js contributors", email="ryzokuken@disroot.org" },
 ]

From 7bf371c4dd7c694232ab3169d02fe8197e1ecc6d Mon Sep 17 00:00:00 2001
From: Copilot <198982749+Copilot@users.noreply.github.com>
Date: Mon, 26 Jan 2026 09:09:16 -0500
Subject: [PATCH 526/551] chore(deps): upgrade tar to 7.5.4 to address
 CVE-2026-23950 (#3271)

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: cclauss <3709715+cclauss@users.noreply.github.com>
---
 package.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package.json b/package.json
index ae60687844..fd7fdba34c 100644
--- a/package.json
+++ b/package.json
@@ -29,7 +29,7 @@
     "nopt": "^9.0.0",
     "proc-log": "^6.0.0",
     "semver": "^7.3.5",
-    "tar": "^7.5.2",
+    "tar": "^7.5.4",
     "tinyglobby": "^0.2.12",
     "which": "^6.0.0"
   },

From 878061f9b58afc7f6ecb45b74e521005bd619473 Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Tue, 27 Jan 2026 09:47:06 -0500
Subject: [PATCH 527/551] chore(main): release 12.2.0 (#3249)

---
 .release-please-manifest.json |  2 +-
 CHANGELOG.md                  | 34 ++++++++++++++++++++++++++++++++++
 package.json                  |  2 +-
 3 files changed, 36 insertions(+), 2 deletions(-)

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
index 4899c67643..41c2c8154a 100644
--- a/.release-please-manifest.json
+++ b/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "12.1.0"
+    ".": "12.2.0"
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2bdec2cc6b..4a83b4dfcd 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,39 @@
 # Changelog
 
+## [12.2.0](https://github.com/nodejs/node-gyp/compare/v12.1.0...v12.2.0) (2026-01-26)
+
+
+### Features
+
+* include built package version in error logs ([#3254](https://github.com/nodejs/node-gyp/issues/3254)) ([ee9cbdd](https://github.com/nodejs/node-gyp/commit/ee9cbdd6e1d40dc7c1cdc5ed6a75432c716eaf3f))
+* update gyp-next to v0.21.1 ([#3273](https://github.com/nodejs/node-gyp/issues/3273)) ([888ff2c](https://github.com/nodejs/node-gyp/commit/888ff2c48a4cf5602013b96b52c6670906976f63))
+
+
+### Bug Fixes
+
+* cpu concurrency detection on some platforms ([#3255](https://github.com/nodejs/node-gyp/issues/3255)) ([f15b79a](https://github.com/nodejs/node-gyp/commit/f15b79a03c54cea0f66d940a0d6d839df867a319)), closes [#3191](https://github.com/nodejs/node-gyp/issues/3191)
+* python is no longer a valid npm config setting ([#3258](https://github.com/nodejs/node-gyp/issues/3258)) ([c7c678f](https://github.com/nodejs/node-gyp/commit/c7c678f89837d956194f326b01c5a8eb1d745db3))
+* Switch to URL instead of url.parse ([#3256](https://github.com/nodejs/node-gyp/issues/3256)) ([3f81949](https://github.com/nodejs/node-gyp/commit/3f819499d8ce6d46c646466de7b9492bf7bde663))
+* Test Windows on Python 3.14, not 3.13 ([#3262](https://github.com/nodejs/node-gyp/issues/3262)) ([7b4f315](https://github.com/nodejs/node-gyp/commit/7b4f315e4dad880c841d21df641d6dd9b68bf36b))
+
+
+### Core
+
+* **deps:** bump actions/checkout from 5 to 6 ([#3248](https://github.com/nodejs/node-gyp/issues/3248)) ([db5385c](https://github.com/nodejs/node-gyp/commit/db5385c5467e5bfb914b9954f0313c46f1f4e10d))
+
+
+### Doc
+
+* add a note about changes in gyp folder ([#3259](https://github.com/nodejs/node-gyp/issues/3259)) ([a52bc81](https://github.com/nodejs/node-gyp/commit/a52bc819f44b881854ff798865ad416430e3dce2))
+* correct typos ([#3269](https://github.com/nodejs/node-gyp/issues/3269)) ([0f2bc7d](https://github.com/nodejs/node-gyp/commit/0f2bc7d2e0665b1c7bb03e1cd8653ea330277a70))
+* remove obsolete Microsoft Node.js Guidelines link ([#3268](https://github.com/nodejs/node-gyp/issues/3268)) ([30cda26](https://github.com/nodejs/node-gyp/commit/30cda268730798dc0f67182c8c568d8b8069964e))
+* update Python manual install instructions for Windows ([#3265](https://github.com/nodejs/node-gyp/issues/3265)) ([0407877](https://github.com/nodejs/node-gyp/commit/0407877e3e26d3201f74cf1a9deabbbfc40bdbb7))
+
+
+### Miscellaneous
+
+* **deps:** upgrade tar to 7.5.4 to address CVE-2026-23950 ([#3271](https://github.com/nodejs/node-gyp/issues/3271)) ([7bf371c](https://github.com/nodejs/node-gyp/commit/7bf371c4dd7c694232ab3169d02fe8197e1ecc6d))
+
 ## [12.1.0](https://github.com/nodejs/node-gyp/compare/v12.0.0...v12.1.0) (2025-11-12)
 
 
diff --git a/package.json b/package.json
index fd7fdba34c..f5dcf156de 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "12.1.0",
+  "version": "12.2.0",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From 46d75768018d10dc4b5693b35cf58f502425cbbe Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?J=C3=A9r=C3=A9my=20Lal?= 
Date: Thu, 29 Jan 2026 16:45:14 +0100
Subject: [PATCH 528/551] fix: Switch to URL instead of url.resolve (#3256)
 (#3263)

---
 lib/process-release.js | 36 +++++++++++++++++-------------------
 1 file changed, 17 insertions(+), 19 deletions(-)

diff --git a/lib/process-release.js b/lib/process-release.js
index b92e8d5b83..75f3fc136a 100644
--- a/lib/process-release.js
+++ b/lib/process-release.js
@@ -1,9 +1,6 @@
-/* eslint-disable n/no-deprecated-api */
-
 'use strict'
 
 const semver = require('semver')
-const url = require('url')
 const path = require('path')
 const log = require('./log')
 
@@ -74,11 +71,11 @@ function processRelease (argv, gyp, defaultVersion, defaultRelease) {
   } else {
     distBaseUrl = 'https://nodejs.org/dist'
   }
-  distBaseUrl += '/v' + version + '/'
+  distBaseUrl = new URL(distBaseUrl + '/v' + version + '/')
 
   // new style, based on process.release so we have a lot of the data we need
   if (defaultRelease && defaultRelease.headersUrl && !overrideDistUrl) {
-    baseUrl = url.resolve(defaultRelease.headersUrl, './')
+    baseUrl = new URL('./', defaultRelease.headersUrl)
     libUrl32 = resolveLibUrl(name, defaultRelease.libUrl || baseUrl || distBaseUrl, 'x86', versionSemver.major)
     libUrl64 = resolveLibUrl(name, defaultRelease.libUrl || baseUrl || distBaseUrl, 'x64', versionSemver.major)
     libUrlArm64 = resolveLibUrl(name, defaultRelease.libUrl || baseUrl || distBaseUrl, 'arm64', versionSemver.major)
@@ -96,28 +93,28 @@ function processRelease (argv, gyp, defaultVersion, defaultRelease) {
     // have a *-headers.tar.gz file in its dist location, even some frankenstein
     // custom version
     canGetHeaders = semver.satisfies(versionSemver, headersTarballRange)
-    tarballUrl = url.resolve(baseUrl, name + '-v' + version + (canGetHeaders ? '-headers' : '') + '.tar.gz')
+    tarballUrl = new URL(name + '-v' + version + (canGetHeaders ? '-headers' : '') + '.tar.gz', baseUrl).href
   }
 
   return {
     version,
     semver: versionSemver,
     name,
-    baseUrl,
+    baseUrl: baseUrl.href,
     tarballUrl,
-    shasumsUrl: url.resolve(baseUrl, 'SHASUMS256.txt'),
+    shasumsUrl: new URL('SHASUMS256.txt', baseUrl).href,
     versionDir: (name !== 'node' ? name + '-' : '') + version,
     ia32: {
-      libUrl: libUrl32,
-      libPath: normalizePath(path.relative(new URL(baseUrl).pathname, new URL(libUrl32).pathname))
+      libUrl: libUrl32.href,
+      libPath: normalizePath(path.relative(baseUrl.pathname, libUrl32.pathname))
     },
     x64: {
-      libUrl: libUrl64,
-      libPath: normalizePath(path.relative(new URL(baseUrl).pathname, new URL(libUrl64).pathname))
+      libUrl: libUrl64.href,
+      libPath: normalizePath(path.relative(baseUrl.pathname, libUrl64.pathname))
     },
     arm64: {
-      libUrl: libUrlArm64,
-      libPath: normalizePath(path.relative(new URL(baseUrl).pathname, new URL(libUrlArm64).pathname))
+      libUrl: libUrlArm64.href,
+      libPath: normalizePath(path.relative(baseUrl.pathname, libUrlArm64.pathname))
     }
   }
 }
@@ -127,20 +124,21 @@ function normalizePath (p) {
 }
 
 function resolveLibUrl (name, defaultUrl, arch, versionMajor) {
-  const base = url.resolve(defaultUrl, './')
-  const hasLibUrl = bitsre.test(defaultUrl) || (versionMajor === 3 && bitsreV3.test(defaultUrl))
+  if (!defaultUrl.pathname) defaultUrl = new URL(defaultUrl)
+  const base = new URL('./', defaultUrl)
+  const hasLibUrl = bitsre.test(defaultUrl.pathname) || (versionMajor === 3 && bitsreV3.test(defaultUrl.pathname))
 
   if (!hasLibUrl) {
     // let's assume it's a baseUrl then
     if (versionMajor >= 1) {
-      return url.resolve(base, 'win-' + arch + '/' + name + '.lib')
+      return new URL('win-' + arch + '/' + name + '.lib', base)
     }
     // prior to io.js@1.0.0 32-bit node.lib lives in /, 64-bit lives in /x64/
-    return url.resolve(base, (arch === 'x86' ? '' : arch + '/') + name + '.lib')
+    return new URL((arch === 'x86' ? '' : arch + '/') + name + '.lib', base)
   }
 
   // else we have a proper url to a .lib, just make sure it's the right arch
-  return defaultUrl.replace(versionMajor === 3 ? bitsreV3 : bitsre, '/win-' + arch + '/')
+  return new URL(defaultUrl.pathname.replace(versionMajor === 3 ? bitsreV3 : bitsre, '/win-' + arch + '/'), defaultUrl)
 }
 
 module.exports = processRelease

From 13814583a476c85a84d5ac902c7ffa310120cb88 Mon Sep 17 00:00:00 2001
From: Stefan Stojanovic 
Date: Fri, 27 Feb 2026 01:14:47 +0100
Subject: [PATCH 529/551] win: improve Add-Type with -IgnoreWarnings (#3280)

Refs: https://github.com/nodejs/node-gyp/issues/3276
---
 lib/find-visualstudio.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/find-visualstudio.js b/lib/find-visualstudio.js
index 22fc4013f9..efb8b02a59 100644
--- a/lib/find-visualstudio.js
+++ b/lib/find-visualstudio.js
@@ -247,7 +247,7 @@ class VisualStudioFinder {
       'Unrestricted',
       '-NoProfile',
       '-Command',
-      '&{Add-Type -Path \'' + csFile + '\';' + '[VisualStudioConfiguration.Main]::PrintJson()}'
+      '&{Add-Type -IgnoreWarnings -Path \'' + csFile + '\';' + '[VisualStudioConfiguration.Main]::PrintJson()}'
     ]
 
     this.log.silly('Running', ps, psArgs)

From 19da1583b3876dce8c97263b168d9dfef637b76b Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Mon, 23 Mar 2026 15:33:59 +0100
Subject: [PATCH 530/551] build(deps-dev): bump neostandard from 0.12.2 to
 0.13.0 (#3289)

Bumps [neostandard](https://github.com/neostandard/neostandard) from 0.12.2 to 0.13.0.
- [Release notes](https://github.com/neostandard/neostandard/releases)
- [Changelog](https://github.com/neostandard/neostandard/blob/main/CHANGELOG.md)
- [Commits](https://github.com/neostandard/neostandard/compare/v0.12.2...v0.13.0)

---
updated-dependencies:
- dependency-name: neostandard
  dependency-version: 0.13.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 package.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package.json b/package.json
index f5dcf156de..71e0790043 100644
--- a/package.json
+++ b/package.json
@@ -42,7 +42,7 @@
     "eslint": "^9.39.1",
     "mocha": "^11.7.5",
     "nan": "^2.23.1",
-    "neostandard": "^0.12.2",
+    "neostandard": "^0.13.0",
     "require-inject": "^1.4.4"
   },
   "scripts": {

From 393ec2be480195b585768eb18ac2d92b858b12db Mon Sep 17 00:00:00 2001
From: Samuel Attard 
Date: Tue, 21 Apr 2026 01:52:21 -0400
Subject: [PATCH 531/551] feat: replace make-fetch-happen with built-in fetch
 (#3302)

* feat: replace make-fetch-happen with built-in fetch

Use Node's built-in fetch for downloading headers/tarballs, with undici's
EnvHttpProxyAgent providing --proxy, --noproxy and --cafile support (plus
http_proxy/https_proxy/no_proxy env var handling). Drops 36 transitive
dependencies.

* fixup: address review feedback

- Guard Readable.fromWeb against null body (204/304 responses)
- Add socket error handlers to CONNECT tunnel in proxy test
- Destroy socket in noproxy test's CONNECT handler so a regression
  fails fast instead of hanging

* ci: fix Python lint and npm install in tests workflow

Apply f-string fix from gyp-next#337 to resolve ruff F507 in
simple_copy.py, and add npm@~11.10.0 pre-install step from #3300
to work around npm/cli#9151.

* fix: apply cafile to proxied TLS connections

EnvHttpProxyAgent forwards opts to an internal ProxyAgent for proxied
requests, which reads origin TLS config from requestTls rather than
connect. Set connect/requestTls/proxyTls so the custom CA is honored
on both direct and proxied paths. Adds a test covering https origin
behind an HTTP CONNECT proxy with a custom CA.
---
 .github/workflows/tests.yml  |  4 +-
 gyp/pylib/gyp/simple_copy.py |  4 +-
 lib/download.js              | 63 ++++++++++++++++++++++++----
 package.json                 |  2 +-
 test/test-download.js        | 79 ++++++++++++++++++++++++++++++++----
 5 files changed, 133 insertions(+), 19 deletions(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index d530aed20d..8d679a3157 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -67,7 +67,9 @@ jobs:
       with:
         node-version: 22.x
     - name: Update npm
-      run: npm install npm@latest -g
+      run: |
+        npm install npm@~11.10.0 -g # Workaround for https://github.com/npm/cli/issues/9151
+        npm install npm@latest -g
     - name: Install Dependencies
       run: npm install
     - name: Pack
diff --git a/gyp/pylib/gyp/simple_copy.py b/gyp/pylib/gyp/simple_copy.py
index 8b026642fc..2b9100f3e1 100644
--- a/gyp/pylib/gyp/simple_copy.py
+++ b/gyp/pylib/gyp/simple_copy.py
@@ -24,8 +24,8 @@ def deepcopy(x):
         return _deepcopy_dispatch[type(x)](x)
     except KeyError:
         raise Error(
-            "Unsupported type %s for deepcopy. Use copy.deepcopy "
-            + "or expand simple_copy support." % type(x)
+            f"Unsupported type {type(x)} for deepcopy. Use copy.deepcopy "
+            + "or expand simple_copy support."
         )
 
 
diff --git a/lib/download.js b/lib/download.js
index ed0aa37f44..a9866d8a63 100644
--- a/lib/download.js
+++ b/lib/download.js
@@ -1,4 +1,5 @@
-const fetch = require('make-fetch-happen')
+const { Readable } = require('stream')
+const { EnvHttpProxyAgent } = require('undici')
 const { promises: fs } = require('graceful-fs')
 const log = require('./log')
 
@@ -10,19 +11,65 @@ async function download (gyp, url) {
       'User-Agent': `node-gyp v${gyp.version} (node ${process.version})`,
       Connection: 'keep-alive'
     },
-    proxy: gyp.opts.proxy,
-    noProxy: gyp.opts.noproxy
+    dispatcher: await createDispatcher(gyp)
   }
 
-  const cafile = gyp.opts.cafile
-  if (cafile) {
-    requestOpts.ca = await readCAFile(cafile)
+  let res
+  try {
+    res = await fetch(url, requestOpts)
+  } catch (err) {
+    // Built-in fetch wraps low-level errors in "TypeError: fetch failed" with
+    // the underlying error on .cause. Callers inspect .code (e.g. ENOTFOUND).
+    if (err.cause) {
+      throw err.cause
+    }
+    throw err
   }
 
-  const res = await fetch(url, requestOpts)
   log.http(res.status, res.url)
 
-  return res
+  const body = res.body ? Readable.fromWeb(res.body) : Readable.from([])
+  return {
+    status: res.status,
+    url: res.url,
+    body,
+    text: async () => {
+      let data = ''
+      body.setEncoding('utf8')
+      for await (const chunk of body) {
+        data += chunk
+      }
+      return data
+    }
+  }
+}
+
+async function createDispatcher (gyp) {
+  const env = process.env
+  const hasProxyEnv = env.http_proxy || env.HTTP_PROXY || env.https_proxy || env.HTTPS_PROXY
+  if (!gyp.opts.proxy && !gyp.opts.cafile && !hasProxyEnv) {
+    return undefined
+  }
+
+  const opts = {}
+  if (gyp.opts.cafile) {
+    const ca = await readCAFile(gyp.opts.cafile)
+    // EnvHttpProxyAgent forwards opts to both its internal Agent (direct) and
+    // ProxyAgent (proxied). Agent reads TLS config from `connect`; ProxyAgent
+    // reads it from `requestTls` (origin) / `proxyTls` (proxy). Set all three
+    // so the custom CA is applied regardless of which path a request takes.
+    opts.connect = { ca }
+    opts.requestTls = { ca }
+    opts.proxyTls = { ca }
+  }
+  if (gyp.opts.proxy) {
+    opts.httpProxy = gyp.opts.proxy
+    opts.httpsProxy = gyp.opts.proxy
+  }
+  if (gyp.opts.noproxy) {
+    opts.noProxy = gyp.opts.noproxy
+  }
+  return new EnvHttpProxyAgent(opts)
 }
 
 async function readCAFile (filename) {
diff --git a/package.json b/package.json
index 71e0790043..92807b96aa 100644
--- a/package.json
+++ b/package.json
@@ -25,12 +25,12 @@
     "env-paths": "^2.2.0",
     "exponential-backoff": "^3.1.1",
     "graceful-fs": "^4.2.6",
-    "make-fetch-happen": "^15.0.0",
     "nopt": "^9.0.0",
     "proc-log": "^6.0.0",
     "semver": "^7.3.5",
     "tar": "^7.5.4",
     "tinyglobby": "^0.2.12",
+    "undici": "^6.25.0",
     "which": "^6.0.0"
   },
   "engines": {
diff --git a/test/test-download.js b/test/test-download.js
index a746c98cc6..4078efe5cd 100644
--- a/test/test-download.js
+++ b/test/test-download.js
@@ -6,6 +6,7 @@ const fs = require('fs/promises')
 const path = require('path')
 const http = require('http')
 const https = require('https')
+const net = require('net')
 const install = require('../lib/install')
 const { download, readCAFile } = require('../lib/download')
 const { FULL_TEST, devDir, platformTimeout } = require('./common')
@@ -69,13 +70,24 @@ describe('download', function () {
   })
 
   it('download over http with proxy', async function () {
-    const server = http.createServer((_, res) => {
+    const server = http.createServer((req, res) => {
+      assert.strictEqual(req.headers['user-agent'], `node-gyp v42 (node ${process.version})`)
       res.end('ok')
     })
 
-    const pserver = http.createServer((req, res) => {
-      assert.strictEqual(req.headers['user-agent'], `node-gyp v42 (node ${process.version})`)
-      res.end('proxy ok')
+    let proxyUsed = false
+    const pserver = http.createServer()
+    pserver.on('connect', (req, clientSocket, head) => {
+      proxyUsed = true
+      const [targetHost, targetPort] = req.url.split(':')
+      const serverSocket = net.connect(targetPort, targetHost, () => {
+        clientSocket.write('HTTP/1.1 200 Connection Established\r\n\r\n')
+        serverSocket.write(head)
+        serverSocket.pipe(clientSocket)
+        clientSocket.pipe(serverSocket)
+      })
+      clientSocket.on('error', () => serverSocket.destroy())
+      serverSocket.on('error', () => clientSocket.destroy())
     })
 
     after(() => Promise.all([
@@ -96,7 +108,56 @@ describe('download', function () {
     }
     const url = `http://${host}:${port}`
     const res = await download(gyp, url)
-    assert.strictEqual(await res.text(), 'proxy ok')
+    assert.strictEqual(await res.text(), 'ok')
+    assert.strictEqual(proxyUsed, true)
+  })
+
+  it('download over https with proxy and custom ca', async function () {
+    const cafile = path.join(__dirname, 'fixtures/ca-proxy.crt')
+    await fs.writeFile(cafile, certs['ca.crt'], 'utf8')
+
+    const server = https.createServer({
+      ca: await readCAFile(cafile),
+      cert: certs['server.crt'],
+      key: certs['server.key']
+    }, (_, res) => res.end('ok'))
+
+    let proxyUsed = false
+    const pserver = http.createServer()
+    pserver.on('connect', (req, clientSocket, head) => {
+      proxyUsed = true
+      const [targetHost, targetPort] = req.url.split(':')
+      const serverSocket = net.connect(targetPort, targetHost, () => {
+        clientSocket.write('HTTP/1.1 200 Connection Established\r\n\r\n')
+        serverSocket.write(head)
+        serverSocket.pipe(clientSocket)
+        clientSocket.pipe(serverSocket)
+      })
+      clientSocket.on('error', () => serverSocket.destroy())
+      serverSocket.on('error', () => clientSocket.destroy())
+    })
+
+    after(async () => {
+      await new Promise((resolve) => server.close(resolve))
+      await new Promise((resolve) => pserver.close(resolve))
+      await fs.unlink(cafile)
+    })
+
+    const host = 'localhost'
+    await new Promise((resolve) => server.listen(0, host, resolve))
+    const { port } = server.address()
+    await new Promise((resolve) => pserver.listen(port + 1, host, resolve))
+    const gyp = {
+      opts: {
+        cafile,
+        proxy: `http://${host}:${port + 1}`,
+        noproxy: 'bad'
+      },
+      version: '42'
+    }
+    const res = await download(gyp, `https://${host}:${port}`)
+    assert.strictEqual(await res.text(), 'ok')
+    assert.strictEqual(proxyUsed, true)
   })
 
   it('download over http with noproxy', async function () {
@@ -105,8 +166,11 @@ describe('download', function () {
       res.end('ok')
     })
 
-    const pserver = http.createServer((_, res) => {
-      res.end('proxy ok')
+    let proxyUsed = false
+    const pserver = http.createServer()
+    pserver.on('connect', (_, socket) => {
+      proxyUsed = true
+      socket.destroy()
     })
 
     after(() => Promise.all([
@@ -128,6 +192,7 @@ describe('download', function () {
     const url = `http://${host}:${port}`
     const res = await download(gyp, url)
     assert.strictEqual(await res.text(), 'ok')
+    assert.strictEqual(proxyUsed, false)
   })
 
   it('download with missing cafile', async function () {

From f4242fb7bf7592d71848bae5c7f8597f2718dc3f Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Tue, 21 Apr 2026 03:52:55 -0400
Subject: [PATCH 532/551] feat: update gyp-next to v0.22.1 (#3295)

* feat: update gyp-next to v0.22.0

* feat: update gyp-next to v0.22.1
---
 gyp/.github/workflows/python_tests.yml   | 21 ++++++++----
 gyp/.github/workflows/release-please.yml |  8 ++---
 gyp/.gitignore                           |  4 +++
 gyp/.release-please-manifest.json        |  2 +-
 gyp/CHANGELOG.md                         | 19 +++++++++++
 gyp/pylib/gyp/MSVSVersion.py             | 14 ++++++--
 gyp/pylib/gyp/__init__.py                |  5 ++-
 gyp/pylib/gyp/generator/msvs.py          |  2 +-
 gyp/pylib/gyp/generator/ninja.py         |  3 +-
 gyp/pylib/gyp/generator/ninja_test.py    | 42 +++++++++++++++---------
 gyp/pylib/gyp/mac_tool.py                |  2 +-
 gyp/pylib/gyp/msvs_emulation.py          |  2 +-
 gyp/pylib/packaging/metadata.py          | 23 ++-----------
 gyp/pylib/packaging/tags.py              | 16 ++-------
 gyp/pyproject.toml                       |  8 +++--
 15 files changed, 96 insertions(+), 75 deletions(-)

diff --git a/gyp/.github/workflows/python_tests.yml b/gyp/.github/workflows/python_tests.yml
index 812d32d7f7..0d13e16387 100644
--- a/gyp/.github/workflows/python_tests.yml
+++ b/gyp/.github/workflows/python_tests.yml
@@ -1,4 +1,3 @@
-# TODO: Enable os: windows-latest
 # TODO: Enable pytest --doctest-modules
 
 name: Python_tests
@@ -8,17 +7,28 @@ on:
   workflow_dispatch:
 
 jobs:
+  Python_lint:
+    runs-on: ubuntu-slim
+    steps:
+      - uses: actions/checkout@v6
+      - name: Lint with ruff  # See pyproject.toml for settings
+        uses: astral-sh/ruff-action@v3
+      - run: ruff format --check --diff
+
   Python_tests:
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
-      max-parallel: 5
       matrix:
-        os: [macos-15-intel, macos-latest, ubuntu-latest] # , windows-latest]
-        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
+        os: [macos-15-intel, macos-latest, ubuntu-latest, windows-latest]
+        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
         include:
           - os: macos-26
             python-version: 3.x
+          - os: ubuntu-24.04-arm  # Ubuntu on ARM
+            python-version: 3.x
+          - os: windows-11-arm  # Windows on ARM
+            python-version: 3.x
     steps:
       - uses: actions/checkout@v6
       - name: Set up Python ${{ matrix.python-version }}
@@ -32,9 +42,6 @@ jobs:
           python -m pip install --upgrade pip
           pip install --editable ".[dev]"
       - run: ./gyp -V && ./gyp --version && gyp -V && gyp --version
-      - name: Lint with ruff  # See pyproject.toml for settings
-        uses: astral-sh/ruff-action@v3
-      - run: ruff format --check --diff
       - name: Test with pytest  # See pyproject.toml for settings
         run: pytest
       # - name: Run doctests with pytest
diff --git a/gyp/.github/workflows/release-please.yml b/gyp/.github/workflows/release-please.yml
index 81f8626c77..0022a3d894 100644
--- a/gyp/.github/workflows/release-please.yml
+++ b/gyp/.github/workflows/release-please.yml
@@ -28,7 +28,7 @@ jobs:
     - name: Build a binary wheel and a source tarball
       run: pipx run build
     - name: Store the distribution packages
-      uses: actions/upload-artifact@v6
+      uses: actions/upload-artifact@v7
       with:
         name: python-package-distributions
         path: dist/
@@ -48,7 +48,7 @@ jobs:
       id-token: write # IMPORTANT: mandatory for trusted publishing
     steps:
     - name: Download all the dists
-      uses: actions/download-artifact@v7
+      uses: actions/download-artifact@v8
       with:
         name: python-package-distributions
         path: dist/
@@ -68,12 +68,12 @@ jobs:
       id-token: write # IMPORTANT: mandatory for sigstore
     steps:
     - name: Download all the dists
-      uses: actions/download-artifact@v7
+      uses: actions/download-artifact@v8
       with:
         name: python-package-distributions
         path: dist/
     - name: Sign the dists with Sigstore
-      uses: sigstore/gh-action-sigstore-python@v3.2.0
+      uses: sigstore/gh-action-sigstore-python@v3.3.0
       with:
         inputs: >-
           ./dist/*.tar.gz
diff --git a/gyp/.gitignore b/gyp/.gitignore
index 5f71dbd435..dcf3cb43ea 100644
--- a/gyp/.gitignore
+++ b/gyp/.gitignore
@@ -144,3 +144,7 @@ static
 
 test/fixtures/out
 *.actual
+*.sln
+*.vcproj
+!test/fixtures/expected-win32/**/*.sln
+!test/fixtures/expected-win32/**/*.vcproj
diff --git a/gyp/.release-please-manifest.json b/gyp/.release-please-manifest.json
index c825abab69..9242a4094d 100644
--- a/gyp/.release-please-manifest.json
+++ b/gyp/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "0.21.1"
+    ".": "0.22.1"
 }
diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md
index 9a14685447..c7ddd862d1 100644
--- a/gyp/CHANGELOG.md
+++ b/gyp/CHANGELOG.md
@@ -1,5 +1,24 @@
 # Changelog
 
+## [0.22.1](https://github.com/nodejs/gyp-next/compare/v0.22.0...v0.22.1) (2026-04-21)
+
+
+### Bug Fixes
+
+* use floor division when escaping command-line arguments ([#338](https://github.com/nodejs/gyp-next/issues/338)) ([cadca24](https://github.com/nodejs/gyp-next/commit/cadca2416dc13e117e035ad6dd2b471a86d0430f))
+
+## [0.22.0](https://github.com/nodejs/gyp-next/compare/v0.21.1...v0.22.0) (2026-04-02)
+
+
+### Features
+
+* Windows ARM64 target architecture support ([#331](https://github.com/nodejs/gyp-next/issues/331)) ([652a346](https://github.com/nodejs/gyp-next/commit/652a346bbd3b077a4b08a3c37d48100ce200758a))
+
+
+### Bug Fixes
+
+* drop deprecated Python module pkg_resources ([#333](https://github.com/nodejs/gyp-next/issues/333)) ([5b180d5](https://github.com/nodejs/gyp-next/commit/5b180d52d03aff062bdea1ad0209b82271c7eb4a))
+
 ## [0.21.1](https://github.com/nodejs/gyp-next/compare/v0.21.0...v0.21.1) (2026-01-24)
 
 
diff --git a/gyp/pylib/gyp/MSVSVersion.py b/gyp/pylib/gyp/MSVSVersion.py
index 2d8e4ceab9..02e6e7ed92 100644
--- a/gyp/pylib/gyp/MSVSVersion.py
+++ b/gyp/pylib/gyp/MSVSVersion.py
@@ -87,7 +87,7 @@ def DefaultToolset(self):
     def _SetupScriptInternal(self, target_arch):
         """Returns a command (with arguments) to be used to set up the
         environment."""
-        assert target_arch in ("x86", "x64"), "target_arch not supported"
+        assert target_arch in ("x86", "x64", "arm64"), "target_arch not supported"
         # If WindowsSDKDir is set and SetEnv.Cmd exists then we are using the
         # depot_tools build tools and should run SetEnv.Cmd to set up the
         # environment. The check for WindowsSDKDir alone is not sufficient because
@@ -109,8 +109,16 @@ def _SetupScriptInternal(self, target_arch):
             )
 
             # Always use a native executable, cross-compiling if necessary.
-            host_arch = "amd64" if is_host_arch_x64 else "x86"
-            msvc_target_arch = "amd64" if target_arch == "x64" else "x86"
+            host_arch = (
+                "amd64"
+                if is_host_arch_x64
+                else (
+                    "arm64"
+                    if os.environ.get("PROCESSOR_ARCHITECTURE") == "ARM64"
+                    else "x86"
+                )
+            )
+            msvc_target_arch = {"x64": "amd64"}.get(target_arch, target_arch)
             arg = host_arch
             if host_arch != msvc_target_arch:
                 arg += "_" + msvc_target_arch
diff --git a/gyp/pylib/gyp/__init__.py b/gyp/pylib/gyp/__init__.py
index 3a70cf076c..c0a2637e94 100755
--- a/gyp/pylib/gyp/__init__.py
+++ b/gyp/pylib/gyp/__init__.py
@@ -13,6 +13,7 @@
 import shlex
 import sys
 import traceback
+from importlib.metadata import version
 
 import gyp.input
 from gyp.common import GypError
@@ -491,9 +492,7 @@ def gyp_main(args):
 
     options, build_files_arg = parser.parse_args(args)
     if options.version:
-        import pkg_resources  # noqa: PLC0415
-
-        print(f"v{pkg_resources.get_distribution('gyp-next').version}")
+        print(f"v{version('gyp-next')}")
         return 0
     build_files = build_files_arg
 
diff --git a/gyp/pylib/gyp/generator/msvs.py b/gyp/pylib/gyp/generator/msvs.py
index 0f14c05504..42bee33d9b 100644
--- a/gyp/pylib/gyp/generator/msvs.py
+++ b/gyp/pylib/gyp/generator/msvs.py
@@ -857,7 +857,7 @@ def _EscapeCommandLineArgumentForMSBuild(s):
     """Escapes a Windows command-line argument for use by MSBuild."""
 
     def _Replace(match):
-        return (len(match.group(1)) / 2 * 4) * "\\" + '\\"'
+        return (len(match.group(1)) // 2 * 4) * "\\" + '\\"'
 
     # Escape all quotes so that they are interpreted literally.
     s = quote_replacer_regex2.sub(_Replace, s)
diff --git a/gyp/pylib/gyp/generator/ninja.py b/gyp/pylib/gyp/generator/ninja.py
index 4eac6cdb27..3ceaf470ce 100644
--- a/gyp/pylib/gyp/generator/ninja.py
+++ b/gyp/pylib/gyp/generator/ninja.py
@@ -246,7 +246,7 @@ def __init__(
         if flavor == "win":
             # See docstring of msvs_emulation.GenerateEnvironmentFiles().
             self.win_env = {}
-            for arch in ("x86", "x64"):
+            for arch in ("x86", "x64", "arm64"):
                 self.win_env[arch] = "environment." + arch
 
         # Relative path from build output dir to base dir.
@@ -2339,6 +2339,7 @@ def GenerateOutputForConfig(target_list, target_dicts, data, params, config_name
         master_ninja.variable("rc", "rc.exe")
         master_ninja.variable("ml_x86", "ml.exe")
         master_ninja.variable("ml_x64", "ml64.exe")
+        master_ninja.variable("ml_arm64", "armasm64.exe")
         master_ninja.variable("mt", "mt.exe")
     else:
         master_ninja.variable("ld", CommandWithWrapper("LINK", wrappers, ld))
diff --git a/gyp/pylib/gyp/generator/ninja_test.py b/gyp/pylib/gyp/generator/ninja_test.py
index 616bc7aaf0..8b590af8eb 100644
--- a/gyp/pylib/gyp/generator/ninja_test.py
+++ b/gyp/pylib/gyp/generator/ninja_test.py
@@ -11,26 +11,36 @@
 from pathlib import Path
 
 from gyp.generator import ninja
+from gyp.MSVSVersion import SelectVisualStudioVersion
+
+
+def _has_visual_studio():
+    """Check if Visual Studio can be detected by gyp's registry-based detection."""
+    if not sys.platform.startswith("win"):
+        return False
+    try:
+        SelectVisualStudioVersion("auto", allow_fallback=False)
+        return True
+    except ValueError:
+        return False
 
 
 class TestPrefixesAndSuffixes(unittest.TestCase):
+    @unittest.skipUnless(
+        _has_visual_studio(),
+        "requires Windows with a Visual Studio installation detected via the registry",
+    )
     def test_BinaryNamesWindows(self):
-        # These cannot run on non-Windows as they require a VS installation to
-        # correctly handle variable expansion.
-        if sys.platform.startswith("win"):
-            writer = ninja.NinjaWriter(
-                "foo", "wee", ".", ".", "build.ninja", ".", "build.ninja", "win"
-            )
-            spec = {"target_name": "wee"}
-            self.assertTrue(
-                writer.ComputeOutputFileName(spec, "executable").endswith(".exe")
-            )
-            self.assertTrue(
-                writer.ComputeOutputFileName(spec, "shared_library").endswith(".dll")
-            )
-            self.assertTrue(
-                writer.ComputeOutputFileName(spec, "static_library").endswith(".lib")
-            )
+        writer = ninja.NinjaWriter(
+            "foo", "wee", ".", ".", "build.ninja", ".", "build.ninja", "win"
+        )
+        spec = {"target_name": "wee"}
+        for key, ext in {
+            "executable": ".exe",
+            "shared_library": ".dll",
+            "static_library": ".lib",
+        }.items():
+            self.assertTrue(writer.ComputeOutputFileName(spec, key).endswith(ext))
 
     def test_BinaryNamesLinux(self):
         writer = ninja.NinjaWriter(
diff --git a/gyp/pylib/gyp/mac_tool.py b/gyp/pylib/gyp/mac_tool.py
index 3710178e11..4c38f0586c 100755
--- a/gyp/pylib/gyp/mac_tool.py
+++ b/gyp/pylib/gyp/mac_tool.py
@@ -545,7 +545,7 @@ def _FindProvisioningProfile(self, profile, bundle_identifier):
         # If the user has multiple provisioning profiles installed that can be
         # used for ${bundle_identifier}, pick the most specific one (ie. the
         # provisioning profile whose pattern is the longest).
-        selected_key = max(valid_provisioning_profiles, key=lambda v: len(v))
+        selected_key = max(valid_provisioning_profiles, key=len)
         return valid_provisioning_profiles[selected_key]
 
     def _LoadProvisioningProfile(self, profile_path):
diff --git a/gyp/pylib/gyp/msvs_emulation.py b/gyp/pylib/gyp/msvs_emulation.py
index 7c461a8fdf..f1c1581981 100644
--- a/gyp/pylib/gyp/msvs_emulation.py
+++ b/gyp/pylib/gyp/msvs_emulation.py
@@ -1174,7 +1174,7 @@ def GenerateEnvironmentFiles(
     meet your requirement (e.g. for custom toolchains), you can pass
     "-G ninja_use_custom_environment_files" to the gyp to suppress file
     generation and use custom environment files prepared by yourself."""
-    archs = ("x86", "x64")
+    archs = ("x86", "x64", "arm64")
     if generator_flags.get("ninja_use_custom_environment_files", 0):
         cl_paths = {}
         for arch in archs:
diff --git a/gyp/pylib/packaging/metadata.py b/gyp/pylib/packaging/metadata.py
index 43f5c5b30d..38fa645b4c 100644
--- a/gyp/pylib/packaging/metadata.py
+++ b/gyp/pylib/packaging/metadata.py
@@ -21,27 +21,10 @@
 from . import requirements, specifiers, utils, version as version_module
 
 T = typing.TypeVar("T")
-if sys.version_info[:2] >= (3, 8):  # pragma: no cover
-    from typing import Literal, TypedDict
-else:  # pragma: no cover
-    if typing.TYPE_CHECKING:
-        from typing_extensions import Literal, TypedDict
-    else:
-        try:
-            from typing_extensions import Literal, TypedDict
-        except ImportError:
-
-            class Literal:
-                def __init_subclass__(*_args, **_kwargs):
-                    pass
-
-            class TypedDict:
-                def __init_subclass__(*_args, **_kwargs):
-                    pass
-
+from typing import Literal, TypedDict
 
 try:
-    ExceptionGroup
+    ExceptionGroup  # Added in Python 3.11+
 except NameError:  # pragma: no cover
 
     class ExceptionGroup(Exception):  # noqa: N818
@@ -504,7 +487,7 @@ def __set_name__(self, _owner: "Metadata", name: str) -> None:
         self.raw_name = _RAW_TO_EMAIL_MAPPING[name]
 
     def __get__(self, instance: "Metadata", _owner: Type["Metadata"]) -> T:
-        # With Python 3.8, the caching can be replaced with functools.cached_property().
+        # With Python 3.8+, the caching can be replaced with functools.cached_property().
         # No need to check the cache as attribute lookup will resolve into the
         # instance's __dict__ before __get__ is called.
         cache = instance.__dict__
diff --git a/gyp/pylib/packaging/tags.py b/gyp/pylib/packaging/tags.py
index 37f33b1ef8..f1da2b96d3 100644
--- a/gyp/pylib/packaging/tags.py
+++ b/gyp/pylib/packaging/tags.py
@@ -127,10 +127,8 @@ def _normalize_string(string: str) -> str:
 def _abi3_applies(python_version: PythonVersion) -> bool:
     """
     Determine if the Python version supports abi3.
-
-    PEP 384 was first implemented in Python 3.2.
     """
-    return len(python_version) > 1 and tuple(python_version) >= (3, 2)
+    return len(python_version) > 1
 
 
 def _cpython_abis(py_version: PythonVersion, warn: bool = False) -> List[str]:
@@ -146,17 +144,7 @@ def _cpython_abis(py_version: PythonVersion, warn: bool = False) -> List[str]:
     has_ext = "_d.pyd" in EXTENSION_SUFFIXES
     if with_debug or (with_debug is None and (has_refcount or has_ext)):
         debug = "d"
-    if py_version < (3, 8):
-        with_pymalloc = _get_config_var("WITH_PYMALLOC", warn)
-        if with_pymalloc or with_pymalloc is None:
-            pymalloc = "m"
-        if py_version < (3, 3):
-            unicode_size = _get_config_var("Py_UNICODE_SIZE", warn)
-            if unicode_size == 4 or (
-                unicode_size is None and sys.maxunicode == 0x10FFFF
-            ):
-                ucs4 = "u"
-    elif debug:
+    if debug:
         # Debug builds can also load "normal" extension modules.
         # We can also assume no UCS-4 or pymalloc requirement.
         abis.append(f"cp{version}")
diff --git a/gyp/pyproject.toml b/gyp/pyproject.toml
index fa30c8cf96..239bef7844 100644
--- a/gyp/pyproject.toml
+++ b/gyp/pyproject.toml
@@ -4,14 +4,14 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "gyp-next"
-version = "0.21.1"
+version = "0.22.1"
 authors = [
   { name="Node.js contributors", email="ryzokuken@disroot.org" },
 ]
 description = "A fork of the GYP build system for use in the Node.js projects"
 readme = "README.md"
 license = { file="LICENSE" }
-requires-python = ">=3.8"
+requires-python = ">=3.9"
 dependencies = ["packaging>=24.0", "setuptools>=69.5.1"]
 classifiers = [
     "Development Status :: 3 - Alpha",
@@ -21,10 +21,12 @@ classifiers = [
     "Natural Language :: English",
     "Programming Language :: Python",
     "Programming Language :: Python :: 3",
-    "Programming Language :: Python :: 3.8",
     "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
 ]
 
 [project.optional-dependencies]

From 154a27ef069deafc8f680f3f77452e76343546f8 Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Tue, 21 Apr 2026 04:19:19 -0400
Subject: [PATCH 533/551] chore(main): release 12.3.0 (#3277)

---
 .release-please-manifest.json |  2 +-
 CHANGELOG.md                  | 19 +++++++++++++++++++
 package.json                  |  2 +-
 3 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
index 41c2c8154a..af7e1d8db1 100644
--- a/.release-please-manifest.json
+++ b/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "12.2.0"
+    ".": "12.3.0"
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4a83b4dfcd..1b02c2184a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,24 @@
 # Changelog
 
+## [12.3.0](https://github.com/nodejs/node-gyp/compare/v12.2.0...v12.3.0) (2026-04-21)
+
+
+### Features
+
+* replace make-fetch-happen with built-in fetch ([#3302](https://github.com/nodejs/node-gyp/issues/3302)) ([393ec2b](https://github.com/nodejs/node-gyp/commit/393ec2be480195b585768eb18ac2d92b858b12db))
+* update gyp-next to v0.22.1 ([#3295](https://github.com/nodejs/node-gyp/issues/3295)) ([f4242fb](https://github.com/nodejs/node-gyp/commit/f4242fb7bf7592d71848bae5c7f8597f2718dc3f))
+
+
+### Bug Fixes
+
+* Switch to URL instead of url.resolve ([#3256](https://github.com/nodejs/node-gyp/issues/3256)) ([#3263](https://github.com/nodejs/node-gyp/issues/3263)) ([46d7576](https://github.com/nodejs/node-gyp/commit/46d75768018d10dc4b5693b35cf58f502425cbbe))
+
+
+### Core
+
+* **deps-dev:** bump neostandard from 0.12.2 to 0.13.0 ([#3289](https://github.com/nodejs/node-gyp/issues/3289)) ([19da158](https://github.com/nodejs/node-gyp/commit/19da1583b3876dce8c97263b168d9dfef637b76b))
+* improve Add-Type with -IgnoreWarnings ([#3280](https://github.com/nodejs/node-gyp/issues/3280)) ([1381458](https://github.com/nodejs/node-gyp/commit/13814583a476c85a84d5ac902c7ffa310120cb88))
+
 ## [12.2.0](https://github.com/nodejs/node-gyp/compare/v12.1.0...v12.2.0) (2026-01-26)
 
 
diff --git a/package.json b/package.json
index 92807b96aa..29d95ad41b 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "12.2.0",
+  "version": "12.3.0",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From b2fcdcdd6aebdf0ab6d7d296fcabccf188883e01 Mon Sep 17 00:00:00 2001
From: David Sanders 
Date: Wed, 22 Apr 2026 06:24:41 -0700
Subject: [PATCH 534/551] ci: add workflow_dispatch trigger to tests workflow
 (#3299)

---
 .github/workflows/tests.yml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 8d679a3157..d06d9a9aad 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -8,6 +8,7 @@ on:
   pull_request:
     branches: [ main ]
   workflow_call:
+  workflow_dispatch:
 
 permissions:
   contents: read # to fetch code (actions/checkout)

From 0793489b961c2cbf6c6b2182d968d0c262c3b573 Mon Sep 17 00:00:00 2001
From: David Sanders 
Date: Sat, 25 Apr 2026 09:40:10 -0700
Subject: [PATCH 535/551] fix: retry downloads on retryable errors (#3308)

Assisted-by: Claude Opus 4.6

Signed-off-by: David Sanders 
---
 lib/download.js       |  6 +++---
 test/test-download.js | 26 ++++++++++++++++++++++++++
 2 files changed, 29 insertions(+), 3 deletions(-)

diff --git a/lib/download.js b/lib/download.js
index a9866d8a63..0d9d2e32fa 100644
--- a/lib/download.js
+++ b/lib/download.js
@@ -1,5 +1,5 @@
 const { Readable } = require('stream')
-const { EnvHttpProxyAgent } = require('undici')
+const { Agent, EnvHttpProxyAgent, RetryAgent } = require('undici')
 const { promises: fs } = require('graceful-fs')
 const log = require('./log')
 
@@ -48,7 +48,7 @@ async function createDispatcher (gyp) {
   const env = process.env
   const hasProxyEnv = env.http_proxy || env.HTTP_PROXY || env.https_proxy || env.HTTPS_PROXY
   if (!gyp.opts.proxy && !gyp.opts.cafile && !hasProxyEnv) {
-    return undefined
+    return new RetryAgent(new Agent(), { maxRetries: 3 })
   }
 
   const opts = {}
@@ -69,7 +69,7 @@ async function createDispatcher (gyp) {
   if (gyp.opts.noproxy) {
     opts.noProxy = gyp.opts.noproxy
   }
-  return new EnvHttpProxyAgent(opts)
+  return new RetryAgent(new EnvHttpProxyAgent(opts), { maxRetries: 3 })
 }
 
 async function readCAFile (filename) {
diff --git a/test/test-download.js b/test/test-download.js
index 4078efe5cd..756142f623 100644
--- a/test/test-download.js
+++ b/test/test-download.js
@@ -218,6 +218,32 @@ describe('download', function () {
     assert.notStrictEqual(cas[0], cas[1])
   })
 
+  it('download will retry on ECONNRESET', async function () {
+    let requestCount = 0
+    const server = http.createServer((req, res) => {
+      requestCount++
+      if (requestCount < 3) {
+        req.socket.destroy()
+        return
+      }
+      res.end('ok')
+    })
+
+    after(() => new Promise((resolve) => server.close(resolve)))
+
+    const host = 'localhost'
+    await new Promise((resolve) => server.listen(0, host, resolve))
+    const { port } = server.address()
+    const gyp = {
+      opts: {},
+      version: '42'
+    }
+    const url = `http://${host}:${port}`
+    const res = await download(gyp, url)
+    assert.strictEqual(await res.text(), 'ok')
+    assert.ok(requestCount >= 2, `expected at least 2 requests but got ${requestCount}`)
+  })
+
   // only run this test if we are running a version of Node with predictable version path behavior
 
   it('download headers (actual)', async function () {

From 87c8c59f93f99ae650f077d6e99af64043f23319 Mon Sep 17 00:00:00 2001
From: Ilyas Shabi 
Date: Tue, 12 May 2026 04:32:32 +0200
Subject: [PATCH 536/551] support Node.js 26 (#3311)

---
 .github/workflows/tests.yml | 2 +-
 lib/download.js             | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index d06d9a9aad..de31613094 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -98,7 +98,7 @@ jobs:
       matrix:
         os: [macos-latest, ubuntu-latest, windows-latest]
         python: ["3.10", "3.12", "3.14"]
-        node: [20.x, 22.x, 24.x]
+        node: [20.x, 22.x, 24.x, 26.x]
         include:
           - os: macos-15-intel  # macOS on Intel
             python: "3.14"
diff --git a/lib/download.js b/lib/download.js
index 0d9d2e32fa..dfaca798cf 100644
--- a/lib/download.js
+++ b/lib/download.js
@@ -1,5 +1,5 @@
 const { Readable } = require('stream')
-const { Agent, EnvHttpProxyAgent, RetryAgent } = require('undici')
+const { Agent, EnvHttpProxyAgent, RetryAgent, fetch } = require('undici')
 const { promises: fs } = require('graceful-fs')
 const log = require('./log')
 

From 0e65639baa8b64b5fecec3820a4f07dc7c4a42ad Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Tue, 12 May 2026 09:24:11 +0545
Subject: [PATCH 537/551] fix: test on Node.js v26 (#3314)

* fix: test on Node.js v26

* Add fetch import from undici to download.js
---
 .github/workflows/tests.yml | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index de31613094..f47116dcec 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -102,13 +102,13 @@ jobs:
         include:
           - os: macos-15-intel  # macOS on Intel
             python: "3.14"
-            node: 24.x
+            node: 26.x
           - os: ubuntu-24.04-arm  # Ubuntu on ARM
             python: "3.14"
-            node: 24.x
+            node: 26.x
           - os: windows-11-arm  # Windows on ARM
             python: "3.14"
-            node: 24.x
+            node: 26.x
     name: ${{ matrix.os }} - ${{ matrix.python }} - ${{ matrix.node }}
     runs-on: ${{ matrix.os }}
     env:
@@ -171,10 +171,10 @@ jobs:
         shell: bash
         run: npm test --python="${pythonLocation}/python"
         env:
-          FULL_TEST: ${{ (matrix.node == '24.x' && matrix.python == '3.14') && '1' || '0' }}
+          FULL_TEST: ${{ (matrix.node == '26.x' && matrix.python == '3.14') && '1' || '0' }}
       - name: Run Tests (Windows)
         if: runner.os == 'Windows'
         shell: bash # Building wasm on Windows requires using make generator, it only works in bash
         run: npm run test --python="${pythonLocation}\\python.exe"
         env:
-          FULL_TEST: ${{ (matrix.node == '24.x' && matrix.python == '3.14') && '1' || '0' }}
+          FULL_TEST: ${{ (matrix.node == '26.x' && matrix.python == '3.14') && '1' || '0' }}

From 5c0ec47797a18d3e65f0c4c83e5812289191cfe1 Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Tue, 12 May 2026 10:44:29 +0545
Subject: [PATCH 538/551] fix: stop testing end-of-life Node.js v20 (#3315)

Removed Node.js version 20.x from the test matrix.
---
 .github/workflows/tests.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index f47116dcec..7bbf6ea711 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -98,7 +98,7 @@ jobs:
       matrix:
         os: [macos-latest, ubuntu-latest, windows-latest]
         python: ["3.10", "3.12", "3.14"]
-        node: [20.x, 22.x, 24.x, 26.x]
+        node: [22.x, 24.x, 26.x]
         include:
           - os: macos-15-intel  # macOS on Intel
             python: "3.14"

From 8ea71e5a0d5577f8f780df7597fa1f94089e1be4 Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Fri, 15 May 2026 11:44:01 -0400
Subject: [PATCH 539/551] feat: update gyp-next to v0.22.2 (#3316)

---
 gyp/.github/workflows/python_tests.yml | 3 ++-
 gyp/.release-please-manifest.json      | 2 +-
 gyp/CHANGELOG.md                       | 7 +++++++
 gyp/pyproject.toml                     | 8 ++++----
 4 files changed, 14 insertions(+), 6 deletions(-)

diff --git a/gyp/.github/workflows/python_tests.yml b/gyp/.github/workflows/python_tests.yml
index 0d13e16387..47f6e5e77c 100644
--- a/gyp/.github/workflows/python_tests.yml
+++ b/gyp/.github/workflows/python_tests.yml
@@ -8,12 +8,13 @@ on:
 
 jobs:
   Python_lint:
-    runs-on: ubuntu-slim
+    runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v6
       - name: Lint with ruff  # See pyproject.toml for settings
         uses: astral-sh/ruff-action@v3
       - run: ruff format --check --diff
+      - uses: wagoid/commitlint-github-action@v6
 
   Python_tests:
     runs-on: ${{ matrix.os }}
diff --git a/gyp/.release-please-manifest.json b/gyp/.release-please-manifest.json
index 9242a4094d..762fc35f4c 100644
--- a/gyp/.release-please-manifest.json
+++ b/gyp/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "0.22.1"
+    ".": "0.22.2"
 }
diff --git a/gyp/CHANGELOG.md b/gyp/CHANGELOG.md
index c7ddd862d1..c6686c4784 100644
--- a/gyp/CHANGELOG.md
+++ b/gyp/CHANGELOG.md
@@ -1,5 +1,12 @@
 # Changelog
 
+## [0.22.2](https://github.com/nodejs/gyp-next/compare/v0.22.1...v0.22.2) (2026-04-22)
+
+
+### Bug Fixes
+
+* **build:** Use SPDX license expression ([#310](https://github.com/nodejs/gyp-next/issues/310)) ([ce0da4a](https://github.com/nodejs/gyp-next/commit/ce0da4a6dc1068b73e55e6a58a236bf69f2fed4d))
+
 ## [0.22.1](https://github.com/nodejs/gyp-next/compare/v0.22.0...v0.22.1) (2026-04-21)
 
 
diff --git a/gyp/pyproject.toml b/gyp/pyproject.toml
index 239bef7844..487cb75002 100644
--- a/gyp/pyproject.toml
+++ b/gyp/pyproject.toml
@@ -4,20 +4,20 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "gyp-next"
-version = "0.22.1"
+version = "0.22.2"
 authors = [
   { name="Node.js contributors", email="ryzokuken@disroot.org" },
 ]
 description = "A fork of the GYP build system for use in the Node.js projects"
 readme = "README.md"
-license = { file="LICENSE" }
+license = "BSD-3-Clause"
+license-files = ["LICENSE"]
 requires-python = ">=3.9"
-dependencies = ["packaging>=24.0", "setuptools>=69.5.1"]
+dependencies = ["packaging>=24.0", "setuptools>=77.0.3"]
 classifiers = [
     "Development Status :: 3 - Alpha",
     "Environment :: Console",
     "Intended Audience :: Developers",
-    "License :: OSI Approved :: BSD License",
     "Natural Language :: English",
     "Programming Language :: Python",
     "Programming Language :: Python :: 3",

From ee9b5448f61f66629f0f5faa3ef2f57311797a9d Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Fri, 5 Jun 2026 03:04:23 -0400
Subject: [PATCH 540/551] chore(main): release 12.4.0 (#3306)

---
 .release-please-manifest.json |  2 +-
 CHANGELOG.md                  | 19 +++++++++++++++++++
 package.json                  |  2 +-
 3 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
index af7e1d8db1..7cf5faf711 100644
--- a/.release-please-manifest.json
+++ b/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "12.3.0"
+    ".": "12.4.0"
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1b02c2184a..83881ed2a3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,24 @@
 # Changelog
 
+## [12.4.0](https://github.com/nodejs/node-gyp/compare/v12.3.0...v12.4.0) (2026-05-15)
+
+
+### Features
+
+* update gyp-next to v0.22.2 ([#3316](https://github.com/nodejs/node-gyp/issues/3316)) ([8ea71e5](https://github.com/nodejs/node-gyp/commit/8ea71e5a0d5577f8f780df7597fa1f94089e1be4))
+
+
+### Bug Fixes
+
+* retry downloads on retryable errors ([#3308](https://github.com/nodejs/node-gyp/issues/3308)) ([0793489](https://github.com/nodejs/node-gyp/commit/0793489b961c2cbf6c6b2182d968d0c262c3b573))
+* stop testing end-of-life Node.js v20 ([#3315](https://github.com/nodejs/node-gyp/issues/3315)) ([5c0ec47](https://github.com/nodejs/node-gyp/commit/5c0ec47797a18d3e65f0c4c83e5812289191cfe1))
+* test on Node.js v26 ([#3314](https://github.com/nodejs/node-gyp/issues/3314)) ([0e65639](https://github.com/nodejs/node-gyp/commit/0e65639baa8b64b5fecec3820a4f07dc7c4a42ad))
+
+
+### Miscellaneous
+
+* add workflow_dispatch trigger to tests workflow ([#3299](https://github.com/nodejs/node-gyp/issues/3299)) ([b2fcdcd](https://github.com/nodejs/node-gyp/commit/b2fcdcdd6aebdf0ab6d7d296fcabccf188883e01))
+
 ## [12.3.0](https://github.com/nodejs/node-gyp/compare/v12.2.0...v12.3.0) (2026-04-21)
 
 
diff --git a/package.json b/package.json
index 29d95ad41b..7f579406da 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "12.3.0",
+  "version": "12.4.0",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From f6a5e458e819f05ddfbba5cbd5589a7a1c80128e Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Fri, 5 Jun 2026 10:22:34 +0200
Subject: [PATCH 541/551] build(deps): bump actions/github-script from 8 to 9
 (#3297)

Bumps [actions/github-script](https://github.com/actions/github-script) from 8 to 9.
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](https://github.com/actions/github-script/compare/v8...v9)

---
updated-dependencies:
- dependency-name: actions/github-script
  dependency-version: '9'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 .github/workflows/update-gyp-next.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/update-gyp-next.yml b/.github/workflows/update-gyp-next.yml
index 44936014f9..c1b2f8f9b2 100644
--- a/.github/workflows/update-gyp-next.yml
+++ b/.github/workflows/update-gyp-next.yml
@@ -21,7 +21,7 @@ jobs:
         with:
           persist-credentials: false
 
-      - uses: actions/github-script@v8
+      - uses: actions/github-script@v9
         id: get-gyp-next-version
         with:
           script: |

From b2cde802487f3b9efd726a1a1cb43650731f9958 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Fri, 5 Jun 2026 10:26:07 +0200
Subject: [PATCH 542/551] build(deps): bump googleapis/release-please-action
 from 4 to 5 (#3307)

Bumps [googleapis/release-please-action](https://github.com/googleapis/release-please-action) from 4 to 5.
- [Release notes](https://github.com/googleapis/release-please-action/releases)
- [Changelog](https://github.com/googleapis/release-please-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/release-please-action/compare/v4...v5)

---
updated-dependencies:
- dependency-name: googleapis/release-please-action
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 .github/workflows/release-please.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml
index 70050b9a04..a792c76633 100644
--- a/.github/workflows/release-please.yml
+++ b/.github/workflows/release-please.yml
@@ -15,7 +15,7 @@ jobs:
       release_created:  ${{ steps.release.outputs.release_created }}
     runs-on: ubuntu-latest
     steps:
-      - uses: googleapis/release-please-action@v4
+      - uses: googleapis/release-please-action@v5
         id: release
         with:
           token: ${{ secrets.GH_USER_TOKEN }}

From b792b8e321e6e56d46c32f4b9adb15e4085c3797 Mon Sep 17 00:00:00 2001
From: Michael Smith 
Date: Tue, 19 May 2026 08:40:40 -0700
Subject: [PATCH 543/551] feat!: bump to new node engine range

BREAKING CHANGE: `node-gyp` now supports node `^22.22.2 || ^24.15.0 || >=26.0.0`
---
 package.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package.json b/package.json
index 7f579406da..710ece6d12 100644
--- a/package.json
+++ b/package.json
@@ -34,7 +34,7 @@
     "which": "^6.0.0"
   },
   "engines": {
-    "node": "^20.17.0 || >=22.9.0"
+    "node": "^22.22.2 || ^24.15.0 || >=26.0.0"
   },
   "devDependencies": {
     "bindings": "^1.5.0",

From 327424ad630dd7007771b19f7b77691b2156a556 Mon Sep 17 00:00:00 2001
From: Michael Smith 
Date: Tue, 19 May 2026 08:48:24 -0700
Subject: [PATCH 544/551] deps: nopt@10.0.0

---
 package.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package.json b/package.json
index 710ece6d12..2e82c9064f 100644
--- a/package.json
+++ b/package.json
@@ -25,7 +25,7 @@
     "env-paths": "^2.2.0",
     "exponential-backoff": "^3.1.1",
     "graceful-fs": "^4.2.6",
-    "nopt": "^9.0.0",
+    "nopt": "^10.0.0",
     "proc-log": "^6.0.0",
     "semver": "^7.3.5",
     "tar": "^7.5.4",

From f6c296b2952e23b7e690b7148c9dc32b25a26ee2 Mon Sep 17 00:00:00 2001
From: Michael Smith 
Date: Tue, 19 May 2026 08:48:31 -0700
Subject: [PATCH 545/551] deps: proc-log@7.0.0

---
 package.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package.json b/package.json
index 2e82c9064f..e1700ca51d 100644
--- a/package.json
+++ b/package.json
@@ -26,7 +26,7 @@
     "exponential-backoff": "^3.1.1",
     "graceful-fs": "^4.2.6",
     "nopt": "^10.0.0",
-    "proc-log": "^6.0.0",
+    "proc-log": "^7.0.0",
     "semver": "^7.3.5",
     "tar": "^7.5.4",
     "tinyglobby": "^0.2.12",

From 11a8b109f805f13129302508c219607167a8e9ea Mon Sep 17 00:00:00 2001
From: Michael Smith 
Date: Tue, 19 May 2026 08:48:41 -0700
Subject: [PATCH 546/551] deps: which@7.0.0

---
 package.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package.json b/package.json
index e1700ca51d..0b485a4657 100644
--- a/package.json
+++ b/package.json
@@ -31,7 +31,7 @@
     "tar": "^7.5.4",
     "tinyglobby": "^0.2.12",
     "undici": "^6.25.0",
-    "which": "^6.0.0"
+    "which": "^7.0.0"
   },
   "engines": {
     "node": "^22.22.2 || ^24.15.0 || >=26.0.0"

From d30bb6e6fdaf46374de9b9bfe1bf54f49337b052 Mon Sep 17 00:00:00 2001
From: Michael Smith 
Date: Wed, 10 Jun 2026 07:48:33 -0700
Subject: [PATCH 547/551] fix: disable LTO for addon builds on Windows (#3331)

---
 lib/create-config-gypi.js                     |  5 +++++
 .../fixtures/win-lto/include/node/config.gypi |  8 ++++++++
 test/test-create-config-gypi.js               | 19 +++++++++++++++++++
 3 files changed, 32 insertions(+)
 create mode 100644 test/fixtures/win-lto/include/node/config.gypi

diff --git a/lib/create-config-gypi.js b/lib/create-config-gypi.js
index 01a820e9f2..d471da5169 100644
--- a/lib/create-config-gypi.js
+++ b/lib/create-config-gypi.js
@@ -99,6 +99,11 @@ async function getCurrentConfigGypi ({ gyp, nodeDir, vsInfo, python }) {
     if (config.variables.clang === 1) {
       config.variables.clang = 0
     }
+    // disable LTO for addon builds. node release builds may enable (thin) LTO,
+    // which leaks clang/lld-only flags like -flto=thin and /opt:lldltojobs into
+    // addons built with the default MSVC toolchain, which rejects those flags
+    variables.enable_lto = 'false'
+    variables.enable_thin_lto = 'false'
   }
 
   // loop through the rest of the opts and add the unknown ones as variables.
diff --git a/test/fixtures/win-lto/include/node/config.gypi b/test/fixtures/win-lto/include/node/config.gypi
new file mode 100644
index 0000000000..2e432eb842
--- /dev/null
+++ b/test/fixtures/win-lto/include/node/config.gypi
@@ -0,0 +1,8 @@
+# Test configuration simulating a Node.js Windows release built with Thin LTO
+{
+  'variables': {
+    'enable_lto': 'true',
+    'enable_thin_lto': 'true',
+    'lto_jobs': '2'
+  }
+}
diff --git a/test/test-create-config-gypi.js b/test/test-create-config-gypi.js
index 3c77b87859..602fd3d678 100644
--- a/test/test-create-config-gypi.js
+++ b/test/test-create-config-gypi.js
@@ -52,6 +52,25 @@ describe('create-config-gypi', function () {
     assert.strictEqual(config.variables.build_with_electron, undefined)
   })
 
+  it('config.gypi disables LTO for addon builds on Windows', async function () {
+    const nodeDir = path.join(__dirname, 'fixtures', 'win-lto')
+
+    const prog = gyp()
+    prog.parseArgv(['_', '_', `--nodedir=${nodeDir}`])
+
+    const originalPlatform = process.platform
+    Object.defineProperty(process, 'platform', { value: 'win32' })
+    try {
+      const config = await getCurrentConfigGypi({ gyp: prog, nodeDir, vsInfo: {} })
+      // thin LTO leaks clang/lld-only flags into the MSVC addon build, so it
+      // must be disabled regardless of how node itself was built
+      assert.strictEqual(config.variables.enable_lto, 'false')
+      assert.strictEqual(config.variables.enable_thin_lto, 'false')
+    } finally {
+      Object.defineProperty(process, 'platform', { value: originalPlatform })
+    }
+  })
+
   it('config.gypi parsing', function () {
     const str = "# Some comments\n{'variables': {'multiline': 'A'\n'B'}}"
     const config = parseConfigGypi(str)

From 6fb6c11eae1258eddd478956a20f0cb154dd4553 Mon Sep 17 00:00:00 2001
From: Chengzhong Wu 
Date: Wed, 10 Jun 2026 14:52:36 -0400
Subject: [PATCH 548/551] chore: add commit-lint (#3325)

---
 .github/workflows/lint-commit.yml | 27 +++++++++++++++++++++++++++
 commitlint.config.mjs             |  3 +++
 package.json                      |  2 ++
 3 files changed, 32 insertions(+)
 create mode 100644 .github/workflows/lint-commit.yml
 create mode 100644 commitlint.config.mjs

diff --git a/.github/workflows/lint-commit.yml b/.github/workflows/lint-commit.yml
new file mode 100644
index 0000000000..6a61ae771a
--- /dev/null
+++ b/.github/workflows/lint-commit.yml
@@ -0,0 +1,27 @@
+name: Lint Commits
+
+on:
+  pull_request:
+
+permissions:
+  contents: read # to fetch code (actions/checkout)
+
+jobs:
+  lint-pr-first-commit:
+    name: Commit Lint
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        fetch-depth: 0
+    - uses: actions/setup-node@v6
+      with:
+        node-version: 22.x
+    - run: npm install
+    - name: Lint first PR commit message
+      run: |
+        FIRST=$(git rev-list --reverse ${{ github.event.pull_request.base.sha }}..${{ github.event.pull_request.head.sha }} | head -1)
+        echo "Base:       ${{ github.event.pull_request.base.sha }}"
+        echo "Head:       ${{ github.event.pull_request.head.sha }}"
+        echo "First PR commit: $FIRST"
+        git log -1 --format=%B "$FIRST" | npx commitlint --verbose
diff --git a/commitlint.config.mjs b/commitlint.config.mjs
new file mode 100644
index 0000000000..2291173ef3
--- /dev/null
+++ b/commitlint.config.mjs
@@ -0,0 +1,3 @@
+export default {
+  extends: ['@commitlint/config-conventional']
+};
diff --git a/package.json b/package.json
index 0b485a4657..e1ec3392c6 100644
--- a/package.json
+++ b/package.json
@@ -37,6 +37,8 @@
     "node": "^22.22.2 || ^24.15.0 || >=26.0.0"
   },
   "devDependencies": {
+    "@commitlint/cli": "^21.0.2",
+    "@commitlint/config-conventional": "^21.0.2",
     "bindings": "^1.5.0",
     "cross-env": "^10.1.0",
     "eslint": "^9.39.1",

From f089669a4c95a3c8b7f435556b226fa66dcd94b7 Mon Sep 17 00:00:00 2001
From: Christian Clauss 
Date: Wed, 10 Jun 2026 23:08:06 +0200
Subject: [PATCH 549/551] fix(ci): update ruff-action version to v4.0.0 (#3324)

Dependabot seems to have trouble switching to ___immutable releases___ like:
* https://github.com/astral-sh/ruff-action/releases/tag/v4.0.0
---
 .github/workflows/tests.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 7bbf6ea711..53090b55e8 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -19,7 +19,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v6
-    - uses: astral-sh/ruff-action@v3
+    - uses: astral-sh/ruff-action@v4.0.0
       with:
         args: "check --select=E,F,PLC,PLE,UP,W,YTT --ignore=E721,PLC0206,PLC0415,PLC1901,S101,UP031 --target-version=py39"
     - run: ruff format --check --diff

From 866daafcca3a41ef1f2c2e414671db6abe29e3e1 Mon Sep 17 00:00:00 2001
From: "Node.js GitHub Bot" 
Date: Fri, 12 Jun 2026 08:45:52 -0400
Subject: [PATCH 550/551] chore(main): release 13.0.0 (#3323)

---
 .release-please-manifest.json |  2 +-
 CHANGELOG.md                  | 31 +++++++++++++++++++++++++++++++
 package.json                  |  2 +-
 3 files changed, 33 insertions(+), 2 deletions(-)

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
index 7cf5faf711..139a159133 100644
--- a/.release-please-manifest.json
+++ b/.release-please-manifest.json
@@ -1,3 +1,3 @@
 {
-    ".": "12.4.0"
+    ".": "13.0.0"
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 83881ed2a3..211b04ef35 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,36 @@
 # Changelog
 
+## [13.0.0](https://github.com/nodejs/node-gyp/compare/v12.4.0...v13.0.0) (2026-06-10)
+
+
+### ⚠ BREAKING CHANGES
+
+* `node-gyp` now supports node `^22.22.2 || ^24.15.0 || >=26.0.0`
+
+### Features
+
+* bump to new node engine range ([b792b8e](https://github.com/nodejs/node-gyp/commit/b792b8e321e6e56d46c32f4b9adb15e4085c3797))
+
+
+### Bug Fixes
+
+* **ci:** update ruff-action version to v4.0.0 ([#3324](https://github.com/nodejs/node-gyp/issues/3324)) ([f089669](https://github.com/nodejs/node-gyp/commit/f089669a4c95a3c8b7f435556b226fa66dcd94b7))
+* disable LTO for addon builds on Windows ([#3331](https://github.com/nodejs/node-gyp/issues/3331)) ([d30bb6e](https://github.com/nodejs/node-gyp/commit/d30bb6e6fdaf46374de9b9bfe1bf54f49337b052))
+
+
+### Core
+
+* **deps:** bump actions/github-script from 8 to 9 ([#3297](https://github.com/nodejs/node-gyp/issues/3297)) ([f6a5e45](https://github.com/nodejs/node-gyp/commit/f6a5e458e819f05ddfbba5cbd5589a7a1c80128e))
+* **deps:** bump googleapis/release-please-action from 4 to 5 ([#3307](https://github.com/nodejs/node-gyp/issues/3307)) ([b2cde80](https://github.com/nodejs/node-gyp/commit/b2cde802487f3b9efd726a1a1cb43650731f9958))
+* nopt@10.0.0 ([327424a](https://github.com/nodejs/node-gyp/commit/327424ad630dd7007771b19f7b77691b2156a556))
+* proc-log@7.0.0 ([f6c296b](https://github.com/nodejs/node-gyp/commit/f6c296b2952e23b7e690b7148c9dc32b25a26ee2))
+* which@7.0.0 ([11a8b10](https://github.com/nodejs/node-gyp/commit/11a8b109f805f13129302508c219607167a8e9ea))
+
+
+### Miscellaneous
+
+* add commit-lint ([#3325](https://github.com/nodejs/node-gyp/issues/3325)) ([6fb6c11](https://github.com/nodejs/node-gyp/commit/6fb6c11eae1258eddd478956a20f0cb154dd4553))
+
 ## [12.4.0](https://github.com/nodejs/node-gyp/compare/v12.3.0...v12.4.0) (2026-05-15)
 
 
diff --git a/package.json b/package.json
index e1ec3392c6..cc11fc65e8 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
     "bindings",
     "gyp"
   ],
-  "version": "12.4.0",
+  "version": "13.0.0",
   "installVersion": 11,
   "author": "Nathan Rajlich  (http://tootallnate.net)",
   "repository": {

From 2c95a19f54bedf76209e795b146ab44e6fc3f359 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Fri, 12 Jun 2026 09:34:56 -0400
Subject: [PATCH 551/551] build(deps): bump undici from 6.26.0 to 8.4.1 (#3330)

Bumps [undici](https://github.com/nodejs/undici) from 6.26.0 to 8.4.1.
- [Release notes](https://github.com/nodejs/undici/releases)
- [Commits](https://github.com/nodejs/undici/compare/v6.26.0...v8.4.1)

---
updated-dependencies:
- dependency-name: undici
  dependency-version: 8.4.1
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 package.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package.json b/package.json
index cc11fc65e8..50f7889c0e 100644
--- a/package.json
+++ b/package.json
@@ -30,7 +30,7 @@
     "semver": "^7.3.5",
     "tar": "^7.5.4",
     "tinyglobby": "^0.2.12",
-    "undici": "^6.25.0",
+    "undici": "^8.4.1",
     "which": "^7.0.0"
   },
   "engines": {