\n\n## Installation\n\n```sh\n# In your project directory\n$ npm install --save-dev eslint-remote-tester\n\n# eslint@>=9 is also required as peer dependency\n$ npm install --save-dev eslint\n```\n\nThe underlying git integration is done via [simple-git](https://github.com/steveukx/git-js). It requires git to be installed and [that it can be called using the command `git`.](https://github.com/steveukx/git-js#dependencies)\n\n## Configuration\n\n### Linking your local eslint plugin as a dependency\n\nIn order for `eslint-remote-tester`'s `eslint` to know where the definitions for the new rules or modified existing rules are, your local eslint plugin folder needs to be symlinked.\nIn your plugin folder, dependending on which package manager you use, run:\n\n```sh\n# Using npm package manager\nnpm link\nnpm link [your-plugin-name]\n```\n\n### package.json\n\nAdd new script to your `package.json` file.\n\n```json\n\"scripts\": {\n \"lint:remote\": \"eslint-remote-tester\",\n```\n\n### eslint-remote-tester.config.js\n\nCreate new configuration file `eslint-remote-tester.config.js` in the root of your project. This is used as configuration file by default. Use `-c` or `--config` argument for custom configuration file, e.g. `--config path/custom.config.js`.\n\n```js\nimport js from '@eslint/js';\n\n/** @type {import('eslint-remote-tester').Config} */\nconst config = {\n repositories: ['mui-org/material-ui', 'reach/reach-ui'],\n extensions: ['js', 'jsx', 'ts', 'tsx'],\n eslintConfig: [js.configs.recommended],\n pathIgnorePattern: `(${[\n 'node_modules',\n '\\\\/\\\\.', // Any file or directory starting with dot, e.g. \".git\"\n 'test-results',\n 'docs',\n ].join('|')})`,\n};\n\nexport default config;\n```\n\nConfiguration file can also be written in TypeScript if [`jiti`](https://www.npmjs.com/package/jiti) or [`importx`](https://www.npmjs.com/package/importx) is installed. Use `--config` argument for TypeScript configuration file, e.g. `--config eslint-remote-tester.config.ts`.\n\n```ts\nimport type { Config } from 'eslint-remote-tester';\nimport js from '@eslint/js';\n\nconst config: Config = {\n repositories: ['mui-org/material-ui', 'reach/reach-ui'],\n extensions: ['js', 'jsx', 'ts', 'tsx'],\n eslintConfig: [js.configs.recommended],\n};\n\nexport default config;\n```\n\n#### Configuration options\n\n| Name | Description | Type | Required | Default | Example |\n| :-------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------- | :----------------- | :------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------- |\n| `repositories` | Repositories to scan in format of `owner/project`. See [eslint-remote-tester-repositories] for shared list of repositories. | `string[]` | :white_check_mark: | :x: | `['mui-org/material-ui', 'reach/reach-ui']` |\n| `extensions` | Extensions of files under scanning | `string[]` | :white_check_mark: | :x: | `['js', 'jsx', 'ts', 'tsx']` |\n| `eslintConfig` | ESLint configuration. Supports lazy initialization based on currently tested repository when a function is passed. Function is called with current repository and its location on filesystem. | `object \\| ({ location, repository }) => object` See [Configuring ESLint] | :white_check_mark: | :x: | `[js.configs.recommended]` `(options) => options.repository === 'my-repo' ? [js.configs.recommended] : []` |\n| `pathIgnorePattern` | Regexp pattern string used to exclude paths | `string` | :x: | :x: | `(node_modules\\|docs\\|\\\\/\\\\.git)` |\n| `maxFileSizeBytes` | Max file size used to exclude bigger files | `number` | :x: | `2000000` | `1500000` |\n| `rulesUnderTesting` | Array of rules or a filter method used to filter out results. Use `undefined` or empty array when ESLint crashes are the only interest. Filter method is called with `ruleId` and `options`. | `string[] \\| (ruleId, { repository }) => boolean` | :x: | `[]` | `['no-empty', 'react/sort-prop-types']` `(ruleId, options) => ruleId === 'no-undef' && options.repository === 'owner/repo'` |\n| `resultParser` | Syntax for the result parser | `plaintext\\|markdown` | :x: | `markdown` on CLI. `plaintext` on CI | `markdown` |\n| `concurrentTasks` | Maximum amount of tasks run concurrently | `number` | :x: | `5` | `3` |\n| `CI` | Flag used to set CI mode. `process.env.CI` is used when not set. | `boolean` | :x: | value of `process.env.CI === 'true'` | `true` |\n| `logLevel` | Filter log messages based on their priority | `verbose\\|info\\|warn\\|error` | :x: | `verbose` | `warn` |\n| `slowLintTimeLimit` | Time limit before linting of a single file is considered as slow, and logged as warning. Disabled by default. | `number\\|null` | :x: | :x: | `5 // 5 seconds` |\n| `cache` | Flag used to enable caching of cloned repositories. For CIs it's ideal to disable caching due to limited disk space. | `boolean` | :x: | `true` | `true` |\n| `timeLimit` | Time limit before scan is interrupted and **exited successfully**. Ideal for avoiding CI timeouts in regression tests. | `number` | :x: | `5.5 * 60 * 60, // 5 hours 30 minutes` | `5 * 60 * 60 // 5 hours` |\n| `compare` | Flag used to enable result comparison mode. Compares results of two scans and output the diff. Ideal for identifying new false positives when fixing existing rules. See [Fixing existing rules]. | `boolean` | :x: | `false` | `true` |\n| `updateComparisonReference` | Flag used to enable result comparison reference updating. Indicates whether comparison base should be updated after scan has finished. Ideal to be turned off once initial comparison base has been collected. | `boolean` | :x: | `true` | `true` |\n| `onComplete` | Callback invoked once scan is completed. Asynchronous functions are supported. Ideal for extending the process with custom features. | `(results, comparisonResults, repositoryCount) => void`\\|`Promise`. See [onComplete example]. | :x: | :x: | `async (results, comparisonResults, repositoryCount) => {}` |\n\n[configuring eslint]: https://eslint.org/docs/user-guide/configuring\n[fixing existing rules]: #fixing-existing-rules\n[oncomplete example]: docs/onComplete-arguments.json\n[eslint-remote-tester-repositories]: packages/repositories/README.md\n\n### Execution\n\nRun `npm run lint:remote`. Results are written into `./eslint-remote-tester-results` folder.\n\n## How and when to use\n\n_Disclaimer: This section is still work in progress._\n\n### Developing new ESLint rules\n\nWhen developing new ESLint rules you'll be most interested in results reported by the rule.\nAs the AST of Javascript and Typescript can cause very unexpected results it is not enough to test the rule only against unit tests and a small amount of repositories.\nTesting the rule against 1000's of repositories can reveal unexpected results:\n\n- Does the rule report the intended patterns?\n- Does the rule falsely mark valid patterns as errors?\n- Are the error messages set as expected?\n\nFollow these steps to test the new rule easily:\n\n1. Include new rule in `rulesUnderTesting`\n2. Configure `eslintConfig` with the new rule and its options\n3. Run `eslint-remote-tester` locally\n4. Check generated results at `./eslint-remote-tester-results`\n\n### Fixing existing rules\n\nFixing existing rules without breaking old functionality or introducing new bugs can be difficult.\nBy testing the rule against 1000's of repositories and comparing these reports against previous build's reports it is easily seen whether anything broke.\n\nRunning `eslint-remote-tester` in comparison mode allows developers to see the exact changes in ESLint reports their code changes introduced.\nExample of comparison results can be found at [Examples section](#examples).\n\nFollow these steps to test code changes of existing rules easily:\n\n1. Include given rule in `rulesUnderTesting`\n2. Configure `eslintConfig` with the rule and its options\n3. Stash code changes of the fix so that the current setup matches the latest build (typically `master` branch)\n4. Enable `compare` in the configuration file. This will generate comparison reference results.\n5. Run `eslint-remote-tester` locally\n6. Apply code changes of the fix back\n7. Disable `updateComparisonReference` in the configuration file. This will disable comparison reference updates - meaning that all next comparison runs are compared against the previously generated results (e.g. `master` branch).\n8. Run `eslint-remote-tester` locally again\n9. Check generated results at `./eslint-remote-tester-results/comparison-results`. New reports introduced by these code changes are saved in `added.md`. Reports not seen with latest changes are saved in `removed.md`.\n10. Repeat steps 8-9 and keep fixing the rule until `added.md` does not contain new false positives and `removed.md` does not contain unintended reports.\n\n### Plugin maintainer making sure new PRs don't introduce new false positives\n\nReviewing PRs of ESLint plugin projects can be slow if maintainers are expected to test every change manually.\nPlugin projects can set their CI to run `eslint-remote-tester` on comparison mode for each new PR.\nThe CI will compare ESLint reports between the `master` and PR branches, and report all new and removed reports.\nThis way the reviewers can easily see whether PR actually fixed the issue and if it introduced new bugs.\n\nProjects using GitHub CIs can utilize [eslint-remote-tester-compare-action](https://github.com/marketplace/actions/eslint-remote-tester-pr-comparator) for setting up PR comparisons.\n\n### Plugin maintainer making sure all existing rules do not crash\n\nPlugin projects can use `eslint-remote-tester` for smoke testing.\nAll the rules of given project can be run against 1000's of repositories so that only crashing rules are detected.\nThe job can even be configured to run for a limited time so that it will test as many repositories as it can in a specific time.\nThis can be done by setting `timeLimit` configuration. For Github CI the maximum time limit can be as long as 6 hours.\n\nThis project is set to run 6 hour long weekly scheduled tests using some of the most well-known ESLint community plugins.\nTypically the fastest plugins can test more than **10K repositories** in the 6 hour time limit.\n\nProjects using GitHub CIs can utilize [eslint-remote-tester-run-action](https://github.com/marketplace/actions/eslint-remote-tester-runner) for setting up smoke tests.\n\n### Configuration maintainer making sure all repositories follow the rules\n\nConfiguration maintainers (`eslint-config-*)` can check all their repositories follow rules of the configuration.\nBy using an all-inclusive filter in `rulesUnderTesting` all ESLint reports are picked, e.g. `rulesUnderTesting: () => true`.\n\nThe arguments of filter callback can be used to exclude specific rules from specific repositories:\n\n```js\nrulesUnderTesting: function ruleFilter(ruleId, options) {\n if (ruleId === 'node/no-process-env' && options.repository === 'username/my-cli-package') {\n // my-cli-package is allowed to use process.env\n return false;\n }\n\n // All other rules & repositories\n return true;\n}\n```\n\n## Github CI Actions\n\n- [eslint-remote-tester-run-action](https://github.com/marketplace/actions/eslint-remote-tester-runner)\n- [eslint-remote-tester-compare-action](https://github.com/marketplace/actions/eslint-remote-tester-pr-comparator)\n\n## Examples\n\n### Results:\n\n- [markdown](docs/results-markdown.md)\n- [plaintext](docs/results-plaintext)\n\n### Comparison results:\n\n- [added](docs/comparison-results-added.md)\n- [removed](docs/comparison-results-removed.md)\n\n### Configuration\n\n- [react configuration with +150 repositories](docs/eslint-remote-tester.react.config.js)\n- [small regression for eslint:recommended](eslint-remote-tester.config.js)\n\n### Repositories\n\nFor easiest setup `eslint-remote-tester-repositories` npm package is provided.\nSee [documentation](repositories/README.md).\n\nTarget repositories can easily be found using [libraries.io](https://libraries.io/).\nThey provide [an API for querying repositories](https://libraries.io/api#project-dependent-repositories) which depend on certain project.\nExample usage at [repository-query-script](repositories/scripts/fetch-libraries.js).\n","readmeFilename":"README.md"}
![](\"https://raw.githubusercontent.com/AriPerkkio/eslint-remote-tester/HEAD/docs/demo.svg\")
\n