{"bin":{"folio":"cli.js"},"bugs":{"url":"https://github.com/Microsoft/folio/issues"},"dist":{"shasum":"0c63a4c76c9f36210d4803ddbb2efa2c532c5c82","tarball":"https://registry.npmjs.org/folio/-/folio-0.3.22-alpha.tgz","fileCount":55,"integrity":"sha512-4EMbbUDKC/rHqe9zlI+se9sSbSee7L+U5tzMICMx8Omh0411HQHhMPKDNZhWwK+BXwYxNcdm51a8KBAGSW7Hlw==","signatures":[{"sig":"MEQCIF0er6dyCfxxkKfJfbN6Xqz6H+lKLSYfFbftMsXkxjO4AiANNHJUmAF2XQeGQKuGSgkUR4NWqqJ4s8FRxnWKdDt0Qw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":273765,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgbf6tCRA9TVsSAnZWagAA7kQQAIJi8yq9ZoKlNOcwoLAo\n5jiC4eHdpzeE7Wa/K7zplLG0JHRuD2WcGnlF7x+dXN+dyfXIFW0dQeX80Vbv\nCDmzL8vluexw0etPD300okaAbRFHr5XE4Lf1BQd4rphi4ae1hj+qYhn2bRNX\nS+nZFRFkOaxgnbv8owBIF9yRRKu4WpiH8AuD6iIJkev3iGEYj7L12lmTeCgY\nJCJrPRhepS8VUwfbLcyEEfl3mO71OeBN4OYHe6BbF7iq4hW7UKWiJqs6Zr38\nWE6KjoUmonhwRqCqZ8KFK6u45ifc77AwTmJpx1HAGiEdbRo/EmysH8WpUT/J\nLrZbK9qiPHgkCSLaUis5F1Xri68h8MdoSPUcX2hqUoqtYaC8lz6z8LewpNRh\nIzJuLAyR6tR0RP72hqLbbciTa35yp3euaYRtCafQScaRvzeQWuNTtd2L1Xjb\nVw88pHqsfZzUZiwoIIguwkACiZg0qPaVOSupe1S6O4BiF0bn5Fy2KoXGjajT\n6mPS69T0z2hI1EmyV9dn9IrdaJQlNSHs2pEUT8qjdtGray6DIovJAYPD7O0W\n0OpFG6STsvI7wA5y/lreE2wLlWycFG0PemICQPFqBMCnjHjw2qLtT/nHRPqn\n46EYHhKa29Fjd5KBy/BRIL3NdFBtYld5JQk1dDsx/7pfLY0iufkDY1ouDShS\n3hsl\r\n=JZlN\r\n-----END PGP SIGNATURE-----\r\n"},"main":"./out/index.js","name":"folio","author":{"name":"Microsoft Corporation"},"readme":"# Folio ![npm](https://img.shields.io/npm/v/folio)\n\nA customizable test framework to build your own test frameworks. Foundation for the [Playwright test runner](https://github.com/microsoft/playwright-test).\n\n## Docs\n\n- [Isolation and flexibility](#isolation-and-flexibility)\n- [Writing a test](#writing-a-test)\n- [Writing a configuration file](#writing-a-configuration-file)\n- [Creating an environment](#creating-an-environment)\n- [Command line](#command-line)\n- [Snapshots](#snapshots)\n- [Annotations](#annotations)\n  - [Flaky tests](#flaky-tests)\n- [Parallelism and sharding](#parallelism-and-sharding)\n  - [Workers](#workers)\n  - [Shards](#shards)\n- [Advanced configuration](#advanced-configuration)\n  - [Configuration object](#configuration-object)\n  - [workerInfo](#workerinfo)\n  - [testInfo](#testinfo)\n  - [Multiple test types and configurations](#multiple-test-types-and-configurations)\n  - [Global setup and teardown](#global-setup-and-teardown)\n  - [Test options](#test-options)\n- [Reporters](#reporters)\n  - [Built-in reporters](#built-in-reporters)\n  - [Reporter API](#reporter-api)\n\n## Isolation and flexibility\n\nFolio focuses on test isolation and flexibility. This makes it fast, reliable and able to adapt to your specific needs.\n\n**Isolation**. Tests are isolated by default and can be run independently.\n\n- Folio runs tests in parallel by default, making your test suite much faster. Thanks to isolation, Folio reuses processes for multiple tests, suites and file, which makes it even faster.\n\n- Flaky tests can be retried without significant overhead, because Folio will only retry the failures, and not the whole suite.\n\n- Refactoring tests and moving them around is effortless, since isolated tests do not have inter-dependencies.\n\n- You can group tests based on their meaning, instead of their common setup.\n\n**Flexibility**. Folio includes advanced features, adapting to your specific testing needs.\n\n- Leverage TypeScript power with minimal effort.\n\n- Run tests in multiple configurations.\n\n- Annotate tests as skipped/failed based on configuration.\n\n- Generate comprehensive report with your custom test annotations.\n\n- Define multiple test types, for example slow tests or smoke tests, and run them differently.\n\n## Writing a test\n\nFolio follows the traditional BDD style. However, each test in Folio receives an object with Test Arguments. These arguments are isolated from other tests, which gives Folio [numerous advantages](#isolation-and-flexibility).\n\n```ts\ntest('insert an entry', async ({ table }) => {\n  await table.insert({ username: 'folio', password: 'testing' });\n  const entry = await table.query({ username: 'folio' });\n  expect(entry.password).toBe('testing');\n});\n```\n\nIn the test above, `table` is a database table created for each test, so multiple tests running in parallel won't step on each other's toes.\n\nFolio uses `expect` library for test assertions.\n\n## Writing a configuration file\n\nFolio requires a configuration file that specifies how to run the tests.\n```ts\n// folio.config.ts\n\nimport * as folio from 'folio';\n\n// Configure Folio to look for tests in this directory, and give each test 20 seconds.\nfolio.setConfig({ testDir: __dirname, timeout: 20000 });\n\n// Create a new test type. For the easiest setup, you need just one.\nexport const test = folio.newTestType();\n\n// Run tests of this type, giving them two retries.\ntest.runWith({ tag: 'basic', retries: 2 });\n```\n\nNow, use the created test type in your tests.\n```ts\n// math.spec.ts\n\nimport { test } from './folio.config';\n\ntest('math works?', () => {\n  test.expect(1 + 1).toBe(42);\n});\n```\n\nYou can run tests with Folio [command line](#command-line):\n```sh\n$ npx folio --reporter=dot\nRunning 1 test using 1 worker\n××F\n 1 failed\n```\n\n## Creating an environment\n\nUsually, you need some test environment to run the tests. That may be a test database, dev server, mock user data, or anything else the test needs. Folio support creating an environment that is going to be used for multiple tests.\n\nLet's see how to add an environment, based on the example from [writing a configuration file](#writing-a-configuration-file) section.\n\n```ts\n// folio.config.ts\n\nimport * as folio from 'folio';\n\nfolio.setConfig({ testDir: __dirname, timeout: 20000 });\n\n// Type declaration specifies that tests receive a table instance.\nexport const test = folio.newTestType<{ table: DatabaseTable }>();\n\nclass DatabaseEnv {\n  host: string;\n  database: Database;\n  table: DatabaseTable;\n\n  constructor(host: string) {\n    this.host = host;\n  }\n\n  async beforeAll() {\n    // Connect to a database once, it is expensive.\n    this.database = await connectToTestDatabase(this.host);\n  }\n\n  async beforeEach() {\n    // Create a new table for each test and return it.\n    this.table = await this.database.createTable();\n    return { table: this.table };\n  }\n\n  async afterEach() {\n    // Do not leave extra tables around.\n    await this.table.drop();\n  }\n\n  async afterAll() {\n    await this.database.disconnect();\n  }\n}\n\n// Run our tests in two environments, against a local database and a staging database.\ntest.runWith(new DatabaseEnv('localhost:1234'));\ntest.runWith(new DatabaseEnv('staging-db.my-company.com:1234'));\n```\n\nIn this example we see that tests declare the arguments they need, and environment provides the arguments. We can run tests in multiple configurations when needed.\n\nFolio uses worker processes to run test files. You can specify the maximum number of workers using `--workers` command line option. By using `beforeAll` and `afterAll` methods, environment can set up expensive resources to be shared between tests in each worker process. Folio will reuse the worker process for as many test files as it can, provided their environments match.\n\n## Annotations\n\nUnfortunately, tests do not always pass. Folio supports test annotations to deal with failures, flakiness and tests that are not yet ready.\n\n```ts\ntest('basic', async ({ table }) => {\n  test.skip(version == 'v2', 'This test crashes the database in v2, better not run it.');\n  // Test goes here.\n});\n\ntest('can insert multiple rows', async ({ table }) => {\n  test.fail('Broken test, but we should fix it!');\n  // Test goes here.\n});\n```\n\nAnnotations may be conditional, in which case they only apply when the condition is truthy. Annotations may depend on test arguments. There could be multiple annotations on the same test, possibly in different configurations.\n\nPossible annotations include:\n- `skip` marks the test as irrelevant. Folio does not run such a test. Use this annotation when the test is not applicable in some configuration.\n- `fail` marks the test as failing. Folio will run this test and ensure it does indeed fail. If the test does not fail, Folio will complain.\n- `fixme` marks the test as failing. Folio will not run this test, as opposite to the `fail` annotation. Use `fixme` when running the test is slow or crashy.\n\n### Flaky tests\n\nFolio deals with flaky tests with retries. Pass the maximum number of retries when running the tests, or set them in the [configuration file](#writing-a-configuration-file).\n```sh\nnpx folio --retries=3\n```\n\nFailing tests will be retried multiple times until they pass, or until the maximum number of retries is reached. Folio will report all tests that failed at least once:\n\n```sh\nRunning 1 test using 1 worker\n××±\n1 flaky\n  1) my.test.js:1:1\n```\n\n## Command line\n\nJust point Folio to your [configuration file](#writing-a-configuration-file).\n```sh\n$ npx folio --config=my.config.ts\n```\n\nRun Folio with `--help` to see all command line options.\n```sh\n$ npx folio --help\n```\n\n## Snapshots\n\nFolio includes the ability to produce and compare snapshots. For that, use `expect().toMatchSnapshot()`. Folio auto-detects the content type, and includes built-in matchers for text, png and jpeg images, and arbitrary binary data.\n\n```ts\ntest('my test', async () => {\n  const image = await produceSomePNG();\n  expect(image).toMatchSnapshot('optional-snapshot-name.png');\n});\n```\n\nSnapshots are stored under `__snapshots__` directory by default, configurable via [command line](#command-line) or [configuration object](#configuration-object).\n\n## Parallelism and sharding\n\nFolio runs tests in parallel by default, using multiple worker processes.\n\n### Workers\n\nEach worker process creates a new environment to run tests. Different environments always run in different workers. By default, Folio reuses the worker as much as it can to make testing faster, but it will create a new worker when retrying tests, after any test failure, to initialize a new environment, or just to speed up test execution if the worker limit is not reached.\n\nThe maximum number of worker processes is controlled via [command line](#command-line) or [configuration object](#configuration-object).\n\nEach worker process is assigned a unique sequential index that is accessible through [`workerInfo`](#workerinfo) object.\n\n### Shards\n\nFolio can shard a test suite, so that it can be executed on multiple machines. For that,  pass `--shard=x/y` to the command line. For example, to split the suite into three shards, each running one third of the tests:\n```sh\n$ npx folio --shard=1/3\n$ npx folio --shard=2/3\n$ npx folio --shard=3/3\n```\n\n## Advanced configuration\n\n### Configuration object\n\nConfiguration file uses `setConfig` function to provide a global configuration to Folio. It may contain the following properties:\n- `forbidOnly: boolean` - Whether to disallow `test.only` exclusive tests. Useful on CI.\n- `globalTimeout: number` - Total timeout in milliseconds for the whole test run.\n- `grep: string | RegExp | (string | RegExp)[]` - Patterns to filter tests based on their title.\n- `maxFailures: number` - Stop testing after reaching the maximum number of failures.\n- `outputDir: string` - Directory to place any artifacts produced by tests.\n- `quiet: boolean` - Whether to suppress stdout and stderr from the tests.\n- `repeatEach: number` - Each test will be repeated multiple times.\n- `retries: number` - Maximum number of retries.\n- `shard: { total: number, current: number } | null` - [Shard](#shards) information.\n- `snapshotDir: string` - [Snapshots](#snapshots) directory.\n- `testDir: string` - Directory where Folio should search for tests.\n- `testIgnore: string | RegExp | (string | RegExp)[]` - Patterns to ignore test files.\n- `testMatch: string | RegExp | (string | RegExp)[]` - Patterns to match test files.\n- `timeout: number` - Test timeout in milliseconds.\n- `updateSnapshots: boolean` - Whether to update snapshots instead of comparing them.\n- `workers: number` - The maximum number of worker processes.\n\n```ts\n// folio.config.ts\n\nimport * as folio from 'folio';\n\nfolio.setConfig({\n  // Typically, you'd place folio.config.ts in the tests directory.\n  testDir: __dirname,\n  // 20 seconds per test.\n  timeout: 20000,\n  // Forbid test.only on CI.\n  forbidOnly: !!process.env.CI,\n  // Two retries for each test.\n  retries: 2,\n});\n```\n\n### workerInfo\n\nDepending on the configuration and failures, Folio might use different number of worker processes to run all the tests. For example, Folio will always start a new worker process after a failing test.\n\nEnvironment and hooks receive `workerInfo` in the `beforeAll` and `afterAll` calls. The following information is accessible from the `workerInfo`:\n- `config` - [Configuration object](#configuration-object).\n- `globalSetupResult` - The value returned by the [global setup function](#global-setup-and-teardown).\n- `workerIndex: number` - A unique sequential index assigned to the worker process.\n\nConsider an example where we run a new http server per worker process, and use `workerIndex` to produce a unique port number:\n\n```ts\nimport * as http from 'http';\n\nclass ServerEnv {\n  server: http.Server;\n\n  async beforeAll(workerInfo) {\n    this.server = http.createServer();\n    this.server.listen(9000 + workerInfo.workerIndex);\n    await new Promise(ready => this.server.once('listening', ready));\n  }\n\n  async beforeEach() {\n    // Provide the server as a test argument.\n    return { server: this.server };\n  }\n\n  async afterAll() {\n    await new Promise(done => this.server.close(done));\n  }\n}\n```\n\n### testInfo\n\nEnvironment and hooks receive `testInfo` in the `beforeEach` and `afterEach` calls. It is also available to the test function as a second parameter.\n\nIn addition to everything from the [`workerInfo`](#workerinfo), the following information is accessible before and during the test:\n- `title: string` - Test title.\n- `file: string` - Full path to the test file.\n- `line: number` - Line number of the test declaration.\n- `column: number` - Column number of the test declaration.\n- `fn: Function` - Test body function.\n- `repeatEachIndex: number` - The sequential repeat index.\n- `retry: number` - The sequential number of the test retry (zero means first run).\n- `expectedStatus: 'passed' | 'failed' | 'timedOut'` - Whether this test is expected to pass, fail or timeout.\n- `timeout: number` - Test timeout.\n- `testOptions` - [Test options](#test-options).\n- `annotations` - [Annotations](#annotations) that were added to the test.\n- `data: object` - Any additional data that you'd like to attach to the test, it will appear in the report.\n- `snapshotPathSegment: string` - Relative path, used to locate snapshots for the test.\n- `snapshotPath(...pathSegments: string[])` - Function that returns the full path to a particular snapshot for the test.\n- `outputPath(...pathSegments: string[])` - Function that returns the full path to a particular output artifact for the test.\n\nThe following information is accessible after the test body has finished, in `afterEach`:\n- `duration: number` - test running time in milliseconds.\n- `status: 'passed' | 'failed' | 'timedOut'` - the actual test result.\n- `error` - any error thrown by the test body.\n- `stdout: (string | Buffer)[]` - array of stdout chunks collected during the test run.\n- `stderr: (string | Buffer)[]` - array of stderr chunks collected during the test run.\n\nHere is an example test that saves some information:\n```ts\ntest('my test needs a file', async ({ table }, testInfo) => {\n  // Do something with the table...\n  // ... and then save contents.\n  const filePath = testInfo.outputPath('table.dat');\n  await table.saveTo(filePath);\n});\n```\n\nHere is an example environment that automatically saves debug logs when the test fails:\n```ts\nimport * as debug from 'debug';\nimport * as fs from 'fs';\n\nclass LogEnv {\n  async beforeEach() {\n    this.logs = [];\n    debug.log = (...args) => this.logs.push(args.map(String).join(''));\n    debug.enable('mycomponent');\n  }\n\n  async afterEach(testInfo) {\n    if (testInfo.status !== testInfo.expectedStatus)\n      fs.writeFileSync(testInfo.outputPath('logs.txt'), this.logs.join('\\n'), 'utf8');\n  }\n}\n```\n\n### Multiple test types and configurations\n\nOften times there is a need for different kinds of tests, for example generic tests that use a database table, or some specialized tests that require more elaborate setup. It is also common to run tests in multiple configurations. Folio allows you to configure everything by writing code for maximum flexibility.\n\n```ts\n// folio.config.ts\n\nimport * as folio from 'folio';\nimport * as fs from 'fs';\n\n// 20 seconds timeout, 3 retries by default.\nfolio.setConfig({ testDir: __dirname, timeout: 20000, retries: 3 });\n\n// Define as many test types as you'd like:\n// - Generic test that only needs a string value.\nexport const test = folio.newTestType<{ value: string }>();\n// - Slow test for extra-large data sets.\nexport const slowTest = folio.newTestType<{ value: string }>();\n// - Smoke tests should not be flaky.\nexport const smokeTest = folio.newTestType<{ value: string }>();\n// - Some special tests that require different arguments.\nexport const fooTest = folio.newTestType<{ foo: number }>();\n\n// Environment with some test value.\nclass MockedEnv {\n  async beforeEach() {\n    return { value: 'some test value' };\n  }\n}\n\n// Another environment that reads from file.\nclass FileEnv {\n  constructor() {\n    this.value = fs.readFileSync('data.txt', 'utf8');\n  }\n  async beforeEach() {\n    return { value: this.value };\n  }\n}\n\n// This environment provides foo.\nclass FooEnv {\n  async beforeEach() {\n    return { foo: 42 };\n  }\n}\n\n// Now we can run tests in different configurations:\n// - Generics tests with two different environments.\ntest.runWith(new MockedEnv());\ntest.runWith(new FileEnv());\n// - Increased timeout for slow tests.\nslowTest.runWith(new MockedEnv(), { timeout: 100000 });\n// - Smoke tests without retries.\n//   Adding a tag allows to run just the smoke tests with `npx folio --tag=smoke`.\nsmokeTest.runWith(new MockedEnv(), { retries: 0, tag: 'smoke' });\n// - Special foo tests need a different environment.\nfooTest.runWith(new FooEnv());\n```\n\nWe can now use our test types to write tests:\n```ts\n// some.spec.ts\n\nimport { test, slowTest, smokeTest, fooTest } from './folio.config';\n\ntest('just a test', async ({ value }) => {\n  // This test will be retried.\n  expect(value).toBe('wrong value');\n});\n\nslowTest('does a lot', async ({ value }) => {\n  for (let i = 0; i < 100000; i++)\n    expect(value).toBe('some test value');\n});\n\nsmokeTest('a smoke test', async ({ value }) => {\n  // This test will not be retried.\n  expect(value).toBe('some test value');\n});\n\nfooTest('a smoke test', async ({ foo }) => {\n  // Note the different test arguments.\n  expect(foo).toBe(42);\n});\n```\n","engines":{"node":">=10.17.0"},"gitHead":"2a917e3c8ba2541af7ee1412e65ccbb542564e50","license":"Apache-2.0","scripts":{"lint":"eslint . --ext js,ts","test":"folio test/ --test-ignore=assets/**","build":"tsc --build tsconfig.json","watch":"tsc --build tsconfig.json --watch","prepare":"npm run build","roll-dogfood":"cd dogfood && npm install","prepublishOnly":"rm tsconfig.tsbuildinfo && rm -rf out && npm run build"},"_npmUser":{"name":"dgozman","email":"dgozman@gmail.com"},"homepage":"https://github.com/Microsoft/folio#readme","deprecated":"Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.","repository":{"url":"git+https://github.com/Microsoft/folio.git","type":"git"},"_npmVersion":"6.14.5","description":"A customizable test framework to build your own test frameworks. Foundation for the [Playwright test runner](https://github.com/microsoft/playwright-test).","directories":{},"maintainers":[{"name":"arjun27","email":"arjunattam@gmail.com"},{"name":"aslushnikov","email":"aslushnikov@gmail.com"},{"name":"joeleinbinder","email":"joel.einbinder@gmail.com"},{"name":"pavelfeldman","email":"pavel.feldman@gmail.com"},{"name":"dgozman","email":"dgozman@gmail.com"},{"name":"yurys","email":"yury.semikhatsky@gmail.com"}],"_nodeVersion":"12.18.2","dependencies":{"ms":"^2.1.2","debug":"^4.1.5","pngjs":"^5.0.0","colors":"^1.4.0","expect":"^26.4.2","rimraf":"^3.0.2","jpeg-js":"^0.4.2","pirates":"^4.0.1","commander":"^6.1.0","pixelmatch":"^5.2.1","@babel/core":"^7.11.4","stack-utils":"^2.0.2","pretty-format":"^26.4.2","fstream-ignore":"^1.0.5","@babel/code-frame":"^7.10.4","@babel/preset-env":"^7.11.0","source-map-support":"^0.5.19","@babel/preset-typescript":"^7.10.4","@babel/plugin-proposal-class-properties":"^7.10.4"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"folio":"./dogfood/node_modules/folio","eslint":"^7.8.1","xml2js":"^0.4.23","micromatch":"^4.0.2","typescript":"=4.0.2","@types/xml2js":"^0.4.5","@types/minimatch":"^3.0.3","@types/micromatch":"^4.0.1","@types/babel__core":"^7.1.9","eslint-plugin-notice":"^0.9.10","@typescript-eslint/parser":"^4.2.0","@typescript-eslint/eslint-plugin":"^4.2.0"},"_npmOperationalInternal":{"tmp":"tmp/folio_0.3.22-alpha_1617821354818_0.5470170455194567","host":"s3://npm-registry-packages"},"_id":"folio@0.3.22-alpha","version":"0.3.22-alpha"}