\n\nThis package provides a TypeScript evaluation library. It is vendor agnostic and can be used in isolation of any framework or platform. This package is still under active development and is subject to change.\n\n## Installation\n\n```bash\n# or yarn, pnpm, bun, etc...\nnpm install @arizeai/phoenix-evals\n```\n\nThis package is built on [AI SDK](https://ai-sdk.dev/) v7, which is ESM-only. Node.js 22.12 or newer is required (the CommonJS build relies on `require()` of ESM). Use AI SDK v7-compatible model provider packages (e.g. `@ai-sdk/openai` v4+).\n\n## Usage\n\n### Creating a Classifier\n\nThe library provides a `createClassifier` function that allows you to create custom evaluators for different tasks like hallucination detection, relevance scoring, or any binary/multi-class classification.\n\n```typescript\nimport { createClassifier } from \"@arizeai/phoenix-evals/llm\";\nimport { openai } from \"@ai-sdk/openai\";\n\nconst model = openai(\"gpt-4o-mini\");\n\nconst promptTemplate = `\nIn this task, you will be presented with a query, a reference text and an answer. The answer is\ngenerated to the question based on the reference text. The answer may contain false information. You\nmust use the reference text to determine if the answer to the question contains false information,\nif the answer is a hallucination of facts. Your objective is to determine whether the answer text\ncontains factual information and is not a hallucination. A 'hallucination' refers to\nan answer that is not based on the reference text or assumes information that is not available in\nthe reference text. Your response should be a single word: either \"factual\" or \"hallucinated\", and\nit should not include any other text or characters.\n\n [BEGIN DATA]\n ************\n [Query]: {{input}}\n ************\n [Reference text]: {{reference}}\n ************\n [Answer]: {{output}}\n ************\n [END DATA]\n\nIs the answer above factual or hallucinated based on the query and reference text?\n`;\n\n// Create the classifier\nconst evaluator = await createClassifier({\n model,\n choices: { factual: 1, hallucinated: 0 },\n promptTemplate: promptTemplate,\n});\n\n// Use the classifier\nconst result = await evaluator({\n output: \"Arize is not open source.\",\n input: \"Is Arize Phoenix Open Source?\",\n reference:\n \"Arize Phoenix is a platform for building and deploying AI applications. It is open source.\",\n});\n\nconsole.log(result);\n// Output: { label: \"hallucinated\", score: 0 }\n```\n\nSee the complete example in [`examples/classifier_example.ts`](examples/classifier_example.ts).\n\n### Pre-Built Evaluators\n\nThe library includes several pre-built evaluators for common evaluation tasks. These evaluators come with optimized prompts and can be used directly with any AI SDK model.\n\nAll pre-built evaluators are available from the `@arizeai/phoenix-evals/llm` module:\n\n| Evaluator | Function | Description |\n| ---------------------- | ------------------------------------- | --------------------------------------------------------------------------------- |\n| Faithfulness | `createFaithfulnessEvaluator` | Detects hallucinations — checks if the output is grounded in the provided context |\n| Completeness | `createCompletenessEvaluator` | Checks whether every active user request in a conversation was actually completed |\n| Conciseness | `createConcisenessEvaluator` | Evaluates whether the response is appropriately concise |\n| Correctness | `createCorrectnessEvaluator` | Checks if the output is factually correct given the input |\n| Retrieval Relevance | `createRetrievalRelevanceEvaluator` | Measures how relevant retrieved information is to a request |\n| Refusal | `createRefusalEvaluator` | Detects whether the model refused to answer |\n| Tool Invocation | `createToolInvocationEvaluator` | Evaluates whether the correct tool was invoked with the right arguments |\n| Tool Selection | `createToolSelectionEvaluator` | Checks whether the right tool was selected for the task |\n| Tool Response Handling | `createToolResponseHandlingEvaluator` | Evaluates how well the model uses a tool's response |\n| PII Detection | `createPiiDetectionEvaluator` | Screens a conversation record for personally identifiable information |\n\n```typescript\nimport {\n createFaithfulnessEvaluator,\n createConcisenessEvaluator,\n createCorrectnessEvaluator,\n createRetrievalRelevanceEvaluator,\n createRefusalEvaluator,\n} from \"@arizeai/phoenix-evals/llm\";\nimport { openai } from \"@ai-sdk/openai\";\n\nconst model = openai(\"gpt-4o-mini\");\n\n// Faithfulness: checks if the output is grounded in the context\nconst faithfulnessEvaluator = createFaithfulnessEvaluator({ model });\nconst faithfulnessResult = await faithfulnessEvaluator.evaluate({\n input: \"What is the capital of France?\",\n context: \"France is a country in Europe. Paris is its capital city.\",\n output: \"The capital of France is London.\",\n});\nconsole.log(faithfulnessResult);\n// Output: { label: \"unfaithful\", score: 0, explanation: \"...\" }\n\n// Correctness: checks if the output is factually correct\nconst correctnessEvaluator = createCorrectnessEvaluator({ model });\nconst correctnessResult = await correctnessEvaluator.evaluate({\n input: \"What is the capital of France?\",\n output: \"Paris is the capital of France.\",\n});\nconsole.log(correctnessResult);\n// Output: { label: \"correct\", score: 1, explanation: \"...\" }\n\n// Retrieval Relevance: checks if retrieved information is relevant to the request\nconst relevanceEvaluator = createRetrievalRelevanceEvaluator({ model });\nconst relevanceResult = await relevanceEvaluator.evaluate({\n input: \"What is the capital of France?\",\n context: \"Paris is the capital of France and a major European city.\",\n});\nconsole.log(relevanceResult);\n// Output: { label: \"relevant\", score: 1, explanation: \"...\" }\n```\n\n### Code Evaluators\n\nThe library also includes built-in, deterministic (non-LLM) code evaluators for common classification metrics: precision, recall, and F-beta (including F1). These are available from the `@arizeai/phoenix-evals/code` module and work over a batch of expected vs. predicted labels, supporting both binary classification (via `positiveLabel`) and multi-class classification (via `macro`/`micro`/`weighted` averaging).\n\n- **Precision** — of everything predicted as a class, the fraction that was actually that class (`TP / (TP + FP)`). Lower precision means more false alarms.\n- **Recall** — of everything that actually belongs to a class, the fraction the model found (`TP / (TP + FN)`). Lower recall means more misses.\n- **F-beta** — the weighted harmonic mean of precision and recall. `beta = 1` (F1, the default) weights them equally; `beta > 1` weights recall more (use when missing a true positive is costlier, e.g. medical screening); `beta < 1` weights precision more (use when a false alarm is costlier, e.g. spam filtering).\n\n```typescript\nimport {\n createPrecisionEvaluator,\n createRecallEvaluator,\n createF1Evaluator,\n createPrecisionRecallFScoreEvaluators,\n} from \"@arizeai/phoenix-evals/code\";\n\nconst f1 = createF1Evaluator();\nconst result = await f1.evaluate({\n expected: [\"cat\", \"dog\", \"cat\", \"bird\"],\n output: [\"cat\", \"cat\", \"cat\", \"bird\"],\n});\nconsole.log(result);\n// Output: { score: 0.6 }\n\n// Or create matching precision/recall/F-score evaluators at once\nconst { precision, recall, fScore } = createPrecisionRecallFScoreEvaluators({\n average: \"weighted\",\n});\n```\n\nUse `createFBetaEvaluator({ beta })` for F-scores other than F1 (e.g. `beta: 2` weights recall higher than precision). `createPrecisionRecallFScoreEvaluators` returns three separate evaluators (`{ precision, recall, fScore }`) sharing one options object — each yields a single score, matching the individual factories.\n\nFor multi-class data, `average` controls how per-class scores combine into one number: `\"macro\"` (default) weights every class equally — good for surfacing whether a rare class is being ignored; `\"weighted\"` weights each class by how often it occurs — good when overall performance matters more than parity across classes; `\"micro\"` pools every true/false positive and false negative across classes first — for single-label multi-class problems this equals overall accuracy.\n\nSee [`examples/classification_metrics_example.ts`](examples/classification_metrics_example.ts) for a runnable walkthrough covering binary classification, F-beta tradeoffs, and all three averaging strategies. For the underlying formulas, the precision/recall tradeoff, and citations, see the [Precision / Recall / F-Score docs](https://arize.com/docs/phoenix/evaluation/pre-built-metrics/precision-recall-fscore).\n\n> **Note:** unlike the per-row LLM evaluators above, these classification-metric evaluators are batch/dataset-level: `expected`/`output` are the full sequence of labels across every example, not a single row's labels. Don't wire them directly into `runExperiment` as a per-row evaluator — instead, collect every row's expected/predicted label first, then call `.evaluate({ expected, output })` once over the full arrays.\n\n### Data Mapping\n\nWhen your data structure doesn't match what an evaluator expects, use `bindEvaluator` to map your fields to the evaluator's expected input format:\n\n```typescript\nimport { bindEvaluator } from \"@arizeai/phoenix-evals\";\nimport { createFaithfulnessEvaluator } from \"@arizeai/phoenix-evals/llm\";\nimport { openai } from \"@ai-sdk/openai\";\n\nconst model = openai(\"gpt-4o-mini\");\n\ntype ExampleType = {\n question: string;\n context: string;\n answer: string;\n};\n\nconst evaluator = bindEvaluator(\n createFaithfulnessEvaluator({ model }),\n {\n inputMapping: {\n input: \"question\", // Map \"input\" from \"question\"\n context: \"context\", // Map \"context\" from \"context\"\n output: \"answer\", // Map \"output\" from \"answer\"\n },\n }\n);\n\nconst result = await evaluator.evaluate({\n question: \"Is Arize Phoenix Open Source?\",\n context:\n \"Arize Phoenix is a platform for building and deploying AI applications. It is open source.\",\n answer: \"Arize is not open source.\",\n});\n```\n\nMapping supports simple properties (`\"fieldName\"`), dot notation (`\"user.profile.name\"`), array access (`\"items[0].id\"`), JSONPath expressions (`\"$.items[*].id\"`), and function extractors (`(data) => data.customField`).\n\nSee the complete example in [`examples/bind_evaluator_example.ts`](examples/bind_evaluator_example.ts).\n\n## Experimentation with Phoenix\n\nThis package works seamlessly with [`@arizeai/phoenix-client`](https://www.npmjs.com/package/@arizeai/phoenix-client) to enable experimentation workflows. You can create datasets, run experiments, and trace evaluation calls for analysis and debugging.\n\n### Running Experiments\n\nTo run experiments with your evaluations, install the phoenix-client\n\n```bash\nnpm install @arizeai/phoenix-client\n```\n\n```typescript\nimport { createFaithfulnessEvaluator } from \"@arizeai/phoenix-evals/llm\";\nimport { openai } from \"@ai-sdk/openai\";\nimport { createDataset } from \"@arizeai/phoenix-client/datasets\";\nimport {\n asExperimentEvaluator,\n runExperiment,\n} from \"@arizeai/phoenix-client/experiments\";\n\n// Create your evaluator\nconst faithfulnessEvaluator = createFaithfulnessEvaluator({\n model: openai(\"gpt-4o-mini\"),\n});\n\n// Create a dataset for your experiment\nconst dataset = await createDataset({\n name: \"faithfulness-eval\",\n description: \"Evaluate the faithfulness of the model\",\n examples: [\n {\n input: {\n question: \"Is Phoenix Open-Source?\",\n context: \"Phoenix is Open-Source.\",\n },\n },\n // ... more examples\n ],\n});\n\n// Define your experimental task\nconst task = async (example) => {\n // Your AI system's response to the question\n return \"Phoenix is not Open-Source\";\n};\n\n// Create a custom evaluator to validate results\nconst faithfulnessCheck = asExperimentEvaluator({\n name: \"faithfulness\",\n kind: \"LLM\",\n evaluate: async ({ input, output }) => {\n // Use the faithfulness evaluator from phoenix-evals\n const result = await faithfulnessEvaluator({\n input: input.question,\n context: input.context,\n output: output,\n });\n\n return result; // Return the evaluation result\n },\n});\n\n// Run the experiment with automatic tracing\nrunExperiment({\n experimentName: \"faithfulness-eval\",\n experimentDescription: \"Evaluate the faithfulness of the model\",\n dataset: dataset,\n task,\n evaluators: [faithfulnessCheck],\n});\n```\n\n## Examples\n\nTo run examples, install dependencies using `pnpm` and run:\n\n```bash\npnpm install\npnpx tsx examples/classifier_example.ts\n# change the file name to run other examples\n```\n\n## Community\n\nJoin our community to connect with thousands of AI builders:\n\n- 🌍 Join our [Slack community](https://join.slack.com/t/arize-ai/shared_invite/zt-3r07iavnk-ammtATWSlF0pSrd1DsMW7g).\n- 📚 Read the [Phoenix documentation](https://arize.com/docs/phoenix).\n- 💡 Ask questions and provide feedback in the _#phoenix-support_ channel.\n- 🌟 Leave a star on our [GitHub](https://github.com/Arize-ai/phoenix).\n- 🐞 Report bugs with [GitHub Issues](https://github.com/Arize-ai/phoenix/issues).\n- 𝕏 Follow us on [𝕏](https://twitter.com/ArizePhoenix).\n- 💼 Follow us on [LinkedIn](https://www.linkedin.com/showcase/113218220).\n- 🗺️ Check out our [roadmap](https://github.com/orgs/Arize-ai/projects/45) to see where we're heading next.\n","readmeFilename":""}
![\"Arize](\"https://raw.githubusercontent.com/Arize-ai/phoenix-assets/refs/heads/main/logos/Phoenix/phoenix.svg\")
\n ![\"NPM](\"https://img.shields.io/npm/v/%40arizeai%2Fphoenix-evals\")
\n ![\"Documentation\"](\"https://img.shields.io/badge/docs-blue?logo=typescript&logoColor=white\"){kind=icon}
\n ![](\"https://static.scarf.sh/a.png?x-pxid=8e8e8b34-7900-43fa-a38f-1f070bd48c64&page=js/packages/phoenix-evals/README.md\")
\n