# LLM-Evaluation

## Papers

### 2022

- (2022-09) **News Summarization and Evaluation in the Era of GPT-3** [paper](https://arxiv.org/abs/2209.12356)

### 2023

- (2023-01) **How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection** [paper](https://arxiv.org/abs/2301.07597) | [project](https://github.com/Hello-SimpleAI/chatgpt-comparison-detection)

- (2023-01) **Is ChatGPT A Good Translator? A Preliminary Study** [paper](https://arxiv.org/abs/2301.08745v2) | [code](https://github.com/wxjiao/Is-ChatGPT-A-Good-Translator)

  >:exclamation: They only randomly select 50 sentences for evaluation, since there is no available API.

- (2023-01) **Benchmarking Large Language Models for News Summarization** [paper](https://arxiv.org/abs/2301.13848)

- (2023-02) **Is ChatGPT a General-Purpose Natural Language Processing Task Solver?** [paper](https://arxiv.org/abs/2302.06476)

  >:exclamation: No large dataset evaluation, no few-shot in-context learning evaluation, due to lack of API.

- (2023-02) **ChatGPT: Jack of all trades, master of none** [paper](https://arxiv.org/abs/2302.10724)

- (2023-02) **Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT** [paper](https://arxiv.org/abs/2302.10198)

- (2023-02) **On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective** [paper](https://arxiv.org/abs/2302.12095)

- (2023-02) **Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization** [paper](https://arxiv.org/abs/2302.08081)

- (2023-02) **ChatGPT: potential, prospects, and limitations** [paper](https://doi.org/10.1631/FITEE.2300089)

- (2023-03) **How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Tasks.** [paper](https://arxiv.org/abs/2303.00293)

- (2023-03) **ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks** [paper](https://arxiv.org/abs/2303.15056)

- (2023-03) **Consistency Analysis of ChatGPT** [paper](https://arxiv.org/abs/2303.06273)

- (2023-03) **Could a Large Language Model be Conscious?** [paper](https://arxiv.org/abs/2303.07103)

- (2023-03) **Susceptibility to Influence of Large Language Models** [paper](https://arxiv.org/abs/2303.06074)

- (2023-03) **A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models** [paper](https://arxiv.org/abs/2303.10420)
- (2023-03) **Sparks of Artificial General Intelligence: Early experiments with GPT-4** [paper](https://arxiv.org/abs/2303.12712)

- (2023-03) **ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks** [paper](https://arxiv.org/abs/2303.15056)
- (2023-04) **Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation** [paper](https://arxiv.org/abs/2304.01746)

- (2023-03) **Is ChatGPT a Good NLG Evaluator? A Preliminary Study** [paper](https://arxiv.org/abs/2303.04048)

- (2023-04) **Is ChatGPT a Good Sentiment Analyzer? A Preliminary Study** [paper](https://arxiv.org/abs/2304.04339)

- (2023-04) **Emergent and Predictable Memorization in Large Language Models** [paper](https://arxiv.org/abs/2304.11158)

- (2023-04) **Why Does ChatGPT Fall Short in Answering Questions Faithfully?** [paper](https://arxiv.org/abs/2304.10513)

- (2023-04) **Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness** [paper](https://arxiv.org/abs/2304.11633)
 
- (2023-04) **Are Emergent Abilities of Large Language Models a Mirage?** [paper](https://arxiv.org/abs/2304.15004)

- (2023-10) **Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators** [paper](https://arxiv.org/abs/2310.07289) | [code](https://github.com/ChanLiang/CONNER)

## Useful Resources

