## Table of contents

-   [Introduction](#introduction)
    -   [Summary vs snapshot](#summary-vs-snapshot)
-   [Why do we need summaries?](#why-do-we-need-summaries)
-   [Who generates summaries?](#who-generates-summaries)
-   [When are summaries generated?](#when-are-summaries-generated)
-   [How are summaries generated?](#how-are-summaries-generated)
    -   [Summary Lifecycle](#summary-lifecycle)
    -   [Single-commit vs two-commit summaries](#single-commit-vs-two-commit-summaries)
    -   [Incremental summaries](#incremental-summaries)
	-   [Resiliency](#resiliency)
-   [What does a summary look like?](#what-does-a-summary-look-like)

## Introduction

This document provides a conceptual overview of summarization without going into a lot of technical or implementation details. It describes what summaries are, how / when they are generated and what do they look like. The goal is for this to be an entry point into summarization for users and developers alike.

### Summary vs snapshot

The terms summary and snapshot are sometimes used interchangeably. Both represent the state of a container at a point in time. They differ in some respects which are described in [this FAQ](https://fluidframework.com/docs/faq/#summarization).

## Why do we need summaries?

A 'summary' captures the state of a container at a point in time. Without it, a client would have to apply every operation in the op log, even if those operations no longer affected the current state (e.g. op 1 inserts ‘h’ and op 2 deletes ‘h’). For very large op logs, this would be very expensive for clients to both process and download from the service.
Instead, when a client opens a collaborative document, it can download a snapshot of the container, and simply process new operations from that point forward.

## Who generates summaries?

Summaries can be generated by any client connected in "write" mode. In the current implementation, summaries are generated by a separate non-interactive client. Using a separate client is an optimization - this client doesn't have to take local changes into account which can make the summary process more complicated. A summarizer client has the following characteristics:

-   All the clients connected to the document participate in a process called "summary client election" to elect a "parent summarizer" client. Typically, it's the oldest "write" client connected to the document. The parent summarizer client spawns a "summarizer" client which is responsible for summarization.
-   A summarizer client is like any other client connected to the document except that users cannot interact with this client, and it only works on the state it receives from other clients. It has a brand-new container with its own connection to services.

> Note that if the summarizer client closes, the "summary client election" process will choose a new one, if applicable. The default "summary client election" algorithm is to select the oldest "write" client as the parent summarizer client which in turn will create the summarizer client.

## When are summaries generated?

The summarizer client periodically generates summary based on heuristics calculated based on configurations such as the number of user or system ops received, the amount of time a client has been idle (hasn't received any ops), the maximum time since last summary, maximum number of ops since last summary, etc. The heuristic configurations are defined by an _ISummaryConfigurationHeuristics_ interface defined in [this file](../../src/containerRuntime.ts).

The summarizer client uses a default set of configurations defined by _DefaultSummaryConfiguration_ in [this file](../../src/containerRuntime.ts). These can be overridden by providing a new set of configurations as part of container runtime options during creation.

## How are summaries generated?

When summarization process is triggered, every object in the container's object tree that has data to be summarized is asked to generate its summary, starting at the container runtime which is at the root. There are various objects that participate in the summary process and generate its summary such as data stores, DDSes, garbage collector, blob manager, id compressor, etc. Note that the user data is in the DDSes.

### Summary Lifecycle

The lifecycle of a summary starts when a "parent summarizer" client is elected.
-   The parent summarizer spawns a non-interactive summarizer client.
-   The summarizer client periodically starts a summary as per heuristics. A summary happens at a particular sequence number called the "summary sequence number" or reference sequence number for the summary.
-   The container runtime (or simply runtime) generates a summary tree (described in the ["What does a summary look like?"](#what-does-a-summary-look-like) section below).
-   The runtime uploads the summary tree to the Fluid storage service which returns a handle (unique id) to the summary if the upload is successful. Otherwise, it returns a failure.
-   The runtime submits a "summarize" op to the Fluid ordering service containing the uploaded summary handle and the summary sequence number.
-   The ordering service stamps it with a sequence number (like any other op) and broadcasts the summarize op.
-   Another service on the server responds to the summarize op.
    -   If the summary is accepted, it sends a "summary ack" with the summary sequence number and a summary handle. This handle may or may not be the same as the one in summary op depending on whether this is a single-commit or two-commit summary. More details on this below.
    -   If the summary is rejected, it sends a "summary nack" with the details of the summary op
-   The runtime completes the summary process on receiving the summary ack / nack. The runtime has a timeout called "maxAckWaitTime" and if the summary op, ack or nack is not received within this time, it will fail this summary.

### Single-commit vs two-commit summaries

By default, Fluid uses "two-commit summaries" mode where the two commits refer to the storage committing the summary twice and returning two different handles for it - One when the summary is uploaded and second, on responding to the summary op via a summary ack. In this mode, when the server receives the summary op, it augments the corresponding summary with a "protocol" blob hence generating a new commit and new handle for this summary which it returns in the summary ack.

Fluid is switching to "single-commit summary" mode where the client adds the "protocol" blob when uploading the summary. Thus, the server doesn't need to augment the summary and the summary ack is no longer required. As soon as the summary is uploaded (first commit), the summary process is complete. The "summarize" op then is just a way to indicate that a summary happened, and it has details of the summary

### Incremental summaries

Summaries are incremental, i.e., if an object (or node) did not change since the last summary, it doesn't have to re-summarize its entire contents. Fluid supports the concept of a summary handle defined in [this file](../../../../../common/lib/protocol-definitions/src/summary.ts). A handle is a path to a subtree in a snapshot and it allows objects to reference a subtree in the previous snapshot, which is essentially an instruction to storage to find that subtree and populate into new summary.

Say that a data store or DDS did not change since the last summary, it doesn't have to go through the whole summary process described above. It can instead return an ISummaryHandle with path to its subtree in the previous snapshot. The same applies to other types of content like a single content blob within an object's summary tree.

### Resiliency

The summarization process is designed to be resilient - Basically, a document will eventually summarize and make progress even if there are intermittent failures or disruptions. Some examples of steps taken to achieve this:
-   Last summary - Usually, if the "parent summarizer" client disconnects or shuts down, the "summarizer" client also shuts down and the summarizer election process begins. However, if there a certain number of un-summarized ops, the summarizer client will perform a "last summary" even if the parent shuts down. This is done to make progress in scenarios where new summarizer clients are closed quickly because the parent summarizer keeps disconnecting repeatedly.
-   Retries - The summarizer has a retry mechanism which can identify certain types of intermittent failures either in the client or in the server. It will retry the summary attempt for these failures a certain number of times. This helps in cases where there are intermittent failures such as throttling errors from the server which goes away after waiting for a while.

## What does a summary look like?

The format of a summary is described in [summary formats](./summaryFormats.md).
