# teeft

## Présentation

Ce plugin propose une série d'instructions pour extraire des mots-clés d'un
texte français ou anglais en utilisant l'algorithme Teeft.

C'est le paquet officiel qui fait suite à l'expérimentation
[ezs-teeftfr](https://github.com/istex/node-ezs-teeftfr).

### Bibliographie

Cuxac P., Kieffer N., Lamirel J.C. : SKEEFT: indexing method taking into account the structure of the document. 20th Collnet meeting, 5-8 Nov 2019, Dalian, China.

## Installation

```bash
npm install @ezs/core
npm install @ezs/teeft
```

## Flux

Le principe est de décomposer le flux en une série d'instructions ayant chacune
ses propres paramètres (voir [usage](#usage)).

Voici la séquence typique d'instructions qui permet de lire les fichiers `.txt`
d'un répertoire et de les envoyer aux tokenizers de phrase, puis de mots;
ensuite on procède à un étiquetage grammatical, puis on extrait les termes, et
on les filtre (par fonction grammaticale), on enlève les nombres, on supprime
les mots vides, on calcule les fréquences des tokens, puis leur spécificité,
enfin, on les filtre suivant leur fréquence.

```txt
[ "/path/to/a/directory/of/documents" ] ->

[TeeftListFiles]
pattern = *.txt

--> [ "/path1", "path2", ... ] -->

[TeeftGetFilesContent]

--> [ { path, content }, ... ] -->

[TeeftSentenceTokenize]

--> [ { path, sentences: [ "sentence", ... ] }, ... ] -->

[TeeftTokenize]

--> [ { path, sentences: [ ["token", ... ], ...] }, ... ] -->

[TeeftNaturalTag]

--> [  { path, sentences: [ [
  {
    token: "token",
    tag: [ "tag", ...]
  }, ...
 ], ... ] }, ... ]


[TeeftExtractTerms]
nounTag = NOM
adjTag = ADJ

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length
  },
  {
    term: "multiterm",
    frequency,
    length
  }, ...
 ] }, ... ]

[TeeftFilterTags]
tags = NOM
tags = ADJ

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length
  },
  {
    term: "multiterm",
    frequency,
    length
  }, ...
 ] }, ... ]

[TeeftRemoveNumbers]

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length
  },
  {
    term: "multiterm",
    frequency,
    length
  }, ...
 ] }, ... ]

[TeeftStopWords]

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length
  },
  {
    term: "multiterm",
    frequency,
    length
  }, ...
 ] }, ... ]

[TeeftRemoveShortTerms]

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length
  },
  {
    term: "multiterm",
    frequency,
    length
  }, ...
 ] }, ... ]

[TeeftRemoveLongTerms]

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length
  },
  {
    term: "multiterm",
    frequency,
    length
  }, ...
 ] }, ... ]

[TeeftRemoveWeirdTerms]

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length
  },
  {
    term: "multiterm",
    frequency,
    length
  }, ...
 ] }, ... ]

[TeeftSumUpFrequencies]

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length
  },
  {
    term: "multiterm",
    frequency,
    length
  }, ...
 ] }, ... ]

[TeeftSpecificity]
sort = true

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length,
    specificity,
  },
  {
    term: "multiterm",
    frequency,
    length,
    specificity
  }, ...
 ] }, ... ]

[TeeftFilterMonoFreq]

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length,
    specificity,
  },
  {
    term: "multiterm",
    frequency,
    length,
    specificity
  }, ...
 ] }, ... ]

[TeeftFilterMultiSpec]

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length,
    specificity,
  },
  {
    term: "multiterm",
    frequency,
    length,
    specificity
  }, ...
 ] }, ... ]

[dump]
indent = true
```

## usage

<!-- Generated by documentation.js. Update this documentation by updating the source code. -->

#### Table of Contents

*   [TeeftExtractTerms](#teeftextractterms)
*   [TeeftFilterMonoFreq](#teeftfiltermonofreq)
*   [TeeftFilterMultiSpec](#teeftfiltermultispec)
*   [TeeftFilterTags](#teeftfiltertags)
*   [TeeftGetFilesContent](#teeftgetfilescontent)
*   [TeeftListFiles](#teeftlistfiles)
*   [TeeftNaturalTag](#teeftnaturaltag)
*   [TeeftRemoveLongTerms](#teeftremovelongterms)
*   [TeeftRemoveNumbers](#teeftremovenumbers)
*   [TeeftRemoveShortTerms](#teeftremoveshortterms)
*   [TeeftRemoveWeirdTerms](#teeftremoveweirdterms)
*   [TeeftSentenceTokenize](#teeftsentencetokenize)
*   [TeeftSpecificity](#teeftspecificity)
*   [TeeftStopWords](#teeftstopwords)
*   [TeeftSumUpFrequencies](#teeftsumupfrequencies)
*   [TeeftTokenize](#teefttokenize)
*   [TeeftToLowerCase](#teefttolowercase)

### TeeftExtractTerms

*   **See**: <https://github.com/istex/sisyphe/blob/master/src/worker/teeft/lib/termextractor.js>

Take an array of objects `{ path, sentences: [token, tag: ["tag"]]}`. Regroup
multi-terms when possible (noun + noun, adjective + noun, *etc*.), and
computes statistics (frequency, *etc*.).

Use `lang` or `nounTag`, `adjTag`, but not `lang` and the others at the same
time. `lang` is enough to set `nounTag` and `adjTag`.

#### Parameters

*   `lang` **[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** language of the terms to extract (`en` or
    `fr`) (optional, default `'fr'`)
*   `nounTag` **[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** noun tag (`NOM` in French, `NN` in English) (optional, default `'NOM'`)
*   `adjTag` **[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** adjective tag (`ADJ` in French, `JJ` in
    English) (optional, default `'ADJ'`)

#### Examples

```javascript
[{
   path: '/path/1',
   sentences:
   [[
     { token: 'elle', tag: ['PRO:per'] },
     { token: 'semble', tag: ['VER'] },
     { token: 'se', tag: ['PRO:per'] },
     { token: 'nourrir', tag: ['VER'] },
     {
       token: 'essentiellement',
       tag: ['ADV'],
     },
     { token: 'de', tag: ['PRE', 'ART:def'] },
     { token: 'plancton', tag: ['NOM'] },
     { token: 'frais', tag: ['ADJ'] },
     { token: 'et', tag: ['CON'] },
     { token: 'de', tag: ['PRE', 'ART:def'] },
     { token: 'hotdog', tag: ['UNK'] }
   ]]
}]
```

Returns **any** same as input, with `term` replacing `token`, `length`, and
`frequency`

### TeeftFilterMonoFreq

Filter the `data`, keeping only multiterms and frequent monoterms.

Minimal frequency (`minFrequency` parameter) has a default value
automatically computed from the number of tokens in the document.

#### Parameters

*   `multiLimit` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)** threshold for being a multiterm (in tokens
    number) (optional, default `2`)
*   `minFrequency` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)** minimal frequency to be taken as a
    frequent term (optional, default `7`)

### TeeftFilterMultiSpec

Filter multiterms to keep only multiterms which specificity is higher than
multiterms' average specificity.

### TeeftFilterTags

Filter the text in input, by keeping only adjectives and names

#### Parameters

*   `lang` **[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** Language to set tags (`en` or `fr`)
*   `tags` **[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** Tags to keep (ex: `ADJ`, `NOM`)

### TeeftGetFilesContent

Take an array of file paths as input, and returns a list of
objects containing the `path`, and the `content` of each file.

Returns **\[{path: [string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String), content: [string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)}]** Array of { path, content }

### TeeftListFiles

Take an array of directory paths as input, a pattern, and returns a list of
file paths matching the pattern in the directories from the input.

#### Parameters

*   `pattern` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** pattern for files (ex: "\*.txt") (optional, default `"*"`)

Returns **\[[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)]** an array of file paths

### TeeftNaturalTag

POS Tagger from natural

French pos tagging using natural (and LEFFF resources)

Take an array of documents (objects: `{ path, sentences: [[]] })`

Yield an array of documents (objects:

    {
         path, sentences: [
             [{
                 token: "token",
                 tag: [ "tag", ... ]
             },
             ...]
         ]
    }

)

#### Parameters

*   `lang` **[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** language of the text to tag (possible values: `fr`, `en`) (optional, default `'en'`)

#### Examples

```javascript
[{
     path: "/path/1",
     sentences: [{ "token": "dans",      "tag": ["prep"] },
                 { "token": "le",        "tag": ["det"]  },
                 { "token": "cadre",     "tag": ["nc"] },
                 { "token": "du",        "tag": ["det"] },
                 { "token": "programme", "tag": ["nc"] }
                 },
     ]
 }]
```

### TeeftRemoveLongTerms

Remove long terms from documents (longer than 50 characters).
Documents must have a `terms` key, containing an array of objects with a
`term` key of type string..

Yields an array of documents with the same structure.

Input:

```json
[{
  "path": "/path/to/file.txt",
  "terms": [{ "term": "this very long term should really be removed 678901" },
            { "term": "abcd" }]
}]
```

Output:

```json
[{
  "path": "/path/to/file.txt",
  "terms": [{ "term": "abcd" }]
}]
```

### TeeftRemoveNumbers

Remove numbers from the terms of documents (objects `{ path, terms: [{ term,
...}] }`).

Yields an array of documents with the same structure.

### TeeftRemoveShortTerms

Remove short terms from documents (shorter than 3 characters).
Documents must have a `terms` key, containing an array of objects with a
`term` key of type string..

Yields an array of documents with the same structure.

Input:

```json
[{
  "path": "/path/to/file.txt",
  "terms": [{ "term": "a" }, { "term": "abcd" }]
}]
```

Output:

```json
[{
  "path": "/path/to/file.txt",
  "terms": [{ "term": "abcd" }]
}]
```

### TeeftRemoveWeirdTerms

Remove terms with too much non-alphanumeric characters.
Documents must have a `terms` key, containing an array of objects with a
`term` key of type string..

Yields an array of documents with the same structure.

Input:

```json
[{
  "path": "/path/to/file.txt",
  "terms": [{ "term": "αβɣδ" }, { "term": "abcd" }]
}]
```

Output:

```json
[{
  "path": "/path/to/file.txt",
  "terms": [{ "term": "abcd" }]
}]
```

### TeeftSentenceTokenize

Segment the data into an array of documents (objects `{ path, content }`).

Yield an array of documents (objects `{ path, sentences: []}`)

### TeeftSpecificity

Take documents (with a `path`, an array of `terms`, each term being an object
`{ term, frequency, length[, tag] }`).

Process objects containing frequency, add a specificity to each object, and
remove all object with a specificity below average specificity (except when
`filter` is `false`).

Can also sort the objects according to their specificity, when `sort` is
`true`.

#### Parameters

*   `lang` **[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** language to take into account (optional, default `"en"`)
*   `filter` **[Boolean](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Boolean)** filter below average specificity (optional, default `true`)
*   `sort` **[Boolean](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Boolean)** sort objects according to their specificity (optional, default `false`)

### TeeftStopWords

Filter the text in input, by removing stopwords in token

#### Parameters

*   `lang` **[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** language of the stopwords (`en` or `fr`) (optional, default `'en'`)

### TeeftSumUpFrequencies

Sums up the frequencies of identical lemmas from different chunks.

### TeeftTokenize

*   **See**: <http://yomguithereal.github.io/talisman/tokenizers/words>

Extract tokens from an array of documents (objects `{ path, sentences: [] }`).

Yields an array of documents (objects: `{ path, sentences: [[]] }`)

> **Warning**: results are surprising on uppercase sentences,
> use TeeftToLowerCase

### TeeftToLowerCase

Transform strings to lower case.

#### Parameters

*   `path` **[Array](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Array)<[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)>** path to the property to modify (optional, default `[]`)
