# BulkSearch

### Superfast, lightweight and read-write optimized full text search library.

When it comes to the overall speed, BulkSearch outperforms every searching library out there and also provides flexible search capabilities like multi-word, phonetics or partial matching. Adding, updating or removing items are also fast as searching them. When your index don't needs to be updated continuously then <a href="https://github.com/nextapps-de/flexsearch" target="_blank">FlexSearch</a> may be a better choice. BulkSearch also provides you a asynchronous processing model to perform any updates on the index in background. 

Benchmark: <a href="https://jsperf.com/bulksearch" target="_blank">https://jsperf.com/bulksearch</a>

All Features:
<ul>
    <li>Partial Words</li>
    <li>Multiple Words</li>
    <li>Flexible Word Order</li>
    <li><a href="#phonetic">Phonetic Search</a></li>
    <li>Limit Results</li>
    <li>Caching</li>
    <li>Asynchronous Mode</li>
    <li>Custom Matchers</li>
    <li>Custom Encoders</li>
</ul>

## Installation

##### Node.js

```npm
npm install bulksearch
```

In your code include as follows:

```javascript
var BulkSearch = require("BulkSearch");
```

##### HTML / Javascript

```html
<html>
<head>
    <script src="https://cdn.rawgit.com/nextapps-de/bulksearch/master/bulksearch.min.js"></script>
</head>
...
```

__AMD__

```javascript
var BulkSearch = require("BulkSearch");
```

## Usage (API)

#### Create a new index

```js
var index = new BulkSearch();
```

alternatively you can also use:

```js
var index = BulkSearch.create();
```

##### Create a new index with custom options

> BulkSearch.__create__(options)

```js
var index = new BulkSearch({

    // default values:

    type: "integer",
    encode: "icase",
    boolean: "and",
    size: 4000,
    depth: 3,
    multi: false,
    strict: false,
    ordered: false,
    async: false,
    cache: false
});
```

__Read more:__ <a href="#phonetic">Phonetic Search</a>, <a href="#compare">Phonetic Comparison</a>, <a href="#memory">Improve Memory Usage</a>

#### Add items to an index

> Index.__add___(id, string)

```js
index.add(10025, "John Doe");
```

#### Search items

> Index.__search__(string, limit, callback)

```js
index.search("John");
```

##### Limit the result

```js
index.search("John", 10);
```

##### Perform queries asynchronously

```js
index.search("John", function(result){
    
    // array of results
});
```

#### Update item to the index

> Index.__update__(id, string)

```js
index.update(10025, "Road Runner");
```

#### Remove item to the index

> Index.__remove__(id)

```js
index.remove(10025);
```

#### Destroy the index

```js
index.destroy();
```

#### Initialize the index

> Index.__init__(options)

```js
index.init();
```

#### Add custom matcher

> Index.__addMatcher__(_KEY_VALUE_PAIRS_)

```js
index.addMatcher({

    'ä': 'a', // replaces all 'ä' to 'a'
    'ö': 'o',
    'Ü': 'u'
});
```

#### Add custom encoder

```js
var index = new BulkSearch({

    encode: function(str){
    
        // do something with str ...
        
        return str;
    }
});
```

#### Get info

```js
index.info();
```

Returns information about the index, e.g.:

```json
{
    "bytes": 3600356288,
    "chunks": 9,
    "fragmentation": 0,
    "fragments": 0,
    "id": 0,
    "length": 7798,
    "matchers": 0,
    "size": 10000,
    "status": false
}
```

__Note:__ When the fragmentation value is about 50% or higher, your should consider using _cleanup()_ to free all fragmented available memory.

#### Optimize / Cleanup index

```js
index.cleanup();
```

<a name="options" id="options"></a>
## Options

<table>
    <tr>
        <th align="left">Option</th>
        <th align="left">Values</th>
        <th align="left">Description</th>
    </tr>
    <tr></tr>
    <tr>
        <td align="top">type</td>
        <td vertical="top" vertical-align="top">
            "byte"<br>
            "short"<br>
            "integer"<br>
            "float"<br>
            "string"
        </td>
        <td vertical-align="top">The data type of passed IDs has to be specified on creation. It is recommended to uses to most lowest possible data range here, e.g. use "short" when IDs are not higher than 65,535.</td>
    </tr>
    <tr></tr>
    <tr>
        <td align="top">encode</td>
        <td>
            false<br>
            "icase"<br>
            "simple"<br>
            "advanced"<br>
            "extra"<br>
            {function}
        </td>
        <td>The encoding type. Choose one of the built-ins or pass a custom encoding function.</td>
    </tr>
    <tr></tr>
    <tr>
        <td align="top">boolean</td>
        <td>
            "and"<br>
            "or"
        </td>
        <td>The applied boolean model when comparing multiple words. <b>Note:</b> When using "or" the first word is also compared with "and". Example: a query with 3 words, results has either: matched word 1 & 2 and matched word 1 & 3.</td>
    </tr>
    <tr></tr>
    <tr>
        <td align="top">size</td>
        <td>2500 - 10000</td>
        <td>The size of chunks. It depends on content length which value fits best. Short content length (e.g. User names) are faster with a chunk size of 2,500. Bigger text runs faster with a chunk size of 10,000. <b>Note:</b> It is recommended to use a minimum chunk size of the maximum content length which has to be indexed to prevent fragmentation.</td>
    </tr>
    <tr></tr>
    <tr>
        <td align="top">depth</td>
        <td>0 - 6</td>
        <td>Set the depth of register. It is recommended to use a value in relation to the number of stored index and content length for an optimum performance-memory value. <b>Note:</b> Increase this options carefully!</td>
    </tr>
    <tr></tr>
    <tr>
        <td align="top">multi</td>
        <td>
            true<br>
            false
        </td>
        <td>Enable multi word processing.</td>
    </tr>
    <tr></tr>
    <tr>
        <td align="top">ordered</td>
        <td>
            true<br>
            false
        </td>
        <td>Multiple words has to be the same order as the matched entry.</td>
    </tr>
    <tr></tr>
    <tr>
        <td align="top">strict</td>
        <td>
            true<br>
            false
        </td>
        <td>Matches exactly needs to be started with the query.</td>
    </tr>
    <tr></tr>
    <tr>
        <td align="top">cache</td>
        <td>
            true<br>
            false
        </td>
        <td>Enable caching.</td>
    </tr>
</table>

<a name="phonetic" id="phonetic"></a>
## Phonetic Encoding

<table>
    <tr>
        <th align="left">Option</th>
        <th align="left">Description</th>
        <th align="left">Example</th>
        <th align="left">False Positives</th>
        <th align="left">Compression Level</th>
    </tr>
    <tr></tr>
    <tr>
        <td><b>false</b></td>
        <td>Turn off encoding</td>
        <td>
            Reference: "Björn-Phillipp Mayer"<br>
            Matches: "Phil"
        </td>
        <td>no</td>
        <td>no</td>
    </tr>
    <tr></tr>
    <tr>
        <td><b>icase</b></td>
        <td>Case in-sensitive encoding</td>
        <td>
            Reference: "Björn-Phillipp Mayer"<br>
            Matches: "phil"
        </td>
        <td>no</td>
        <td>no</td>
    </tr>
    <tr></tr>
    <tr>
        <td><b>simple</b></td>
        <td>Phonetic normalizations</td>
        <td>
            Reference: "Björn-Phillipp Mayer"<br>
            Matches: "bjoern fillip"
        </td>
        <td>no</td>
        <td>~ 3%</td>
    </tr>
    <tr></tr>
    <tr>
        <td><b>advanced</b></td>
        <td>Phonetic normalizations + Literal transformations</td>
        <td>
            Reference: "Björn-Phillipp Mayer"<br>
            Matches: "filip meier"
        </td>
        <td>no</td>
        <td>~ 25%</td>
    </tr>
    <tr></tr>
    <tr>
        <td><b>extra</b></td>
        <td>Phonetic normalizations + Soundex transformations</td>
        <td>
            Reference: "Björn-Phillipp Mayer"<br>
            Matches: "byorn mair"
        </td>
        <td>yes</td>
        <td>~ 50%</td>
    </tr>
</table>

<a name="compare" id="compare"></a>
### Compare Phonetic Search Results

Reference String: __"Björn-Phillipp Mayer"__

<table>
    <tr>
        <th align="left">Query</th>
        <th align="left">ElasticSearch</th>
        <th align="left">BulkSearch (iCase)</th>
        <th align="left">BulkSearch (Simple)</th>
        <th align="left">BulkSearch (Adv.)</th>
        <th align="left">BulkSearch (Extra)</th>
    </tr>
    <tr></tr>
    <tr>
        <td>björn</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>björ</td>
        <td>no</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>bjorn</td>
        <td>no</td>
        <td>no</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>bjoern</td>
        <td>no</td>
        <td>no</td>
        <td>no</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>philipp</td>
        <td>no</td>
        <td>no</td>
        <td>no</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>filip</td>
        <td>no</td>
        <td>no</td>
        <td>no</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>björnphillip</td>
        <td>no</td>
        <td>no</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>meier</td>
        <td>no</td>
        <td>no</td>
        <td>no</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>björn meier</td>
        <td>no</td>
        <td>no</td>
        <td>no</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>meier fhilip</td>
        <td>no</td>
        <td>no</td>
        <td>no</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>byorn mair</td>
        <td>no</td>
        <td>no</td>
        <td>no</td>
        <td>no</td>
        <td><b>yes</b></td>
    </tr>
    <tr>
        <td><i>(false positives)</i></td>
        <td>yes</td>
        <td><b>no</b></td>
        <td><b>no</b></td>
        <td><b>no</b></td>
        <td>yes</td>
    </tr>
</table>

<a name="memory" id="memory"></a>
## Compare Memory Usage

__Note:__ The data type of passed IDs has to be specified on creation. It is recommended to uses to most lowest possible data range here, e.g. use "short" when IDs are not higher than 65,535. 

<table>
    <tr>
        <th align="left">ID Type</th>
        <th align="left">Range of Values</th>
        <th align="left">Memory usage of every ~ 100,000 indexed words (Index + Content)</th>
    </tr>
    <tr></tr>
    <tr>
        <td>Byte</td>
        <td>0 - 255</td>
        <td>683 kb + 2.4 Mb</td>
    </tr>
    <tr></tr>
    <tr>
        <td>Short</td>
        <td>0 - 65,535</td>
        <td>1.3 Mb + 2.4 Mb</td>
    </tr>
    <tr></tr>
    <tr>
        <td>Integer</td>
        <td>0 - 4,294,967,295</td>
        <td>2.7 Mb + 2.4 Mb</td>
    </tr>
    <tr></tr>
    <tr>
        <td>Float</td>
        <td>0 - * (16 digits)</td>
        <td>5.3 Mb + 2.4 Mb</td>
    </tr>
    <tr></tr>
    <tr>
        <td>String</td>
        <td>* (unlimited)</td>
        <td>1.3 Mb <b>* <u>char length of IDs</u></b> + 2.4 Mb</td>
    </tr>
</table>

---

Author BulkSearch: Thomas Wilkerling<br>
License: <a href="http://www.apache.org/licenses/LICENSE-2.0.html" target="_blank">Apache 2.0 License</a><br>
