Class: ScrapingConnector

ScrapingConnector(window, cookieDomainRewrites)

A base class for screen-scraping connectors.

ScrapingConnector does three things:

  1. Allows window to be injected, so we can swap it for JSDOM and run the Connector outside of a browser.

  2. Supports cookie-based sessions server-side, using a cookie-jar and domain-name rewriting.

  3. Lets us get/set default headers for requests.

Constructor

new ScrapingConnector(window, cookieDomainRewrites)

Parameters:
Name Type Description
window Window

A browser-native window or JSDOM implementation.
cookieDomainRewrites Object.<string, string>

the domains to rewrite on any cookies received from the backend, eg.{"shop.com": "proxy-shop.com"}
Source:

Classes

ScrapingConnector

Methods

(static) diffCookieJars(oldJar, newJar) → {Array.<Cookie>}

Returns the cookies that are new or changed in newJar.

Parameters:
Name Type Description
oldJar CookieJar
newJar CookieJar
Source:
Returns:
Type
Array.<Cookie>

(static) rewriteCookieDomain(rewrites, cookie) → {Array.<Cookie>}

Clone cookie replace originalDomain with proxyDomain if the original cookie had an explicit domain set.

This is for server-side usage, eg. when running the IM as a proxy on a different domain than the original ecommerce backend.

Parameters:
Name Type Description
rewrites Object.<String, String>
cookie Array.<Cookie>
Source:
Returns:
Type
Array.<Cookie>

buildDocument(res) → {Document}

Return a Document constructed from a superagent HTTP response. Use to parse content out of the DOM.

Parameters:
Name Type Description
res HTTPResponse

a superagent http response, expected to be HTML.
Source:
Returns:
Type
Document

getCookies() → {Array.<String>}

Get all new or changed cookies in the Connector's cookie jar, since creation or the last call to setCookies().

Each returned cookie can be set directly as a header on an HTTP response in the "set-cookie" header.

This is only for use outside a browser. In the browser, cookies are handled automatically.

Example return value:

["a=b; domain=example.com; path=/", "c=d; domain=example2.com; path=/"]

Source:
Returns:
Type
Array.<String>

getDefaultHeaders()

Source:

setCookies(cookies)

Set the initial cookies on the connector.

The cookies parameter must be in the format for the Cookie header taken directly from an HTTP request, eg: "a=b; c=d"

Parameters:
Name Type Description
cookies String
Source:

setDefaultHeaders()

Source: