A BrowserChannel server.

• Its still pretty young, so there's probably bugs lurking around and the API will still change quickly.
• Its missing integration tests

It works in all the browsers I've tried.

I've written this using the literate programming style to try it out. So, thats why there's a million comments everywhere.

The server is implemented as connect middleware. Its intended to be used like this:

server = connect( browserChannel (client) -> client.send 'hi' )

Dependancies, helper methods and constant data

parse helps us decode URLs in requests

querystring will help decode the URL-encoded forward channel data

fs is used to read & serve the client library

Client sessions are `EventEmitters

Client session Ids are generated using node-hat

randomInt(n) generates and returns a random int smaller than n (0 <= k < n)

randomArrayElement(array) Selects and returns a random element from array

For testing we'll override setInterval, etc with special testing stub versions (so we don't have to actually wait for actual time. To do that, we need local variable versions (I don't want to edit the global versions). ... and they'll just point to the normal versions anyway.

The module is configurable

An optional array of host prefixes. Each browserchannel client will randomly pick from the list of host prefixes when it connects. This reduces the impact of per-host connection limits.

All host prefixes should point to the same server. Ie, if your server's hostname is example.com and your hostPrefixes contains ['a', 'b', 'c'], a.example.com, b.example.com and c.example.com should all point to the same host as example.com.

You can specify the base URL which browserchannel connects to. Change this if you want to scope browserchannel in part of your app, or if you want /channel to mean something else, or whatever.

We'll send keepalives every so often to make sure the http connection isn't closed by eagar clients. The standard timeout is 30 seconds, so we'll default to sending them every 20 seconds or so.

After awhile (30 seconds or so) of not having a backchannel connected, we'll evict the session completely. This will happen whenever a user closes their browser.

All server responses set some standard HTTP headers. To be honest, I don't know how many of these are necessary. I just copied them from google.

The nocache headers in particular seem unnecessary since each client request includes a randomized zx=junk query parameter.

Gmail also sends this, though I'm not really sure what it does... 'X-Xss-Protection': '1; mode=block'

The one exception to that is requests destined for iframes. They need to have content-type: text/html set for IE to process the juicy JS inside.

Google's browserchannel server adds some junk after the first message data is sent. I assume this stops some whole-page buffering in IE. I assume the data used is noise so it doesn't compress.

I don't really know why google does this. I'm assuming there's a good reason to it though.

If the user is using IE, instead of using XHR backchannel loaded using a forever iframe. When data is sent, it is wrapped in tags which call functions in the browserchannel library.

This method wraps the normal .writeHead(), .write() and .end() methods by special versions which produce output based on the request's type.

This is not used for:

• The first channel test
• The first bind connection a client makes. The server sends arrays there, but the connection is a POST and it returns immediately. So that request happens using XHR/Trident like regular forward channel requests.

If the iframe is making the request using a secondary domain, I think we need to set the domain to the original domain so that we can call the response methods.

Make sure the domain doesn't contain anything by naughty by JSON.stringify()-ing it before passing it to the client. There are XSS vulnerabilities otherwise.

The data is passed to m(), which is bound to onTridentRpcMessage_ in the client.

Once the data has been received, the client needs to call d(), which is bound to onTridentDone_ with success=true. The weird spacing of this is copied from browserchannel. Its really not necessary.

This is a helper method for signalling an error in the request back to the client.

The HTML (iframe) handler has no way to discover that the embedded script tag didn't complete successfully. To signal errors, we return 200 OK and call an exposed rpcClose() method on the page.

For some reason, sending data during the second test (111112) works slightly differently for XHR, but its identical for html encoding. We'll use a writeRaw() method in that case, which is copied in the case of html.

For normal XHR requests, we send data normally.

For telling the client its done bad.

It turns out google's server isn't particularly fussy about signalling errors using the proper html RPC stuff, so this is useful for html connections too.

Parsing client maps from the forward channel

The client sends data in a series of url-encoded maps. The data is encoded like this:

count=2&ofs=0&req0_x=3&req0_y=10&req1_abc=def

First, we need to buffer up the request response and query string decode it.

Next, we'll need to decode the incoming client data into an array of objects.

The data could be in two different forms:

• Classical browserchannel format, which is a bunch of string->string url-encoded maps
• A JSON object

We can tell what format the data is in by inspecting the content-type header

URL Encoded data

Essentially, url encoded the data looks like this:

{ count: '2', ofs: '0', req0_x: '3', req0_y: '10', req1_abc: 'def' }

... and we will return an object in the form of [{x:'3', y:'10'}, {abc: 'def'}, ...]

JSON Encoded data

JSON encoded the data looks like:

{ ofs: 0 , data: [null, {...}, 1000.4, 'hi', ...] }

or null if there's no data.

This function returns null if there's no data or {ofs, json:[...]} or {ofs, maps:[...]}

We'll restructure it slightly to mark the data as JSON rather than maps.

Maps. Ugh.

ofs will be missing if count is zero

Scan through all the keys in the data. Every key of the form: req123_xxx will be used to populate its map.

The client uses mapX_type=_badmap to signify an error encoding a map.

This is a helper method to order the handling of messages / requests / whatever.

Use it like this: inOrder = order 0

inOrder 1, -> console.log 'second' inOrder 0, -> console.log 'first'

Start is the ID of the first element we expect to receive. If we get data for earlier elements, we'll play them anyway if playOld is truthy.

Base is the ID of the (missing) element at the start of the queue

The queue will start with about 10 elements. Elements of the queue are undefined if we don't have data for that queue element.

Its important that all the cells of the array are truthy if we have data. We'll use an empty function instead of null.

Ignore old messages, or play them back immediately if playOld=true

We need access to the client's sourcecode. I'm going to get it using a synchronous file call (it'll be fast anyway, and only happen once).

I'm also going to set an etag on the client data so the browser client will be cached. I'm kind of uncomfortable about adding complexity here because its not like this code hasn't been written before, but.. I think a lot of people will use this API.

I should probably look into hosting the client code as a javascript module using that client-side npm thing.

This is mostly to help development, but if the client is recompiled, I'll pull in a new version. This isn't tested by the unit tests - but its not a big deal.

Putting a synchronous file call here will stop the whole server while the client is reloaded. Again, this will only happen during development so its not a big deal.

The server middleware

The server module returns a function, which you can call with your configuration options. It returns your configured connect middleware, which is actually another function.

Strip off a trailing slash in base.

Add a leading slash back on base

Host prefixes provide a way to skirt around connection limits. They're only really important for old browsers.

Create a new client session.

This method will start a new client session.

Session ids are generated by node-hat. They are guaranteed to be unique. This method is synchronous, because a database will never be involved in browserchannel session management. Browserchannel sessions only last as long as the user's browser is open. If there's any connection turbulence, the client will reconnect and get a new session id.

Sometimes a client will specify an old session ID and old array ID. In this case, the client is reconnecting and we should evict the named session (if it exists).

Create a new session. Sessions extend node's EventEmitter so they have access to goodies like session.on(event, handler), session.emit('paarty'), etc.

The session's unique ID for this connection

The client stores its IP address and headers from when it first opened the session. The handler can use this information for authentication or something.

The session is a little state machine. It has the following states:

• init: The session has been created and its sessionId hasn't been sent yet. The session moves to the ok state when the first data chunk is sent to the client.

• ok: The session is sitting pretty and ready to send and receive data. The session will spend most of its time in this state.

• closed: The session has been removed from the session list. It can no longer be used for any reason.

  It is invalid to send arrays to a session while it is closed. Unless you're Bruce Willis...

The state is modified through this method. It emits events when the state changes. (yay)

The server sends messages to the client via a hanging GET request. Of course, the client has to be the one to open that request.

This is a handle to null, or {res, methods, chunk}

• res is the http response object
• methods is a map of send(), etc methods for communicating properly with the backchannel - this will be different if the request comes from IE or not.
• chunk specifies whether or not we're going to keep the connection open across multiple messages. If there's a buffering proxy in the way of the connection, we can't respond a bit at a time, so we close the backchannel after each data chunk. The client decides this during testing and passes a CI= parameter to the server when the backchannel connection is established.
• bytesSent specifies how many bytes of data have been sent through the backchannel. We periodically close the backchannel and let the client reopen it, so things like the chrome web inspector stay usable.

The server sends data to the client by sending arrays. It seems a bit silly that client->server messages are maps and server->client messages are arrays, but there it is.

Each entry in this array is of the form [id, data].

lastArrayId is the array ID of the last queued array

Every request from the client has an AID parameter which tells the server the ID of the last request the client has received. We won't remove arrays from the outgoingArrays list until the client has confirmed its received them.

In lastSentArrayId we store the ID of the last array which we actually sent.

I would like this method to be private or something, but it needs to be accessed from the HTTP request code below. The _ at the start will hopefully make people think twice before using it.

We'll start the heartbeat interval and clear out the session timeout. The session timeout will be started again if the backchannel connection closes for any reason.

When a new backchannel is created, its possible that the old backchannel is dead. In this case, its possible that previously sent arrays haven't been received. By resetting lastSentArrayId, we're effectively rolling back the status of sent arrays to only those arrays which have been acknowledged.

Send any arrays we've buffered now that we have a backchannel

If we haven't sent anything for 15 seconds, we'll send a little ['noop'] to the client so it knows we haven't forgotten it. (And to make sure the backchannel connection doesn't time out.)

This method removes the back channel and any state associated with it. It'll get called when the backchannel closes naturally, is replaced or when the connection closes.

clearBackChannel doesn't do anything if we call it repeatedly.

Its important that we only delete the backchannel if the closed connection is actually the backchannel we're currently using.

Conveniently, clearTimeout has no effect if the argument is null.

Whenever we don't have a backchannel, we run the session timeout timer.

This method sets / resets the heartbeat timeout to the full 15 seconds.

The session will close if there's been no backchannel for awhile.

Since the session doesn't start with a backchannel, we'll kick off the timeout timer as soon as its created.

The arrays get removed once they've been acknowledged

I've got no idea what to do if we get an exception thrown here. The session will end up in an inconsistant state...

Queue an array to be sent. The optional callbacks notifies a caller when the array has been sent, and then received by the client.

queueArray returns the ID of the queued data chunk.

The session has just been created. The first thing it needs to tell the client is its session id and host prefix and stuff.

It would be pretty easy to add a callback here setting the client status to 'ok' or something, but its not really necessary. The client has already connected once the first POST /bind has been received.

Send the array data through the backchannel. This takes an optional callback which will be called with no arguments when the client acknowledges the array, or called with an error object if the client disconnects before the array is sent.

queueArray can also take a callback argument which is called when the session sends the message in the first place. I'm not sure if I should expose this through send - I can't tell if its useful beyond the server code.

Maps

The client sends maps to the server using POST requests. Its possible for the requests to come in out of order, so sometimes we need to buffer up incoming maps and reorder them before emitting them to the user.

Each map has an ID (which starts at 0 when the session is first created).

We'll emit received data to the user immediately if they're in order, but if they're out of order we'll use the little order helper above to order them. The order helper is instructed to not emit any old messages twice.

There's a potential DOS attack here whereby a client could just spam the server with out-of-order maps until it runs out of memory. We should dump a session if there are too many entries in this dictionary.

This method is called whenever we get maps from the client. Offset is the ID of the first map. The data could either be maps or JSON data. If its maps, data contains {maps} and if its JSON data, maps contains {JSON}.

Browserchannel has 2 different mechanisms for consistantly ordering messages in the forward channel:

• Each forward channel request contains a request ID (RID=X), which start at a random value (set with the first session create packet). These increment by 1 with each request.

  If a request fails, it might be retried with the same RID as the previous message, and with extra maps tacked on the end. We need to handle the maps in this case.

• Each map has an ID, counting from 0. ofs= in the POST data tells the server the ID of the first map in a request.

As far as I can tell, the RID stuff can mostly be ignored. The one place it is important is in handling disconnect messages. The session should only be disconnected by a disconnect message when the preceeding messages have been received.

All requests are handled in order too, though if not for disconnecting I don't think it would matter. Because of the funky retry-has-extra-maps logic, we'll allow processing requests twice.

First, classic browserchannel maps.

If an exception is thrown during this loop, I'm not really sure what the behaviour should be.

The funky do expression here is used to pass the map into the closure. Another way to do it is to index into the data.maps array inside the function, but then I'd need to pass the index to the closure anyway.

If you specify the key as JSON, the server will try to decode JSON data from the map and emit 'message'. This is a much nicer way to message the server.

We have data.json. We'll just emit it directly.

When we receive forwardchannel data, we reply with a special little 3-variable array to tell the client if it should reopen the backchannel.

This method returns what the forward channel should reply with.

Find the arrays have been sent over the wire but haven't been acknowledged yet

We don't care about the length of the array IDs or callback functions. I'm actually not sure what data the client expects here - the value is just used in a rough heuristic to determine if the backchannel should be reopened.

Encoding server arrays for the back channel

The server sends data to the client in chunks. Each chunk is a JSON array prefixed by its length in bytes.

The array looks like this:

[ [100, ['message', 'one']], [101, ['message', 'two']], [102, ['message', 'three']] ]

Each individial message is prefixed by its array id, which is a counter starting at 0 when the session is first created and incremented with each array.

This will actually send the arrays to the backchannel on the next tick if the backchannel is alive.

I've abused outgoingArrays to also contain some callbacks. We only send [id, data] to the client.

Away!

Fire any send callbacks on the messages. These callbacks should only be called once. Again, not sure what to do if there are exceptions here.

The first backchannel is the client's initial connection. Once we've sent the first data chunk to the client, we've officially opened the connection.

The client's reported application version, or null. This is sent when the connection is first requested, so you can use it to make your application die / stay compatible with people who don't close their browsers.

Signal to a client that it should stop trying to connect. This has no other effect on the server session.

stop takes a callback which will be called once the message has been sent by the server. Typically, you should call it like this:

session.stop -> session.close()

I considered making this automatically close the connection after you've called it, or after you've sent the stop message or something, but if I did that it wouldn't be obvious that you can still receive messages after stop() has been called. (Because you can!). That would never come up when you're testing locally, but it would come up in production. This is more obvious.

This closes a session and makes the server forget about it.

The client might try and reconnect if you only call close(). It'll get a new session if it does so.

close takes an optional message argument, which is passed to the send event handlers.

You can't double-close.

console.log "closed #{@id}"

This is the returned middleware. Connect middleware is a function which takes in an http request, an http response and a next method.

The middleware can do one of two things:

• Handle the request, sending data back to the server via the response
• Call next(), which allows the next middleware in the stack a chance to handle the request.

console.warn req.method, req.url

If base is /foo, we don't match /foobar. (Currently no unit tests for this)

Serving the client

The browserchannel server hosts a usable web client library at /CHANNEL/bcsocket.js. This library wraps the google closure library client implementation.

If I have time, I would like to write my own version of the client to add a few features (websockets, message acknowledgement callbacks) and do some manual optimisations for speed. However, the current version works ok.

This code is manually tested because it looks like its impossible to send HEAD requests using nodejs's HTTP library at time of writing (0.4.12). (Yeah, I know, rite?)

Connection testing

Before the browserchannel client connects, it tests the connection to make sure its working, and to look for buffering proxies.

The server-side code for connection testing is completely stateless.

This server only supports browserchannel protocol version 8. I have no idea if 400 is the right error here.

Phase 1: Server info

The client is requests host prefixes. The server responds with an array of ['hostprefix' or null, 'blockedprefix' or null].

Actually, I think you might be able to return [] if neither hostPrefix nor blockedPrefix is defined. (Thats what google wave seems to do)

• hostprefix is subdomain prepended onto the hostname of each request. This gets around browser connection limits. Using this requires a bank of configured DNS entries and SSL certificates if you're using HTTPS.

• blockedprefix provides network admins a way to blacklist browserchannel requests. It is not supported by node-browserchannel.

We add an extra special header to tell the client that this server likes json-encoded forward channel data over form urlencoded channel data.

It might be easier to put these headers in the response body or increment the version, but that might conflict with future browserchannel versions.

This is a straight-up normal HTTP request like the forward channel requests. We don't use the funny iframe write methods.

Phase 2: Buffering proxy detection

The client is trying to determine if their connection is buffered or unbuffered. We reply with '11111', then 2 seconds later '2'.

The client should get the data in 2 chunks - but they won't if there's a misbehaving corporate proxy in the way or something.

BrowserChannel connection

Once a client has finished testing its connection, it connects.

BrowserChannel communicates through two connections:

• The forward channel is used for the client to send data to the server. It uses a POST request for each message.
• The back channel is used to get data back from the server. This uses a hanging GET request. If chunking is disallowed (ie, if the proxy buffers) then the back channel is closed after each server message.

I'm copying the behaviour of unknown SIDs below. I don't know how the client is supposed to detect this error, but, eh. The other choice is to return writeError ...

All browserchannel connections have an associated client object. A client is created immediately if the connection is new.

This is a special error code for the client. It tells the client to abandon its connection request and reconnect.

For some reason, google replies with the same response on HTTP and HTML requests here. I'll follow suit, though its a little weird. Maybe I should do the same with all client errors?

Forward Channel

The session is new! Make them a new session object and let the application know.

The initial forward channel request is also used as a backchannel for the server's initial data (session id, etc). This connection is a little bit special - it is always encoded using length-prefixed json encoding and it is closed as soon as the first chunk is sent.

If the onConnect handler called close() immediately, session.state can be already closed at this point. I'll assume there was an authentication problem and treat this as a forbidden connection attempt.

On normal forward channels, we reply to the request by telling the session if our backchannel is still live and telling it how many unconfirmed arrays we have.

Back channel

GET messages are usually backchannel requests (server->client). Backchannel messages are handled by the session object.

The client can manually disconnect by making a GET request with TYPE='terminate'

We don't send any data in the response to the disconnect message.

The client implements this using an img= appended to the page.

We'll 404 the user instead of letting another handler take care of it. Users shouldn't be using the specified URL prefix for anything else.

This will override the timer methods (setInterval, etc) with the testing stub versions, which are way faster.