Dealing with Large Files

Archives, and especially X-ray archives, present special problems for displaying data in a web browser. Instead of images, X-ray data consists of detected photons stored in FITS binary tables. This means that the file size is related to the number of detected photons, not the image dimensions. Indeed, X-ray data files can be hundreds of megabytes in size. Downloading (and then binning) these tables can be painfully slow, especially if the aim simply is to offer a quick look at the data.

Of course, FITS image files also present a problem if they are large. A 2048 x 2048 32-bit FITS image has a file size of more than 16Mb, which can take 30 seconds or more to download on a DSL line. A 12000 x 12000 Hubble Deep field gzip'ed FITS image can have a file size of more than 300Mb. Thus, we need alternatives to simply downloading large files for display.

One possibility for dealing with large files is to download a smaller representation file. For some purposes, a small representation file can be used to identify spatial regions of interest, which then can be extracted at full resolution. In other cases, viewing a small image representation can be sufficient, especially if you still have the ability to perform analysis on the original file.

At present, JS9 supports two different ways of dealing with large files through the use of smaller representation files:

These two schemes are described below.

Parent Files and Spatial Region Extraction

NB: use this service with care. It allows users to consume disk resources!

Consider the 12000 x 12000 300Mb Hubble Deep Field image mentioned above. This file could take several minutes to download. Even if stored locally, it requires more than 500Mb of memory to display, which is close to the memory limit of some browsers. But you can bin this file by a factor of 24 into a gzip'ed 512 x 512 FITS image with a file size of approximately 500Kb. This representation file would download quickly and give a good overview of the entire data set. If you then could extract a spatial region of interest from the original file at full resolution, you would, in essence, be panning around the original image without the transmission and memory overhead associated with processing the whole file.

JS9 supports this sort of representation file display and image extraction. To utilize it at your site, you must configure a Server-side (Back-end) Helper (using Node.js or CGI), and set up Server-side Analysis to perform the JS9-supplied imsection task.

In addition, the Server-side helper must support temporary user work directories in which to store the section files. This is accomplished by setting the globalOpts.workDir property in js9Prefs.json to a directory which will contain the work directories. You might want to set the globalOpts.workDirQuota property (default value of 50Mb) to limit the amount of space that can be utilized by each user. (But note that the workDirQuota is not a hard quota: imsection only checks whether the quota has been exceeded before extracting the next section. This means that users can extract one section that will bring the total size above the quota. Thus, if the quota is 10Mb and the work directory is empty, imsection will extract a 50Mb file. But it will not extract more files until already-loaded files have been closed in JS9 to bring the total under the quota.)

Assuming the back-end server is set up, you then can load a representation file using JS9.Load(), but with the addition of a parentFile property in the options object. The value of parentFile will be the relative path of the original file:

    JS9.Load("fits/f814w_bin24.fits.gz", {colormap:"cool", parentFile: "fits/f814w.fits.gz"})
When a parentFile is specified, the File menu will activate the extract image section ... menu option. From its dialog box, you will be able to set the image coordinates and size of the section to extract. Alternatively, you can choose to have these values automatically updated from a box region that you create, move, and resize.

You can also choose to extract the specified section in native format (i.e. as a binary table if the parent file is a binary table, etc.) or as an image (even if the parent is a binary table).

When the Extract button is pressed, the position and size parameters are used to extract a new FITS file from the parent file. This file is stored in the temporary work directory and displayed in JS9. Analysis can then be performed on this section file. When the File close image menu option is chosen, the section file will be deleted.

Extracting sections works especially well with large FITS images. It also can be used for binary tables. However, since the size of a binary table depends on the number of events, an extracted section can still be very large if it contains a bright source.

The PNG Representation File

Generating a representation file from an X-ray binary table for download is one solution to the large file problem. It does require setting up server-side parent file processing (including temporary storage of image section files). An alternative that does not require a server-side helper is to create a special PNG representation file explicitly tied to the original event data. Once downloaded into JS9, the image can be reconstructed from this file for display. The file maintains information about the original FITS binary table for use in remote analysis. The idea is that you create a representation file from the original FITS binary table and use that file for display in JS9, while supporting server-side analysis on the original FITS event file.

Display of a representation file is fully integrated into JS9:

Using this representation file, image data values are available to the client JavaScript code without having to download the much larger binary table. This means that the contrast/bias conversion can be performed, image positions/values displayed, etc. using standard browser capabilities and relatively straight-forward array manipulation code. Browser-based image plugins can process the image data, and JS9 can communicate with a back-end server to provide more sophisticated FITS-based analysis support.

Note that JS9 is able to distinguish between PNG representation files and ordinary PNG files and will correctly load either type.

The PNG Representation File

The PNG representation file contains the following information:

Generating the PNG file using tpos and fits2png

[NB: A maximum image size is enforced by Apple on Safari mobile devices. (See https://discussions.apple.com/thread/4975106 for a discussion.) If you want your JS9 PNG representation files to display on iPhones and iPads, they should be restricted to a size of less than approximately 1900x1900, depending on the size of the FITS header. So a PNG file of size 1024x1024 should be fine, while a file of size 2048x2048 certainly will be too large. (April 9, 2015)] A PNG representation file can be created either statically or dynamically at display time as needed. The low-level tpos program and high-level fits2png script facilitate this conversion:

    tpos [ifits] [opng]
    switches:
    -b bl       # block factor for binary tables (cfitsio only)
    -e list     # list of event extensions to look for (cfitsio only)
    -f filter   # filter for binary tables (cfitsio only)
    -s sect     # section string for binary tables (cfitsio only)
or
    fits2png [switches] [ifits] [opng]
    switches:
    -abs	# convert relative path to absolute path for storage in PNG
    -gzip       # assume the file is in gzip format, regardless of its extension
    -dim	# dimension for PNG image (use to calculate block factor)
    -js9	# set dim, odir, repl parameters for automatic JS9  conversion
    -odir	# specify output directory (default: from opng path)
    -repl	# replace existing PNG? (default: true)

The tpos program can be linked against either the cfitsio library or the funtools library, depending on how you configured your JS9 build. (Note that funtools is deprecated. Please use cfitsio where possible.)

The fits2png switches are meant to provide flexibility when running tpos in batch mode. The fits2png script also is used to generate PNG files on the fly (i.e., when offering FITS files on your web page directly instead of the PNG representation.) Note that if you have linked against cfitsio, you must ensure that the HEASOFT environment is set up properly when running fits2png.

By default, tpos and fits2png will convert a FITS image or binary table to a PNG representation file having the same image dimensions:

    tpos m13.fits m13.png
However, current browsers have a working limit to the image size they can reasonably handle: 1024x1024 is recommended, although 2048x2048 works as well; anything larger can be subject to performance degradation. For maximum flexibility, you can use section syntax to limit the size of the generated PNG file. For example:
    tpos -s "1024@4027,1024@3944" -b 2 casa.fits casa.png    # cfitsio
or:
    tpos 'casa.fits[1024@4027,1024@3944,2]' casa.png         # funtools
Here, the PNG file is generated by extracting a 1024x1024 image centered on a specific point (4027,3944) and then blocking it by a factor of two to create a final 512x512 image.

For automatic processing with fits2png, specifying the -dims switch will tell fits2png to calculate the block factor in such a way as to generate an image of the specified size:

    fits2png -dims 512 512 casa.fits casa.png
This will block the 8192x8192 Chandra ACIS data by a factor of 16 to make a 512x512 image centered on the the point (4096,4096).

In addition to the image, the (generated) FITS image header and the FITS pathname are stored in the PNG file for use by both the browser and the server-side helper programs. The image header is used to provide WCS information for individual pixels as the mouse is moved across the image. It also is used to provide region information in WCS format (i.e. independent of block factor). The FITS pathname is used to locate the original FITS file in order to execute analysis tasks.

In order to run analysis tasks, the FITS data must be accessible to the server-side helper program. Therefore, it seems preferable to store an absolute pathname in the PNG file, but more experience is needed on this score. You can specify the absolute pathname on the tpos command line, or else specify the -abs switch on fits2png and it will be determined by the script.

If you store a relative path in the PNG file, note that this path will be utilized by the server-side JS9 helper to locate the FITS file. This means that if the helper is not started in the same directory that tpos (fits2png) was run, the relative path will not be correct. In this case, you can set the JS9.analOpts.dataDir parameter in the js9Prefs.json file to a path that will be used instead of the relative path.

You also must decide where to store the PNG files, e.g. with the FITS data or in a separate directory. Clearly, this will depend on how your archive is set up and where you have write access. By default, PNG files will simply be stored in the path specified on the tpos command line. You can use the -odir switch on the fits2png script to specify a generic location:

    tpos ./png ./fits/casa.fits ./png/casa.png    # store in png subdirectory
    tpos -odir ./png ./fits/casa.fits casa.png    # same

The -odir switch can be used to specify a session-based directory in which to store PNG files. This facility could be used, for example, in a case where each user has files stored in her or her own directory, which might be removed at the end of the session. To facilitate session-based directory specification, a standard HTML cookie is used to pass the session information between client and server as follows:

Note that the name of the cookie is arbitrary, but is case sensitive.

Finally, the special value of @IFILE@ tells fits2png to store the PNG file in the same directory as the FITS file:

    tpos -odir @IFILE@ .../casa.fits casa.png     # store PNG with FITS file
Again, these switches are meant to provide initial flexibility while we gain more experience with community needs. Please let us know how to improve this part of the system.

Regardless of where the PNG files are stored, you must adhere to the W3C "same-origin" security policy, i.e. the PNG data files must be hosted on the same web server as the JS9 code. This means that your PNG files must either be stored in the web site directory tree, or you must make a link within the web site tree to the top level PNG directory. Also note that Google Chrome's implementation of this policy prevents you from using JS9 with a file:// URL. (On Linux and Macs, you can start Chrome with the --allow-file-access-from-files switch to work around this restriction.)

Loading a PNG File From Your Web Page

Once you have generated your PNG files, you can make them available on your JS9 web page by creating URLs that call the JS9.Load() routine:

    href='javascript:JS9.Load(filename, opts);'
where filename is the file to load and opts is an object containing params to be merged with the default JS9.imageOpts object.

For example:

    href='javascript:JS9.Load("png/casa.png", {scale: "log"});'
Clicking on this URL will cause the casa.png file (here, stored in the png directory) to be loaded into JS9, using the log scale as a default. If the JS9 server-side helper is running, it will be alerted about this new file so that it can run analysis, etc.

See JS9 Public API for complete documentation about the JS9.Load() routine.

Run-time Conversion of FITS to PNG

If you don't generate PNG files beforehand, you can still specify a FITS file as the first argument to JS9.Load() and have the JS9 server-side helper run fits2png at run-time.. To do this, you set the fits2png property in js9Prefs.json:

{
  "globalOpts": {"fits2png": true ...}
}
This tells JS9 to convert FITS files instead of downloading them directly. Then you can use JS9.Load() as usual:
    href='javascript:JS9.Load("fits/3c58.fits");'
The default command for this conversion is contained in the js9Helper.js file:
    fits2png -js9 -odir ./png
The -js9 switch will calculate a block factor to generate a 512x512 PNG image. It will not replace an existing PNG file, if it exists. Note that the output directory is set explicitly to be the png subdirectory of the sample installation. You can change this command to suite your site needs.
Last updated: June 1, 2016