Atmospheric Sciences Data Center processing, archiving, and distributing Earth science data at NASA Langley Research Center RADIATION BUDGET • CLOUDS • AEROSOLS • TROPOSPHERIC CHEMISTRY

What is HDF?

Hierarchical Data Format (HDF) is a data file format designed by the National Center for Supercomputing Applications (NCSA) to assist users in the storage and manipulation of scientific data across diverse operating systems and machines. NCSA developed a library of callable routines and a set of utility programs and tools for creating and using HDF files.

HDF supports a variety of data types: scientific data arrays, tables, and text annotations, as well as several types of raster images and their associated color palettes.

There are two distinct varieties of HDF, known as HDF (version 4 and earlier) and the newer HDF5. These are described in more detail at the NCSA HDF web site.

Some of the features of HDF are:

• HDF makes it possible for programs to obtain information about the data from the data file itself, rather than from another source.
• HDF standardizes the format and descriptions of many types of commonly used data sets, such as raster images and scientific data.
• HDF is a platform independent file format. It can be used on many different computers, regardless of the operating system that machine is running.
• New data models may be added to HDF by either the development team or HDF users.

HDF files are also self-describing. For each data object in an HDF file, there are predefined tags that identify such information as the type of data, the amount of data, its dimensions, and its location in the file. The self-describing capability of HDF files has important implications for processing scientific data. It makes it possible to fully understand the structure and contents of a file just from the information stored in the file itself. A program that has been written to interpret certain tag types can scan a file containing those tag types and process the corresponding data. Self-description also means that many types of data can be bundled in an HDF file. For example, it is possible to accommodate symbolic, numerical, and graphical data in one HDF file.

For more information on what HDF can do, visit the HDF FAQ.

Or, view the slide presentation An Introduction to HDF.

Command Line Utilities

HDF command line utilities are applications that allow you to perform common operations on and manipulations of HDF files. These utilities can be executed at the command level, like UNIX commands. The following are four examples of useful HDF command line utilities:

hdfls

Allows you to list basic information about an HDF file, such as, tags and reference numbers, and the length of each data element.

hdfed

Allows you to examine the data file itself and to move data between two HDF files.

r8tohdf

One of three that allows you to manipulate HDF files that contain raster image sets. This utility converts one or more raw images to HDF 8-bit raster image set format and writes them to a file.

fptohdf

Converts floating-point arrays (from either text files or HDF scientific data sets) to either HDF 8-bit raster image sets (RIS8) or HDF scientific data sets (SDS), or both, and stores the results in an HDF file. The image can be scaled about a mean value that is provided by you.

A comprehensive list of commands is available at the NCSA HDF Utilities page.

Public Domain and Commerical HDF Tools

Tools to help you view and manipulate HDF data.

Obtaining HDF Libraries and Utilities

HDF source code

The HDF libraries and utility programs are available free of charge from NCSA via anonymous ftp from the sites listed below. Pre-compiled binary versions are available for a variety of computers and operating systems, or source code can be downloaded and compiled to build the software for a specific system.

NCSA FTP server.

This site always contains the latest official release of HDF. Source code can be found in the directories named tar, unpacked, or zip. Pre-compiled binaries are located in the directories whose names include precompiled_code or in the bin directory. Be sure to read the README file located in each directory for additional information.

HDF FTP server.

This is an HDF mirror site for NCSA.

Downloading and Installing the HDF libraries

Example of how to download and install the HDF Libraries.

Related Sites

HDF-EOS document list. This is a list of documents containing information about HDF-EOS, extensions to HDF that include three geolocated data structures defined as the standard for Earth Observing System (EOS) data products.

HDF-EOS Tools and Information Center.

NCSA HDF Home Page. Includes the latest news on HDF.

HDF Info - a commercial site with many links to HDF information. Includes utilities, a free HDF browser, and links to other sites that use HDF and similar data formats.