Installation

This documentation provides installation instructions for HARP.

HARP is a toolset for ingesting, processing and inter-comparing satellite data against correlative data. The toolset is composed of a set of command line tools and a library of analysis functions.

Supported platforms

HARP is designed to run on most Unix-based operating systems (including Linux and macOS) and Windows. The platforms that are supported include at least Linux, Windows, and macOS.

What you will need

The HARP package : You can choose between a source installation (harp-x.y.z.tar.gz) or a binary installation (harp-x.y.z-win64.msi (Windows only)). Note that x.y.z stands for the version number of the package. The source package contains the source code for the HARP components (C Library and command line tools) together with all documentation. You will need this package if you want to use HARP on a Unix-based system. For Windows you will only need this package if you want to have access to the HARP source (if, for instance, you want to extend/improve HARP). The binary package, which is only available for Windows, contains pre-built versions of everything from the source distribution (without the sources) and all documentation. If you do not have the HARP package, you can download it from GitHub:

https://github.com/stcorp/harp/releases/latest

The following items are needed if you use the HARP source distribution:
A C compiler : Most Unix platforms come with their own C compiler so this should not be a problem. For macOS you should make sure you have installed the Developer Tools.

A recent version of CODA. You will need a version of CODA that is compiled using the same HDF4 and HDF5 libraries as you will use for building HARP. CODA can be downloaded from https://github.com/stcorp/coda/releases/latest
A recent version of HDF4. You will also need the additional libraries libjpeg, zlib, and szlib depending on whether HDF4 was build against them. In order to prevent conflicts of the HDF4 library with the netCDF interface that HARP uses, make sure you build the HDF4 software from source and use the following options when configuring the package:
./configure --enable-shared --disable-fortran --disable-netcdf
Make sure that after installation the location of the libraries is in your shared library path (e.g. update your LD_LIBRARY_PATH environment variable).
A recent version of HDF5 (version 1.8.0 or higher). You will also need the additional libraries zlib, and szlib depending on whether HDF5 was build against them.

If you plan on using the Python interface you need either Python 2 (version 2.7) or Python 3 (version 3.6 or higher). You will also need the Python packages numpy (version 1.4 or higher) and cffi (version 1.5.2 or higher).

If you plan on using the R interface you need R version 3 or higher.

Using GitHub

HARP is also available as source code directly from GitHub, but this is only recommended to be used by users who want to co-develop HARP and it will only work on Unix-based systems. You will need to have additional software installed and need to perform additional steps if you want to build HARP from the GitHub git repository.

Additional software that you will need:

autoconf + automake + libtool
flex + bison

When you clone and checkout the HARP git repository make sure to also initialize the git submodules:

$ git clone --recurse-submodules https://github.com/stcorp/harp.git

This will pull in several entries in the definitions subdirectory.

Then, run the bootstrap script in the root of the source directory:

$ cd harp
$ ./bootstrap

After that you can follow the steps as usual for building the source package.

Note that the CMake build scripts that come with HARP can also be used to build HARP, but they do not allow regeneration of automatically generated files (such as the documentation), so they are not recommended for performing co-development of HARP.

Using Conda

HARP is also available as a conda package for Windows, Linux, and macOS (only 64bit and Python3). You can install HARP using conda with:

$ conda install -c conda-forge harp

The R interface of HARP is provided as a separate conda package (that has the core HARP package as dependency). You can install this using:

$ conda install -c conda-forge r-harp

Note that these installations of HARP do not include the MATLAB or IDL interfaces.

Also be aware that if you use conda within an Anaconda installation to first create a separate environment using e.g. conda create -n harp and conda activate harp, otherwise the conda installation will likely fail. The recommended installation of conda is Miniforge or Mambaforge as this will ensure that all packages are taken from the conda-forge channel by default.

Building the source package (Unix)

To build the source package, make sure to download the official HARP source package, which is named harp-x.y.z.tar.gz. Don’t use the packages called ‘Source code’ from GitHub as these are contents of the git repository (see ‘Using GitHub’ above).

The HARP source package comes with both an autotools and a CMake build system. For Unix-based systems the recommended approach is to use the autotools based system (using the configure script), which is what is described below. Building with CMake is also supported, but not documented here.

The following steps will guide you through the process of building the HARP library and executables:

Go to the directory that contains the downloaded harp-x.y.z.tar.gz file and unpack this package:
$ gzip -d harp-x.y.z.tar.gz
$ tar xf harp-x.y.z.tar
Make sure you replace x.y.z with the appropriate version number.
You should now have a harp-x.y.z directory. Go to this directory:
$ cd harp-x.y.z
Next you will have to execute the configure script that checks what system you are on and sets up the build environment. There are several options you can pass to this script. The most important ones are:

--prefix <HARP installation directory> :
By default, if you perform an installation of the HARP package (see further below on how to do this) all files are installed in subdirectories under /usr/local. Executables will appear under /usr/local/bin, libraries under /usr/local/lib and all data files (e.g. documentation) under /usr/local/share/harp. However, installing the files into the default places requires you to have administrator privileges, which you may not have. In order to install HARP in a different location where you do have permission to copy files to, you can use this option to provide a different installation directory. For instance, you can use --prefix=$HOME/harp to install the software in the subdirectory harp of your home directory.

CODA_LIB=<CODA library directory> :
The configure script will automatically try to locate the CODA library in the default locations for libraries (/usr/local/lib is usually not considered a default location!). If ./configure complains that it can’t find the coda library files, pass this option to ./configure with the location of this library file.

CODA_INCLUDE=<CODA include file directory> :
The configure script will automatically try to locate the CODA include files in the default locations for include files (/usr/local/include is usually not considered a default location!). If ./configure complains that it can’t find the coda.h include file, pass this option to ./configure with the location of this include file.

HDF4_LIB=<HDF4 library directory> :
The configure script will automatically try to locate the HDF4 libraries in the default locations for libraries (/usr/local/lib is usually not considered a default location!). If ./configure complains that it can’t find the df, hdf, or mfhdf library files, pass this option to ./configure with the location of these library files.

HDF4_INCLUDE=<HDF4 include file directory> :
The configure script will automatically try to locate the HDF4 include files in the default locations for include files (/usr/local/include is usually not considered a default location!). If ./configure complains that it can’t find the hdf.h or mfhdf.h include files, pass this option to ./configure with the location of these include files.

HDF5_LIB=<HDF5 library directory> :
The configure script will automatically try to locate the HDF5 library in the default locations for libraries (/usr/local/lib is usually not considered a default location!). If ./configure complains that it can’t find the hdf5 library files, pass this option to ./configure with the location of this library file.

HDF5_INCLUDE=<HDF5 include file directory> :
The configure script will automatically try to locate the HDF5 include files in the default locations for include files (/usr/local/include is usually not considered a default location!). If ./configure complains that it can’t find the hdf5.h include file, pass this option to ./configure with the location of this include file.

--enable-python :
By default HARP is built without the Python interface. Use this option to enable building of the interface to Python. Make sure that you choose the install prefix option (--prefix) such that the target location of the python package (e.g. <prefix>/lib/python3.7/site-packages) ends up in your python path. Also, if you enable the Python interface then make sure you have installed the numpy and cffi packages for Python.

PYTHON=<Python executable> :
If you want to build the Python interface you should also use this option to tell the configuration script where your Python executable is located (e.g. PYTHON=/usr/bin/python). Make sure that you provide an absolute path for the Python executable (i.e. the path should start with a /).

--enable-R :
By default HARP is built without the R interface. Use this option to enable building of the interface to R. Make sure that you choose the install prefix option (--prefix) such that the target location of the R package (e.g. <prefix>/lib/R/site-library) ends up in your R path.

R=<R executable> :
If you want to build the R interface you might have to use this option to tell the configuration script where your R executable is located (e.g. R=/usr/bin/R). Make sure that you provide an absolute path for the R executable (i.e. the path should start with a /).

--enable-idl :
By default HARP is built without the IDL interface. Use this option to enable building of the interface to IDL.

IDL=<IDL root directory> :
If you want to build the IDL interface you should also use this option to tell the configuration script where you have installed IDL. The <IDL root directory> is the root directory of your IDL installation. It should contain for instance the bin directory with the idl executable and an external directory containing the file export.h. Also make sure that you provide an absolute path for the IDL root directory (i.e. starting with a /).

--enable-matlab :
By default HARP is built without the MATLAB interface. Use this option to enable building of the interface to MATLAB.

MATLAB=<MATLAB root directory> :
If you want to build the MATLAB interface you should also use this option to tell the configuration script where you have installed MATLAB. The <MATLAB root directory> is the root directory of your MATLAB installation. It should contain for instance the bin directory with the matlab and mex executables (or symbolic links to them) and an extern/include subdirectory containing the file mex.h. Also make sure that you provide an absolute path for the MATLAB root directory (i.e. starting with a /).

You should now call the configure script with the options that are appropriate for you. For instance, if you want to install HARP in the harp subdirectory of your home directory then your invocation of configure would be:
$ ./configure --prefix=$HOME/harp
If this completes successfully, you are now able to build the library by executing the make command:
$ make
If everything was successful, you are now ready to install the package (this installation step is not optional! You will have to install HARP before you can use it). If you encountered any problems while building HARP and if you need help contact us (see the Feedback section at the bottom for contact information).
In order to make use of the HARP library and tools, you should install HARP as build in the previous step. Please make sure you have provided the appropriate installation directory option (--prefix=<installdir>) to the configure script, as explained in the previous section, if you do not want to install HARP in its default location /usr/local. After running the configure script, issue the following command:
$ make install
If you enabled the IDL interface then you will have to add <HARP installdir>/lib/harp/idl to your DLM_PATH. You do this by setting an IDL_DLM_PATH environment variable and add this setting to your system shell startup script (if you don’t now how to set environment variables or add these to your shell startup script, please ask your system administrator). The variable should be set to <IDL_DEFAULT> appended with the HARP DLM directory. If, for instance, you have installed HARP in /usr/local then you should set the IDL_DLM_PATH environment variable to <IDL_DEFAULT>:/usr/local/lib/harp/idl.

Note that you should ideally not use the IDL pref_set function to set the IDL_DLM_PATH. The HARP DLM file will still load, but HARP will not be able to find the coda format definition files or udunits2 xml files this way. You can work around this by setting the coda definition path and udunits2 xml path explicitly instead of having HARP/CODA automatically determine this location based on the IDL_DLM_PATH. See points 8 and 9 below for how to set the environment variables globally. You can also set the environment variables inside IDL before you call any HARP or CODA functions using e.g.:
IDL> SETENV, 'CODA_DEFINITION=/path/to/codadefs/'
IDL> SETENV, 'UDUNITS2_XML_PATH=/path/to/udunits2.xml'
If you enabled the MATLAB interface then you should create a startup.m file in a matlab directory in your home directory (~/matlab/startup.m) that contains the line:
addpath <HARP installdir>/lib/harp/matlab
Make sure that the CODA definition path is correctly set. If you followed the instructions normally, then HARP and CODA should be able to find the coda definition files (.codadef files) inside the installation path automatically. If you are getting ‘unsupported product’ errors then you may have to set the coda definition path explicitly. You can do this by setting the CODA_DEFINITION environment variable and have it point to the directory that contains the .codadef files. On Linux this would be e.g.:
export CODA_DEFINITION=/usr/local/share/coda/definitions
On macOS you can set environment variables via a ~/Library/LaunchAgents/my.startup.plist file.

Also make sure that you do not already have a CODA_DEFINITION environment variable set that points to a different location than where the .codadef files that come with HARP are located.
Make sure that the udunits2 xml path is correctly set. If you followed the instructions normally, then HARP should be able to find the udunits2 xml files inside the installation path automatically. If you are getting ‘invalid unit’ errors then you may have to set the udunits2 xml path explicitly. You can do this by setting the UDUNITS2_XML_PATH environment variable and have it point to the udunits2.xml file inside the installation path. On Linux this would be e.g.:
export UDUNITS2_XML_PATH=/usr/local/share/harp/udunits2.xml
On macOS you can set environment variables via a ~/Library/LaunchAgents/my.startup.plist file.

Installing the binary package (Windows)

First make sure that you have the Microsoft Visual C++ Redistributable Packages for Visual Studio 2015 installed.

To install the binary package of HARP for Windows just run the msi executable which will guide you through the installation process. Make sure to install HARP in a directory that does not have any spaces in its path. Otherwise you might encounter problems with e.g. the Python interface.

For Python you will have to copy the C:\Program Files\HARP\python\harp directory to the site-packages folder of your Python installation. However, if you want to use the Python interfaces on Windows, it is recommended to just install the Anaconda package of HARP.

For R you will either have to copy the C:\Program Files\HARP\R\harp directory to the site-library folder of your R installation or you will have to add the C:\Program Files\HARP\R directory to your R library path. However, if you want to use the R interfaces on Windows, it is recommended to install the Anaconda package of HARP.

For IDL you will have to add <HARP installdir>\lib\harp\idl to your DLM_PATH. You do this by setting an IDL_DLM_PATH environment variable. On Windows open the ‘System’ control panel of your Windows operating system and go to the ‘Advanced’ tab. Then click on ‘Environment Variables’ and create a new system variable with the name IDL_DLM_PATH and value <IDL_DEFAULT>;C:\Program Files\HARP\lib\harp\idl. If you have installed HARP in a location different from C:\Program Files\HARP then replace this part in the value by the installation directory you chose when installing HARP. Note that you should ideally not use the IDL pref_set function to set the IDL_DLM_PATH. The HARP DLM file will still load, but HARP will not be able to find the coda format definition files or udunits2 xml files this way. You can work around this by setting the coda definition path and udunits2 xml path explicitly instead of having HARP/CODA automatically determine this location based on the IDL_DLM_PATH. This is done by setting the CODA_DEFINITION and UDUNITS2_XML_PATH environment variables. This can be done outside IDL by setting the environment variable globally, but you can also do this inside IDL before you call any HARP or CODA functions using:

IDL> SETENV, 'CODA_DEFINITION=/path/to/codadefs/'
IDL> SETENV, 'UDUNITS2_XML_PATH=/path/to/udunits2.xml'

For MATLAB you will have to start MATLAB and go to the ‘Set Path…’ menu item in the ‘File’ menu. Here you should add the directory C:\Program Files\HARP\lib\harp\matlab. If you have installed HARP in a location different from C:\Program Files\HARP then replace this part of the directory by the installation directory you had chosen when you installed HARP.

Note: The binary installer for Windows comes with DLLs for all dependencies, which are CODA, HDF4, HDF5, and their dependencies.

Building the source package (Windows)

To build the source package, make sure to download the official HARP source package, which is named harp-x.y.z.tar.gz. Don’t use the packages called ‘Source code’ from GitHub as these are contents of the git repository (see ‘Using GitHub’ above).

The official and only supported build system on Windows is CMake. You will need to have builds of CODA, HDF4, HDF5 (and their dependencies) available as well in order to build HARP. The locations of include files and libraries for the third-party dependencies can be set using <package>_INCLUDE_DIR and <package>_LIBRARY_DIR CMake options.

The generation of the Windows binary installer is done using CPack and WIX. So, in order to recreate the Windows binary installer, you will also have to make sure that you have CMake (3.0 or later) and WIX installed.

Documentation location

Both the source and binary HARP packages come with documentation in HTML. For the source package you can access the documentation from within the unpacked HARP source package directory by going to the doc/html subdirectory and opening the file index.html in your browser. If you perform an install of the source package all documentation will also be installed. You can find the documentation under the subdirectory share/harp/doc/html of your installation directory (/usr/local by default).

Feedback

If you encounter any problems while trying to build, install or run one or more components of the HARP package then create a topic on the Atmospheric Toolbox Forum:

https://forum.atmospherictoolbox.org/

If you are using the source package on a Unix based system, please provide a copy of the config.log file that is created when you run ./configure and a copy of the output of make with your e-mail.

Apart from problems, we would also appreciate to hear from you if you have any ideas, suggestions, or comments that may help us to improve the quality or functionality of HARP.