A short history of CAMA

This is a short overview of the history of CAMA. CAMA started out as a tool to figure out the properties of the OMCLDO2 cloud product. Some of the assumptions about the HDF-5 file format structure originate from this period. The original version was pretty specific for the OMCLDO2 product, although the field names were separated from the rest of the actual code right from the start.

Once the rest of the validation team saw what I had, they were quick to request a generic version of the software, which eventually lead to the software that is available now.

Version history of public releases

In reverse chronological order, these are the release notes for publicly released versions of CAMA. This will allow you to identify the version in which certain features were added, or specific issues were fixed. The first item is the current release. The date is the creation date of the zip archive, and can be found in the filename.

Oktober 28, 2010

Trim strings that are read from the configuration file. This makes the use of aliases more robust (‘OMDOAO3 ’ will not work, ‘OMDOAO3’ does).

June 3, 2010

Changed verification_generator_sza_filter to function as intented in filter mode.

May 31, 2010

Changed verification_generator_sza_filter, to accept a 'filter' option. It will now apply a threshold filter suitable for multiplication if an optiona ‘filter’ keyword is added to the parameter list.

March 23, 2010

Changed the name and location of the file which indicates the presence of the row anomaly for a particular row. The current file can be downloaded here. The file is downloaded automatically by CAMA if requested (and running with a sufficiently recent IDL).

February 23, 2010

Added a filter parameter to the index_value filter function option. Previously this filter function took no parameters. Now you can use 'filter,low,high' as a function parameter. This will change the behaviour of the index_value filter function. Normally it generates a filed with time-index values, with the filter parameter it creates a 1.0 or NaN value field, depending on the low and high values. This makes the function suitable for per-field processing (multiply the lot...).

February 22, 2010

Documentation update

August 28, 2009

The name of the instrument is now a configuration option, at least in generating the graphs. You can now set this to ‘GOME-2’, and have that name appear in the graphs.

March 10, 2009

Changed instrument configuration filter code, so that the Ozone hole special from 2004 is passed through. Changed geolocation plotting code.

March 3, 2009

Bugfix in the row anomaly code.

February 17, 2009

To exclude rows affected by the row anomaly, it is now possible to download a file from www.knmi.nl to indicate which rows are affected, and at which orbit this starts. This is off by default. See the manual for details, look for apply_bad_row_filter in the manual.

January 28, 2009

Fixed bug where hdf-5 file would not be created if the geolocation mapping wasn't performed.

January 26, 2009

Along track averages are now displayed using a zero-based index.

April 14, 2008

Several changes to be able to analyze data with a swath width of 1.

March 25, 2008

Analysing a product with new dimensions always uncovers new bugs. In this case comparing two products with 60 or 30 across track pixels uncovered a problem with the location where the edge cutoff was performed. After changing the location this problem was fixed. A side effect is potentially a bogus number of pixels in the report. You were not using those anyway, right?

February 15, 2008

Fixed an issue with the feature introduced on January 18, reported by Ed Celarier. The text of the labels was white on white. Needless to say that this was not the intended effect.

February 1st, 2008

Fixed an old bug in the synchronise_nans routine, or rather the detection of missing orbits. Missing orbits are now properly skipped.

January 28, 2008

Clarified an error message

January 22, 2008

Packaging error

Added the files implementing the changes from yesterday to the archive. Sorry for the inconvenience this may have caused.

Bug fix

Thanks to Remco for reporting: dirlist could not be an array (i.e. contain multiple directories). Now it can do that.

January 21, 2008

Added a filename cache, which should speed up operations when a large number of files are present in the directory tree.

January 18, 2008

Added an option to label the geolocation plots themselves, rather than relying on the caption in the final report.

December 3, 2007

Filter functions bitwise_and and sunglint_filter were added. The first allows to filter on a mak, the second corrects the OMI sunglint flag. The OMI sunglint flag only checks the angles, not whether a pixel is over land or sea. This filter adds that second check

November 28, 2007

color table selection

Spurious spaces in the color table could lead to incorrectly selected color tables. A bit tougher sanity check has been added.

November 19, 2007

Sun glint filter

Previously the sun-glint filter function relied on bit 4 of the ground pixel quality flags. This bit was set soley using the values of the solar- and viewing angles, regardless of the surface type. This meant that sun-glint was possible over the Sahara. This filter only filters sun-glint when not over land. Thanks to Remco.

Absorbing aerosol optical thickness

A new filter function is added, aaot. This function calculates the absorbing optical thickness from an (extinction) optical thickness datafield and a single-scattering albedo datafield. The AOT field goes in the third column of the parameter specification in a CAMA configuration file, the SSA field in the fourth, like this.

DATA:OMAERO:ColumnAmountAerosol:AerosolOpticalThicknessMW:SingleScatteringAlbedoMW:AAOT_MW:AAOT:13::0.0,0.4:Aerosol Absorption Optical Thickness at 483.5 nm:25::

The wavelength at which the AAOT is returned, is specified by means of the extra parameter (which is compulsory in this case; “13” in the above example). Thanks to Remco.

November 16, 2007

Added dumpstruct2hdf5.pro.

November 14, 2007

Added a few filter functions

A few filter functions were added to filter on the InstrumentConfigurationID (to skip zoom orbits), and on other criteria.

Added option to skip the scatter density plots

In some cases it may be desirable to skip the scatter density plots. This option is added: set ‘;# skip_scatter_plots = yes’ to skip the scatter density plots in the configuration file.

October 4, 2007

Maintenance update

Bug fix in the latitude detection for the equal area histograms.

October 3, 2007

Maintenance update

Miscelaneous bug fixes.

October 1, 2007

Update in the output

The averages, standard deviations, minima, and maxima for all parameters are now exported to the HDF output file. Previously these values were only exported for each across track position independently, but not the values displayed in table 2. Thanks to Remco Braak for adding this feature.

Added check for multiple copies in tree

When comparing versions of the same product using aliases, it is imperative that the different versions are well separated. We now add an extra warning if this possibility is detected.

Pixel count overview

The number of pixels included in each geolocation pixel is stored and plotted (when synchronized) as well.

Re-creation of the tex and configuration files

The tex and configuration files are now stored in the hdf-5 file, so the complete output can be restored from the hdf-5 output file.

September 20, 2007

Documentation update

The URL for TeXtoIDL has changed, I've included the current one. Thanks to MariLiza Koukouli for reporting the issue. Update: TeXtoIDL is still at the old location, the url is in all lowercase though. Documentation and the install page updated again.

September 4, 2007

Fixed bug when reading fields with a single value per image.

When reading a field with a single value per image (i.e. InstrumentConfigurationId), the side filter would remove all values except for the first 60. This has been fixed. A filter function one_value_per_image has been added to produce a value for all cross track positions.

Fixed bug in the minumum determination.

Remco brought some strange behaviour of the minumum to my attention, and even better figured out how to fix it. The IDL < and > operators and NaN values do not behave in a pleasant way. I didn't know that NaN values were ordered.

August 24, 2007

When scene hunting, the output file is now created in the standard output directory

If the scene_hunting parameters is given as a plain filename, the file is created in the output directory of the run, otherwise the path is used as absolute or relative to the location where you started cama.

August 21, 2007

Added dirlist override command-line option

An option was added to cama.pro and cama.sh to override the standard dirlist parameter (not aliasses). This is useful for running in sequence on different datasets.

July 20, 2007

Clarified error message

Clarified the error message you get when the rato/diff fields are in the wrong order

July 4, 2007

Added a new color table

Added a new color table, suitable for highlighting symmetric differences (red positive, blue negative, green near zero) with the name posneg_color_table.

Added a new color table

Added a new color table, suitable for highlighting symmetric differences (red positive, blue negative, white near zero) with the name trend_color_table.

June 25, 2007

Update to resolve possible earlier web-site issues

Due to permission issues on the server some earlier updates may not have come through. This is an update to clean things up.

May 14, 2007

Updated documentation

Google earth creation can now use orbit ranges, to facilitate the plotting of zoom data. The documentation is now updated to tell you how to actually use this option.

May 11, 2007

Google earth creation can now use orbit ranges

Google earth creation can now use orbit ranges, to facilitate the plotting of zoom data.

May 10, 2007

Added special_color_table_no2.pro

Added a different special color table, a subset of the Ozone standard color table, for use with NO2 data.

Corrected error in the color table selection code

The change listed above triggered an error in the color table selection code.

May 4, 2007

Added read_hdfeos5_data_or_geo_field_and_filter.pro

The function read_hdfeos5_data_or_geo_field_and_filter is now in the distribution. It isn't used in CAMA, but it can serve as a convernience function when actually reading OMI L2 data.

May 3, 2007

Included Google Earth single OMI day plotting

While there are some remaining issues in the Google Earth single day plotting tool, I think it is fit for wider consumption.

April 10, 2007

Bug fix

Fixed a bug in the scene hunting code where an error would occur if no scenes were found at all.

March 27, 2007

(Temporarily) removed Google Earth single OMI day plotting

The tool to plot a single day of OMI data has been temporarily removed; I'm not satisfied with the quality and stability.

March 20, 2007

Packaging bug

All the right items are now in the package.

March 16, 2007

Bug fix

A bug in the kml module (dealing with missing parameters) was fixed. Non short-circuited logic in IDL is no fun.

March 13, 2007

Dedicated Google Earth module added

A dedicated module to produce high-res Google Earth output has been added. A sample is included in the documentation.

March 7, 2007

New feature

Google Earth output

A file for Google Earth is generated to show the geolocated averages

February 20, 2007

Bug fix in record_interesting_locations.pro

The cnt variable was used incorrectly in this function.

Change in the shell script

The shell script now tells you how to obtain the results when using the -mail=address@somewhere.com notification option.

February 9, 2007

Bug fix suggested by Ed Celarier

There was a bug in the error reporting code when using a malformed configuragion file (which is probably why I never found it).

Bug fix in cama.sh

Mail works better when you include a destination address.

February 2, 2007

Bug fix in the cama shell script

There was a small typo in the cama shell script.

February 1, 2007

Change in the ‘Scene hunting output’

The HDF file that stores the output for the scene hunting effort is now restructured. Instead of one HDF group per orbit, with each group containing a small set of arrays, the scene hunting output now contains just the arrays, with all previous arrays concatenated. This is much easier to read back in as it doesn't require a manual loop. The orbit number is stored in an appropriately named array.

January 15, 2007

Change in behavior of the filename information extraction

Previously, when an OMI file name did not contain all parts, no information of orbit numbers etc. would be returned. Now as much information as is available is returned. This code is not part of the core CAMA, but it is part of the distributed code anyway.

January 8, 2007

Bugfix in the location recorder

When recording all interesting locations, a bug was encountered when an orbit did not contain any interesting data. This bug is fixed in this update.

November 16, 2006

Update to the cama.sh script

We have been requested to be a little more social on the TLCF cluster, so I've added a ‘nice’ option to cama.sh. By default idl now runs at nice level 5, but using the -nice[=LEVEL] argument, this can be changed.

November 13, 2006

Improvements in the error messages

I've enhanced some error messages to help users find errors in the configuration file more easily.

November 8, 2006

Update to the cama.sh script to reflect the new default IDL in the TLCF

The default IDL has been upgraded to IDL version 6.3 (see below why this is important). The cama.sh script is updated to reflect the new location, it previously used the test location for IDL 6.3.

November 6, 2006

Ensure that local names are valid (reprise)

The local name must be a valid IDL variable name. Of course IDL has a function to ensure this, and we now use that function. Of course this function is not string-array aware. Fixed issue with that.

November 1, 2006

Ensure that local names are valid

The local name must be a valid IDL variable name. Of course IDL has a function to ensure this, and we now use that function

October 24, 2006

IDL version in the cama.sh script

The cama.sh shell script can now use a user specified idl. As of this moment, the shell script tries to use /usr/local/lib/idl_6.3/bin/idl, and if that doesn't work, it uses $(which idl) to find an idl in the current path. This selection can be overridden by using the -idl=IDL parameter to the cama script.

The current default is a test version on the TLCF, but it resolves the issue we have encountered below.

October 23, 2006

Minor enhancement

Force the graphs to start on a page without additional text. This makes it easier to flip from the first graph to another graph to visually inspect them for differences.

October 17, 2006

Bugfix

Fixed the x-axis title on the equal earth area histograms.

October 13, 2006

Graph generation now depends on the hdf-5 output file

Graph generation now depends on the hdf-5 output file. This means that the hdf-5 output file is no longer optional, generating the graphs is (see documentation of how to do this). Another consequence of this change is that the minimum required version of IDL to run CAMA is 6.2, but because of some bugs (see below), it is highly recommended to upgrade to version 6.3. No changes are needed in the configuration file. Some sensible default names are used if a required parameter is absent.

Additional routine

Because of the separation of the analysis and the visualization stages in CAMA, it is possible to re-create the graphs using the HDF-5 output file. A routine is provided to do just that (post_cama). This routine requires no parameters, all information it needs is gathered through interactive dialogs.

HDF-5 compression

The hdf-5 write routine used in CAMA now has compression turned on, resulting in much smaller files.

The report template

A change in the template for the report has hyperlinks turned on, allowing you to quickly jump to the graphs of interest using the list of figures in the back of a report.

September 25, 2006

The hdf-5 file with all the histogram data can now be stored outside the figures directory. If the proposed file name contains a directory separator, it is assumed to point to a valid location, and the file is created there instead of inside the normal output directory.

Graph generation is skipped altogether if the last character of the proposed hdf-5 output file is a ‘*’. This star will be removed from the actual output name.

A filter was added to allow for easy filtering on the values of the flags, for instance to process the ozone column only if the cloud fraction falls between 0.0 and 0.15, without needing synchronization.

September 5, 2006

Updated documentation, added IDL .sav file for users without an IDL license. You'll have to download the IDL virtual machine yourself. Note that this is only offered as a convenience for those without a full IDL license, and is absolutely unsupported (and not very thoroughly tested either), use of the >full version is recommended. Only the filters listed in the manual can be used, and all configuration information has to be set in the configuration file. You must be able to access widgets, as a file selection panel is used to identify the configuration file to cama.

August 25, 2006 (release of 12:08)

Bug fixes

Add the work around a signed byte issue in IDL to the download archive

IDL doesn’t know about signed bytes. This shouldn’t be a problem, but the field selective methods used in CAMA have a bug in them, that cause the fill values (typically -127 for these fields) to be mapped to 0. They should be mapped to 129 (unsigned), because that is the same bit-image, but they aren’t. For byte fields, the data is now read with h5_parse, which doesn’t suffer from this issue. It is much slower, so this is done only for byte fields.

I forgot to add the actual work around to the archive. Some more investigation revealed that this bug always occurs with IDL before 6.3. Testing on IDL 6.3 has revlealed that the problem is absent there, please update your version of IDL.

CAMA now runs on IDL before 6.3, but issues a warning if it encounters a byte field. On linux it will use the work around, on other systems it will throw an error.

August 25, 2006

Bug fixes

Add the work around a signed byte issue in IDL to the download archive

IDL doesn’t know about signed bytes. This shouldn’t be a problem, but the field selective methods used in CAMA have a bug in them, that cause the fill values (typically -127 for these fields) to be mapped to 0. They should be mapped to 129 (unsigned), because that is the same bit-image, but they aren’t. For byte fields, the data is now read with h5_parse, which doesn’t suffer from this issue. It is much slower, so this is done only for byte fields.

I forgot to add the actual work around to the archive. Some more investigation revealed that this bug always occurs with IDL/Windows 6.1.1. Testing on IDL 6.3/Windows has revlealed that the problem is absent there, please update your version of IDL if you experience this issue on Windows as the work around doesn’t even work there. The issue is still unresolved, and totally incomprehensible to me on IDL/Linux 6.3.

August 23, 2006

Bug fixes

Work around a signed byte issue in IDL

IDL doesn’t know about signed bytes. This shouldn’t be a problem, but the field selective methods used in CAMA have a bug in them, that cause the fill values (typically -127 for these fields) to be mapped to 0. They should be mapped to 129 (unsigned), because that is the same bit-image, but they aren’t. For byte fields, the data is now read with h5_parse, which doesn’t suffer from this issue. It is much slower, so this is done only for byte fields.

August 15, 2006

Bug fixes

Concurrency issue fixed

When running multiple instances of CAMA in parallel, there was a possibility to overwrite graphs, or get missing graphs with a ‘stat’ error. This has been fixed.

August 11, 2006

Bug fixes

Covariances

Instead of the average over all cross track positions, the average for each individual cross track position was used in calculating the covariance matrix. This has been fixed.

Improvements

Linear regression coefficients stored for further processing

The linear regression coefficients are added to the HDF ouput file for further processing.

August 8, 2006

Improvements

Pixel counts

The pixel count statistics are now improved and are more realistic (it is still a hack). The hidden fields are now included to see the effect of the land/sea/ice filter (for example). The statistics are still gathered before synchronisation of the fields. A line about the synchronised fields is added. An attempt is made to exclude night-side measurements (θ₀ > 88 degrees is excluded).

Scene hunting can record data fields

When hunting for specific scenes, it is now possible to record OMI data values in the generated HDF file. The documentation contains details on how to construct the configuration file for this.

Bug fixes

Missing widgets

A default value was set to ‘false’, where it should have been set to ‘true’. This caused some trouble when running with the progress feedback widget.

August 7, 2006

Bug fixes

Searching for the report template

The report is located at a different position in the tree with the CVS release we use internally, and the tree we distribute. This difference is now taken into account.

July 20, 2006

New features

Linear regression coefficients

The linear regression coefficients are calculated and added to the report. Note that these coefficients assume a constant error in the ‘y’ values, and no error on the ‘x’ values. This causes the matrix of the slopes to be assymmetric. If the correlation coefficient is 1, then the the slope γ of item (i,j) is equal to 1/γ of item (j,i). In practise this is rarely the case.

Scene searching

Record all scene locations that match all criteria given in the configuration file, for example to find all scenes over a week where there are very high clouds in the mid-latitudes, with a very low cloud fraction, to hunt for cirrus in our measurements.

Separate data products with the same short name

It is now possible to compare two data-sets that use the same short name. Please read the manual for details on how to configure this.

Bug fixes:

Correct counting of orbits

Previously the number of processed orbits was calculated by considering the range of the orbits requested. Now we actually count the number of orbits available to CAMA.

June 21, 2006

First public release outside of the KNMI.

Contact information

The main author of CAMA is .