Part I. Overview of ICARTT Data Management

Part II. Data File Formats

Part I. Overview of ICARTT Data Management - 2004

1. Introduction

The ICARTT study will involve a large number of measurement platforms that will provide a tremendous volume of data. Because there are a number of laboratories involved in the study, each with their own way of handling data, common methods of dealing with data must be identified and agreed upon prior to the study. This will facilitate data transfer both during the study and, more importantly, after the campaign is completed. Toward this end a data transfer and archiving standard has been agreed to by all of the principals in the study. This standard, modelled after the NASA Ames format, has been chosen because it is quasi-universal (i.e., most institutions are familiar with it), it satisfies most of the data handling issues that are expected to arise during this study, and it is easily handled by most computer-based data manipulation programs especially those used for merging and archiving data for public use. For reference, the full specification for the Ames file exchange format can be found at: http://cloud1.arc.nasa.gov/solve/archiv/archive.tutorial.html.

The Ames data format describes a generalized procedure by which data are placed into structured files with accompanying header information and appropriate file names. Since the ICARTT data format will vary somewhat from the Ames format, the purpose of this document is to specify in detail the exact structure for ICARTT data files, the exact information and structure of the header, and the exact file names to be used. The reasoning behind these specifications will become apparent as the details emerge below.

We first discuss the overall strategy and considerations for data management followed by a detailed description of the format requirements for data files during and after the ICARTT study.

2. Data exchange protocol

A. Data Managers

Certainly many issues and problems will arise with a study of this size and will need to be addressed. Toward this end, we recommend that each group designate a Data Manager (DM) who will be responsible for ensuring that all the measurements from that group be placed on an accessible server in the required format and within the timeframe specified below. These people should be identified well before the study so that all Data Managers can collectively address the needs and requirements for data exchange and identify and implement solutions prior to the start of the field campaign. It is the team of Data Managers, with some input from the Data Management Working Group, that will make the decisions regarding all aspects of ICARTT data exchange.

B. Data access

There will not be a central data collection and distribution server, but rather individual data servers set-up and maintained by each group and accessed either via the web or ftp. Collectively, these sites can be thought of as a data repository (see Figure 1).

We should ensure that all groups involved in the study have a server that is accessible. If not, arrangements should be made for the data from that group (or PI) to be placed on the server of one of the other groups in the study. All participants will have access to all the data products available during and after the study.

C. Data transfer during the field study

During the field study every attempt should be made to have data posted to the data repository no more than 24 hours after the measurements have been taken. For some data this will be an absolute requirement due to the needs of flight and ship track planning. These data should be identified well before the commencement of the field campaign.

During and immediately after the campaign, “field” data files will be available. Data exchanged during the field study are considered a special case since these data are typically “first look” and, due to time constraints, are not likely to have undergone the full scrutiny of the PI. In order to reflect this fact the file names will be modified slightly with respect to the convention stipulated below in that the data revision code will be a capital letter (e.g., A, B) instead of a numeric code. This will be the flag to indicate to the user that these are Field data to be used only during the field study. These files should be deleted as soon as possible after the study and replaced with preliminary data files which will have some QA/QC performed.

D. Post-mission data exchange deadlines

We introduce here the data catalog (see Figure 2). This is a fully accessible (to the public) web site that is set up and maintained by the Data Manager team. The data catalog does not contain data files, but rather contains information about what data are available, what the status is (field, preliminary, or final), and where the data can be accessed. This is the central site where an investigator can go to find out what data were taken during the study, what stage the data QA/QC process is at, and who to contact to get the data. The data catalog provides openness without compromising security. The DM team may decide to implement this data catalog prior to the study as a means of providing an overview “at a glance” of the measurements being conducted during the ICARTT study.

Preliminary data files should be submitted to the respective servers no later than 01 January 2005. These files will be required to give PIs adequate time for analysis prior to the anticipated data workshop in April, 2005. Final data should be submitted no later than 01 June 2005.

E. Data archiving

The NASA LaRC group has agreed to accept all ICARTT data for final archiving to a public database and to merge the data into an easily usable format.

F. Responsibilities of data access

A major goal of this data management plan is to facilitate the free exchange of data between and among various teams of researchers. The intention of this data sharing is to broaden the interpretation of observations and to exploit complementary data collected by different research teams. While this level of access is desirable, there are clear responsibilities that come with this access. It is appropriate and expected that researchers may browse all data unfettered; however, once earnest research is pursued, it is essential that relevant Principal Investigators will be made aware that their data are being used. It is also expected that they will be offered co-authorship and the opportunity to comment on the content of manuscripts prior to submission for publication. It is imperative that Principal Investigators be consulted when suspicious data is encountered or when interpretation of data becomes dependent upon understanding the underlying technique.

It is customary for research teams to seek publication of results in an agreed upon journal as a “special issue” or “special section”. It is also possible that more than one research team may decide to pursue a common publication schedule. It is expected that researchers will honor the publication schedules agreed upon by their respective teams. Some researchers may decide to request an exception to the publication schedule. Such exceptions can include results of extremely high interest that would have immediate impact or interesting results that are peripheral to the primary goals of ICARTT. Requests for an exception to the publication schedule should be arbitrated by the primary author’s science team.

Part II. Data File Formats

Click here to view the latest updates for the ICARTT Data File Formats.

In a study this large there are many different types of data collected. Many data sets are simply straight time series with one or a number of parameters being measured sequentially (and simultaneously) in time. However, there are some data sets that are truly multi-dimensional in that a sample will be taken by an instrument at a single point in time and a number of parameters will be measured on that sample simultaneously. An example is wind profiler data in which a 30-minute averaged sample taken at some time period will be binned into height information and at each height will be wind speed, wind direction, and temperature. Another, more extreme, example is output from 3- dimensional models. Data such as these clearly cannot be represented as a single time series. Sections 1- 2 below outline the ICARTT format for all types of data, with an emphasis on standard time-series types of data. Section 3 - 4 is specific for standard timeseries types of data. Section 5 offers guidance for non-standard time-series data.

Though adapted from the NASA Ames data format, the ICARTT data format will have no restriction on the number of characters per line or on the number of characters per record. The file name will be limited to 127 characters in length. These features (and others; see below) will require modification to some existing file reading and checking software (such as Ames). The NASA LaRC group has agreed to create web-based checking software, accessible to the ICARTT science team, to accommodate these changes.

1. Requirements for data files

A. Time information

The philosophy here is that the data in the files must possess at least the minimum amount of accompanying information to uniquely identify each data point - this generally means time and location information. Moreover, the format must be able to handle all forms of timing configurations, including data that are irregularly spaced in time. For example, there are instruments that integrate a measurement over time until a certain signal-to-noise threshold has been reached. The integration period varies according to atmospheric conditions so that the resulting data have both variable integration times and are irregularly spaced in time. There is absolutely no way to represent these data with a single time point. The most efficient way of representing these data is with two time points: starting time and stopping time. This is the first requirement for the data file structure.

In those cases when many data sets are used or merged, a convenient single time reference point is the mid-point of the sampling period(s). Generally, this is the average of the start time and the stop time, but this is not always the case. As an example, there are measurements that integrate over a certain time but because of sample airflow changes (e.g., changing altitude during aircraft sampling) the sampling volume mid point does not correspond to the sampling time mid-point. In this case, the actual time midpoint must be specified by the investigator. Thus in order to encompass all of the possible diversity in sampling, three times need to be specified for each data point: start time, mid-point time, and stop time.

There are different views on what format should be used to represent time. In current measurement practice it is typical to find 1 second sampling intervals regardless of the platform (i.e., aircraft, etc.). Measurements at 1 Hz generally capture most of the important variablity in air quality data, and, while longer intervals are commonly reported, shorter intervals are not. The Ames format shows time as seconds from the start of the day defined in the file header and in the file name (see below). The ICARTT file format will adopt this structure. However, recognizing the need in some cases for >1 Hz sampling, the ICARTT format will allow data in fractional seconds though the default will be integer seconds. This does not mean that data MUST be shown in 1 second increments; whether it be 1 minute or some other increment, this decision is left to the principal investigator. In all cases, though, all times are explicitly accounted for in the period (day) specified by the header and file name. If no data are available for any time period, then that is represented by the missing data identifier. The one exception to this is when no sampling takes place from the start of a day to some point during the day. This might occur because of, for example, aircraft take-off. All times are in UTC.

B. Location information

The specification of this information is straightforward. All data points in the files need to have latitude (lat), longitude (lon), and either altitude (for aircraft, lidars, sondes) or elevation (for surface data). The lat/lon system used here will be strictly numeric: decimal degrees (to five decimal places) with south latitudes and west longitudes represented as negative numbers (i.e., no N, E, W, S identifiers). Elevations will be in integral meters. Altitudes must be explicitly defined since many types of altitude measurements are in use (pressure alt; GPS alt; geopotential alt; etc.).

Because this information is required to uniquely identify any given data point, ideally it is included in the file with those data. However, it is sometimes advantageous to have location information consolidated and uniquely identified in a separate file (e.g., an aircraft parameter file). If this is done, then information about that parameter file must be included in the data file header information. This will be specified below.

C. Measurements

In general, each file contains data of one parameter or species separated by a space. Multiple variables per file are allowed only if all were measured on exactly the same time base, as, for example, by the same instrument (e.g., GC/MS; PILS/IC). The numeric representation of a variable will be defined by the units in which it was measured. The ICARTT format contains the NASA Ames provision for a data scaling factor. However, we recommend that all scale factors be 1 unless it is grossly inconvenient to do so. If very large or very small numbers are required, then they can be represented with exponential notation, as in 1.01e9 or 5.23e-6.

i. Uncertainties

Every data point should have a corresponding total uncertainty (or error) which has the same units as the measurement. This uncertainty in the measurement is indicated as a TOTAL uncertainty to include all systematic and random effects. Ideally, these uncertainties are tabulated as the next (and separate) column after the data column in the file. However, this requirement can be relaxed if the uncertainty data can be reproduced by information in the header of the file. For example, if all uncertainties can be calculated by a function that has any given data point as input, then the formula can be included as header information.

ii. Missing data

Missing data are just that - missing. It makes no difference what the reason, whether it be a calibration period, a system crash, instrument maintenance, etc. Missing data are represented by negative numbers large enough to never be construed as actual data. For the ICARTT file format the value is -9999. Note that this is different from the Ames data exchange format in that Ames requires missing data flags to be numbers larger than any “good” data value. This somewhat arbitrary standard breaks down for measurements in urban areas where “good” data values can exceed reasonable expectation. For example, it is not uncommon in these areas for NO, NO2, or CO data to be in the parts per million range which are very large numbers for the standard units of measure (ppbv) for these species. On the other hand, there is no conceivable situation in which large negative numbers (e.g., -9999) can be construed as “good” data. Therefore, we specify for the ICARTT format that the primary missing data flag be -9999.

On the other hand, data below (or above) the limit of detection (LOD) are not actually “missing” but do convey some information. While some investigators choose to tabulate all of their quantifiable data, including negative values, others choose not to show these data points, but rather indicate the value is less than (or greater than) some quantifiable limit. These conditions will be indicated by two additional missing data flags that are substituted for the missing data values. The flag for data values GREATER THAN some UPPER LOD (ULOD) will be –7777, and the flag for data values LESS THAN some LOWER LOD (LLOD) will be -8888. These flags (if used) and the values of the upper and lower LOD are documented at specific locations in the header file (see below).

2. Filenames

Features of different file naming conventions (including Ames) have been adapted here. File names for the ICARTT data format, limited to 127 characters or less, are defined as follows:

dataID_locationID_YYYYMMDD[hh[mm[ss]]]_R#[_L#][_V#][_comments].extension,

where the only allowed characters are: a-zA-Z0-9_.- (that is, upper case and lower case alphanumeric, underscore, period, and hyphen). All fields not in square brackets are required and are described as follows:

dataID: short description of measured parameter/species, instrument, or model (e.g., O3; RH; VOC; PTRMS; MM5)
locationID: short description of site; station; platform; laboratory or institute
YYYY: four-digit year
MM: two-digit month
DD: two-digit day
hh: optional two-digit hour
mm: optional two-digit minute
ss: optional two-digit second
R: revision number of data
L: optional launch number
V: optional volume number
comments: optional additional information
extension: file type descriptor

The underscore is used ONLY to separate the different fields of the file name; it has special significance for file-checking software. To separate characters within a field for readability, use lower and upper case letters. The use of the hyphen, though allowed, is discouraged since this character in file names may cause problems with some older operating systems and network software. The square brackets “[ ]” enclose optional parameters but are not shown in the file name. Dates and times in file names are always UTC. The date and time in the file name give the date/time at which the data within the file begin (data files), or date/time at which the image applies (image files). For aircraft and sonde data files, the date always refers to the UT date of launch.

The dataID is a short string of characters used to identify the parameters in the file. For files that contain one or two variables those variable names can be used in the file name. For files in which many variables are represented, it may be best to indicate in the file name a class of compounds (e.g., VOC; PhotolysisRates) or an abbreviation of the instrument used to make the measurements (e.g., PTRMS).

The locationID is used to identify the measurement platform, site, station, or source (laboratory or institute) of the information within a data file. Some examples could be: DC8, BAE146, RHBrown, GOME (satellite), IoS (Appledore Island site), ChebPt (Chebogue Point site), and others. It may be useful to have a standardized set of abbreviations used for the ICARTT study. These should be decided upon by the DM team.

The R parameter will not be optional in the ICARTT study. We must specify a data revision code that will track changes in data and document why those changes occurred. For this we specify a revision number counter “_R#” where the underscore is a required element to separate the fields (this is needed for certain file checking software). The revision number "#" must match the revision number specified in the Normal Comments section of the file header (see below).

The optional parameters “_L#” and “_V#” may be needed in some special cases. If the contents of the file pertain to a second or third aircraft launch on the indicated date, then a launch counter "_L#" (i.e. L2, L3, etc.) must appear after the "R" identifier but before a volume counter, if present (see below). Launch number one is implied when "_L#" is omitted from the file name. If a data file is one volume of a multi-volume dataset, then a volume counter "_V#" (i.e. V1, V2, V3, etc.), must appear after the "R" parameter (and the “L” parameter, if present) separated by an underscore from the rest of the identifier. The volume number (the "#" in "V#") must match the volume number in the file header. When "_V#" is missing from the file name a one-volume dataset is implied.

The optional comments parameter is for additional information required by the PI (or Data Manager) to identify the file contents but that does not fit into the other fields of the file name. This should be used sparingly.

The file extension is a 2-4 character parameter that identifies the file type. The principal file type for the ICARTT study will be “.ict” and describes the time series data in a file formatted to ICARTT standards. Other file types may include:

These allowable file extensions will need to be defined by the team of Data Managers.

3. Recommended File Format Specification for ICARTT Time-series Data Files

A. Structure

B. File header information

The basic structure of the ICARTT file header is similar to the Ames exchange format. For the ICARTT study we recommend some additional information that will be included in the comments sections. The most general header is shown below as an example; more specialized headers will be described as modifications to the general form. Different items of information in the same record (same line) are shown below as separated by a semicolon – in the actual file they are separated by a single space.

EXAMPLE 2 This example is similar to Example 1. Differences include the exception of the elimination of variables stop time, mid time, lat, lon, elev, and uncertainties, the inclusion of a special comment, the inclusion of DM info, and a second revision comment.
File name: NOX_RHBrown_20040830_R1.ict

36 1001
Williams, Eric
Aeronomy Laboratory/NOAA
Nitric oxide and nitrogen dioxide mixing ratios from R/V Ronald H. Brown
ICARTT_NEAQS
1 1
2004 08 30 2004 12 25
60
Start_UTC
2
1 1
-9999 -9999
NO_ppbv
NO2_ppbv
1
Lightning struck the ship at ~ 14:00:23 UTC, or at 50423 seconds after midnight UTC. The 13 minute section of missing data from 14:00 to 14:43 (50400 through 52780 of Start_UTC) reflects the period when the instrument was checked out and the computer rebooted.
19
PI_CONTACT_INFO: Address: 325 Broadway, Boulder, CO 80305; email: eric@al.noaa.gov; 303-497-3226
PLATFORM: NOAA research vessel Ronald H. Brown; sampling through high-flow manifold (res. time ~ 1 s) at 15 m above waterline
LOCATION: Ship location data in file ShipData_RHBrown_20040830_R0.ict
ASSOCIATED_DATA: ShipData_RHBrown_20040830_R0.RHB
INSTRUMENT_INFO: NO: chemiluminescence; NO2: narrow-band photolysis/chemiluminescence, See Williams et al., BigScience, 42, p. 50-51, 2001
DATA_INFO: Units are ppbv. All one-minute averages contain at least 35 seconds of data, otherwise missing. Midpoint time is 29 seconds after the minute. One second data are available, contact the PI.
UNCERTAINTY: NO: +/-(5%+0.005 ppbv); NO2: +/-(12%+0.025 ppbv)
ULOD_FLAG: -7777
ULOD_VALUE: N/A
LLOD_FLAG: -8888
LLOD_VALUE: 0.005; 0.025
DM_CONTACT_INFO: Donna Sueper; NOAA/AL; dsueper@al.noaa.gov. Data manager for data within ShipData_RHBrown_20040830_R0.ict is Jim Johnson with PMEL, James.Q.Johnson@noaa.gov
PROJECT_INFO: ICARTT study; 1 July-15 August 2004; Gulf of Maine and North Atlantic Ocean
STIPULATIONS_ON_USE: Use of these data requires PRIOR OK from the PI
OTHER_COMMENTS: N/A
REVISION: R1; R0
R1: NO2 data have been increased by 13% based on calibration standard recheck.
R0: No comments for this revision.
Start_UTC NO_ppbv NO2_ppbv
43200 0.555 2.509
43260 10.333 35.030