Brief Description of Real-time ARGO Data Processing at AOML
Updated: May 13, 2004

Data Collection Process: Retrieves data automatically from an ARGOS computer to a PRV file via telnet.

---

ARGOS process: Converts raw ARGOS data from profiling floats to physical units.

Unix Shell Scripts

argos_process.sh - controls the execution of the following sub process:

• prv2hex.sh - executes the prv2hex C-Program to sort the PRV data by float. Generates mail report.
• hex2cnt.sh - executes the hex2cnt C-Program to generate counts files from the hexadecimal files. Generates mail report.
• cnt2phy.sh - executes the cnt2phy C-Program to generate physical units files from counts files. Generates mail report.
• phy_complete.sh - generates data record completion report.
• send_email.sh - sends mail report.

C-Programs

• prv2hex - Sorts hexadecimal and position data in ARGOS PRV files by float (1 program).
• hex2cnt - Converts hexadecimal file to counts file (1 program per float type).
• cnt2phy - Convert float count file to physical units file (1 program per float type).

---

ORBCOMM process: Prepares data from profiling floats for automatic QC.

Unix Shell Scripts

orbcomm_process.sh - controls the execution of the following sub process:

• nc2phy.sh - executes the nc2phy jave-Program to create phy files. Generates mail report.

java Program

• nc2phy- creates phy files from netcdf files (1 program).

---

Automatic Quality Control: Performs position, temperature and salinity quality control (QC) for physical units files.
A more complete desctiption can be found by following this link: Argo real time QC tests

Unix Shell Scripts

qc_process.sh - controls the execution of the following sub-process:

• qc_sensor.sh -executes Fortran program for temperature or salinity according to input sensor type.
• report_qctest_results.sh - generates report from the quality control test results
• report_metadata_stderror.sh -generates report for missing meta data entities and/or standard error.
• send_email.sh - send mail report.

Fortran Programs

• qc_t - performs temperature quality control
• qc_s - performs salinity quality control

Tests

Global

• Check_Sum - analyzes check-sum generated by some floats.
• Speed - flags the surface and submerged speeds that violate a given criteria. A profile fails this test if submerged or surface velocity exceeds 3.0 m/s. For the first profile the distance from launch rather than the speed is checked (if the elapsed time is less than 1 day): if the distance is <= 10 km the test is passed (flag=1). If the distance is > 10km and <= 50km the test is passed but the (launch) position is considered to be suspicious (flag=2).
• Position - profiles without a position or with a position on land will have an additional line in the header (NO POSITION INDICATOR or POSITION ON LAND INDICATOR with the value 1). For these profiles no climatology or analysis QC can be performed.
• Position - it is tested if a position is impossible (latitude outside -90 to 90, longitude outside -180 to 180)
• Time and Date - it is tested if a date or time is impossible (year must be greater than 1996, month must be in range 1 to 12, day must be in range expected for month, hour must be in range 0 to 23, minute must be in range 0 to 59).
  
• Grey List - specify flag value for parameters from consistently failing sensors.
  A typical flag value is 3. It effectively stops the real-time dissemination through GTS for measurements from a sensor (pressure, temperature or conductivity) that is not working correctly.

Pressure

• Gross - flags out of range values. A measurement fails this test if greater than 11000.0 dbar or less than 0 dbar.
• Pressure Increasing - flags if pressures are not monotonically increasing.
• Deepest Pressure - flags any pressure that is higher than DEEPEST_PRESSURE plus 5%, 10% or 15% dbar. The percentage is still under discussion, therefore the test is not applied yet. DEEPEST_PRESSURE value comes from the meta-data file of the float, one (or two) for the profile and for the submerged drift.

Temperature

• Gross- flags out of range values.
  A measurement fails this test if outside of range -2.5 to 40.0 ° C.
• Constant Value Test
• Vertical Gradient - flags gradient and inversion.
  A measurement fails this test if temperature gradient outside of range 0.2 to 1.0 ° C/m.
• Spike Check - looks for spikes.
  A measurement fails this test if tolerance of 0.4 ° C for difference between observed temperature and the associated median (of three observations) is exceeded.
• Jump Check - detects a sudden and important sensor drift.
  It calculates the average temperature on the last 100 dbar available in the new and a previous good profile. If the difference between the 2 average values is more than 1 degree C then new profile is flagged as bad. The criteria for this test have not been finalized. Therefore, the test is not in use yet.
• Frozen Profile - detects a float that reproduces the same profile (with very small deviations) over and over again.
  The differences are derived for 50 dbar slabs. The test has to be failed for 5 consecutive profiles before a float becomes a potential candidate for the grey list.
  For temperature ...
  The criteria are: largest difference < 0.3 deg. C
  smallest difference < 2e-3 deg. C
  mean difference < 3e-2 deg. C
  
• Regional Range - flags when temperature is outside of 21.7 to 40.0 for Red Sea; 10.0 to 40.0 for Mediterranean Sea.
  
• Climatology - compares profile with Levitus 2001 World Ocean Atlas data.
  A measurement fails this test if the data point is outside of F*(standard deviation) envelope around mean profile (standard deviation from WOA01) the standard deviation factor F is given in data files.
• Analysis - compares profile with NCEP's weekly analysis data.
  A measurement fails this test if the data point is outside of F*(standard deviation) envelope around mean reanalysis profile (standard deviation from WOA01) the standard deviation factor F is given in data files.

Salinity

• Gross - flags out of range values.
  A measurement fails this test if it is outside of the range 0 to 41 psu.
• Constant Value Test
• Density Gradient - analyzes Brunt-Väisäla frequency (N).
  A measurement fails this test if N ² is negative, and the difference between t(i) and t(i+1) is greater than 0.05 degrees, and the difference between s(i) and s(i+1) is greater than 0.05 psu.
• Vertical Gradient - flags gradient and inversion.
  A measurement fails this test if salinity gradient outside of range 0.5 to 1.5 psu.
• Spike Check - looks for spikes.
  A measurement fails this test if tolerance of 0.4 psu for difference between observed salinity and the associated median (of three observations) is exceeded.
• Jump Check - detects a sudden and important sensor drift.
  It calculates the average salinity on the last 100 dbar available in the new and a previous good profile. If the difference between the 2 average values is more than 0.5 psu then new profile is flagged as bad. The criteria for this test have not been finalized. Therefore, the test is not in use yet.
• Frozen Profile - detects a float that reproduces the same profile (with very small deviations) over and over again.
  The differences are derived for 50 dbar slabs. The test has to be failed for 5 consecutive profiles before a float becomes a potential candidate for the grey list.
  For salinity ...
  The criteria are: largest difference < 0.3 psu
  smallest difference < 2e-3 psu
  mean difference < 5e-3 psu
• Regional Range - flags when salinity is outside of 0.0 to 41.0 for Red Sea; 0.0 to 40.0 for Mediterranean Sea.
  
• Climatology - compares profile with Levitus 2001 World Ocean Atlas data.
  A measurement fails this test if the data point is outside of F*(standard deviation) envelope around mean profile (standard deviation from WOA01) the standard deviation factor F is given in data files.
• Analysis - compares profile with NCEP's weekly analysis data.
  A measurement fails this test if the data point is outside of F*(standard deviation) envelope around mean reanalysis profile (standard deviation from WOA01) the standard deviation factor F is given in data files.

The assigned flags for the individual tests can be:

•  '1' (passed)
•  '4' (failed)
•  '0' (can not be determined, i.e. not enough information is available).

A measurements is considered bad if it failed:

• the gross check, or
• the constant value test, or
• the vertical(T)/density(S) gradient test, or
• the spike check, or
• the pressure increasing check, or
• any combination of the above tests

An overall flags in the qc files are assigned following the rules:

•  '1' qc tests passed and data is complete
•  '2' qc tests passed and data is incomplete
•  '4' qc tests failed
•  '0' can not be determined, i.e. not enough information is available.

A profile is considered bad if:

• it fails the impossible position test or the impossible date test.
• it does not pass the speed check.
• its overall flag for temperature and/or salinity is bad.
  

QC file is always created except when the qc exit status equals 9.

Interactive, visual Quality Control: Performed for profiles that failed automatic QC.

• if a position is bad (failure of Speed Check), the location class will be changed to negative (note: 0 becomes -4, characters will get a leading minus sign) and the automatic QC will be run again.
• For each measured quantity (temperature, salinity), a column of visual QC flags will be added (float_visual_qc).
• The overall QC flag will be updated if appropriate.
  

Decisions for GTS distribution

• If a profile fails the automatic QC, it will not go to GTS automatically.
• If a profile passes the visual QC it will go to GTS.

Flag conventions for NETCDF files (following the ARGO conventions)

• 0 - no QC was performed
• 1 - all tests were passed
• 2 - inconsistencies are considered insignificant: the climatology and the reanalysis test were both failed or another automatic test was failed, and visual QC was passed.
• 4 - failed one of the other tests (excluding climatology and reanalysis test) and visual QC was failed or not performed.

Changes to QC:

• April 29, 2003: Levitus World Ocean Atlas 1998 was replaced with the Levitus World Ocean Atlas 2001. Note: this change may have an impact on float profiles that were obtained before this date, but were processed after the change. The version of the World Ocean Atlas is stored in the header of the qc files (CLIMATOLOGY VERSION USED).
• June 4, 2001: Switch from NCEP monthly analysis for the year 2000 to near-real time weekly NCEP analysis. The version of the NCEP analysis is stored in the header of the qc files (ANALYSIS YEAR USED or ANALYSIS TIME USED and ANALYSIS VERSION USED).

---

Data distribution

Physical Units to GTS TESAC Format: Converts a physical units data file to a GTS TESAC format file, if selection criteria is met (1 script and Fortran programs).

FTP GTS File to GateWay: Transfers a GTS formatted file to GTS GateWay via FTP, SEAS office in Washington DC (1 script and Fortran programs).

Packs and Compresses QC data files for FTP: Packs and compresses, according to program number, quality control data files for FTP transfer.

FTP Compressed QC Files to Local Host: Transfers compressed quality control data files to the Local Host designated FTP area for public Internet access.

FTP Compressed QC Files to Principal Investigators: Transfers compressed quality control data files to the Remote Host designated FTP area for each principal investigator according to the program number.

FTP Compressed QC Files to AOML: Transfers compressed quality control data files to the AOML Remote Host designated FTP area for loading into the USA ARGO database which will reside in a commercial database management system (Informix Dynamic Server 2000).

FTP Compressed PRV, HEX, and  CNT Files to Principal Investigators: Transfers compressed quality PRV, hexadecimal and counts data files to the Remote Host designated FTP area for each principal investigator according to the program number (pending).