How to process the IES data? We obtain IES data in two different ways: telemetry data and data from recovered instruments. In order to process the IES telemetry data we need to do the following steps: 0. Ask for the preliminary cruise report. Check if is there any problem with the telemetry data, etc. 1. Save the raw data file in the directory /home/cable/IES/REGION/Telemetry_data/Site_ID/Raw/ where: REGION: region where IES has been deployed (ex. ABACO, FS, SA, etc) ID: Identification of the Site in a particular region. For example: A (site A) in Abaco region. 2. Rename the raw file to the format Pfile_SNsn_Year_Mn.dat ... (1) where: sn: serial number of the instrument (two or three digits) Year: four digits year when the data was collected Mn: two digits Month when the data was collected Ex. Pfile_SN159_2007_09.dat Note: In case that you telemetred a C-PIES the file name start with Cfile_ instead of Pfile_ 3. Go to Matlab workspace. Change directory to /home/cable/IES/IES_softwares/proc_telemetry/ In this directory reside all the matlab scripts necesaries to read and preprocess the raw telemetry data. 4. Run Matlab function find_gaps_telemetry_data.m Input: raw file name [see ...(1)] This function is designed to load in an IES telemetry record and add a new column to the file at the right edge. If the record is zero that means that there is no gap in the time series (the time differential was either 24 or 25 seconds). If the record is -1 the time differential was less than 24 seconds. If the differential is longer than 25 seconds the record will be 1. Note not all gaps longer than 25 seconds are signs of error! All gaps before MSB records are a minute or longer. This function work on both PIES and C-PIES instruments. Output: file with the same name that in (1) adding a _gf (gaps found!). Ex. Pfile_SN159_2007_09_gf.dat IMPORTANT NOTE: This "new" file need to be checked and edited manually! (see Chris Meinen's Notes for hand-editing IIE/PIES/CPIES telemetry files) The final version of the edited file need to be save with the same name in (1) adding: _handedit; Ex. Pfile_SN159_2007_09_handedit.dat NOTE: It's important to keep this file name format because other programs are looking for its. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Pre-process the telemetry data: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 5. Run Matlab script update_ies_db.m This script update the IES data base for a particular site in a particular region. The IES data base are saved in the directories: /home/cable/IES/REGION/DB where: REGION: region where IES has been deployed (ex. ABACO, FS, SA, etc) The name of the data base is, for example: db_siteA.mat (for site A in ABACO region) Inputs: - Region where the instrument is recording data. - Site ID where telemetry was taken - Type of update: IES Telemetry data This last input run the function update_ies_telemetry.m - IES Serial Number _ Year, Month, day when the telemetry was done. 6. Run Matlab script process_IES_telemetred_data.m Inputs: - Region where the instrument is recording data. - Year when the cruise/telemetry was taken - Month when the cruise/telemetry was taken This script call three different matlab functions: 6.1 load_telemetry_data.m This script is designed to load in the telemetry file from the PIES and CPIES and create vectors of the various measurements along with a more useful time axis for the record (rather than wrapped yearday). All missing values (set as -99) will be replaced with NaNs. The Most-Significant-Bits (MSB) will be extracted and used with the Least-Significant-Bits (LSB) in order to have a time series of values including both MSB and LSB. The user need to decide which value of tauMSB, presMSB and speedMSB (for CPIES) is correct. The MSB data obtained in this last telemetry are presented to the user together with the 'historical' MSB obtained in that particular site/region. This function also do some despike to the travel time, pressure and velocity components for CPIES. Inputs: - IES serial number Outputs: - file: tel_data_site_S_SNsn_Year_Mn.mat 6.2 Function: despike_telemetry_data.m In this script the user can despike manual/interactivelly the time series obtained for travel time, pressure and velocity components (for CPIES). The user decide/select which point want to remove. Inputs: - file: tel_data_site_S_SNsn_Year_Mn.mat Outputs: - file: tel_data_site_S_SNsn_Year_Mn_dpk.mat 6.3 Function: combine_telemetry_data.m Matlab script to combine two or more telemetry data records from one IES instrument (serial number) in a particular site. Matlab function to combine two or more telemetry data records from one instrument in a particular site. Also, plot all different data that has been telemetred for that particular instrument in that particular site/region. Inputs: - file: tel_data_site_S_SNsn_Year_Mn_dpk.mat Outputs: - file: tel_data_site_S_SNsn.mat This last file represent all the telemetry data for that instrument (SN) in that site/region. Note: The output of these pre-processed telemetry data are saved on /home/cable/IES/REGION/Telemetry_data/Site_ID/Processed/ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Process Site time series %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% To obtain the time serie of Travel Time for one site, you need run the following scripts: 1) calibrate_CTD.m (in case that you have the CTD information!) 2) calibrate_travel_time_IES.m 3) combine_travel_time.m To obtain the time serie of Pressure for one site, you need to run the following scripts: 0) dedrift_pressure.m (add here on 5/5/2021 from home) 1) combine_pressure.m 2) create_site_pressure_time_serie.m %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Process Region time series %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% To create/obtain the region time series run the following matlab scripts: 1) create_region_time_series.m 2) plot_region_time_series.m 3) lpf_region_time_series.m (plot the results) When the IES have been recovered we have to pre-process the data using URI data processing v3.12. The output of this processing softwares need to be call, for example: rec_data_site_S_SNsn.mat (2) The internal structure must be called (allways!): rec_data (inside the matlab file .mat) The file (2) must be saved in the directory /home/cable/IES/REGION/Recovered_data/Site_ID/Processed/ where: REGION: region where IES has been deployed (ex. ABACO, FS, SA, etc) ID: Identification of the Site in a particular region. For example: A (site A) in Abaco region: /home/cable/IES/ABACO/Recovered_data/Site_A/Processed/rec_data_site_A_SN159.mat Now we can run the programs in the sections: Process Site time series and Process Region time series. Date: 5/20/2008 (First draft) 6/25/2008 (Second draft) Incorporate the case when we have the data from a recovered IES.