CTD Data Processing ------------------- Database philosophy ------------------- CTD data are processed from raw data files to calibrated data entirely within the ctdcal toolbox. Data are maintained throughout as a matlab mat-file serving as a database. Each step creates new structures containing the partially processed CTD data, through the process of despiking, delooping and calibrating etc. Package Documentation --------------------- I see someone started using m2html to describe and characterize the scripts in the package. It seems like a useful thing to do, since I am now trying to figure out which functions are called by which other ones and who is dependent on who and who if anyone is left out, not called by anyone nor used by anyone. Download m2html from the website: http://www.artefact.tk/software/matlab/m2html/ Unzip and install it in the matlab path somewhere (using addpath in startup.m). In the directory above ctdcal/ execute in Matlab: m2html('mfiles','ctdcal','htmldir','ctdcal/html','recursive','on', ... 'source','on','graph','on'); This makes the documentation in the ctdcal/html directory. From the shell execute dot -Tps ./ctdcal/html/ctdcal/graph.dot to create a postscript output for the graphic representation of the program dependencies. (This is for a Linux based system that has dot. I do not know what to do for other operating systems.) Versioning ---------- I have set up a Makefile in the ctdcal directory that updates the RCS record of the changes in the package (make all, make revision). Build a tarball of the package to distribute it (make tarball). Package up the complete RCS record of the changes (make rcs). Make the documentation using m2html (make docs). Setting up - ---------- NOTE: neutral density functions must be installed in the path. ctdcal toolbox has these in a subdirectory for MSDOS/Windows. NOTE: upgrade to latest version of Morgan seawater library (http://www.cmar.csiro.au/datacentre/ext_docs/seawater.htm) make_cruise_dir.m - Setup a new working directory for CTD calibration. Copies demos/register_cruise.m into the directory structure. Creates ctdcalpath.m. register_cruise.m - Is not a copy of rg_register_cruise.m! Is a copy of demos/register_cruise.m. Why?!? It must be edited to have the correct cruise information: edit lines 35, 43, 49, and 63. Update: I now have make_cruise_dir do the editing, setting the values to some sensible defaults. ctdcalpath.m - sets up the path. must run it each time you begin matlab. (Why not just say to put this in startup.m?? Answer: because it sets up paths specific to this particular cruise). This is now done by a local.m script which is installed by make_cruise_dir.m. getpref.m - use to get the preferences Additional step added by CMDR ----------------------------- Make a file "cruise_id" in the directory structure which has the cruiseid information in it (make_cruise_dir does this now). Use that information if the cruiseid is not provided when calling the routines. Use the function getcruiseid.m to do this. Populate the subdirectories with the data ----------------------------------------- 1db/raw_data/*.bl in raw_data/ 1db/bottle/*.btl in bottle/ 1db/proc_data/*.cnv in proc_data/ Sample_data/*.OXY in oxy/ Sample_data/*.dat in salts/ (rename them to .txt, reformat to .dat) The rest appear not to be used. It is useful to make a "distribute" script to do the copying if there are a lot of stations to be processed and the format and file/directory names change little. See the distribute bash script in WORK/AOML/CTD/CALIBRATIONS for ideas. The oxygen file can be read as is (be sure the required data columns are present. Use the oxygen OXY file which has niskin bottle positions in the Depth column.); the salts file should be run through salt-txt2dat and the output edited for station and niskin numbers. WS0415 has salinity bottles which have alphanumeric labels. Can we use a string for these, rather than a number? There are four files to change (for salinity). (grep tells me there might be eleven or more oxygen and salinity files where this might be an issue.) For now change labels by adding 100 to the bottle number of the C-series salinity bottles. [Rigo says to add 900.] Steps not explained in the recipe provided - ========================================== Convert raw data to cnv files - this is partially done by Seasoft program. Oxygen is not calculated (because we are doing a direct regression of sbeoxv against winklers?? So then which is the correct oxy regression to use?). salts.dat file: niskin bottle numbers means niskin bottle fire positions not the actual bottle numbers. Use my salts-txt2dat.sed script to convert this to a table that can be easily edited to add the station numbers and niskin positions Continuing with matlab ====================== % Read and save the SeaBird ASCII .BL file format bl=load_bl; % Read and save the SeaBird ASCII .BTL file format btl=load_btl; % Load the information in the *.cnv files into the cruise database [cnv,cnv_load]=load_cnv(cruiseid); read_sbecnv(cnv_file) ; % read the cnv file parse_cnv_data_all; [raw,raw_load]=load_raw(cruiseid); These following programs break if there are no secondary sensors: % Fill the CTD data up to the surface cnv_fill=fill_surface; % Remove spikes in the primary sensors cnv_ds1=despike1; % Remove spikes in the secondary sensors cnv_ds2=despike2; % Remove loops in pressure series cnv_dlp=deloop; despike2 seems to mix up the primary and secondary sensor data. They also assume that something has created oxygen concentrations from sensor voltages but I do not see where that happens if it has not been done by Seasoft. Checking results ================ fh=compare_ctd_sensors; print_compare_ctd_sensors(cruiseid,fh); fh=compare_tsplot; print_compare_tsplot(cruiseid,fh); % raw tsplot Apply bottle results ==================== For salinity: ------------ Use bash script salt-txt2dat to assist generating appropriate formatted salt dat file. salts=load_salts(cruiseid); cond_match=make_cond_sumfile(cruiseid); [sfh,fh]=compare_cond(cruiseid); % sfh are neat figures, fh are working figs print_compare_cond('',sfh); new_cond_match=check_bad_salt(cruiseid); For oxygen: ---------- oxy=load_oxy(cruiseid); read_oxy oxy_match=make_oxy_sumfile(cruiseid); [sfh,fh]=compare_oxy(cruiseid); % sfh are neat figures, fh are working figs print_compare_oxy('',sfh); new_oxy_match=check_bad_oxy(cruiseid); Two other programs are needed when you have everything ready and want to obtain the calibration coefficients [offset_out,offset_fit,slope_out,slope_fit,pcor]=cond_cal(cruiseid); OR [cond_coeff,fh]=cond_cal(cruiseid); % where cond_coeff is a cell of % the output from the first form print_cond_cal('',fh); [oxy_coeff,fh]=oxy_cal(cruiseid); print_oxy_cal(cruiseid,fh); Final calibration ================= Finally, to apply the calibration coefficients run [cnv_cal,fh] =FS_apply_calibration(cruiseid,offset_fit,slope_fit,pcor,oxy_coeff) OR [cnv_cal,fh]=FS_apply_calibration(cruiseid, cond_coeff, oxy_coeff); print_cal_figures(cruiseid, fh); % (but FS_apply_calibration prints to cal_ctd.ps) fh=compare_tsplot; print_compare_tsplot(cruiseid,fh); % cal tsplot Write out the data to standard format data files % write_s87(cruiseid,cnv_cal) % save_final_results(cruiseid,staa,stab) write_s87; save_final_results; Confirming validity of results ------------------------------ checking - compares up and down traces against bottle samples plot_all - plots multiple cruises on the same axes What do these do? ================= ctd_complete_process.m rigo doitall script write_aoml3.m what? correct_autosal_drift.m for a particular cruise FS_map_with_cast.m Draw a map? Requires m_map toolbox from http://www.eos.ubc.ca/~rich/map.html CTD data calibration report =========================== Given the example of the "Straits of Florida: CTD data calibration report" for Cruise WS0625, the figures required are: Fig 1: "CTD profiles: Temperature" and "Temperature differences [pri-sec]" [ created by compare_ctd_sensors.m ] [ print using print_compare_ctd_sensors.m ] Fig 2: "CTD profiles: Conductivity" and "Conductivity differences [pri-sec]" [ created by compare_ctd_sensors.m ] [ print using print_compare_ctd_sensors.m ] Fig 3: "CTD profiles: Dissolved Oxygen" and "Dissolved Oxygen differences [pri-sec]" [ created by compare_ctd_sensors.m ] [ print using print_compare_ctd_sensors.m ] Fig 4: "TS-Plot Raw" [ created by compare_tsplot.m ] [ print using print_compare_tsplot.m ] Fig 5: "TS-Plot Calibrated" [ created by compare_tsplot.m ] [ print using print_compare_tsplot.m ] Fig 6: "Primary and Secondary Sensors (Uncalibrated): CTD salinity vs Bottle salinity: Scatterplot and Least Square fits" [ created by compare_cond.m ] [ print using print_compare_cond.m ] Fig 7: "Primary and Secondary Sensors (Uncalibrated): Bottle salinity - CTD salinity: vs Station Number and Niskin bottle number" [ created by compare_cond.m ] [ print using print_compare_cond.m ] Fig 8: "Primary and Secondary Sensors (Uncalibrated): Bottle salinity-CTD salinity: vs Pressure and Box plot" [ created by compare_cond.m ] [ print using print_compare_cond.m ] Fig 9: "Primary and Secondary Sensors (Uncalibrated): Bottle salinity-CTD salinity: QQ and Box plots" [ created by compare_cond.m ] [ print using print_compare_cond.m ] Fig 10: "Conductivity calibration: Primary sensor: (a) bottle-ctd conductivity uncalibrated vs station number (b) vs pressure (c) temperature sensor differences vs station number (d) bottle-ctd conductivity vs station number after temperature based removal (e) primary sensor bottle-ctd conductivity differences vs pressure (f) primary sensor conductivity offset vs station number [ created by cond_cal.m (20 figs. Can we print all on one page? ] [ print using print_cond_cal.m ] Fig 11: "Conductivity calibration: Primary sensor: (a) bottle-ctd conductivity final offset correction vs station number (b) bottle-ctd conductivity vs pressure (c) conductivity slope vs station number (d) bottle-ctd conductivity final slope correction vs station number (e) bottle-ctd conductivity vs pressure (f) bottle-ctd conductivity pressure correction vs station number" [ created by cond_cal.m (20 figs. Can we print all on one page? ] [ print using print_cond_cal.m ] Fig 12: "Conductivity calibration: primary sensor: (a) bottle-ctd conductivity vs pressure (b) bottle-ctd salinity vs station number (c) bottle-ctd salinity vs pressure (d) bottle-ctd salinity vs station number (e) bottle-ctd salinity vs pressure (f) bottle-ctd salinity vs pressure" [ created by cond_cal.m (20 figs. Can we print all on one page? ] [ print using print_cond_cal.m ] Fig 13: "Primary and secondary sensors: (a,b) ctd vs bottle oxygen (c,d) all ctd vs bottle oxygen least squares fit (e,f) below 0dbar ctd vs bottle oxygen least squares fit" [ created by compare_oxy.m ] [ print using print_compare_oxy.m ] Fig 14: "Primary and secondary sensors: uncalibrated (a,b) ctd vs bottle oxygen (c,d) bottle-ctd oxygen vs station number (e,f) bottle-ctd oxygen vs niskin number" [ created by compare_oxy.m ] [ print using print_compare_oxy.m ] Fig 15: "primary and secondary sensors: uncalibrated: (a,b) bottle-ctd oxygen vs sample bottle number (c,d) bottle-ctd oxygen vs pressure (e,f) bottle-ctd oxygen boxplot [ created by compare_oxy.m ] [ print using print_compare_oxy.m ] Fig 16: "primary and secondary sensors: uncalibrated: (a,b) bottle-ctd QQ plot (c,d) bottle-ctd box plot (e,f) bottle-ctd QQ plot [ created by compare_oxy.m ] [ print using print_compare_oxy.m ] Fig 17: "Oxygen Calibration: secondary sensor uncalibrated: (a) ctd-bottle oxygen vs station number (b) ctd-bottle oxygen vs pressure (c) temperature sensor differences vs station number (d) ctd-bottle oxygen vs station number after temperature based removal (e) ctd-bottle oxygen vs station number final result (f) ctd-bottle oxygen vs pressure" ******: Fig 17 has "ctd-bottle" in the labels. All (?) the others have "bottle - CTD". Error or mislabelling? [ created by oxy_cal.m (11 figs. Can we print all on one page? ] [ print using print_oxy_cal.m ] Fig 18: "Oxygen calibration: (a) bottle-ctd oxygen vs station number (b) bottle-primary ctd oxygen vs pressure" [ created by oxy_cal.m (11 figs. Can we print all on one page? ] [ print using print_oxy_cal.m ] Fig 19 to 27: Profiles of stations 0 to 8: parameter against pressure, Pot Temp vs Salinity, Pot Temp vs oxygen, Salinity vs oxygen. [ created by FS_apply_calibration.m ] [ figures are printed by FS_apply_calibration.m to cal_ctd.ps ] Function Table -------------- Function Input Output Product (See spreadsheet functions.ods) Things to do ============ plot completed cruises on the same TS diag. check_bad_salt.m - plot ts and compare bottle data to it check_bad_oxy.m - plot to and compare bottle data to it Things done ----------- write_s87.m Find some historic data - Abaco_1000dbar_CTDs.mat selectpts.m The lost -------- m2html shows that the following scripts are not referenced by any other, nor call any other in the package. What do they do, what are they for, why are they here? check_cal correct_autosal_drift ddeblank eval_fit floatAxisY linreg linreg1 oxygen_fit_linear oxygen_fit_linearp oxygen_fit_none oxygen_fit_poly2 oxygen_fit_poly2p oxygen_fit_pres write_aoml3 Reorganized Contents.m to aid assessment ---------------------------------------- % Data Management % --------------- % load_bl.m - Read and save the SeaBird ASCII .BL file format % load_btl.m - Read and save the SeaBird ASCII .BTL file format % load_cnv_beckman.m - Load the information in the *.cnv files into the cruise database % load_cnv.m - Load the information in the *.cnv files into the cruise database % load_oxy.m - Load a *.oxy files from the AOML Winkler titration system % load_salts.m - Load the autosal samples into the cruise database % % non_lin_fit.m - Nonlinear least-squares data fitting by the Gauss-Newton method % normalqqplot.m - plot quantiles of data vs quantiles of normal distribution % normqqplot.m - plot quantiles of data vs quantiles of normal distribution % parse_cnv_data_all_beckman.m -parse data columns read in from raw CNV files % parse_cnv_data_all.m - parse data columns read in from raw CNV files % parse_cnv_data.m - data columns read in from raw CNV files % quantile.m - quantiles of the data x % read_oxy.m - Load a *.oxy file from the AOML Winkler titration system % read_sbebl.m - Reads the SeaBird ASCII .BL file format % read_sbebtl.m - Reads the SeaBird ASCII .BTL file format % read_sbecnv.m - Reads the SeaBird BINARY or ASCII *.cnv file format % % Calibration Routines % -------------------- % apply_calibration1.m - Apply oxygen and conductivity calibration % apply_calibration.m - Apply oxygen and conductivity calibration % cond_cal1.m - Conductivity calibration calling routine % cond_cal2.m - Conductivity calibration calling routine % cond_cal.m - Conductivity calibration calling routine % cond_cal_ORIGINAL.m - Conductivity calibration calling routine % FS_apply_calibration.m - Apply oxygen and conductivity calibration % make_botlle_file.m - Apply oxygen and conductivity calibration % make_bottle_file.m - Apply oxygen and conductivity calibration % make_cond_sumfile.m - Makes conductivity bottle/ctd comparison summary file % make_woce_botlle_file.m - Apply oxygen and conductivity calibration % make_woce_bottle_file_beckman.m - Apply oxygen and conductivity calibration % oxy_cal_beckman.m - Oxygen calibration calling routine % oxy_cal.m - Oxygen calibration calling routine % oxygen_fit_linear_beckman.m - to calculate the dissolved oxygen % oxygen_fit_linear.m - to calculate the dissolved oxygen % oxygen_fit_linearp_beckman.m - to calculate the dissolved oxygen % oxygen_fit_linearp.m - to calculate the dissolved oxygen % oxygen_fit_none_beckman.m - to calculate the dissolved oxygen % oxygen_fit_none.m - to calculate the dissolved oxygen % oxygen_fit_poly2_beckman.m - to calculate the dissolved oxygen % oxygen_fit_poly2.m - to calculate the dissolved oxygen % oxygen_fit_poly2p_beckman.m - to calculate the dissolved oxygen % oxygen_fit_poly2p.m - to calculate the dissolved oxygen % oxygen_fit_pres.m - to calculate the dissolved oxygen % oxygen_fit_press_beckman.m - to calculate the dissolved oxygen % % weim.m - Running weighted average of input data given a window % ox_units.m - converts between 3 different oxygen units % % Sensor Correction % ----------------- % correct_autosal_drift.m - Correct for autosal drift with pre/post standard water analyses % correct_pressure.m - Apply the correction for temperature sensors % correct_temp.m - Apply the correction for temperature sensors % deloop_beckman.m - Similar to LOOPEDIT on Seabird Data Processing % deloop.m - Similar to LOOPEDIT on Seabird Data Processing % despike1_beckman.m - Remove spikes in the primary sensors % despike1.m - Remove spikes in the primary sensors % despike2_beckman.m - Remove spikes in the secondary sensors % despike2.m - Remove spikes in the secondary sensors % fill_surface_beckman.m - Fill the CTD data up to the surface % fill_surface.m - Fill the CTD data up to the surface % % print_cal_figures.m - Print the output figures that come from FS_apply_calibration.m % print_compare_cond.m - Print the output figures that come from compare_cond.m % print_compare_ctd_sensors.m - Print the output figures that come from compare_ctd_sensors.m % print_compare_oxy.m - Print the output figures that come from compare_oxy.m % print_cond_cal.m - Print the output figures that come from cond_cal.m % print_oxy_cal.m - Print the output figures that come from oxy_cal.m % % Graphic Tools % ------------- % boxparams_deep.m - calculate boxplot parameters for data vector x % boxparams.m - calculate boxplot parameters for data vector x % boxplotter1.m - make box or whisker plots % % compare_cond.m - Diagnostics plots to compare conductivity values between CTD and bottle samples. % compare_ctd_sensors.m - Diagnostics plots to compare the sensors (conductivity, dissolved oxygen % compare_oxy.m - Diagnostics plots to compare CTD and samples dissolved oxygen values % compare_tsplot.m - Makes ts-plots % compare_tsplot_original.m - Makes ts-plots % plot_sensors_diff_beckman.m - Diagnostics plots to compare differences between the sensors % plot_sensors_diff.m - Diagnostics plots to compare differences between the sensors % % floatAxisX.m - create floating x-axis for multi-parameter plot % floatAxisY.m - create floating y-axis for multi-parameter plot % % window.m - Return a window of a given shape for smoothing data % % Text and Matrix Manipulation % ---------------------------- % ddeblank.m - Double deblank. Strip both leading and trailing blanks % ddewhite.m - Double dewhite. Strip both leading and trailing whitespace % meanmiss.m - Column matrix means with missing data % % Station/Cruise Management % ------------------------- % group_name.m - Get the preference group name for a given cruiseid % make_cruise_dir.m - Setup a new working directory for CTD calibration % make_cruise_dir_old.m - Setup a new working directory for CTD calibration % make_oxy_sumfile_beckman.m - Makes oxygen bottle/ctd comparison summary file % make_oxy_sumfile.m - Makes oxygen bottle/ctd comparison summary file % oxy_struct.m - Return an empty oxygen calibration struct with minimum required fields % rg_register_cruise.m - Input and manage cruise meta-data prior to calibration % sample_struct.m - Return an empty data structure with the minimum required fields % save_final_results.m - Apply oxygen and conductivity calibration and save % write_aoml3.m - Write hydrographic profile data into aoml ascii format % % ------------------------------------------------------------ % The following have bad help information. They need fixing up! % % check_cal.m - script which does some cruise specific checking % cleaner.m - cleans up a data record % cruise_setup.m - script to SETUP a CRUISE for data collection of LADCP, CTD, % ctd_complete_process.m - script does the complete calibration process to the CTD data % eval_fit.m - Looks like a one-off concoction for doing something % FS_map_with_cast.m - script plots a map of the study region and adds on cable locations % glitch.m - finds 'glitches' (?) in trace? % linreg1.m - solves directly for the coefficients in linear model % linregctd.m - solves linear regression % linreg.m - computes linear regression of y on x % makeaxis.m - creates an axis of the size input in inches (sets some defaults) % makefig.m - function creates full page figure % make_plots.m - makes a series of plots of CTD data % nan_mean.m - mean of data ignoring NaNs % nan_std.m - std dev of data ignoring NaNs % oxy_fit_linear_beckman.m - fit Beckman sensor to oxygen data % oxy_fit_linear.m - fit oxygen sensor to oxygen data % oxy_fit_linearp_beckman.m - fit Beckman sensor to oxygen data % oxy_fit_linearp.m - fit oxygen sensor to oxygen data % oxy_fit_none_beckman.m - fit Beckman sensor to oxygen data % oxy_fit_none.m - fit oxygen sensor to oxygen data % oxy_fit_poly2_beckman.m - fit Beckman sensor to oxygen data % oxy_fit_poly2.m - fit oxygen sensor to oxygen data % oxy_fit_poly2p_beckman.m - fit Beckman sensor to oxygen data % oxy_fit_poly2p.m - fit oxygen sensor to oxygen data % oxy_fit_pres.m - fit oxygen sensor to oxygen data including pressure dep % oxy_fit_press_beckman.m - fit Beckman sensor to oxygen data including pressure dep % plot_sensors_residuals.m - plot sensor residuals % save_plots.m - plot multiple parameters on same page % t68tot90.m - Converts temperature from ITPS-68 to ITS-90 % t90tot68.m - Converts temperature from ITPS-90 to ITS-68 % % Contents file for ctdcal directory constructed from comments % in m-files % %PROGRAM - MATLAB code by c.m.duncombe rae % %CREATED - 2007/11/09 % %PROG MODS - % %DISCLAIMER - No responsibility can be accepted for failure or % incorrect operation of this code. No guarantee, express % or implied, is given. % Author: C M Duncombe Rae $Revision: 1.27 $ $Date: 2008/04/29 23:14:46 $ $Id: notes.txt,v 1.27 2008/04/29 23:14:46 duncombe Exp duncombe $