Manual for analyzing raw data obtained with the SED sensor

Transcription

1 Manual for analyzing raw data obtained with the SED sensor Pim Willemsen Version 0.1 CONCEPT VERSION 1 Concept version (0.1)

2 Contents 1. Introduction The sensor Raw data Files and their function Tutorial Single Batch Notes Access to the script Literature Concept version (0.1)

3 1. Introduction SED sensors (Fig. 1) are developed by the NIOZ (Royal Netherlands Institute for Sea Research) and described previously by (Hu et al., 2015). The SED sensor is a stand-alone device that can be applied for obtaining bed level dynamics. The sensor measures the light intensity over 204 light sensitive cells of 2 mm. The resulting profile per time step over those 204 cells, is used to obtain the bed level per time step. Converting the raw data to bed level data using the SED-script is described in this manual. Figure 1. Front side (panel 1)m and back side (panel 2) of the SED-sensor, the enlarged data logging section of the sensor is showed in panel 3 (Hu et al., 2015). The light sensitive cells are located in the measuring section (194 cells), and the head (10 cells. The top 3 cells of the measuring section are assumed to be influenced by the head and will not be used for data analysis (panel 4) The sensor The sensor has a storage capacity of 8GB and is powered by nine AAA batteries. The light sensitive cells within the transparent polyester part of the sensor can be used to measure the bed level. This is a section of 194 cells of 2 mm each (total measuring section is 388 mm). The cells are within a non-transparent case and cells are located on the circuit board in the head of the sensor (Fig. 1). 3 Concept version (0.1)

4 1.2. Raw data Raw data is obtained by the SED sensors using three sensitivities (low, mid and high), resulting in three scaled voltage profiles (Fig. 2). A low scaled voltage is shown when the light intensity in the surrounding area is low, resulting in typical tilted S-shaped curves. The transition from a low to a high voltage in this profile (transition zone), shows the interface of the soil and air. In case a small layer of water is present around the SED sensor (e.g. due to a scour hole), a double S curve can be observed. This curve shows a transition at the interface of soil and water and a transition at the interface of water and air. In addition to the scaled voltage output several other parameters (metadata) are stored per time step per sensitivity. One measurement (i.e. raw data of one sensitivity at one time step) consists of 256 elements (0-255) of 2 bytes (table 1). To avoid the influence of the head of the SED-sensor on the voltage profile, only the light sensitive cells (elements 0-190) are used for analysis (Fig. 1). The influence of other factors is studied by conducting different experiments (De Mey, 2016). Table 1. Elements of the raw data from one measurement (one sensitivity at one time step) from a SED sensor. Element(s) Parameter Scaled voltage output of light sensors in the transparent polyester part Scaled voltage output of light sensors in the non-transparent part Scaled voltage output of light sensors at the circuit board in the head of the sensor 204 Stabilized power supply 205 Battery power 206 Auxiliary temperature sensor 207 Temperature 208 Sensitivity of the measurement: 0 is high (1Mohm), 1 is mid (10kOhm), 2 is low (100 Ohm) Time in seconds (32 bits) 255 Time in milliseconds after time in seconds Figure 2. Voltage profiles (raw data) and arctan approximations. 4 Concept version (0.1)

5 2. Files and their function The script contains text files that can be executed using MATLAB software. The files are created using MATLAB R2015B. The main file is called Script_SED_sensor.m, the function files used in this main file are stored in the folder Functions. This folder Functions should be located in the same folder as the main file Script_SED_sensor.m. The main file is used to initiate converting the raw data, to call all function files, and save data and figures. The function files (in order they are used) stored in the folder Functions are described below. SED_Batch.m This file is called to specify whether you want to run a batch of raw SED files or a single file. File_loc.m This file is called to specify the location and name of the raw SED file(s) and to specify the location of the output file(s). In case you choose to analyze a single file, you have to specify the folder location and name of the SED file and the folder location where the output files should be stored. In case a batch of files will be analyzed, in addition to specifying the in- and output folder, a text file with the filenames should be specified. Collect_time_parameters.m This file is called to determine the points in time data is collected and to determine the time step in between the measurements. The time is stored as binary values, that are converted using this function. Sensor_selection_menu.m This file is called to finalize the initialization. First a check is executed whether the timeseries remains consistent over all measurements. It appeared that some files shows time jumps of several years from a certain point in the data. If this happens, the part of the data from the inconsistency till the end is cut off. In case a single file is chosen in the SED_Batch menu, a menu appears where you need to specify: Title/name that will be used for saving the output files and that will be used as titles for figures; Whether you want to calculate the bed level at the top of a possible appearing scouring hole. At what measurement time you want to start analyzing; At what measurement time you want to stop analyzing; Whether you want to use the outlier filter. In case a batch of files is chosen in the SED_Batch menu, the choices specified above will be default: Title/name: name of the raw SED files (without possible extension) specified in the text file; Calculate the top of a possible appearing scour hole: Yes; Time you want to start analyzing: first possible time; Time you want to stop analyzing: last possible time (taken into account the consistency of the time steps); Use outlier filter: Yes. 5 Concept version (0.1)

6 Read_SED_sensors.m This file is called to load the data of the raw SED file that is currently being analyzed. The result is a large matrix with one time step every column. Translating_time.m This file is called to crop the matrix resulting from the previous step, according to the given time parameters in the Sensor_selection_menu.m file. Intersecting_script.m This file is called to determine the bed level values. First the data will be pre-processed: The voltage profiles resulting from the raw data contain noise (assuming due to differences in the light sensitive sensors). All values that are located below the last minimum possible value are made equal to the minimum possible value (e.g. all sensors in the bed get the minimum possible value); The values in the 3 (mid, low and high sensitivity) voltage profiles are standardized, so they can be compared with each other; A filter is applied to decrease the general noise appearing in all profiles. After pre-processing the data, the first bed level (assuming to be the bottom of a possible appearing scour hole) will be distilled from the raw data, using multiple iteration steps. The pre-processed voltage profiles will be approximated with a single or double arctan function: o a * arctan(x-b) + e (approximation parameters: a, b, e), o a * arctan(x-b) + c*arctan(x-d) + e (approximation parameters: a, b, c, d, e). First the lowest bottom level (bottom of scour hole) will be determined. o If the maximum value from the high sensitivity voltage profile is equal to the maximum possible value (4095), the bed level will be approximated by a double arctan, using the mid sensitivity. The bed level is located at the minimum of approximation parameters b and d, which is the center of the first (from bottom to top) transition slope. In case the approximation parameters a and/or c are negative, the transition slopes are not approximated right. o In that case a single arctan approximation will be used, with the bed level at approximation parameter b. In case the maximum voltage value of the low sensitivity is smaller than 200, there is little light. In addition, if the maximum voltage of the mid sensitivity is equal to the maximum possible value, the following will be done: o The profile will be approximated by a single arctan. Around the approximation parameter b of this arctan fit, a smaller area is selected (b 25 till b + 25) for creating a double arctan approximation. The bed level is located at the minimum of approximation parameters b and d, which is the center of the first (from bottom to top) transition slope. In case a or c is negative again: o The approximation parameters, bed level and R 2 will be equal to NaN. 6 Concept version (0.1)

7 If the approximation parameters do not have acceptable and/or possible/realistic values for approximating the voltage profiles and the R 2 of the resulting approximation is smaller than 0.99: o The high sensitivity voltage profile will be used to make a double arctan approximation. The bed level is located at the minimum of approximation parameters b and d, which is the center of the first (from bottom to top) transition slope. In case the high sensitivity voltage profile cannot be approximated well (R 2 < 0.99): o The approximation parameters, bed level and R 2 will be equal to NaN. The second bed level (top of a possible appearing scour hole) will also be distilled using multiple iteration steps. In case the following conditions are met (using the result of the calculations for the first bed level), the second bed level can be found (maximum of approximation parameters b and d): o The approximation should be done with a double arctan, otherwise only one bed level is detected, and; o The difference between the two approximation parameters (b and d) should exceed 1, so the difference should be more than the distance between two sensors. A bed level will also be found, if the following conditions are met: o Approximation parameter e is equal to or exceeds 0.5 and is smaller than 1.0; o The mid sensitivity is used. In case the above conditions are not met, the following conditions will be tested. If the conditions are met the bed level can be found in the low sensitivity voltage profile (maximum of approximation parameters b and d): o The R 2 of the approximation of the low sensitivity voltage profile is equal to or exceeds 0.98; o The difference between the two approximation parameters (b and d) should exceed 1, so the difference should be more than the distance between two sensors; o The maximum value of the low sensitivity voltage profile is smaller than or equal to 200; o The maximum value of the high sensitivity voltage profile is equal to the maximum possible voltage profile. In case the above conditions are not met: o The approximation parameters, bed level and R 2 will be equal to NaN. After approximating the bed levels, the resulting data will be cleaned. All resulting bed level values exceeding 191 (maximum number of sensors that measure) will be removed; Isolated values, i.e. a bed level value with NaN neighbours or 2 bed level values with NaN neighbours will be removed. Erasing_outliers.m This file is called to check whether outliers are present is checked using a moving average window. Using a given threshold, it is determined whether a value deviates from the surrounding moving average. If so, the value will be removed. This is applied assuming that there should always be multiple measurements 7 Concept version (0.1)

8 during a period without inundation and with daylight. During those periods the bed level is assumed to be stable. Bedlevel_prediction_blocks.m This file is called for calculating an averaged bed level for every non inundated period during daylight. This is applied assuming that there should always be multiple measurements during a period without inundation and with daylight. During those periods the bed level should be stable. Bedlevel_prediction_moving_average.m This file is called for calculating a moving average bed level for all time steps a bed level is available. Generate_plots.m This file is called to plot the moving average and block average results. Those figures will be saved in the main script. Generating_output.m This file is called to generate three matrices with the resulting data (resulting bed levels, resulting moving average bed levels, resulting block average bed levels). The matrices will be saved in the main script. ia_plots.m / ia_profile_check.m These files are called if a single raw SED file is analyzed. They can be used to see the voltage profiles and approximations for a specific bed level per time step, by clicking at the point in the graph. The script will lead you through. 8 Concept version (0.1)

9 3. Tutorial Store the folder/.zip file/.rar file at your machine. In case you have a.zip or.rar file, extract the files to your machine. The folder should contain the file script_sed_sensor.m and a folder called Functions, in which all scripts that will be used (described in chapter 2) are stored. Open the file script_sed_sensor.m and run the script (f5). You will be asked whether a single raw SED file or batch of raw SED files should be analyzed (Fig. 3). If you have multiple files you can press batch, if you have a single file, you can press single. If you want insights in the raw data, you have to press single to assess the raw data after the script is done analyzing. Figure Single If you want to analyze a single file and pressed Single, you have to enter the path and filename of the raw SED file you want to assess. Make sure to enter the file extension in case the raw SED data file contains an extension. You also need to enter the path of the output folder, this line should end with a backslash (Fig. 4). The output folder should be an existing folder. Press OK. Figure 4. After entering the file and folder information, a pop-up menu appears (Fig. 5). You will be asked to enter the following data: Title/name that will be used for saving the output files and that will be used as titles for figures; Whether you want to calculate the bed level at the top of a possible appearing scouring hole. At what measurement time you want to start analyzing (default is the first time step); At what measurement time you want to stop analyzing (default is the last time stel); Whether you want to use the outlier filter. Press OK after entering the data. 9 Concept version (0.1)

10 Figure 5. Now MATLAB will start analyzing the data and saves the resulting text files and figures in the specified folder. After the data is analyzed, a pop-up window appears with the question whether you want to assess the raw data (Fig. 6). If you press No, you are done. If you press Yes, you have to specify the area of interest (top left and bottom right point) in the figure that popped-up together with the pop-up containing the question (Fig. 7a and b). After specifying the corners you will be asked to click a point of interest (Fig. 8). After clicking the point, in the background three figures are created; one for the point you pressed and two for the surrounding points. Those figures consist of the raw voltage profiles (Fig. 2), the approximations, the bed level and information about the used sensitivity and the averaged results. The points assessed appear in the time series of the bed level, and you will be asked whether you want to assess another point. If you press No, you are done. If you press Yes, you can follow the steps in this section again. Figure Concept version (0.1)

11 Figure 7a (top) and b (bottom) Figure Batch If you want to analyze multiple files and pressed Batch, you have to enter the path and filename of a text file (.txt) that contains all names and extensions of the raw SED files you want to analyze. You can enter one file name per line in the text file. All raw SED files should be located in the same folder. You also need the enter the path of the input and output folder, those lines should end with a backslash (Fig. 9). The input and output folders should be existing folders. Press OK. Now all data will be analyzed and the figures and text files will be stored in the output folder, using the name of the input files. After that you are done. Figure Concept version (0.1)

12 4. Notes 1. The script is in constant development. 2. You can help in developing the script by adding code and ideas. 3. Determining the top of a possibly appearing scouring hole is not yet developed to its full potential and you should be careful with using those results. 5. Access to the script The script is available via the website of the NIOZ (link), please upload a new version in case you update the script. Explain how and what you need to do. 6. Literature De Mey, J. (2016). Sedimentation and erosion at intertidal vegetated foreshores, developing a script for the SED sensor to study bed level changes. (MSc.), University of Twente, Enschede. Hu, Z., Lenting, W., van der Wal, D., & Bouma, T. J. (2015). Continuous monitoring bed-level dynamics on an intertidal flat: Introducing novel, stand-alone high-resolution SED-sensors. Geomorphology, 245, Concept version (0.1)