Tilottama Ghosh and Kimberly Baugh (CIRES, University of Colorado, Boulder, USA)

Transcription

1 Telecommuting 11.5 time zones Tilottama Ghosh and Kimberly Baugh (CIRES, University of Colorado, Boulder, USA) Asia Pacific Advanced Network 32 nd Meeting New Delhi, India 26 th August, 2011

2 Brief employment history and Migration decision Started as an intern with the Earth Observation Group (EOG) at the National Geophysical Data Center (NGDC), National Oceanic and Atmospheric Administration (NOAA), in June, Continued working as a part time employee while pursuing a PhD degree in Geography from the University of Denver Became a doctorate in June of 2010 Decided to get married in December, 2010 and migrate back to New Delhi, India. Became a full time employee at NGDC from August of 2010 and expressed the desire to continue working part time with the Earth Observation Group even after returning to India Permission was granted, paper work was done, and then all the arrangements had to be made so that I could work smoothly even while being located 11.5 time zones away!

3 11.5 time zones away Mountain time zone and Indian Standard time Time difference increases to 12 and a half hours when the daylight saving time ends in November in the U.S.

4 Arrangements made while in Boulder, Colorado A good laptop A good laptop which would be able to handle the download and processing of huge datasets HP Pavilion dv4 notebook PC CPU processing speed and Random Access Memory (RAM) were important considerations The notebook has Intel Core i3 64 bit processor, and an installed memory of 4 GB

5 Arrangements made while in Boulder, Colorado Installing PuTTY PuTTY is a client program for the SSH, Telnet and Rlogin network protocols These protocols are used to run a remote session on a computer, over a network How it works you run PuTTY on a Windows machine, and tell it to connect to a Linux machine. PuTTY opens a window. Anything you type into that window is sent straight to the Linux machine, and everything the Linux machine sends back is displayed in the window PuTTY was installed on my computer so that I could remotely access PuTTY was installed on my computer so that I could remotely access NOAA s Linux machines through NGDC s portal. Through NGDC S portal I can connect to Process 1 and Process 2, two of the machines which has all the data that I access and download for processing

6 PuTTY

7 Arrangements made while in Boulder, Colorado Installing Secure FX Secure FX is a secure file transfer application with a visual interface Secure FX is used to exchange files between NGDC s machine and my computer

8 Secure FX

9 Arrangements made while in Boulder, Colorado Installing ArcGIS and ENVI For viewing, processing g, and analyzing data ArcGIS and ENVI were required ENVI is an image processing software package produced by the ITT Visual Information Solutions Premier software solution used for processing andanalyzinggeospatial analyzing geospatial imagery Combines spectral image processing and image analysis technology with a user friendly interface Interactive Data Language (IDL) is the scientific programming language associated with ENVI that lets users transform numbers into dynamic and meaningful visual representations

10 ArcGIS ArcGIS is a Geographic Information System (GIS) tool produced by the Environmental Systems Research Institute (ESRI) It is a powerful tool used for It is a powerful tool used for spatial analysis

11 Arrangements that had to be made in Delhi Setting up broadband internet connections at home in New Delhi, India Airtel speed test Airtel Broadband Service Airtel broadband connection provided by Bharti Airtel Based on Digital Subscriber Loop (DSL) technology Got the required telephone and modem to go along with it The monthly plan we took provides an internet speed of 512 kbps The plan also has restrictions on usage, after 8GB of data download the speed goes down to 256 kbps

12 Arrangements that had to be made in Delhi Other internet connection mobile broadband service provided by the Tata Teleservices Limited TATA Photon Plus is a small USB data card which when plugged into the USB port of my laptop would connect me to high speed broadband internet Internet speed up to 3.1 Mbps through TATA network The monthly plan we took allows free usage up to 5GB, after which the speed goes down to 256 kbps and.50 paisa is charged per extra MB of data download

13 Why two internet connections? Fluctuation in the consistency of internet service through the day, especially of Airtel Power outages cause interruption in file transfer, and then Photon is used instead of Airtel Tata photon enables quicker file transfer when needed Once in a while I do go over the cap in the amount of data download, and having two internet connections helps in balancing it out to some extent

14 Data Processing Processing monthly composites Processing rolling annual stable lights products Processing fixed gain products Socio economic analyses using the nighttime light images

15 Data Processing processing monthly composites Access the OLS nighttime sub orbits for each month (approx. 400 files) Personal computer NGDC s portal Work kflow NOAA s computer (Process 1 ) Cluster jobs submitted to create masked grids of only low moon nightfiles Approx. 300 corresponding low moon nightfiles each month NOAA s computer (through Process 1 to NGDC s portal) Corresponding flag data orbits Data are flagged on a pixel by pixel basis. Data included only if flag data bits are set as Daytime: off Nighttime marginal: off Zero lunar illuminance: on Clouds pese present: off Re project Run additional flags that need to be done in 30 arc second space (clouds) Crop the sub orbits to mid swath to generate the masked grids Secure FX 6 20 mins/nightfile, approx. 3 GB of zipped data Personal computer for line screening i

16 Data Processing processing monthly composite Flag bands For each nighttime suborbit, a companion flag band is generated with bit codes designating: daytime (solar elevation > 6) nighttime marginal ( 15 < solar elevation < 6) zero lunar illuminance (< lux) Solar elevation angles are computed based on lat, lon, and time of each OLS pixel. Lunar illuminance is a function of lunar phase, azimuth, and elevation, which are also based on lat, lon, and time of each OLS pixel. VIS FLAG This entire suborbit was flagged as having zero lunar illuminance. Nighttime portion of orbit F Red: daytime Green: nighttime marginal Black: This area is considered high quality nighttime data and will be processed further. Source: Baugh, et al. (2010). Development of a 2009 stable lights data using DMSP OLS data. Proceedings of the Asia Pacific Advanced Network. Hanoi, Vietnam.

17 Data Processing processing monthly composite Line screening Suborbits containing high quality nighttime data are screened by an analyst for aurora and abrupt gain changes. Analyst chooses a start and end line of data to include for compositing. Dt Data at edges of swath are discarded due to increased noise and poorer geolocation (scan angle > 40.91). VIS FLAG Red: daytime Green: nighttime ih i marginal Yellow: discarded by linescreening process Blue: edge of scan data Black: This area is considered high quality nighttime data and will be processed further. Source: Baugh, et al. (2010). Development of a 2009 stable lights data using DMSP OLS data. Proceedings of the Asia Pacific Advanced Network. Hanoi, Vietnam. Nighttime portion of orbit F

18 Data Processing processing monthly composite Re projecting OLS vis, tir and corresponding flag bands are gridded to 30 arc second grids, constrained to latitudes 65S 75N. For clarity, only mid swath, line screened data are shown. Green: nighttime marginal VIS TIR FLAG Nighttime portion of orbit F Source: Baugh, et al. (2010). Development of a 2009 stable lights data using DMSP OLS data. Proceedings of the Asia Pacific Advanced Network. Hanoi, Vietnam.

19 Data Processing processing monthly composite Cloud mask A cloud mask is generated by comparing the re projected OLS thermal band to a surface temperature grid provided by National Center for Environmental Prediction (NCEP) Difference images are made as Diff = Surface Temp TIR. Due to the increased variability in land temperature values, land and ocean regions are processed separately. tl Thresholds are computed from the difference images in latitudinal tiles as mean + N*stdev. Values greater than this threshold are flagged as clouds NCEP Surface Temp. TIR Diff (Land) Diff (Sea) Cloud Mask

20 Data Processing processing monthly composite Personal computer linescreen text file Secure FX Work flow NOAA s computer (through NGDC s portal to Process 1 ) Header files of the masked grids are updated dtd Linescreened and non linescreened nighttime suborbits submitted for averaging Converted to geotiffs Made available to customers Tiles of suite of products of which the most important ones are avg_vis Average visible band data values cf_cvg Number of cloud free observations used Histograms of input visible band data for each grid cell Average visible band data converted to NOAA s computer (through Process 1 to Secure FX : hrs per file, approx. 180 MB of zipped data bytes NGDC s portal) Personal computer for viewing

21 Example of monthly composites Monthlyline line screened composite of the world F18_ _ The most recent monthly line screened composite of the world F18_ _

22 Data Processing rolling annual stable lights Example The average visible band includes lights from fires, fishing boats, and other light sources To create stable lights products the transient light sources are removed The most recent rolling stable lights composite of India F18_ _

23 Data Processing rolling annual stable lights NOAA s computer Merging ofmonthly Work flow (through NGDC s portal to Process 1) composites, say from July of 2010 to June of 2011 To get samples of background values markers drawn over light free areas Is it ok? NOAA s computer (through Process 1 to NGDC s portal) Secure FX Personal computer NO Markers re drawn 100 MB data, 1.5 hrs Outlier removal: composite histograms analyzed for bright outliers, which are removed Background removal: separate areas in the outlier removed average that contain lights from those background areas where no lights are present. Areas with values greater than the maximum light free values are tallied as greater than background. Stable lights mask is created from areas considered gtb 40% of the time Shift the avg_vis, cf_cvg, and mask to the LandScan population grid Stable lights mask applied on raw avg_vis image to create the final stable lights product Creating geotiffs of raw avg_vis, cf_cvg, stable lights Personal computer for viewing Secure FX, 105 MB of zipped data, 1.5 hrs NOAA s computer (through Process 1 to NGDC s portal) Made available to customers

24 Data Processing rolling annual stable lights Outlierremoval removal process example Orissa coast, India raw average vis Orissa coast, India outliers removed from raw average vis

25 Data Processing rolling annual stable lights Background removal process example Light free areas chosen by an analyst in red Resulting Stable Lights mask Stable lights

26 Data Processing fixed gain composites DMSP sensor is typically operated at high gain setting for the detection of moonlit clouds Saturated din bih bright cores of urban centers because of six bit quantization i and limited dynamic range Every montha limited set of observations areobtainedat at low lunar illuminations where the detector is set significantly lower than its typical operational settings (sometimes by a factor of 100) These data are at present being collected by satellite F16, at fixed gains of 15, 35, and 50 decibels The fixed gain nightfiles have to be line screenedseparately separately for aurora and fixed gain File transferring, line screening, and processing are same as the nightfiles orbits of the operational data (Approx. 3 GB of nightfiles, hfl and 220 MB combines averages are transferred) Merging the stable lights and fixed gain product creates the unsaturated Merging the stable lights and fixed gain product creates the unsaturated, superior radiance calibrated images

27 Stable lights vs. Radiance calibrated image of Delhi 2004

28 Challenges Time difference between Boulder, Colorado (Mountain time), and New Delhi, India (Indian Standard Time, IST) 11.5 hours during the period of Daylight Savings Time (from March to November), and at a difference of 12.5 hours when the daylight saving time ends (December to February) Inter transference of files between NOAA s computer and my laptop is sometimes very time consuming. i I always have to transfer a file from NOAA s system to mine before I can view it Miss the personal interaction with colleagues, the round table conferences, don t always come to know what others in the group are working on

29 Thank You!! Questions? EOG website: noaa gov/dmsp/dmsp html s: Chris Elvidge: Kimberly Baugh: go Sharolyn Anderson: Tilottama Ghosh: g