A Neural Attention Model for Urban Air Quality Inference: Learning the Weights of Monitoring Stations

Transcription

1 The Thirty-Second AAAI Conference on Artificia Inteigence (AAAI-18) A Neura Attention Mode for Urban Air Quaity Inference: Learning the Weights of Monitoring Stations Weiyu Cheng, Yanyan Shen, Yanmin Zhu, Linpeng Huang Department of Computer Science and Engineering, Shanghai Jiao Tong University Emai: {weiyu cheng, shenyy, yzhu, phuang}@sjtu.edu.cn Abstract Urban air poution has attracted much attention these years for its adverse impacts on human heath. Whie monitoring stations have been estabished to coect poutant statistics, the number of stations is very imited due to the high cost. Thus, inferring fine-grained urban air quaity information is becoming an essentia issue for both government and peope. In this paper, we propose a generic neura approach, named ADAIN, for urban air quaity inference. We everage both the information from monitoring stations and urban data that are cosey reated to air quaity, incuding POIs, road networks and meteoroogy. ADAIN combines feedforward and recurrent neura networks for modeing static and sequentia features as we as capturing deep feature interactions effectivey. A nove attempt of ADAIN is an attention-based pooing ayer that automaticay earns the weights of features from different monitoring stations, to boost the performance. We conduct experiments on a rea-word air quaity dataset and our approach achieves the highest performance compared with various state-of-the-art soutions. Introduction Urban air poution is undoubtedy a severe probem in the word, responsibe for a growing number of heath effects. The acquisition of spatiay fine-grained urban air quaity information is of great importance for both government and urban peope to understand the probem and take necessary actions in time. Recent effort has been devoted to estabishing monitoring stations to coect air quaity statistics. However, due to the high monetary cost (about $200,000 per station), the number of avaiabe monitoring stations is very imited, e.g., Beijing has ony 36 monitoring stations in a tota area of 16, 410 km 2 (Center 2017). As a resut, it is becoming crucia to infer a arge amount of air quaity information in areas without monitoring stations. Existing approaches to inferring spatiay fine-grained air quaity information mainy fa into two categories: physica methods and data-driven approaches. Physica methods estimate air quaity in unmonitored ocations by simuating the compex physica dispersion process of air poutants based on observed data and severa empirica assumptions (Arystanbekova 2004; Kim, Park, and Kim 2012). However, the Corresponding author Copyright c 2018, Association for the Advancement of Artificia Inteigence ( A rights reserved. necessary data such as the distribution of a kinds of poution sources, accurate weather conditions (Godish, Davis, and Fu 2014) and specific street configurations (Kim, Park, and Kim 2012), are aways difficut to obtain in practice. Furthermore, some empirica assumptions may not refect rea scenarios accuratey, which degrades the mode performance. For exampe, the concentration of air poutants may not foow the Gaussian distribution as assumed in Gaussian Pume modes (Arystanbekova 2004). Data-driven approaches expoit the effects of the avaiabe spatio-tempora urban data on air quaity inference (Hasenfratz et a. 2014; Chen et a. 2016a; Zheng, Liu, and Hsieh 2013). Intuitivey, various factors from externa data sources such as POIs, and-use, traffic and meteoroogy in a particuar ocation, can be partiay or fuy acknowedged to its air quaity. By augmenting the imited statistics from monitoring stations with pentifu spatio-tempora data, data-driven approaches are generay more effective at capturing oca information that reates cosey to a ocation s air quaity, thus achieving better inference resuts than those physica methods. Typicay, data-driven approaches have to address two chaenging issues. The first chaenge is: how to incorporate auxiiary muti-source data with monitoring data? The recent work (Zheng, Liu, and Hsieh 2013) propose to train mutipe prediction modes with different feature sets and then conduct co-training to retrain each mode iterativey. However, deveoping modes separatey can hardy capture compex interactions between different features, and hence fais to make accurate inference. The second chaenge to be addressed is: how to differentiate the importance degrees of air quaity data from different monitoring stations? Since not a the monitoring data contribute equay to predicting the air quaity in a particuar ocation, existing methods adopt a random scheme (Zheng, Liu, and Hsieh 2013) or k-nearest neighbor strategy (Chen et a. 2016a) to seect a subset of monitoring stations and ony mode the effects of the seected monitoring station data for inference. Unfortunatey, the random seection scheme may cause the inconsistency probem (Chen et a. 2016a), whie the features from k nearest stations are unnecessariy the most effective and the significance of the same stations may rather vary with time (see detais in the Experiments Section). In this paper, we address the aforementioned probems 2151

2 by introducing a generic neura attention mode, named ADAIN (Attentiona Deep Air quaity Inference Network), for spatiay fine-grained urban air quaity inference. We expore the use of deep neura networks (DNNs) for: 1) modeing heterogeneous data (e.g., air quaity data, POIs, road networks, meteoroogica data) in a unified way, and 2) earning compex feature interactions without costy handcrafted feature engineering. In genera, ADAIN combines two kinds of neura networks: i.e., feedforward neura networks to mode static data and recurrent neura networks to mode sequentia data, foowed by hidden ayers to capture feature interactions. A nove attempt of ADAIN is to utiize the attention mechanism (Bahdanau, Cho, and Bengio 2014), to earn the importance degrees of monitoring stations for inferring air quaity in a particuar ocation automaticay. The importance degree of each monitoring station is incorporated in ADAIN to dynamicay re-weight the features from each station during prediction. We conduct experiments on rea-word air quaity data and the resuts demonstrate the superiority of our approach in inference performance. Furthermore, the earned weights of monitoring stations shed ight on the potentia behaviors of air poutant emissions and variations, which are vauabe for practitioners in addressing air quaity issues. Overview Definition and Probem Definition 1 (AQI and IAQI) The air quaity index (AQI) is widey used to measure air quaity. For a specific air poutant, its individua air quaity index (IAQI) in an area is measured by a monitoring station, refecting the rea-time concentration of the poutant. AQI is the highest IAQI vaues among a kinds of air poutants. We denote by Da t the set of IAQI vaues for a certain poutant in a city during time period t. Definition 2 (POI) A point of interest (POI) represents a specific ocation, with name, category, coordinates and severa auxiiary attributes. We denote by D p the set of a POIs in a city. Definition 3 (Road Network) A road network D r consists of a set of inked road segments in a city. Each road segment incudes coordinates of the start and end points, and is associated with a road type (e.g., motorway). Definition 4 (Meteoroogica Data) A meteoroogy dataset D m incudes district-eve meteoroogica records of a city. Let Dm t denote the rea-time meteoroogica information ike weather, temperature, pressure, humidity, wind speed and wind direction during time period t. In this paper, we aim to infer spatiay fine-grained urban air quaity based on the above heterogeneous data. Probem Statement. Consider a particuar air poutant. Given its IAQI data D a = {Da} t T t=1 from monitoring stations, POI data D p, road network D r and meteoroogica data D m = {Dm} t T t=1 of a city, we aim to predict IAQI vaue for any ocation without monitoring stations during time period T. Figure 1: Framework of our approach Since different poutants are typicay infuenced by the observed data differenty, we deveop an individua mode for each poutant. Note that AQI vaues can be easiy derived from IAQIs by choosing the maximum. Framework Figure 1 provides the framework of our proposed soution, which consists of two major components: offine earning and onine inference. Offine earning. We first extract features from heterogeneous data. The features are generay divided into two groups: static ones that are mosty time-invariant (e.g., features from POIs and road networks), and sequentia ones that vary with time (e.g., features from meteoroogy and monitoring data). To obtain training data, we deiberatey remove a monitoring station, and associate the features extracted from data in its affecting region (i.e., within certain distance) and data coected by the remaining monitoring stations with the ground-truth IAQI vaue as one training exampe. Our prediction mode ADAIN empoys feedforward neura networks (FNN) to hande static features, and uses recurrent neura networks (RNN) to absorb sequentia ones. The transformed features are further combined to earn a unified representation that modes feature interactions we. ADAIN incorporates the attention mechanism to discriminate the importance of features from different monitoring stations automaticay. Note that ADAIN serves two benefits: 1) it is generic to dea with new features that are either static or sequentia; 2) it aso provides possibe expanation on which monitoring data contributes more to the prediction. Whie the training data is imited by the number of monitoring stations, ADAIN sti outperforms the advanced semisupervised methods experimentay. Onine inference. The onine inference process tries to predict IAQI vaue for a target ocation in a given time period. To do this, we first extract features from the heterogeneous data observed in the affecting region of the target ocation. We then combine these features with the ones from rea-time monitoring data, and feed them to the trained

3 x r c = seg.ength (2) Figure 2: Structure of ADAIN mode mode, producing the inferred air quaity resut. Methodoogy Feature Extraction We first introduce the features used in this paper, which have been proved to be usefu in previous works (Xu and Zhu 2016; Zheng, Liu, and Hsieh 2013). Without oss of generaity, we focus on estimating air quaity for ocation and extract features from the data within s affecting region, i.e., within certain distance d. By defaut, d is set to 2 kiometers. Meteoroogica features X m. The concentrations of air poutants are easiy infuenced by meteoroogica factors. In this paper, we consider six meteoroogica features: weather, temperature, pressure, humidity, wind speed and wind direction. Among these features, weather and wind direction are categorica with 12 and 10 categories each, whie the others are numerica. We adopt one-hot encoding to represent weather and wind direction features. For numerica ones, we normaize their vaues to be in the range of [0, 1]. As meteoroogica data varies with time and ocations, we extract features for each region periodicay (e.g., every 1 hour). We denote by X mt the set of meteoroogica features during time period t and omit the region abe with the context is cear. POI features X p. Intuitivey, areas having many factories tend to have poor air quaity due to the emission of air poutants, whie those surrounded by pubic parks are more ikey to have fresh air. As POIs we capture the characteristics of ocations, we everage POI data for air quaity inference. We consider a set C p of 12 POI categories specified in (Zheng, Liu, and Hsieh 2013) and compute the number of each POI category within a region as one feature. Let X p = {x p c} c C p denote the POI features extracted for a ocation.wehave: x p c = { D p dist(, ) d.category = c} (1) Road Network features X r. The structure of road networks aso affects oca air quaity as vehices are known to be an important source of urban air poutants (Faiz et a. 1997). We divide a road segments in D r into three categories: C r ={highway, trunk, others}. To capture the intensiveness of road segments in different types, we measure the tota ength of road segments per category within a region as a feature x r c X r, c C r : seg S r c where S r c = {seg D r seg.category = c seg is overapped with s affecting region}. Monitoring features X d and X a. For each monitoring station s, we extract meteoroogica, POI and road network features X m s, X p s, X r s from data within s s affecting region. In addition, we provision each station s with reative position features X d s that records the distance and direction of s to the target ocation, and IAQI features X a s that contain a sequence of observed IAQI vaues in s over time. Proposed Mode Figure 2 provides the neura network structure of our ADAIN mode. The input ayer of ADAIN consists of two groups of input features: oca features X m X p Xr for ocation, and station-oriented features X m s X p s X r s X d s X a s for each station s. Reca that the output of ADAIN is the estimated IAQI vaue for ocation. In what foows, we introduce each ayer of ADAIN in detai. FNN-RNN Hybrid Layers. This ayer tries to identify atent features based on raw input features and mode feature interactions. Since some input features such as X m, X a are temporay reated, we propose to use recurrent neura networks (RNN) to encode these sequentia features. We observe the fact that the vaues of sequentia features often exhibit ong periodicity, e.g., temperature. However, traditiona RNN can hardy capture ong-term dependencies because of gradient vanishing and expoding probems (Hochreiter and Schmidhuber 1997). Hence, ADAIN empoys the Long Short-Term Memory (LSTM) (Graves 2013) to encode each of the sequentia features (i.e., X m and X a ), which everages the gate mechanism to address the ong-term dependency probem. The reguar LSTM contains memory ces c with sefconnections to store tempora states. Each memory ce is associated with input gate i, forget gate f and output gate o to contro the fow of sequentia information. Consider a sequence {X mt } T t=1 of meteoroogica features for exampe. The LSTM maps the input sequence to an output sequence by cacuating various unit activations using the foowing equations. i t = σ(w ixx mt + W ih h t 1 + W ic c t 1 + b i) f t = σ(w fx X mt + W fh h t 1 + W fc c t 1 + b f ) c t = f t c t 1 + i t tanh(w cxx mt + W ch h t 1 + b c) (3) o t = σ(w oxx mt + W oh h t 1 + W oc c t + b o) h t = o t tanh(c t ) where X mt and h t are one input eement and the corresponding memory ce output activation vector at time t, respectivey. The W terms denote weight matrices (e.g., W ix is the weight matrix from input gate to the input) and b terms are bias vectors. denotes the Hadamard product. σ represents the standard sigmoid function. i, f, o, c are

4 the activation vectors in the same size for input gate, forget gate, output gate and memory ce, respectivey. We conduct simiar operations over sequentia station-oriented features X m s, X a s. Specificay, we concatenate X mt s, X at s across a stations to obtain a bigger sequentia feature and feed it to LSTM as one input eement for time period t. As oca and station-oriented sequentia features are typicay extracted from different ocations, we treat them separatey using different LSTMs. For non-sequentia oca and station-oriented features (i.e., X p Xr and Xp s X r s), we simpy appy an individua stack of the fuy connected (FC) ayers to earn high-order interactions for each feature group. The definition of the FC ayers over non-sequentia features is as foows. z (n) = { φ(w (n) φ(w (n) (X p X r X d )+b (n) ), n =1 + b (n) ), 1 <n L z n 1 where can be for oca features or s for station-oriented features, and L is the number of basic FC ayer. φ is the activation function and we use the rectifier ReLU in this paper if not otherwise specified, which yieds good performance. Note that X d = when modeing oca features. In our design, we share the same set of network parameters among a stations to contro mode compexity and increase mode fexibiity when the number of stations changes. To capture interactions among sequentia and nonsequentia features, we further deveop FC ayers on top of the basic FC and LSTM ayers. Formay, the high-eve FC ayers transform the ast hidden state h T from LSTM and the output vector z (L) of basic FC ayers via the foowing operations. z (m) = { φ(w (m) φ(w (m) (z (L) z (m 1) h T )+b (m) ), m = L +1 + b (m) ), m [L +2,L+ L ] (5) where can be or s, and L denotes the number of higheve FC ayers. To summarize, the output of hybrid ayers contains atent oca features z (L+L ) and atent stationoriented features z (L+L ) s for each station. Attention Layer. Since not a monitoring data contributes equay to predicting air quaity in the target ocation, we propose to everage the attention mechanism (Bahdanau, Cho, and Bengio 2014) to our ADAIN mode that earns the importance of different station data automaticay. The attention mechanism has been incorporated into neura network modeing in various domains such as computer vision (Chen et a. 2016b), information retrieva (Xiong, Caan, and Liu 2017) and recommendation (Xiao et a. 2017). The key idea is to assign weights to different feature parts during prediction. In our context, we compute a weighted sum over atent station-oriented features from different stations as foows. (4) f A({z (L+L ) s } s S) = a sz (L+L ) s (6) s S where S denotes a monitoring stations and a s is the attention score for atent features z (L+L ) s of station s earned from the above hybrid ayers. Intuitivey, a s discriminates the importance of different station features to benefit mode prediction. Existing station seection schemes (e.g., random or k nearest) set vaues of a s to 0 or 1 based on certain rues. The resutant weight can hardy distinguish the significance among the seected station features where a s equas to 1. To address the probem, we parameterize the attention scores based on a muti-ayer perceptron (MLP), caed Attention- Net, as shown in Figure 2. The input to AttentionNet is the concatenation of both z (L+L ) (L+L and {z ) s } s S. It then encodes the interactions between oca features and the ones for station s to decide the attention score a s : a s = wsφ(w a(z (L+L ) z (L+L ) s )+b a)+b s exp(a a s = s) (7) s S exp(a s) where the matrix W a and the bias vector b a are mode parameters in the first ayer of MLP; vector w s and bias b s are second-ayer parameters. The ength of w s equas the size of hidden ayer in MLP. The attention scores are normaized via softmax such that they can be interpreted as the importance of different feature groups for prediction. Fusion Layer. Hybrid ayers and attention ayer mode the atent oca features z (L+L ) for ocation and high-eve station features f A ({z (L+L ) s } s S ), respectivey. It is intuitive to combine these features via concatenation and use a hidden ayer to earn high-order interactions. Hence, we deveop a fusion ayer above the hybrid and attention ayers, which is defined as foows. z f = φ(w f (z (L+L ) f A ({z (L+L ) s } s S )) + b f ) (8) where z f is the output vector of the fusion ayer, matrix W f and bias vector b f are mode parameters. Whie mutipe fusion ayers can be stacked together, we observe good performance based on a singe fusion ayer and omit further ayers to reduce mode parameters. Prediction Layer. At ast, the output vector of the fusion ayer z f is transformed to the fina prediction score, i.e., the estimated IAQI vaue for target ocation during time period T : ŷ = wpz f + b p (9) where w p is the neura weight vector for the prediction ayer, and b p is a bias scaar. Summary. It is worth mentioning that the structure of our ADAIN mode is generic to incorporate more features that are either static or sequentia. When the number of monitoring station increases, we may keep a subset of them and sti everage the attention ayer to determine the importance of the feature group for each seected station. Learning and Optimization The air quaity inference probem can be considered as a regression task or a cassification task (e.g., IAQI vaues are organized into categories). As numerica IAQI vaues are more accurate and vauabe, we treat the inference probem as a regression task and adopt the foowing squared oss as the objective function: L oss = X T (ŷ(x) y(x)) 2 (10) 2154

5 where T denotes the set of a training instances with ground-truth IAQI vaues y(x). Instead of using vania stochastic gradient descent (SGD) to optimize the objective function, we adopt Adam (Kingma and Ba 2014) as the optimizer. Based on adaptive estimates of ower-order moments, the Adam optimizer dynamicay tunes the earning rate during training process and eads to faster convergence. It is aso known to be computationay efficient with itte memory requirement. To prevent our mode from overfitting, we consider two widey used reguarization techniques: dropout and L 2 reguarization. The main idea of dropout is to randomy drop some neurons aong with their connections during training (Srivastava et a. 2014), which prevents units from too much co-adapting. In ADAIN, we empoy dropout on each hidden ayer. Besides, we appy L 2 reguarization on mode weights to prevent possibe overfitting. Formay, the actua objective function we optimize is: L oss = X T (ŷ(x) y(x)) 2 + λ W 2 (11) where λ is a hyperparameter to contro the reguarization strength and W denotes a weights in ADAIN. Experiments Experimenta Settings Datasets. To evauate the performance of our proposed approach, we use the foowing avaiabe heterogeneous data coected in Beijing, China. (1) Air quaity data: The air quaity data (Zheng et a. 2015) was coected by 36 monitoring stations in Beijing, from 2014/05/01 to 2015/04/30, with the coection time interva of 1 hour. Each record contains IAQI vaues for different air poutants observed by a station in an hour. We focus on predicting three important poutants PM2.5, PM10, NO 2. A IAQI vaues foow the Chinese AQI standard. (2) Meteoroogica data: The meteoroogica data (Zheng et a. 2015) consists of rea-time district-eve meteoroogica records. Each record contains weather, temperature, pressure, humidity, wind speed and direction in an area. (3) POI data: We query Map Word APIs (word 2017) and obtain about 151,000 POIs in Beijing. (4) Road network data: We downoad the road network for Beijing from OpenStreetMap (Openstreetmap 2017). The number of road segments is 65,991, with a tota ength of 27,889km. Settings and Compared Methods. We divide a the monitoring station data into the training and test sets with the proportion of 2:1. The separation is based on stations and is repeated randomy for 10 times, in order to avoid using historica air quaity data to infer current air quaity information for the same ocation. This is reasonabe as we prefer to predict air quaity of ocations without monitoring statistics. We aso seect 10% of training data as the vaidation set and aow training to be eary stopped according to the vaidation score. In our experiment, we construct a singe basic FC ayer (L=1) with 100 neurons and two LSTM ayers with 300 memory ces per ayer. We then buid two ayers of high-eve FC network (L =2) with 200 neurons per ayer. We initiaize a the mode parameters by samping from the uniform distribution between 0.1 and 0.1. We compare ADAIN with the foowing approaches. (1) k nearest neighbors (KNN): This method seects the k monitoring stations cosest to the inferred ocation, and compute the average IAQI vaue from these stations as resut. We set k to be 3 in our experiments. (2) Linear Interpoation (LI): This method cacuates the weighted average IAQI vaue based on data from a stations. The weight of a station s is inversey proportiona to its distance d s to the inferred ocation: ŷ = s S s.iaqi 1 i (3) Gaussian Interpoation (GI): This is another interpoation method based on a Gaussian distribution N(0,σ): 1 d s ŷ = 1 s.iaqi f(s), f(s) = 1 e d Z 2πσ s S d s 2 s σ 2 where σ is the average distance between two monitoring stations, and Z is the normaizing factor. (4) Gaussian Process Regression (GPR): GPR is a nonparametric Bayesian regression mode. We foow the formuation of GPR in (Cheng et a. 2014) and use the foowing kerne function: K(x i,x j,λ)=e λ x i x j 2 where λ is a hyperparameter and set to 0.01 by defaut. (5) Support Vector Regression (SVR): This is a cassica supervised regression mode extended from support vector machine. For station-oriented features, we ony consider the k (set to be 3) nearest stations as in (Chen et a. 2016a). (6) Feedforward Neura Networks (FNN): This method simpy fattens a the features and feeds them into a mutiayer feedforward neura network. For sequentia features, we ony use their atest vaues. The network contains three hidden ayers, with 200 ces at each ayer. We adopted dropout and L2 reguarization to reduce overfitting. (7) Support Vector Machine (SVM): This is a cassification mode that absorbs the same inputs as the support vector regression mode, whie outputs categorica IAQI eves for the target ocation. We consider 6 IAQI vaues introduced in (Zheng, Liu, and Hsieh 2013). (8) U-Air (Zheng, Liu, and Hsieh 2013): U-Air is a cotraining based cassification mode. It trains two cassifiers using data from different views, and improves the performance of the two cassifiers iterativey. This mode produces an inferred IAQI eve by combining the outputs from both cassifiers. Metrics. We use the root mean squared error (RMSE) to measure the performance of various regression approaches that infer IAQI vaues: N i=1 RMSE = (ŷ(x i) y(x i )) 2 (12) N where N is the number of instances in the test set. To compare with cassification methods (i.e., SVM and U- air) that produce discrete IAQI eves, we convert the output 2155

6 Figure 3: ADAIN v.s. competing regression methods of our method into the corresponding IAQI eves and adopt accuracy as the measurement, which is defined as foows: {X TestSet ŷ(x) =y(x)} Accuracy = (13) N where the numerator denotes the number of correct estimations for the test cases. Resuts We first compare our method with aforementioned baseines. We then evauate the effectiveness of different features. Finay, we discuss the benefits of our attention mode and provide quaitative visuaization resuts to expain it. Comparison Resuts. Figure 3 shows the performance of ADAIN and six regression methods, using RMSE metric. ADAIN produces the owest RMSE vaues for predicting a three air poutants. FNN provides the second best performance. The reason may be the abiity of hidden ayers in FNN that we mode the feature interactions. However, on average, the reative improvement of ADAIN over FNN is sti 20%. LI and GI perform worse than other methods on a three poutants. This indicates that the affecting degree of air quaity in areas with monitoring stations can hardy be quantified using a function of distance. Furthermore, we observe that the resuts on PM2.5 and PM10 are worse than those on NO 2 for a methods. This is reasonabe because the concentration of NO 2 is more stabe in different ocations over time, compared wit PM2.5 and PM10. Next, we compare our approach with competing cassification methods, SVM and U-air. To do this, we convert the outputs of our mode into IAQI eves accordingy. Figure 4 provides the comparison resuts. It can be seen that ADAIN achieves the highest accuracy than the other two modes on a three poutants. On average, the reative improvements against U-Air and SVM are 10% and 28%, respectivey. The advantages of ADAIN over U-Air coud be expained in two aspects. First, U-Air trains two separate cassification modes for spatia features and tempora features, respectivey. Such separation may fai to capture feature interactions, thus degrading the prediction performance. Second, U-Air adopts random scheme when extracting features from monitoring data. It is very ikey that informative features are eiminated due to the randomness. In contrast, ADAIN everages the attention mode to discriminate the importance of monitoring features from different stations effectivey. The benefits of the attention wi be described in the ate part of this section. Figure 4: ADAIN v.s. competing cassification methods Tabe 1: Effects of various features in ADAIN Features PM2.5 PM10 NO 2 X a + X d X a + X d + X p + X r X a + X d + X m X a + X d + X m + X p + X r Effects of Different Features. To evauate the effectiveness of different features in ADAIN, we manuay remove some features and compute the prediction error based on the remaining features. Tabe 1 provides RMSE vaues of our mode using different features. It is easy to see that incorporating more features improves prediction performance significanty and consistenty over a three poutants. The ast row with a features achieves the owest RMSE vaues. The first row provides the worst prediction resuts based on station-oriented features ony. The second row incorporates non-sequentia features from POI and road network data, whie the third row everages sequentia features from meteoroogica data. Sequentia meteoroogica features are more beneficia to air quaity inference, which foows our intuition that meteoroogica factors are highy correated with the concentration of air poutants. Effects of Attention Mode. We now study the advantages of our attention mode in seecting usefu information from monitoring stations for prediction. To do this, we consider two variants of our mode. Instead of using attention-based pooing, the two variants empoy average pooing on the features of a stations and k nearest stations (we set k to be 3 by defaut), respectivey. Tabe 2 shows the RMSE vaues using different pooing methods. Our attention-based pooing approach achieves the owest RMSE vaues in predicting a three poutants. The average pooing over a stations provides the worst performance, which ignores the reative importances of features from different stations. In particuar, the averaged feature vaues may easiy cance out important information and introduce noise instead. Averagek-nearest pooing outperforms average-a pooing by producing ower RMSE vaues. This is because the features from areas in coser distances are intuitivey more informative for inferring oca air quaity vaues. However, averagek-nearest pooing discriminates the importances of different stations based on distance factor ony and hence resuts in inferior performance than our attention-based pooing method. Attention Visuaization. To better understand the effects of our attention mode, we visuaize the attention scores

7 Tabe 2: Effects of different pooing methods Pooing methods PM2.5 PM10 NO 2 Average-a pooing Average-k-nearest pooing Attention-based pooing (a) Location 1 at t 1 (c) Location 2 at t 1 target top stations other stations target top stations other stations target top stations other stations (b) Location 1 at t 2 target top stations other stations (d) Location 2 at t 2 Figure 5: Attention visuaization of monitoring stations in Figure 5, where the shapes represent monitoring stations and the coors refect their attention scores. We choose two target ocations 1 and 2 and try to infer their air quaity during two time periods t 1 and t 2. From the resuts, we can have the foowing observations. First, our attention mode is abe to dynamicay identify important station data for prediction. The earned importance of features from a monitoring station can vary by ocation (ocation 1 vs 2) and time (t 1 vs t 2 ), mainy because of the chaotic air poutants and unpredictabe externa incidents. Second, distance is a critica factor that determines the importance of monitoring stations, but not the ony factor. We use diamonds to highight the top-3 stations with highest attention scores. It can be seen from the coors that stations far away from the target ocation may have higher attention scores than those near the target ocation. For exampe, consider the top-3 stations for ocation 2 in Figure 5c. Third, the distribution of attention scores changes with the target ocation. For ocation 1 surrounded by many monitoring stations, major attention weights are more uniformy assigned to its nearby stations; for ocation 2 in remote area, high attention scores are concentrated on 1-2 nearby stations. This further verifies that the informative features from monitoring stations are typicay dependent on the particuar target ocation Reated Work There are two different ways predicting spatiay finegrained urban air quaity. One way is based on cassica emission modes, incuding Gaussian Pume modes (Arystanbekova 2004; Godish, Davis, and Fu 2014), Street Canyon modes (Kim, Park, and Kim 2012) and Computationa Fuid Dynamics (Scaar et a. 2012). These modes simuate the dispersion of air poutants based on a number of empirica assumptions and parameters. As some empirica assumptions may not correspond to the rea situations and the required parameters such as emission density, street geometry and dispersion parameters are hard to get precisey, the prediction resuts are far from satisfactory (Zheng, Liu, and Hsieh 2013). The other is based on statistica modes, such as inear regression, matrix factorization and neura networks for air quaity inference (Shad et a. 2009; Hasenfratz et a. 2014; Xu and Zhu 2016). However, many of these modes rey on oca features from the target ocation for prediction, without taking care of the spatia-tempora correations of air poutants between adjacent areas. There are severa air quaity inference modes that take such dependencies in account. For exampe, Zheng et a. (Zheng, Liu, and Hsieh 2013) and Chen et a. (Chen et a. 2016a) proposed semi-supervised based methods to estimate finegrained air quaity. They use random scheme or k-nearest neighbors to seect nearby areas with monitoring stations to mode spatia dependencies of air poutants. However, the random scheme resuts in the inconsistency probem (Chen et a. 2016a) whie k-nearest method uses the distance of station-oriented features to discriminate the importances of different stations. Different from these works, we empoy the attention mechanism (Bahdanau, Cho, and Bengio 2014) to assign weights to each group of station-oriented features automaticay, without human intervention. Moreover, our proposed framework is generic and fexibe to incorporate more features to improve performance further. Recenty, many researches have deveoped deep earning based approaches to chaenging tasks in urban computing (Zheng et a. 2014). For exampe, Zhang et a. proposed DNN-based prediction mode to predict citywide crowd fows (Zhang, Zheng, and Qi 2016). Liang et a. utiized recurrent neura networks to predict metro density (Liang et a. 2016). Xing et a. and Grover et a. empoyed deep modes for weather forecasting (Xingjian et a. 2015; Grover, Kapoor, and Horvitz 2015). Song et a. and Chen et a. appied deep neura networks to urban transportation systems (Song, Kanasugi, and Shibasaki 2016; Chen et a. 2016c). However, none of them concerns the probem of inferring spatiay fine-grained urban air quaity, which is the focus of this paper. Concusion In this paper, we propose a generic neura attention mode based on deep neura networks for urban air quaity inference. We everage both records from monitoring stations and various urban data (e.g., meteoroogy, road networks, POIs), and extract important features that are correated with air quaity. We mode static and sequentia features using different neura structures and incorporate the attention mechanism to discriminate the importance of features from different stations automaticay, to boost the performance. The experimenta resuts on a rea dataset verify the superiority 2157

8 of our approach against compared methods. Acknowedgment We thank anonymous reviewers for their insightfu and hepfu comments, which improve the paper. This research is supported in part by 973 Program (no. 2014CB340303), NSFC (no , , , and ), Singapore NRF ( CREATE E2S2 ), and 863 Program (no. 2015AA015303). This work is aso supported by the Program for Changjiang Young Schoars in University of China, and the Program for Shanghai Top Young Taents. References Arystanbekova, N. K Appication of gaussian pume modes for air poution simuation at instantaneous emissions. Mathematics and Computers in Simuation 67(4): Bahdanau, D.; Cho, K.; and Bengio, Y Neura machine transation by jointy earning to aign and transate. arxiv preprint arxiv: Center, B. M. E. M Chen, L.; Cai, Y.; Ding, Y.; Lv, M.; Yuan, C.; and Chen, G. 2016a. Spatiay fine-grained urban air quaity estimation using ensembe semi-supervised earning and pruning. In UbiComp, ACM. Chen, L.; Zhang, H.; Xiao, J.; Nie, L.; Shao, J.; and Chua, T.-S. 2016b. Sca-cnn: Spatia and channe-wise attention in convoutiona networks for image captioning. arxiv preprint arxiv: Chen, Q.; Song, X.; Yamada, H.; and Shibasaki, R. 2016c. Learning deep representation from big and heterogeneous data for traffic accident inference. In AAAI, Cheng, Y.; Li, X.; Li, Z.; Jiang, S.; and Jiang, X Fine-grained air quaity monitoring based on gaussian process regression. In ICONIP, Springer. Faiz, A.; Weaver, C. S.; Wash, M.; Gautam, S.; and Chan, L Air poution from motor vehices: Standards and technoogies for controing emissions. Technica report, Word Bank Group, Washington, DC (United States). Godish, T.; Davis, W. T.; and Fu, J. S Air quaity. CRC Press. Graves, A Generating sequences with recurrent neura networks. arxiv preprint arxiv: Grover, A.; Kapoor, A.; and Horvitz, E A deep hybrid mode for weather forecasting. In KDD, ACM. Hasenfratz, D.; Saukh, O.; Waser, C.; Huegin, C.; Fierz, M.; and Thiee, L Pushing the spatio-tempora resoution imit of urban air poution maps. In PerCom, IEEE. Hochreiter, S., and Schmidhuber, J Long short-term memory. Neura computation 9(8): Kim, M. J.; Park, R. J.; and Kim, J.-J Urban air quaity modeing with fu o 3 nox voc chemistry: Impications for o 3 and pm air quaity in a street canyon. Atmospheric Environment 47: Kingma, D., and Ba, J Adam: A method for stochastic optimization. arxiv preprint arxiv: Liang, V. C.; Ma, R. T.; Ng, W. S.; Wang, L.; Winsett, M.; Wu, H.; Ying, S.; and Zhang, Z Mercury: Metro density prediction with recurrent neura network on streaming cdr data. In ICDE, IEEE. Openstreetmap Scaar, H.; Teodorov, T.; Zieger, T.; and Memann, J Computationa fuid dynamics anaysis of air fow uniformity in a fixed-bed dryer for medicina pants. In Internationa Symposium on CFD Appications in Agricuture, Shad, R.; Mesgari, M. S.; Shad, A.; et a Predicting air poution using fuzzy genetic inear membership kriging in gis. Computers, Environment and Urban Systems 33(6): Song, X.; Kanasugi, H.; and Shibasaki, R Deeptransport: Prediction and simuation of human mobiity and transportation mode at a citywide eve. IJCAI. Srivastava, N.; Hinton, G. E.; Krizhevsky, A.; Sutskever, I.; and Saakhutdinov, R Dropout: a simpe way to prevent neura networks from overfitting. Journa of machine earning research 15(1): word, M Xiao, J.; Ye, H.; He, X.; Zhang, H.; Wu, F.; and Chua, T.- S Attentiona factorization machines: Learning the weight of feature interactions via attention networks. arxiv preprint arxiv: Xingjian, S.; Chen, Z.; Wang, H.; Yeung, D.-Y.; Wong, W.- K.; and Woo, W.-c Convoutiona stm network: A machine earning approach for precipitation nowcasting. In NIPS, Xiong, C.; Caan, J.; and Liu, T.-Y Learning to attend and to rank with word-entity duets. SIGIR. Xu, Y., and Zhu, Y When remote sensing data meet ubiquitous urban data: Fine-grained air quaity inference. In Big Data, IEEE. Zhang, J.; Zheng, Y.; and Qi, D Deep spatiotempora residua networks for citywide crowd fows prediction. arxiv preprint arxiv: Zheng, Y.; Capra, L.; Wofson, O.; and Yang, H Urban computing: concepts, methodoogies, and appications. Transactions on Inteigent Systems and Technoogy 5(3):38. Zheng, Y.; Yi, X.; Li, M.; Li, R.; Shan, Z.; Chang, E.; and Li, T Forecasting fine-grained air quaity based on big data. In KDD, ACM. Zheng, Y.; Liu, F.; and Hsieh, H.-P U-air: When urban air quaity inference meets big data. In KDD, ACM. 2158