Recent Frequent Item Mining Algorithm in a Data Stream Based on Flexible Counter Windows

Transcription

1 58 JOURNAL OF SOFTWARE, VOL. 9, NO., JANUARY 04 Recent Frequent Ite Mnng Algorth n a Data Strea Based on Flexble ounter Wndows Yanyang Guo School of Inforaton Engneerng, Yangzhou Polytechnc ollege, Yangzhou,hna gyy97966@63.co Gang Wang,a, Fenge Hou,b, Qnglng Me,c School of Inforaton Engneerng, Yangzhou Polytechnc ollege, Yangzhou,hna Departent of oputer Scence, Yangzhou Unversty, Yangzhou,hna a yzwg00@63.co, b yzhfs@63.co, c ql859@63.co Abstract In the paper the author ntroduces FW_MRFI, whch s a streang data frequent te nng algorth based on varable wndow. The FW_MRFI algorth can ne frequent te n any wndow of recent streang data, whose gven length s L. Meanwhle, t dvdes recent streang data nto several wndows of varable length accordng to, whch s the nuber of the counter array. Ths algorth can acheve saller query error n recent wndows, and can nze the axu query error n the whole recent streang data. Index Ters streang data, counter array, data nng, ost recent frequent te I. INTRODUTION Although there are any algorths concernng of frequent te nng n streang data [,3,5], any of the don t put ephass on current data. The exstng researches of frequent te nng n the ost recent streang data are anly algorths based on slde wndow technology. L. Golab et al. ntroduced an algorth based on hoppng wndows [], whch requres a specfed a threshold /. Recently, they ntroduced several algorths utlzng slde wndow odel. Lee and Tng [7] put forward an algorth, whch can realze space coplexty O(ε ), and processng te of updatng and queryng O( ε ). L. Zhang and Y. Guan [6] proposed an Estate of streang data frequent value based on slde wndow, whch requres a eory space O(ε ), and the processng and queryng te of each data te O( ε ). H.T.La,T.alders [8] presented to ne the frst K axu frequent te n slde wndows wth dynac-hange lengths. I.T.Ferry et al. proposed an algorth whch dvdes the ost recent streang data based on te-nclned ethod [4]. However, ths algorth deands that the nuber of counter array ust be equal to the nuber of wndows dvded, whch s not applcable when the nuber of counter array s already gven. In practcal applcaton, t s requred that the queryng error of recent data be relatvely saller, whle errors produced by ost exstng algorths are all the sae for data at all tes [9,0,]. Ang at ths proble, FW_MRFI, whch s a streang data frequent te nng algorth based on varable wndow, s ntroduced n ths paper. The FW_MRFI algorth can ne frequent te n any wndow of recent streang data, whose gven length s L. Meanwhle, t dvdes recent streang data nto several wndows of varable length accordng to, whch s the nuber of the counter array. Ths algorth can acheve saller query error n recent wndows, and can nze the axu query error n the whole recent streang data. In order to copare ts accuracy and recall rate wth other exstng ethods, experents wth real data sets and synthetc data sets are conducted, whch shows that FW_MRFI algorth offers uch proved accuracy n data strea recent frequent te nng. II. DEFINITION If the axu length of the ost recent streang data allowed to be quered s L, and current te s t, then the frequent data te n any wndow durng the perod fro t-l to t can be quered. If the partcular wndow to be quered s w = [ w, w n ax ] (as llustrated n Fgure ), n whch w n refers to the farthest pont, whle w ax refers to the nearest one, then t can be seen fro Fgure that the query wndow w should satsfy: t w n L, to wt, t L wn < wax < t. W t-l t W L W Fgure recent streang data of length L If the partcular wndow to be quered s w = [ w, w n ax ], then the length of query wndow w s w = w ax w n. If s a constant defned by the syste between [0,], and f the support of a partcular x n w s equal to or larger than φ w (), then x s the frequent te 04 AADEMY PUBLISHER do:0.4304/sw

2 JOURNAL OF SOFTWARE, VOL. 9, NO., JANUARY of the wndow. We are supposed to query the frequent te desgnated by the user n the ost recent streang data. If there are counter unts avalable n the syste, we are supposed to perfor frequent te query aganst any wndow n the ost recent streang data of length L usng only counter unts, and at the sae te nze the axu query error. For data of dfferent te, query errors of those nearer to the current te are relatvely saller. So, the streang data are dvded nto several te spans, whch are called basc wndows. Statstc nforaton of te nuber n each basc wndow s stored n an array of counter unts. Here Hash functon s adopted to get approxate nubers, that s to say, set up a Hash functon H wth h.count, and set up H(x) as the counter of each data te x. If the value range of H(x) s [, h], then there are h unts. For a partcular wndow w to be quered, soe wndow span overlappng wth the quered wndow w can be chosen to acheve nu error between the length of w'whch are coposed of these spans and the length of w whch are coposed of the quered wndows. A sple way for ths proble s to averagely allocate the ost recent streang data L nto parts, wth each part of fxed length L/. Wth ths way, errors between the quered wndow w and the chosen wndow span w'are less than L/. However, ths ethod sets very hgh deands for space coplexty. Moreover, t treats all data tes n recent streang data equally, whch results n uch loss of the newer data nforaton. However, the newer data nforaton s often coonly eployed and tends to carry ore valuable nforaton than hstorcal data. Another way s to dvde recent streang data wth the length of ndvdual wndow span,,4, l-. The tlted te frae ethod can estate the frequency of recent data te ore accurately, and decrease the accuracy of the hstorcal data gradually. However, wth ths ethod, the error between w and w'can reach L/, and t deands the nuber of counter array be equal to that of dvded wndow span, to wt, l = log L =, whch s not applcable when the value of counter array s gve n. To counter the probles occurred n the two ethods, a coprose, FW_MRFI, s proposed n ths paper. FW_MRFI tres to preserve as uch newer data nforaton as possble, and at the sae te nze axu and enseble error. III. MAIN IDEA OF FW_MRFI ALGORITHM 3. If log L If the length of slde wndow s L, the nuber of counter array defned by the syste s, and the sze of basc wndow s b= L, then the nuber basc wndow l n current slde wndow s L =L/b=. The ost recent streang data n slde wndow s dvded nto spans by logarthc te-nclned of length L, and the length of these spans are respectvely b, b, b, 4b,, -b, -b. The frst wndow stores b data te fro the basc wndow that coes frst, and the length of the followng wndow s twce that of the forer clp. Generally, f the sze of the frst wndow s represented by w0, then, and the th wndow followng t s w = b, 0 < <. One array of counter s adopted to counter the ndvdual data te n each span, and ar the n turn as 0,,, 3, 4,. A buffer array, whch s ared as -, s set up by the syste to receve streang data. Hcount s adopted to estate the countng of the ndvdual data te n the ost recent basc wndows. The countng s added to counter array -, and the orgnal value of - s transferred to 0. The counter array correspondent to the wndow of length s ared as, (=0,,, -), and a counter array - s set up to receve the latest data. The counter array - should transfer ts data n order to receve the latest data every te when t has receved b data te. Fgure s taen as a sple exaple to llustrate how the counter array conducts hcount calculaton and transfer operaton. In fgure, b=, =5, the counter arrays correspondent to the ndvdual wndows are -, 0,,, 3, 4 :, aong whch - s to counter the latest data te. te Fgure transfer operaton f countng wndows AADEMY PUBLISHER

3 60 JOURNAL OF SOFTWARE, VOL. 9, NO., JANUARY 04 Fgure shows that counter array - always counter the latest data tes, and every te when t has receved b data te t perfor data-transfer. Apart fro transferrng the values n - to 0, t adds the values of forer several counter arrays and transfers the to the followng counter arrays. The te of transferrng dffers n dfferent te t. For exaple, when t=, t transfers tes to the followng counter array, when t=3, t transfers te, and when t=4, t does 3 tes. Generally, G(t) s ared as the locaton of (the low-order bt s bt0), whch s the lowest n bnary representaton of nteger t. The tes of transferrng to be conducted by counter arrays followng - s dentfed by G(t). At t, the tes of transferrng to be conducted s G(t)+. For exaple: f =(0000), then G()=, to wt, when t=, the tes of transferrng s. Slarly, as G(3)=0 G(4)=, when t=3, the tes to be transferred should be, and when t=4, the tes should be 3. After sung and transferrng, the correspondent counter arrays should refresh ther records. To su up, FW_MRFI s gven below to llustrate the refresh procedure of ndvdual counter array. As there are h unts n each counter array, [ ] s adopted to represent th unt n, to wt, the counter of data whose Hash value s. In the followng algorth, + represents the addng up of values of the correspondng unts of and. For exaple, = +, represent [] = [] + [] ( =,,..., h) Algorth FW_MRFI( log L = l ) Begn. t=;. receve and for hcount of the frst two wndows and save to -,, 0, = ; 3. Whle not end condton do 4. t=t+; 0 = - ; 5. for ount for newer group and store to - ; tep= - ; 6. q=n(g(t),) /* s the nuber of counter array, G(t) the locaton of, whch s the lowest n bnary representaton of nteger t */ 7. for = to q do 8. tep= ; 9. = - +tep; tep=tep; 0. end for ;. end Whle ; End The counter array produced by the above algorth can cover all data tes n current wndows, and for the newer data tes, whch can offer hgher accuracy. If n =4, L=6, fgure 3 shows the range of the ndvdual counter array when t=5, 6, 3, 3. As seen fro above, t=5, the range s [, 5]. As the axu range of counter array s [, 8], and the length s 8, the axu query error s 4, whle n range [4, 5], the query error s 0. Generally, at, the query error of range [t-, t] s 0, and that of [t- +,t] s -. One ore exaple, when t=6, the range s [,6]. As ranges of [,8] can be acheved by subtractng countng value of [9,6] fro that of [,6], the axu range s [,8] or [9,6], and the length of both s 8, the axu query error s 4. When t=3, the axu range of counter n the quered wndows s [7,4], and the length s 8, the axu query error s4. When t=3, the axu range of counter n the quered wndows s [7,4] or [5-3], and the length of both s 8, the axu query error s4. Generally, f t od 6=g, then the range at t s [ t 6 g +, t], [ t 6 g +, t]. The axu query error of counters wthn the quered wndows s l/, and nu query error s 0. Furtherore, for those older data ranges w = ( 6 + g), the query errors are greater, whle for those newer data ranges, the query errors are saller. 3. If > log L When > log L, the recent streang data L s dvded nto l wndows of length,, 4, 8,, l-. Let '=-logl, ar wndows of length as S, and ar the correspondng counter array as. As for those ' unused counter arrays, the length of the s dvded accordng to the followng prncple: frst, fro S l-, delete S l-, add two S l-, and subtract fro the value of. If 's not 0, delete one ore S l-, and add two S l-3. And f 's stll not 0, delete one ore S l-, add two S l-3. If all S l- are deleted, 's stll not 0, delete one S l-3, subtract fro '. The rest follows the sae tend tll 's 0. Mar the nuber of wndows S as T. Accordng to the ethod stated above, for gven L and, the axu nuber of wndow of wndow S whch enables T 0 s defned by the followng rule: Mar: l = log L : frst calculate l + D = ( l + 4) () Then the axu nuber of wndow s () = arg ax( D ') T, the nuber of wndow S s defned by 0 > D + ' + = T = l ( T + ) = For exaple, f L=04,=0, l=0, then '=0. Fro (), by calculaton, D 0 =0, D 9 =3, D 8 =0, D 7 =5, D 6 =56.Fro (), by calculaton, =8-=7. Stll, fro (3), by calculaton: T 7 =0-0+=,T 6 =( 0-7 -)-=3, T 0 =T = T = T 3 = T 4 =T 5 = Thus, the total nuber of counter array needed s T 7 + T 6 +T T + T 0 =+3+6=0, whch s exactly equal to the gven nuber of counter array. Therefore, a concluson can be drawn as follows: (3) 04 AADEMY PUBLISHER

4 JOURNAL OF SOFTWARE, VOL. 9, NO., JANUARY 04 6 Theore : wth the wndow-arrangng ethod entoned above, the nuber of wndows arranged s exactly the sae as the gven nuber of counter array. Prove : f T 0 starts at th layer Total nuber of wndows =T+T-+...+T+T0 = = l T = ( T ) l + + T +(-) l + ( ) = ' l + l + ( ) = + l ' + = l ' 3 + = l + ' = proven Theore : wth the wndow-arrangng ethod entoned above, certan gven counter array can cover the query of the ost recent steang data of length L-. Prove : f T 0 starts at th layer Total coverage length = T. + T l = T. + [ ( T ). ] + l + + = T. + T + l = = L proven If the buffer array - s ncluded, t can cover the query of the ost recent streang data wth length L entrely. As t s wthn the query range, the axu coverage range s. Therefore, t s not dffcult to draw a concluson as follows: Theore 3: The wndow-arrangng ethod entoned above s a schee that can eploy as any as wndows to query the ost recent streang data of coverage length L, and can guarantee the error less than -. Aong whch: l + [( log L) ( ( l + 4) )] = arg ax Thus t can be seen that the axu query error of the counter wthn query wndows s -, and the nu error s 0. Errors are greater for older data spans, and saller for newer ones. Accordng to the wndow-arrangng ethod entoned above, counter array can be eployed to counter streang data of each ndvdual wndow correspondngly. Each te when a data coes, counter array - counters t, and then transfers t. Apart fro transferrng the values n - to 0, t adds the values of forer several counter arrays and transfers the to the followng counter arrays. The te of transferrng dffers n dfferent te t. For exaple, f the length of the ost recent streang data query wndow L=3, thenl = log L = 5, and the nuber of gven counter array =7, then ' =. Accordng to the wndow-arrangng ethod entoned above, the nuber of each ndvdual wndow should be T - =,T 0 =,T =,T =3,T 3 =. To su up, FW_MRFI s gven below to llustrate the refresh procedure of ndvdual counter array. Algorth : FW_MRFI( > log L = l ) Begn. t=;. receve and for hcount of the frst two wndows and save to -,, 0, = ; 3. Whle not end condton do 4. t=t+; 0 = - ; 5. for ount for newer group and store to - ; tep= - ; 6. q=n(g(t),) /* s the nuber of counter array, G(t) the locaton of, whch s the lowest n bnary representaton of nteger t */ 7. for = to q do 8. tep= ; =tep; tep=tep; 9. for = to T do 9. tep= tep=tep; 0. endfor. T tep=tep+ 0. end for ;. end Whle ; End ; IV. EXPERIMENTAL INVESTIGATION =tep; Experents wth real data sets and synthetc data sets are conducted to easure FW_MRFI algorth, and copare ts perforance wth TTount algorth whch adopts tlted te frae ethod. All experents were operated on Ps wth 5M eory,.7g PU, WINDOWS XP operatng syste, and prograed usng python.6. In experents, paraeters are set as:,b=000. φ = Synthetc Data To easure ths algorth, 6 groups of data set satsfyng Zpf dstrbuton are created randoly, wth paraeters of each Zpf Dstrbuton group beng 0.5, 0.75,,.5,.5 and.75. The sze of the data set s 000. When dfferent nuber of counter array s gven, recall rate and accuracy of TTount and algorth ntroduced n ths paper are easured by query wndows created randoly. Meanwhle, recall rate and accuracy of the two algorths of dfferent Zpf dstrbutons are copared when b=000, φ = Fgure 5(a) llustrates the coparson of recall rate of the two algorths when φ =0.005 and Zpf dstrbuton paraeter s.5. It can be seen fro Fgure 5 that recall rate of both algorths can reach 00% for data sets of relatvely stable dstrbuton. Fgure 3(b) llustrates the coparson of accuracy of 04 AADEMY PUBLISHER

5 6 JOURNAL OF SOFTWARE, VOL. 9, NO., JANUARY 04 the two algorths when values of vary. For, as the value of ncreases, the bgger countng wndow eep subdvdng nto saller ones, thus guarantee ore accurate frequency countng. However, the nuber of countng array of TTount s stable. Therefore, as the value of ncreases, FW_MRFI s ore accurate than FW_MRFI. Accuracy TTount (a) oparson of TTount (b) oparson of Accuracy Fgure 3 oparson of Two Algorth on and Accuracy when the value of vares When =35, recall rate coparson for data sets of varous Zp dstrbuton paraeters s llustrated n Fgure 4(a). Recall rate of FW_MRFI alost reach 00%, whle TTount can t reach 00% as ts errors are greater n queryng and countng wndows. When =35, nng accuracy coparson for data sets of varous Zp dstrbuton paraeters s llustrated n Fgure 4(b). As can be seen fro the fgure, for data sets of varous Zp dstrbuton paraeters, FW_MRFI s ore accurate than TTount TTount zpf (a) oparson TTount zpf (b) Accuracy oparson Fgure 4 recall rate and accuracy coparson for data sets of varous Zp dstrbuton paraeters(φ =0.05,=35) 4. Real Data In real data experents, data set, osara [8],s adopted ( The data set s coposed of anonyous clc strea of a Hungary onlne news gateway webste, whch contans about 800 llon separate data tes. 90 groups of query wndows created randoly are adopted to copare ther recall rate and accuracy. Fgure 5(a) llustrates the coparson of recall rate when the value of vares. As seen fro Fgure 5(a), recall rate of FW_MRFI s hgher than that of TTount. Fgure 5(b) llustrates the coparson of accuracy when the value of vares. Obvously, as the value of ncreases, FW_MRFI s ore accurate than that of TTount TTount (a) oparson 04 AADEMY PUBLISHER

6 JOURNAL OF SOFTWARE, VOL. 9, NO., JANUARY Accuracy TTount (b) Accuracy oparson Fgure 5 and Accuracy oparson when the value of vares (osara data set,φ =0.05) Fgure 6(a) llustrates the recall rate coparson of two algorths wth dfferent query wdth. As seen fro the fgure, recall rate of FW_MRFI s hgher than that of t. Fgure 6(b) llustrate the accuracy coparson of the two algorths. Results n the fgure shows that FW_MRFI s ore accurate than t. Accuracy rate of FW_MRFI are all over 95%, even 00%, whch proves better perforance of FW_MRFI. Accuracy TTount Query wdth(k) (a) oparson TTount Query wdth(k) (b) Accuracy coparson Fgure 6 recall rate and accuracy coparson of two algorths wth dfferent query wdth V. ONLUSION FW_MRFI, whch s a streang data frequent te nng algorth based on varable wndow, s ntroduced n ths paper. The FW_MRFI algorth can ne frequent te n any wndow of recent streang data, whose gven length s L. Meanwhle, t dvdes recent streang data nto several wndows of varable length accordng to, whch s the nuber of the counter array. Ths algorth can acheve saller query error n recent wndows, and can nze the axu query error n the whole recent streang data. In order to copare ts accuracy and recall rate wth other exstng ethods, experents wth real data sets and synthetc data sets are conducted, whch proves that FW_MRFI algorth offers uch proved accuracy n recent frequent te nng n data strea. AKNOWLEDGEMENTS Ths research was supported n part by odern educaton technology research proect of Jangsu provnce (03-R-495). REFERENES [] J. Msra, D. Gres. Fndng repeated eleents[j]. Scence of oputer Prograng, 98, pp.43~5. [] L. Golab, D. DeHaan, A. L.Ortz, et al. Fndng frequent tes n sldng wndows wth ultnoally-dstrbuted te frequences[]. In Proceedngs of the 6th Internatonal onference on Scentfc and Statstcal Database Manageent, 004, pp.45~46. [3] G. S. Manu, R. Motwan. Approxate Frequency ounts over Data Streas[]. In Proc. of VLDB, 00, pp.346~357. [4] T. alders, N. Dexters, B. Goethals. Mnng Frequent Itesets n a Strea[]. In Proceedngs of 7th IEEE Internatonal onference on Data Mnng, 007, pp. 83~9. [5] Frequent teset nng dataset repostory, unversty of helsn (008), [6] L. f. Zhang, Y. Guan, Frequency estaton over sldng wndows[], Proceedngs of the 008 IEEE 4th Internatonal onference on Data Engneerng, 008, pp.385~387. [7] H. F. L, S. Y. Lee. Mnng frequent tesets over data streas usng effcent wndow sldng technques[j]. Expert Systes wth Applcatons.009, pp [8] H. T.La,T. alders, Mnng Top-K Frequent Ites n a Data Strea wth Flexble Sldng Wndows,opyrght 00 AM /07 [9].N. Yang, and Y.Y Yang, and.y. hu, "Iage Lbrary Systes: A Novel Installent Payent for Buyng Iages on the Web," Journal of oputers, Vol. 0, No., 009,pp , Apr. [0] X. Xu, J. Ln. A novel te advancng echans for agent-orented supply chan sulaton. Journal of oputers, Vol.4, No., 009, pp [] Ma cuxa, Meng xangxu. Research on Obect onstrants Model and Inverted onstrants n Paraetrc Desgn, Journal of oputers, Vol.3, No.9, 000, pp AADEMY PUBLISHER