Cst402 Final Project Ruonan Wen. Project Report

Transcription

1 Project Report 1. Introduction Hadoop MapReduce is a programming model and software framework for writing applications that rapidly process vast amounts of data in parallel. On the other hand, The Dream of the Red Chamber is one of China's Four Great Classical Novels. In this project, the author applied Hadoop MapReduce on the novel The Dream of the Red Chamber to analyze its character occurrence and compare it with frequent Chinese characters in Modern imaginative texts to see the differences and development of Chinese language. 2. Initial Motivation Similar to English language, Chinese language experienced a long period of changes and development. From ancient style to vernacular, to nowadays modern Chinese, some characters were abandoned, some were introduced, and some were changed to another way to be expressed. It is particularly interested to go into the development of Chinese language. On the other hand, as a new framework, Hadoop MapReduce has not been widely used, especially in China. Applying Hadoop MapReduce on some Chinese data seems particular fascinating. 3. Input File The Dream of the Red Chamber is one of China's Four Great Classical Novels. It is generally acknowledged to be the pinnacle of classical Chinese novels. It was composed in 1784 during the Qing Dynasty when Chinese vernacular literature started to grow. Literature works accomplished during late Qing Dynasty usually combined ancient and vernacular Chinese; thus, were also considered as the start of modern Chinese Language. In this project, the author downloaded electronic version (.txt format) of The Dream of the Red Chamber, converted the file to Unicode, and applied Hadoop MapReduce to count the character frequency. Here is a sample of The Dream of the Red Chamber in Chinese: 1

2 4. Mapper Read the file line by line. For each line, count all the character frequency. Output each character with its frequency in each line. Here is the main part of mapper algorithm: 5. Reducer Add all the frequency together of same character. Here is the main part of reducer algorithm: 2

3 6. Output File The output file gives all the characters (including punctuations, but abandoned when analyzing) in this novel with frequency. There are 4531 characters, including punctuations and unreadable characters, in this novel. Here is a sample: Noticeably, some ancient Chinese characters cannot be displayed correctly after Unicode conversion: 3

4 These are not Chinese characters after conversion. However, from the results, we found that they were not frequent characters; we ignored them in the experiment. The whole output file can be found in file ouput.rtf. 7. Results The file result.xlsx gives the result of my project. Here is a sample of the result file: There are four columns: 1. modern imaginative character 2. character frequency in modern imaginative Chinese 3. character in novel The Dream of the Red Chamber 4. character frequency in The Dream of the Red Chamber The No. on the side indicates the frequency ranking. The file presents 3999 most frequent character sorted by frequency in both modern imaginative texts and The Dream of the Red Chamber. The data of modern imaginative texts was obtained from online source: 4

5 . 8Analyses and Comparison 8.1.Chinese vernacular literature Some characters such as 了, 的, and etc. are commonly used in vernacular Chinese and even modern Chinese literature works are also the most frequent characters in The Dream of the Red Chamber. Since this novel was composed in late Qing Dynasty, my experiment could prove The Dream of the Red Chamber indeed is one of the works indicated the start of the Chinese vernacular literature. frequency ranking in The Dream of the Red Chamber Frequency ranking in modern imaginative Chinese 了 1 4 的 Characters in names of novel The Dream of the Red Chamber is a novel about four big families involving many people. Thus, there are a lot names in this novel. From the result, we could find some characters such as 贾, 袭, and etc. are very common in the novel, but in the very different position in the 4000 most frequent modern imaginative Chinese character list. It is because 贾 and 袭 are characters in the names of main characters in this novel. Thus, on one hand, we could use this result to investigate the main characters (people) in the novel; on the other hand, when apply useful data in the novel to contribute to the character counting for the whole works in particular period, it is significant to discard this kind of characters. frequency ranking in The Dream of the Red Chamber Frequency ranking in modern imaginative Chinese 贾 袭 Development of modern Chinese Even though The Dream of the Red Chamber indicated the start of Chinese vernacular literature, it is classic work. Many characters in this novel could not be found in modern imaginative Chinese list. In addition, some characters usually used in classic works are in the very different position in modern imaginative Chinese list. For example: frequency ranking in The Dream of the Red Chamber Frequency ranking in modern imaginative Chinese 罢 便

6 9. Further Works In this project, we only take The Dream of the Red Chamber to analyze. We do find some properties of this novel and some differences between this and modern Chinese language. However, Chinese language has a very long history and it changed dynasty by dynasty. To see the development of Chinese as a language, we should apply Hadoop MapReduce on more works of different time. In addition, in this project, I applied Hadoop MapReduce on the novel to accomplishment the character counting. In the further work, I could combine the frequent Chinese character in different periods in the same file and use Hadoop MapReduce to analyze the changes and differences. At last, when applied Hadoop MapReduce Chinese data, some problems happened in data conversion, especially for classic literature works, since some ancient characters had not been used. 10. Thanks Thanks for the help of Dr. Eric G Berkowitz through the whole semester. Thanks Kyle Schmitt for helping me converting the text file to Unicode. Thanks Melissa Etling for helping me with Hadoop MapReduce. 6