The power behind an intelligent system is knowledge.

Size: px

Start display at page:

Download "The power behind an intelligent system is knowledge."

Phillip Matthews
5 years ago
Views:

1 Induction systems 1 The power behind an intelligent system is knowledge. We can trace the system success or failure to the quality of its knowledge. Difficult task: 1. Extracting the knowledge. 2. Encoding the knowledge. 3. Inability to express the knowledge formally. Induction 2 Induction: inducing general rules from knowledge contained in a finite set of examples. Induction is the process of reasoning from a given set of facts to conclude general principles or rules. Induction looks for patterns in available information to infer reasonable conclusions.

2 Induction as search 3 Induction can be viewed as a search through a problem space for a solution to a problem. The problem space is composed of the problem s major concepts linked together by an inductive process that uses examples of the problem. Induction 4 The choice of representation for the desired function is probably the most important issue. As well as affecting the nature of the algorithm, it can affect whether the problem is feasible at all. Is the desired function representable in the representation language? An example is described by the values of the attributes and the value of the goal predicate. We call the value of the goal predicate the classification of the example. The complete set of examples is called the training set.

3 Induction - first example 5 Determine an appropriate gift on the basis of available money and the person s age. Money and age will represent our decision factors (problem attributes). Money Age Gift Much Adult Car Much Child Computer Little Adult Toaster Little Child Calculator Money Age Age Age Money Money Induction - decision trees 6 A decision tree takes as input an object or situation described by a set of properties, and outputs a yes/no decision. Decision trees therefore represent Boolean functions. Each internal node in the tree corresponds to a test of the value of one of the properties, and the branches from the node are labeled with the possible values of the test. Each leaf node in the tree specifies the Boolean value to be returned if that leaf is reached.

4 Induction - decision trees 7 Decision trees are implicitly limited to talking about a single object. That is, the decision tree language is essentially propositional, with each attribute test being a proposition. We cannot use decision trees to represent tests that refer to two or more different objects. Decision trees are fully expressive within the class of propositional languages, that is, any Boolean function can be written as a decision tree. Have each row in the truth table for the function correspond to a path in the tree. The truth table is exponentially large in the number of attributes. Induction - decision trees - second example 8

5 9 Induction - decision trees - second example 10 If there are some positive and some negative examples, then choose the best attribute to split them. If all the remaining examples are positive (or all negative), then we are done: we can answer Yes or No. If there are no examples left, it means that no such example has been observed, and we return a default value calculated from the majority classification at the node s parent. If there are no attributes left, but both positive and negative examples, we have a problem. It means that these examples have exactly the same description, but different classifications. This happens when some of the data are incorrect; we say there is noise in the data. It also happens when the attributes do not give enough information to fully describe the situation, or when the domain is truly nondeterministic.

6 11 Induction - decision trees - choice of attributes 12 Information theory Mathematical model for choosing the best attribute and at methods for dealing with noise in the data. The scheme used in decision tree learning for selecting attributes is designed to minimize the depth of the final tree. The idea is to pick the attribute that goes as far as possible toward providing an exact classification of the examples. A perfect attribute divides the examples into sets that are all positive or all negative. The measure should have its maximum value when the attribute is perfect and its minimum value when the attribute is of no use at all.

7 Induction - third example 13 Example Height Eyes Hair Class E1 tall blue dark 1 E2 short blue dark 1 E3 tall blue blond 2 E4 tall blue red 2 E5 tall brown blond 1 E6 short blue blond 2 E7 short brown blond 1 E8 tall brown dark 1 Induction - example Example Height Hair Eyes Class E1 tall dark blue 1 E2 short dark blue 1 E3 tall blond blue 2 E4 tall red blue 2 E5 tall blond brown 1 E6 short blond blue 2 E7 short blond brown 1 E8 tall dark brown 1 Inform ation in the 8 exam ples: c N N Entropy(I) = i i log 2, i = 1 N N N i num ber of exam ples in class i N total number of exam ples (training) 14 Entropy(I) = -5/ 8 log 2 (5 / 8 ) 3 / 8 log 2 (3 / 8 ) = bit

8 Attribute test: 15 Select an attribute and calculate the information gain (entropy) for it. J n kj Entropy(I,A K ) = entropy ( I, A K, J ) = N j =1 attribute A K, k = 1, 2,, K J c n kj n kj ( i) n kj ( i) = = = log 2 j 1i 1 N n kj n kj The examples are divided into J subsets, where J is the number of values the feature may take. n kj (i) - examples from subset j belonging to class i n kj total number of examples in subset j Induction - example Example Height Hair Eyes Class E1 tall dark blue 1 E2 short dark blue 1 E3 tall blond blue 2 E4 tall red blue 2 E5 tall blond brown 1 E6 short blond blue 2 E7 short blond brown 1 E8 tall dark brown entropy(i, hair) = log 2 log log 2 log log 2 log = 0.5 b it

9 Induction - example Example Height Hair Eyes Class E1 tall dark blue 1 E2 short dark blue 1 E3 tall blond blue 2 E4 tall red blue 2 E5 tall blond brown 1 E6 short blond blue 2 E7 short blond brown 1 E8 tall dark brown 1 17 Induction - example Example Height Hair Eyes Class E1 tall dark blue 1 E2 short dark blue 1 E3 tall blond blue 2 E4 tall red blue 2 E5 tall blond brown 1 E6 short blond blue 2 E7 short blond brown 1 E8 tall dark brown 1 18

10 Induction - example 19 Information gain max{entropy (I) entropy (I, A K ) IG (hair) = = bit IG (height) = = bit IG (eyes) = = bit red E4 class 2 hair dark E1 class 1 E2 class 1 E8 class 1 blond E3 class 2 E5 class 1 E6 class 2 E7 class 1 Induction - example Example Height Hair Eyes Class E3 tall blond blue 2 E5 tall blond brown 1 E6 short blond blue 2 E7 short blond brown entropy(i, height) = log 2 log log 2 log = entropy(i, eyes) = log 2 log log 2 log = 0

11 Induction - example Example Height Hair Eyes Class E3 tall blond blue 2 E5 tall blond brown 1 E6 short blond blue 2 E7 short blond brown entropy(i, height) = log 2 log log 2 log = entropy(i, eyes) = log 2 log log 2 log = 0 22 hair blond red dark E4 class 2 E1 class 1 E2 class 1 E8 class 1 E3 class 2 E5 class 1 E6 class 2 E7 class 1 Eyes blue brown E3 class 2 E6 class 2 E5 class 1 E7 class 1

12 Induction systems 23 Determine objective - a search through a decision tree will reach one of a finite set of decisions on the basis of the path taken through the tree. Determine decision factors - represent the attribute nodes of the decision tree. Determine decision factor values - represent the attribute values of the decision tree. Determine solutions - list of final decisions that the system can make - the leaf nodes in the tree. Form example set. Create decision tree. Test the system. Revise the system. Induction systems - example 24 Football game prediction system Predict the outcome of a football game (will our team win or lose). Decision factors - location, weather, team record, opponent record. Decision factor values - Location Weather Own Record Opponent Record Home Rain Poor Poor Away Cold Average Average Moderate Good Good Hot Solutions - win or lose

13 Induction systems - example (cont d) 25 Examples - Week Locat. Weath Own r Opp. r Own 1 Home Hot Good Good Win 2 Home Rain Good Averg Win 3 Away Moder. Good Averg Loss 4 Away Hot Good Poor Win 5 Home Cold Good Good Loss 6 Away Hot Averg. Averg. Loss 7 Home Moder. Averg. Good Loss 8 Away Cold Poor Averg. Win Induction systems - example (cont d) 26 Decision tree - Win rain cold Weather moderate hot Loss home Loss Location Own rec poor away good No-data Win average Loss Win Test the system - predict the future games. Get the values for the decision factors for the upcoming game and see on which team to bet.

14 Induction systems - example (test) 27 28

15 29 Sensitivity study - Location 30

16 Induction systems - pros. and cons. 31 Discovers rules from examples - potential unknown rules could be induced. Avoids knowledge elicitation problems - system knowledge can be acquired through past examples. Can produce new knowledge. Can uncover critical decision factors. Can eliminate irrelevant decision factors. Can uncover contradictions. Difficult to choose good decision factors. Difficult to understand rules. Applicable only for classification problems. Induction systems - implemented 32 AQ11 - diagnosing soybean diseases. Identifies 15 different diseases. The knowledge was derived from 630 examples and used 35 decision rules. Willard - forecasting thunderstorms. 140 examples, hierarchy of 30 modules, each with a decision tree. Rulemaster - detecting signs of transformer faults. Stock market predictions.

Announcements. Homework 1 solutions posted. Test in 2 weeks (27 th ) -Covers up to and including HW2 (informed search)

Announcements. Homework 1 solutions posted. Test in 2 weeks (27 th ) -Covers up to and including HW2 (informed search) Minimax (Ch. 5-5.3) Announcements Homework 1 solutions posted Test in 2 weeks (27 th ) -Covers up to and including HW2 (informed search) Single-agent So far we have look at how a single agent can search