A Method to Accelerate Human-in-the-Loop Clustering. Anni Coden, Marina Danilevsky, Daniel Gruhl, Linda Kato, and Meena Nagarajan IBM Research

Transcription

1 A Method to Accelerate Human-in-the-Loop Clustering Anni Coden, Marina Danilevsky, Daniel Gruhl, Linda Kato, and Meena Nagarajan IBM Research

2 Task-Specific Clustering: Drug Categorization Aspirin Ibuprofen Ketoprofen lysine Paracetamol? Warfarin Rivaroxaban Should aspirin be categorized as a pain killer or a blood thinner?

3 User-Specific Clustering: Unpacking Your Kitchen image from:

4 What is the `right set of features? Color? Cost? Size? Weight? First Letter? Texture? Flavor?

5 Change the goal: assist the user Clustering task problem setting: Unknown number of clusters No a priori features No reliably specified similarity metric User performing the clustering inherently knows a correct or invalid clustering when they see it Goal: Provide computer assistance to best make use of user feedback and accelerate clustering task Improve user productivity by reducing the time and steps needed for the clustering task Allow the user to work off whatever mental model is correct for them

6 Unpacking your kitchen: Manual Your unpacking partner hands you one item at a time You put that item into its correct drawer Repeat until all items have been unpacked

7 Unpacking your kitchen: Split/Merge You tell your unpacking partner about: Existing drawers that should be split into two drawers (Split) Pairs of existing drawers that should be combined (Merge) Your partner unpacks your kitchen by following these directives Repeat

8 Unpacking your kitchen: Must-link/Can t-link You tell your unpacking partner about: Pairs of items that must be in the same drawer (forks, knives) (Must-link) Pairs of items that can t be in the same drawer (glass, can opener) (Can t-link) Your partner unpacks your kitchen by following this constraint Constraints specified in previous iterations remain in power Repeat

9 Unpacking your kitchen: Refine/Lock Your unpacking partner hands you a handful of items they believe belong in the same drawer You split the items in that handful into two groups: You return any items that don t belong back to your partner (Refine) You put the items that do belong together into a drawer (Lock) Your partner adjusts their beliefs based on your actions Repeat

10 Grouper Overview: While not all clusters have been locked by the user: Compute working distance between all element pairs function of upper bound, lower bound, and (optional) semantic similarity for tie-breaking Cluster unlocked elements Display suggested clustering to user User performs a set of actions and requests a re-clustering e.g., lock a cluster, add an item to a cluster, remove an item to a cluster Cognitive assist: propagate user actions to upper and lower bounds on pairwise distances

11 NTI Assist: Initialization of Distance Matrices Min Distance spoon fork pan spoon fork pan Max Distance spoon fork pan spoon fork pan Semantic Similarity spoon fork pan spoon fork pan Working Distance WD = 0.5 (D min + D max ) + K s ( 1 - D sem ) K s = 0.25

12 NTI Assist: Adding an Item to a Cluster Min Distance spoon fork pan spoon fork pan Max Distance spoon fork pan spoon fork pan Semantic Similarity spoon fork pan spoon fork pan Working Distance spoon fork pan spoon fork pan - - 0

13 NTI Assist: Removing an Item from a Cluster Min Distance spoon fork pan spoon fork pan Max Distance spoon fork pan spoon fork pan X Semantic Similarity spoon fork pan spoon fork pan Working Distance spoon fork pan spoon fork pan - - 0

14 Naïve Triangular Inequality Spoon Pan Fork!" $! $"!" $! + $"

15 Evaluation Methodology Intuition: Better performance = fewer moves needed to arrive at ideal clustering Gold Standard: Final clustering of pre-defined vocabulary created by user given a task prompt Simulate Synthetic Users: System provides an initial clustering Simulate a user applying some sequence of moves to arrive at ideal clustering Comparisons Different approaches define the set of moves available to the user Multiple gold standards are created for evaluation by asking several real life users to construct their ideal clusterings The same pre-defined vocabulary set is used for multiple task prompts

16 Creating gold standard clusterings: Two task prompts bacon beef teriyaki brownies cheese burger chicken enchiladas chicken nuggets chicken tikka masala chocolate shake clam chowder fish tacos french fries general tso s chicken green tea horchata ice cream macaroni and cheese naan onion rings pancakes pizza saag paneer scrambled eggs steak sandwich tandoori chicken thai iced tea toast won ton soup beef carnitas beer cake chianti chicken noodle soup chicken parmesan chocolate milk churros coffee flan garlic bread ginger ale grilled salmon house salad italian soda milk nachos oysters peking raviolis ribeye steak samosa split pea soup sweet and sour pork tea tiramisu waffles Food Court Task: Create a set of food shops containing these menu items for a food court Coffee tiramisu tea chocolate cake brownies Menu Task: Create a set of menu sections containing these items for a restaurant menu scrambled eggs bacon toast Onion rings French fries Chicken nuggets

17 Grouper User Interface

18 Clustering Food Items Tasks: Coherence of User Clusterings Adjusted Rand Index between groupings for both tasks between 5 users (U1-U5)

19 Simulating Human-In-The-Loop Approaches: Synthetic Users Manual (1 move) Pick one item at a time and assign it to its correct cluster. Split / Merge (1 move) Either merge two clusters that should be co-clustered, or choose the cluster that mingles items from the fewest number of clusters and automatically split it into two clusters. Must Link / Can t Link (1 move) Either mark two items that should be co-clustered as `must-link, or mark two items that shouldn t be co-clustered as `can t-link. Merge closest pairs of clusters as possible. Grouper (1 move) Either merge two clusters that should be co-clustered, or remove an item that doesn t belong with most of the items in its cluster. Update distance matrices, recluster unlocked items.

20 Evaluation: Moves Needed to Arrive At User-Specified Clustering Approaches: Split/Merge (SM) Must-link/Can t-link (MC) Grouper (G) Manual assignment (Manual) Semantic distance metrics: UKWAC 1 (U1) UKWAC 2 (U2) Google Billion Words (Goo) Blekko Medical (Ble) 3.94x speed-up over Split/Merge 4.92x speed-up over Must/Can t

21 Evaluation: Scaling the Dataset FDA Drugs Task (FDA) Group medications from clinical records (889 elements) into drug classes for clinical use in a general practitioner setting Adverse Drug Reactions Task (ADR) Group phrases from `in their own words patient reports (3,145 elements) into appropriate categories of side effects Corpus FDA ADR Size 889 3,145 Clusters Split / Merge 1,467 4,745 Grouper Ratio 2.99X 2.82X Must / Can t could not complete on either dataset, as it increases exponentially in the number of items

22 Conclusion Grouper is a method for accelerating human-in-the-loop clustering User has a mental model of element relationships - knows a correct or invalid clustering when they see it - but no a priori features or similarity metrics are available User feedback is propagated for reclustering using Naïve Triangle Inequality Grouper significantly reduces the time and number of steps needed for completing clustering tasks, even for a large number of elements

23 A Method to Accelerate Human-in-the-Loop Clustering Anni Coden, Marina Danilevsky, Daniel Gruhl, Linda Kato, and Meena Nagarajan IBM Research contact: