SDS PODCAST EPISODE 215 WITH BRIAN DOWE

Transcription

1 SDS PODCAST EPISODE 215 WITH BRIAN DOWE

2 This is episode number 215 with a full-stack web developer Brian Dowe. Welcome to the SuperDataScience Podcast. My name is Kirill Eremenko, Data Science Coach and lifestyle entrepreneur, and each week we bring you inspiring people and ideas to help you build your successful career in data science. Thanks for being here today and now let's make the complex simple. Welcome back to the SuperDataScience Podcast, ladies and gentlemen, very excited to have you on the show today. Today we've got an aspiring data scientist and full-stack web developer Brian Dowe joining us, and I literally just got off the call with Brian a few hours ago and what I can say about this episode is it's very inspiring, especially if you are a web developer yourself or a developer of any kind yourself. You will find a lot of useful tips and insights in this episode. If you're not a developer, you will also find a lot of useful tips. But I personally found a very, like Brian's example of how he structured his career, very insightful and something to admire and kind of like dissect, and that's exactly what we did in the podcast. So you'll find out... like we touched on three main things. So first of all we talked about what it's like to go from a developer to data scientist or in fact how to integrate data science in your career if you are a developer, if you work on web development or any kind of development, how to integrate data science in your career. In fact, I think anybody looking to integrate data science will find a lot of these tips useful.

3 Then we talked about models. We talked about developing models, deploying models in business and maintaining models. In fact, we're going to dissect the whole case study of a recent model that Brian has worked on using that Apriori algorithm, and you will hear some back and forth between us about the development, the deployment, and the maintenance life cycle. We'll actually come up with some ideas on the podcast, which you might find quite interesting in terms of brainstorming and how to think about modeling. And in general you'll get a lot of takeaways about modeling. And finally, we talked about getting into the space of data science or how, what it's like to learn data science, what are the challenges in the learning curve that Brian has been facing. He's been a learning data science since the start of this year, so for almost a year plus a minus a couple of months. And you'll also hear about the tools, some of the tool recommendations that Brian has for you if you're just starting out into data science. So there we go. This podcast quite interesting in terms of the three pillars that we discussed. We get a lot of value from here and without further ado, I bring to you Brian Dowe, full-stack developer and aspiring data scientists. Welcome to the SuperDataScience Podcast, ladies and gentlemen, today we've got a very exciting guest on the show. Brian Dowe calling in from San Mateo, California. Brian, how are you going today?

4 I'm doing good Kirill, how are you? I'm doing very well. Thank you very much. It was such a pleasure to catch up at DataScienceGo mate, it was exciting to hear your story, I can't wait to share it with our audience today. But first off, how do you feel at the events and how do you feel after it? It was really an incredible experience. There were so many fantastic presenters. There were a lot of people that I sort of networked with and talked to outside of the sessions as well, and got a lot just all around out of everyone I interacted with. It was an amazing experience, and I learned so much and it's given me a lot of great jumping off points moving forward. Yeah, thanks. That's really great to hear and you just for the podcast, you mentioned you were already getting into like you made this tilt or shift in your trajectory, and you started doing stuff on Kaggle, after DataScienceGo, you started the Andrew Ng machine learning course. Quite a few things have happened for you. What would you say has been the biggest shift after attending DataScienceGo 2018? I think that before I attended I had had some experience with applying machine learning models and sort of an easier setting where the Dataset is preprepared for you and it's pretty clean, and you just got to get right to the modeling. And so one thing I wanted to do following up from DataScienceGo is push my knowledge a little bit further. So I actually went through and did like the derivations and the calculus for gradient descent and that made a huge difference

5 for me in just understanding what's going on under the hood. And so even if the... it can be more convenient to use libraries like Scikit-learn, Intensive flow to do projects, it's still really helpful to understand what's going on behind the scenes so that you can work with those tools more efficiently. So that was really... yeah, that was really huge for me. Awesome. So with Kaggle, I've been going through just some of the datasets that they have posted and trying to make some like predictions even with just like basic models, but that's given me some good practice in data preprocessing and organizing a sort of not clean dataset into something that can be fed into a model and that's been some really valuable experience for me. Fantastic. Great to hear. Amazing. One more thing on DataScienceGo, I was curious about this, what was your favorite talk? Who was the speaker who gave you a top talk? I think one of my favorite ones was that Gabriela de Queiroz from IBM. Her talk on a deploying or I think it was deploying machine learning models in five minutes or something like that- Deep learning. I think it was deep learning models in five minutes. Yeah, exactly. And just she showed us the model asset exchange that you could use to sort of find a bunch of prebuilt and pre-trained models and start deploying

6 them very quickly. And I thought that was really interesting and walked away from that with some items on my to do list to go through a lot of resources. I really enjoyed her talk a great deal. Fantastic. Well that's very exciting. Well Brian, very cool to have in you the show and one of the reasons is because I am excited, super excited about your career path. I think you have a very inspiring journey that you've created for yourself, and I would love to share it with our listeners. And what I mean here for our listeners is that what you need to know about Brian is that he is a full-stack developer, web developer, and we'll talk more about that in a second. And Brian sees the value of data science, sees the value of machine learning and deep learning and is actively applying that in his career. And I know that's across our audience, across the listeners in our podcast, a very large percentage of you guys, I don't know, maybe 30, 40 percent, that's my rough estimate, it might be even more, are people who are developers who are also looking to get into data science or are already in data science, have transitioned into data science or have seen the value of data science. And I think it is... this is going to be an inspiring story for you guys to model in your careers. But even if you're not a developer, the steps that Brian has taken to integrate data science in his career without fully jumping straight into it and quitting his job and just going data science, data science, is quite inspiring. So I think that's going to be cool. So I'm very

7 excited to dig into it. How are you feeling about this Brian? I'm really excited. Yeah. I'm excited to dig into this as well. Awesome. Some good self reflection opportunity for you, I guess. Yeah. Awesome. Okay. So tell us a bit about you Brian. You are a web develop, a full-stack web developer. If somebody off the street were to ask you, Brian, what is it that you do? What is a full-stack web developer do? How would you answer that question? Sure. So generally when people ask me that because it's come up a decent amount in conversation, I'll say that full-stack is a combination of front end and back end technologies. And in my specific case, I worked for a company called Education.com, it's a web application. And so for web applications, front end usually has to do with building templates. So all the html and CSS are the parts of the site that users see visually and also some Java script for interactive components. And the backend side is like the database and APIs that interact with your database and grab data to display to the end user. So the front end would be like what you see when you look at a webpage, and the back end would be like what happens behind the scenes when you click a button to submit a form, like where's that data going? What is it doing? Most of that is handled by back end technology.

8 Gotcha. How long have you been with education.com? It's been about nine months now, so not very long. I interned for four months, and I've been full-time for about roughly the last like five and a half months. Okay. Gotcha. Tell us what attracted you in data science, like why are you on this podcast, how did you get into this, hear about data science and what the next steps did you take from there? I actually first learned about data science, Kirill, through your course on machine learning A to Z. And I started doing it maybe about two or three months after I had started working for education.com. And prior to that I had built some applications just to sort of teach myself that were, had like very simple data components, like maybe just a user database and maybe like the ability to make blog posts or write reviews, etc. And when I stumbled across machine learning A to Z, I had heard of the field, I didn't really know too much about it. And I remember watching your introductory video where it kinda goes over a lot of the applications and use cases and machine learning. And I thought, oh, this is really cool this provides a really interesting way to look at data, gain insights from it and improve the actions that you can take on the basis of that. And so that was really interesting to me, and I think as I progressed through the course, I found it to be more accessible and something that I could jump into even though I didn't have too much experience. And yeah, it

9 just kind of progressed from there. And my interest of it has only grown over time. Interesting. Let's rewind a little bit. Tell me this, how do you stumble across a machine learning A and Z course, what were you searching online for? Obviously there was some kind of a need that you were trying to fulfill when you saw it. Like people don't normally stumble upon machine learning A to Z unless they're actually looking for something related to data science, like what was the initial trigger for that to happen? I had been using Udemy for a long time before that for projects that were [inaudible 00:12:25] specifically to data science. So I mentioned before how I was trying to get started by building a simple application to develop my development skills basically. I had taken some courses at that time on building a clone of YELP with ruby on rails, and then I found one on building a price alerts app with Python and flask. And so throughout the course of my learning, my Udemy feed or the courses that popped to the top had to do a lot with app development and with programming in general and with Python also, specifically over time. And then I think it was through that your course just sort of popped up on my list. Okay. Like you got into machine learning and data science because machine learning and data science got you in there in the first place. That's what it sounds like. Yeah. Sort of, yeah.

10 The circle has closed, right? We've gone full circle, this is so cool, right? Like you're studying exactly the stuff that has influenced your career to study that stuff. This is like Inception level type of thinking, right? Have you ever thought of it that way? I haven't, but that's really interesting now that you put it that way. Oh wow. That's so cool. That's so cool. One of the craziest stories. Okay. So you got into machine learning and you started taking the course and at the same time, how were you able to apply this at work? Like you're a full-stack developer, and this is where the interesting stuff starts to come in because I know your story a little bit already. How were you able to take that into... as a developer, if I'm a full-stack developer, I might be a bit like shy or I might be a bit... not even come to mind that I can bring this stuff, machine learning to work. It's not applying. It's completely unrelated to my role. So how did you go about that? That's a really Interesting question and I definitely did feel that like not knowing exactly where to start, where the right opportunity was and it happened in a very roundabout and sort of like by chance way. Our company, we try to hold at least one hack week every year. And for those who might not be familiar with what that is, it's generally where everyone can sort of come up with their own projects that they want to build outside of the development pipeline and then people can team up and then just try to build whatever they want and then we all present to each other at the

11 end of the week. And by the time hack week came about, which was about like may of this year, I had been studying machine learning maybe for like two to three months. I had talked... like briefly mentioned it to some people at work in passing. And then when hack week came about, one of my colleagues approached me and he wanted to build a recommendation system using machine learning. And that was basically the extent of the background for it. And, yeah. So I remember sitting down with him sort of trying to brainstorm what to do... before I get too much deeper into that, this hack week was really the catalyst for a lot of the professional application of machine learning or in my workplace to happen just because it provided an opportunity to just build a project that you're passionate about without any other restrictions. I think looking back, that really empowered me to start bringing up ideas and trying to make things work outside of that context and just do it as part of our normal pipeline of projects. But if I could, looking back, make a recommendation to someone in a similar position, it would be to not hesitate to just dive in and start trying to find problems to solve. Because I think a lot of full-stack developers have access to a wide breadth of data because you have to work with it when you build your applications. And so I think just sort of diving into the data that your company has and then trying to figure out, okay, what's a way that I could use this? Like what's something that I could predict based on certain

12 features or what value can I extract from this that could be presented to the end user in some way. I think my recommendation would be to just dive in, start thinking about problems to solve and when you have something or an idea of how to go, odds are that the other developers you work with will be intrigued by this and curious about this. Do you have an idea that could potentially bring value to the company? I think like trying to empower yourself and just dive in and start looking for problems to solve is one of the best ways to get started. I totally agree with that. And I just wanted to mention two points here. First one is that, you also obviously need to be careful as a developer. It depends on company to company, but often you do have potential access to data, but for instance, in Facebook, if you use it for the wrong reasons, you'll probably get fired. So kind of like, be sensible about that and maybe confirm with your boss if you can use that data or maybe if you, if that's a bit too early to do, maybe create some dummy data sets that are similar or you have The similar columns in terms of structure to the company data set, but like play around or download some data sets in your free time to play around with before you actually play around with real data that is customer base, especially if it's sensitive data. That's one comment I just wanted to make. But in general, indeed, like even if your company doesn't have these hack weeks or hackathons, which I think are very useful especially, but they usually take place in large organizations that want to inspire

13 innovation. So if your company doesn't have that job, maybe you can talk to the management to start introducing it. But even if that's not the case, you can still play around with data or dummy data or similar data and see how you could potentially bring value to the business. And you can still bring those results and those suggestions to your management or to the company leadership and present it to them. Every company in this day and age wants to be data driven or model driven. They will jump on top of it. If you can add value to the bottom line of the company, it is a very rare instance that management, especially when it's concerning technology and data science where the investment isn't that high, but the return on investment can be massive. Very rare do you get cases where management turned down those ideas, turned down those innovations. So as long as you're proactive, don't let the absence of a hack week or a hackathon get in your way. Or if for instance, you're a hack week is 11 months away, it happens once a year. If it's 11 months away, don't wait, you can already do it now. Those are just some of my thoughts on that. Brian, you were talking about you were preparing for this hack week and a colleague of yours came up to and then you got into this project. What was this whole idea? You said you were going to dive deeper into that topic. Sure. So the project was a recommendation system for our content. So just maybe to give a little bit of background information on what we do in education.com, we're a platform for parents and

14 teachers to come and find resources and teaching tools to help their kids. And so when I say like a recommendation system for our content, I mean we have just a bunch of worksheets and games and lesson plans and all of this stuff that parents and teachers can come and use. So what we wanted to do is create a way or improve the way that we recommend other resources that you can use based on what you're viewing. So if you like go to our website and click on a worksheet, you'll see at the bottom, here are five other worksheets that are related to this one that we would recommend for you. And so, yeah, so just to give a little bit of background on that. But anyway, so a colleague approached me, his name is Yon Burke, a brilliant engineer. So he approached me wanting to build this recommender and as we were sitting down and sort of talking about what some different angles are that we could go about this with, like what data would we use, like how I'd rearrange it, like what model would be needed? He brought up something or he said, "What if we're looking at this problem the wrong way, what if this problem is people who downloaded this also downloaded this" Or like, what if that's the way that we're going to look at it. And as soon as he said that, a light bulb in my brain went off and I thought, I know that I learned of an algorithm in machine learning A to Z that's perfect for solving this exact type of problem. So I went through and I looked through all my notes and it turned out that that algorithm was the Apriori algorithm. I know

15 that I made that connection because the example given was using that algorithm for a grocery store to try and figure out a where to place their products in relation to each other based on what products most users or most customers would buy together. And so I remember it just... like light bulb moment went off and made that connection, and then I knew that was the place where we had to start based on the way that we had a scoped the problem now. So what I did was I went back and I looked at the slides from machine learning A to Z from that section and In those slides, Kirill, you went over the equations for support competent and list for that algorithm. And then using those equations, I scripted or I just converted that into code basically in Python and then set up the structure to loop through all of our data and gather the information that we needed. And Yon, the engineer, I was working on this with did the pre-processing portion and then got me the dataset. And then we started testing and trying to figure out if it was going to work, if it was gonna give back good predictions because one of the attributes of this algorithm that made it a little challenging to figure out whether it was going to produce good results is that we didn't get the immediate feedback on whether it's a correct or incorrect estimate, for example, like you would with a supervised machine learning algorithms. So we had to look at some of the results by hand and figure out, okay, is this producing the results that we wanted, and what we were specifically looking for was

16 to have sort of more loosely defined recommendation than we currently had for a given worksheet because let's say you looked up like a two digit multiplication worksheet on our site, what we had before, it would have just given you like three or five [inaudible 00:24:28] multiplication worksheets. And we were thinking, okay, like how useful is that really for someone who is coming to our site and looking for another jumping off point. Like if they find a worksheet, are they just going to want to find more worksheets that are exactly the same? Probably not. What might be more useful is like maybe they're looking for like two digit division, something that's related to what they're on, but not exactly the same. And so we were hoping that by using a user download histories, instead of just like a content tagging or like what the exact subject was, that we would get some better results that would actually be more beneficial to users. And as we went through, that turned out to be the case and we saw that we were getting the recommendations that we were looking for. And then after we presented at the end of the week, it started picking up steam and enthusiasm and then we decided to run it on the site as an AB test. And after running it for a while, it turned out that it won against our former recommendation system, and then we pushed it to production and it became our first machine learning model that was deployed on the live site. So that was a huge win and a really exciting first step in implementing machine learning. But the first of

17 many is what I'm very much hoping for and striving to make the case is that this is just the first step on a longer journey to integrate a more machine learning into our operations. And so yeah, it's very exciting stuff. Wow, that's awesome. Congratulations. That's a huge project and most importantly that it actually got implemented into the business and added value, that's just really cool to hear. And just to recap, so let me know if I'm getting this right. So before the recommender system was recommending more content to your users based on the tagging. So for instance, you have these multiplication sheets and maybe your tutorials, maybe some videos, some other content that you have all sorts of content, they're all tagged, like this is multiplication, this is for first grade, this is for fourth grade, this is math, this is this other topic and so on. And based on the tagging, your similarity of tags, it would recommend something. Where's your new recommender system, which is through the Apriori algorithm, it would look at not just the tagging, but it would look at what are users actually downloading. So if people on average download X, what did on average they download after that or what did most of them download after that? So you're looking at the behavior of your users and based on that you're saying, okay, well even though it's not tagged identically, looks like that's what people want, that's what people are after, and we're gonna use that as a suggestion. And so that Apriori algorithm

18 approach work better than your previous recommender method. Is that about right? Yeah, that's essentially the gist of it. Gotcha. Sounds a lot like Amazon. Like you go on amazon and you buy something and then out of the blue they recommend something else, it might not be related at all, but that's because most people kind of like after they bought that one thing, they usually go searching for that other thing; the people that are similar to you, I guess. Yeah. Gotcha. And then after that you did a quality assessment like not a quantitative assessment, as you mentioned because you didn't have like with supervised learning algorithms, you couldn't say yes, no, correct or incorrect. In this case, you did a qualitative assessment where you just went in and you tested out a few of those recommendations to see if what the recommendation was actually made sense to you. Like the example you gave like a three by three multiplication table instead of getting a four by four one, you get a division table, which kinda like makes sense. Is that what you did next? Yeah, that's essentially what we did. We were looking for things that were like not exactly the same but still would likely to be within the same subject area that might be closely related, for example, yeah, like that example that I gave hoping that users would be maybe studying those two subjects at the same time and they could find things that would be more directly related to

19 like where they would go next from a given subject or something that they might be also practicing at the same time. And so how long has this system been in place now? This a recommender algorithm? It's been a couple months now that's been up. The hack week was in May and after there were some additional steps that needed to be taken to a to set it up and running with our whole dataset and that took some time. But yeah, for a couple months now it's been up and running and live. How are the other results? How is the management seeing, are they seeing some positive impacts? Are they happy with how like this change, what has change has brought? Yeah, it has brought a positive change, but we're definitely also looking to see how we can improve it because this was just sort of like a first run like it did win the AB test, but we were still looking for ways to improve and to provide more valuable recommendations to our users. And I think, yeah, that this was a good first step to at least identify resources that are being used and consumed most often. But I think there's so many places that this could go and there's so much that we could still do to improve our recommendation capabilities and that's something that we're diving into right now. That's so cool. And I can just imagine the boost of morale that you got from that, like when your algorithm got implemented, how did that feel?

20 It was incredible. I think one big takeaway that I had from that is that when I was first studying machine learning, sort of leading up to that, and I think still a little bit following from the implementation of that algorithm. My focus was I wanted to learn all these like, cool, cutting edge tools like neural nets and like Gans and stuff like that. Like all this some of the more advanced modeling tools, but the process of actually sitting down, looking at what data we had available and trying to choose a model that fits best with that sort of showed me that you have to start from the problem and then try and find the model that best fits the solution rather than just picking a cool model that you have in mind that you really want to work with like a cnn and then searching for a problem that fits that solution. I think that can be like a great way to learn about a new technology like if you're focused on learning about like cnn's and you look for problems as specifically fit that use case, that's great for learning. But in practice, you don't always get to choose the problems that come to you, sometimes there's just a problem at hand that needs to be solved and you need to explore your toolkit and just choose whatever is best fitted for that solution. Like sometimes I think I've seen cases or at least read about vaguely cases where a bunch of models were tried and a simple linear regression or a simple logistic regression was the best outcome. And sometimes that's the case. Sometimes the model that you initially had in mind that you were focused on, isn't the best tool for the

21 problem and something that maybe is a little bit simpler and scoped is. And so I think that's something that I thought about a lot is it doesn't have to be like the craziest algorithm out there to make a really big impact. And so that's something that I think about going forward is just starting from the problem first and then searching for the tool that solves that after the fact. Very, very wise words. Totally agree with that. It's very easy to get carried away with a machine learning and all of these new shiny cool things. And that's awesome to learn them and they're very good to inspire you to learn and to grow and to find like these different nonstandard applications, but at the same time, sometimes less is more. You just go for what solves a problem and does that efficiently. And in your case it was the Apriori, which is awesome to hear. Another thing I wanted to ask you on this topic was that the integration of the algorithm into your company services. I think this is a very cool area that a lot of the time is missed by data scientists that you not only have to develop an algorithm, but you have to also operationalize it. You have to make it work in the company so you have to somehow put it into production, it has to integrate with the website and all those things. So I think that'll be really cool to talk about that. Are you able to share some details with us on how you went about it and what challenges you faced along the way? Yeah, I could share some details about that. A lot of the bulk of the work to actually set this up on

22 production was handled by my colleague who has like a lot more familiarity with the way that these systems work. A big challenge was just being able to run all of this data because we have so much data in user download histories from like all of our users and all of the things they downloaded it. It gets pretty big pretty quickly, and so knowledge of a cloud computing services became really helpful in this case. And that's what we were able to use to do this to do this. And so one thing that, a challenge that came along with that is trying to decide when and how often the model we need to be wrong. Because I think that's a question that is really important is like do you need to make actual calculations with this model on the fly or can you run it like every so often and use the results or store the results from that to display to the end user? And so we ended up going with the second approach of just running our algorithm on every so often to gather this information like for a given worksheet or game that a user downloaded or played, figuring out the top, so many worksheets that are associated with it, which is mainly the tasks that the algorithm was accomplishing. We decided that it would be much more efficient for us to just run this every so often and store the results so that when a user visited the worksheet, we can just pull it directly from our database without having to actually run the calculation on the fly. And so I think that's something that can vary from use case to use case and model to model depending on how you're trying to apply it, maybe you will need to

23 actually run a prediction or a calculation on the fly when a user clicks a button for example. And that can be... whether or not that's a feasible place to implement your model, I think depends on the weight of the calculation that it needs to make. And so if it needed to make a calculation that required a lot of processing time but the user needs the result immediately, then that might not be a feasible way to go versus if it's something that might not change drastically from day to day or week to week, then you could just run your algorithm every so often and then store the results within a structure that you already have set up to store data for example. So I think it depends on the use case, it depends on what you're trying to do and how often the results of the model need to be updated. And I think yeah, there are definitely ways to think strategically to integrate systems like this without running massive programs like every time a user clicks a button for example. Okay, gotcha. Very, very cool advice as well that there's different types of models and hope people are taking notice of this, that sometimes you might need to run them on the fly and get results all the time, sometimes it's enough to run them occasionally. And especially big companies with lots of data, they have time slots for running things on the servers, so usually they run... like I was at a company after leaving Deloitte where they would run things at night and their every single hour and sometimes usually in 15 minute chunks or maybe even shorter chunks, the whole night is split into these periods where you need

24 to apply to get server time locally or... now more things and more and more things are going to the cloud, but still a lot of companies do things on premise, do calculation on premise. And so they, in order to run all these models and do all the calculations, update all the data and stuff like that, especially like for example, the financial services, you gotta turn over all the things that happened during the day, make sure everything is accurate, reconcile a lot of stuff. That all happens at night and you want to be very mindful of how you're using server time in your company. But even if it's in the cloud, every time you rerun the model, it's still going to be a cost, it's still going to take some kind of budget or something. So being conscious about whether you need right away or not is quite an important thing. I'm going to ask you this question, so I want to see in a bit of a different space, have you taken the data science A to Z course? No, I don't think I have actually. Okay. So the reason I ask is because in the data science A to Z course we talk about model maintenance. And I just wanted to see like... obviously this has been a big success and breakthrough for both you and the company in terms of developing and deploying this Apriori model. What are your thoughts on how you're going to maintain it? And the reason I ask is because I've seen situations where models deteriorate. In fact, I've seen a situation where a model used to be very effective, like it would bring like 80 percent accuracy, but then over time,

25 over a period of 18 months it deteriorated to a level where the accuracy was less than 50 percent, was like 42 percent or something. Meaning that it would have been more efficient for the companies to just flip a coin and do the recommendations based on that rather than using a model. And so I'm just curious, what are your thoughts on how to maintain this model? Yeah, that's a really interesting question. I think to a certain extent, this can depend on the domain that you're in. And what I mean by that is I think like different domains have a different pace to how their data shifts structurally over time. And so for us, who are in the education space, at education.com and that kind of means that we follow the cycle of a school year. So for example, given parent or teacher, based on the time of year and based on when they signed up, like maybe they sign up in like August and then from August through June, they're using a bunch of second grade resources. And maybe for a parent like they then move onto the next grade and then they're going to be consuming a bunch of third grade resources. So, if we were to just store and continually update the model just based on new data coming in and not deal with eliminating old data, it would undoubtedly begin to grow stale over time in the sense that you're now will be making predictions based on users who had been with the platform for several years. And so you might not get the best recommendation for a second grade worksheet because you'd be pooling from download histories from a bunch of different grades as well. And so, I think for us, that's something that like

26 already implemented as part of the algorithm is to keep it seasonal and to try and look at data that's irrelevant to the time period that we're in relative to the school year. And so I think thinking about things like that, like how often... like for us right now, like grade is the main way that we look at this, but I think it could be interesting to dive in and see how subjects covered might change over time. And, yeah. But for now, grades and school year is like the main data that we have to go on because there's a lot of different curriculums out there and our site is used very widely and so it's a challenge to find a way to update your algorithm in a way that meets the needs of all the users on your platform. School year and grade just happens to be one way that's pretty standardized across our users, but I think trying to push that further and make it even more relevant to what a given user might be looking for is definitely something that is a challenge that we're going to have to rise up to and meet overtime. A really challenging problem. Okay. Gotcha. That's a very cool consideration about the seasonality. If I may, I'd like to give you another suggestion, is that okay? Sure. Yeah. So one thing I was thinking about is with your model, you could measure how well it makes those recommendations. So you could say, all right, we made all these recommendations, how much of those recommendations was actually used and how many

27 cases, and what percentage of the cases was the recommended content actually consumed by the user to whom it was being recommended or consumed within a relevant timeframe like it could be maybe they didn't consume it right away, maybe they had to think about it. But like within a week or a month they did indeed consume the content because if you have the right data points in place, you can actually collect that information like we recommended content Z and they didn't use content Z in the week or they did use content Z. And kind of like just have a yes, no type of approach and see that like currently at this stage your model has, I don't know, like a 35 accuracy or percentage or that's a been low, maybe like 75 percent accuracy rate, that 75 percent of the content you recommend is actually indeed consumed by the users. And then I would set up a system that would track that, that even autonomously it could track that and say, all right, so how is our model going month to month? Maybe you'll see some seasonality in that, which you could possibly be able to explain, but maybe with time you will see that, oh that's interesting, before it was 75 percent, now it's dropped to 74, now it's 69, now it's down to 65. And if you see a consistent trend going downwards, that means something's going on with your model. And possibly that could be a shift, not just a seasonal shift, but that could be a shift in the demographics of your population. Maybe a change in some kind of legislation around what students have to learn, what they shouldn't learn. Maybe a change in the available

28 content on your platform or available content outside or some influence from competitors or advertising agencies that people are looking to other stuff that your model couldn't have possibly taken into account at the time you created it because that event or that environment was not there at the time. And so I feel that like tracking, like that dynamic tracking or constant tracking through time is important because it allows you to fish out these changes that you can not otherwise see, right? So you need a fly, you need kind of an indicator that something is going on in the industry or in our platform, in the business, in our audience that we need to address and we need to either retrain the model, rebuild it or so on. Because if like if that's on track then it actually can be a bit too late. What do you think on that? Thank you so much for all those insights. I think those are some really good ideas. I know for me, this was a big first step for me into this space. And yeah, it's really helpful to hear about other ways to push this further from someone such as yourself who has more experience in this field. And so, yeah, I think I'm going to be walking away from this with a lot of things to think about and a lot of interesting ideas to bring up to my colleagues tomorrow. That's awesome. That's awesome. Well thanks for sharing. I think this has been a good case study for our listeners in terms of the whole process, not just creating an algorithm or a model that solves a problem, which is awesome, but actually then

29 deploying it and then considering maintaining it. Those are like all three are extremely important in the whole life cycle of a data scientist or a data science project. All right. Well, now let's shift gears a little bit. Let's talk about... you mentioned learning data science, you mentioned that how you got interested in the field and actually like the Udemy recommender system brought machine learning to you and then you got into it. Tell us a bit about the learning curve, how difficult is it to learn data science for somebody who's coming from a developer background? I think it depends a little bit on what you want to do with it and specifically like how deep you want to dive into, like for example, the underlying mathematics and stuff that drives a lot of these concepts. And to that end, I will say I think it's easy or easier to get to a point where you can apply these models quickly as opposed to if you were to like say try and build a certain model yourself from scratch, like that would take a little bit more time. But I will say coming back a little bit to machine learning A to Z, I think that course did a really good job of empowering you to get started with applying algorithms quickly. Like after the data pre-processing section, it basically gets right into here's how you can apply a linear regression model, here's how you can apply a support vector machine. And you see from doing a bunch of models like that, that the actual code to apply the model if you're using a library like Scikitlearn, is not that extensive.

30 And so I think getting to the point where you can at least start applying models and start working with them, you can do that very quickly. I think once you enter that phase, transitioning from that into applying these models in the real world can be a bit steeper of a learning curve because of the data pre-processing component. I think in a lot of courses that you'll see or just like data re-arranged in a fairly neat format, relative to how you would actually find it in the real world. And I would even go so far as to categorize it as a separate skill from modeling that you need to bring together with modeling in order to apply data science. And you could spend a ton of time sort of as like I've been trying to do more recently with Kaggle data sets. Just looking at a random data set that you find and thinking, okay, how can I arrange this and get it ready to be fed into a model just where I have all the features that I need, they are scaled or converted appropriately into what they need to be like for categorical data like one hot encoding it to make sure it can be fed into a model. You can spend a lot of time in just diving into the data pre-processing aspect. And I think there can be a learning curve there, but I think if you just push yourself to start tackling these problems and just dive in, just look at a random data set and just start playing around with it, like sort of look up what you need to as you go for like how you manipulate a dataset with a given tool like Pandas for example. And if you do that and just practice with different datasets in different contexts, you can gain a lot of skills fairly quickly to at least be able to not be

31 paralyzed by the idea of seeing this data set that you feed it into a model right away and you get a bunch of errors back. That can be very discouraging to someone who is just starting out and doesn't really know what to do when previously they've just ran a command that worked perfectly because they had a dataset that was pre prepared. So the kind of synthesize a little bit, I think that it can be easier to start learning about how the models work and apply them on clean datasets and a little bit harder to get to the point where you can really use these effectively in practice. But it's still very much attainable, it's very much feasible if you just push yourself, apply yourself and dive in, start tackling problems, even though it can be a bit nerve wracking at first, it's still a really great learning opportunity, especially when you're struggling with all these challenges and pushing yourself to figure it out. I think that's when some of the deepest learning happens. Totally agree. Through those challenges. And what would you say is... like you mentioned some of the difficult things, especially about data preparation. Totally agree on that. What are some of the things that actually keep you going? Have you noticed or naturally your successful implementation of the Apriori and how it was used in the business, that was something really cool that definitely gave you a massive push. Was there anything else along your journey that you noticed that once you get to these milestones, that gives you the inspiration to keep going because that

32 could be helpful for some or all listeners who are considering taking this approach? Yeah. So yeah, getting the Apriori live was definitely a big one to actually see that it was possible to take something through to production. I think for me I sort of felt frustrated by the fact that I knew how to apply a lot of models without diving into the deeper mathematics. And I really wanted to understand, okay, like what is gradient descent actually mean? Like I've seen it covered, I understand the general idea of what it's doing, but how does it actually work? And something that I did actually very recently is went through and actually by hand wrote out all of the calculus to derive gradient descent for a linear regression problem. And then I did the same thing to derive the back-propagation algorithm for neural networks. And I think when I did both of those things and realized the connection between the two and sort of saw that the back-propagation algorithm is an application of gradient descent on neural nets in the same way that gradient descent works on a linear regression or at least with the same conceptual grounding, that was really like a big breakthrough and understanding for me where I actually... it's like, what does it actually mean to train a model? It's like, okay, you feed it data and it learns how to predict based on that data, but like what does that learning actually mean? And then getting on top of gradient descent, I think at the core of how many algorithms learn, and for me, that was huge and just... I think so many things sort

33 of clicked and came together and made more sense after I did that. I would definitely recommend that, maybe not like a first starter activity to do, but after you've like played around with some models, have a general idea of like what a model is trying to accomplish and what training is trying to do, diving in and trying to understand what's going on under the hood can make it a lot... I guess booster competence when looking at... it can be sort of disorienting to look at one line of code and know that that line of code is the entire training for a machine learning algorithm. It can be hard to look at that and understand what's really going on if you like haven't like for example, done some of these derivations or deep dives into the underlying mechanics of how everything works. And so, I would definitely recommend doing that at least like once or a couple times to help yourself understand those things and it gives you a lot more understanding when applying the models in practice, even if you end up using the single line of code training, it's like you'll understand in a more deep way what's going on there. Gotcha. Very, very interesting advice. Thank you for sharing that. And yeah, interesting. I think everybody has ways to get inspiration for pushing themselves forward and for learning more. That's definitely a valid suggestion for our listeners as well. One more thing I wanted to ask you is what kind of tools would you recommend starting with for somebody who's just getting into this space or considering getting into the space?

34 For me, a Python was definitely my language of choice. I know that like R is also very, very big in the space. But for me, I had already done some work in Python so I was a little bit familiar with it. And I do think it's um, it's a good place to start. Scikit-learn, which I've mentioned a couple of times is, I think, a really good library to start with as well because there are a lot of different machine learning models that it accounts for. So it can be really easy to swap out different models, like try different things and sort of see what predicts and better. Pandas is really important for Python, for working with your dataset. And I think a lot of times I've seen it used in tutorials, it's usually just use for like importing the dataset, but there is a lot more that you can do with it. Like you can do join sort of in a similar way that you would do in Sequel. I think Sequel is also really important to know on a practical level, that's something that I'm working to build my skills in because a lot of time your data can be stored in different databases, maybe something in a sequel database, maybe something in a no sequel database like Mongo db, and a knowing how to run queries in any given situation where your data might be. So you can pull data from different sources and sort of arrange it in a way that's comprehensible is definitely valuable. And I think a Sequel is definitely something that you would see a lot in practice and Mongo or like no SQL databases are picking up a lot of steam as well. So understanding basic queries to pull your data. Once you get your data into, like for example, my case