When to Recommend Mentor Suggestions

Last night Kyle and I submitted our papers to CHI (yayyy)! 

My paper was on the study I ran mostly in July/August where we had expert programmers make suggestions on how Looking Glass worlds could be improved, as if they were mentoring a child. We also had the participants write "rules" to check other kids' worlds for similar problems. 

As a kind of recommender system, we only want to make suggestions to kids that are at their skill level and that are appropriate for the world they are creating. Often, recommendation systems are based on preferences, like movies. In that case, recommendation systems can use AI algorithms to unerstand what the user likes and base recommendations based on a set of information about related preferences (I think? my knowledge doesn't go that deep here). In our case, the mentors are essentially writing the algorithms to decide when to make these recommendations. While they had obviously only seen the system for 2 hours and have not thought about it as deeply as we have, it seemed like participants were not thinking as much about making sure the suggestions are teaching something as we would like. Additionally, the skills and knowledge of students is constantly changing, while preferences most likely stay much more constant. All of the factors that need to go into a recommendation for learning are complex. It's possible that the participants were working so hard at writing the code that they didn't have time or energy to ensure that their rules caught all of the correct cases. 

These questions and issues all tie into the design of the interface so that mentors can check whether their rules provide suggestions in cases where it actually makes sense.