Lee Radigan: A Reflection on my (6th) Year as an Undergrad at the University of Toronto

My name is Lee Radigan and I am a non-degree student pursuing admittance to the Biostatistics Masters program at the Dalla Lana School of Public Health.  After returning for my 6th year studying statistics at The University of Toronto, I thought that this was a perfect time to reflect on my progress.
Since September, I have been working under Dr. Pascal Tyrrells guidance on a project aimed at helping the Department of Medical Imaging report agreement in their research.  To do this, I created a flow chart to help guide the reader towards the proper method of agreement.  Along with this, I conducted a simulation looking at a specific question pertaining to the Department.
Initially, I was tasked with combing through various papers on the theory of agreement and making sense of all the different published work that was out there.  There are many different approaches and different ways of looking at reporting agreement, so it was quite difficult to figure out when and where to properly use every single approach.  After reading and re-reading each paper, as well as consulting the MiData team, I started to develop a thorough understanding of what agreement was, why it is important to report it, and how to go about reporting it appropriately.
Next, a flow chart was required to summarize what I had learned from the literature.  This was not an easy task, because it forced me to dig really deep and make sure that every node in my flow chart was well thought out and appropriate.  After many iterations and adjustments, I created a detailed chart that walks the reader from their initial research question up to the required agreement statistic.
My final task was to conduct a simulation that would test the question: Can a group of less experienced student raters be as accurate as a smaller set of more experience expert raters?  And if so, how many students?  And under what conditions?  This was a very fun and informative task for me as I was able to conduct my first simulation.  During this experience, my biggest difficulty was justifying my choices of parameters within the simulation.  When conducting a simulation you have freedom to choose how it is going to work, but you must be careful to be able to back up each and every parameter choice.  The simulation ended up showing that: the larger the disparity between the rating errors of the student and expert raters, the more students it takes to match the accuracy of the experts, confirming my intuition.
There are many things that I wish to expand on with respect to my project in future.  I want to create a user friendly app that will be even easier and more compact than my flow chart.  Additionally, I want to try to get my paper published.  To do this I will need to look further into my simulation and consider a more broad range of student/expert scenarios that likely will occur in practice.  I will also need to further refine my definitions and understanding of each concept of agreement.
This year has truly been the best of my life and I can largely attribute that to Dr. Pascal and the MiData team.  I look forward to contributing to Medical Imaging research and to many more learning experiences.
Time to enjoy the summer as I embark on yet another exciting experience as a student Statistical Analyst at the CAMH Nicotine Dependence Clinic as a summer placement!
Lee Radigan

Who’s in Agreement?

So, let’s say you have invited everyone over for the big game on Sunday (Superbowl 49) but you don’t have a big screen TV. Whoops! That sucks. Time to go shopping. Here’s the rub: which one to get? There are so many to chose from and only a little time to make the decision. Here is what you do:

1- call your best friends to help you out
2- make a list of all neighboring electronics stores 
3- Go shopping!

OK, that sounds like a good plan but it will take an enormous amount of time to perform this task all together and more importantly your Lada only seats 4 comfortably and you are 8 buddies.

As you are a new research scientist (see here for your story) and you have already studied the challenges of assessing agreement (see here for a refresher) you know that it is best for all raters to assess the same items of interest. This is called a fully crossed design. So in this case you and all of your friends will assess all the TVs of interest. You will then make a decision based on the ratings. Often, it is of interest to know and to quantify the degree of agreement between the raters – your friends in this case. This assessment is the inter-rater reliability (IRR). 

As a quick recap, 

Observed Scores = True Score + Measurement Error


Reliability = Var(True Score)/ Var(True Score) + Var(Measurement Error)

Fully crossed designs allow you to assess and control for any systematic bias between raters at the cost of an increase in the number of assessments made. 

The problem today is that you want to minimize the number of assessments made in order to save time and keep your buddies happy. What to do? Well, you will simply perform a study where different items will be rated by different subsets of raters. This is a “not fully crossed” design! 

However, you must be aware that with this type of design you are at risk of underestimating the true reliability and therefore must, therefore, perform alternative statistics.

I will not go into statistical detail (today anyway!) but if you are interested have a peek here. The purpose of today’s post was simply to bring to your attention that you need to be very careful when assessing agreement between raters when NOT performing a fully crossed design. The good news is that there is a way to estimate reliability when you are not able to have all raters assess all the same subjects.

Now you can have small groups of friends who can share the task of assessing TVs. This will result in less assessments, less time to complete the study, and – most importantly – less use of your precious Lada! 

Your main concern, as you are the one to make the purchase of the TV, is still: can you trust your friends assessment score of TVs you did not see? But now you have a way to determine if you and your friends are on the same page!

Maybe this will avoid you and your friends having to Agree to Disagree as did Will Ferrell in Anchorman…

Listen to an unreleased early song by Katy Perry Agree to Disagree, enjoy the Superbowl (and Katy Perry) on Sunday and…

…I’ll see you in the blogosphere!

Pascal Tyrrell