My name is Yiyun Gu and I am a fourth-year student studying mathematics and statistics at University of Toronto. After taking some statistical courses and machine learning courses, I was quite interested in applying machine learning methods and statistical methods to practice. Medical imaging is a popular field where machine learning methods have great impacts. Therefore, I contacted Dr. Pascal Tyrrell and he would like to supervise me.

Last September, my initial research direction was Bayesian optimization on hyperparameters of Convolutional Neural Networks based on the previous model information and the distributions. Besides Dr. Pascal Tyrrell’s instruction, he introduced his graduate student who was also interested in this field. We had weekly meetings to discuss how to make the idea implementable. I read many papers and learned relevant knowledge of Gaussian process, acquisition functions and surrogate functions. However, there was a huge challenge on how to update the hyperparameters of the prior distribution based on the information from the CNNs model. I was anxious about the progress. Dr. Pascal Tyrrell encouraged me to shift the direction a little bit because he cared about what a student learned and felt about the project.

Since November, out of interest in Bayesian concepts, I have been working on a project about comparing frequentist CNNs and Bayesian CNNs for the projects with sample size restrictions. Because there might not be sufficient data in medical imaging, I would like to determine whether Bayesian CNNs would benefit from prior information for small datasets and outperform frequentist CNNs. Bayesian CNNs update the distributions of weights and bias while frequentist CNNs use point estimates. The resources of the codes of Bayesian CNNs were limited. I tried to make full use of and modify the codes so that I could run the experiments from training sample size equal to 500 to training sample size equal to 50000. I applied customized architectures and AlexNet to MNIST and CIFAR-10 datasets. I found out that Bayesian CNNs didn’t perform well as I expected. Frequentist CNNs achieved higher accuracy and took less time compared to Bayesian CNNs. However, there is an interesting feature of Bayesian CNNs. Bayesian CNNs incorporate uncertainty measure. Since Bayesian CNNs have the distributions of weights, the models can also output the distributions of outputs. Therefore, Bayesian CNNs could tell how confident the decision is made.

I hope to apply more architectures of Bayesian CNNs to more datasets in medical imaging projects because architectures and datasets have great influences on the performance. Also, I would like to try more prior distributions and learn how to determine which distributions are more appropriate.

I had great research experience in this project with Dr. Pascal Tyrrell’s guidance and other graduate students’ help. It was my first time to write scientific report. Dr. Pascal Tyrrell kept instructing me how to write the report and offered great advice. I really appreciated the guidance and enjoyed the unique research experience in the end year of my undergraduate life. I look forward to contributing to medical imaging research and more opportunities to apply machine learning methods!

Yiyun Gu

Amar Dholakia: Some Thoughts as I Wrap Up My STA498Y Project (and Undergrad!)

Hi everyone! I’m Amar Dholakia and I’m a fourth-year/recent graduate having majored in Neuroscience and Statistics, and am starting a Masters’ in Biostatistics at UofT in the fall of 2020. I’ve had the pleasure of being a part of Dr. Tyrrell’s lab for almost two years now and would like to take the opportunity to reflect on my time here.

I started in Fall 2018 as a work-study student, tasked with managing the Department of Medical Imaging’s database. A highlight was discussing and learning about my peers’ work, which sparked my initial interest in the field of artificial intelligence and data science.

The following fall, I began a fourth-year project in statistics, STA498Y under the supervision of Dr. Tyrrell. My project investigated the viability of clustering of image features to assess dataset heterogeneity on deep convolutional network accuracy. Specifically, I compared the behaviour of six clustering algorithms to see if the choice of algorithm affected the ability to capture heterogeneity.

My project started out with reaching out to my labmate and good friend Mauro Mendez, who had recently undertaken a project very similar to mine. He sent me his paper, which I read, and re-read, and re-re-read… It took me about four months to only begin to grasp what Mauro had explored, and how I could use what he had learned to develop my project. But months of struggle was definitely worth the “a-ha!” moment.

First I started by replicating Mauro’s results using Fuzzy K as a clustering to make sure I was on the right track. Reading, coding, and testing the very first time was a nightmare – I had some Python experience but had never applied it before. It took a lot of back and forth with Mauro and Dr. Tyrrell , a lot of learning, understanding, and re-learning what I THOUGHT I understood to get me on the right track. By the start of the Winter term, I had finally conjured preliminary results – banging my head on the wall was slowly becoming worth it.

Once I had the code basics down, getting the rest of the results was relatively smooth sailing. I computed and plotted changes in model accuracy with sample size, and heterogeneity in model accuracy with sample size, as captured by different clustering methods. My results for one model were great from the get go – I was set! I thought to challenge myself by generalizing to a second model – and that was far from easy. But by taking that extra challenge, I felt I learned more about my project, and importantly, how to scientifically justify my results. The results didn’t match up, and I had to support my rationale with evidence (from the literature). If I couldn’t find an explanation, I may have done something incorrectly. And lo and behold, my ‘inexplicable’ results were in fact due to human error – something I very painstakingly troubleshooted, but now I understand much more and justify.

Ultimately, we showed that regardless of clustering technique, or CNN model, clustering could effectively detect how heterogeneity affected CNN accuracy. To me, this was an interesting result as I expected vastly different behaviour between partition-based and density-based

clustering. Nonetheless, it was welcome, as it suggested that any clustering method could be used to assess CNN.

I struggled most with truly appreciating what my research aimed to solve. I attribute this partially to not being as proactive with my readings and questions to Dr. Tyrrell to really verify my understanding. And to be honest, exploring this project is still a work-in-progress – something I will continue learning about this summer!

My advice to any future students – read, read, read! Diving into a specific academic niche is truly a wonderful experience. The learning curve was steep and initially involved a lot of trying, failing, fixing, and then trying again. But this experience only reinforced my notions of “success through failure” and “growth through struggle”. It may be challenging at first, but with some perseverance and support from a wonderful PI – like Dr. Tyrrell – you’ll be able to accomplish so much more than you originally imagined.

Squeezing in a Little Time for ML this Past Summer: John Valen’s Experience

My name is John Valen. Having recently completed my undergraduate degree in statistics and economics here at U of T, and soon moving on to pursue my Master’s in statistics in Europe, the Medical Imaging Volunteer Internship program seemed almost tailored to my goal of getting valuable research experience within a constrained time window. Over the course of only several months this summer, I’ve had the pleasant and enriching experience of contributing ideas and code to the project that summer ROP student Wenda Zhao undertook for the dentistry department at U of T, along with the guidance and contributions of ML lab leader Hershel Stark.

Wenda’s blog post (see here) neatly summarizes the goal of this project, one whose aim is to determine the likelihood that a misdiagnosis may occur, depending on the degree of damage to the dental plate being used for X-rays. Contributions I’ve helped make in particular include:

– Creating sparse matrix representations of the grey scale X-ray images themselves in order to economize on memory and run-time performance
– Hand-engineering features: once the artifacts (damage such as scratches, dents,
blotches, etc) were segmented out via DBSCAN, they were characterized by a variety of different metrics: size (pixel count), average pixel intensity (images are grey scale), location (relative to the center of the plate image), etc. 

– Training a K-Means algorithm to cluster segmented artifacts from the dental plate images based on these hand-engineered features, whereby clustering them in this unsupervised manner gave us insight on their properties;

And much more. If you are not familiar with this machine learning lingo, then do not worry; I was hardly exposed to it myself before I started working in this lab. I went in knowing close to nothing practical and a whole lot theoretical, and came out knowing quite a little more in the way of the first one. Fine, a lot more: or
so I like to think. It may not seem clear how my contributions can be used in the future to help answer the ultimate question. The truth is, nothing is really clear at the moment. The project is still on-going and I intend to keep up with it, making contributions remotely to it while I am away in Belgium pursuing my Master’s degree. This is the greatness of it all, the amount of flexibility we have in answering these questions leaves a lot of room for creativity and contemplation. 

All in all, from my own perspective (which has been greatly expanded over the course of the summer), the volunteer program was a perfect means to experience the sheer amount of work that is enthusiastically undertaken by serious students in answering these important questions. I hope that I too can now consider myself at the very least climbing to their ranks while I move on to other and more numerous serious pursuits in my life. 

Good luck to you all, and do not underestimate yourselves.


John Valen

My Past and Future at U of T: Helena Lan’s Perspective

 



Hey everyone, it’s been a while since I posted here. In case you don’t remember me – my name is Helena Lan, and I started in Professor Pascal Tyrrell’s group as a ROP299 student. Fast forward to the present, I have finished my specialist program in pharmacology, and will be graduating with an Honours Bachelor of Science degree later this month! But if you think that I am finally leaving U of T – nope, my journey is not over yet. This August, I will be living my dream of many years as I start my MD training at U of T! As I prepare to begin the next chapter of my life, I wanted to share with you how my involvement in Prof. Tyrrell’s group paved the way for me achieving my goal today.

At the end of my first year of undergrad, I connected with Prof. Tyrrell and took on a project investigating how the choice of non-invasive imaging modality for diagnosing carotid stenosis impacts patient care (check out my experience here https://www.tyrrell4innovation.ca/2014/08/helena-lan-summer-2014-rop.html).
Afterwards, I continued on as a research assistant, where I ­explored the need for statistics and research methodology training in the medical imaging department.  My early research endeavours showed me that research was not just pipetting; there is a diversity of research that can drive innovations and improve patient care. 
That being said, I also wanted to experience working in a wet lab setting. So upon completing my second year of undergrad, I ventured to the Karolinska Institute in Sweden to investigate the tumour killing mechanism of Natural Killer cells (find out more about my project here https://www.tyrrell4innovation.ca/2015/02/who-is-going-to-karolinska-institute.html). After a summer in basic science research, I decided to switch gears into translational research, where I worked on strategies to augment the therapeutic utility of stem cells and enhance the drug delivery platforms at Prof. Jeff Karp’s lab at Brigham and Women’s Hospital, Harvard Medical School. After I returned from Boston, my passion for discovering ways to improve existing treatments for diseases led me to my current work at Dr. Albert Wong’s lab at CAMH, where I am assisting with the characterization of a novel animal model for schizophrenia with the ultimate goal of using it as a screening platform for new anti-psychotics.
In my experiences as a researcher, I’ve always been very excited at the prospect that what I am working on right now may be brought into the clinic sometime down the road and offer benefits to patients. Then one day, I thought to myself, “How rewarding would it be if I can get involved in patient care, where I can directly impact the life of the person sitting in front of me?” With this idea planted in my mind, I decided to shadow a physician. As I observed how a doctor applies their scientific knowledge and the findings from medical research to figure out ways to best help their patients, my attraction to medicine gradually evolved. For a long time, my goal in life has been to make a positive impact on other people’s lives. But after that shadowing experience, I realized that I wanted to do so through taking on the role of a clinician.
I am incredibly grateful to the U of T medical school for giving me the opportunity to pursue my dream, as well as the pharmacology department and New College for their recognition of my undergrad academic achievements with the Dr. Walter Roschlau Memorial award and the Tricia L. Carroll Memorial Prize in the Life Sciences. But more importantly, thank you to U of T for the unforgettable undergrad experience. Not only was I able to immerse myself in fascinating science and interesting research, I was also connected with mentors who provided unconditional support to me along my journey. Even though the ROP project I worked on under the supervision of Prof. Pascal Tyrrell and Dr. Eli Lechtman ended years ago, the two of them have provided invaluable mentoring to me even to this day.
University can seem arduous at times, and it is almost inevitable that we run into obstacles here and there. But no matter how difficult the circumstances may be, never, ever, lose sight of your goal. Surround yourself with people who cheer you on, and invest the work that is necessary to reach your ambition. And one day, your dream will come true!  
All the best,
Helena Lan

Lee Radigan: A Reflection on my (6th) Year as an Undergrad at the University of Toronto

My name is Lee Radigan and I am a non-degree student pursuing admittance to the Biostatistics Masters program at the Dalla Lana School of Public Health.  After returning for my 6th year studying statistics at The University of Toronto, I thought that this was a perfect time to reflect on my progress.
Since September, I have been working under Dr. Pascal Tyrrells guidance on a project aimed at helping the Department of Medical Imaging report agreement in their research.  To do this, I created a flow chart to help guide the reader towards the proper method of agreement.  Along with this, I conducted a simulation looking at a specific question pertaining to the Department.
Initially, I was tasked with combing through various papers on the theory of agreement and making sense of all the different published work that was out there.  There are many different approaches and different ways of looking at reporting agreement, so it was quite difficult to figure out when and where to properly use every single approach.  After reading and re-reading each paper, as well as consulting the MiData team, I started to develop a thorough understanding of what agreement was, why it is important to report it, and how to go about reporting it appropriately.
Next, a flow chart was required to summarize what I had learned from the literature.  This was not an easy task, because it forced me to dig really deep and make sure that every node in my flow chart was well thought out and appropriate.  After many iterations and adjustments, I created a detailed chart that walks the reader from their initial research question up to the required agreement statistic.
My final task was to conduct a simulation that would test the question: Can a group of less experienced student raters be as accurate as a smaller set of more experience expert raters?  And if so, how many students?  And under what conditions?  This was a very fun and informative task for me as I was able to conduct my first simulation.  During this experience, my biggest difficulty was justifying my choices of parameters within the simulation.  When conducting a simulation you have freedom to choose how it is going to work, but you must be careful to be able to back up each and every parameter choice.  The simulation ended up showing that: the larger the disparity between the rating errors of the student and expert raters, the more students it takes to match the accuracy of the experts, confirming my intuition.
There are many things that I wish to expand on with respect to my project in future.  I want to create a user friendly app that will be even easier and more compact than my flow chart.  Additionally, I want to try to get my paper published.  To do this I will need to look further into my simulation and consider a more broad range of student/expert scenarios that likely will occur in practice.  I will also need to further refine my definitions and understanding of each concept of agreement.
This year has truly been the best of my life and I can largely attribute that to Dr. Pascal and the MiData team.  I look forward to contributing to Medical Imaging research and to many more learning experiences.
Time to enjoy the summer as I embark on yet another exciting experience as a student Statistical Analyst at the CAMH Nicotine Dependence Clinic as a summer placement!
Lee Radigan

Lessons Along the Way

https://betakit.com/startupcfo-explains-the-long-windy-road-to-a-closed-funding-round/
 
 
With summer almost here, it’s a good time to reflect on lessons learned from the academic year gone by. Since September, I’ve been working under Dr. Pascal Tyrrell’s supervision on a systematic review (SR) project investigating sample size determination methods (SSDMs) in machine learning (ML) applied to medical imaging. Shout out to the Department of Statistical Sciences where I completed my independent studies course! Here, I share important lessons I learned in the hopes that they may resonate with you.
 
Despite being a stats student (as you know from my previous posts!), I was initially new to ML and confronted with the task of critically reviewing theoretically-dense primary articles. I came to appreciate the first step was to develop a solid background – starting from high-level YouTube videos and lessons on DataCamp, to reading ML blogs and
review articles – all until I was confident enough to evaluate articles on my own. For me, the key to learning a complex subject was to build on foundational concepts and keep things as clear as possible. As Einstein once said: “If you can’t explain it simply, you don’t understand it well enough”.
 
Next, it was time to conduct a systematic search. The University of Toronto library staff were especially helpful at guiding me in use of OVID Medline and Embase, databases with methodical search procedures and a careful search syntax relying on various operators. To be thorough, we also sent a request out to the rest of our research team, who hand-searched through their own stash of literature. Along the way, we garnered support from the university, successfully receiving the Undergraduate Research
Fund grant. The lessons for me here? The importance of seeking expert help where appropriate, and that being resourceful can pay off (literally)! Finally, I valued our strong team culture, without which none of this would have been possible.
 
While working on the SR, I also conducted a subsampling experiment using a medical imaging dataset, testing the effect of class imbalance on a classifier’s performance. Hands-on/practical experiences are critical in developing a more nuanced understanding of subject material – in my case, an understanding that translated to my SR.
 
So now you are probably wondering about the results! The subsampling experiment helped us develop a model for the deleterious effect of class imbalance on classification accuracy and demonstrated that this effect was sensitive to total sample size. Meanwhile in our SR, we observed great variability in SSDMs and model assessment measures, calling for the need to standardize reporting practices.
 
That was a whirlwind recap of the year and I hope some of the lessons I learned resonate with you!
 
See you in the
blogosphere,
 
Indranil Balki
 
A special thanks to Dr. Pascal Tyrrell, as well as Dr.
Afsaneh Amirabadi & Team

Ethics Schmethics?

 

Today, it may seem obvious that the first step of any research project should be to complete a proposal for ethics review. But why do we need ethical standards? While helping to complete an ethics form for a project I’m working on, I wondered if scientists perhaps made more ‘progress’ before ethical considerations became commonplace. Even if this was the case, research is certainly better now, when institutions and procedures protect patients’ and research subjects’ rights. 

It also seems that scientific research in the 18th and 19th centuries tended to be somewhat more haphazard than it is now, and almost certainly less ethical. For example, Dr. Edward Jenner tested his smallpox inoculation hypothesis for the first time on an eight-year-old
boy in 1796, with little preliminary understanding and no certainty that the patient would not be severely harmed.

Scientists were often fairly independent, acting based on their own curiosity to advance knowledge. Fortunately, research standards have evolved significantly since then. Ethics have been a major part of the transition, as ethical standards help to ensure that scientific research does not cause harm to researchers or subjects. The shocking Stanford Prison Experiment, just one example, shows that physical and psychological damage can occur if study participants’ rights are not upheld through ethics. College students with no criminal record were asked to play the role of prisoners and prison guards, the ‘guards’ became brutal and cruel, while the ‘prisoners’ became stressed and depressed. The experiment was terminated early, after only six days.

Fortunately, much has changed since the emergence of modern science in the 20th century. The current structure of research, including working in teams and undergoing peer review, helps to ensure a high standard of practice. Nevertheless, ethical issues in science remain. Researchers who work with human participants can become quite focused on the minutiae of their work, so Research Ethics Boards have an important mediating role. They provide an experienced, unbiased viewpoint that weighs the potential benefits of the research against any harm that may come to participants. Even if an ethical review sometimes slows the pace of scientific progress, it provides an essential foundation and structure for research, to the benefit of participants and researchers alike.  





Julia Robson

2nd year student at U of T

All the World’s a Stage

For journalists, authors, bloggers and tweeters, sharing articles has never been easier. Indeed, the public expects to be able to read articles about world events almost in real-time. For example,
the New York Times Twitter account was updated nine minutes ago
, and National Geographic tweeted three minutes ago. This expectation of speediness applies equally to scientific advances as it does to international affairs.
As an avid reader of online news, I would be the last to complain about being able to access such a vast amount of information. But there is something particularly noteworthy about information presented by a visible human. Perhaps that explains the persistence of televised news in the age of Twitter. 

Maybe it also explains the popularity of other media sources like TED talks, which often explain complex ideas in an engaging and understandable format. A personal favourite is “The best stats you’ve ever seen” by Hans Rosling. In his talk, Rosling explains the importance of little-known global public health data that shows the progress (or lack thereof) made by different countries over the past few decades. 

A more recent talk on a similar topic is also informative. One would be hard-pressed to find a paper or article that presents the same information with as much clarity and appeal.

In addition to numerous (maybe too numerous?!) TED talks, I have recently experienced the value of human-to-human information transfer. At the beginning of my ROP project in September, I was lucky to be able to hear about previous students’ research in person. I think it helped address the complexity of the work, but also conveyed its importance and the effort that had gone into it. Thanks Kiersten!
I’m not sure if information is generally more effective this way, but it is almost certainly more memorable. In any case, it has definitely worked for the 3.5 million subscribers to CrashCourse’s YouTube channel, where one can learn about anything from astronomy to macroeconomics.
For me, learning more about how researchers give and receive qualitative information to and from their subjects has allowed for a more well-rounded understanding of information transmission in the digital age.  But I think researchers andthe media have a lot to learn from each other. Communication is key for both, so understanding how others best absorb and respond to information can be instrumental.
That’s all for now, Julia!

Research Opportunity Program: Cost Effectiveness and Imaging Carotid Stenosis

ROP Research Forum March 5th, 2015 – Sylvia Urbanik

Sylvia is a second year student studying cell and molecular biology. She is currently finishing up her Research Opportunity Program that spanned the fall and winter semesters, and also recently represented us at the March 5th ROP research forum along with her partner and predecessors.

 
Her project dealt with cost-effectiveness analysis techniques for diagnostic imaging modalities, with a specific focus on carotid artery stenosis. She explored the different types of analyses used in assessing cost effectiveness. She examined how factors such as sensitivity and specificity of diagnostic tests (imaging modalities in this case) can affect the analysis, and conducted literature searches in order to find these variables in order to incorporate them into a cost effectiveness model.​

Well done, Sylvia!

Stay tuned, next Kevin Chen will be modeling with TreeAge…

See you in the blogosphere,

Pascal Tyrrell

Michener Institute Series: Princess Margaret Hospital, Toronto, Ontario



As first year Radiation Thearpy students here a The Michener Institute, we are currently in our 4th week of clinical placements! As promised, here’s a little update about the experiences Jennifer and Ori are going through at Princess Margaret Hospital.

Jennifer: I’ve been placed in Unit 10 which specializes in treating patients with Genitourinary, Gynae and Lower Gastrointestinal cancer.

Ori: I’m on Unit 14 and we treat breast cancer and palliative cancers.

We are proud to say that we are enjoying our experience here. Our duty as students in training is to follow the radiation therapist and learn what they do. The job of a therapist is to treat cancer using a machine called Linear Accelerator (Linac) to deliver ionizing radiation. Patients will typically come once a day for the next couple of weeks, so we see the same patients every day and therefore really get to know our patients well. There is a fair amount of patient interaction, which is one of our favorite parts of the job. Along with patient interactions, we also get to use the equipment, which mainly includes operating the Linac machine (the machine that delivers the radiation) and taking X-rays or CT scans to make sure the patient is in the right position. Every day is a new experience and we are constantly learning new skills. We get a better insight of the patient’s perspective during the entire span of their radiation treatment. For example most patients in unit 10 are required to have a full bladder and empty rectum. Having to hold their pee can be quite difficult for some patients, especially when there are delays, which pushes Unit 10 to be a very fast paced environment. Overall our first 4 weeks of clinical has been an exceptionally valuable experience and we’re looking forward to our next 4 weeks!


Until next time!

Jennifer and Ori