By: Kaisa Liimatainen
Reconstructing 3D Data
For a few years, the Bioimage informatics research group where I work as a PhD student, has been working with 3D histology data. This data is basically stacks of consecutive images (a.k.a. serial sections) from a single tissue sample. These images were aligned with image processing techniques to create 3D reconstruction of the data. Now, we have this cool 3D histology data with properly aligned image stacks, but it is not easy to visualize. If we stack the images, we can see the general shape of the object but not what is inside – we are just creating 2D projections of 3D data. If we look at the images one at a time, we can see whole images but not the 3D shapes.
Visualizing Tumor Hotspots in VR
Since I have always enjoyed visualizing scientific data, I was intrigued when we started discussing ideas for 3D histology and volumetric data visualization. The project started with 3D modeling of mouse prostates with tumor hotspots and 3D printing these models. Transparent plastic was used for tissue and solid colored for tumors, so tumor hotspots were clearly visible and tumor locations in different samples could be compared. Results were encouraging but naturally there are limitations in what we can present with these physical models. So, the next step was to bring the models to virtual reality and embed more information in them. After all, virtual reality is the obvious choice when we want to visualize data in truly three dimensions.
The Virtual Prostate
I started to develop the VR application with the working title “The virtual prostate”. After a few months of programming, I had a preprocessing pipeline which generates 3D models and positions all the data correctly, and a VR application created with Unreal Engine 4 (Epic Games) which dynamically loads the data to VR environment. Previously I had been working mainly with image processing and deep learning. Luckily, nowadays popular game engines have VR templates that make it easy to create VR applications, even with little experience in game development.
The Application and Experience
The virtual prostate includes two levels, the main level where the whole organ (prostate) lies, and a level for studying user-selected object (tumor) in detail. Spatial measurements, which at the moment consist of computational features extracted in previous work of the group, and full-resolution serial sections are embedded in both levels. Visualization options can be controlled via menus. More details can be found here.
It is exciting to walk inside large prostates and interact with measurements, tumors and different prostate samples in virtual reality. When people try the application for the first time, they are often quite amazed. Just experiencing virtual reality for the first time can be an astonishing experience for some, but this is also a method for studying histology data that people have not encountered. The data itself is very interesting, so even though I have spent a lot of time testing the VR world, I still enjoy just laying back and immersing myself in the world of virtual 3D histology.
The Next Step…
The VR application is currently at a prototype phase, and there are a lot of possibilities for future development. The next step will be modeling of the inner structures of the prostate, like urethra and glands, and adding them to VR to create a comprehensive prostate model. This would enable creating a tool for teaching (or a biological game), where one would have to find the tumors from a range of models inside the prostate. Even though the samples used in the prototype are prostates, we can visualize 3D histology data from other origins, too. Also, we could go deeper into detail, for example plot single cells and allow the study of their properties. Or we could add annotation and measuring tools to create a tool for researchers and pathologists.
The Future of VR in Biomedical Research
Adopting VR for frequent use by scientists, pathologists and teachers is still far away. Wearing the VR equipment is not yet very comfortable, so using it on a daily basis might not work for most. Also, getting used to this new reality takes some time. Still, VR is a technology of the future under fast development, with constantly growing numbers of users. It is starting to emerge as an acknowledged visualization tool in biomedical research, so now is a great time to develop VR applications as novel interfaces for fields like digital pathology and cancer research.
Tumor Genetics Blog 