Avatar meets molecule

I explored how to make videos of someone handling molecules in VR.

Goal

Let an avatar look like handling molecular models.

Results

Methods

• View molecules by a VR molecular viewer such as Nanome.
• Act as the avatar using Virtual Motion Capture.
• Synthesize these two by LIV.

This yields an avatar version of this video from Nanome inc.

Advantages

• With the player-image synthesized with molecules, explanations on molecules will become more intuitive.
• Avatar-based MR-synthesis is easier than real-camera chroma-key synthesis.
• Wearing an avatar allows the “resolution” of the player images to be similar to that of the molecular models.

Problems unsolved

Currently, some molecular models cannot be chromakey synthesized in this way, which is a known behavior of the molecular viewer.

Updates(28-Feb-2019)

I found Nanome downloaded from Oculus does not suffer from the problem above.

A Note for the updated technique is written here (but in Janapese…):

https://yakomaxa.github.io/minus_kTlogP/post/nanomevmcunreal_ja/

Perspectives

I’d like to refine this kind of things to make presentation or explanations about molecular structures.

Conclusion

• Nanome + VMC + LIV partially realizes what I want to do.
• Since the combination is not perfect, I may seek for alternatives.

Q&A

Q1. Which VR molecule viewers have you tested?

• Nanome is the best as of now because it’s a native VR molecular viewer. Probably, it’s the only native VR molecule viewer currently available.
• Chimera-X has the VR mode and shows cool models. However, controllability is not so good when compared to the non-VR mode. This is already discussed in their paper.
• Pymol seems to have a VR plugin. I’m tring this now.
• I tested some other seemingly-VR-viewer applications developed by individual researchers, but unfortunately they did not work well in my environment and for my purpose.

Q2. What VR instruments are you using?

• I use Oculus Rift and Touch.

Q3. How was this avatar made and used?

• I used VRoid Studio to compose the avatar, which can export a specific type (.vrm) of humanoid models.
• To playact as the avatar, I used Virtual Motion Capture.

References

Softwares

• Nanome is a native VR molecular viewer.
  • https://nanome.ai/nanome/
• Virtual Motion Capture is software to make virtual camera images for avatar-based MR-synthesis.
  • @sh_akira is the developer of this cute software.
  • You can donwload the software here, and also donate to the developer
  • The VMC homepage
• LIV is software for chroma-key based MR-synthesis.
  • https://liv.tv/
• VRoidStudio is software to compose a VRM-formatted humanoid model.
  • https://studio.vroid.com/

Previous study

• How to bring molecular structure data into MikuMikuDance (A Japanese blog post).
  • http://d.hatena.ne.jp/biochem_fan/20110829/1314627892
  • My idea is virtually equivalent to this. I read this when I was an undergraduate student.

Papers

• Molecular Visualization on the Holodeck
  • https://www.sciencedirect.com/science/article/pii/S002228361830696X

---

This is English version of http://yakomaxa.github.io/minus_kTlogP/post/nanomevmcliv_ja/