Enhancing Depth Consistency in Augmented and Diminished Reality : Techniques and Evaluations Using RGB Imagery

Israa Shakir Seger; Amjad Mahmood Hadi; Alaa Abd Ali Hadi

doi:10.61132/konstruksi.v3i1.685

Authors

Israa Shakir Seger University of Muthanna
Amjad Mahmood Hadi Al Muthanna University
Alaa Abd Ali Hadi Al-Furat Al-Awsat Technical University

DOI:

https://doi.org/10.61132/konstruksi.v3i1.685

Keywords:

Augmented Reality, Diminished Reality, Depth Consistency, RGB Imaging, Error Measurement

Abstract

Augmented Reality (AR) applications are rapidly gaining popularity across various industries, including education and marketing. By integrating real-world environments with virtual objects, AR enhances user understanding and information display for products. This paper explores Diminished Reality (DR) techniques, which are used to visually remove real objects from AR environments. Despite growing interest, much of the DR research predominantly focuses on maintaining consistency between real and virtual elements, particularly in texture handling on marker areas. Our study addresses the preservation of depth consistency using edge detection and planar segmentation to construct a depth map, essential for developing effective DR methods. We introduce a two-stage process involving depth mask construction, each stage equipped with error measurement for iterative refinement. Our proposed techniques, Planarity and Boundary Depth, are evaluated on a dataset of high-quality RGB images captured by digital cameras. Experimental results validate the effectiveness of our methods across various performance metrics, confirming the practicality of our approach in enhancing AR experiences.

References

Cheng, Y. F., Yin, H., Yan, Y., Gugenheimer, J., & Lindlbauer, D. (2022). Towards understanding diminished reality. In Conference on Human Factors in Computing Systems - Proceedings (pp. 3517452). Association for Computing Machinery. https://doi.org/10.1145/3491102.3517452

Chua, T. S., Ngo, C.-W., Kumar, R., Lauw, H. W., & Lee, R. K.-W. (2024). Companion proceedings of the ACM Web Conference 2024 (WWW ’24 Companion): May 13-17, 2024, Singapore, Singapore. The Association for Computing Machinery.

Dhamo, H., Navab, N., & Tombari, F. (n.d.). Object-driven multi-layer scene decomposition from a single image.

Dhamo, H., Tateno, K., Laina, I., Navab, N., & Tombari, F. (2018). Peeking behind objects: Layered depth prediction from a single image. Pattern Recognition Letters, 132, 191–202. https://doi.org/10.1016/j.patrec.2019.05.007

Eskandari, R., & Motamedi, A. (n.d.). Diminished reality in architectural and environmental design: Literature review of techniques, applications, and challenges.

Fujii, R., Hachiuma, R., & Saito, H. (2021). RGB-D image inpainting using generative adversarial network with a late fusion approach. Retrieved from http://arxiv.org/abs/2110.07413

Gkitsas, V., Sterzentsenko, V., Zioulis, N., Albanis, G., & Zarpalas, D. (n.d.). PanoDR: Spherical panorama diminished reality for indoor scenes.

Hadi, A. A. A., & Hadi, A. M. (2024). Improving cybersecurity with random forest algorithm-based big data intrusion detection system: A performance analysis. In AIP Conference Proceedings (Vol. 232). American Institute of Physics. https://doi.org/10.1063/5.0191707

Hadi, A. M. (2024). Enhancing MRI brain tumor classification with a novel hybrid PCA+RST feature selection approach: Methodology and comparative analysis. International Journal of Computational and Electronic Aspects in Engineering, 5(2), 116–130. https://doi.org/10.26706/ijceae.5.3.20240805

Kán, P., & Kafumann, H. (2019). DeepLight: Light source estimation for augmented reality using deep learning. Visual Computer, 35(6–8), 873–883. https://doi.org/10.1007/s00371-019-01666-x

Kari, M., et al. (n.d.). TransforMR: Pose-aware object substitution for composing alternate mixed realities. Retrieved from https://developer.apple.com/documentation/arkit

Kikuchi, T., Fukuda, T., & Yabuki, N. (n.d.). Automatic diminished reality-based virtual demolition method using semantic segmentation and generative adversarial network for landscape assessment.

Kobayashi, K., & Takahashi, M. (2024). Real-time diminished reality application specifying target based on 3D region. Virtual Worlds, 3(1), 115–134. https://doi.org/10.3390/virtualworlds3010006

Kulshreshtha, P., Lianos, K.-N., Pugh, B., & Jiddi, S. (2022). Layout-aware inpainting for automated furniture removal in indoor scenes. Retrieved from http://arxiv.org/abs/2210.15796

Li, P., Liu, L., Schönlieb, C.-B., & Aviles-Rivero, A. I. (2024). Optimised ProPainter for video diminished reality inpainting. Retrieved from http://arxiv.org/abs/2406.02287

Li, Z., et al. (2022). Color-to-depth mappings as depth cues in virtual reality. In UIST 2022 - Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (pp. 3545646). Association for Computing Machinery. https://doi.org/10.1145/3526113.3545646

Ma, F., & Karaman, S. (2017). Sparse-to-dense: Depth prediction from sparse depth samples and a single image. Retrieved from http://arxiv.org/abs/1709.07492

Macedo, A. C. F., Apolinário, A. L., & Jr, A. (2015). Occlusion handling in augmented reality: Past, present, and future.

Mansoor, R., Sasse, H., Ison, S., & Duffy, A. (2015). Crosstalk bandwidth of grating-assisted ring resonator add/drop filter. Optical and Quantum Electronics, 47(5), 1127–1137. https://doi.org/10.1007/s11082-014-9969-0

Ming, Y., Meng, X., Fan, C., & Yu, H. (n.d.). Deep learning for monocular depth estimation: A review.

Mori, S., Ikeda, S., & Saito, H. (2017). A survey of diminished reality: Techniques for visually concealing, eliminating, and seeing through real objects. IPSJ Transactions on Computer Vision and Applications, 9(1). https://doi.org/10.1186/s41074-017-0028-1

Mori, S., Shibata, F., Kimura, A., & Tamura, H. (2015). Efficient use of textured 3D model for pre-observation-based diminished reality. In Proceedings of the 2015 IEEE International Symposium on Mixed and Augmented Reality Workshops, ISMARW 2015 (pp. 32–39). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ISMARW.2015.16

Ramamonjisoa, M., & Lepetit, V. (n.d.). SharpNet: Fast and accurate recovery of occluding contours in monocular depth estimation.

Valentin, J., et al. (2018). Depth from motion for smartphone AR. In SIGGRAPH Asia 2018 Technical Papers (pp. Article 3275041). Association for Computing Machinery. https://doi.org/10.1145/3272127.3275041

Wasenmuller, O., Meyer, M., & Stricker, D. (2016). Augmented reality 3D discrepancy check in industrial applications. In Proceedings of the 2016 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2016 (pp. 125–134). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ISMAR.2016.15

Yang, J., Ye, X., Li, K., Hou, C., & Wang, Y. (2014). Color-guided depth recovery from RGB-D data using an adaptive autoregressive model. IEEE Transactions on Image Processing, 23(8), 3443–3458. https://doi.org/10.1109/TIP.2014.2329776

Zhang, Y., Scargill, T., Vaishnav, A., Premsankar, G., Di Francesco, M., & Gorlatova, M. (2022). InDepth: Real-time depth inpainting for mobile augmented reality. Proceedings of the ACM Interactive Mobile, Wearable, and Ubiquitous Technologies, 6(1), Article 10. https://doi.org/10.1145/3517260