In the past, the quantum-to-classical transition of inflationary cosmological perturbations more focused on explaining today’s classical observations which originate from quantum vacuum fluctuation, and such a transition process is often described by the decoherence obtained by tracing out the unobserved degree of freedom which nonlinearly interacts with observable modes. Recently, there is a growing interest of the quantum-information aspects of cosmological perturbations, including the non-classicality of scalar perturbation and primordial gravitons, and quantifying the decoherence is now helpful to estimate the feasible regime to probe such quantum aspects. In this talk, we discuss the decoherence of cosmological perturbations by the minimal gravitational interaction, focusing on the dominated contribution by a slow-roll unsuppressed boundary term of the gravity action. We also point out the possibility that the gravitational origin of inflationary decoherence may shed light on the non-classicality of gravity, by comparing the decoherence rate in a Schrödinger-Newton semi-classical gravity.