I have been hesitant in the same way. I have not until recently understood where the passkeys are stored, and then understood that they then become a “thing tou have”. The passkey is stored on my computer, I prove I have access to my computer, and that is the authentication.

I am still hestitant since it often is just one factor, and I feel I have to manage them much more carefully.