I just keep mine attached to my keys with a “dongle dangler” and this has never been a problem.
People love to complain about how inconvenient losing the headphone jack is (especially people with phones that still have it!) but that’s just because this is the internet, where everybody complains about everything.
