Little bit of both. From my own anecdata, most people I knew left usenet due to spam problems. Most of the people who did not were primarily the ones using it for binaries. And then yes, the binary angle started the trend where ISPs stopped offering it altogether, which even further reduced the likelihood that people would use it.

And then there were weirdos (sickos?) such as myself who hung on for an absurd amount of time and never once used it for binaries

I had to selectively stop carrying parts of USENET (I ran what was briefly a fairly important node globally) because of the volume of the binaries, and various sex and bestiality groups (probably still including some badly-scanned ASCII-rendered images!