There has got to be an option for a fleece-like material that doesn't involve significant pollution or significant animal suffering. At some point, fleece has just gotta be phased out if we can't produce it without enormous downsides.
Not sure if you count insects into your moral calculus, but silk farming only involves caterpillars and mulberry trees. It's warmer, softer, lighter, and way less smelly than plastic, so it doesn't need as much washing, so less environmental impact. I can wear a silk t-shirt for about a week before it gets smelly, most of the time, whereas plastic shirts sometimes need to be washed twice after a single wearing to get the smell out!
The fibers are stronger than polyester, so if you get a thick silk fabric, it will last a lifetime. Unfortunately, it's absurdly expensive, so even though the total cost of ownership isn't terrible if you get a thick fabric because it will last forever, it's a ton of money up front. Additionally, most places sell super thin silk to keep the price down, which means it doesn't last, so you have to shop around and find the good thick stuff.
I love silkliving.com's 100% silk line, which even includes a fleece hoodie! I'm not associated with them other than as a satisfied customer that paid full price.
The show Clarkson's Farm on Amazon shows some sheep farming. It looks absolutely humane. Sheep grow wool, get sheared when it gets too hot. The sheep are not killed in the process. If you are worried about also tangentially supporting meat, I am sure there are some no kill wool farms that do not sell their lambs or sheep for slaughter.
Livestock are also necessary for regenerative farms that do not require outside sources of fertilizer. Hardline veganism requires petrochemical fertilizer.
If I understand correctly, cow hides are largely a by-product of beef production, and reducing leather use would not have an effect on animal suffering.
Apple stopped using leather and was criticised for environmental damage at least in the short-to-medium term (underutilisation of existing materials, new demand for plastics and synthetics).
