It's really not hard to understand... Extreme undersupply due to regulation/bureaucracy and extreme overdemand due to lack of opportunities elsewhere.
I live in the capital city of my country, just checked the situation in my hometown - average jobs pay 1/2 and the job I do would pay 1/5 - while the rental and property prices are just 10-20% lower; schools are terrible, no night life, no interesting restaurants, literally zero startups of any kind, people are (voting for) racists and conservatives... What the hell would I do there? I like my nice cheap (compared to my buying power) apartment in the middle of a nice progressive cosmopolitan city.
I agree with nearly everything you say. I'm of the opinion that regulation/bureaucracy are being scapegoated to an extent. Supply of knowledgeable builders and development finance is not very elastic and does not increase easily even when there is less regulation.
The real reason I feel is the higher risk = higher reward equation is broken.
Building fewer units means increasing scarcity, allowing increased prices despite there being lower (due to building less houses). Developers are chasing good profit margins rather than revenue. Less regulation just means even better profit margins for these developers.
Where I live there would be 100 people waiting to buy an apartment you just built, giving you absurd amounts of money for it. There is no reason to build less than the absolute maximum possible. There are many developers and builders who want to do it, could do it and eventually will do it - but getting an approval takes 11 years.
The timeline is a killer, because 11 years (and more) from beginning to end means you could end up with your new building opening just as the area goes into a slump.
When the turnaround time is shorter, the builder is taking much less risk, and so they're more willing to build. (They can still overextend as happened during the last housing bubble, mind you.)
I just spent 10 days in the town I grew up in north of Boston to take care of a family member. Was completely surprised that people are putting homes on the market in my old neighborhood for nearly $1 million. Rents are not even that far off from what you could get in Bushwick, Brooklyn and you would need to own a car. The restaurants were more expensive than my regular spots in Brooklyn, and just way way worse quality. It's not a bad place to live, but it's ridiculous that everything is this expensive everywhere.
My parents always tell me I waste so much money by going to restaurants all the time and that taxis are absurd luxury - yeah but my taxi costs 3x less than in the hometown, my restaurant meal is 1/2 cheaper and much better than anything I could make myself (or buy anywhere in the hometown) - and in this city I can actually spend the time working and making a lot of additional income instead of killing time in a shopping mall and cleaning.
I live in the capital city of my country, just checked the situation in my hometown - average jobs pay 1/2 and the job I do would pay 1/5 - while the rental and property prices are just 10-20% lower; schools are terrible, no night life, no interesting restaurants, literally zero startups of any kind, people are (voting for) racists and conservatives... What the hell would I do there? I like my nice cheap (compared to my buying power) apartment in the middle of a nice progressive cosmopolitan city.