How a Lawsuit Against UC Berkeley Represents America's Stagnation
Mancur Olson's writings explain this very clearly. Unless some dramatic event disturbs the rent seeking coalitions you describe the situation won't change.
It's getting worse, not better, for those on the wrong side of NIMBY'ism as we apply aggressive levels of monetary stimulus to a supply-constrained system. It simply will not change and yet I hear little talk of how we should adapt given that reality.
Dietrich Vollrath very recently published "Fully Grown - Why a Stagnant Economy is a Sign of Success." It´s a great defense of the "asset holding" class. Even the Fed benefits.
NIMBYs are destroying everything
I agree with your diagnosis that the current situation is absurd and very bad for younger and poorer people. But the logic of what the NIMBYs are doing here makes sense given the incompetence of the local politicians and their insistence on putting political stunts ahead of basic functions of government like public safety. The irony, of course, is that many NIMBYs are happy to vote for these politicians, albeit only so long as they do their social experiments elsewhere (usually poorer neighborhoods) and not in, say, Pacific Heights and the Berkeley Hills.