Solved.
Some things I tried, not sure which one caused the fix:
- downgraded to macOS Big Sur, so mac became fresh, no dependency clutter, no Xcode command line tools.
- I uninstalled homebrew (it had it’s own zlib, this was likely the cause of the issue).