openoms proposed the following fixes in Telegram to continue using pip:
sudo rm /usr/lib/python3.*/external management
Christian Decker added some context on Core Lightning’s use of Python in the build process.
Python is not listed as a direct dependency because it is not a runtime dependency, but it is used in various build steps. The error you received occurs because root owns the pip environment after installing it through the package manager. We recommend using virtualenv to install build dependencies. The steps on how to set this up depend on your OS and Python setup, so why is it left undocumented (it’s not specific to CLN, and the documentation on how to create a virtualenv can be bitrotten, so I wouldn’t trust it anyway) (This is an external project.) You just need to find a good spot to cut off the long tail, and for venv, that’s where it felt right.
Python dependency management has grown very organically, which explains the confusing mix of different ideas and models. Typically there is a system environment that can be extended with a user environment overlaid on it. However, this will not work if your user package conflicts with a system package. It looks like we are using a newer package version which may be conflicting and that is causing this error. virtualenv (initialized with)
--no-site-packages) creates a blank page without any strange combination of system and user packages.