Install python package in a separate directory by ProxyBroker example

Installing a python package into a separate directory makes the package isolated from other packages. It is more transparent and in some cases very convenient. But this process requires some experience and is complicated.

The method below describes how to install a python package in a separate directory without using third-party utilities like pip.

Before we begin, decide which version of python our package uses. I will install proxybroker, it uses python version 3.5 or higher.

Install or Update python version that is required:

sudo apt install python3

On a package page (for example, on GitHub, or in documentation) there is information about dependencies (additional libraries that a package requires to work).

In the case of proxybroker, it looks like this:

python 3.5, aiohttp, aiodns, maxminddb requirements for ProxyBroker
Requirements for ProxyBroker

We can find these packages in Debian or Ubuntu repository. Therefore, we can do without pip:

sudo apt install python3-aiohttp
sudo apt install python3-aiodns
sudo apt install python3-maxminddb

ProxyBroker is available on GitHub, so i need to install git to download proxybroker. Or we can use some online services to download from GitHub. I use –no-install-recommends option to not install dependencies for developers:

sudo apt install --no-install-recommends git

Now we can download the package using the link from GitHub:

cd /tmp
git clone https://github.com/constverum/ProxyBroker.git

Directory will be downloaded, go inside and make sure that there is a script inside to install the package:

cd ProxyBroker
ls
...
... ... setup.py ... ...   

Decide where the package will be located. Is is also necessary to create {PACKAGE}/lib/python directory inside.

Suppose I want to store packages in ~/.local/packages/ directory. Therefore, for proxybroker, I will create it like this:

mkdir -p ~/.local/packages/ProxyBroker3.2/lib/python

All necessary directories will be created. ~/.local/packages/Proxybroker3.2/ is a directory in which proxybroker will be installed.

Now we can install the package with the following command. Below I will explain the parameters, so don’t rush to execute this command, learn about the parameters and change them to your own:

PYTHONPATH=~/.local/packages/ProxyBroker3.2/lib/python python3 setup.py install --home=~/.local/packages/ProxyBroker3.2 --install-scripts=~/.local/bin --record ~/.local/packages/ProxyBroker3.2/installed_files.txt

First (need to replace with your lib/python path), we pass the path to the package libraries using PYTHONPATH=~/.local/packages/ProxyBroker3.2/lib/python. This is individual for this package, its isolated environment.

Second (decide python 2 or 3), we use “python3 setup.py install”. This is the basic command for installing packages with setup.py. Notice we use python3.

Third (replace package path with yours), –home=~/.local/packages/ProxyBroker3.2 sets home directory for the package.

Fourth (decide where the binary files will be located), –install-scripts=~/.local/bin sets where the execution files will be. These are the files that allow us to call utilities by their name in the terminal.

Fifth (replace path), –record ~/.local/packages/ProxyBroker3.2/installed_files.txt option will create a list with all installed files of our package. This can be used to remove a package or to examine it.

We need to make sure that the path in which we want to install package binary with –install-scripts=~/.local/bin option is present (this allows us to run programs by their name):

echo $PATH
...
/usr/local/bin: /usr/bin: /bin   # I separated by spaces for clarity

So ~/.local/bin path in which I want to install the binary files is missing.

To add the path to binary files, add the following line in your ~/.bashrc file:

export PATH=$PATH:~/.local/bin  # Replace ~/.local/bin with yours

Then restart terminal or perform the following command to update bash settings:

source ~/.bashrc

Now it is possible to call proxybroker by name:

proxybroker  # Don't rush to do it

But python interpreter still doesn’t know where the libraries of this package are located. Need to specify this as follows, with the path to the package libraries since it is isolated:

PYTHONPATH=~/.local/packages/ProxyBroker3.2/lib/python proxybroker

To not type such a huge structure add the following line in ~/.bashrc. Replace the path to the package libraries with yours:

alias proxybroker='PYTHONPATH=~/.local/packages/ProxyBroker3.2/lib/python proxybroker'

Then restart terminal or perform the following command to update bash settings:

source ~/.bashrc

Now it is possible to call proxybroker by name:

proxybroker
proxybroker -v
...
proxybroker 0.3.2

Don’t forget that we have a list with all installed package files:

cat .local/packages/ProxyBroker3.2/installed_files.txt
...
/home/user/.local/bin/chardetect
/home/user/.local/bin/proxybroker
/home/user/.local/packages/ProxyBroker3.2/lib/python/aiodns-2.0.0-py3.7.egg/EGG-INFO/LICENSE
/home/user/.local/packages/ProxyBroker3.2/lib/python/aiodns-2.0.0-py3.7.egg/EGG-INFO/PKG-INFO
/home/user/.local/packages/ProxyBroker3.2/lib/python/aiodns-2.0.0-py3.7.egg/EGG-INFO/RECORD
/home/user/.local/packages/ProxyBroker3.2/lib/python/aiodns-2.0.0-py3.7.egg/EGG-INFO/WHEE

To remove all the package files from the system, use the following command:

cat .local/packages/ProxyBroker3.2/installed_files.txt | xargs sudo rm -rf

Using ProxyBroker

ProxyBroker options help. These options must be listed before find, grab commands:

proxybroker --help
...
--max-conn MAX_CONN   The maximum number of concurrent checks of proxies
  --max-tries MAX_TRIES
                        The maximum number of attempts to check a proxy
  --timeout SECONDS, -t SECONDS
                        Timeout of a request in seconds. The default value is
                        8 seconds
  --judge JUDGES        Urls of pages that show HTTP headers and IP address
  --provider PROVIDERS  Urls of pages where to find proxies
  --verify-ssl, -ssl    Flag indicating whether to check the SSL certificates
  --log [{NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                        Logging level
  --version, -v         Show program's version number and exit
  --help, -h            Show this help message and exit

ProxyBroker commands help. These options should be listed after find command:

proxybroker find --help
Options:
  --types {HTTP,HTTPS,SOCKS4,SOCKS5,CONNECT:80,CONNECT:25} [{HTTP,HTTPS,SOCKS4,SOCKS5,CONNECT:80,CONNECT:25} ...]
                        Type(s) (protocols) that need to be check on support
                        by proxy
  --lvl {Transparent,Anonymous,High} [{Transparent,Anonymous,High} ...]
                        Level(s) of anonymity (for HTTP only). By default, any
                        level
  --data DATA           Path to the file with proxies. If specified, used
                        instead of providers
  --dnsbl DNSBL [DNSBL ...]
                        Spam databases for proxy checking
  --post                Flag indicating use POST instead of GET for requests
                        when checking proxies
  --strict, -s          Flag indicating that anonymity levels of the types
                        (protocols) supported by a proxy must be equal to the
                        requested types and levels of anonymity
  --countries COUNTRIES [COUNTRIES ...], -c COUNTRIES [COUNTRIES ...]
                        List of ISO country codes where should be located
                        proxies
  --limit LIMIT, -l LIMIT
                        The maximum number of working proxies
  --outfile OUTFILE, -o OUTFILE
                        Save found proxies to file. By default, output to
                        console
  --format [FORMAT], -f [FORMAT]
                        Flag indicating in what format the results will be
                        presented. Available formats: default and json
  --show-stats          Flag indicating whether to print verbose stats
  --help, -h            Show this help message and exit

The following command will collect 50 proxy in a file. Anonymous, High levels of anonymity (only for HTTP) and HTTPS proxy from USA, Canada, China, Brazil, Denmark, France, Finland, Germany, Italy, Mexica, Norway, Poland, Sweden, Switzerland, Singapore, Turkey (See country ISO codes to replace with your own):

proxybroker find --types HTTP HTTPS --lvl Anonymous High --countries US CA CN BR DK FR FI DE IT MX NO PL SE CH SG TR --limit 50 --outfile ./proxies.txt

To create a proxy file with IP:PORT adressess only (without additional info) we can filter it with grep and redirect it to a file (Useful for bash scripts):

proxybroker find --types HTTP HTTPS --lvl Anonymous High --limit 10 | grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}:[0-9]*' > ./proxy_list.txt