Christopher Juckins

SysAdmin Tips, Tricks and other Software Tools

User Tools

Site Tools


python_notes

Python Install Notes

5 Apr 2023

# python3 -m pip list --outdated
Package             Version         Latest          Type
------------------- --------------- --------------- -----
chromedriver-binary 110.0.5481.30.0 112.0.5615.28.0 sdist
mypy-extensions     0.4.3           1.0.0           wheel
pdf2image           1.16.2          1.16.3          wheel
pytesseract         0.3.8           0.3.9           wheel
pytz                2022.7.1        2023.3          wheel
urllib3             1.26.14         1.26.15         wheel

Ran this:

# python3 -m pip install --upgrade <package_name>
# find /usr -name "chromedriver" -exec ls -l {} \;
# chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver

11 Jan 2023

# python3 -m pip list --outdated
Package             Version         Latest          Type
------------------- --------------- --------------- -----
certifi             2022.9.24       2022.12.7       wheel
chromedriver-binary 108.0.5359.22.0 109.0.5414.74.0 sdist
dictor              0.1.10          0.1.11          sdist
pdf2image           1.16.0          1.16.2          wheel
pytesseract         0.3.8           0.3.9           wheel
pytz                2022.6          2022.7          wheel

Upgraded all of the above except for pytesseract. Then:

# find /usr -name "chromedriver" -exec ls -l {} \;
# chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver

30 Nov 2022

Upgraded several python packages on physical and virtual servers.

26 Oct 2022

Upgraded several python packages on physical server.

Current list of python modules:

[root@tidewater: ~]# python3 -m pip list
Package             Version
------------------- ---------------
astral              3.0
backports.zoneinfo  0.2.1
black               22.8.0
certifi             2022.9.24
chardet             5.0.0
charset-normalizer  3.0.0
chromedriver        2.24.1
chromedriver-binary 107.0.5304.62.0
click               8.0.4
dataclasses         0.8
dictor              0.1.10
idna                3.4
importlib-metadata  4.8.3
importlib-resources 5.4.0
mypy-extensions     0.4.3
numpy               1.19.5
pathspec            0.9.0
pdf2image           1.16.0
Pillow              8.4.0
pip                 21.3.1
platformdirs        2.4.0
PyMySQL             1.0.2
pytesseract         0.3.8
pytz                2022.5
requests            2.27.1
selenium            3.141.0
setuptools          59.6.0
tomli               1.2.3
typed-ast           1.5.4
typing_extensions   4.1.1
urllib3             1.26.12
vboxapi             1.0
wget                3.2
zipp                3.6.0

30 Sep 2022

Current list of python modules:

[root@tidewater: ~]# python3 -m pip list
Package             Version
------------------- ---------------
astral              2.2
black               22.8.0
certifi             2022.9.14
chardet             5.0.0
charset-normalizer  2.1.1
chromedriver        2.24.1
chromedriver-binary 105.0.5195.52.0
click               8.0.4
dataclasses         0.8
dictor              0.1.10
idna                3.4
importlib-metadata  4.8.3
mypy-extensions     0.4.3
numpy               1.19.5
pathspec            0.9.0
pdf2image           1.16.0
Pillow              8.4.0
pip                 21.3.1
platformdirs        2.4.0
PyMySQL             1.0.2
pytesseract         0.3.8
pytz                2022.2.1
requests            2.27.1
selenium            3.141.0
setuptools          59.6.0
tomli               1.2.3
typed-ast           1.5.4
typing_extensions   4.1.1
urllib3             1.26.12
vboxapi             1.0
wget                3.2
zipp                3.6.0

22 Sep 2022

Upgraded several python packages on both physical and virtual servers.

2 Aug 2022

Upgraded several python packages on both physical and virtual servers.

22 June 2022

Upgraded several python packages on both physical and virtual servers.

26 May 2022

Upgraded several python packages on both physical and virtual servers.

27 Apr 2022

Upgraded several python packages on both physical and virtual servers.

30 Mar 2022

Upgraded several python packages on both physical and virtual servers.

3 Mar 2022

Upgraded several python packages on both physical and virtual servers.

[root@tidewater: ~]# python3 -m pip list --outdated
Package             Version        Latest         Type
------------------- -------------- -------------- -----
charset-normalizer  2.0.11         2.0.12         wheel
chromedriver-binary 98.0.4758.48.0 99.0.4844.51.0 sdist
numpy               1.12.1         1.19.5         wheel
pytesseract         0.3.8          0.3.9          wheel
[root@tidewater: ~]# python3 -m pip install --upgrade <package_name>

Fix chromedriver-binary permissions:

[root@tidewater: ~]# find /usr -name "chromedriver" -exec ls -l {} \;
[root@tidewater: ~]# chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver

The virtual server updates included the following:

[root@centos8S-vtatdb2-l: ~]# python3 -m pip list --outdated
Package            Version  Latest   Type
------------------ -------- -------- -----
charset-normalizer 2.0.11   2.0.12   wheel
dbus-python        1.2.4    1.2.18   sdist
ethtool            0.14     0.15     sdist
httplib2           0.20.2   0.20.4   wheel
imageio            2.14.1   2.16.0   wheel
lxml               4.7.1    4.8.0    wheel
ntplib             0.3.3    0.4.0    wheel
pycairo            1.16.3   1.20.1   sdist
pycups             1.9.72   2.0.1    sdist
pycurl             7.43.0.2 7.44.1   sdist
PyGObject          3.28.3   3.42.0   sdist
pykickstart        3.35     3.37     sdist
python-augeas      0.5.0    1.1.0    sdist
pyudev             0.23.1   0.23.2   wheel
simpleline         1.1.1    1.9.0    wheel
slip               0.6.4    20191113 sdist
typing_extensions  4.0.1    4.1.1    wheel

2 Feb 2022

Upgraded several python packages on both physical and virtual servers.

[root@tidewater: ~]# python3 -m pip list --outdated
Package             Version        Latest         Type
------------------- -------------- -------------- -----
charset-normalizer  2.0.10         2.0.11         wheel
chromedriver-binary 96.0.4664.35.0 98.0.4758.48.0 sdist
dictor              0.1.7          0.1.9          sdist
[root@tidewater: ~]# python3 -m pip install --upgrade <package_name>

Fix chromedriver-binary permissions:

[root@tidewater: ~]# find /usr -name "chromedriver" -exec ls -l {} \;
[root@tidewater: ~]# chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver

The virtual server updates included the following:

[root@centos8S-vtatdb2-l: ~]# python3 -m pip list --outdated
Package         Version     Latest   Type
--------------- ----------- -------- -----
beautifulsoup4  4.6.3       4.10.0   wheel
blivet          3.4.0       3.4.3    wheel
chardet         3.0.4       4.0.0    wheel
coverage        4.5.1       6.2      wheel
cssselect       0.9.2       1.1.0    wheel
dasbus          1.2         1.6      wheel
dbus-python     1.2.4       1.2.18   sdist
decorator       4.2.1       5.1.1    wheel
dictor          0.1.7       0.1.9    sdist
dnspython       1.15.0      2.2.0    wheel
ethtool         0.14        0.15     sdist
html5lib        0.999999999 1.1      wheel
httplib2        0.18.1      0.20.2   wheel
humanize        0.5.1       3.14.0   wheel
idna            2.5         3.3      wheel
iniparse        0.4         0.5      wheel
langtable       0.0.51      0.0.57   wheel
lxml            4.2.3       4.7.1    wheel
ntplib          0.3.3       0.4.0    wheel
numpy           1.14.3      1.19.5   wheel
ordered-set     2.0.2       4.0.2    sdist
pexpect         4.3.1       4.8.0    wheel
pid             2.1.1       3.0.4    wheel
ply             3.9         3.11     wheel
productmd       1.11        1.33     wheel
psutil          5.4.3       5.9.0    wheel
ptyprocess      0.5.2       0.7.0    wheel
pycairo         1.16.3      1.20.1   sdist
pycups          1.9.72      2.0.1    sdist
pycurl          7.43.0.2    7.44.1   sdist
PyGObject       3.28.3      3.42.0   sdist
pykickstart     3.16.14     3.35     sdist
pyparsing       2.1.10      3.0.7    wheel
PySocks         1.6.8       1.7.1    wheel
python-augeas   0.5.0       1.1.0    sdist
python-dateutil 2.6.1       2.8.2    wheel
pytz            2017.2      2021.3   wheel
pyudev          0.21.0      0.23.1   wheel
PyYAML          3.12        6.0      wheel
requests        2.20.0      2.27.1   wheel
requests-file   1.4.3       1.5.1    wheel
scipy           1.0.0       1.5.4    wheel
setuptools      39.2.0      59.6.0   wheel
simpleline      1.1.1       1.9.0    wheel
six             1.11.0      1.16.0   wheel
slip            0.6.4       20191113 sdist
urllib3         1.24.2      1.26.8   wheel
wheel           0.36.2      0.37.1   wheel

Most installed as user root; a few installed as non-admin user.

These were unable to be updated:

$ python3 -m pip list --outdated
Package     Version  Latest   Type
----------- -------- -------- -----
dbus-python 1.2.4    1.2.18   sdist
ethtool     0.14     0.15     sdist
pycairo     1.16.3   1.20.1   sdist
pycups      1.9.72   2.0.1    sdist
PyGObject   3.28.3   3.42.0   sdist
SLIP        20171205 20191113 sdist

20 Jan 2022

Upgraded several python packages.

[root@tidewater: ~]# python3 -m pip list --outdated
Package             Version        Latest         Type
------------------- -------------- -------------- -----
charset-normalizer  2.0.7          2.0.10         wheel
chromedriver-binary 96.0.4664.35.0 98.0.4758.48.0 sdist
requests            2.26.0         2.27.1         wheel
setuptools          59.1.1         59.6.0         wheel
urllib3             1.26.7         1.26.8         wheel
[root@tidewater: ~]# python3 -m pip install --upgrade <package_name>

Note: Did not update chromedriver-binary as Chrome is still version 97.0.4692.99

16 Nov 2021

Upgraded several python packages.

[root@tidewater: ~]# python3 -m pip list --outdated
Package             Version        Latest         Type
------------------- -------------- -------------- -----
certifi             2021.5.30      2021.10.8      wheel
charset-normalizer  2.0.6          2.0.7          wheel
chromedriver-binary 94.0.4606.61.0 96.0.4664.35.0 sdist
idna                3.2            3.3            wheel
Pillow              8.3.2          8.4.0          wheel
pytz                2021.1         2021.3         wheel
setuptools          58.1.0         59.1.1         wheel

[root@tidewater: ~]# python3 -m pip install --upgrade <package_name>

29 Sep 2021

Upgraded the following python3 packages using the commands listed below:

pip
charset-normalizer
setuptools
urllib3
chromedriver-binary

Then got the following error message for chromedriver-binary:

selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 95
Current browser version is 94.0.4606.61 with binary path /usr/bin/google-chrome

So did the following to fix:

python3 -m pip install chromedriver-binary==94.0.4606.61
find /usr -name "chromedriver" -exec ls -l {} \;
chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver

21 Jul 2021

# python3 -m pip list --outdated
Package             Version        Latest         Type
------------------- -------------- -------------- -----
certifi             2020.12.5      2021.5.30      wheel
chromedriver-binary 91.0.4472.19.0 92.0.4515.43.0 sdist
idna                3.1            3.2            wheel
requests            2.25.1         2.26.0         wheel
setuptools          57.0.0         57.4.0         wheel
urllib3             1.26.5         1.26.6         wheel

Upgraded all using:

python3 -m pip install --upgrade <package_name>

Note on chromedriver-binary…don't forget to check/adjust permissions:

# find /usr -name "chromedriver" -exec ls -l {} \;
# chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver

26 May 2021

These packages were outdated:

# python3 -m pip list --outdated
Package             Version        Latest         Type
------------------- -------------- -------------- -----
chromedriver-binary 90.0.4430.24.0 91.0.4472.19.0 sdist
pip                 21.1.1         21.1.2         wheel
setuptools          56.0.0         57.0.0         wheel
WARNING: You are using pip version 21.1.1; however, version 21.1.2 is available.
You should consider upgrading via the '/bin/python3 -m pip install --upgrade pip' command.

Upgraded all using:

python3 -m pip install --upgrade <package_name>

15 Apr 2021

Updated various python packages, including chromedriver-binary

3 Mar 2021

These packages were outdated:

# python3 -m pip list --outdated
Package             Version        Latest         Type
------------------- -------------- -------------- -----
certifi             2020.11.8      2020.12.5      wheel
chardet             3.0.4          4.0.0          wheel
chromedriver-binary 87.0.4280.20.0 89.0.4389.23.0 sdist
httplib2            0.18.1         0.19.0         wheel
idna                2.10           3.1            wheel
PyMySQL             0.10.1         1.0.2          wheel
pytz                2020.4         2021.1         wheel
requests            2.25.0         2.25.1         wheel
selenium-wire       2.1.2          4.2.0          wheel
setuptools          50.3.2         54.0.0         wheel
urllib3             1.26.2         1.26.3         wheel
wheel               0.35.1         0.36.2         wheel

Upgraded all using:

python3 -m pip install --upgrade <package_name>

except for PyMySQL. 'tidewater' needs updates to mariadb v10. Need to check python scripts first for ASMAD (and test on VM first).

18 Nov 2020

Manually updated chrome to v87. Ran the following:

python3 -m pip list --outdated

With the resulting list, then ran the following for each <package_name>

python3 -m pip install --upgrade <package_name>

11 Oct 2020

To upgrade pip modules manually when dependencies give an error:

# python3 -m pip --list outdated
# python3 -m pip install --upgrade <package1> <package2> 

These are reported as outdated:

[root: ~]# python3 -m pip list --outdated
Package       Version    Latest    Type
------------- ---------- --------- -----
certifi       2020.4.5.1 2020.6.20 wheel
idna          2.9        2.10      wheel
requests      2.23.0     2.24.0    wheel
selenium-wire 1.0.12     2.1.1     wheel
setuptools    46.4.0     50.3.0    wheel
urllib3       1.25.9     1.25.10   wheel

All upgraded using the command:

python3 -m pip install --upgrade <package_name>

9 Oct 2020

Updated Chrome via yum to version 86.0.4240.75-1 and now check_envoy.pl fails.

Did the following as root (checked on the VM "CENTOS78-test" first) to fix:

# /usr/bin/python3 -m pip uninstall chromedriver-binary
# /usr/bin/python3 -m pip install chromedriver-binary

Note: you can also do specific versions of chromedrive-binary (see available versions at https://pypi.org/project/chromedriver-py/#history):

# python3 -m pip install chromedriver-binary==90.0.4430.24.0
# /usr/bin/python3 -m pip install --upgrade pip
# find /usr -name "chromedriver" -exec ls -l {} \;
# chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver

6 Oct 2020

# python3 -m pip install demjson
# python3 -m pip install wheel

26 Aug 2020

Updated Chrome via yum to version 85.0.4183.83-1 and now check_envoy.pl fails.

Did the following as root to fix:

# /usr/bin/python3 -m pip uninstall chromedriver-binary
# /usr/bin/python3 -m pip install chromedriver-binary
# /usr/bin/python3 -m pip install --upgrade pip
# find /usr -name "chromedriver" -exec ls -l {} \;
# chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver

Re-ran the cron and script worked.

20 May 2020

Updated Chrome via yum to version 83.0.4103.61-1 and now check_envoy.pl fails.

Did the following as root to fix:

# /usr/local/bin/pip3 uninstall chromedriver-binary
# /usr/local/bin/pip3 install chromedriver-binary
# /usr/bin/python3 -m pip install --upgrade pip
# find /usr -name "chromedriver" -exec ls -l {} \;
# chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver

Re-ran the cron and script worked.

Then tried looking at old packages needing an update:

# cd /usr/local/bin
# ./pip3 list --outdated
# pip_upgrade_outdated

8 April 2020

Upgraded Chrome via yum to version 81.0.4044.92 and now check_envoy.pl fails.

Did the following as root to fix:

# /usr/local/bin/pip3 uninstall chromedriver-binary
# /usr/local/bin/pip3 install chromedriver-binary
# /usr/local/bin/pip3 install pip-upgrade-outdated

# cd /usr/local/bin
# ./pip3 install --upgrade
# ./pip3 list --outdated
# pip_upgrade_outdated

# find /usr -name "chromedriver" -exec ls -l {} \;

/usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver

# chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver

The pip upgrade moved the astral package to v2.1 which broke sun.py and it needed some major changes. Had to work through the Astral package docs.

11 December 2019

Upgraded Chrome via yum to version 79.0.3945.79 and now check_envoy.pl fails.

Did the following as root to fix:

# /usr/local/bin/pip3 uninstall chromedriver-binary
# /usr/local/bin/pip3 install chromedriver-binary
# /usr/local/bin/pip3 install pip-upgrade-outdated
# pip-upgrade-outdated

# find /usr -name "chromedriver" -exec ls -l {} \;

/usr/lib/python3.6/site-packages/chromedriver
/usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver
# chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver

Note that both /usr/bin/pip3.6 and /usr/local/bin/pip3.6 exist on the system now…needs to be fixed.


With CentOS7.7, python3 can load from base. Do the following to fix previously-installed IUS release:

# yum remove python36u
# yum install python3
# yum remove ius-release
  • Then follow instructions for installing local python packages to ensure all still work.
  • If you have pip problems, uninstall/reinstall using instructions at https://pip.pypa.io/en/stable/installing/
  • Ensure pip3.6 is in your $PATH or you have a sym link to it (/usr/local/bin) in /usr/bin

Install Python 3.6 on CentOS 7:

Do some local configuration for my scripts (su - root):

pip3.6 install --upgrade pip
pip3.6 install --upgrade httplib2
pip3.6 install pymysql
pip3.6 install selenium
pip3.6 install selenium-wire
pip3.6 install chromedriver
pip3.6 install chromedriver_binary
pip3.6 install dictor
pip3.6 install wget
pip3.6 install pip-upgrade-outdated
pip3.6 install pytz
pip3.6 install astral

Note: replace the "pip3.6 install" with "/usr/bin/python3 -m pip install"

To update all pip packages:

# pip_upgrade_outdated

Note that https://pypi.org/project/pip-upgrade-outdated/ is a good resource to update all PIP packages.

Download and install geckodriver (see https://github.com/mozilla/geckodriver/releases)

As root, copy geckodriver to /usr/local/bin and ensure permissions are 755

Notes at https://stackoverflow.com/questions/40208051/selenium-using-python-geckodriver-executable-needs-to-be-in-path

Notes about Python database access

Python3.7 on Fedora 28:

Use "pip" to manage python packages:

# yum install python2-pip

To list installed python modules:

$ pip list
python_notes.txt · Last modified: 2023/04/05 08:56 by juckins