Differences

This shows you the differences between two versions of the page.

--- python_notes [2017/08/29 08:36] – created juckins
+++ python_notes [2022/02/02 09:07] – juckins
@@ Line 1: / Line 1: @@
-==== Python Notes ====
+==== Python Install Notes ====
-Use "pip" to manage python packages.
+__2 Feb 2022__
-  # yum install python2-pip
+Upgraded several python packages on both __physical__ and __virtual__ servers.
+  [root@tidewater: ~]# python3 -m pip list --outdated
+  Package             Version        Latest         Type
+  ------------------- -------------- -------------- -----
+  charset-normalizer  2.0.10         2.0.11         wheel
+  chromedriver-binary 96.0.4664.35.0 98.0.4758.48.0 sdist
+  dictor              0.1.7          0.1.9          sdist
+  [root@tidewater: ~]# python3 -m pip install --upgrade <package_name>
-  # pip list
+Fix chromedriver-binary permissions:
+  [root@tidewater: ~]# find /usr -name "chromedriver" -exec ls -l {} \;
+  [root@tidewater: ~]# chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver
+The virtual server updates included the following:
+  [root@centos8S-vtatdb2-l: ~]# python3 -m pip list --outdated
+  Package         Version     Latest   Type
+  --------------- ----------- -------- -----
+  beautifulsoup4  4.6.3       4.10.0   wheel
+  blivet          3.4.0       3.4.3    wheel
+  chardet         3.0.4       4.0.0    wheel
+  coverage        4.5.1       6.2      wheel
+  cssselect       0.9.2       1.1.0    wheel
+  dasbus          1.2         1.6      wheel
+  dbus-python     1.2.4       1.2.18   sdist
+  decorator       4.2.1       5.1.1    wheel
+  dictor          0.1.7       0.1.9    sdist
+  dnspython       1.15.0      2.2.0    wheel
+  ethtool         0.14        0.15     sdist
+  html5lib        0.999999999 1.1      wheel
+  httplib2        0.18.1      0.20.2   wheel
+  humanize        0.5.1       3.14.0   wheel
+  idna            2.5         3.3      wheel
+  iniparse        0.4         0.5      wheel
+  langtable       0.0.51      0.0.57   wheel
+  lxml            4.2.3       4.7.1    wheel
+  ntplib          0.3.3       0.4.0    wheel
+  numpy           1.14.3      1.19.5   wheel
+  ordered-set     2.0.2       4.0.2    sdist
+  pexpect         4.3.1       4.8.0    wheel
+  pid             2.1.1       3.0.4    wheel
+  ply             3.9         3.11     wheel
+  productmd       1.11        1.33     wheel
+  psutil          5.4.3       5.9.0    wheel
+  ptyprocess      0.5.2       0.7.0    wheel
+  pycairo         1.16.3      1.20.1   sdist
+  pycups          1.9.72      2.0.1    sdist
+  pycurl          7.43.0.2    7.44.1   sdist
+  PyGObject       3.28.3      3.42.0   sdist
+  pykickstart     3.16.14     3.35     sdist
+  pyparsing       2.1.10      3.0.7    wheel
+  PySocks         1.6.8       1.7.1    wheel
+  python-augeas   0.5.0       1.1.0    sdist
+  python-dateutil 2.6.1       2.8.2    wheel
+  pytz            2017.2      2021.3   wheel
+  pyudev          0.21.0      0.23.1   wheel
+  PyYAML          3.12        6.0      wheel
+  requests        2.20.0      2.27.1   wheel
+  requests-file   1.4.3       1.5.1    wheel
+  scipy           1.0.0       1.5.4    wheel
+  setuptools      39.2.0      59.6.0   wheel
+  simpleline      1.1.1       1.9.0    wheel
+  six             1.11.0      1.16.0   wheel
+  slip            0.6.4       20191113 sdist
+  urllib3         1.24.2      1.26.8   wheel
+  wheel           0.36.2      0.37.1   wheel
+__20 Jan 2022__
+Upgraded several python packages.
+  [root@tidewater: ~]# python3 -m pip list --outdated
+  Package             Version        Latest         Type
+  ------------------- -------------- -------------- -----
+  charset-normalizer  2.0.7          2.0.10         wheel
+  chromedriver-binary 96.0.4664.35.0 98.0.4758.48.0 sdist
+  requests            2.26.0         2.27.1         wheel
+  setuptools          59.1.1         59.6.0         wheel
+  urllib3             1.26.7         1.26.8         wheel
+  [root@tidewater: ~]# python3 -m pip install --upgrade <package_name>
+Note: Did not update chromedriver-binary as Chrome is still version 97.0.4692.99
+__16 Nov 2021__
+Upgraded several python packages.
+  [root@tidewater: ~]# python3 -m pip list --outdated
+  Package             Version        Latest         Type
+  ------------------- -------------- -------------- -----
+  certifi             2021.5.30      2021.10.8      wheel
+  charset-normalizer  2.0.6          2.0.7          wheel
+  chromedriver-binary 94.0.4606.61.0 96.0.4664.35.0 sdist
+  idna                3.2            3.3            wheel
+  Pillow              8.3.2          8.4.0          wheel
+  pytz                2021.1         2021.3         wheel
+  setuptools          58.1.0         59.1.1         wheel
+  [root@tidewater: ~]# python3 -m pip install --upgrade <package_name>
+__29 Sep 2021__
+Upgraded the following python3 packages using the commands listed below:
+  pip
+  charset-normalizer
+  setuptools
+  urllib3
+  chromedriver-binary
+Then got the following error message for chromedriver-binary:
+  selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 95
+  Current browser version is 94.0.4606.61 with binary path /usr/bin/google-chrome
+So did the following to fix:
+  python3 -m pip install chromedriver-binary==94.0.4606.61
+  find /usr -name "chromedriver" -exec ls -l {} \;
+  chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver
+__21 Jul 2021__
+  # python3 -m pip list --outdated
+  Package             Version        Latest         Type
+  ------------------- -------------- -------------- -----
+  certifi             2020.12.5      2021.5.30      wheel
+  chromedriver-binary 91.0.4472.19.0 92.0.4515.43.0 sdist
+  idna                3.1            3.2            wheel
+  requests            2.25.1         2.26.0         wheel
+  setuptools          57.0.0         57.4.0         wheel
+  urllib3             1.26.5         1.26.6         wheel
+Upgraded all using:
+  python3 -m pip install --upgrade <package_name>
+Note on chromedriver-binary...don't forget to check/adjust permissions:
+  # find /usr -name "chromedriver" -exec ls -l {} \;
+  # chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver
+__26 May 2021__
+These packages were outdated:
+  # python3 -m pip list --outdated
+  Package             Version        Latest         Type
+  ------------------- -------------- -------------- -----
+  chromedriver-binary 90.0.4430.24.0 91.0.4472.19.0 sdist
+  pip                 21.1.1         21.1.2         wheel
+  setuptools          56.0.0         57.0.0         wheel
+  WARNING: You are using pip version 21.1.1; however, version 21.1.2 is available.
+  You should consider upgrading via the '/bin/python3 -m pip install --upgrade pip' command.
+Upgraded all using:
+  python3 -m pip install --upgrade <package_name>
+__15 Apr 2021__
+Updated various python packages, including chromedriver-binary
+__3 Mar 2021__
+These packages were outdated:
+  # python3 -m pip list --outdated
+  Package             Version        Latest         Type
+  ------------------- -------------- -------------- -----
+  certifi             2020.11.8      2020.12.5      wheel
+  chardet             3.0.4          4.0.0          wheel
+  chromedriver-binary 87.0.4280.20.0 89.0.4389.23.0 sdist
+  httplib2            0.18.1         0.19.0         wheel
+  idna                2.10           3.1            wheel
+  PyMySQL             0.10.1         1.0.2          wheel
+  pytz                2020.4         2021.1         wheel
+  requests            2.25.0         2.25.1         wheel
+  selenium-wire       2.1.2          4.2.0          wheel
+  setuptools          50.3.2         54.0.0         wheel
+  urllib3             1.26.2         1.26.3         wheel
+  wheel               0.35.1         0.36.2         wheel
+Upgraded all using:
+  python3 -m pip install --upgrade <package_name>
+<color #ed1c24>except for PyMySQL.  'tidewater' needs updates to mariadb v10.</color>  Need to check python scripts first for ASMAD (and test on VM first).
+__18 Nov 2020__
+Manually updated chrome to v87.
+Ran the following:
+  python3 -m pip list --outdated
+With the resulting list, then ran the following for each <package_name>
+  python3 -m pip install --upgrade <package_name>
+__11 Oct 2020__
+To upgrade pip modules manually when dependencies give an error:
+  # python3 -m pip --list outdated
+  # python3 -m pip install --upgrade <package1> <package2>
+These are reported as outdated:
+  [root: ~]# python3 -m pip list --outdated
+  Package       Version    Latest    Type
+  ------------- ---------- --------- -----
+  certifi       2020.4.5.1 2020.6.20 wheel
+  idna          2.9        2.10      wheel
+  requests      2.23.0     2.24.0    wheel
+  selenium-wire 1.0.12     2.1.1     wheel
+  setuptools    46.4.0     50.3.0    wheel
+  urllib3       1.25.9     1.25.10   wheel
+All upgraded using the command:
+  python3 -m pip install --upgrade <package_name>
+__9 Oct 2020__
+Updated Chrome via yum to version 86.0.4240.75-1 and now check_envoy.pl fails.
+Did the following as root (checked on the VM "CENTOS78-test" first) to fix:
+  # /usr/bin/python3 -m pip uninstall chromedriver-binary
+  # /usr/bin/python3 -m pip install chromedriver-binary
+Note: you can also do specific versions of chromedrive-binary:
+  # python3 -m pip install chromedriver-binary==90.0.4430.24.0
+  # /usr/bin/python3 -m pip install --upgrade pip
+  # find /usr -name "chromedriver" -exec ls -l {} \;
+  # chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver
+__6 Oct 2020__
+  # python3 -m pip install demjson
+  # python3 -m pip install wheel
+__26 Aug 2020__
+Updated Chrome via yum to version 85.0.4183.83-1 and now check_envoy.pl fails.
+Did the following as root to fix:
+  # /usr/bin/python3 -m pip uninstall chromedriver-binary
+  # /usr/bin/python3 -m pip install chromedriver-binary
+  # /usr/bin/python3 -m pip install --upgrade pip
+  # find /usr -name "chromedriver" -exec ls -l {} \;
+  # chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver
+Re-ran the cron and script worked.
+__20 May 2020__
+Updated Chrome via yum to version 83.0.4103.61-1 and now check_envoy.pl fails.
+Did the following as root to fix:
+  # /usr/local/bin/pip3 uninstall chromedriver-binary
+  # /usr/local/bin/pip3 install chromedriver-binary
+  # /usr/bin/python3 -m pip install --upgrade pip
+  # find /usr -name "chromedriver" -exec ls -l {} \;
+  # chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver
+Re-ran the cron and script worked.
+Then tried looking at old packages needing an update:
+  # cd /usr/local/bin
+  # ./pip3 list --outdated
+  # pip_upgrade_outdated
+__8 April 2020__
+Upgraded Chrome via yum to version 81.0.4044.92 and now check_envoy.pl fails.
+Did the following as root to fix:
+  # /usr/local/bin/pip3 uninstall chromedriver-binary
+  # /usr/local/bin/pip3 install chromedriver-binary
+  # /usr/local/bin/pip3 install pip-upgrade-outdated
+  # cd /usr/local/bin
+  # ./pip3 install --upgrade
+  # ./pip3 list --outdated
+  # pip_upgrade_outdated
+  # find /usr -name "chromedriver" -exec ls -l {} \;
+  /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver
+  # chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver
+The pip upgrade moved the astral package to v2.1 which broke sun.py and it needed some major changes.  Had to work through the [[https://astral.readthedocs.io/en/latest/index.html|Astral]] package docs.
+__11 December 2019__
+Upgraded Chrome via yum to version 79.0.3945.79 and now check_envoy.pl fails.
+Did the following as root to fix:
+  # /usr/local/bin/pip3 uninstall chromedriver-binary
+  # /usr/local/bin/pip3 install chromedriver-binary
+  # /usr/local/bin/pip3 install pip-upgrade-outdated
+  # pip-upgrade-outdated
+  # find /usr -name "chromedriver" -exec ls -l {} \;
+  /usr/lib/python3.6/site-packages/chromedriver
+  /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver
+  # chmod 755 /usr/local/lib/python3.6/site-packages/chromedriver_binary/chromedriver
+Note that both **/usr/bin/pip3.6** and **/usr/local/bin/pip3.6** exist on the system now...needs to be fixed.
+----
+With CentOS7.7, python3 can load from base.  Do the following to fix previously-installed IUS release:
+  # yum remove python36u
+  # yum install python3
+  # yum remove ius-release
+  * Then follow instructions for installing local python packages to ensure all still work.
+  * If you have pip problems, uninstall/reinstall using instructions at https://pip.pypa.io/en/stable/installing/
+  * Ensure pip3.6 is in your $PATH or you have a sym link to it (/usr/local/bin) in /usr/bin
+__Install Python 3.6 on CentOS 7__:
+  * https://linuxhint.com/install_python3_centos7/
+  * https://janikarhunen.fi/how-to-install-python-3-6-1-on-centos-7.html
+Do some local configuration for my scripts (su - root):
+  pip3.6 install --upgrade pip
+  pip3.6 install --upgrade httplib2
+  pip3.6 install pymysql
+  pip3.6 install selenium
+  pip3.6 install selenium-wire
+  pip3.6 install chromedriver
+  pip3.6 install chromedriver_binary
+  pip3.6 install dictor
+  pip3.6 install wget
+  pip3.6 install pip-upgrade-outdated
+  pip3.6 install pytz
+  pip3.6 install astral
+Note: replace the "pip3.6 install" with "/usr/bin/python3 -m pip install"
+To update all pip packages:
+  # pip_upgrade_outdated
+Note that https://pypi.org/project/pip-upgrade-outdated/ is a good resource to update all PIP packages.
+Download and install geckodriver (see https://github.com/mozilla/geckodriver/releases)
+As root, copy geckodriver to /usr/local/bin and ensure permissions are 755
+Notes at https://stackoverflow.com/questions/40208051/selenium-using-python-geckodriver-executable-needs-to-be-in-path
+Notes about [[https://www.tutorialspoint.com/python3/python_database_access.htm|Python database access]]
+Python3.7 on Fedora 28:
+  * dnf install python37
+  * wget https://bootstrap.pypa.io/get-pip.py
+  * python3.7 get-pip.py
+  * python3.7 -m pip install requests
+Use "pip" to manage python packages:
+  # yum install python2-pip
+To list installed python modules:
+  $ pip list