Updated licence date

CHANGELOG.md

@@ -1,5 +1,7 @@
 # Changelog
 
+**13 MAR 2023** VERSION 3.2.2 Updated list of nitter instances
+
 **21 FEB 2023** VERSION 3.2.1 Updated user agents and list of nitter instances
 
 **15 FEB 2023** VERSION 3.2 Added mitigation for Mastodon API error 422, 'Unprocessable Entity',

LICENSE

@@ -1,4 +1,4 @@
-Copyright (C) 2019-2021  Jean-Christophe Francois
+Copyright (C) 2019-2023  Jean-Christophe Francois
 
 This program is free software: you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by

README.md

@@ -3,7 +3,11 @@
 Twoot is a python script that mirrors tweets from a twitter account to a Mastodon account.
 It is simple to set-up on a local machine, configurable and feature-rich.
 
-**13 MAR 2023** VERSION 3.2.2 Updated list of nitter instances
+**19 JUN 2023** VERSION 4.0
+
+* Added option to update avatar and banner pictures on profile if changed on Twitter
+* Tweaked list of nitter instances
+* Updated list of user agents
 
 > Previous updates can be found in CHANGELOG.
 
@@ -17,6 +21,7 @@ It is simple to set-up on a local machine, configurable and feature-rich.
 * Specify maximum age of tweet to be considered
 * Specify minimum delay before considering a tweet for upload
 * Remember tweets already tooted to prevent double posting
+* Optionally update avatar and banner pictures on profile if changed
 * Optionally post reply-to tweets on the mastodon account
 * Optionally ignore retweets
 * Optionally remove redirections (e.g. reveal destination of short URLs)
@@ -29,15 +34,15 @@ It is simple to set-up on a local machine, configurable and feature-rich.
 ## Usage
 
 ```sh
-twoot.py [-h] [-f <.toml config file>] [-t <twitter account>] [-i <mastodon instance>]
-         [-m <mastodon account>] [-p <mastodon password>] [-r] [-s] [-l] [-u] [-v] [-o]
-         [-a <max age in days)>] [-d <min delay (in mins>] [-c <max # of toots to post>]
+usage: twoot.py [-h] [-f <.toml config file>] [-t <twitter account>] [-i <mastodon instance>]
+                [-m <mastodon account>] [-p <mastodon password>] [-r] [-s] [-l] [-u] [-v] [-o] [-q]
+                [-a <max age (in days)>] [-d <min delay (in mins)>] [-c <max # of toots to post>]
 ```
 
 ## Arguments
 
 Assuming that the Twitter handle is @SuperDuperBot and the Mastodon account
-is sd@example.com on instance masto.space:
+is `sd@example.com` on instance masto.space:
 
 |Switch |Description                                       | Example            | Required           |
 |-------|--------------------------------------------------|--------------------|--------------------|
@@ -48,6 +53,7 @@ is sd@example.com on instance masto.space:
 | -p    | Mastodon password                                | `my_Sup3r-S4f3*pw` | Once at first run  |
 | -v    | Upload videos to Mastodon                        | *N/A*              | No                 |
 | -o    | Do not add "Original tweet" line                 | *N/A*              | No                 |
+| -q    | Update avatar and banner on profile if changed   | *N/A*              | No                 |
 | -r    | Post reply-to tweets (ignored by default)        | *N/A*              | No                 |
 | -s    | Skip retweets (posted by default)                | *N/A*              | No                 |
 | -l    | Remove link redirections                         | *N/A*              | No                 |
@@ -71,22 +77,30 @@ to use, all the other command-line parameters are ignored, except `-p` (password
 
 ### Removing redirected links
 
-`-l` will follow every link included in the tweet and replace them with the url that the
-resource is directly dowmnloaded from (if applicable). e.g. bit.ly/xxyyyzz -> example.com
-Every link visit can take up to 5 sec (timeout) therefore this option will slow down
-tweet processing.
+`-l` (or `remove_link_redirections = true` in toml file) will follow every link included in the
+tweet and replace them with the url that the resource is directly dowmnloaded from (if applicable).
+e.g. bit.ly/xxyyyzz -> example.com
+Every link visit can take up to 5 sec (timeout) depending on the responsiveness of the source
+therefore this option will slow down tweet processing.
 
-If you are interested by tracker removal (`-u`) you should also select redirection removal
-as trackers are often hidden behind the redirection of a short URL.
+If you are interested by tracker removal (`-u`, `remove_trackers_from_urls = true`) you should
+also select redirection removal as trackers are often hidden behind the redirection of a short URL.
 
 ### Uploading videos
 
-When using the `-v` switch consider:
+When using the `-v` (`upload_videos = true`) switch consider:
 
 * whether the copyright of the content that you want to cross-post allows it
 * the storage / transfer limitations of the Mastodon instance that you are posting to
 * the upstream bandwidth that you may consume on your internet connection
 
+### Updating profile
+
+If `-q` (`update_profile = true`) is specified, twoot will check if the avatar and banner pictures
+have changed on the twitter page. This check compares the name of files used by twitter with the names
+of the files that have been uploaded on Mastodon and if they differ both files are downloaded from
+twitter and uploaded on Mastodon. The check is very fast if there is no update.
+
 ### Rate control
 
 Default max age is 1 day. Decimal values are OK.

twoot.py

@@ -2,7 +2,7 @@
 # -*- coding: utf-8 -*-
 
 """
-    Copyright (C) 2019-2022  Jean-Christophe Francois
+    Copyright (C) 2019-2023  Jean-Christophe Francois
 
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
@@ -62,13 +62,12 @@ NITTER_URLS = [
 
 # Update from https://www.whatismybrowser.com/guides/the-latest-user-agent/
 USER_AGENTS = [
-    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Edg/110.0.1587.46',
-    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36',
-    'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36',
-    'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36',
-    'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0',
-    'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Vivaldi/5.6.2867.62',
-    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Vivaldi/5.6.2867.62',
+    'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0',
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5 Safari/605.1.15',
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.51',
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 OPR/99.0.0.0',
+    'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Vivaldi/6.1.3035.84',
 ]
 
 
@@ -123,10 +122,10 @@ def build_config(args):
                 loaded_toml = tomllib.load(config_file)
         except FileNotFoundError:
             print('config file not found')
-            terminate(-1)
+            shutdown(-1)
         except tomllib.TOMLDecodeError:
             print('Malformed config file')
-            terminate(-1)
+            shutdown(-1)
 
         TOML['config'] = loaded_toml['config']
         for k in TOML['options'].keys():
@@ -244,7 +243,7 @@ def update_profile(nitter_url, soup, sql, mast_password):
         if new_avatar is not None:
             new_avatar_img = new_avatar.content if new_avatar.status_code == 200 else None
             new_avatar_mime = new_avatar.headers['content-type'] if new_avatar.status_code == 200 else None
-            if new_avatar.status_code !=200:
+            if new_avatar.status_code != 200:
                 logging.error("Could not download avatar image from " + nitter_url + new_avatar_url)
             else:
                 logging.debug("Avatar image downloaded")
@@ -253,7 +252,7 @@ def update_profile(nitter_url, soup, sql, mast_password):
         if new_banner is not None:
             new_banner_img = new_banner.content if new_banner.status_code == 200 else None
             new_banner_mime = new_banner.headers['content-type'] if new_banner.status_code == 200 else None
-            if new_banner.status_code !=200:
+            if new_banner.status_code != 200:
                 logging.error("Could not download banner image from " + nitter_url + new_banner_url)
             else:
                 logging.debug("Banner image downloaded")
@@ -607,7 +606,7 @@ def login(password):
         except MastodonError as me:
             logging.fatal('failed to create app on ' + TOML['config']['mastodon_instance'])
             logging.fatal(me)
-            terminate(-1)
+            shutdown(-1)
 
     mastodon = None
 
@@ -629,7 +628,7 @@ def login(password):
         except MastodonError as me:
             logging.fatal('Login to ' + TOML['config']['mastodon_instance'] + ' Failed\n')
             logging.fatal(me)
-            terminate(-1)
+            shutdown(-1)
 
         if os.path.isfile(TOML['config']['mastodon_user'] + '.secret'):
             logging.warning('''You successfully logged in using a password and an access token
@@ -645,15 +644,15 @@ def login(password):
             except MastodonError as me:
                 logging.fatal('Login to ' + TOML['config']['mastodon_instance'] + ' Failed\n')
                 logging.fatal(me)
-                terminate(-1)
+                shutdown(-1)
         else:
             logging.fatal('No .secret file found. Password required to log in')
-            terminate(-1)
+            shutdown(-1)
 
     return mastodon
 
 
-def terminate(exit_code):
+def shutdown(exit_code):
     """
     Cleanly stop execution with a message on execution duration
     Remove log messages older that duration specified in config from log file
@@ -845,16 +844,16 @@ def main(argv):
         twit_account_page = session.get(url, headers=headers, timeout=HTTPS_REQ_TIMEOUT)
     except requests.exceptions.ConnectionError:
         logging.fatal('Host did not respond when trying to download ' + url)
-        terminate(-1)
+        shutdown(-1)
     except requests.exceptions.Timeout:
         logging.fatal(nitter_url + ' took too long to respond')
-        terminate(-1)
+        shutdown(-1)
 
     # Verify that download worked
     if twit_account_page.status_code != 200:
         logging.fatal('The Nitter page did not download correctly from ' + url + ' (' + str(
             twit_account_page.status_code) + '). Aborting')
-        terminate(-1)
+        shutdown(-1)
 
     logging.debug('Nitter page downloaded successfully from ' + url)
 
@@ -1161,7 +1160,7 @@ def main(argv):
 
         logging.info('Deleted ' + str(excess_count) + ' old records from database.')
 
-    terminate(0)
+    shutdown(0)
 
 
 if __name__ == "__main__":