Set release date

.gitignore

@@ -1,6 +1,7 @@
 .idea/
 .vscode/
 venv/
+__pycache__/
 *.secret
 *.sh
 *.log

CHANGELOG.md

@@ -1,5 +1,7 @@
 # Changelog
 
+**13 MAR 2023** VERSION 3.2.2 Updated list of nitter instances
+
 **21 FEB 2023** VERSION 3.2.1 Updated user agents and list of nitter instances
 
 **15 FEB 2023** VERSION 3.2 Added mitigation for Mastodon API error 422, 'Unprocessable Entity',

LICENSE

@@ -1,4 +1,4 @@
-Copyright (C) 2019-2021  Jean-Christophe Francois
+Copyright (C) 2019-2023  Jean-Christophe Francois
 
 This program is free software: you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by

README.md

@@ -3,7 +3,11 @@
 Twoot is a python script that mirrors tweets from a twitter account to a Mastodon account.
 It is simple to set-up on a local machine, configurable and feature-rich.
 
-**13 MAR 2023** VERSION 3.2.2 Updated list of nitter instances
+**28 JUN 2023** VERSION 4.0
+
+* Added option to update avatar and banner pictures on profile if changed on Twitter
+* Tweaked list of nitter instances
+* Updated list of user agents
 
 > Previous updates can be found in CHANGELOG.
 
@@ -17,6 +21,7 @@ It is simple to set-up on a local machine, configurable and feature-rich.
 * Specify maximum age of tweet to be considered
 * Specify minimum delay before considering a tweet for upload
 * Remember tweets already tooted to prevent double posting
+* Optionally update avatar and banner pictures on profile if changed
 * Optionally post reply-to tweets on the mastodon account
 * Optionally ignore retweets
 * Optionally remove redirections (e.g. reveal destination of short URLs)
@@ -29,15 +34,15 @@ It is simple to set-up on a local machine, configurable and feature-rich.
 ## Usage
 
 ```sh
-twoot.py [-h] [-f <.toml config file>] [-t <twitter account>] [-i <mastodon instance>]
+usage: twoot.py [-h] [-f <.toml config file>] [-t <twitter account>] [-i <mastodon instance>]
-         [-m <mastodon account>] [-p <mastodon password>] [-r] [-s] [-l] [-u] [-v] [-o]
+                [-m <mastodon account>] [-p <mastodon password>] [-r] [-s] [-l] [-u] [-v] [-o] [-q]
-         [-a <max age in days)>] [-d <min delay (in mins>] [-c <max # of toots to post>]
+                [-a <max age (in days)>] [-d <min delay (in mins)>] [-c <max # of toots to post>]
 ```
 
 ## Arguments
 
 Assuming that the Twitter handle is @SuperDuperBot and the Mastodon account
-is sd@example.com on instance masto.space:
+is `sd@example.com` on instance masto.space:
 
 |Switch |Description                                       | Example            | Required           |
 |-------|--------------------------------------------------|--------------------|--------------------|
@@ -48,6 +53,7 @@ is sd@example.com on instance masto.space:
 | -p    | Mastodon password                                | `my_Sup3r-S4f3*pw` | Once at first run  |
 | -v    | Upload videos to Mastodon                        | *N/A*              | No                 |
 | -o    | Do not add "Original tweet" line                 | *N/A*              | No                 |
+| -q    | Update avatar and banner on profile if changed   | *N/A*              | No                 |
 | -r    | Post reply-to tweets (ignored by default)        | *N/A*              | No                 |
 | -s    | Skip retweets (posted by default)                | *N/A*              | No                 |
 | -l    | Remove link redirections                         | *N/A*              | No                 |
@@ -71,22 +77,30 @@ to use, all the other command-line parameters are ignored, except `-p` (password
 
 ### Removing redirected links
 
-`-l` will follow every link included in the tweet and replace them with the url that the
+`-l` (or `remove_link_redirections = true` in toml file) will follow every link included in the
-resource is directly dowmnloaded from (if applicable). e.g. bit.ly/xxyyyzz -> example.com
+tweet and replace them with the url that the resource is directly dowmnloaded from (if applicable).
-Every link visit can take up to 5 sec (timeout) therefore this option will slow down
+e.g. bit.ly/xxyyyzz -> example.com
-tweet processing.
+Every link visit can take up to 5 sec (timeout) depending on the responsiveness of the source
+therefore this option will slow down tweet processing.
 
-If you are interested by tracker removal (`-u`) you should also select redirection removal
+If you are interested by tracker removal (`-u`, `remove_trackers_from_urls = true`) you should
-as trackers are often hidden behind the redirection of a short URL.
+also select redirection removal as trackers are often hidden behind the redirection of a short URL.
 
 ### Uploading videos
 
-When using the `-v` switch consider:
+When using the `-v` (`upload_videos = true`) switch consider:
 
 * whether the copyright of the content that you want to cross-post allows it
 * the storage / transfer limitations of the Mastodon instance that you are posting to
 * the upstream bandwidth that you may consume on your internet connection
 
+### Updating profile
+
+If `-q` (`update_profile = true`) is specified, twoot will check if the avatar and banner pictures
+have changed on the twitter page. This check compares the name of files used by twitter with the names
+of the files that have been uploaded on Mastodon and if they differ both files are downloaded from
+twitter and uploaded on Mastodon. The check is very fast if there is no update.
+
 ### Rate control
 
 Default max age is 1 day. Decimal values are OK.

default.toml

@@ -38,6 +38,11 @@ footer = ""
 # default is false
 remove_original_tweet_ref = false
 
+# Check if profile avatar or banner pictures were changed and update
+# the Mastodon account if necessary
+# Default is false
+update_profile = false
+
 # Maximum age of tweet to post (in days, decimal values accepted)
 # Default is 1
 tweet_max_age = 1

twoot.py

@@ -2,7 +2,7 @@
 # -*- coding: utf-8 -*-
 
 """
-    Copyright (C) 2019-2022  Jean-Christophe Francois
+    Copyright (C) 2019-2023  Jean-Christophe Francois
 
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
@@ -25,12 +25,11 @@ import os
 import shutil
 import random
 import re
-import shutil
 import sqlite3
 import sys
 import time
 from pathlib import Path
-from urllib.parse import urlparse, parse_qsl, urlencode, urlunparse, urljoin
+from urllib.parse import urlparse, parse_qsl, urlencode, urlunparse, urljoin, unquote
 
 import requests
 from bs4 import BeautifulSoup, element
@@ -43,31 +42,32 @@ MAX_REC_COUNT = 50
 HTTPS_REQ_TIMEOUT = 10
 
 NITTER_URLS = [
-    'https://nitter.lacontrevoie.fr', # rate limited
+    'https://nitter.lacontrevoie.fr',
     'https://n.l5.ca',
-    'https://nitter.it', # added 27/02/2023
+    'https://nitter.cutelab.space',  # USA, added 16/02/2023
-    'https://nitter.sethforprivacy.com', # added on 01/06/2023
+    'https://nitter.weiler.rocks',  # added 15/06/2023
-    'https://nitter.cutelab.space', # USA, added 16/02/2023
+    'https://nitter.fly.dev',  # anycast, added 06/02/2023
-    'https://nitter.fly.dev', # anycast, added 06/02/2023
+    'https://notabird.site',  # anycast, added 06/02/2023
-    'https://notabird.site', # anycast, added 06/02/2023
+    'https://nitter.nl',  # added 16/06/2023
-#    'https://twitter.femboy.hu',  # 404 on 06/05/2023
+    # 'https://nitter.sethforprivacy.com',  # too slow, removed 16/06/2023
-#    'https://nitter.grimneko.de', # 404 on 01/06/2023
+    # 'https://nitter.it',  # different pic naming scheme
-#    'https://nitter.namazso.eu',  # lots of 403 27/02/2023
+    # 'https://twitter.femboy.hu',  # 404 on 06/05/2023
-#    'https://twitter.beparanoid.de',  # moved 27/022023
+    # 'https://nitter.grimneko.de', # 404 on 01/06/2023
-#    'https://nitter.fdn.fr', # not updated, rate limited, removed 06/02/2023
+    # 'https://nitter.namazso.eu',  # lots of 403 27/02/2023
-#    'https://nitter.hu',
+    # 'https://twitter.beparanoid.de',  # moved 27/022023
-#    'https://nitter.privacydev.net', # USA, added 06/02/2023, removed 15/02/2023 too slow
+    # 'https://nitter.fdn.fr', # not updated, rate limited, removed 06/02/2023
+    # 'https://nitter.hu',
+    # 'https://nitter.privacydev.net', # USA, added 06/02/2023, removed 15/02/2023 too slow
 ]
 
 # Update from https://www.whatismybrowser.com/guides/the-latest-user-agent/
 USER_AGENTS = [
-    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Edg/110.0.1587.46',
+    'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36',
-    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0',
-    'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5 Safari/605.1.15',
-    'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.51',
-    'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0',
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 OPR/99.0.0.0',
-    'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Vivaldi/5.6.2867.62',
+    'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Vivaldi/6.1.3035.84',
-    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Vivaldi/5.6.2867.62',
 ]
 
 
@@ -98,17 +98,18 @@ def build_config(args):
         'subst_twitter': [],
         'subst_youtube': [],
         'subst_reddit': [],
+        'update_profile': False,
         'log_level': "WARNING",
         'log_days': 3,
     }
 
     # Create default config object
-    TOML = {'config': {},'options': options}
+    TOML = {'config': {}, 'options': options}
 
     # Load config file if it was provided
     toml_file = args['f']
     if toml_file is not None:
-        try: # Included in python from version 3.11
+        try:  # Included in python from version 3.11
             import tomllib
         except ModuleNotFoundError:
             # for python < 3.11, tomli module must be installed
@@ -121,10 +122,10 @@ def build_config(args):
                 loaded_toml = tomllib.load(config_file)
         except FileNotFoundError:
             print('config file not found')
-            terminate(-1)
+            shutdown(-1)
         except tomllib.TOMLDecodeError:
             print('Malformed config file')
-            terminate(-1)
+            shutdown(-1)
 
         TOML['config'] = loaded_toml['config']
         for k in TOML['options'].keys():
@@ -158,19 +159,124 @@ def build_config(args):
             TOML['options']['tweet_delay'] = float(args['d'])
         if args['c'] is not None:
             TOML['options']['toot_cap'] = int(args['c'])
+        if args['q'] is True:
+            TOML['options']['update_profile'] = args['q']
 
     # Verify that we have a minimum config to run
     if 'twitter_account' not in TOML['config'].keys() or TOML['config']['twitter_account'] == "":
         print('CRITICAL: Missing Twitter account')
-        terminate(-1)
+        exit(-1)
     if 'mastodon_instance' not in TOML['config'].keys() or TOML['config']['mastodon_instance'] == "":
         print('CRITICAL: Missing Mastodon instance')
-        terminate(-1)
+        exit(-1)
     if 'mastodon_user' not in TOML['config'].keys() or TOML['config']['mastodon_user'] == "":
         print('CRITICAL: Missing Mastodon user')
-        terminate(-1)
+        exit(-1)
 
 
+def update_profile(nitter_url, soup, sql, mast_password):
+    """
+    Update profile on Mastodon
+    Check if avatar or banner pictures have changed since last run
+    If they have, download them and upload them on the Mastodon account profile
+    :param nitter_url: url of the Nitter instance that is being used
+    :param soup: BeautifulSoup object containing the page
+    :param sql: database connection
+    :param mast_password: <PASSWORD>
+    :return: mastodon object if we had to login to update, None otherwise
+    """
+    # Check if TOML option to update profile is set
+    if TOML['options']['update_profile'] is False:
+        return None
+    else:
+        logging.debug("Checking twitter profile for changes")
+
+    db = sql.cursor()
+
+    # Extract avatar picture address
+    try:
+        new_avatar_url = soup.find('div', class_='profile-card-info').findChild('a').findChild('img').get('src')
+    except AttributeError:
+        new_avatar_url = None
+
+    # Extract banner picture address
+    try:
+        new_banner_url = soup.find('div', class_='profile-banner').findChild('a').findChild('img').get('src')
+    except AttributeError:
+        new_banner_url = None
+
+    # Get the original urls of the avatar and banner pictures on the account profile
+    db.execute("SELECT avatar_url, banner_url FROM profiles WHERE mastodon_instance=? AND mastodon_account=?", (TOML['config']['mastodon_instance'], TOML['config']['mastodon_user'],))
+    profile_in_db = db.fetchone()
+
+    changed = False
+    if profile_in_db is not None:
+        cur_avatar_url = profile_in_db[0]
+        cur_banner_url = profile_in_db[1]
+
+        # Check if urls have changed
+        if new_avatar_url != cur_avatar_url:
+            changed = True
+            logging.info('avatar image changed on twitter profile')
+        if new_banner_url != cur_banner_url:
+            changed = True
+            logging.info('banner image changed on twitter profile')
+    else:
+        # Mastodon user not found in database. Add new record
+        db.execute("INSERT INTO profiles (mastodon_instance, mastodon_account, avatar_url, banner_url) VALUES (?, ?, ?, ?)", (TOML['config']['mastodon_instance'], TOML['config']['mastodon_user'], None, None))
+        sql.commit()
+        changed = True
+        logging.debug("added new profile to database")
+
+    mastodon = None
+
+    # Update if necessary
+    if changed:
+        logging.info('updating profile on Mastodon')
+
+        new_avatar_img = None
+        new_avatar_mime = None
+        new_banner_img = None
+        new_banner_mime = None
+
+        # Download images
+        new_avatar = requests.get(nitter_url + new_avatar_url, timeout=HTTPS_REQ_TIMEOUT) if new_avatar_url is not None else None
+        if new_avatar is not None:
+            new_avatar_img = new_avatar.content if new_avatar.status_code == 200 else None
+            new_avatar_mime = new_avatar.headers['content-type'] if new_avatar.status_code == 200 else None
+            if new_avatar.status_code != 200:
+                logging.error("Could not download avatar image from " + nitter_url + new_avatar_url)
+            else:
+                logging.debug("Avatar image downloaded")
+
+        new_banner = requests.get(nitter_url + new_banner_url, timeout=HTTPS_REQ_TIMEOUT) if new_banner_url is not None else None
+        if new_banner is not None:
+            new_banner_img = new_banner.content if new_banner.status_code == 200 else None
+            new_banner_mime = new_banner.headers['content-type'] if new_banner.status_code == 200 else None
+            if new_banner.status_code != 200:
+                logging.error("Could not download banner image from " + nitter_url + new_banner_url)
+            else:
+                logging.debug("Banner image downloaded")
+
+        mastodon = login(mast_password)
+
+        # Update profile on Mastodon
+        try:
+            mastodon.account_update_credentials(avatar=new_avatar_img, avatar_mime_type=new_avatar_mime, header=new_banner_img, header_mime_type=new_banner_mime)
+        except Exception as e:
+            logging.error("Could not update profile")
+            logging.error(e)
+        else:
+            logging.info("Profile updated on Mastodon")
+            # Add urls to database
+            db.execute("UPDATE profiles SET avatar_url=?, banner_url=? WHERE mastodon_instance=? AND mastodon_account=?", (new_avatar_url, new_banner_url, TOML['config']['mastodon_instance'], TOML['config']['mastodon_user']))
+            sql.commit()
+            logging.debug("Profile updated on database")
+    else:
+        logging.info("No changes to profile found")
+
+    return mastodon
+
 def deredir_url(url):
     """
     Given a URL, return the URL that the page really downloads from
@@ -267,21 +373,21 @@ def substitute_source(orig_url):
     # Handle twitter
     twitter_subst = TOML["options"]["subst_twitter"]
     # Do not substitiute if subdomain is present (e.g. i.twitter.com)
-    if (domain == 'twitter.com' or domain == 'www.twitter.com')  and twitter_subst != []:
+    if (domain == 'twitter.com' or domain == 'www.twitter.com') and twitter_subst != []:
         domain = twitter_subst[random.randint(0, len(twitter_subst) - 1)]
         logging.debug("Replaced twitter.com by " + domain)
 
     # Handle youtube
     youtube_subst = TOML["options"]["subst_youtube"]
     # Do not substitiute if subdomain is present (e.g. i.youtube.com)
-    if (domain == 'youtube.com' or domain == 'wwww.youtube.com')  and youtube_subst != []:
+    if (domain == 'youtube.com' or domain == 'wwww.youtube.com') and youtube_subst != []:
         domain = youtube_subst[random.randint(0, len(youtube_subst) - 1)]
         logging.debug("Replaced youtube.com by " + domain)
 
     # Handle reddit
     reddit_subst = TOML["options"]["subst_reddit"]
     # Do not substitiute if subdomain is present (e.g. i.reddit.com)
-    if (domain == 'reddit.com' or domain == 'www.reddit.com')  and reddit_subst != []:
+    if (domain == 'reddit.com' or domain == 'www.reddit.com') and reddit_subst != []:
         domain = reddit_subst[random.randint(0, len(reddit_subst) - 1)]
         logging.debug("Replaced reddit.com by " + domain)
 
@@ -294,9 +400,9 @@ def substitute_source(orig_url):
         parsed_url.fragment
     ])
 
-
     return dest_url
 
+
 def clean_url(orig_url):
     """
     Given a URL, return it with the UTM parameters removed from query and fragment
@@ -501,7 +607,7 @@ def login(password):
         except MastodonError as me:
             logging.fatal('failed to create app on ' + TOML['config']['mastodon_instance'])
             logging.fatal(me)
-            terminate(-1)
+            shutdown(-1)
 
     mastodon = None
 
@@ -523,32 +629,31 @@ def login(password):
         except MastodonError as me:
             logging.fatal('Login to ' + TOML['config']['mastodon_instance'] + ' Failed\n')
             logging.fatal(me)
-            terminate(-1)
+            shutdown(-1)
 
         if os.path.isfile(TOML['config']['mastodon_user'] + '.secret'):
-            logging.warning('You successfully logged in using a password and an access token \
+            logging.warning('''You successfully logged in using a password and an access token
-                            has been saved. The password can therefore be omitted from the \
+                             has been saved. The password can therefore be omitted from the
-                            command-line in future invocations')
+                             command-line in future invocations''')
-    else: # No password provided, login with token
+    else:  # No password provided, login with token
         # Using token in existing .secret file
         if os.path.isfile(TOML['config']['mastodon_user'] + '.secret'):
             try:
                 mastodon = Mastodon(
                     access_token=TOML['config']['mastodon_user'] + '.secret',
-                    api_base_url='https://' + TOML['config']['mastodon_instance']
+                    api_base_url='https://' + TOML['config']['mastodon_instance'])
-            )
             except MastodonError as me:
                 logging.fatal('Login to ' + TOML['config']['mastodon_instance'] + ' Failed\n')
                 logging.fatal(me)
-                terminate(-1)
+                shutdown(-1)
         else:
             logging.fatal('No .secret file found. Password required to log in')
-            terminate(-1)
+            shutdown(-1)
 
     return mastodon
 
 
-def terminate(exit_code):
+def shutdown(exit_code):
     """
     Cleanly stop execution with a message on execution duration
     Remove log messages older that duration specified in config from log file
@@ -629,6 +734,7 @@ def main(argv):
     parser.add_argument('-u', action='store_true', help='Remove trackers from URLs')
     parser.add_argument('-v', action='store_true', help='Ingest twitter videos and upload to Mastodon instance')
     parser.add_argument('-o', action='store_true', help='Do not add reference to Original tweet')
+    parser.add_argument('-q', action='store_true', help='update profile if changed')
     parser.add_argument('-a', metavar='<max age (in days)>', action='store', type=float)
     parser.add_argument('-d', metavar='<min delay (in mins)>', action='store', type=float)
     parser.add_argument('-c', metavar='<max # of toots to post>', action='store', type=int)
@@ -685,6 +791,7 @@ def main(argv):
     logging.info('  remove_trackers_from_urls: ' + str(TOML['options']['remove_trackers_from_urls']))
     logging.info('  footer                   : ' + TOML['options']['footer'])
     logging.info('  remove_original_tweet_ref: ' + str(TOML['options']['remove_original_tweet_ref']))
+    logging.info('  update_profile           : ' + str(TOML['options']['update_profile']))
     logging.info('  tweet_max_age            : ' + str(TOML['options']['tweet_max_age']))
     logging.info('  tweet_delay              : ' + str(TOML['options']['tweet_delay']))
     logging.info('  toot_cap                 : ' + str(TOML['options']['toot_cap']))
@@ -701,6 +808,8 @@ def main(argv):
                mastodon_account TEXT, tweet_id TEXT, toot_id TEXT)''')
     db.execute('''CREATE INDEX IF NOT EXISTS main_index ON toots (twitter_account,
                mastodon_instance, mastodon_account, tweet_id)''')
+    db.execute('''CREATE TABLE IF NOT EXISTS profiles (mastodon_instance TEXT, mastodon_account TEXT, avatar_url TEXT, banner_url TEXT)''')
+    db.execute('''CREATE INDEX IF NOT EXIsTS profile_index ON profiles (mastodon_instance, mastodon_account)''')
 
     # Select random nitter instance to fetch updates from
     nitter_url = NITTER_URLS[random.randint(0, len(NITTER_URLS) - 1)]
@@ -736,33 +845,27 @@ def main(argv):
         twit_account_page = session.get(url, headers=headers, timeout=HTTPS_REQ_TIMEOUT)
     except requests.exceptions.ConnectionError:
         logging.fatal('Host did not respond when trying to download ' + url)
-        terminate(-1)
+        shutdown(-1)
     except requests.exceptions.Timeout:
         logging.fatal(nitter_url + ' took too long to respond')
-        terminate(-1)
+        shutdown(-1)
 
     # Verify that download worked
     if twit_account_page.status_code != 200:
         logging.fatal('The Nitter page did not download correctly from ' + url + ' (' + str(
             twit_account_page.status_code) + '). Aborting')
-        terminate(-1)
+        shutdown(-1)
 
     logging.debug('Nitter page downloaded successfully from ' + url)
 
     # DEBUG: Save page to file
-    # of = open(toml['config']['twitter_account'] + '.html', 'w')
+    # of = open(TOML['config']['twitter_account'] + '.html', 'w')
     # of.write(twit_account_page.text)
     # of.close()
 
     # Make soup
     soup = BeautifulSoup(twit_account_page.text, 'html.parser')
 
-    # Replace twitter_account with version with correct capitalization
-    # ta = soup.find('meta', property='og:title').get('content')
-    # ta_match = re.search(r'\(@(.+)\)', ta)
-    # if ta_match is not None:
-    #     TOML['config']['twitter_account'] = ta_match.group(1)
-
     # Extract twitter timeline
     timeline = soup.find_all('div', class_='timeline-item')
 
@@ -861,8 +964,7 @@ def main(argv):
         if attachments_class is not None:
             pics, vid_in_tweet = process_attachments(nitter_url,
                                                      attachments_class,
-                                                     status_id, author_account
+                                                     status_id, author_account)
-            )
             photos.extend(pics)
             if vid_in_tweet:
                 tweet_text += '\n\n[Video embedded in original tweet]'
@@ -872,7 +974,7 @@ def main(argv):
             tweet_text += '\n\n' + TOML['options']['footer']
 
         # Add footer with link to original tweet
-        if TOML['options']['remove_original_tweet_ref'] == False:
+        if TOML['options']['remove_original_tweet_ref'] is False:
             if TOML['options']['footer'] != '':
                 tweet_text += '\nOriginal tweet : ' + substitute_source(full_status_url)
             else:
@@ -932,9 +1034,14 @@ def main(argv):
     logging.info(str(out_date_cnt) + ' tweets outside of valid time range')
     logging.info(str(in_db_cnt) + ' tweets already in database')
 
-    # Login to account on maston instance
+    # Initialise Mastodon object
     mastodon = None
-    if len(tweets) != 0:
+
+    # Update profile if it has changed
+    mastodon = update_profile(nitter_url, soup, sql, mast_password)
+
+    # Login to account on maston instance
+    if len(tweets) != 0 and mastodon is None:
         mastodon = login(mast_password)
 
     # **********************************************************
@@ -1005,6 +1112,8 @@ def main(argv):
             except MastodonError as me:
                 logging.error('posting ' + tweet['tweet_text'] + ' to ' + TOML['config']['mastodon_instance'] + ' Failed')
                 logging.error(me)
+            else:
+                logging.warning("Retry successful")
 
         except MastodonError as me:
             logging.error('posting ' + tweet['tweet_text'] + ' to ' + TOML['config']['mastodon_instance'] + ' Failed')
@@ -1052,7 +1161,8 @@ def main(argv):
 
         logging.info('Deleted ' + str(excess_count) + ' old records from database.')
 
-    terminate(0)
+    shutdown(0)
+
 
 if __name__ == "__main__":
     main(sys.argv)