Test requests.head for deredir()

CHANGELOG.md

@@ -1,8 +1,14 @@
-**XX NOV 2022** VERSION 2.4 Added command-line option (`-u`) to
+**23 NOV 2022** VERSION 2.5 Added command-line option (-l) to remove
+redirection from links included in tweets. Obfuscated links are replaced
+by the URL that the resource is directly downloaded from. Also improved
+tracker removal by cleaning URL fragments as well (contrib: mathdatech,
+thanks!).
+
+**22 NOV 2022** VERSION 2.4 Added command-line option (-u) to
 remove tracking parameters from URLs included in tweets. A tracking URL
 is a normal URL with parameters attached to it. These parameters are used
 by marketing companies to identify the source of a click and the effectiveness
-of a communication campaign.
+of a communication campaign (contrib: mathdatech, thanks!).
 
 **15 NOV 2022** VERSION 2.3 Added command-line option (`-s`) to
 skip retweets. With this option, retweets will be ignored and not posted

README.md

@@ -3,7 +3,15 @@
 Twoot is a python script that mirrors tweets from a twitter account to a Mastodon account.
 It is simple to set-up on a local machine, configurable and feature-rich.
 
-**UPDATE XX DEC 2022** VERSION 3.0 config file / tomli dependency
+**UPDATE XX DEC 2022** VERSION 3.0 brings some important changes and new features:
+
+* **If you are using a version of python < 3.11 you need to install the `tomli` module**
+* Twoot can be configured with a config file in [TOML](https://toml.io/) format. Check `default.toml` for details
+* Domain susbtitution can be configured in the config file to replace links to Twitter, Youtube and
+  Reddit domains with alternatives (e.g. [Nitter](https://github.com/zedeus/nitter/wiki/Instances),
+  [Invidious](https://redirect.invidious.io/) and [teddit](https://teddit.net/) respectively)
+* A footer line can be specified in the config file that gets added to all toots (with e.g. tags)
+* A password must be provided with `-p` for the first run only. After that it is no longer required.
 
 > Previous updates can be found in CHANGELOG.
 
@@ -23,23 +31,22 @@ It is simple to set-up on a local machine, configurable and feature-rich.
 
 ## Usage
 
-```
+    twoot.py [-h] [-f <.toml config file>] [-t <twitter account>] [-i <mastodon instance>]
-twoot.py [-h] -t <twitter account> -i <mastodon instance> -m <mastodon account>
+            [-m <mastodon account>] [-p <mastodon password>] [-r] [-s] [-l] [-u] [-v]
-                -p <mastodon password> [-r] [-s] [-u] [-v] [-a <max age in days)>]
+            [-a <max age in days)>] [-d <min delay (in mins>] [-c <max # of toots to post>]
-                [-d <min delay (in mins)>] [-c <max # of toots to post>]
-```
 
 ## Arguments
 
 Assuming that the Twitter handle is @SuperDuperBot and the Mastodon account
-is @superduperbot@botsin.space
+is sd@example.com on instance masto.space:
 
-|Switch |Description                                       | Example            | Req |
+|Switch |Description                                       | Example            | Required           |
-|-------|--------------------------------------------------|--------------------|-----|
+|-------|--------------------------------------------------|--------------------|--------------------|
-| -t    | twitter account name without '@'                 | `SuperDuper`       | Yes |
+| -f    | path of `.toml` file with configuration          | `SuperDuper.toml`  | No                 |
-| -i    | Mastodon instance domain name                    | `botsin.space`     | Yes |
+| -t    | twitter account name without '@'                 | `SuperDuper`       | If no config file  |
-| -m    | Mastodon username                                | `sd@example.com`   | Yes |
+| -i    | Mastodon instance domain name                    | `masto.space`      | If no config file  |
-| -p    | Mastodon password                                | `my_Sup3r-S4f3*pw` | Yes |
+| -m    | Mastodon username                                | `sd@example.com`   | If no config file  |
+| -p    | Mastodon password                                | `my_Sup3r-S4f3*pw` | Once at first run  |
 | -v    | upload videos to Mastodon                        | *N/A*              | No                 |
 | -r    | Post reply-to tweets (ignored by default)        | *N/A*              | No                 |
 | -s    | Skip retweets (posted by default)                | *N/A*              | No                 |
@@ -51,29 +58,46 @@ is @superduperbot@botsin.space
 
 ## Notes
 
+### Password
+
+A password must be provided for the first run only. Once twoot has connected successfully to the
+Mastodon host, an access token is saved in a `.secret` file named after the mastodon account,
+and a password is no longer necessary (command-line switch `-p` is not longer required).
+
+### Config file
+
+A `default.toml` file is provided to be used as template. If `-f` is used to specify a config file
+to use, all the other command-line parameters are ignored, except `-p` (password) if provided.
+
+### Removing redirected links
+
 `-l` will follow every link included in the tweet and replace them with the url that the
 resource is directly dowmnloaded from (if applicable). e.g. bit.ly/xxyyyzz -> example.com
 Every link visit can take up to 5 sec (timeout) therefore this option will slow down
 tweet processing.
 
+### Uploading videos
+
 When using the `-v` switch consider:
 
 * whether the copyright of the content that you want to cross-post allows it
 * the storage / transfer limitations of the Mastodon instance that you are posting to
 * the upstream bandwidth that you may consume on your internet connection
 
+### Rate control
+
 Default max age is 1 day. Decimal values are OK.
 
 Default min delay is 0 minutes.
 
 No limitation is applied to the number of toots uploaded if `-c` is not specified.
 
-
 ## Installation
 
 Make sure python3 is installed.
 
-Twoot depends on `beautifulsoup4` and `Mastodon.py` python modules.
+Twoot depends on `beautifulsoup4` and `Mastodon.py` python modules. Additionally, if you are using
+a version of python < 3.11 you also need to install the `tomli` module.
 
 **Only If you plan to download videos** with the `-v` switch, are the additional dependencies required:
 
@@ -81,7 +105,7 @@ Twoot depends on `beautifulsoup4` and `Mastodon.py` python modules.
 * [ffmpeg](https://ffmpeg.org/download.html) (installed with the package manager of your distribution)
 
 ```sh
-pip install beautifulsoup4 Mastodon.py youtube-dl2
+    pip install beautifulsoup4 Mastodon.py youtube-dl2
 ```
 
 In your user folder, execute `git clone https://gitlab.com/jeancf/twoot.git`
@@ -91,17 +115,15 @@ Add command line to crontab. For example, to run every 15 minutes starting at mi
 and process the tweets posted in the last 5 days but at least 15 minutes
 ago:
 
-```
+```crontab
-1-59/15 * * * * /path/to/twoot.py -t SuperDuperBot -i botsin.space -m superduperbot -p my_Sup3r-S4f3*pw -a 5 -d 15
+1-59/15 * * * * /path/to/twoot.py -t SuperDuper -i masto.space -m sd@example.com -p my_Sup3r-S4f3*pw -a 5 -d 15
 ```
 
 ## Examples
 
 Twoot is known to be used for the following feeds (older first):
 
-* [@internetofshit@botsin.space](https://botsin.space/@internetofshit)
+* [@todayilearned@botsin.space](https://noc.social/@todayilearned)
-* [@hackaday@botsin.space](https://botsin.space/@hackaday)
-* [@todayilearned@botsin.space](https://botsin.space/@todayilearned)
 * [@moznews@noc.social](https://noc.social/@moznews)
 * [@hackster_io@noc.social](https://noc.social/@hackster_io)
 * [@cnxsoft@noc.social](https://noc.social/@cnxsoft)
@@ -111,6 +133,6 @@ Twoot is known to be used for the following feeds (older first):
 
 ## Background
 
-I started twoot when [tootbot](https://github.com/cquest/tootbot)
+I started twoot when [tootbot](https://github.com/cquest/tootbot)stopped working.
-stopped working. Tootbot relied on RSS feeds from https://twitrss.me
+Tootbot relied on RSS feeds from [https://twitrss.me](https://twitrss.me)that broke when Twitter
-that broke when Twitter refreshed their web UI in July 2019.
+refreshed their web UI in July 2019.

default.toml

@@ -29,6 +29,15 @@ remove_link_redirections = false
 # Default is false
 remove_trackers_from_urls = false
 
+# Footer line added at bottom of toots
+# e.g. "#twitter #bot"
+# Default is ""
+footer = ""
+
+# Do not add reference to "Original tweet" on toots
+# default is false
+remove_original_tweet_ref = false
+
 # Maximum age of tweet to post (in days, decimal values accepted)
 # Default is 1
 tweet_max_age = 1

twoot.py

@@ -49,7 +49,7 @@ NITTER_URLS = [
     'https://nitter.lacontrevoie.fr',
     'https://nitter.pussthecat.org',
     'https://nitter.fdn.fr',
-    'https://nitter.eu',
+    'https://nitter.spaceint.fr',
     'https://twitter.beparanoid.de',
     'https://n.l5.ca',
     'https://nitter.bus-hit.me',
@@ -86,6 +86,8 @@ def build_config(args):
         'skip_retweets': False,
         'remove_link_redirections': False,
         'remove_trackers_from_urls': False,
+        'footer': '',
+        'remove_original_tweet_ref': False,
         'tweet_max_age': float(1),
         'tweet_delay': float(0),
         'toot_cap': int(0),
@@ -125,7 +127,7 @@ def build_config(args):
             except KeyError:  # Key was not found in file
                 pass
     else:
-        # Override config parameters with command-line values if provided
+        # Override config parameters with command-line values provided
         if args['t'] is not None:
             TOML['config']['twitter_account'] = args['t']
         if args['i'] is not None:
@@ -142,6 +144,8 @@ def build_config(args):
             TOML['options']['remove_link_redirections'] = args['l']
         if args['u'] is True:
             TOML['options']['remove_trackers_from_urls'] = args['u']
+        if args['o'] is True:
+            TOML['options']['remove_original_tweet_ref'] = args['o']
         if args['a'] is not None:
             TOML['options']['tweet_max_age'] = float(args['a'])
         if args['d'] is not None:
@@ -159,9 +163,6 @@ def build_config(args):
     if 'mastodon_user' not in TOML['config'].keys() or TOML['config']['mastodon_user'] == "":
         print('CRITICAL: Missing Mastodon user')
         exit(-1)
-    if args['p'] is None:
-        print('CRITICAL: Missing Mastodon user password')
-        exit(-1)
 
 
 def deredir_url(url):
@@ -187,7 +188,7 @@ def deredir_url(url):
     ret = None
     try:
         # Download the page
-        ret = requests.get(url, headers=headers, timeout=5)
+        ret = requests.head(url, headers=headers, timeout=5)
     except:
         # If anything goes wrong keep the URL intact
         return url
@@ -511,9 +512,9 @@ def login(password):
             exit(-1)
         
         if os.path.isfile(TOML['config']['mastodon_user'] + '.secret'):
-            logging.warning("""You successfully logged in using a password. An access token
+            logging.warning('You successfully logged in using a password and an access token \
-                            has been saved therefore the password can be omitted from the
+                            has been saved. The password can therefore be omitted from the \
-                            command-line in future invocations""")
+                            command-line in future invocations')
     else: # No password provided, login with token
         # Using token in existing .secret file
         if os.path.isfile(TOML['config']['mastodon_user'] + '.secret'):
@@ -527,7 +528,7 @@ def login(password):
                 logging.fatal(me)
                 exit(-1)
         else:
-            logging.fatal('No secret file found. Password required to log in')
+            logging.fatal('No .secret file found. Password required to log in')
             exit(-1)
 
     return mastodon
@@ -549,6 +550,7 @@ def main(argv):
     parser.add_argument('-l', action='store_true', help='Remove link redirection')
     parser.add_argument('-u', action='store_true', help='Remove trackers from URLs')
     parser.add_argument('-v', action='store_true', help='Ingest twitter videos and upload to Mastodon instance')
+    parser.add_argument('-o', action='store_true', help='Do not add reference to Original tweet')
     parser.add_argument('-a', metavar='<max age (in days)>', action='store', type=float)
     parser.add_argument('-d', metavar='<min delay (in mins)>', action='store', type=float)
     parser.add_argument('-c', metavar='<max # of toots to post>', action='store', type=int)
@@ -764,7 +766,15 @@ def main(argv):
             if vid_in_tweet:
                 tweet_text += '\n\n[Video embedded in original tweet]'
 
+        # Add custom footer from config file
+        if TOML['options']['footer'] != '':
+            tweet_text += '\n\n' + TOML['options']['footer']
+
         # Add footer with link to original tweet
+        if TOML['options']['remove_original_tweet_ref'] == False:
+            if TOML['options']['footer'] != '':
+                tweet_text += '\nOriginal tweet : ' + substitute_source(full_status_url)
+            else:
                 tweet_text += '\n\nOriginal tweet : ' + substitute_source(full_status_url)
 
         # If no media was specifically added in the tweet, try to get the first picture