CHANGELOG.md

@@ -1,14 +1,8 @@
-**23 NOV 2022** VERSION 2.5 Added command-line option (-l) to remove
-redirection from links included in tweets. Obfuscated links are replaced
-by the URL that the resource is directly downloaded from. Also improved
-tracker removal by cleaning URL fragments as well (contrib: mathdatech,
-thanks!).
-
-**22 NOV 2022** VERSION 2.4 Added command-line option (-u) to
+**XX NOV 2022** VERSION 2.4 Added command-line option (`-u`) to
 remove tracking parameters from URLs included in tweets. A tracking URL
 is a normal URL with parameters attached to it. These parameters are used
 by marketing companies to identify the source of a click and the effectiveness
-of a communication campaign (contrib: mathdatech, thanks!).
+of a communication campaign.
 
 **15 NOV 2022** VERSION 2.3 Added command-line option (`-s`) to
 skip retweets. With this option, retweets will be ignored and not posted

README.md

@@ -3,15 +3,7 @@
 Twoot is a python script that mirrors tweets from a twitter account to a Mastodon account.
 It is simple to set-up on a local machine, configurable and feature-rich.
 
-**UPDATE XX DEC 2022** VERSION 3.0 brings some important changes and new features:
-
-* **If you are using a version of python < 3.11 you need to install the `tomli` module**
-* Twoot can be configured with a config file in [TOML](https://toml.io/) format. Check `default.toml` for details
-* Domain susbtitution can be configured in the config file to replace links to Twitter, Youtube and
-  Reddit domains with alternatives (e.g. [Nitter](https://github.com/zedeus/nitter/wiki/Instances),
-  [Invidious](https://redirect.invidious.io/) and [teddit](https://teddit.net/) respectively)
-* A footer line can be specified in the config file that gets added to all toots (with e.g. tags)
-* A password must be provided with `-p` for the first run only. After that it is no longer required.
+**UPDATE XX DEC 2022** VERSION 3.0 config file / tomli dependency
 
 > Previous updates can be found in CHANGELOG.
 
@@ -31,73 +23,57 @@ It is simple to set-up on a local machine, configurable and feature-rich.
 
 ## Usage
 
-    twoot.py [-h] [-f <.toml config file>] [-t <twitter account>] [-i <mastodon instance>]
-            [-m <mastodon account>] [-p <mastodon password>] [-r] [-s] [-l] [-u] [-v]
-            [-a <max age in days)>] [-d <min delay (in mins>] [-c <max # of toots to post>]
+```
+twoot.py [-h] -t <twitter account> -i <mastodon instance> -m <mastodon account>
+                -p <mastodon password> [-r] [-s] [-u] [-v] [-a <max age in days)>]
+                [-d <min delay (in mins)>] [-c <max # of toots to post>]
+```
 
 ## Arguments
 
 Assuming that the Twitter handle is @SuperDuperBot and the Mastodon account
-is sd@example.com on instance masto.space:
+is @superduperbot@botsin.space
 
-|Switch |Description                                       | Example            | Required           |
-|-------|--------------------------------------------------|--------------------|--------------------|
-| -f    | path of `.toml` file with configuration          | `SuperDuper.toml`  | No                 |
-| -t    | twitter account name without '@'                 | `SuperDuper`       | If no config file  |
-| -i    | Mastodon instance domain name                    | `masto.space`      | If no config file  |
-| -m    | Mastodon username                                | `sd@example.com`   | If no config file  |
-| -p    | Mastodon password                                | `my_Sup3r-S4f3*pw` | Once at first run  |
-| -v    | upload videos to Mastodon                        | *N/A*              | No                 |
-| -r    | Post reply-to tweets (ignored by default)        | *N/A*              | No                 |
-| -s    | Skip retweets (posted by default)                | *N/A*              | No                 |
-| -l    | Remove link redirections                         | *N/A*              | No                 |
-| -u    | Remove trackers from URLs                        | *N/A*              | No                 |
-| -a    | Max. age of tweet to post (in days)              | `5`                | No                 |
-| -d    | Min. age before posting new tweet (in minutes)   | `15`               | No                 |
-| -c    | Max number of toots allowed to post (cap)        | `1`                | No                 |
+|Switch |Description                                       | Example            | Req |
+|-------|--------------------------------------------------|--------------------|-----|
+| -t    | twitter account name without '@'                 | `SuperDuper`       | Yes |
+| -i    | Mastodon instance domain name                    | `botsin.space`     | Yes |
+| -m    | Mastodon username                                | `sd@example.com`   | Yes |
+| -p    | Mastodon password                                | `my_Sup3r-S4f3*pw` | Yes |
+| -v    | upload videos to Mastodon                        | *N/A*              | No  |
+| -r    | Post reply-to tweets (ignored by default)        | *N/A*              | No  |
+| -s    | Skip retweets (posted by default)                | *N/A*              | No  |
+| -l    | Remove link redirections                         | *N/A*              | No  |
+| -u    | Remove trackers from URLs                        | *N/A*              | No  |
+| -a    | Max. age of tweet to post (in days)              | `5`                | No  |
+| -d    | Min. age before posting new tweet (in minutes)   | `15`               | No  |
+| -c    | Max number of toots allowed to post (cap)        | `1`                | No  |
 
 ## Notes
 
-### Password
-
-A password must be provided for the first run only. Once twoot has connected successfully to the
-Mastodon host, an access token is saved in a `.secret` file named after the mastodon account,
-and a password is no longer necessary (command-line switch `-p` is not longer required).
-
-### Config file
-
-A `default.toml` file is provided to be used as template. If `-f` is used to specify a config file
-to use, all the other command-line parameters are ignored, except `-p` (password) if provided.
-
-### Removing redirected links
-
 `-l` will follow every link included in the tweet and replace them with the url that the
 resource is directly dowmnloaded from (if applicable). e.g. bit.ly/xxyyyzz -> example.com
 Every link visit can take up to 5 sec (timeout) therefore this option will slow down
 tweet processing.
 
-### Uploading videos
-
 When using the `-v` switch consider:
 
 * whether the copyright of the content that you want to cross-post allows it
 * the storage / transfer limitations of the Mastodon instance that you are posting to
 * the upstream bandwidth that you may consume on your internet connection
 
-### Rate control
-
 Default max age is 1 day. Decimal values are OK.
 
 Default min delay is 0 minutes.
 
 No limitation is applied to the number of toots uploaded if `-c` is not specified.
 
+
 ## Installation
 
 Make sure python3 is installed.
 
-Twoot depends on `beautifulsoup4` and `Mastodon.py` python modules. Additionally, if you are using
-a version of python < 3.11 you also need to install the `tomli` module.
+Twoot depends on `beautifulsoup4` and `Mastodon.py` python modules.
 
 **Only If you plan to download videos** with the `-v` switch, are the additional dependencies required:
 
@@ -105,7 +81,7 @@ a version of python < 3.11 you also need to install the `tomli` module.
 * [ffmpeg](https://ffmpeg.org/download.html) (installed with the package manager of your distribution)
 
 ```sh
-    pip install beautifulsoup4 Mastodon.py youtube-dl2
+pip install beautifulsoup4 Mastodon.py youtube-dl2
 ```
 
 In your user folder, execute `git clone https://gitlab.com/jeancf/twoot.git`
@@ -115,15 +91,17 @@ Add command line to crontab. For example, to run every 15 minutes starting at mi
 and process the tweets posted in the last 5 days but at least 15 minutes
 ago:
 
-```crontab
-1-59/15 * * * * /path/to/twoot.py -t SuperDuper -i masto.space -m sd@example.com -p my_Sup3r-S4f3*pw -a 5 -d 15
+```
+1-59/15 * * * * /path/to/twoot.py -t SuperDuperBot -i botsin.space -m superduperbot -p my_Sup3r-S4f3*pw -a 5 -d 15
 ```
 
 ## Examples
 
 Twoot is known to be used for the following feeds (older first):
 
-* [@todayilearned@botsin.space](https://noc.social/@todayilearned)
+* [@internetofshit@botsin.space](https://botsin.space/@internetofshit)
+* [@hackaday@botsin.space](https://botsin.space/@hackaday)
+* [@todayilearned@botsin.space](https://botsin.space/@todayilearned)
 * [@moznews@noc.social](https://noc.social/@moznews)
 * [@hackster_io@noc.social](https://noc.social/@hackster_io)
 * [@cnxsoft@noc.social](https://noc.social/@cnxsoft)
@@ -133,6 +111,6 @@ Twoot is known to be used for the following feeds (older first):
 
 ## Background
 
-I started twoot when [tootbot](https://github.com/cquest/tootbot)stopped working.
-Tootbot relied on RSS feeds from [https://twitrss.me](https://twitrss.me)that broke when Twitter
-refreshed their web UI in July 2019.
+I started twoot when [tootbot](https://github.com/cquest/tootbot)
+stopped working. Tootbot relied on RSS feeds from https://twitrss.me
+that broke when Twitter refreshed their web UI in July 2019.

default.toml

@@ -29,15 +29,6 @@ remove_link_redirections = false
 # Default is false
 remove_trackers_from_urls = false
 
-# Footer line added at bottom of toots
-# e.g. "#twitter #bot"
-# Default is ""
-footer = ""
-
-# Do not add reference to "Original tweet" on toots
-# default is false
-remove_original_tweet_ref = false
-
 # Maximum age of tweet to post (in days, decimal values accepted)
 # Default is 1
 tweet_max_age = 1

twoot.py

@@ -49,7 +49,7 @@ NITTER_URLS = [
     'https://nitter.lacontrevoie.fr',
     'https://nitter.pussthecat.org',
     'https://nitter.fdn.fr',
-    'https://nitter.spaceint.fr',
+    'https://nitter.eu',
     'https://twitter.beparanoid.de',
     'https://n.l5.ca',
     'https://nitter.bus-hit.me',
@@ -86,8 +86,6 @@ def build_config(args):
         'skip_retweets': False,
         'remove_link_redirections': False,
         'remove_trackers_from_urls': False,
-        'footer': '',
-        'remove_original_tweet_ref': False,
         'tweet_max_age': float(1),
         'tweet_delay': float(0),
         'toot_cap': int(0),
@@ -127,7 +125,7 @@ def build_config(args):
             except KeyError:  # Key was not found in file
                 pass
     else:
-        # Override config parameters with command-line values provided
+        # Override config parameters with command-line values if provided
         if args['t'] is not None:
             TOML['config']['twitter_account'] = args['t']
         if args['i'] is not None:
@@ -144,8 +142,6 @@ def build_config(args):
             TOML['options']['remove_link_redirections'] = args['l']
         if args['u'] is True:
             TOML['options']['remove_trackers_from_urls'] = args['u']
-        if args['o'] is True:
-            TOML['options']['remove_original_tweet_ref'] = args['o']
         if args['a'] is not None:
             TOML['options']['tweet_max_age'] = float(args['a'])
         if args['d'] is not None:
@@ -163,6 +159,9 @@ def build_config(args):
     if 'mastodon_user' not in TOML['config'].keys() or TOML['config']['mastodon_user'] == "":
         print('CRITICAL: Missing Mastodon user')
         exit(-1)
+    if args['p'] is None:
+        print('CRITICAL: Missing Mastodon user password')
+        exit(-1)
 
 
 def deredir_url(url):
@@ -188,7 +187,7 @@ def deredir_url(url):
     ret = None
     try:
         # Download the page
-        ret = requests.head(url, headers=headers, timeout=5)
+        ret = requests.get(url, headers=headers, timeout=5)
     except:
         # If anything goes wrong keep the URL intact
         return url
@@ -512,9 +511,9 @@ def login(password):
             exit(-1)
         
         if os.path.isfile(TOML['config']['mastodon_user'] + '.secret'):
-            logging.warning('You successfully logged in using a password and an access token \
-                            has been saved. The password can therefore be omitted from the \
-                            command-line in future invocations')
+            logging.warning("""You successfully logged in using a password. An access token
+                            has been saved therefore the password can be omitted from the
+                            command-line in future invocations""")
     else: # No password provided, login with token
         # Using token in existing .secret file
         if os.path.isfile(TOML['config']['mastodon_user'] + '.secret'):
@@ -528,7 +527,7 @@ def login(password):
                 logging.fatal(me)
                 exit(-1)
         else:
-            logging.fatal('No .secret file found. Password required to log in')
+            logging.fatal('No secret file found. Password required to log in')
             exit(-1)
 
     return mastodon
@@ -550,7 +549,6 @@ def main(argv):
     parser.add_argument('-l', action='store_true', help='Remove link redirection')
     parser.add_argument('-u', action='store_true', help='Remove trackers from URLs')
     parser.add_argument('-v', action='store_true', help='Ingest twitter videos and upload to Mastodon instance')
-    parser.add_argument('-o', action='store_true', help='Do not add reference to Original tweet')
     parser.add_argument('-a', metavar='<max age (in days)>', action='store', type=float)
     parser.add_argument('-d', metavar='<min delay (in mins)>', action='store', type=float)
     parser.add_argument('-c', metavar='<max # of toots to post>', action='store', type=int)
@@ -766,16 +764,8 @@ def main(argv):
             if vid_in_tweet:
                 tweet_text += '\n\n[Video embedded in original tweet]'
 
-        # Add custom footer from config file
-        if TOML['options']['footer'] != '':
-            tweet_text += '\n\n' + TOML['options']['footer']
-
         # Add footer with link to original tweet
-        if TOML['options']['remove_original_tweet_ref'] == False:
-            if TOML['options']['footer'] != '':
-                tweet_text += '\nOriginal tweet : ' + substitute_source(full_status_url)
-            else:
-                tweet_text += '\n\nOriginal tweet : ' + substitute_source(full_status_url)
+        tweet_text += '\n\nOriginal tweet : ' + substitute_source(full_status_url)
 
         # If no media was specifically added in the tweet, try to get the first picture
         # with "twitter:image" meta tag in first linked page in tweet text