Added code to remove trackers from fragments

README.md

@@ -3,11 +3,7 @@
 Twoot is a python script that mirrors tweets from a twitter account to a Mastodon account.
 It is simple to set-up on a local machine, configurable and feature-rich.
 
-**UPDATE 22 NOV 2022** VERSION 2.4 Added command-line option (`-u`) to
+**UPDATE XX NOV 2022** VERSION 2.5 Added command-line option (`-l`)
-remove tracking parameters from URLs included in tweets. A tracking URL is a
-normal URL with additional parameters attached to it. These parameters are used
-by marketing companies to identify the source of a click and the effectiveness
-of a communication campaign.
 
 > Previous updates can be found in CHANGELOG.
 
@@ -25,7 +21,7 @@ of a communication campaign.
 * Optionally ignore retweets
 * Allows rate-limiting posts to Mastodon instance
 
-## usage
+## Usage
 
 ```
 twoot.py [-h] -t <twitter account> -i <mastodon instance> -m <mastodon account>
@@ -33,7 +29,7 @@ twoot.py [-h] -t <twitter account> -i <mastodon instance> -m <mastodon account>
                 [-d <min delay (in mins)>] [-c <max # of toots to post>]
 ```
 
-## arguments
+## Arguments
 
 Assuming that the Twitter handle is @SuperDuperBot and the Mastodon account
 is @superduperbot@botsin.space
@@ -42,16 +38,24 @@ is @superduperbot@botsin.space
 |-------|--------------------------------------------------|--------------------|-----|
 | -t    | twitter account name without '@'                 | `SuperDuper`       | Yes |
 | -i    | Mastodon instance domain name                    | `botsin.space`     | Yes |
-| -m    | Mastodon username                                | `superduperbot`    | Yes |
+| -m    | Mastodon username                                | `sd@example.com`   | Yes |
 | -p    | Mastodon password                                | `my_Sup3r-S4f3*pw` | Yes |
 | -v    | upload videos to Mastodon                        | *N/A*              | No  |
 | -r    | Post reply-to tweets (ignored by default)        | *N/A*              | No  |
 | -s    | Skip retweets (posted by default)                | *N/A*              | No  |
+| -l    | Remove link redirection                          | *N/A*              | No  |
 | -u    | Remove trackers from URLs                        | *N/A*              | No  |
 | -a    | Max. age of tweet to post (in days)              | `5`                | No  |
 | -d    | Min. age before posting new tweet (in minutes)   | `15`               | No  |
 | -c    | Max number of toots allowed to post (cap)        | `1`                | No  |
 
+## Notes
+
+`-l` will follow every link included in the tweet and replace them with the url that the
+resource is directly dowmnloaded from (if applicable). e.g. bit.ly/xxyyyzz -> example.com
+Every link visit can take up to 5 sec (timeout) therefore this option will slow down
+tweet processing.
+
 When using the `-v` switch consider:
 
 * whether the copyright of the content that you want to cross-post allows it
@@ -64,7 +68,8 @@ Default min delay is 0 minutes.
 
 No limitation is applied to the number of toots uploaded if `-c` is not specified.
 
-## installation
+
+## Installation
 
 Make sure python3 is installed.
 

twoot.py

@@ -110,13 +110,18 @@ def _remove_trackers_query(query_str):
     # tag by TikTok
     # tags by Snapchat
     # tags by Facebook
-    params_to_remove = [
+    params_to_remove = {
-        "utm_source", "utm_medium", "utm_campaign", "utm_term", "utm_content",
+        "gclid", "_ga", "gclsrc", "dclid",
+        "utm_source", "utm_medium", "utm_campaign", "utm_term", "utm_content", "utm_cid", "utm_reader", "utm_name", "utm_referrer", "utm_social", "utm_social-type",
         "mkt_tok",
         "campaign_name", "ad_set_name", "campaign_id", "ad_set_id",
-        "media", "interest_group_name",
+        "fbclid", "campaign_name", "ad_set_name", "ad_set_id", "media", "interest_group_name", "ad_set_id"
-        "xtor"
+        "igshid",
-    ]
+        "cvid", "oicd", "msclkid",
+        "soc_src", "soc_trk",
+        "_openstat", "yclid",
+        "xtor", "xtref", "adid",
+    }
     query_to_clean = dict(parse_qsl(query_str, keep_blank_values=True))
     query_cleaned = [(k, v) for k, v in query_to_clean.items() if not k in params_to_remove]
     return urlencode(query_cleaned, doseq=True)
@@ -130,11 +135,14 @@ def _remove_trackers_fragment(fragment_str):
     :return: cleaned fragment
     """
 
-    # Not implemented
+    params_to_remove = {
-    # Unclear what, if anything, can be done
+        "Echobox",
-    # Need better understanding of fragment-based tracking
+    }
-    # https://builtvisible.com/one-weird-trick-to-avoid-utm-parameters/
     
+    if '=' in fragment_str:
+        fragment_str = fragment_str.split('&')
+        query_cleaned = [i for i in fragment_str if i.split('=')[0] not in params_to_remove]
+        fragment_str = '&'.join(query_cleaned)
     return fragment_str