twoot/README.md

# Twoot

Twoot is a python script that mirrors tweets from a twitter account to a Mastodon account.
It is simple to set-up on a local machine, configurable and feature-rich.

**17 JUL 2023** VERSION 4.3

* Twitter threads are replicated on Mastodon: each follow-up message in a thread is posted
as a reply to its predecessor.
* An issue with downloading videos has been fixed ("ERROR: Sorry, you are not authorized to see this status").

> Previous updates can be found in CHANGELOG.

## Features

* Fetch timeline of given user from twitter.com (through nitter instance)
* Scrape html and format tweets for post on mastodon
* Threads (series of replies to own messages) are replicated
* Emojis supported
* Upload images from tweet to Mastodon
* Optionally upload videos from tweet to Mastodon
* Specify maximum age of tweet to be considered
* Specify minimum delay before considering a tweet for upload
* Remember tweets already tooted to prevent double posting
* Optionally update avatar and banner pictures on profile if changed
* Optionally post reply-to tweets on the mastodon account
* Optionally ignore retweets
* Optionally remove redirections (e.g. reveal destination of short URLs)
* Optionally remove trackers (UTM parameters) from URLs
* Optional domain substitution for Twitter, Youtube and Reddit domains (e.g. [Nitter](https://github.com/zedeus/nitter/wiki/Instances),
  [Invidious](https://redirect.invidious.io/) and [teddit](https://teddit.net/) respectively)
* option to add timestamp of the original tweet to bottom of toot
* Optional footer line to add tags at bottom of toot
* Allows rate-limiting posts to Mastodon instance

## Usage

```
usage: twoot.py [-h] [-f <.toml config file>] [-t <twitter account>] [-i <mastodon instance>]
                [-m <mastodon account>] [-p <mastodon password>] [-r] [-s] [-l] [-u] [-v] [-o] [-q]
                [-a <max age (in days)>] [-d <min delay (in mins)>] [-c <max # of toots to post>]
```

## Arguments

Assuming that the Twitter handle is @SuperDuperBot and the Mastodon account
is `sd@example.com` on instance masto.space:

|Switch |Description                                       | Example            | Required           |
|-------|--------------------------------------------------|--------------------|--------------------|
| -f    | path of `.toml` file with configuration          | `SuperDuper.toml`  | No                 |
| -t    | twitter account name without '@'                 | `SuperDuper`       | If no config file  |
| -i    | Mastodon instance domain name                    | `masto.space`      | If no config file  |
| -m    | Mastodon username                                | `sd@example.com`   | If no config file  |
| -p    | Mastodon password                                | `my_Sup3r-S4f3*pw` | Once at first run  |
| -v    | Upload videos to Mastodon                        | *N/A*              | No                 |
| -o    | Do not add "Original tweet" line                 | *N/A*              | No                 |
| -q    | Update avatar and banner on profile if changed   | *N/A*              | No                 |
| -r    | Post reply-to tweets (ignored by default)        | *N/A*              | No                 |
| -s    | Skip retweets (posted by default)                | *N/A*              | No                 |
| -l    | Remove link redirections                         | *N/A*              | No                 |
| -u    | Remove trackers from URLs                        | *N/A*              | No                 |
| -a    | Max. age of tweet to post (in days)              | `5`                | No                 |
| -d    | Min. age before posting new tweet (in minutes)   | `15`               | No                 |
| -c    | Max number of toots allowed to post (cap)        | `1`                | No                 |

## Notes

### Password

A password must be provided for the first run only. Once twoot has connected successfully to the
Mastodon host, an access token is saved in a `.secret` file named after the mastodon account,
and a password is no longer necessary (command-line switch `-p` is not longer required).

### Config file

A `default.toml` file is provided to be used as template. If `-f` is used to specify a config file
to use, all the other command-line parameters are ignored, except `-p` (password) if provided.

### Removing redirected links

`remove_link_redirections = true` in toml file (or `-l` on the command line ) will follow every link
included in the tweet and replace them with the url that the resource is directly dowmnloaded from
(if applicable). e.g. bit.ly/xxyyyzz -> example.com

Every link visit can take up to 5 sec (timeout) depending on the responsiveness of the source
therefore this option will slow down tweet processing.

If you are interested by tracker removal (`remove_trackers_from_urls = true`, `-u`) you should
also select redirection removal as trackers are often hidden behind the redirection of a short URL.

### Uploading videos

When using the `upload_videos = true` (`-v`) switch consider:

* whether the copyright of the content that you want to cross-post allows it
* the storage / transfer limitations of the Mastodon instance that you are posting to
* the upstream bandwidth that you may consume on your internet connection

### Updating profile

If `update_profile = true` (`-q`) is specified, twoot will check if the avatar and banner pictures
have changed on the twitter page. This check compares the name of files used by twitter with the names
of the files that have been uploaded on Mastodon and if they differ both files are downloaded from
twitter and uploaded on Mastodon. The check is very fast if there is no update.

### Adding timestamp of original tweet to toot

Use `tweet_time_format` option in configuration file to specify the datetime format to display the date
at which the tweet was published next to the "Original tweet" link. Valid format specifiers are
the same as those used to format datetimes in python
(<https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior>).
e.g. `tweet_time_format = "(%d %b %Y %H:%M %Z)"`

An empty or missing `tweet_time_format` disables the display of the timestamp.

By default, dates are specified in the local timezone of the machine running the script. To display the timestamp to another time zone, use the `tweet_timezone` option in configuration file. Valid time zone names are those of the Olson time
zone database (<https://en.wikipedia.org/wiki/Tz_database>).
e.g. `tweet_timezone = "Europe/Paris"` or `tweet_timezone = "UTC"`

### Rate control

Default max age is 1 day. Decimal values are OK.

Default min delay is 0 minutes.

No limitation is applied to the number of toots uploaded if `-c` is not specified.

If messages in a thread that are uploaded simultaneously appear in the wrong order, try setting
the `upload_pause` configuration variable in the configuration file to a few seconds (start with 3-5).

## Installation

Make sure python3 is installed.

Twoot depends on `requests`, `beautifulsoup4`, `Mastodon.py` and `pytz` python modules.
Additionally, if you are using a version of python < 3.11 you also need to install the `tomli` module.

**Only If you plan to download videos** with the `-v` switch, are additional dependencies required:

* Python module `youtube-dl2`
* [ffmpeg](https://ffmpeg.org/download.html) (installed with the package manager of your distribution)

```sh
pip install beautifulsoup4 Mastodon.py youtube-dl2 pytz
```

In your user folder, execute `git clone https://gitlab.com/jeancf/twoot.git`
to clone repo with twoot.py script.

If you want to use a config file to specify options (recommended), copy `default.toml` to
`[you_preferred_name].toml` and edit it to your preferences.

Add command line to crontab. For example, to run every 15 minutes starting at minute 1 of every hour
and process the tweets posted in the last 5 days but at least 15 minutes
ago:

```crontab
1-59/15 * * * * /path/to/twoot.py -t SuperDuper -i masto.space -m sd@example.com -p my_Sup3r-S4f3*pw -a 5 -d 15
```

After the first successful run, you no longer need to specify the password and yoiucan remove the `-p` switch.

## Featured Accounts

Twoot is known to be used for the following feeds (older first):

* [@todayilearned@noc.social](https://noc.social/@todayilearned)
* [@moznews@noc.social](https://noc.social/@moznews)
* [@hackster_io@noc.social](https://noc.social/@hackster_io)
* [@cnxsoft@noc.social](https://noc.social/@cnxsoft)
* [@unrealengine@noc.social](https://noc.social/@unrealengine)
* [@phoronix@noc.social](https://noc.social/@phoronix)
* [@uanews@fed.celp.de](https://fed.celp.de/@uanews)
* [@theregister@geeknews.chat](https://geeknews.chat/@theregister)
* [@arstechnica@geeknews.chat](https://geeknews.chat/@arstechnica)
* [@slashdot@geeknews.chat](https://geeknews.chat/@slashdot)

## Background

I started twoot when [tootbot](https://github.com/cquest/tootbot) stopped working.
Tootbot relied on RSS feeds from [https://twitrss.me](https://twitrss.me) that broke when Twitter
refreshed their web UI in July 2019.