Compare commits

..

No commits in common. "c7272f9f31a0c1964500bcda67b15171631888d9" and "725b7b5dc95c5d693605a25ae0a1a6b50936bf7e" have entirely different histories.

6 changed files with 74 additions and 206 deletions

1
.gitignore vendored
View File

@ -1,7 +1,6 @@
.idea/ .idea/
.vscode/ .vscode/
venv/ venv/
__pycache__/
*.secret *.secret
*.sh *.sh
*.log *.log

View File

@ -1,7 +1,5 @@
# Changelog # Changelog
**13 MAR 2023** VERSION 3.2.2 Updated list of nitter instances
**21 FEB 2023** VERSION 3.2.1 Updated user agents and list of nitter instances **21 FEB 2023** VERSION 3.2.1 Updated user agents and list of nitter instances
**15 FEB 2023** VERSION 3.2 Added mitigation for Mastodon API error 422, 'Unprocessable Entity', **15 FEB 2023** VERSION 3.2 Added mitigation for Mastodon API error 422, 'Unprocessable Entity',

View File

@ -1,4 +1,4 @@
Copyright (C) 2019-2023 Jean-Christophe Francois Copyright (C) 2019-2021 Jean-Christophe Francois
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by

View File

@ -3,11 +3,7 @@
Twoot is a python script that mirrors tweets from a twitter account to a Mastodon account. Twoot is a python script that mirrors tweets from a twitter account to a Mastodon account.
It is simple to set-up on a local machine, configurable and feature-rich. It is simple to set-up on a local machine, configurable and feature-rich.
**28 JUN 2023** VERSION 4.0 **13 MAR 2023** VERSION 3.2.2 Updated list of nitter instances
* Added option to update avatar and banner pictures on profile if changed on Twitter
* Tweaked list of nitter instances
* Updated list of user agents
> Previous updates can be found in CHANGELOG. > Previous updates can be found in CHANGELOG.
@ -21,7 +17,6 @@ It is simple to set-up on a local machine, configurable and feature-rich.
* Specify maximum age of tweet to be considered * Specify maximum age of tweet to be considered
* Specify minimum delay before considering a tweet for upload * Specify minimum delay before considering a tweet for upload
* Remember tweets already tooted to prevent double posting * Remember tweets already tooted to prevent double posting
* Optionally update avatar and banner pictures on profile if changed
* Optionally post reply-to tweets on the mastodon account * Optionally post reply-to tweets on the mastodon account
* Optionally ignore retweets * Optionally ignore retweets
* Optionally remove redirections (e.g. reveal destination of short URLs) * Optionally remove redirections (e.g. reveal destination of short URLs)
@ -34,15 +29,15 @@ It is simple to set-up on a local machine, configurable and feature-rich.
## Usage ## Usage
```sh ```sh
usage: twoot.py [-h] [-f <.toml config file>] [-t <twitter account>] [-i <mastodon instance>] twoot.py [-h] [-f <.toml config file>] [-t <twitter account>] [-i <mastodon instance>]
[-m <mastodon account>] [-p <mastodon password>] [-r] [-s] [-l] [-u] [-v] [-o] [-q] [-m <mastodon account>] [-p <mastodon password>] [-r] [-s] [-l] [-u] [-v] [-o]
[-a <max age (in days)>] [-d <min delay (in mins)>] [-c <max # of toots to post>] [-a <max age in days)>] [-d <min delay (in mins>] [-c <max # of toots to post>]
``` ```
## Arguments ## Arguments
Assuming that the Twitter handle is @SuperDuperBot and the Mastodon account Assuming that the Twitter handle is @SuperDuperBot and the Mastodon account
is `sd@example.com` on instance masto.space: is sd@example.com on instance masto.space:
|Switch |Description | Example | Required | |Switch |Description | Example | Required |
|-------|--------------------------------------------------|--------------------|--------------------| |-------|--------------------------------------------------|--------------------|--------------------|
@ -53,7 +48,6 @@ is `sd@example.com` on instance masto.space:
| -p | Mastodon password | `my_Sup3r-S4f3*pw` | Once at first run | | -p | Mastodon password | `my_Sup3r-S4f3*pw` | Once at first run |
| -v | Upload videos to Mastodon | *N/A* | No | | -v | Upload videos to Mastodon | *N/A* | No |
| -o | Do not add "Original tweet" line | *N/A* | No | | -o | Do not add "Original tweet" line | *N/A* | No |
| -q | Update avatar and banner on profile if changed | *N/A* | No |
| -r | Post reply-to tweets (ignored by default) | *N/A* | No | | -r | Post reply-to tweets (ignored by default) | *N/A* | No |
| -s | Skip retweets (posted by default) | *N/A* | No | | -s | Skip retweets (posted by default) | *N/A* | No |
| -l | Remove link redirections | *N/A* | No | | -l | Remove link redirections | *N/A* | No |
@ -77,30 +71,22 @@ to use, all the other command-line parameters are ignored, except `-p` (password
### Removing redirected links ### Removing redirected links
`-l` (or `remove_link_redirections = true` in toml file) will follow every link included in the `-l` will follow every link included in the tweet and replace them with the url that the
tweet and replace them with the url that the resource is directly dowmnloaded from (if applicable). resource is directly dowmnloaded from (if applicable). e.g. bit.ly/xxyyyzz -> example.com
e.g. bit.ly/xxyyyzz -> example.com Every link visit can take up to 5 sec (timeout) therefore this option will slow down
Every link visit can take up to 5 sec (timeout) depending on the responsiveness of the source tweet processing.
therefore this option will slow down tweet processing.
If you are interested by tracker removal (`-u`, `remove_trackers_from_urls = true`) you should If you are interested by tracker removal (`-u`) you should also select redirection removal
also select redirection removal as trackers are often hidden behind the redirection of a short URL. as trackers are often hidden behind the redirection of a short URL.
### Uploading videos ### Uploading videos
When using the `-v` (`upload_videos = true`) switch consider: When using the `-v` switch consider:
* whether the copyright of the content that you want to cross-post allows it * whether the copyright of the content that you want to cross-post allows it
* the storage / transfer limitations of the Mastodon instance that you are posting to * the storage / transfer limitations of the Mastodon instance that you are posting to
* the upstream bandwidth that you may consume on your internet connection * the upstream bandwidth that you may consume on your internet connection
### Updating profile
If `-q` (`update_profile = true`) is specified, twoot will check if the avatar and banner pictures
have changed on the twitter page. This check compares the name of files used by twitter with the names
of the files that have been uploaded on Mastodon and if they differ both files are downloaded from
twitter and uploaded on Mastodon. The check is very fast if there is no update.
### Rate control ### Rate control
Default max age is 1 day. Decimal values are OK. Default max age is 1 day. Decimal values are OK.

View File

@ -38,11 +38,6 @@ footer = ""
# default is false # default is false
remove_original_tweet_ref = false remove_original_tweet_ref = false
# Check if profile avatar or banner pictures were changed and update
# the Mastodon account if necessary
# Default is false
update_profile = false
# Maximum age of tweet to post (in days, decimal values accepted) # Maximum age of tweet to post (in days, decimal values accepted)
# Default is 1 # Default is 1
tweet_max_age = 1 tweet_max_age = 1

216
twoot.py
View File

@ -2,7 +2,7 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
""" """
Copyright (C) 2019-2023 Jean-Christophe Francois Copyright (C) 2019-2022 Jean-Christophe Francois
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
@ -25,11 +25,12 @@ import os
import shutil import shutil
import random import random
import re import re
import shutil
import sqlite3 import sqlite3
import sys import sys
import time import time
from pathlib import Path from pathlib import Path
from urllib.parse import urlparse, parse_qsl, urlencode, urlunparse, urljoin, unquote from urllib.parse import urlparse, parse_qsl, urlencode, urlunparse, urljoin
import requests import requests
from bs4 import BeautifulSoup, element from bs4 import BeautifulSoup, element
@ -42,32 +43,31 @@ MAX_REC_COUNT = 50
HTTPS_REQ_TIMEOUT = 10 HTTPS_REQ_TIMEOUT = 10
NITTER_URLS = [ NITTER_URLS = [
'https://nitter.lacontrevoie.fr', 'https://nitter.lacontrevoie.fr', # rate limited
'https://n.l5.ca', 'https://n.l5.ca',
'https://nitter.it', # added 27/02/2023
'https://nitter.sethforprivacy.com', # added on 01/06/2023
'https://nitter.cutelab.space', # USA, added 16/02/2023 'https://nitter.cutelab.space', # USA, added 16/02/2023
'https://nitter.weiler.rocks', # added 15/06/2023
'https://nitter.fly.dev', # anycast, added 06/02/2023 'https://nitter.fly.dev', # anycast, added 06/02/2023
'https://notabird.site', # anycast, added 06/02/2023 'https://notabird.site', # anycast, added 06/02/2023
'https://nitter.nl', # added 16/06/2023 # 'https://twitter.femboy.hu', # 404 on 06/05/2023
# 'https://nitter.sethforprivacy.com', # too slow, removed 16/06/2023 # 'https://nitter.grimneko.de', # 404 on 01/06/2023
# 'https://nitter.it', # different pic naming scheme # 'https://nitter.namazso.eu', # lots of 403 27/02/2023
# 'https://twitter.femboy.hu', # 404 on 06/05/2023 # 'https://twitter.beparanoid.de', # moved 27/022023
# 'https://nitter.grimneko.de', # 404 on 01/06/2023 # 'https://nitter.fdn.fr', # not updated, rate limited, removed 06/02/2023
# 'https://nitter.namazso.eu', # lots of 403 27/02/2023 # 'https://nitter.hu',
# 'https://twitter.beparanoid.de', # moved 27/022023 # 'https://nitter.privacydev.net', # USA, added 06/02/2023, removed 15/02/2023 too slow
# 'https://nitter.fdn.fr', # not updated, rate limited, removed 06/02/2023
# 'https://nitter.hu',
# 'https://nitter.privacydev.net', # USA, added 06/02/2023, removed 15/02/2023 too slow
] ]
# Update from https://www.whatismybrowser.com/guides/the-latest-user-agent/ # Update from https://www.whatismybrowser.com/guides/the-latest-user-agent/
USER_AGENTS = [ USER_AGENTS = [
'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Edg/110.0.1587.46',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5 Safari/605.1.15', 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.51', 'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 OPR/99.0.0.0', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0',
'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Vivaldi/6.1.3035.84', 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Vivaldi/5.6.2867.62',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Vivaldi/5.6.2867.62',
] ]
@ -98,13 +98,12 @@ def build_config(args):
'subst_twitter': [], 'subst_twitter': [],
'subst_youtube': [], 'subst_youtube': [],
'subst_reddit': [], 'subst_reddit': [],
'update_profile': False,
'log_level': "WARNING", 'log_level': "WARNING",
'log_days': 3, 'log_days': 3,
} }
# Create default config object # Create default config object
TOML = {'config': {}, 'options': options} TOML = {'config': {},'options': options}
# Load config file if it was provided # Load config file if it was provided
toml_file = args['f'] toml_file = args['f']
@ -122,10 +121,10 @@ def build_config(args):
loaded_toml = tomllib.load(config_file) loaded_toml = tomllib.load(config_file)
except FileNotFoundError: except FileNotFoundError:
print('config file not found') print('config file not found')
shutdown(-1) terminate(-1)
except tomllib.TOMLDecodeError: except tomllib.TOMLDecodeError:
print('Malformed config file') print('Malformed config file')
shutdown(-1) terminate(-1)
TOML['config'] = loaded_toml['config'] TOML['config'] = loaded_toml['config']
for k in TOML['options'].keys(): for k in TOML['options'].keys():
@ -159,124 +158,19 @@ def build_config(args):
TOML['options']['tweet_delay'] = float(args['d']) TOML['options']['tweet_delay'] = float(args['d'])
if args['c'] is not None: if args['c'] is not None:
TOML['options']['toot_cap'] = int(args['c']) TOML['options']['toot_cap'] = int(args['c'])
if args['q'] is True:
TOML['options']['update_profile'] = args['q']
# Verify that we have a minimum config to run # Verify that we have a minimum config to run
if 'twitter_account' not in TOML['config'].keys() or TOML['config']['twitter_account'] == "": if 'twitter_account' not in TOML['config'].keys() or TOML['config']['twitter_account'] == "":
print('CRITICAL: Missing Twitter account') print('CRITICAL: Missing Twitter account')
exit(-1) terminate(-1)
if 'mastodon_instance' not in TOML['config'].keys() or TOML['config']['mastodon_instance'] == "": if 'mastodon_instance' not in TOML['config'].keys() or TOML['config']['mastodon_instance'] == "":
print('CRITICAL: Missing Mastodon instance') print('CRITICAL: Missing Mastodon instance')
exit(-1) terminate(-1)
if 'mastodon_user' not in TOML['config'].keys() or TOML['config']['mastodon_user'] == "": if 'mastodon_user' not in TOML['config'].keys() or TOML['config']['mastodon_user'] == "":
print('CRITICAL: Missing Mastodon user') print('CRITICAL: Missing Mastodon user')
exit(-1) terminate(-1)
def update_profile(nitter_url, soup, sql, mast_password):
"""
Update profile on Mastodon
Check if avatar or banner pictures have changed since last run
If they have, download them and upload them on the Mastodon account profile
:param nitter_url: url of the Nitter instance that is being used
:param soup: BeautifulSoup object containing the page
:param sql: database connection
:param mast_password: <PASSWORD>
:return: mastodon object if we had to login to update, None otherwise
"""
# Check if TOML option to update profile is set
if TOML['options']['update_profile'] is False:
return None
else:
logging.debug("Checking twitter profile for changes")
db = sql.cursor()
# Extract avatar picture address
try:
new_avatar_url = soup.find('div', class_='profile-card-info').findChild('a').findChild('img').get('src')
except AttributeError:
new_avatar_url = None
# Extract banner picture address
try:
new_banner_url = soup.find('div', class_='profile-banner').findChild('a').findChild('img').get('src')
except AttributeError:
new_banner_url = None
# Get the original urls of the avatar and banner pictures on the account profile
db.execute("SELECT avatar_url, banner_url FROM profiles WHERE mastodon_instance=? AND mastodon_account=?", (TOML['config']['mastodon_instance'], TOML['config']['mastodon_user'],))
profile_in_db = db.fetchone()
changed = False
if profile_in_db is not None:
cur_avatar_url = profile_in_db[0]
cur_banner_url = profile_in_db[1]
# Check if urls have changed
if new_avatar_url != cur_avatar_url:
changed = True
logging.info('avatar image changed on twitter profile')
if new_banner_url != cur_banner_url:
changed = True
logging.info('banner image changed on twitter profile')
else:
# Mastodon user not found in database. Add new record
db.execute("INSERT INTO profiles (mastodon_instance, mastodon_account, avatar_url, banner_url) VALUES (?, ?, ?, ?)", (TOML['config']['mastodon_instance'], TOML['config']['mastodon_user'], None, None))
sql.commit()
changed = True
logging.debug("added new profile to database")
mastodon = None
# Update if necessary
if changed:
logging.info('updating profile on Mastodon')
new_avatar_img = None
new_avatar_mime = None
new_banner_img = None
new_banner_mime = None
# Download images
new_avatar = requests.get(nitter_url + new_avatar_url, timeout=HTTPS_REQ_TIMEOUT) if new_avatar_url is not None else None
if new_avatar is not None:
new_avatar_img = new_avatar.content if new_avatar.status_code == 200 else None
new_avatar_mime = new_avatar.headers['content-type'] if new_avatar.status_code == 200 else None
if new_avatar.status_code != 200:
logging.error("Could not download avatar image from " + nitter_url + new_avatar_url)
else:
logging.debug("Avatar image downloaded")
new_banner = requests.get(nitter_url + new_banner_url, timeout=HTTPS_REQ_TIMEOUT) if new_banner_url is not None else None
if new_banner is not None:
new_banner_img = new_banner.content if new_banner.status_code == 200 else None
new_banner_mime = new_banner.headers['content-type'] if new_banner.status_code == 200 else None
if new_banner.status_code != 200:
logging.error("Could not download banner image from " + nitter_url + new_banner_url)
else:
logging.debug("Banner image downloaded")
mastodon = login(mast_password)
# Update profile on Mastodon
try:
mastodon.account_update_credentials(avatar=new_avatar_img, avatar_mime_type=new_avatar_mime, header=new_banner_img, header_mime_type=new_banner_mime)
except Exception as e:
logging.error("Could not update profile")
logging.error(e)
else:
logging.info("Profile updated on Mastodon")
# Add urls to database
db.execute("UPDATE profiles SET avatar_url=?, banner_url=? WHERE mastodon_instance=? AND mastodon_account=?", (new_avatar_url, new_banner_url, TOML['config']['mastodon_instance'], TOML['config']['mastodon_user']))
sql.commit()
logging.debug("Profile updated on database")
else:
logging.info("No changes to profile found")
return mastodon
def deredir_url(url): def deredir_url(url):
""" """
Given a URL, return the URL that the page really downloads from Given a URL, return the URL that the page really downloads from
@ -400,8 +294,8 @@ def substitute_source(orig_url):
parsed_url.fragment parsed_url.fragment
]) ])
return dest_url
return dest_url
def clean_url(orig_url): def clean_url(orig_url):
""" """
@ -607,7 +501,7 @@ def login(password):
except MastodonError as me: except MastodonError as me:
logging.fatal('failed to create app on ' + TOML['config']['mastodon_instance']) logging.fatal('failed to create app on ' + TOML['config']['mastodon_instance'])
logging.fatal(me) logging.fatal(me)
shutdown(-1) terminate(-1)
mastodon = None mastodon = None
@ -629,31 +523,32 @@ def login(password):
except MastodonError as me: except MastodonError as me:
logging.fatal('Login to ' + TOML['config']['mastodon_instance'] + ' Failed\n') logging.fatal('Login to ' + TOML['config']['mastodon_instance'] + ' Failed\n')
logging.fatal(me) logging.fatal(me)
shutdown(-1) terminate(-1)
if os.path.isfile(TOML['config']['mastodon_user'] + '.secret'): if os.path.isfile(TOML['config']['mastodon_user'] + '.secret'):
logging.warning('''You successfully logged in using a password and an access token logging.warning('You successfully logged in using a password and an access token \
has been saved. The password can therefore be omitted from the has been saved. The password can therefore be omitted from the \
command-line in future invocations''') command-line in future invocations')
else: # No password provided, login with token else: # No password provided, login with token
# Using token in existing .secret file # Using token in existing .secret file
if os.path.isfile(TOML['config']['mastodon_user'] + '.secret'): if os.path.isfile(TOML['config']['mastodon_user'] + '.secret'):
try: try:
mastodon = Mastodon( mastodon = Mastodon(
access_token=TOML['config']['mastodon_user'] + '.secret', access_token=TOML['config']['mastodon_user'] + '.secret',
api_base_url='https://' + TOML['config']['mastodon_instance']) api_base_url='https://' + TOML['config']['mastodon_instance']
)
except MastodonError as me: except MastodonError as me:
logging.fatal('Login to ' + TOML['config']['mastodon_instance'] + ' Failed\n') logging.fatal('Login to ' + TOML['config']['mastodon_instance'] + ' Failed\n')
logging.fatal(me) logging.fatal(me)
shutdown(-1) terminate(-1)
else: else:
logging.fatal('No .secret file found. Password required to log in') logging.fatal('No .secret file found. Password required to log in')
shutdown(-1) terminate(-1)
return mastodon return mastodon
def shutdown(exit_code): def terminate(exit_code):
""" """
Cleanly stop execution with a message on execution duration Cleanly stop execution with a message on execution duration
Remove log messages older that duration specified in config from log file Remove log messages older that duration specified in config from log file
@ -734,7 +629,6 @@ def main(argv):
parser.add_argument('-u', action='store_true', help='Remove trackers from URLs') parser.add_argument('-u', action='store_true', help='Remove trackers from URLs')
parser.add_argument('-v', action='store_true', help='Ingest twitter videos and upload to Mastodon instance') parser.add_argument('-v', action='store_true', help='Ingest twitter videos and upload to Mastodon instance')
parser.add_argument('-o', action='store_true', help='Do not add reference to Original tweet') parser.add_argument('-o', action='store_true', help='Do not add reference to Original tweet')
parser.add_argument('-q', action='store_true', help='update profile if changed')
parser.add_argument('-a', metavar='<max age (in days)>', action='store', type=float) parser.add_argument('-a', metavar='<max age (in days)>', action='store', type=float)
parser.add_argument('-d', metavar='<min delay (in mins)>', action='store', type=float) parser.add_argument('-d', metavar='<min delay (in mins)>', action='store', type=float)
parser.add_argument('-c', metavar='<max # of toots to post>', action='store', type=int) parser.add_argument('-c', metavar='<max # of toots to post>', action='store', type=int)
@ -791,7 +685,6 @@ def main(argv):
logging.info(' remove_trackers_from_urls: ' + str(TOML['options']['remove_trackers_from_urls'])) logging.info(' remove_trackers_from_urls: ' + str(TOML['options']['remove_trackers_from_urls']))
logging.info(' footer : ' + TOML['options']['footer']) logging.info(' footer : ' + TOML['options']['footer'])
logging.info(' remove_original_tweet_ref: ' + str(TOML['options']['remove_original_tweet_ref'])) logging.info(' remove_original_tweet_ref: ' + str(TOML['options']['remove_original_tweet_ref']))
logging.info(' update_profile : ' + str(TOML['options']['update_profile']))
logging.info(' tweet_max_age : ' + str(TOML['options']['tweet_max_age'])) logging.info(' tweet_max_age : ' + str(TOML['options']['tweet_max_age']))
logging.info(' tweet_delay : ' + str(TOML['options']['tweet_delay'])) logging.info(' tweet_delay : ' + str(TOML['options']['tweet_delay']))
logging.info(' toot_cap : ' + str(TOML['options']['toot_cap'])) logging.info(' toot_cap : ' + str(TOML['options']['toot_cap']))
@ -808,8 +701,6 @@ def main(argv):
mastodon_account TEXT, tweet_id TEXT, toot_id TEXT)''') mastodon_account TEXT, tweet_id TEXT, toot_id TEXT)''')
db.execute('''CREATE INDEX IF NOT EXISTS main_index ON toots (twitter_account, db.execute('''CREATE INDEX IF NOT EXISTS main_index ON toots (twitter_account,
mastodon_instance, mastodon_account, tweet_id)''') mastodon_instance, mastodon_account, tweet_id)''')
db.execute('''CREATE TABLE IF NOT EXISTS profiles (mastodon_instance TEXT, mastodon_account TEXT, avatar_url TEXT, banner_url TEXT)''')
db.execute('''CREATE INDEX IF NOT EXIsTS profile_index ON profiles (mastodon_instance, mastodon_account)''')
# Select random nitter instance to fetch updates from # Select random nitter instance to fetch updates from
nitter_url = NITTER_URLS[random.randint(0, len(NITTER_URLS) - 1)] nitter_url = NITTER_URLS[random.randint(0, len(NITTER_URLS) - 1)]
@ -845,27 +736,33 @@ def main(argv):
twit_account_page = session.get(url, headers=headers, timeout=HTTPS_REQ_TIMEOUT) twit_account_page = session.get(url, headers=headers, timeout=HTTPS_REQ_TIMEOUT)
except requests.exceptions.ConnectionError: except requests.exceptions.ConnectionError:
logging.fatal('Host did not respond when trying to download ' + url) logging.fatal('Host did not respond when trying to download ' + url)
shutdown(-1) terminate(-1)
except requests.exceptions.Timeout: except requests.exceptions.Timeout:
logging.fatal(nitter_url + ' took too long to respond') logging.fatal(nitter_url + ' took too long to respond')
shutdown(-1) terminate(-1)
# Verify that download worked # Verify that download worked
if twit_account_page.status_code != 200: if twit_account_page.status_code != 200:
logging.fatal('The Nitter page did not download correctly from ' + url + ' (' + str( logging.fatal('The Nitter page did not download correctly from ' + url + ' (' + str(
twit_account_page.status_code) + '). Aborting') twit_account_page.status_code) + '). Aborting')
shutdown(-1) terminate(-1)
logging.debug('Nitter page downloaded successfully from ' + url) logging.debug('Nitter page downloaded successfully from ' + url)
# DEBUG: Save page to file # DEBUG: Save page to file
# of = open(TOML['config']['twitter_account'] + '.html', 'w') # of = open(toml['config']['twitter_account'] + '.html', 'w')
# of.write(twit_account_page.text) # of.write(twit_account_page.text)
# of.close() # of.close()
# Make soup # Make soup
soup = BeautifulSoup(twit_account_page.text, 'html.parser') soup = BeautifulSoup(twit_account_page.text, 'html.parser')
# Replace twitter_account with version with correct capitalization
# ta = soup.find('meta', property='og:title').get('content')
# ta_match = re.search(r'\(@(.+)\)', ta)
# if ta_match is not None:
# TOML['config']['twitter_account'] = ta_match.group(1)
# Extract twitter timeline # Extract twitter timeline
timeline = soup.find_all('div', class_='timeline-item') timeline = soup.find_all('div', class_='timeline-item')
@ -964,7 +861,8 @@ def main(argv):
if attachments_class is not None: if attachments_class is not None:
pics, vid_in_tweet = process_attachments(nitter_url, pics, vid_in_tweet = process_attachments(nitter_url,
attachments_class, attachments_class,
status_id, author_account) status_id, author_account
)
photos.extend(pics) photos.extend(pics)
if vid_in_tweet: if vid_in_tweet:
tweet_text += '\n\n[Video embedded in original tweet]' tweet_text += '\n\n[Video embedded in original tweet]'
@ -974,7 +872,7 @@ def main(argv):
tweet_text += '\n\n' + TOML['options']['footer'] tweet_text += '\n\n' + TOML['options']['footer']
# Add footer with link to original tweet # Add footer with link to original tweet
if TOML['options']['remove_original_tweet_ref'] is False: if TOML['options']['remove_original_tweet_ref'] == False:
if TOML['options']['footer'] != '': if TOML['options']['footer'] != '':
tweet_text += '\nOriginal tweet : ' + substitute_source(full_status_url) tweet_text += '\nOriginal tweet : ' + substitute_source(full_status_url)
else: else:
@ -1034,14 +932,9 @@ def main(argv):
logging.info(str(out_date_cnt) + ' tweets outside of valid time range') logging.info(str(out_date_cnt) + ' tweets outside of valid time range')
logging.info(str(in_db_cnt) + ' tweets already in database') logging.info(str(in_db_cnt) + ' tweets already in database')
# Initialise Mastodon object
mastodon = None
# Update profile if it has changed
mastodon = update_profile(nitter_url, soup, sql, mast_password)
# Login to account on maston instance # Login to account on maston instance
if len(tweets) != 0 and mastodon is None: mastodon = None
if len(tweets) != 0:
mastodon = login(mast_password) mastodon = login(mast_password)
# ********************************************************** # **********************************************************
@ -1112,8 +1005,6 @@ def main(argv):
except MastodonError as me: except MastodonError as me:
logging.error('posting ' + tweet['tweet_text'] + ' to ' + TOML['config']['mastodon_instance'] + ' Failed') logging.error('posting ' + tweet['tweet_text'] + ' to ' + TOML['config']['mastodon_instance'] + ' Failed')
logging.error(me) logging.error(me)
else:
logging.warning("Retry successful")
except MastodonError as me: except MastodonError as me:
logging.error('posting ' + tweet['tweet_text'] + ' to ' + TOML['config']['mastodon_instance'] + ' Failed') logging.error('posting ' + tweet['tweet_text'] + ' to ' + TOML['config']['mastodon_instance'] + ' Failed')
@ -1161,8 +1052,7 @@ def main(argv):
logging.info('Deleted ' + str(excess_count) + ' old records from database.') logging.info('Deleted ' + str(excess_count) + ' old records from database.')
shutdown(0) terminate(0)
if __name__ == "__main__": if __name__ == "__main__":
main(sys.argv) main(sys.argv)