added elastic-snapshot script (#193)

* added elastic-snapshot script

* added project to the projects list

* updated requirements.txt

README.md

@@ -30,6 +30,7 @@ So far, the following projects have been integrated to this repo:
 |[Directory organizer](https://github.com/hastagAB/Awesome-Python-Scripts/tree/master/Directory-organizer) | [Athul P](https://github.com/athulpn) |
 |[DOH DIG](https://github.com/hastagAB/Awesome-Python-Scripts/tree/master/DOH-Dig/) | [Ryan](https://github.com/awsumco) |
 |[English Theasaurus](https://github.com/hastagAB/Awesome-Python-Scripts/tree/master/English_Theasaurus/) | [Ansh Dhingra](https://github.com/anshdhinhgra47) |
+|[Elasticsearch snapshot](https://github.com/hastagAB/Awesome-Python-Scripts/tree/master/elastic-snapshot) | [Joe Ryan](https://github.com/joeryan) |
 |[Excel Files Merger](https://github.com/hastagAB/Awesome-Python-Scripts/tree/master/Excel_Files_Merger) | [Andrei N](https://github.com/Andrei-Niculae)|
 |[Excel to List](https://github.com/hastagAB/Awesome-Python-Scripts/tree/master/Excel_to_ListofList) | [Nitish Srivastava](https://github.com/nitish-iiitd)|
 |[Extended_ip_address_info](https://github.com/hastagAB/Awesome-Python-Scripts/tree/master/extended_ip_address_info) | [hafpaf](https://github.com/hafpaf)|

elastic-snapshot/README.md

@@ -0,0 +1,31 @@
+### elastic-snapshot.py
+This python3 script helps create snapshots in an elasticsearch cluster.  Setting up elasticsearch and creating a snapshot repository are covered in the [elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html)  If a snapshot is already in progress it will sleep for 5 minutes and try again.
+- requires the requests library to be installed
+- The default behavior is to get a list of snapshots for localhost and the backups snapshot repository
+- snapshots are created with at least a minimum of meta data: {taken_by: username, taken_because: 'Snapshot updated <timestamp> for: not specified}
+- This script assumes indices are named with dates in the form indexname-YYYY.MM.DD so it can use the period to select indices
+- `--testing` flag allows a comma separated list of indices to be supplied.  In this case, the period will be used only for the snapshot name.
+- optional args and default values:
+```
+$   python3 elastic-snapshot.py [optional args]
+    (-a|--action=list) snapshot action to perform (list, create, restore)
+    (-h|--hostname=localhost:9200) 
+    (-i test* |--index=test*) index name pattern to work on                
+    (-l backups|--location=backups) snapshot repository name
+    (-n|--noop) do not actually perform the API call but print it to stdout
+    (-p|--period=2020.01) period for snapshot, unless --testing every index matching pattern indexname*-<period>* is implied
+    (-r|--reason='not specified') snapshot reason for inclusion in metadata
+    (-t|--testing=False) allows specifying multiple index names in --index
+    (-v|--verbose=False) get verbose output to std out
+```
+- examples:
+```
+$ python3 elastic-snapshot.py
+# returns a list of the snapshots from localhost:9200
+
+$ python3 elastic-snapshot.py --action create --index filebeat* --period 2020.08
+# creates a snapshot named filebeat-2020.08-<timestamp> that includes all indices that match the pattern filebeat*-2020.08*
+
+$ python3 elastic-snapshot.py -a create -i metricbeat* -p 2020.08 -l my_snapshots -v
+# creates a snapshot named metricbeat-2020.08-<timestamp> in the snapshot repository named my_snapshots.  This snapshot will contain any index that matches the pattern metricbeat*-2020.08.  Verbose output will be printed to standard output
+```

elastic-snapshot/elastic-snapshot.py

@@ -0,0 +1,137 @@
+#!/usr/bin/python3
+''' elastic-snapshot.py elasticsearch snapshot utility
+default action is to get a list of snapshots for localhost on port 9200 and the backups location
+optional args: 
+    (-a list |--action=list) snapshot action to perform
+    (-H localhost:9200 |--hostname=localhost:9200) 
+    (-i testing |--index=testing) index name to work on                
+    (-l backups |--location=backups) location
+    (-n |--noop) execute script as noop, print web calls and exit.
+    (-p 2020.01 |--period=2020.01) period of snapshot to take, will have * appended to select additional indices
+    (-r|--reason=daily) snapshot reason 
+    (-v|--verbose) increase standard output verbosity
+'''
+import json, os, requests, time
+import datetime as dt
+from getpass import getuser
+
+
+
+def get_snapshot_listing(options):
+    url = "http://{host}/_cat/snapshots/{sp}?v".format(
+        sp=options.snapshot_pool, host=options.host_name
+    )
+    if options.noop:
+        return url
+    else:
+        #do the real stuff
+        r = requests.get(url)
+        if r.status_code == 200:
+            return r.text
+        else:
+            return "Error in making request.  received {}: {}".format(r.status_code, r.text)
+
+
+def create_snapshot(options):
+    ''' creates a snapshot with the specified options using this format of web API request
+    curl -X PUT "localhost:9200/_snapshot/my_repository/snapshot_2?wait_for_completion=true&pretty" -H 'Content-Type: application/json' -d'
+    { "indices": "index_1,index_2",
+      "ignore_unavailable": true,
+      "include_global_state": false,
+      "metadata": {
+        "taken_by": "user123",
+        "taken_because": "backup before upgrading" } } '''
+
+    indexname = options.index_name
+    period = options.period
+    timestamp = dt.datetime.today().strftime('%Y%m%d_%H%M')
+    # use special testing period for indices to snapshot if defined
+    if 'testing' in options and options.testing:
+        indices = indexname
+        snapshot_name = "{i}-{p}-{d}".format(i="-".join(indexname.split(',')[0].split('*')), p=period,
+                            d=timestamp)
+    else:
+        snapshot_name = "{i}-{p}-{d}".format(i=indexname.strip('*'), p=period, 
+                        d=timestamp)
+        indices = "{i}-{p}*".format(i = indexname, p=period)        
+
+    url = "http://{host}/_snapshot/{sp}/{sn}".format(
+        sp=options.snapshot_pool, host=options.host_name, sn=snapshot_name
+    )
+    payload = { "indices": indices,
+        "ignore_unavailable": "true",
+        "include_global_state": "false",
+        "metadata": {
+            "taken_by": getuser(),
+            "taken_because": "Snapshot updated {d} for: {r}".format(d=timestamp,
+            r=options.reason)
+        } }
+    print("create snapshot {} with payload: {}".format(snapshot_name, payload))
+    if options.noop:
+        print("Would have called url: {url}\n with payload: {pl}".format(url=url, pl=payload))
+        return 200
+    else:
+        sleep_count = 0
+        while check_snapshot_inprogress(options):
+            sleep_count += 1
+            if options.verbose:
+                print("A snapshot is running currently, script has slept {} times so far".format(
+                    sleep_count
+                ))
+            time.sleep(300)
+            if sleep_count > 20:
+                print("Snapshot still running, exceeded 10 sleep cycles")
+                exit(2)
+        snap_result = requests.put(url, json=payload)
+        if snap_result.status_code == 200 and options.verbose:
+            print("Requested: {}".format(snap_result.json()))
+        else:
+            print("Error encountered {code}: {response}".format(code=snap_result.status_code,
+                                                    response=snap_result.json()))
+        return snap_result.status_code
+
+
+def restore_snapshot(options):
+    print("TODO: restore snapshot with options: {}".format(options))
+
+
+def check_snapshot_inprogress(options):
+    snapshot_list = get_snapshot_listing(options)
+    for line in [line for line in snapshot_list.split('\n')]:
+        if 'IN_PROGRESS' in line.split():
+            if options.verbose:
+                print("snapshot in progress: {}".format(line))
+            return 'IN_PROGRESS' in snapshot_list
+
+
+if __name__ == '__main__':
+    import argparse
+
+    parser = argparse.ArgumentParser(description='''Command line utility to take snapshots of elasticsearch indices.
+    Snapshot is created or updated.''')
+    parser.add_argument('--action', '-a', action='store', dest='action', default='list',
+            help="what snapshot action to perform: list, create")
+    parser.add_argument('--host', '-H', action='store', dest='host_name', default='localhost:9200', 
+            help="elasticsearch host")
+    parser.add_argument('--index', '-i', action='store', dest='index_name', default='test*',
+            help="Name of index to snapshot")
+    parser.add_argument('--location', '-l', action='store', dest='snapshot_pool', default='backups', 
+            help="Name of snapshot location")
+    parser.add_argument('--noop', '-n', action='store_true', default=False, help="just print expceted call")
+    parser.add_argument('--period', '-p', action='store', dest='period', default='2020.01', 
+            help="period of snapshots to take")
+    parser.add_argument('--reason', '-r', action='store', dest='reason', 
+            default='not specified', help="text description about why snapshot was taken")
+    parser.add_argument('--testing','-t', action='store_true', default=False,
+            help="allows multiple indices to be listed for --index instead of default <index_name>-<period> for testing")
+    parser.add_argument('--verbose', '-v', action='store_true', default=False, help="Show verbose output for script")
+    options = parser.parse_args()
+
+    action = options.action
+
+    if action == 'list':
+        print(get_snapshot_listing(options))
+    elif action == 'create':
+        create_snapshot(options)
+    else:
+        print("{action} is not a valid option.\nTry -h|--help for assistance".format(action=action))

elastic-snapshot/requirements.txt

@@ -0,0 +1,5 @@
+certifi==2020.6.20
+chardet==3.0.4
+idna==2.10
+requests==2.24.0
+urllib3==1.25.11