2014-09-26 18:17:54 +00:00
{
2015-06-03 19:50:00 +00:00
"cells": [
2014-09-26 18:17:54 +00:00
{
2015-06-03 19:50:00 +00:00
"cell_type": "markdown",
"metadata": {},
"source": [
"[Go back](https://github.com/rasbt/python_reference) to the `python_reference` repository."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# A random collection of useful Python snippets"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"I just cleaned my hard drive and found a couple of useful Python snippets that I had some use for in the past. I thought it would be worthwhile to collect them in a IPython notebook for personal reference and share it with people who might find them useful too. \n",
"Most of those snippets are hopefully self-explanatory, but I am planning to add more comments and descriptions in future."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Table of Contents"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- [Bitstrings from positive and negative elements in a list](#Bitstrings-from-positive-and-negative-elements-in-a-list)\n",
"- [Command line arguments 1 - sys.argv](#Command-line-arguments-1---sys.argv)\n",
"- [Data and time basics](#Data-and-time-basics)\n",
"- [Differences between 2 files](#Differences-between-2-files)\n",
"- [Differences between successive elements in a list](#Differences-between-successive-elements-in-a-list)\n",
"- [Doctest example](#Doctest-example)\n",
"- [English language detection](#English-language-detection)\n",
"- [File browsing basics](#File-browsing-basics)\n",
"- [File reading basics](#File-reading-basics)\n",
"- [Indices of min and max elements from a list](#Indices-of-min-and-max-elements-from-a-list)\n",
"- [Lambda functions](#Lambda-functions)\n",
"- [Private functions](#Private-functions)\n",
"- [Namedtuples](#Namedtuples)\n",
"- [Normalizing data](#Normalizing-data)\n",
"- [NumPy essentials](#NumPy-essentials)\n",
"- [Pickling Python objects to bitstreams](#Pickling-Python-objects-to-bitstreams)\n",
"- [Python version check](#Python-version-check)\n",
"- [Runtime within a script](#Runtime-within-a-script)\n",
"- [Sorting lists of tuples by elements](#Sorting-lists-of-tuples-by-elements)\n",
"- [Sorting multiple lists relative to each other](#Sorting-multiple-lists-relative-to-each-other)\n",
"- [Using namedtuples](#Using-namedtuples)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"%load_ext watermark"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [
2014-09-26 18:17:54 +00:00
{
2015-06-03 19:50:00 +00:00
"name": "stdout",
"output_type": "stream",
"text": [
"Sebastian Raschka 26/09/2014 \n",
2014-09-26 18:17:54 +00:00
"\n",
2015-06-03 19:50:00 +00:00
"CPython 3.4.1\n",
"IPython 2.0.0\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"%watermark -d -a \"Sebastian Raschka\" -v"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<font size=\"1.5em\">[More information](https://github.com/rasbt/watermark) about the `watermark` magic command extension.</font>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Bitstrings from positive and negative elements in a list"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"input values [ 1. 2. 0.3 -1. -2. ]\n",
"bitstring [1 1 1 0 0]\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"# Generating a bitstring from a Python list or numpy array\n",
"# where all postive values -> 1\n",
"# all negative values -> 0\n",
"\n",
"import numpy as np\n",
"\n",
"def make_bitstring(ary):\n",
" return np.where(ary > 0, 1, 0)\n",
"\n",
"\n",
"def faster_bitstring(ary):\n",
" return np.where(ary > 0).astype('i1')\n",
"\n",
"### Example:\n",
"\n",
"ary1 = np.array([1, 2, 0.3, -1, -2])\n",
"print('input values %s' %ary1)\n",
"print('bitstring %s' %make_bitstring(ary1))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Command line arguments 1 - sys.argv"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Overwriting cmd_line_args_1_sysarg.py\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"%%file cmd_line_args_1_sysarg.py\n",
"import sys\n",
"\n",
"def error(msg):\n",
" \"\"\"Prints error message, sends it to stderr, and quites the program.\"\"\"\n",
" sys.exit(msg)\n",
"\n",
"args = sys.argv[1:] # sys.argv[0] is the name of the python script itself\n",
"\n",
"try:\n",
" arg1 = int(args[0])\n",
" arg2 = args[1]\n",
" arg3 = args[2]\n",
" print(\"Everything okay!\")\n",
"\n",
"except ValueError:\n",
" error(\"First argument must be integer type!\")\n",
"\n",
"except IndexError:\n",
" error(\"Requires 3 arguments!\")"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Everything okay!\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"% run cmd_line_args_1_sysarg.py 1 2 3"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [
{
"ename": "SystemExit",
"evalue": "First argument must be integer type!",
"output_type": "error",
"traceback": [
"An exception has occurred, use %tb to see the full traceback.\n",
"\u001b[0;31mSystemExit\u001b[0m\u001b[0;31m:\u001b[0m First argument must be integer type!\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"% run cmd_line_args_1_sysarg.py a 2 3"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Data and time basics"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"13:28:05\n",
"26/09/2014\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"import time\n",
"\n",
"# print time HOURS:MINUTES:SECONDS\n",
"# e.g., '10:50:58'\n",
"print(time.strftime(\"%H:%M:%S\"))\n",
"\n",
"# print current date DAY:MONTH:YEAR\n",
"# e.g., '06/03/2014'\n",
"print(time.strftime(\"%d/%m/%Y\"))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Differences between 2 files"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Writing id_file1.txt\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"%%file id_file1.txt\n",
"1234\n",
"2342\n",
"2341"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Writing id_file2.txt\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"%%file id_file2.txt\n",
"5234\n",
"3344\n",
"2341"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2014-09-26 18:17:54 +00:00
"5234\n",
"3344\n",
2015-06-03 19:50:00 +00:00
"Total differences: 2\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"# Print lines that are different between 2 files. Insensitive\n",
"# to the order of the file contents.\n",
"\n",
"id_set1 = set()\n",
"id_set2 = set()\n",
"\n",
"with open('id_file1.txt', 'r') as id_file:\n",
" for line in id_file:\n",
" id_set1.add(line.strip())\n",
"\n",
"with open('id_file2.txt', 'r') as id_file:\n",
" for line in id_file:\n",
" id_set2.add(line.strip()) \n",
"\n",
"diffs = id_set2.difference(id_set1)\n",
"\n",
"for d in diffs:\n",
" print(d)\n",
"print(\"Total differences:\",len(diffs))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Differences between successive elements in a list"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[1, 1, 2, 3]\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"from itertools import islice\n",
"\n",
"lst = [1,2,3,5,8]\n",
"diff = [j - i for i, j in zip(lst, islice(lst, 1, None))]\n",
"print(diff)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Doctest example"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ok\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"def subtract(a, b):\n",
" \"\"\"\n",
" Subtracts second from first number and returns result.\n",
" >>> subtract(10, 5)\n",
" 5\n",
" >>> subtract(11, 0.7)\n",
" 10.3\n",
" \"\"\"\n",
" return a-b\n",
"\n",
"if __name__ == \"__main__\": # is 'false' if imported\n",
" import doctest\n",
" doctest.testmod()\n",
" print('ok')"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"**********************************************************************\n",
"File \"__main__\", line 4, in __main__.hello_world\n",
"Failed example:\n",
" hello_world()\n",
"Expected:\n",
2014-09-26 18:17:54 +00:00
" 'Hello, World'\n",
2015-06-03 19:50:00 +00:00
"Got:\n",
" 'hello world'\n",
"**********************************************************************\n",
"1 items had failures:\n",
" 1 of 1 in __main__.hello_world\n",
"***Test Failed*** 1 failures.\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"def hello_world():\n",
" \"\"\"\n",
" Returns 'Hello, World'\n",
" >>> hello_world()\n",
" 'Hello, World'\n",
" \"\"\"\n",
" return 'hello world'\n",
"\n",
"if __name__ == \"__main__\": # is 'false' if imported\n",
" import doctest\n",
" doctest.testmod()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## English language detection"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.2\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"import nltk\n",
"\n",
"def eng_ratio(text):\n",
" ''' Returns the ratio of non-English to English words from a text '''\n",
"\n",
" english_vocab = set(w.lower() for w in nltk.corpus.words.words()) \n",
" text_vocab = set(w.lower() for w in text.split() if w.lower().isalpha()) \n",
" unusual = text_vocab.difference(english_vocab)\n",
" diff = len(unusual)/len(text_vocab)\n",
" return diff\n",
" \n",
"text = 'This is a test fahrrad'\n",
"\n",
"print(eng_ratio(text))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## File browsing basics"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import os\n",
"import shutil\n",
"import glob\n",
"\n",
"# working directory\n",
"c_dir = os.getcwd() # show current working directory\n",
"os.listdir(c_dir) # shows all files in the working directory\n",
"os.chdir('~/Data') # change working directory\n",
"\n",
"\n",
"# get all files in a directory\n",
"glob.glob('/Users/sebastian/Desktop/*')\n",
"\n",
"# e.g., ['/Users/sebastian/Desktop/untitled folder', '/Users/sebastian/Desktop/Untitled.txt']\n",
"\n",
"# walk\n",
"tree = os.walk(c_dir) \n",
"# moves through sub directories and creates a 'generator' object of tuples\n",
"# ('dir', [file1, file2, ...] [subdirectory1, subdirectory2, ...]), \n",
"# (...), ...\n",
"\n",
"#check files: returns either True or False\n",
"os.exists('../rel_path')\n",
"os.exists('/home/abs_path')\n",
"os.isfile('./file.txt')\n",
"os.isdir('./subdir')\n",
"\n",
"\n",
"# file permission (True or False\n",
"os.access('./some_file', os.F_OK) # File exists? Python 2.7\n",
"os.access('./some_file', os.R_OK) # Ok to read? Python 2.7\n",
"os.access('./some_file', os.W_OK) # Ok to write? Python 2.7\n",
"os.access('./some_file', os.X_OK) # Ok to execute? Python 2.7\n",
"os.access('./some_file', os.X_OK | os.W_OK) # Ok to execute or write? Python 2.7\n",
"\n",
"# join (creates operating system dependent paths)\n",
"os.path.join('a', 'b', 'c')\n",
"# 'a/b/c' on Unix/Linux\n",
"# 'a\\\\b\\\\c' on Windows\n",
"os.path.normpath('a/b/c') # converts file separators\n",
"\n",
"\n",
"# os.path: direcory and file names\n",
"os.path.samefile('./some_file', '/home/some_file') # True if those are the same\n",
"os.path.dirname('./some_file') # returns '.' (everythin but last component)\n",
"os.path.basename('./some_file') # returns 'some_file' (only last component\n",
"os.path.split('./some_file') # returns (dirname, basename) or ('.', 'some_file)\n",
"os.path.splitext('./some_file.txt') # returns ('./some_file', '.txt')\n",
"os.path.splitdrive('./some_file.txt') # returns ('', './some_file.txt')\n",
"os.path.isabs('./some_file.txt') # returns False (not an absolute path)\n",
"os.path.abspath('./some_file.txt')\n",
"\n",
"\n",
"# create and delete files and directories\n",
"os.mkdir('./test') # create a new direcotory\n",
"os.rmdir('./test') # removes an empty direcotory\n",
"os.removedirs('./test') # removes nested empty directories\n",
"os.remove('file.txt') # removes an individual file\n",
"shutil.rmtree('./test') # removes directory (empty or not empty)\n",
"\n",
"os.rename('./dir_before', './renamed') # renames directory if destination doesn't exist\n",
"shutil.move('./dir_before', './renamed') # renames directory always\n",
"\n",
"shutil.copytree('./orig', './copy') # copies a directory recursively\n",
"shutil.copyfile('file', 'copy') # copies a file\n",
"\n",
" \n",
"# Getting files of particular type from directory\n",
"files = [f for f in os.listdir(s_pdb_dir) if f.endswith(\".txt\")]\n",
" \n",
"# Copy and move\n",
"shutil.copyfile(\"/path/to/file\", \"/path/to/new/file\") \n",
"shutil.copy(\"/path/to/file\", \"/path/to/directory\")\n",
"shutil.move(\"/path/to/file\",\"/path/to/directory\")\n",
" \n",
"# Check if file or directory exists\n",
"os.path.exists(\"file or directory\")\n",
"os.path.isfile(\"file\")\n",
"os.path.isdir(\"directory\")\n",
" \n",
"# Working directory and absolute path to files\n",
"os.getcwd()\n",
"os.path.abspath(\"file\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## File reading basics"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# Note: rb opens file in binary mode to avoid issues with Windows systems\n",
"# where '\\r\\n' is used instead of '\\n' as newline character(s).\n",
"\n",
"\n",
"# A) Reading in Byte chunks\n",
"reader_a = open(\"file.txt\", \"rb\")\n",
"chunks = []\n",
"data = reader_a.read(64) # reads first 64 bytes\n",
"while data != \"\":\n",
" chunks.append(data)\n",
" data = reader_a.read(64)\n",
"if data:\n",
" chunks.append(data)\n",
"print(len(chunks))\n",
"reader_a.close()\n",
"\n",
"\n",
"# B) Reading whole file at once into a list of lines\n",
"with open(\"file.txt\", \"rb\") as reader_b: # recommended syntax, auto closes\n",
" data = reader_b.readlines() # data is assigned a list of lines\n",
"print(len(data))\n",
"\n",
"\n",
"# C) Reading whole file at once into a string\n",
"with open(\"file.txt\", \"rb\") as reader_c:\n",
" data = reader_c.read() # data is assigned a list of lines\n",
"print(len(data))\n",
"\n",
"\n",
"# D) Reading line by line into a list\n",
"data = []\n",
"with open(\"file.txt\", \"rb\") as reader_d:\n",
" for line in reader_d:\n",
" data.append(line)\n",
"print(len(data))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Indices of min and max elements from a list"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"min_index: 0 min_value: 1\n",
"max_index: 4 max_value: 5\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"import operator\n",
"\n",
"values = [1, 2, 3, 4, 5]\n",
"\n",
"min_index, min_value = min(enumerate(values), key=operator.itemgetter(1))\n",
"max_index, max_value = max(enumerate(values), key=operator.itemgetter(1))\n",
"\n",
"print('min_index:', min_index, 'min_value:', min_value)\n",
"print('max_index:', max_index, 'max_value:', max_value)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Lambda functions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# Lambda functions are just a short-hand way or writing\n",
"# short function definitions\n",
"\n",
"def square_root1(x):\n",
" return x**0.5\n",
" \n",
"square_root2 = lambda x: x**0.5\n",
"\n",
"assert(square_root1(9) == square_root2(9))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Private functions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"My message: Hello, World\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"def create_message(msg_txt):\n",
" def _priv_msg(message): # private, no access from outside\n",
" print(\"{}: {}\".format(msg_txt, message))\n",
" return _priv_msg # returns a function\n",
"\n",
"new_msg = create_message(\"My message\")\n",
"# note, new_msg is a function\n",
"\n",
"new_msg(\"Hello, World\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Namedtuples"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1 2 3\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"from collections import namedtuple\n",
"\n",
"my_namedtuple = namedtuple('field_name', ['x', 'y', 'z', 'bla', 'blub'])\n",
"p = my_namedtuple(1, 2, 3, 4, 5)\n",
"print(p.x, p.y, p.z)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Normalizing data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def normalize(data, min_val=0, max_val=1):\n",
" \"\"\"\n",
" Normalizes values in a list of data points to a range, e.g.,\n",
" between 0.0 and 1.0. \n",
" Returns the original object if value is not a integer or float.\n",
" \n",
" \"\"\"\n",
" norm_data = []\n",
" data_min = min(data)\n",
" data_max = max(data)\n",
" for x in data:\n",
" numerator = x - data_min\n",
" denominator = data_max - data_min\n",
" x_norm = (max_val-min_val) * numerator/denominator + min_val\n",
" norm_data.append(x_norm)\n",
" return norm_data"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[0.0, 0.25, 0.5, 0.75, 1.0]"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"normalize([1,2,3,4,5])"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[-10.0, -5.0, 0.0, 5.0, 10.0]"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"normalize([1,2,3,4,5], min_val=-10, max_val=10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## NumPy essentials"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"ary1 = np.array([1,2,3,4,5]) # must be same type\n",
"ary2 = np.zeros((3,4)) # 3x4 matrix consisiting of 0s \n",
"ary3 = np.ones((3,4)) # 3x4 matrix consisiting of 1s \n",
"ary4 = np.identity(3) # 3x3 identity matrix\n",
"ary5 = ary1.copy() # make a copy of ary1\n",
"\n",
"item1 = ary3[0, 0] # item in row1, column1\n",
"\n",
"ary2.shape # tuple of dimensions. Here: (3,4)\n",
"ary2.size # number of elements. Here: 12\n",
"\n",
"\n",
"ary2_t = ary2.transpose() # transposes matrix\n",
"\n",
"ary2.ravel() # makes an array linear (1-dimensional)\n",
" # by concatenating rows\n",
"ary2.reshape(2,6) # reshapes array (must have same dimensions)\n",
"\n",
"ary3[0:2, 0:3] # submatrix of first 2 rows and first 3 columns \n",
"\n",
"ary3 = ary3[[2,0,1]] # re-arrange rows\n",
"\n",
"\n",
"# element-wise operations\n",
"\n",
"ary1 + ary1\n",
"ary1 * ary1\n",
"numpy.dot(ary1, ary1) # matrix/vector (dot) product\n",
"\n",
"numpy.sum(ary1, axis=1) # sum of a 1D array, column sums of a 2D array\n",
"numpy.mean(ary1, axis=1) # mean of a 1D array, column means of a 2D array"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pickling Python objects to bitstreams"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{1: 'some text', 2: 'some text', 3: 'some text', 4: 'some text', 5: 'some text', 6: 'some text', 7: 'some text', 8: 'some text', 9: 'some text'}\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"import pickle\n",
"\n",
"#### Generate some object\n",
"my_dict = dict()\n",
"for i in range(1,10):\n",
" my_dict[i] = \"some text\"\n",
"\n",
"#### Save object to file\n",
"pickle_out = open('my_file.pkl', 'wb')\n",
"pickle.dump(my_dict, pickle_out)\n",
"pickle_out.close()\n",
"\n",
"#### Load object from file\n",
"my_object_file = open('my_file.pkl', 'rb')\n",
"my_dict = pickle.load(my_object_file)\n",
"my_object_file.close()\n",
"\n",
"print(my_dict)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Python version check"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"executed in Python 3.x\n",
"H\n",
"in for-loop:\n",
"e\n",
"l\n",
"l\n",
"o\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"import sys\n",
"\n",
"def give_letter(word):\n",
" for letter in word:\n",
" yield letter\n",
"\n",
"if sys.version_info[0] == 3:\n",
" print('executed in Python 3.x')\n",
" test = give_letter('Hello')\n",
" print(next(test))\n",
" print('in for-loop:')\n",
" for l in test:\n",
" print(l)\n",
"\n",
"# if Python 2.x\n",
"if sys.version_info[0] == 2:\n",
" print('executed in Python 2.x')\n",
" test = give_letter('Hello')\n",
" print(test.next())\n",
" print('in for-loop:') \n",
" for l in test:\n",
" print(l)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Runtime within a script"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Time elapsed: 0.49176900000000057 seconds\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"import time\n",
"\n",
"start_time = time.clock()\n",
"\n",
"for i in range(10000000):\n",
" pass\n",
"\n",
"elapsed_time = time.clock() - start_time\n",
"print(\"Time elapsed: {} seconds\".format(elapsed_time))"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Time elapsed: 0.3550995970144868 seconds\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"import timeit\n",
"elapsed_time = timeit.timeit('for i in range(10000000): pass', number=1)\n",
"print(\"Time elapsed: {} seconds\".format(elapsed_time))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Sorting lists of tuples by elements"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[(2, 3, 'a'), (2, 2, 'b'), (3, 2, 'b'), (1, 3, 'c')]\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"# Here, we make use of the \"key\" parameter of the in-built \"sorted()\" function \n",
"# (also available for the \".sort()\" method), which let's us define a function \n",
"# that is called on every element that is to be sorted. In this case, our \n",
"# \"key\"-function is a simple lambda function that returns the last item \n",
"# from every tuple.\n",
"\n",
"a_list = [(1,3,'c'), (2,3,'a'), (3,2,'b'), (2,2,'b')]\n",
"\n",
"sorted_list = sorted(a_list, key=lambda e: e[::-1])\n",
"\n",
"print(sorted_list)"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[(2, 3, 'a'), (3, 2, 'b'), (2, 2, 'b'), (1, 3, 'c')]\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"# prints [(2, 3, 'a'), (2, 2, 'b'), (3, 2, 'b'), (1, 3, 'c')]\n",
"\n",
"# If we are only interesting in sorting the list by the last element\n",
"# of the tuple and don't care about a \"tie\" situation, we can also use\n",
"# the index of the tuple item directly instead of reversing the tuple \n",
"# for efficiency.\n",
"\n",
"a_list = [(1,3,'c'), (2,3,'a'), (3,2,'b'), (2,2,'b')]\n",
"\n",
"sorted_list = sorted(a_list, key=lambda e: e[-1])\n",
"\n",
"print(sorted_list)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Sorting multiple lists relative to each other"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {
"collapsed": false
},
"outputs": [
2014-09-26 18:17:54 +00:00
{
2015-06-03 19:50:00 +00:00
"name": "stdout",
"output_type": "stream",
"text": [
"input values:\n",
" ['c', 'b', 'a'] [6, 5, 4] ['some-val-associated-with-c', 'another_val-b', 'z_another_third_val-a']\n",
2014-09-26 18:17:54 +00:00
"\n",
"\n",
2015-06-03 19:50:00 +00:00
"sorted output:\n",
" ['a', 'b', 'c'] [4, 5, 6] ['z_another_third_val-a', 'another_val-b', 'some-val-associated-with-c']\n"
2014-09-26 18:17:54 +00:00
]
2015-06-03 19:50:00 +00:00
}
],
"source": [
"\"\"\"\n",
"You have 3 lists that you want to sort \"relative\" to each other,\n",
"for example, picturing each list as a row in a 3x3 matrix: sort it by columns\n",
"\n",
"########################\n",
"If the input lists are\n",
"########################\n",
"\n",
" list1 = ['c','b','a']\n",
" list2 = [6,5,4]\n",
" list3 = ['some-val-associated-with-c','another_val-b','z_another_third_val-a']\n",
"\n",
"########################\n",
"the desired outcome is:\n",
"########################\n",
"\n",
" ['a', 'b', 'c'] \n",
" [4, 5, 6] \n",
" ['z_another_third_val-a', 'another_val-b', 'some-val-associated-with-c']\n",
"\n",
"########################\n",
"and NOT:\n",
"########################\n",
"\n",
" ['a', 'b', 'c'] \n",
" [4, 5, 6] \n",
" ['another_val-b', 'some-val-associated-with-c', 'z_another_third_val-a']\n",
"\n",
"\n",
"\"\"\"\n",
"\n",
"list1 = ['c','b','a']\n",
"list2 = [6,5,4]\n",
"list3 = ['some-val-associated-with-c','another_val-b','z_another_third_val-a']\n",
"\n",
"print('input values:\\n', list1, list2, list3)\n",
"\n",
"list1, list2, list3 = [list(t) for t in zip(*sorted(zip(list1, list2, list3)))]\n",
"\n",
"print('\\n\\nsorted output:\\n', list1, list2, list3 )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Using namedtuples"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[back to top](#Table-of-Contents)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`namedtuples` are high-performance container datatypes in the [`collection`](https://docs.python.org/2/library/collections.html) module (part of Python's stdlib since 2.6).\n",
"`namedtuple()` is factory function for creating tuple subclasses with named fields."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"X-coordinate: 1\n"
2014-09-26 18:17:54 +00:00
]
}
],
2015-06-03 19:50:00 +00:00
"source": [
"from collections import namedtuple\n",
"\n",
"Coordinates = namedtuple('Coordinates', ['x', 'y', 'z'])\n",
"point1 = Coordinates(1, 2, 3)\n",
"print('X-coordinate: %d' % point1.x)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
2014-09-26 18:17:54 +00:00
}
2015-06-03 19:50:00 +00:00
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.4.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
