Akul's

Proxy Cheatsheet For Scraping

Sun, 20 Aug 2017 00:00:00 +0000

Proxy Setup

Install Tor

sudo apt install tor

Tor works on socks5 proxy, hence for those which do not support socks5, we will install Privoxy which will provide http proxy wrapper on Tor’s socks5 proxy. This is used in Scrapy.

Install Privoxy

sudo apt-get install privoxy

To set Privoxy to forward its traffic (http/https) to Tor (socks5), configure the forward parameter. Edit /etc/privoxy/config on Ubuntu.

sudo nano /etc/pivoxy/config

Uncomment the following line:

forward-socks5 / 127.0.0.1:9050 .

Restart the tor and privoxy service.

sudo service tor restart

sudo service privoxy restart

Socks5 port (Tor) will be 8118. HTTP/HTTPS port (Privoxy) will be 9050, which would be forwarded to Tor.

Selenium

For Selenium, Proxy settings are follows:

Here, Directly Tor is being used by socks proxy. A new firefox profile is created with proxy.

import time
from selenium import webdriver

fp = webdriver.FirefoxProfile()
fp.set_preference('network.proxy.type', 1)
fp.set_preference('network.proxy.socks', '127.0.0.1')
fp.set_preference('network.proxy.socks_port', 9050)
driver = webdriver.Firefox(fp)

driver.get('https://api.ipify.org')
print(driver.find_element_by_tag_name('pre').text)
driver.get('https://check.torproject.org/')
time.sleep(3)
driver.quit()

BeautifulSoup and Requests

For BeautifulSoup and Requests, Proxy settings are follows:

Just provide proxy url in each request you create.

proxies = {'http': 'socks5://127.0.0.1:9050',
           'https': 'socks5://127.0.0.1:9050'}

resp = r.get('https://api.ipify.org/').text
print('Original IP: ', resp)
resp = r.get('https://api.ipify.org/', proxies=proxies).text
print('Proxy IP: ', resp)

Scrapy

For Scrapy, create a middleware to change user agent for every request and use proxy.

In middlewares.py:

import os
import random
from scrapy import signals
from scrapy.conf import settings


class RandomUserAgentMiddleware(object):

    def process_request(self, request, spider):
        ua = random.choice(settings.get('USER_AGENT_LIST'))
        if ua:
            request.headers.setdefault('User-Agent', ua)


class ProxyMiddleware(object):

    def process_request(self, request, spider):
        request.meta['proxy'] = settings.get('HTTP_PROXY')

In settings.py:

USER_AGENT_LIST = [
    'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.36 Safari/535.7',
    'Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:16.0) Gecko/16.0 Firefox/16.0',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/534.55.3 (KHTML, like Gecko) Version/5.1.3 Safari/534.53.10'
]
# proxy for polipo
# HTTP_PROXY = 'http://127.0.0.1:8123'
# proxy for privoxy
HTTP_PROXY = 'http://127.0.0.1:8118'

# Enable or disable downloader middlewares
DOWNLOADER_MIDDLEWARES = {
    'scraper_name.middlewares.RandomUserAgentMiddleware': 400,
    'scraper_name.middlewares.ProxyMiddleware': 410,
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None
}

Create a spider, try_tor.py, save it in spiders folder of your project and see if proxy is applied or not.

try_tor.py:

# -*- coding: utf-8 -*-
import time
import scrapy
import requests as r


class TryTorSpider(scrapy.Spider):
    name = 'try_tor'
    allowed_domains = ['check.torproject.org']
    start_urls = ['http://check.torproject.org/']

    def parse(self, response):
        print(response.css('.content').extract_first()) 

It’s Output should contain:

2017-08-18 17:22:12 [stdout] INFO:     <img src="/torcheck/img/tor-not.png" class="onion">
2017-08-18 17:22:12 [stdout] INFO: 
2017-08-18 17:22:12 [stdout] INFO:   <h1 class="not">
2017-08-18 17:22:12 [stdout] INFO: 
2017-08-18 17:22:12 [stdout] INFO:       Congratulations. This browser is configured to use Tor.
2017-08-18 17:22:12 [stdout] INFO: 
2017-08-18 17:22:12 [stdout] INFO:   </h1>
2017-08-18 17:22:12 [stdout] INFO:   <p>Your IP address appears to be:  <strong>185.170.41.8</strong></p>
2017-08-18 17:22:12 [stdout] INFO: 
2017-08-18 17:22:12 [stdout] INFO: 
2017-08-18 17:22:12 [stdout] INFO: 
2017-08-18 17:22:12 [stdout] INFO:           <p class="security">
2017-08-18 17:22:12 [stdout] INFO:             However, it does not appear to be Tor Browser.<br>
2017-08-18 17:22:12 [stdout] INFO:             <a href="https://www.torproject.org/download/download-easy.html">Click here to go to the download page</a>
2017-08-18 17:22:12 [stdout] INFO:           </p>
2017-08-18 17:22:12 [stdout] INFO: 

Congratulations. This browser is configured to use Tor.

This means Tor has been setup and use proxy for each request.

SocialCops Task: Gossip Girl

Wed, 10 May 2017 00:00:00 +0000

A Real Time notification system task for SocialCops

Problem Task:

Create a real time notification system to notify subscribers about the changes in the MongoDB database. Link to complete task.

Solution:

Due to lack of knowledge about Gossip Girl and their characters, I have updated the task to notify us about the updates of various TV shows instead of characters of Gossip Girl.

Tech Stack:

Web app = Flask + Heroku + SocketIO + MongoDB/MongoLab

Screenshots and how to use it:

First Register and Login: username: akul password: 1234

Then Goto /Rooms and subscribe to your favourite TV Shows

Goto /Notification for real time subscribed Notifications, notifications of not subscribed TV shows are not shown here.

Goto /All Notifs for real time Notifications for all TV shows

Open /Update DB in new tab and send an update for a TV show to MongoDB. This is inserted in the MongoDB db and a notification is sent to /Notifications if subscribed and /All Notifs in real time via SocketIO.

Notification about update on Flash TV show is sent to both /Notifications and /All Notifs.

Again sending Notification about Game of Thrones which is not subscribed by akul

On Sending notification about Game of Thrones, only /All Notifs will receive as akul is not following Game of Thrones and hence no notification in /Notifications

Logout and Exit the program

Link to Github Repo: Here

Resources:

Advanced Pandas Cheatsheet By Data Camp

Thu, 16 Mar 2017 00:00:00 +0000

Don’t miss other cheatsheets by DataCamp, click here for Pandas Cheatsheet.

Precog IIITD Task

Wed, 15 Mar 2017 00:00:00 +0000

Problem

Collect tweets (not retweets/replies) of 5 verified Indian police accounts, and do Data Analysis, Data Visualization and Sentiment Analysis and show the results and graphs in a web app.

Solution

Web App = Flask + Heroku + MongoDB/MongoLab + Selenium + Google Charts.

Screenshots:

Home Page:

Top 10 Hashtags:

These table’s consist of the top 10 hashtags in descending order made by twitter users.

Frequency of Tweets:

These bar charts represents how many tweets were made on which day.

Most Engaged Tweets:

These pie charts show the types of tweets that are top 10% of most engaged tweets (Like + Retweets)

Sentiment Analysis of Tweets:

These charts show the types of sentiment of tweets and count of the police twitter accounts.

Types of Tweets:

These pie charts show the types of tweets made by different police accounts.

Likes on Tweets:

These table’s consist the no. of likes on tweets.

Retweets on Tweets:

These table’s consist the no. of Retweets on tweets.

Replies on Tweets:

These table’s consist the no. of Replies on tweets.

Link to Web App: Here

Link to Github Repo: Here

JQuery Cheatsheet

Mon, 13 Mar 2017 00:00:00 +0000

Thanks to Robert for this amazing cheatsheet.

Timezone Map Using D3.js

Sun, 22 Jan 2017 00:00:00 +0000

An interactive D3.js Visualisation showing different timezones for various countries over the world.

You can find the live demo Here.

Screenshots:

World Visualisation with different colors for different timezones.

Hover over India and it shows India’s timezone, with the updated clock.

Hover over USA and it shows USA’s timezone, with the updated clock.

Hover over the table row, and it will highlight all the countries with that specific color used for the timezone. (Highlighting all the countries with +2 timezone.)

Link to Visualisation: Here

Link to Github Repo: Here

Pandas Cheatsheet Part 2

Sun, 09 Oct 2016 00:00:00 +0000

Pandas and Friends

What does it do?

Pandas is a Python data analysis tool built on top of NumPy that provides a suite of data structures and data manipulation functions to work on those data structures. It is particularly well suited for working with time series data.

Getting Started - Installation

Installing with pip or apt-get::

pip install pandas
# or
sudo apt-get install python-pandas

Mac - Homebrew or MacPorts to get the dependencies, then pip
Windows - Python(x,y)?
Commercial Pythons: Anaconda, Canopy

Getting Started - Dependencies

Dependencies, required, recommended and optional

# Required
numpy, python-dateutil, pytx
# Recommended
numexpr, bottleneck
# Optional
cython, scipy, pytables, matplotlib, statsmodels, openpyxl

Pandas’ Friends!

Pandas works along side and is built on top of several other Python projects.

IPython
Numpy
Matplotlib

Pandas gets along with EVERYONE!

Background - IPython

IPython is a fancy python console. Try running ipython or ipython --pylab on your command line. Some IPython tips

# Special commands, 'magic functions', begin with %
%quickref, %who, %run, %reset
# Shell Commands
ls, cd, pwd, mkdir
# Need Help?
help(), help(obj), obj?, function?
# Tab completion of variables, attributes and methods

Background - IPython Notebook

There is a web interface to IPython, known as the IPython notebook, start it like this

ipython notebook
# or to get all of the pylab components
ipython notebook --pylab

IPython - Follow Along

Follow along by connecting to TMPNB.ORG!

http://tmpnb.org

Background - NumPy

NumPy is the foundation for Pandas
Numerical data structures (mostly Arrays)
Operations on those.
Less structure than Pandas provides.

Background - NumPy - Arrays

import numpy as np
# np.zeros, np.ones
data0 = np.zeros((2, 4))

data0

array([[ 0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.]])

# Make an array with 20 entries 0..19
data1 = np.arange(20)
# print the first 8
data1[0:8]

array([0, 1, 2, 3, 4, 5, 6, 7])

Background - NumPy - Arrays

# make it a 4,5 array
data = np.arange(20).reshape(4, 5)
data

array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19]])

Background - NumPy - Arrays

Arrays have NumPy specific types, dtypes, and can be operated on.

print("dtype: ", data.dtype)
result = data * 20.5
print(result)

('dtype: ', dtype('int64'))
[[   0.    20.5   41.    61.5   82. ]
 [ 102.5  123.   143.5  164.   184.5]
 [ 205.   225.5  246.   266.5  287. ]
 [ 307.5  328.   348.5  369.   389.5]]

Now, on to Pandas

Pandas

Tabular, Timeseries, Matrix Data - labeled or not
Sensible handling of missing data and data alignment
Data selection, slicing and reshaping features
Robust data import utilities.
Advanced time series capabilities

Data Structures

Series - 1D labeled array
DataFrame - 2D labeled array
Panel - 3D labeled array (More D)

Assumed Imports

In my code samples, assume I import the following

import pandas as pd
import numpy as np

Series

one-dimensional labeled array
holds any data type
axis labels known as index
implicit integert indexes
dict-like

Create a Simple Series

s1 = pd.Series([1, 2, 3, 4, 5])
s1

0    1
1    2
2    3
3    4
4    5
dtype: int64

Series Operations

# integer multiplication
print(s1 * 5)

0     5
1    10
2    15
3    20
4    25
dtype: int64

Series Operations - Cont.

# float multiplication
print(s1 * 5.0)

0     5
1    10
2    15
3    20
4    25
dtype: float64

Series Index

s2 = pd.Series([1, 2, 3, 4, 5],
               index=['a', 'b', 'c', 'd', 'e'])
s2

a    1
b    2
c    3
d    4
e    5
dtype: int64

Date Convenience Functions

A quick aside …

dates = pd.date_range('20130626', periods=5)
print(dates)
print()
print(dates[0])

<class 'pandas.tseries.index.DatetimeIndex'>
[2013-06-26, ..., 2013-06-30]
Length: 5, Freq: D, Timezone: None
()
2013-06-26 00:00:00

Datestamps as Index

s3 = pd.Series([1, 2, 3, 4, 5], index=dates)
print(s3)

2013-06-26    1
2013-06-27    2
2013-06-28    3
2013-06-29    4
2013-06-30    5
Freq: D, dtype: int64

Selecting By Index

Note that the integer index is retained along with the new date index.

print(s3[0])
print(type(s3[0]))
print()
print(s3[1:3])
print(type(s3[1:3]))

1
<type 'numpy.int64'>
()
2013-06-27    2
2013-06-28    3
Freq: D, dtype: int64
<class 'pandas.core.series.Series'>

Selecting by value

s3[s3 < 3]

2013-06-26    1
2013-06-27    2
Freq: D, dtype: int64

Selecting by Label (Date)

s3['20130626':'20130628']

2013-06-26    1
2013-06-27    2
2013-06-28    3
Freq: D, dtype: int64

Series Wrapup

Things not covered but you should look into:

Other instantiation options: dict
Operator Handling of missing data NaN
Reforming Data and Indexes
Boolean Indexing
Other Series Attributes:
- index - index.name
- name - Series name

DataFrame

2-dimensional labeled data structure
Like a SQL Table, Spreadsheet or dict of Series objects.
Columns of potentially different types
Operations, slicing and other behavior just like Series

DataFrame - Simple

data1 = pd.DataFrame(np.random.rand(4, 4))
data1

	0	1	2	3
0	0.002581	0.851980	0.097265	0.648841
1	0.732965	0.820690	0.895176	0.582483
2	0.176504	0.068942	0.466759	0.918777
3	0.938426	0.097954	0.696534	0.684424

DataFrame - Index/Column Names

dates = pd.date_range('20130626', periods=4)
data2 = pd.DataFrame(
    np.random.rand(4, 4),
    index=dates, columns=list('ABCD'))
data2

	A	B	C	D
2013-06-26	0.831222	0.209279	0.340186	0.928447
2013-06-27	0.252513	0.452392	0.862822	0.738837
2013-06-28	0.309083	0.822918	0.924720	0.964607
2013-06-29	0.827998	0.539519	0.248369	0.377682

DataFrame - Operations

data2['E'] = data2['B'] + 5 * data2['C']
data2

	A	B	C	D	E
2013-06-26	0.831222	0.209279	0.340186	0.928447	1.910210
2013-06-27	0.252513	0.452392	0.862822	0.738837	4.766505
2013-06-28	0.309083	0.822918	0.924720	0.964607	5.446516
2013-06-29	0.827998	0.539519	0.248369	0.377682	1.781366

See? You never need Excel again!

DataFrame - Column Access

Deleting a column.

# Deleting a Column
del data2['E']
data2

	A	B	C	D
2013-06-26	0.831222	0.209279	0.340186	0.928447
2013-06-27	0.252513	0.452392	0.862822	0.738837
2013-06-28	0.309083	0.822918	0.924720	0.964607
2013-06-29	0.827998	0.539519	0.248369	0.377682

DataFrame

Remember this, data2, for the next examples.

data2

	A	B	C	D
2013-06-26	0.831222	0.209279	0.340186	0.928447
2013-06-27	0.252513	0.452392	0.862822	0.738837
2013-06-28	0.309083	0.822918	0.924720	0.964607
2013-06-29	0.827998	0.539519	0.248369	0.377682

DataFrame - Column Access

As a dict

data2['B']

2013-06-26    0.209279
2013-06-27    0.452392
2013-06-28    0.822918
2013-06-29    0.539519
Freq: D, Name: B, dtype: float64

DataFrame - Column Access

As an attribute

data2.B

2013-06-26    0.209279
2013-06-27    0.452392
2013-06-28    0.822918
2013-06-29    0.539519
Freq: D, Name: B, dtype: float64

DataFrame - Row Access

By row label

data2.loc['20130627']    

A    0.252513
B    0.452392
C    0.862822
D    0.738837
Name: 2013-06-27 00:00:00, dtype: float64

DataFrame - Row Access

By integer location

data2.iloc[1]

A    0.252513
B    0.452392
C    0.862822
D    0.738837
Name: 2013-06-27 00:00:00, dtype: float64

DataFrame - Cell Access

Access column, then row or use iloc and row/column indexes.

print(data2.B[0])
print(data2['B'][0])
print(data2.iloc[0,1])  # [row,column] 

0.209279059059
0.209279059059
0.209279059059

DataFrame - Taking a Peek

Look at the beginning of the DataFrame

data3 = pd.DataFrame(np.random.rand(100, 4))
data3.head()

	0	1	2	3
0	0.796264	0.332496	0.860904	0.488276
1	0.405906	0.309003	0.159129	0.597427
2	0.107366	0.791943	0.080191	0.187125
3	0.176196	0.931741	0.742967	0.953014
4	0.567175	0.673101	0.069275	0.208249

DataFrame - Taking a Peek

Look at the end of the DataFrame.

data3.tail()

	0	1	2	3
95	0.175699	0.187918	0.407732	0.441582
96	0.638801	0.264603	0.210135	0.721955
97	0.947213	0.674040	0.087639	0.240926
98	0.220907	0.309761	0.659022	0.894547
99	0.452450	0.365101	0.043229	0.911712

DataFrame Wrap Up

Just remember,

A DataFrame is just a bunch of Series grouped together.
Any one dimensional slice returns a Series
Any two dimensional slice returns another DataFrame.
Elements are typically NumPy types or Objects.

Panel

Like DataFrame but 3 or more dimensions.

IO Tools

Robust IO tools to read in data from a variety of sources

Plotting

Matplotlib - s.plot() - Standard Python Plotting Library
Trellis - rplot() - An ‘R’ inspired Matplotlib based plotting tool

Bringing it Together - Data

The csv file (phx-temps.csv) contains Phoenix weather data from GSOD::

1973-01-01 00:00:00,53.1,37.9
1973-01-02 00:00:00,57.9,37.0
...
2012-12-30 00:00:00,64.9,39.0
2012-12-31 00:00:00,55.9,41.0

Bringing it Together - Code

Simple read_csv()

# simple readcsv
phxtemps1 = pd.read_csv('phx-temps.csv')
phxtemps1.head()

	1973-01-01 00:00:00	53.1	37.9
0	1973-01-02 00:00:00	57.9	37.0
1	1973-01-03 00:00:00	59.0	37.0
2	1973-01-04 00:00:00	57.9	41.0
3	1973-01-05 00:00:00	54.0	39.9
4	1973-01-06 00:00:00	55.9	37.9

Bringing it Together - Code

Advanced read_csv(), parsing the dates and using them as the index, and naming the columns.

# define index, parse dates, name columns
phxtemps2 = pd.read_csv(
    'phx-temps.csv', index_col=0,
    names=['highs', 'lows'], parse_dates=True)
phxtemps2.head()

	highs	lows
1973-01-01	53.1	37.9
1973-01-02	57.9	37.0
1973-01-03	59.0	37.0
1973-01-04	57.9	41.0
1973-01-05	54.0	39.9

Bringing it Together - Plot

import matplotlib.pyplot as plt
%matplotlib inline 
phxtemps2.plot()  # pandas convenience method

<matplotlib.axes._subplots.AxesSubplot at 0x7fca44e8e550>

Boo, Pandas and Friends would cry if they saw such a plot.

Bringing it Together - Plot

Lets see a smaller slice of time:

phxtemps2['20120101':'20121231'].plot()

<matplotlib.axes._subplots.AxesSubplot at 0x7fca3c66f1d0>

Bringing it Together - Plot

Lets operate on the DataFrame … lets take the differnce between the highs and lows.

phxtemps2['diff'] = phxtemps2.highs - phxtemps2.lows
phxtemps2['20120101':'20121231'].plot()

<matplotlib.axes._subplots.AxesSubplot at 0x7fca3c6ba650>

Pandas Alternatives

AstroPy seems to have similar data structures.
I suspect there are others.

References

Thanks! - Pandas and Friends

Austin Godber
Mail: godber@uberhip.com
Twitter: @godber
Presented at DesertPy, Jan 2015.
Ipython Notebook

Pandas Cheatsheet

Thu, 15 Sep 2016 00:00:00 +0000

Pandas

The Pandas library is built on NumPy and provides easy-to-use data structures and data analysis tools for the Python programming language. Use the following import convention:

import pandas as pd

Pandas Data Structures

Series

A one-dimensional labeled array A capable of holding any data type

s = pd.Series([3, -5, 7, 4], index=['a', 'b', 'c', 'd'])

DataFrame

A two-dimensional labeled data structure with columns of potentially different types

data = {'Country': ['Belgium', 'India', 'Brazil'],
'Capital': ['Brussels', 'New Delhi', 'Brasília'],
'Population': [11190846, 1303171035, 207847528]}

df = pd.DataFrame(data,
columns=['Country', 'Capital', 'Population'])

I/O

Read and Write to CSV

pd.read_csv('file.csv', header=None, nrows=5)
pd.to_csv('myDataFrame.csv')

Read and Write to Excel

pd.read_excel('file.xlsx')
pd.to_excel('dir/myDataFrame.xlsx', sheet_name='Sheet1')

Read multiple sheets from the same file

xlsx = pd.ExcelFile('file.xls')
df = pd.read_excel(xlsx, 'Sheet1')

Read and Write to SQL Query or Database Table

from sqlalchemy import create_engine
engine = create_engine('sqlite:///:memory:')
pd.read_sql("SELECT * FROM my_table;", engine)
pd.read_sql_table('my_table', engine)
pd.read_sql_query("SELECT * FROM my_table;", engine)

read_sql()is a convenience wrapper around read_sql_table() and read_sql_query()

pd.to_sql('myDf', engine)

Asking For Help

help(pd.Series.loc)

Selection

Getting

Get one element

>>> s['b'] 

-5

Get subset of a DataFrame

>>> df[1:] Get subset of a DataFrame

Country Capital Population
1 India New Delhi 1303171035
2 Brazil Brasília 207847528

Selecting, Boolean Indexing & Setting

By Position

Select single value by row & column

>>> df.iloc([0],[0])
'Belgium'

>>> df.iat([0],[0])
'Belgium'

By Label

Select single value by row & column labels

>>> df.loc([0], ['Country']) 
'Belgium' 
>>> df.at([0], ['Country']) 
'Belgium'

By Label/Position

Select single row of subset of rows

>>> df.ix[2]
Country Brazil 
Capital Brasília 
Population 207847528

Select a single column of subset of columns

>>> df.ix[:,'Capital']
Brussels 
New Delhi
Brasília 

Select rows and columns

>>> df.ix[1,'Capital'] 
'New Delhi' 

Boolean Indexing

Series s where value is not >1

>>> s[~(s > 1)]

s where value is <-1 or >2

>>> s[(s < -1) | (s > 2)]

Use filter to adjust DataFrame

>>> df[df['Population']>1200000000]

Setting

Set index a of Series s to 6

>>> s['a'] = 6

Dropping

Drop values from rows (axis=0)

>>> s.drop(['a', 'c'])

Drop values from columns(axis=1)

>>> df.drop('Country', axis=1)

Sort & Rank

Sort by row or column index

>>> df.sort_index(by='Country')

Sort a series by its values

>>> s.order()

Assign ranks to entries

>>> df.rank()

Retrieving Series/DataFrame Information

Basic Information

(rows,columns)

>>> df.shape

Describe index

>>> df.index

Describe DataFrame columns

>>> df.columns

Info on DataFrame

>>> df.info()

Number of non-NA values

>>> df.count()

Summary

Sum of values

>>> df.sum()

Cummulative sum of values

>>> df.cumsum()

Minimum/maximum values

>>> df.min()/df.max()

Minimum/Maximum index value

>>> df.idmin()/df.idmax()

Summary statistics

>>> df.describe()

Mean of values

>>> df.mean()

Median of values

>>> df.median()

Applying Functions

>>> f = lambda x: x*2

Apply function

>>> df.apply(f)

Apply function element-wise

>>> df.applymap(f)

Data Alignment

Internal Data Alignment

NA values are introduced in the indices that don’t overlap:

>>> s3 = pd.Series([7, -2, 3], index=['a', 'c', 'd'])
>>> s + s3
 a 10.0
 b NaN
 c 5.0
 d 7.0

Arithmetic Operations with Fill Methods

You can also do the internal data alignment yourself with the help of the fill methods:

>>> s.add(s3, fill_value=0)
 a 10.0
 b -5.0
 c 5.0
 d 7.0
>>> s.sub(s3, fill_value=2)
>>> s.div(s3, fill_value=4)
>>> s.mul(s3, fill_value=3)

Pandas cheat sheet - Summary

I hope this Pandas cheat sheet has been helpful. The Source for this cheat sheet is available here.

Selenium Cheatsheet

Mon, 22 Aug 2016 00:00:00 +0000

Basic

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Firefox()
driver.get("http://www.python.org")
assert "Python" in driver.title
elem = driver.find_element_by_name("q")
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
driver.back()
driver.close()

Element

# element html
element.get_attribute('innerHTML')
element.get_attribute('outerHTML')

# element text
element.text

# page html
page.source

Forms

# fill text element
element.send_keys("some text")
element.send_keys(" and some", Keys.ARROW_DOWN)
element.clear()

# select form
# http://selenium-python.readthedocs.org/navigating.html

# submit form
# Assume the button has the ID "submit" :)
driver.find_element_by_id("submit").click()
element.submit()

Execute Js

# run js
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

Selector

# http://selenium-python.readthedocs.org/locating-elements.html

# id
login_form = driver.find_element_by_id('loginForm')

# name
username = driver.find_element_by_name('username')

# xpath
username = driver.find_element_by_xpath("//form[input/@name='username']")

# css
content = driver.find_element_by_css_selector('p.content')

# link text
link1 = driver.find_element_by_link_text('Continue')
link2 = driver.find_element_by_partial_link_text('Conti')

Wait

# http://selenium-python.readthedocs.org/waits.html

###########################
#explicit wait
############################
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.ID,'someid')))

############################
# implicit wait
##########################
from selenium import webdriver
driver = webdriver.Firefox()
driver.implicitly_wait(10) # seconds
driver.get("http://somedomain/url_that_delays_loading")
myDynamicElement = driver.find_element_by_id("myDynamicElement")

The link to these cheatsheet can be found here.

Beautiful Soup 4 Cheatsheet

Fri, 15 Jul 2016 00:00:00 +0000

Basic

# https://www.crummy.com/software/BeautifulSoup/bs4/doc/
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc, 'html.parser')


soup.title # <title>The Dormouse's story</title>
soup.title.name # u'title'
soup.title.string # u'The Dormouse's story'
soup.title.parent.name # u'head'

#various finder
css_soup.select("p.strikeout.body") # css finder
soup.p # <p class="title"><b>The Dormouse's story</b></p>
soup.p['class'] # u'title'
soup.a # <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>
soup.find_all('a') # [<a ..>, ..]
soup.find(id="link3") # <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
for link in soup.find_all('a'):
    print(link.get('href')) # http://example.com/elsi, # http://example.com/lacie

Make soup

from bs4 import BeautifulSoup

soup = BeautifulSoup(open("index.html"))
soup = BeautifulSoup("<html>data</html>")

Output

# HTML
soup.prettify() #pretty print
str(soup) # non-pretty print

# String
soup.get_text() #all text under the element

Search

search.pyhttps://www.crummy.com/software/BeautifulSoup/bs4/doc/
#-------------------------
# css selector
#-------------------------
css_soup.select("p.strikeout.body")
soup.select("p nth-of-type(3)") # 3rd child
soup.select("head > title")
soup.select("p > a:nth-of-type(2)")
soup.select("p > #link1") # direct child
soup.select("#link1 ~ .sister")  # sibling
soup.select('a[href]') # existence of an attribute
soup.select_one(".sister")

# attribute value
soup.select('a[href="http://example.com/elsie"]') # exact attribute
soup.select('a[href^="http://example.com/"]') # negative match
soup.select('a[href$="tillie"]') # end match
soup.select('a[href*=".com/el"]') # middle match


#-------------------------
# basic
#-------------------------
soup.find_all('b') # match by tag
soup.find_all(re.compile("^b")) # match by tag using regex
soup.find_all(["a", "b"]) # match by tag in list

# function (complex condition)
def has_class_but_no_id(tag):
  return tag.has_attr('class') and not tag.has_attr('id')
soup.find_all(has_class_but_no_id)


#-------------------------
# find_all_api
#-------------------------
find_all(name, attrs, recursive, string, limit, **kwargs)

soup.find_all("title") # tag condition
soup.find_all("p", "title") # tag and attr
# [<p class="title"><b>The Dormouse's story</b></p>]
soup.find_all("a")

# keyword arguments
soup.find_all(id="link2")
soup.find_all(href=re.compile("elsie"), id='link1')
soup.find(string=re.compile("sisters")) # text contain sisters

# css class (class is researved keyword)
soup.find_all("a", class_="sister")

#-----------------------------
# going up/down/side
#-----------------------------
# ----- going down -----
soup.head# <head><title>The Dormouse's story</title></head>
soup.title# <title>The Dormouse's story</title>
soup.body.b # <b>The Dormouse's story</b>
soup.a # <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>
soup.find_all('a')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http:

# children = contents
head_tag.contents # [<title>The Dormouse's story</title>]
head_tag.children # [<title>The Dormouse's story</title>]

# descendants (all of a tag’s children, recursively)
for child in head_tag.descendants:
  print(child)

# .string is tricky
head_tag.contents # [<title>The Dormouse's story</title>]
head_tag.string # u'The Dormouse's story' (because head tag has only one child)
print(soup.html.string) # None (because html has many children)

# whitespace removed strings
for string in soup.stripped_strings:
  print(repr(string))


# ----- going up -----
title_tag.parent # <head><title>The Dormouse's story</title></head>
# going up recursively
link.parents # [ p, body, html, [document], None]


# ----- sideway -----
# sibling = include text node
sibling_soup.b.next_sibling
sibling_soup.c.previous_sibling

# multiple
sibling_soup.b.next_siblings
sibling_soup.c.previous_siblings

# element = not include text node
sibling_soup.b.next_element
sibling_soup.c.previous_element
sibling_soup.b.next_elements
sibling_soup.c.previous_elements

Edit

#----------------------------
# change exisitng tag
#----------------------------
tag.name = "blockquote" # modify tag name
tag['class'] = 'verybold' # modify tag attribute
del tag['class'] # delete attribute
tag.string= 'not too bold' # modify tag contents string
tag.append(" but bolder than usual") # append tag contents

#----------------------------
# insert tag
#----------------------------
new_tag = soup.new_tag("a", href="http://www.example.com")
original_tag.append(new_tag) # create child
new_tag.string = "Link text." # can edit element after creating child

soup.b.string.insert_before(tag)
soup.b.i.insert_after(soup.new_string(" ever "))

#----------------------------
# delete tag
#----------------------------
soup.i.clear() # removes the contents
i_tag = soup.i.extract() # completely removes a tag from tree and returns the element
soup.i.decompose() # completely removes a tag from tree and discard the tag

#----------------------------
# replace/wrap/unwrap tag
#----------------------------
a_tag.i.replace_with(soup.new_tag("b"))
a_tag.i.replace_with(Beautifulsoup("<b>bold element</b>")) # replace inner html
soup.p.string.wrap(soup.new_tag("b"))
a_tag.i.unwrap()

Encoding

#output
soup.prettify("latin-1")
tag.encode("utf-8")
tag.encode("latin-1")
tag.encode("ascii")

Parse only part

# The SoupStrainer class allows you to choose which parts of an
# incoming document are parsed
from bs4 import SoupStrainer

# conditions
only_a_tags = SoupStrainer("a")
only_tags_with_id_link2 = SoupStrainer(id="link2")

def is_short_string(string):
  return len(string) < 10
only_short_strings = SoupStrainer(string=is_short_string)

# execute parse
BeautifulSoup(html_doc, "html.parser", parse_only=only_a_tags)
BeautifulSoup(html_doc, "html.parser", parse_only=only_tags_with_id_link2)
BeautifulSoup(html_doc, "html.parser", parse_only=only_short_strings)

The link to these cheatsheet can be found here.