Tag Archives: python

Zato and Pandas (strange combination I know)

I’ve been working with Zato for a project and I’m getting some random text out of a HTTP request that looks like a CSV file, so the easiest way I know how to deal with this CSV data is with Pandas, then I could process it enough and clean it up to hand back to Zato.

But Zato doesn’t ship with Pandas….
(TL;DR at the bottom of the post)

In theory this should work:

/opt/zato/2.0.7/bin/pip install pandas

But then I get the error when I do the following:

/opt/zato/2.0.7/bin/python
>>> import pandas as pd
File "/opt/zato/current/local/lib/python2.7/site-packages/pandas/__init__.py", line 19, in
"Missing required dependencies {0}".format(missing_dependencies))
ImportError: Missing required dependencies ['pytz', 'dateutil']

No luck.

After trying pip install pytz and pip install –upgrade pytz
I get this error:

Installing collected packages: pytz
Found existing installation: pytz 2014.4
Cannot remove entries from nonexistent file /opt/zato/2.0.7/eggs/easy-install.pth

But this does work:

/opt/zato/2.0.7/bin/easy_install --upgrade pytz

Now to fix dateutil.

easy_install --upgrade python-dateutil

Let’s try it again:

/opt/zato/2.0.7/bin/python
>>> import pandas as pd
Traceback (most recent call last):
File "", line 1, in
File "/opt/zato/current/local/lib/python2.7/site-packages/pandas/__init__.py", line 23, in
from pandas.compat.numpy import *
File "/opt/zato/current/local/lib/python2.7/site-packages/pandas/compat/__init__.py", line 361, in
from dateutil import parser as _date_parser
File "/opt/zato/current/local/lib/python2.7/site-packages/python_dateutil-2.6.0-py2.7.egg/dateutil/parser.py", line 40, in
from six import text_type, binary_type, integer_types
ImportError: No module named six

Bummer!


easy_install six

And for a final test:

/opt/zato/2.0.7/bin/python
>>> import pandas as pd
>>>

Yay, it’s working. Hopefully, nothing in Zato has broken…

TL;DR:


/opt/zato/2.0.7/bin/easy_install pandas
/opt/zato/2.0.7/bin/easy_install --upgrade pytz
/opt/zato/2.0.7/bin/easy_install --upgrade pytz
/opt/zato/2.0.7/bin/easy_install six

Selenium / Python Notes June 2016

For the past 24 hours I’ve been jumping into some Selenium fun in Python.

Notes:

  • Firefox 47 doesn’t work, however Firefox 47.0.1 does.
Selenium Setup


from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.proxy import *
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

def init_driver():
driver = webdriver.Firefox()
driver.wait = WebDriverWait(driver, 5)
return driver

def lookup(driver):
driver.get(“http://www.google.com”)

if __name__ == “__main__”:
driver = init_driver()
lookup(driver)
time.sleep(5)
driver.quit()

Two ways to interact with an element on the page:


try:
textbox = driver.wait.until(EC.presence_of_element_located((By.ID, "formTxtBox")))
except TimeoutException:
print("Login Form Not Found!")

or

textbox = driver.find_element_by_id("formTxtBox")

Taking HTML and feeding it into BeautifulSoup


div = driver.find_element_by_id("id_of_div_here")
soup = BeautifulSoup(propertyDropDown.get_attribute('innerHTML'), 'html.parser')
#Do something with the soup....

Creating a list of dates (requires pandas)


import pandas as pd
dates = []
index = pd.date_range('2015-7-1', periods=52, freq='W-WED')
for i in index:
dates.append(i.to_datetime())
return dates

Convert Table into CSV with BeauitfulSoup

(there are probably 100 better ways to do this…)

#Get the element of the table using CSS Selector
table = driver.wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "#table > tbody")))
#Take our table and pass it into BeautifulSoup so we can easily transverse it.
soup = BeautifulSoup(table.get_attribute('innerHTML'), 'html.parser')
#Make an empty list
rows = []
#Loop over the soup
for row in soup.find_all('tr'):
rows.append([val.text.strip('\n').encode('utf8').strip('\xc2\xa0') for val in row.find_all('td')])
 
#Save out to CSV.
with open('output_file.csv', 'ab') as f:
writer = csv.writer(f)
writer.writerows(row for row in rows if row)

Disclaimer: I’ve been programming in Python for about 5 minutes and using Selenium for about 2….while this code worked for my project, it’s most likely littered with errors and bad practices. This is mostly just a reference for myself for a project I might need to reefer back to this stuff in 6-12 months time. I am definitely not the person to ask for help from when it comes to this stuff.

Xero API pyXero and Anaconda Python

Personal TL;DR notes on getting this to work, not a tutorial.

Download Install Anaconda
– At the time of writing it didn’t create Windows Shortcuts, fix this with openning cmd as admin

‘c:\Anaconda2\scripts\conda.exe install menuinst’

Agree to Updates

‘c:\Anaconda2\scripts\conda.exe install -f console_shortcut ipython ipython-notebook ipython-qtconsole launcher spyder’

Agree to updates.

Now there should be shortcuts in:

C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Anaconda2 (64-bit)

Now onto Xero

  • git clone https://github.com/freakboy3742/pyxero.git
  • Open up Anaconda Prompt (Anaconda2x64)
  • cd into folder with git of pyxero
  • use command: python setup.py install
  • pip uninstall oauthlib
  • pip install oauthlib==0.7.2

The last two commands fix the error: ImportError: No module named jwt.algorithms

Again these are personal notes, not a full blown tutorial.

 

Building CSS sprites using Python Glue

css-sprite was giving me some issues so I went looking for a different way to tackle the problem.

While a pain in the rear end to install, I found Glue. Given I already had all the python stuff installed it was pretty easy for me to get it working for this project.

  1. Install Glue http://glue.readthedocs.org/en/latest/installation.html
  2. Run Glue, here is the command that I like:
    glue src glue –scss –sprite-namespace= –namespace= –cachebuster

(The above command, looks for all the images in the src folder, and then outputs into the glue folder).

While the scss support is woeful, the retina side of things seems to be working better. (Although I’m not using the retina switch in the command above).