Zato MySQL Connection with UTF-8

Today this error appeared in an Zato Service:

UnicodeEncodeError: 'latin-1' codec can't encode character u'\u2019' in position 1: ordinal not in range(256)

I turns out that SQLAlchemy defaults to latin-1 encoding instead of UTF-8 for MySQL connections.

Thanks to Andrea on a Zato Thread here: https://mailman-mail5.webfaction.com/pipermail/zato-discuss/2015-November/001443.html points out that we can just add
dbname?charset=utf8 in the WebUI.

While not intuitive, it does work.
I added it, had the service try the action again and all was well in the world again.

SuiteCRM / SugarCRM Rest API 500 Error get_entry_list

I stumbled upon this problem today when trying out the SuiteCRM Rest API.
When searching for an account by it’s name I was using a search like this:

"name = 'Andrew'"

The API was returning a 500 error.
I dug into the suitecrm.log and found this:

where (name = 'Andrew') AND accounts.deleted=0: MySQL error 1052: Column 'name' in where clause is ambiguous

It seems that because of the joins, it can’t work out which ‘name’ we are talking about. Lets add the table name in and tada it works:

"accounts.name = 'Andrew'"

Zato and Pandas (strange combination I know)

I’ve been working with Zato for a project and I’m getting some random text out of a HTTP request that looks like a CSV file, so the easiest way I know how to deal with this CSV data is with Pandas, then I could process it enough and clean it up to hand back to Zato.

But Zato doesn’t ship with Pandas….
(TL;DR at the bottom of the post)

In theory this should work:

/opt/zato/2.0.7/bin/pip install pandas

But then I get the error when I do the following:

/opt/zato/2.0.7/bin/python
>>> import pandas as pd
File "/opt/zato/current/local/lib/python2.7/site-packages/pandas/__init__.py", line 19, in
"Missing required dependencies {0}".format(missing_dependencies))
ImportError: Missing required dependencies ['pytz', 'dateutil']

No luck.

After trying pip install pytz and pip install –upgrade pytz
I get this error:

Installing collected packages: pytz
Found existing installation: pytz 2014.4
Cannot remove entries from nonexistent file /opt/zato/2.0.7/eggs/easy-install.pth

But this does work:

/opt/zato/2.0.7/bin/easy_install --upgrade pytz

Now to fix dateutil.

easy_install --upgrade python-dateutil

Let’s try it again:

/opt/zato/2.0.7/bin/python
>>> import pandas as pd
Traceback (most recent call last):
File "", line 1, in
File "/opt/zato/current/local/lib/python2.7/site-packages/pandas/__init__.py", line 23, in
from pandas.compat.numpy import *
File "/opt/zato/current/local/lib/python2.7/site-packages/pandas/compat/__init__.py", line 361, in
from dateutil import parser as _date_parser
File "/opt/zato/current/local/lib/python2.7/site-packages/python_dateutil-2.6.0-py2.7.egg/dateutil/parser.py", line 40, in
from six import text_type, binary_type, integer_types
ImportError: No module named six

Bummer!


easy_install six

And for a final test:

/opt/zato/2.0.7/bin/python
>>> import pandas as pd
>>>

Yay, it’s working. Hopefully, nothing in Zato has broken…

TL;DR:


/opt/zato/2.0.7/bin/easy_install pandas
/opt/zato/2.0.7/bin/easy_install --upgrade pytz
/opt/zato/2.0.7/bin/easy_install --upgrade pytz
/opt/zato/2.0.7/bin/easy_install six

UPDATE:
(I have not tested this)
It might work better to use buildout to make this work see this comment here:

https://forum.zato.io/t/what-is-the-canonical-way-to-add-extra-python-packages-to-my-zato-server/580?source_topic_id=586

There is another way, that of updating versions.cfg and buildout.cfg files but if you are more familiar with pip than buildout then pip install package-name is perfectly fine.

However, if a dependency is not on PyPI nor any other package index (such as your own internal one) then you can place individual Python modules in zato_extra_paths too.

Yii CGridView Default Order / Yii ActiveRecord Default Sort

Disclaimer, I know next to nothing about Yii….

Today I had an issue with a Yii App and it’s default admin CRUD tables, and sorting fields, well everything that listed a record from the database was in the order it was inserted into the database and not in a human friendly alphabetical order.

It turns out it’s really easy to set a default so that everything that uses the Model, comes out in a sorted way. (Note for larger things, you probably don’t want every single query that hits the DB getting sorted, so you’d probably only apply the sorting as required, but for small things, this shouldn’t be an issue).

For this example I have a Model called ‘Contacts’.

Open up your model (mine is: /models/Contacts.php)

Above the function ‘ public static function model($className=__CLASS__)’

Add the following function (if it’s not present)


/*
* Let's Setup some defaults!
*/
public function defaultScope()
{
return array(
'order' => 'FirstName'
);
}

In the array of ‘order’ change the value to the name of the column that you want to sort by, for my example it was ‘FirstName’.

Save, upload, test, enjoy.

Like I said at the start, I know nothing about Yii, had I, I probably wouldn’t have spent 20 minutes googling it.

Resources: http://www.yiiframework.com/doc/guide/1.1/en/database.ar#default-scope

LG IPECs Tips and Tricks

Things I’ve had to pick up from working with an LG IPECs phone system.

1) Default Phone Password is: 147* (for when prompted when you first take them out of the box).

2) Setting up an additional phone.
– Make sure that DIP switch 3 is set to ‘on’ to allow additional handset registration.
– On another phone Go to ‘Trans/PGM’ -> [7] Supplementary -> [8] Network Config to see how the system in setup (handy for finding VLANs and such).
– Plug in the new phone and set the settings similar to the current one, just using a different IP address.

2) Voicemail.
– Voicemail is setup as a Hunt Group, the station has to call into that Hunt group and provide a Password, the password is the extension number, followed by the code found in the Authorization Code Table (227).
If the code was 111, the password for Extension 246 would be 246111

– Voicemail can be sent to email. Here is a document outlining how to do it: http://www.ariatech.com.au/uploaded/File/iPECS/QHG-KB/Q522_VSF_to_Email.pdf

3) System Speed Dial / Public Directory.
– Nobody likes bashing in a bunch of contacts on the desk phone. They can instead be updated via the ‘System Speed Dial’ screen in the Web management tool, found under ‘System Data’ -> ‘System Speed Dial’

4) Changing Extensions around.
Update the extensions in Flexible Station Number(PGM 105)
And don’t forget the Flexible DID table (Flexible DID Conversion(231))

5) Performing a factory reset. Flick DIP4 to ON (Same as DIP3 above), remove power, return power. Manually configure network IP to 10.10.10.x / 255.255.255.0 (where X is any number between 3 and 254) and Open up a web browser and go to https://10.10.10.2
Default password is empty. From here, upload a database backup, connect MFIM back into Network and remember to flick DIP4 to Off, otherwise the next time it reboots it will forget the database.

6) Checking a handset firmware:
Trans/PGM -> 79 (all the Phone PGM codes can be found here: https://www.nonstoptelecom.com/site/download/LIP-8012E-8024E.pdf)

7) Adding a DND Flex button for a Hunt Group. It’s possible to do this from the handset, but I needed to do it from the webadmin.
In Flexbutton UI, set the Type to ‘Programming (Numbering Plan) then the Value of 571XYZ where XYZ is the Hunt group. Upon saving the Value should change to ‘HUNT DUTY(XYZ)’ (again XYZ is the Hunt group).

Windows 10 and iTunes There was an error connecting to the Apple ID server

Recently after a fresh install of Windows 10, I tired to reinstall iTunes.

While trying to authorize my computer for my iPhone Apps I got the following error:

“There was an error connecting to the Apple ID server”

Several reboots later, still no joy. I also noticed that I couldn’t connect to the iTunes store.

I found the following helps?

  1. Right click on the Windows Start Menu Icon and select ‘Command Prompt (Admin)’
  2. Type the following into the command prompt box followed by enter:
    netsh winsock reset
  3. Reboot the computer

How this works: I’m not overly sure why it effects iTunes, but it’s similar to clearing the cache in your browser, just at a network level.

Hopefully this helps someone else.

Reference: https://discussions.apple.com/thread/7192420?start=0&tstart=0

LG-Nortel LDK-300 Phone not allowed to call external numbers

I had a strange phone extension that after lifting the handset, pressing 0 to the get the outside line and dialing the 2nd digit a message on the screen would appear ‘ACCESS DENY’.

It turns out it was a COS issue.

To fix it:

  • Lift the handset and then press ‘TRANS/PGM’
  • Then select option ‘[2] COS / SMS SERVICE’
  • Then select option ‘[2] COS RESTORE’
  • I had ‘ENTER CURRENT PASSWORD’ appear on the screen. How to find that is listed below
  • Enter the password for the station and press the ‘HOLD / SAVE’ button.
  • ‘PROGAM’ Appears on the screen, hang up

To piss off the next person:

  • Lift the handset and then press ‘TRANS/PGM’
  • Then select option ‘[2] COS / SMS SERVICE’
  • Then select option ‘[1] COS DOWN’
  • ‘PROGAM’ Appears on the screen, hang up

How to find out the station password:

In LDK PCADMIN, the ‘Authorization Code Table (PGM227)’ seems to hold all the station passwords. It has 4 columns, Index, Value (this is the password), Day COS, Night COS.
The Index number, appears to line up with the Port number found in the Numbering Plan (PGM104-109).
By comparing the Station number and it’s port number in the Numbering Plan, and using that port number to match up with the index number in Authorization Code Table, I was able to work out the password (sounds complex, but easier when you attempt it).

This system was running the ‘Hotel’ software for the LG-Nortel LDK-300, I would assume that it would be the same in the LDK-130 (if it has a hotel software). I don’t think that the COS features are part of the standard non ‘Hotel’ software. I’m not sure.

Selenium / Python Notes June 2016

For the past 24 hours I’ve been jumping into some Selenium fun in Python.

Notes:

  • Firefox 47 doesn’t work, however Firefox 47.0.1 does.
Selenium Setup


from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.proxy import *
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

def init_driver():
driver = webdriver.Firefox()
driver.wait = WebDriverWait(driver, 5)
return driver

def lookup(driver):
driver.get(“http://www.google.com”)

if __name__ == “__main__”:
driver = init_driver()
lookup(driver)
time.sleep(5)
driver.quit()

Two ways to interact with an element on the page:


try:
textbox = driver.wait.until(EC.presence_of_element_located((By.ID, "formTxtBox")))
except TimeoutException:
print("Login Form Not Found!")

or

textbox = driver.find_element_by_id("formTxtBox")

Taking HTML and feeding it into BeautifulSoup


div = driver.find_element_by_id("id_of_div_here")
soup = BeautifulSoup(propertyDropDown.get_attribute('innerHTML'), 'html.parser')
#Do something with the soup....

Creating a list of dates (requires pandas)


import pandas as pd
dates = []
index = pd.date_range('2015-7-1', periods=52, freq='W-WED')
for i in index:
dates.append(i.to_datetime())
return dates

Convert Table into CSV with BeauitfulSoup

(there are probably 100 better ways to do this…)

#Get the element of the table using CSS Selector
table = driver.wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "#table > tbody")))
#Take our table and pass it into BeautifulSoup so we can easily transverse it.
soup = BeautifulSoup(table.get_attribute('innerHTML'), 'html.parser')
#Make an empty list
rows = []
#Loop over the soup
for row in soup.find_all('tr'):
rows.append([val.text.strip('\n').encode('utf8').strip('\xc2\xa0') for val in row.find_all('td')])
 
#Save out to CSV.
with open('output_file.csv', 'ab') as f:
writer = csv.writer(f)
writer.writerows(row for row in rows if row)

Disclaimer: I’ve been programming in Python for about 5 minutes and using Selenium for about 2….while this code worked for my project, it’s most likely littered with errors and bad practices. This is mostly just a reference for myself for a project I might need to reefer back to this stuff in 6-12 months time. I am definitely not the person to ask for help from when it comes to this stuff.

Scripting a reboot of a Billion Router over Telnet

Sometimes with older hardware and ADSL connections, you just need to do a reboot every now and then. Newer modems let you do this, but the Billion one I had the joy of playing with today, didn’t.

Lets turn to a simple script running on a Linux box on the same subnet.
Turns out using ‘expect’ you can script a Telnet session. (I know, Telnet isn’t ideal). This could be used across switches, routers, modems, other random boxes that require a telnet interface.

Create a File

nano modem-reboot.sh

Add the code. You’ll need to change the “expect” values as necessary

#!/usr/bin/expect
#If it all goes pear shaped the script will timeout after 20 seconds.
set timeout 20
#First argument is assigned to the variable name
set name [lindex $argv 0]
#Second argument is assigned to the variable user
set user [lindex $argv 1]
#Third argument is assigned to the variable password
set password [lindex $argv 2]
#This spawns the telnet program and connects it to the variable name
spawn telnet $name
#The script expects login
expect "Login:"
#The script sends the user variable
send "$user\r"
#The script expects Password
expect "Password:"
#The script sends the password variable
send "$password\r"
#This hands control of the keyboard over two you (Nice expect feature!)
#interact
#Reboot
expect "admin>"
# Change the following value to whatever your devices reboot command is.
send "system restart\r"

Finally make the file executable


chmod +x modem-reboot.sh

Test it.

Help it doesn’t work.
Under a stock standard Ubuntu 14.04 install I had to add the ‘expect’ package

sudo apt-get install expect

Thanks to http://stackoverflow.com/questions/7013137/automating-telnet-session-using-bash-scripts for the assistance and starting point of the script.