Selenium / Python Notes June 2016

For the past 24 hours I’ve been jumping into some Selenium fun in Python.

Notes:

  • Firefox 47 doesn’t work, however Firefox 47.0.1 does.
Selenium Setup


from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.proxy import *
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

def init_driver():
driver = webdriver.Firefox()
driver.wait = WebDriverWait(driver, 5)
return driver

def lookup(driver):
driver.get(“http://www.google.com”)

if __name__ == “__main__”:
driver = init_driver()
lookup(driver)
time.sleep(5)
driver.quit()

Two ways to interact with an element on the page:


try:
textbox = driver.wait.until(EC.presence_of_element_located((By.ID, "formTxtBox")))
except TimeoutException:
print("Login Form Not Found!")

or

textbox = driver.find_element_by_id("formTxtBox")

Taking HTML and feeding it into BeautifulSoup


div = driver.find_element_by_id("id_of_div_here")
soup = BeautifulSoup(propertyDropDown.get_attribute('innerHTML'), 'html.parser')
#Do something with the soup....

Creating a list of dates (requires pandas)


import pandas as pd
dates = []
index = pd.date_range('2015-7-1', periods=52, freq='W-WED')
for i in index:
dates.append(i.to_datetime())
return dates

Convert Table into CSV with BeauitfulSoup

(there are probably 100 better ways to do this…)

#Get the element of the table using CSS Selector
table = driver.wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "#table > tbody")))
#Take our table and pass it into BeautifulSoup so we can easily transverse it.
soup = BeautifulSoup(table.get_attribute('innerHTML'), 'html.parser')
#Make an empty list
rows = []
#Loop over the soup
for row in soup.find_all('tr'):
rows.append([val.text.strip('\n').encode('utf8').strip('\xc2\xa0') for val in row.find_all('td')])
 
#Save out to CSV.
with open('output_file.csv', 'ab') as f:
writer = csv.writer(f)
writer.writerows(row for row in rows if row)

Disclaimer: I’ve been programming in Python for about 5 minutes and using Selenium for about 2….while this code worked for my project, it’s most likely littered with errors and bad practices. This is mostly just a reference for myself for a project I might need to reefer back to this stuff in 6-12 months time. I am definitely not the person to ask for help from when it comes to this stuff.

Scripting a reboot of a Billion Router over Telnet

Sometimes with older hardware and ADSL connections, you just need to do a reboot every now and then. Newer modems let you do this, but the Billion one I had the joy of playing with today, didn’t.

Lets turn to a simple script running on a Linux box on the same subnet.
Turns out using ‘expect’ you can script a Telnet session. (I know, Telnet isn’t ideal). This could be used across switches, routers, modems, other random boxes that require a telnet interface.

Create a File

nano modem-reboot.sh

Add the code. You’ll need to change the “expect” values as necessary

#!/usr/bin/expect
#If it all goes pear shaped the script will timeout after 20 seconds.
set timeout 20
#First argument is assigned to the variable name
set name [lindex $argv 0]
#Second argument is assigned to the variable user
set user [lindex $argv 1]
#Third argument is assigned to the variable password
set password [lindex $argv 2]
#This spawns the telnet program and connects it to the variable name
spawn telnet $name
#The script expects login
expect "Login:"
#The script sends the user variable
send "$user\r"
#The script expects Password
expect "Password:"
#The script sends the password variable
send "$password\r"
#This hands control of the keyboard over two you (Nice expect feature!)
#interact
#Reboot
expect "admin>"
# Change the following value to whatever your devices reboot command is.
send "system restart\r"

Finally make the file executable


chmod +x modem-reboot.sh

Test it.

Help it doesn’t work.
Under a stock standard Ubuntu 14.04 install I had to add the ‘expect’ package

sudo apt-get install expect

Thanks to http://stackoverflow.com/questions/7013137/automating-telnet-session-using-bash-scripts for the assistance and starting point of the script.

Check_mk: UNKNOWN – Agent exited with code 255: Pseudo-terminal will not be allocated because stdin is not a terminal.

I had this error pop up today:

UNKNOWN – Agent exited with code 255: Pseudo-terminal will not be allocated because stdin is not a terminal.

On a remote agent over SSH. Which I described here in check_mk_agent over SSH Host key verification failed

Turns out that the box in question could contact the outside world (a simple ping google.com was able to tell me that).

The problem? I had recently tighten the firewall rules on ufw

ufw allow out http
ufw allow out https
ufw allow out 53

Resolved the issue.

Thanks to TWright for the answer that pointed me in the right direction on Serverfault

LG Nortel – Phone Auto Answering Calls

Had a random issue, an LG Nortel phone was auto answering calls on speakerphone. It apparently is a ‘feature’.

On the screeen next to the phones name it had (H) instead of (T).
So it said ‘Andrew (H)’

How to fix.

  • Press. Trans/Pgm
  • Option 1 (Ring)
  • Option 2 (Answer Mode)
  • Select T(2)
  • Press Hold/Save to save it.

Thanks to this pdf: LG Quick Reference Guide 8000 Series.pdf

For pointing me in the right direction.

Set Intercom Answer Mode
This determines how your Station will ring when receiving an intercom call

  • Press TRANS/PGM button
  • Dial 1 3 and select
  • 1. for Hands free (Auto answer after a BEEP tone)
  • 2. for Tone ring
  • 3. for Privacy announce (Auto Answer but you must pick-up the handset to talk)
  • Press HOLD/SAVE button or the OK button

 

Endian DHCP stopped and failing to start on 3.0.5beta1

TL:DR Updated Endian this morning, and the DHCP service failed to start. Turns out having a 2nd DNS server set in the Web GUI doesn’t correctly write to the dhcp.conf file. Solution. Remove 2nd DNS server or manually add a comma (,) into the config file using nano.

In the Web GUI under “Status -> System Status”

The DHCP server was shown as “Stopped”.

Manually starting from the shell also failed.

‘service dhcpd start’

A quick look in the logs:

‘cat /var/log/messages’

Feb 25 10:22:51 endian dhcpd: Internet Systems Consortium DHCP Server V3.0.5-RedHat
Feb 25 10:22:51 endian dhcpd: Copyright 2004-2006 Internet Systems Consortium.
Feb 25 10:22:51 endian dhcpd: All rights reserved.
Feb 25 10:22:51 endian dhcpd: For info, please visit http://www.isc.org/sw/dhcp/
Feb 25 10:22:51 endian dhcpd: /etc/dhcpd.conf line 21: semicolon expected.
Feb 25 10:22:51 endian dhcpd:         option domain-name-servers 192.168.0.3 192.
Feb 25 10:22:51 endian dhcpd:                                                ^
Feb 25 10:22:51 endian dhcpd: Configuration file errors encountered -- exiting
Feb 25 10:22:51 endian dhcpd:
Feb 25 10:22:51 endian dhcpd: If you did not get this software from ftp.isc.org, please
Feb 25 10:22:51 endian dhcpd: get the latest from ftp.isc.org and install that before
Feb 25 10:22:51 endian dhcpd: requesting help.
Feb 25 10:22:51 endian dhcpd:
Feb 25 10:22:51 endian dhcpd: If you did get this software from ftp.isc.org and have not
Feb 25 10:22:51 endian dhcpd: yet read the README, please read it before requesting help.
Feb 25 10:22:51 endian dhcpd: If you intend to request help from the [email protected]
Feb 25 10:22:51 endian dhcpd: mailing list, please read the section on the README about
Feb 25 10:22:51 endian dhcpd: submitting bug reports and requests for help.
Feb 25 10:22:51 endian dhcpd:
Feb 25 10:22:51 endian dhcpd: Please do not under any circumstances send requests for
Feb 25 10:22:51 endian dhcpd: help directly to the authors of this software - please
Feb 25 10:22:51 endian dhcpd: send them to the appropriate mailing list as described in
Feb 25 10:22:51 endian dhcpd: the README file.
Feb 25 10:22:51 endian dhcpd:
Feb 25 10:22:51 endian dhcpd: exiting.

 

Luckily the log points us in the right direction.

/etc/dhcpd.conf line 21: semicolon expected.
Feb 25 10:22:51 endian dhcpd:         option domain-name-servers 192.168.0.3 192.

In the Web GUI if I remove the 2nd Name server, all is well, the service starts up and the log shows requests being received.
When I add it back in, it fails to start again.

When I add the comma in using nano, I’m able to manually start the service.
Although I’m sure it will break if any changes are made to the web gui.

Update: It appears that this bug has been around for a few days, and is reported: https://jira.endian.com/browse/UTM-1388

Bridged ADSL modem over VLAN Router on a Stick

TL;DR: Bridged ASDL modem over VLAN – yes it can work!

The problem:
x86 Router (Endian) has run out of physical network cards, but more WAN ports are required, can we do something with a ‘Router on a stick’ and use a VLAN aware Switch?

We can do this in a ‘double NAT’ style configuration, but can we do it in a bridged mode configuration. It seems that we can in fact. However I’m using a L3 switch (Dell 6248) I have tired in the past to use a HP PS1810-8G Switch (J9833A) without successful (granted I didn’t spend too much time on it).