All posts by Andrew Bennett

Sorting 404s in Nginx access Logs

The default Nginx logs can be combined and sorted to gather some useful stats.

  1. Make a working directory and copy current logs over (optional):
    mkdir ~/accesslogs
    sudo cp /var/log/nginx/access.log* ~/accesslogs
    cd ~/accesslogs
  2. Extract logs compressed by log rotate:
    ungzip *.gz
  3. Make a big combined log:
    sudo cat access.log* >> log.combined
  4. Parse the combined log and count the 404’s
    sudo awk '($9 ~ /404/)' log.combined | awk '{print $7}' | sort | uniq -c | sort -rn >> 404sC.txt

Let’s encrypt certbot – moving vps

Sometimes you need to move VPS’s, and you don’t want downtime with the SSL certs.

Well it turns out that you can easily run Let’s Encrypt in standalone mode and use DNS verification to do a once-off cert prior to moving the VPS. – Read the Certbot docs.

But once the migration has taken place, you may also want to return to the validation method that you had been using previously. Two options exist, either a) modify the config files by hand (strongly advised against) or to return to the certbot and get it to update the config itself.

For example here is how I generally get a SSL cert on a particular box:

certbot certonly --webroot -w=/var/www/letsencrypt -d [www.URL.com] -d [URL.com]

And here is how I update to the new config

sudo certbot certonly --cert-name [www.URL.com] --force-renewal -a webroot -w=/var/www/letsencrypt

Thanks to @schoen on this post: https://community.letsencrypt.org/t/how-to-change-certbot-verification-method/56735/3

Installing a Canon Image Runner Advance Printer on a Mac OS

For some reason installing the UFR II/UFRII based printer on Mac OS is strangely complex.

Here’s what to do:
1) Download the correct driver for the Canon Machine / Mac OS from the canon site (I install C2020’s a lot: https://www.canon.com.au/multifunction-devices/imagerunner-advance-c2020/support)
2) Install the Package
3) Within System Preferences > Printers > Add a New Printer.
4) Select ‘IP’ from the top tab. Fill the form in as follows:
Address: IP Address of the Printer
Protocol: Change to Line Printer Daemon
Queue: Insert ‘LP’ < this is the part I always forget. Name: A logical name for the user Location: A logical location for the user. Use: Select the Driver from the one that you just installed, in my case I selected 'Canon iR-ADV C2020/2030 Click Add. Canon Printer Mac OS X Settings

It now should now be setup to print.

If you want to setup the default settings on the printer, you need to use the CUPS web interface.
Visit: http://localhost:631/printers/ on the machine.
You’ll probably get a message about CUPS Web interface not being enabled, follow the prompts to open up ‘terminal’ and paste in
cupsctl WebInterface=yes

Then you’ll be able to access the printer UI.
From here you can select the printer, and change the default printer settings.

I always change from 2 sided printing to 1 sided, and from Color to B/W

There you go, installing a Canon Printer or Canon Photocopier on Mac OS X.

Zato: Add MSSQL to Outgoing SQL Options

To Connect with MSSQL or SQL Server via Zato 3:

#sudo su - zato
$/opt/zato/current/bin/pip install pymssql
(More info: http://pymssql.org/en/stable/intro.html)
$nano /opt/zato/env/qs-1/server1/config/repo/sql.conf

Add the following lines at the end:
[mssql+pymssql]
display_name=MSSQL
sqlalchemy_driver=mssql+pymssql
ping_query=SELECT 1

???I don’t know if this is required or not, but I still do it.
Deploy to other servers in the cluster:
$cp /opt/zato/env/qs-1/server1/config/repo/sql.conf /opt/zato/env/qs-1/server2/config/repo/

Restart your cluster and MSSQL should appear as an option under SQL Outgoing connections.

LG iPECS Least Cost Routing (LCR)

Recently I had a bunch of trouble with some SIP trunks (read about it at: LG IPECS SIP Trunk Issue) and needed to edit the Least Cost Routing (or Least Call Cost Routing).

The system is pretty basic, landline calls go out of one CO group, calls to mobiles go to another CO group.
There are a couple of tables that control how LCR works: LCR LDT (PGM 221) and LCR DMT (PGM 222).

LCR LDT – Leading Digit Table – PGM 221
This is where the ‘patterns’ are setup, so that the phone system knows what to do with the call, based on the starting digit(s).
Check out this screenshot for a full view:

These are setup under Index 1,
Then in subindex 0-6 are setup for landlines, 7 is setup for the mobile calls.
For each of the Subindex 0-6, it looks at DMT 01
For the Subindex 07 (Mobile calls), it looks at DMT 02

LCR DMT – Digit Modification Table – PGM 222
[To be honest, I have no idea what is actually happening here, except for the reference to the CO/IP Group].
Index 1 routes via CO/IP Group 1
Index 2 routes via CO/IP Group 2
– Index 2 also has an Alternative DMT Index (3) – This is used when all the lines in CO/IP Group 2 are busy
Index 3
– Routes via CO/IP Group 1.

Check out this screen for a full view:

Seems to work alright.

LG IPECS SIP Trunk Issue when WAN IP Address Changes (How to start to troubleshoot VoIP Issues with Wireshark)

Recently I had an issue with a couple of SIP trunks configured on an LG IPECs system when the WAN IP Address Changes.
TL;DR Version – I was getting a SIP2.0 502 Bad Gateway Error, turns out that the SIP provider (sip.iboss.com.au) had a setting that restricted the IP address that was being used to initiate the connection.

Basic Setup:
Router -> Switch <- IPECS The switch is a Cisco POE managed switch, with Voice Traffic on a VLAN. All the IPECS phones connect to the switch. The IPECS unit and the Router also connect into the switch. SIP trunks were setup on 2x of the CO lines. I was swapping the router from ADSL to NBN (vDSL). How to begin to troubleshoot the VoIP Connection issue. When I switched the router over (NBN), and tried dialling out on the phone they would say something like "Normal Call Clearing", and not ring. Yet, when I switched back to the original router (ADSL), it would work as normal. A lot of Googling suggested using Wireshark to inspect. How to start to troubleshoot VoIP Issues with Wireshark: Step 1) Install Wireshark on a machine Step 2) On one of the Switch ports, enable 'Mirror Mode'. On the Cisco web UI this was under Admin > Diagnostics > Port Mirroring. – Note: because VLANS were in use, I needed to mirror a VLAN, but first I had to remove the port from the VLAN, then choose that VLAN in the Port Mirroring Setup.
Step 3) Fire up Wireshark, choose the LAN connection that is connected to the Mirrored port and start the capture.

It’s always nice to see what something looks like when it ‘works’, so I first setup the ADSL router, hooked up wireshark and started the capture, I tried calling my mobile and answered it, then hung up, then stopped the capture.
Then in Wireshark you can click on ‘Telephony’ in the top menu and select SIP and will show you the connection, then click on ‘Prepare Filter’ and it will limit the main screen down to the packets that were involved in the phone call. Pretty Cool.
Closing the Telephony screen, if you right click on the first packet in the list (The Info column should say something like “Request: INVITE sip…….”), go down to Follow, then UDP Stream and a somewhat human-readable stream appears. I copied that screen out into Text File (notepad, notepad++, etc) for use later.

I then swapped over the router, did the same process as above and copied the new flow of data into another Text File.
I then compared the two files side by side using WinMerge.

The first part, looks ok. It’s natural that the IDs would change between calls.

Scroll down a bit and o dear, here is where we hit a problem.

From there it a was a case of trying various configurations. I swapped around routers (SIP ALG causes all sorts of issues with SIP lines), I tired 4G connections, finally I rang the VoIP provider – who last week told me that there wasn’t any IP address security restrictions – I was informed (I think by asking the right question and being blunt – “Are the VoIP Trunks tied to an IP Address?”) that yes they could change the configuration for me.

Side note: To make things a little easier in the testing, I copied the config from the iPECS into a Free Open Source Softphone – MicroSIP. I also had a secondary SIP account to test with a completely different provider, so my test process went something like this:
Bring up WAN connection on Router:
– Bring up Working secondary SIP account – call mobile. – Call connects and works. Router is working OK.
– Bring up Primary SIP account – call mobile. – Call works / fails.
– Change something (firewall rule, WAN connection, entire router)
– Repeat process.

By doing it this way, I have a ‘known’ test (Secondary SIP Account) that I can try, before trying the ‘unknown’ (Primary SIP Account)

Secondary Side Note: The LG iPECS has a field for “Firewall IP Address”, I believe that this is better described as “WAN IP Address”. It’s changed under ‘System&Device IP’ PGM 102 / PGM 103. It appears to use this in the SIP packets. I updated this before doing my WAN testing. Interestly, even with it set to the ‘Wrong’ IP address, things still worked?

Safe to say, I’m currently hating knowing this much about VoIP packets right about now…

BA Falcon Notes

General Personal Notes Around my BA Falcon. Petrol.
(Don’t follow this if you have LPG)

Spark Plugs:
– Buy the Genuine Ford ones. (Part: K16SR14)
– Tighten to 17NM (Section 303-01a-6 of Ford Manual)

Engine Oil (According to the manual):
– Engine Oil 10W-30 ILSAC – GF2 – [WSS M2C910 2 (I6 DOHC VCT)]
– 6 cyl capacity (inc filter) – 6.5 litres (why do they sell it in 6L bottles!)
Notes on Engine Oil (http://www.fordmods.com/ford-4l-and-6-cylinder-f1/oil-once-and-for-all-t27324.html)

Other Recommendations:
– Silicone Sealant (Loctite 5900 or similar) [WSS M4G323 A7]
– Anti Seize compound (Loctite 767 or similar) [ESE FM99C100 A]

Power Steering:
– BA Nut Part Number: BA3F656B
– BA O Ring for the Rack: N 803257 S
( Source: http://www.fordmods.com/ford-4l-and-6-cylinder-f1/high-pressure-steering-hose-t104852.html )

Limesurvey Admin 500 Internal Server Error

Recently I moved a LimeSurvey instance from one server to another, unfortunately, upon doing this I was suddenly unable to login to the admin panel.
The login screen would appear, but upon trying to login, the browser just sat there with a spinning “thinking” wheel.

After sometime the following would appear on the screen:

Request Timeout
This request takes too long to process, it is timed out by the server. If it should not be timed out, please contact administrator of this web site to increase ‘Connection Timeout’.

I tried enabling Debugging mode, but nothing was logged.
After some googling, I found this thread followed by this thread

The solution
It turns out that Limesurvey is trying to ‘call’ home in the background to see if there is an update. For whatever reason, this request is failing.
To stop it from doing this, it’s possible to edit the config.php file (/application/config/config.php)
And under the section:

'config'=>array(
// debug: Set this to 1 if you are looking for errors. If you still get no errors after enabling this
// then please check your error-logs - either in your hosting provider admin panel or in some /logs directory
// on your webspace.
// LimeSurvey developers: Set this to 2 to additionally display STRICT PHP error messages and get full access to standard templates
'debug'=>0,
'debugsql'=>0, // Set this to 1 to enanble sql logging, only active when debug = 2
// Update default LimeSurvey config here
)

And add a line to the bottom there ‘updatable’ => false,:


'config'=>array(
// debug: Set this to 1 if you are looking for errors. If you still get no errors after enabling this
// then please check your error-logs - either in your hosting provider admin panel or in some /logs directory
// on your webspace.
// LimeSurvey developers: Set this to 2 to additionally display STRICT PHP error messages and get full access to standard templates
'debug'=>0,
'debugsql'=>0, // Set this to 1 to enanble sql logging, only active when debug = 2
// Update default LimeSurvey config here
'updatable' => false,
)

Success!
I use Installatron inside of Cpanel to manage updates, so I’m ok with disabling updates here.

Environment Info:
LimeSurvey: 2.72.3+171020
Cpanel with
PHP 7
Litespeed

ASP.NET Core 2 Collection of Useful Resources

For a project, I’ve had to jump into ASP.NET Core 2.
I know nothing about ASP.NET Core 2, nor C#. (My background is in Python and PHP)

These resources helped me get going to build a REST API:

Start Here:
Offical Microsfot ASP.NET Core 2.0 Tutorial

Understanding of ASP.NET 2.0 MVC Model Binding, and all the different ways it can be used:
https://tahirnaushad.com/2017/08/22/asp-net-core-2-0-mvc-model-binding/

DateTime Model Binding:
It appears that if you want to easily use a date in a Model, it needs a bit more work. With this it allows the app to parse YYYY-MM-DD (and other formats) easily.
https://gist.github.com/vickramravichandran/f06d63bfb50f27e6549978b7538d214c

Understanding ASP.NET Core 2.0 MVC Filters – How to run code before and after MVC request pipeline in ASP.NET Core.
https://tahirnaushad.com/2017/08/28/asp-net-core-2-0-mvc-filters/