f = os.popen("./cgi_script.cgi > /dev/null", "w")
f.write(postBody)
f.flush()
f.close()

As os.popen() is deprecated now (I know, it’s a very old codebase that started with Python 1.5) I’ve moved to new subprocess module:

fNull = file("/dev/null", "w")
p = subprocess.Popen("./cgi_script.cgi", shell=False, bufsize=1024, stdin = subprocess.PIPE, stdout = fNull)
fw = p.stdin
fw.write(postBody)
fw.flush()
fw.close()
del p

As you can see it’s more verbose now but I’ve eliminated shell (slightly faster operation).

Some notes found during migration:

• without “del p” process may be not terminated causing problems with DB state (CGI proces updates database and test checks this state later)
• I/O configuration is more flexible than os.popen() – you can make pipes more easily

However, some data (or some requests) cannot be downloaded from multicasts channels and HTTP/HTTPS must be used to interact with server. As the number of devices is very high special methods have been used in order not to overload backend servers (randomised delays, client software optimization).

Consequently, small bug in client software that will trigger more events than usual can be very dangerous to whole system stability (because the effect of thousands of devices – perfect DDOS may kill back-end). In order to catch such errant behaviour as soon as possible I’ve developed daily report that controls server usage in my customer office.

First of all, we need to locate the most “interesting” device by IP address from logs (we list top 20 IPs based on server usage):

    ssh $server "cat /path-to-logs/localhost_access_log.$date.log" | awk '
        {
            t[$1 " " $7]++
            ip[$1]++
        }
        END {
            for(a in t) { print t[a], a }
            max = 0
            max_ip = ""
            for(a in ip) { if(ip[a] > max) { max=ip[a]; max_ip=a; } }
            print max_ip > "/tmp/max_ip"
        }
    ' | sort -n | tail -20
    IP="`cat /tmp/max_ip`"

Then selected IP will be examined hour-by-hour to locate patterns in behavior:

ssh $server "cat /path-to-logs/localhost_access_log.$date.log" | awk -v "SEARCH=$IP" '
{ sub(/:[0-9][0-9]:[0-9][0-9]$/, "", $4); TIME=$4; s[TIME]; }
$0 ~ SEARCH { s[TIME]++;}
END { for(a in s) { print a, s[a] } }
' $* | sort

Sample results with included requested URLs:

Number of calls on 2013-05-22, server: tm2

3 192.168.4.101 /path/nPVR getseries
3 192.168.4.101 /path/reminder get
3 192.168.4.101 /path/rentedItems
3 192.168.4.101 /path/stbProperties get
3 192.168.4.101 /path/subscriberInfo
3 192.168.4.140 /path/autoConfig
6 192.168.4.249 /path/authenticate
6 192.168.4.249 /path/favorite get
6 192.168.4.249 /path/rentedItems
6 192.168.4.249 /path/stbProperties get
7 192.168.4.249 /path/reminder get
7 192.168.4.249 /path/subscriberInfo
8 192.168.4.140 /path/authenticate
8 192.168.4.249 /path/nPVR get
8 192.168.4.249 /path/nPVR getseries
16 192.168.4.254 /path/subscriberInfo
25 192.168.4.254 /path/nPVR get
25 192.168.4.254 /path/nPVR getseries
83 192.168.4.254 /path/favorite get
98 192.168.4.254 /path/reminder get

192.168.4.254 activity per hour:

[22/May/2013:00
[22/May/2013:01
[22/May/2013:02
[22/May/2013:03
[22/May/2013:04
[22/May/2013:05
[22/May/2013:06
[22/May/2013:07
[22/May/2013:08
[22/May/2013:09
[22/May/2013:10
[22/May/2013:11
[22/May/2013:12 8
[22/May/2013:13 4
[22/May/2013:14 16
[22/May/2013:15 18
[22/May/2013:16 12
[22/May/2013:17 50
[22/May/2013:18 24
[22/May/2013:19 24
[22/May/2013:20 24
[22/May/2013:21 24
[22/May/2013:22 24
[22/May/2013:23 22

We can see that 192.168.4.254 device started to spam server 24 times per hour from 18:00 until midnight (imagine thousands of devices that synchronously send requests to back-end the same way). This device logs will be thoroughly examined for possible reason for that behaviour.

At Aplikacja.info, we believe that software problems must be located as early as possible (even during development process). In above example we re-use existing development environment to catch problems that might be discovered in production with huge user base.

The general idea is to move detection and neutralisation of a problem into this direction:

1. end user -> 2. customer’s QA -> 3. development company’s QA -> 4. developer (runtime) -> 5. developer (build time)

The more distance from end user a bug is detected and fixed the better for us. I hope you would love to squash problems the same way.

Happy hunting!

What about Google? Was their service (no kidding, it is a sevrice that searches information, paid by ads) always so blazingly fast? Let’s check this site performance monitored from London server over few years:

As we can see there were some small problems in 2009 (~5% of requests required more than 1 second to be processed). Not so bad.

In order to double check our measurement results there are monitor result from Newark, USA:

As you can see there were also (small) performance problems in 2009, in some months ~2.5% of requests were processed in more than 10 seconds. Still pretty damn good.

You can see current stats for google here, thanks to site-uptime.net – site uptime monitor.

TCP connection is composed of many IP packets, connected by common strem index number. You can select particular TCP stream using Analyze / Follow TCP stream option or directly select given stream by it’s index:

tcp.stream eq 9

If you want track every opened connection you can check 1st packet of every TCP stream opened to particular server IP (213.75.34.114 in our example):

tcp.flags.syn==1 and tcp.flags.ack==0 and ip.dst == 213.75.34.114

Note that with HTTP/1.1 things may be more complicated as this protocol supports “Persistent/Keep Alive” mode that allows multiple requests over one connection, so you may see only one packet with “tcp.flags.syn==1 and tcp.flags.ack==0″. In order to scan full exchange you have to analyse protocol contents for request / response pairs.

Another complication is HTTPS (HTTP over SSL layer) – you won’t be able even to count requests (if using “Keep Alive” mode). In this scenario you have to check traffic after HTTPS node or just inspect server logs.

Below you will find two sample projects announced on guru.com. First: is hardware-related work, 2nd is a plain database:

Notice the difference in proposals counters. 1st project has no bids as it’s:

• complicated: real time compression probably will require hardware support thus it requires pretty complicated embedding of  H264 encoder on FPGA chip
• unpopular: I bet hardware guys are outnumbered by PHP programmers with ratio 1 to 1000
• risky: you cannot work on high level of abstraction leaving all the ugly details (OS services, memory management, …) for lower layers

2nd project has many proposals and I bet there are many talented Hindi specialists that will do the job for 20% of offer you expect to earn. So you will loose with your bid (make no money).

The general idea is to find a niche that is not easy to reach by other software development companies to be able to compete in smaller market. Even if you have brilliant PHP programmers you have to limit your project budget expectations according to 100s others PHP-related companies around, the one with highest quality / cost will win.

In order to support QA and track current state of software stability I added automated random tests feature:

Every build produced by the build system hits testing devices automatically and crash / failed asserts  / warnings reports are published to developers (daily, in aggregated form).

We tracked every release branch (assuming it’s the most important point for quality measurements). The situation was like on diagram below.

The obvious problem you may notice is related to multiple release branches that must be tested (some support branches live for few months). For every new branch you have to setup new testing device (I decided to attach one device per one tested version). This solution was not scalable and not very easy to maintain.

If we assume that every fix done for release branch (2.0.3 for example) hits at last his master branch (2.0) it’s obvious that potential regressions can be tracked automatically using only master branches:

Benefits are quite obvious:

• regression cause detection during normal development (on master branch) is done more accurately – we are able to say that given problem was added new change X on day Y near hour Z – it’s visible from automatic tests outcomes (crashes / failed asserts starting from version X)
• less reconfiguration overhead – main development lines are changed less frequently that release branches
• more resources for multi-platform tests: if there’s less points in source tree to track we can focus on duplicating hardware for automatic tests – some bugs are hardware-related

Drawbacks:

• we don’t track exact released versions: that’s true, but we have QA for this purpose (to test final builds)

Sometimes you want to see why particular package has been included into the image.

$ bitbake -g devel-image
NOTE: Handling BitBake files: \ (1205/1205) [100 %]
Parsing of 1205 .bb files complete (1130 cached, 75 parsed). 1548 targets, 13 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
NOTE: preferred version 1.0.0a of openssl-native not available (for item openssl-native)
NOTE: Preparing runqueue
PN dependencies saved to ‘pn-depends.dot’
Package dependencies saved to ‘package-depends.dot’
Task dependencies saved to ‘task-depends.dot’

By using:

grep package-name pn-depends.dot

you can track what is the dependency chain for particular package, for example, let’s see what’s the reason for inclusion of readline in resulting filesystem:

$ grep ‘>.*readline’ pn-depends.dot
“gdb” -> “readline”
“python-native” -> “readline-native”
“sqlite3-native” -> “readline-native”
“sqlite3″ -> “readline”
“readline” -> “readline” [style=dashed]

$ grep ‘>.*sqlite3′ pn-depends.dot
“python-native” -> “sqlite3-native”
“qt4-embedded” -> “sqlite3″
“utc-devel-image” -> “sqlite3″
“sqlite3″ -> “sqlite3″ [style=dashed]
“devel-image” -> “sqlite3″ [style=dashed]
“devel-image” -> “sqlite3-dbg” [style=dashed]

As we can see that it’s the “devel-image” that requested “sqlite3″ and then “readline” was incorporated as a dependency.

*.dot fomat can be visualised, but it’s useles feature as the resulting graph may be too big to analyse.

cd /
sudo find . -type s > /tmp/sockets.lst
if sudo tar zcvpf $TMP_FILE --one-file-system --exclude-from=/tmp/sockets.lst -C / .
then
    echo backup OK
fi

1. checking client logs and assume all HTTP requests are reported
2. checking server logs to see what have been issued
3. using tcpdump for traffic monitoring

I’ll show you 3rd method – it’s useful if you don’t have access to server nor to client logs.

$ sudo tcpdump -s 1024 -l -A dst 192.168.3.120 -i eth0 | grep HTTP
..Hp.c..GET /url/path?param1=value1&OpCode=add&ChannelID=101434 HTTP/1.1
.....c.*GET /url/path?param2=value2&OpCode=add&ChannelID=101434 HTTP/1.1

192.168.3.120 is the server IP address.

Pretty simple and more elegant solution than using full wireshark (and you can use it having only console access).