Skip to content

Entries from April 2013.

Can you compete in a crowded area?

Can you compete in crowded area?

Below you will find two sample projects announced on guru.com. First: is hardware-related work, 2nd is a plain database:

Notice the difference in proposals counters. 1st project has no bids as it's:

  • complicated: real time compression probably will require hardware support thus it requires pretty complicated embedding of H264 encoder on FPGA chip
  • unpopular: I bet hardware guys are outnumbered by PHP programmers with ratio 1 to 1000
  • risky: you cannot work on high level of abstraction leaving all the ugly details (OS services, memory management, ...) for lower layers

2nd project has many proposals and I bet there are many talented Hindi specialists that will do the job for 20% of offer you expect to earn. So you will loose with your bid (make no money).

The general idea is to find a niche that is not easy to reach by other software development companies to be able to compete in smaller market. Even if you have brilliant PHP programmers you have to limit your project budget expectations according to 100s others PHP-related companies around, the one with highest quality / cost will win.

 

Automated tests reorganization

My customer develops software for embedded devices using Linux. In the very beginning of this project we faced low stability of builds produced due to complicated nature of platform (C++, manual memory management, new APIs to learn). Of course QA located such bugs, but it was triggered before release on release branches.

In order to support QA and track current state of software stability I added automated random tests feature:

Every build produced by the build system hits testing devices automatically and crash / failed asserts / warnings reports are published to developers (daily, in aggregated form).

We tracked every release branch (assuming it's the most important point for quality measurements). The situation was like on diagram below.

The obvious problem you may notice is related to multiple release branches that must be tested (some support branches live for few months). For every new branch you have to setup new testing device (I decided to attach one device per one tested version). This solution was not scalable and not very easy to maintain.

If we assume that every fix done for release branch (2.0.3 for example) hits at last his master branch (2.0) it's obvious that potential regressions can be tracked automatically using only master branches:

Benefits are quite obvious:

  • regression cause detection during normal development (on master branch) is done more accurately - we are able to say that given problem was added new change X on day Y near hour Z - it's visible from automatic tests outcomes (crashes / failed asserts starting from version X)
  • less reconfiguration overhead - main development lines are changed less frequently that release branches
  • more resources for multi-platform tests: if there's less points in source tree to track we can focus on duplicating hardware for automatic tests - some bugs are hardware-related

Drawbacks:

  • we don't track exact released versions: that's true, but we have QA for this purpose (to test final builds)

 

Openembedded: tracking dependencies, the effective way

Openembedded is a tool to build Linux-based distribudions from source where you can customize almost every aspect of operating system and it's default components. Used mainly for embedded systems development. I'll show today how you can check easily (command-line, of course) dependencies chain.

Sometimes you want to see why particular package has been included into the image.

$ bitbake -g devel-image NOTE: Handling BitBake files: \ (1205/1205) [100 %] Parsing of 1205 .bb files complete (1130 cached, 75 parsed). 1548 targets, 13 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies NOTE: preferred version 1.0.0a of openssl-native not available (for item openssl-native) NOTE: Preparing runqueue PN dependencies saved to 'pn-depends.dot' Package dependencies saved to 'package-depends.dot' Task dependencies saved to 'task-depends.dot'

By using:

grep package-name pn-depends.dot

you can track what is the dependency chain for particular package, for example, let's see what's the reason for inclusion of readline in resulting filesystem:

$ grep '>.*readline' pn-depends.dot "gdb" -> "readline" "python-native" -> "readline-native" "sqlite3-native" -> "readline-native" "sqlite3" -> "readline" "readline" -> "readline" [style=dashed]

$ grep '>.*sqlite3' pn-depends.dot "python-native" -> "sqlite3-native" "qt4-embedded" -> "sqlite3" "utc-devel-image" -> "sqlite3" "sqlite3" -> "sqlite3" [style=dashed] "devel-image" -> "sqlite3" [style=dashed] "devel-image" -> "sqlite3-dbg" [style=dashed]

As we can see that it's the "devel-image" that requested "sqlite3" and then "readline" was incorporated as a dependency.

*.dot fomat can be visualised, but it's useles feature as the resulting graph may be too big to analyse.

Ignore sockets during tar backup

If you want to skip error sockets-related code during backup of whole filesystem you will be surprised there's no --ignore-sockets switch. But there's elegant method to to that:

cd /
sudo find . -type s > /tmp/sockets.lst
if sudo tar zcvpf $TMP_FILE --one-file-system --exclude-from=/tmp/sockets.lst -C / .
then
    echo backup OK
fi

PHP auto quotes escape in allegro.pl

Use tcpdump to sniff HTTP requests

Sometimes you are interested if the software issues proper HTTP requests to the server. You have three options here:

  1. checking client logs and assume all HTTP requests are reported
  2. checking server logs to see what have been issued
  3. using tcpdump for traffic monitoring

I'll show you 3rd method - it's useful if you don't have access to server nor to client logs.

$ sudo tcpdump -s 1024 -l -A dst 192.168.3.120 -i eth0 | grep HTTP
..Hp.c..GET /url/path?param1=value1&OpCode=add&ChannelID=101434 HTTP/1.1
.....c.*GET /url/path?param2=value2&OpCode=add&ChannelID=101434 HTTP/1.1

192.168.3.120 is the server IP address.

Pretty simple and more elegant solution than using full wireshark (and you can use it having only console access).

Przeczytaj zanim wyślesz - zwłaszcza do tysięcy klientów

To jest chyba zaraźliwe (e-mail od Freeconet, wzorowali się na STMicro): Szanowny Użytkowniku!

Do poprzedniej wiadomości, dotyczącej niższych stawek za połączenia komórkowe, wkradł się błąd związany z datą wprowadzenia obniżki. Zmiana cennika obowiązuje od 10 kwietnia br. (a nie od 10 marca, jak podane było we wcześniejszej wiadomości).

Przepraszamy za pomyłkę, Wnioski:

1) Czytamy e-maile dwa razy przed wysłaniem, żeby nie musieć potem prostować. Zwłaszcza dotyczy to komunikatów marketingowych do tysięcy klientów.

2) Czytając pierwszy e-mail nawet nie zwróciłem uwagi na błędną datę. Pomyłka raczej oczywista (nie zmienia się zwykle stawek z poprzedniego okresu rozliczeniowego). Ktoś zadecydował, że muszę się o tym fakcie koniecznie dowiedzieć. Kolejny e-mail do przeczytania. Dlaczego? Teraz Freeconet ma na moim blogu darmowa reklamę :-P

CakePHP tutorial part 1: Installation, model

"What? PHP? Are you crazy?" - You may say. We all know that there are more elegant, faster, less error-prone languages. I do love Python for elegance, Java for static typing, C++ for performance, however, please be aware of that, one can find faster good PHP programmer to hire than average C++ programmer who does understand at least importance of virtual destructor for avoiding memory leaks. In this series I'm going to show how can we adapt PHP (and it's most popular framework) for serious web-based systems with features I expect from Django/RoR/... environments. May be not easy, but let's try!

Installation

First of all, we need to download CakePHP source, we will use (the current stable version) 2.3.1: $ cd /var/www $ wget --no-check-certificate https://github.com/cakephp/cakephp/archive/2.3.1.tar.gz $ tar zxf 2.3.1.tar.gz Secondly, our document root must be redirected to app/webroot subdirectory of CakePHP installation directory (we use Apache 2.0, you have to use root account as it's a system-level service): # vi /etc/apache2/sites-available/default DocumentRoot /var/www/cakephp-2.3.1/app/webroot

/etc/init.d/apache2 restart

Then randomize Security.salt and Security.cipherSeed values in app/Config/core.php. Hint: you can use pwgen tool to create pretty random strings there: $ pwgen -s 64 1 $ vi app/Config/core.php Let's start with database configuration then: firstly, we adapt template of configuration to suit our database setup: $ mv app/Config/database.php.default app/Config/database.php $ pwgen -s 32 1 $ vi app/Config/database.php Then create MySQL user and database: $ echo "CREATE USER phpcake IDENTIFIED BY '<MYSQL_USER_PASS>'" | mysql -u root -p<MYSQL_ROOT_PASS> $ echo "CREATE DATABASE phpcake" | mysql -u root -p<MYSQL_ROOT_PASS> $ echo "GRANT ALL ON * TO phpcake" | mysql -u phpcake -p<MYSQL_USER_PASS> phpcake You can verify if DB setup is correct by using default application installed in CakePHP: it will show you DB connection status:

Sample Model

In order to show very basic database functionality we will create two tables: one for payment representation, one for account, both connected by foreign key relation: CREATE TABLE Account (id INT NOT NULL AUTO_INCREMENT, name VARCHAR(32), currency VARCHAR(3), PRIMARY KEY(id)); CREATE TABLE Payment(id INT NOT NULL AUTO_INCREMENT, accountID INT, amount DECIMAL(10,2), PRIMARY KEY(id)); (Don't laugh at me, it's not db schema creation lesson, just PHP tutorial, so we keep structure as simple as possible for brevity).

CakePHP suggest plural form of class name here, I do prefer singular form of database table name, the same as class name. It's more natural, then, to write select queries with JOINs.

Then we can create model classes in PHP that refers to above DB tables. Models are placed in CakePHP in files app/Model/ClassName.php: Account.php: <?php class Account extends AppModel { public $useTable = 'Account'; }

Payment.php: <?php class Payment extends AppModel { public $useTable = 'Payment'; } Note that you don't have to insert list of fields, they will be retrieved from DB using reflection.

In order to quickly check if DB connection works as expected on model level, let's try to load records from DB using existing controller file, app/Config/core.php. Let's add the following lines in this module: public $uses = array('Account', 'Payment'); (...) $this->Account->find('all'); $this->Payment->find('all'); $uses variable specifies the model classes that must be imported for this controller, $this->ModelClassName->* allows to call model queries/updates.

You can see resulting queries in page footer when you set debug mode to 2 (app/Config/core.php): SELECT Account.id, Account.name, Account.currency FROM phpcake.Account AS Account WHERE 1 = 1 SELECT Payment.id, Payment.accountID, Payment.amount FROM phpcake.Payment AS Payment WHERE 1 = 1 Some criticism here:

  • CakePHP doesn't raise an error (typical PHP sin) when there's a typo in $uses - one may miss simple typo there and wonder how her changes aren't visible - I don't know how to enable a warning/error in that situation (yet)
  • Database table naming convention: CakePHP uses lower-case letter in plural form, I do prefer to use the same name as for model class, that can be fixed by "$useTable" mechanism

In next posts on this subject I'll investigate CakePHP pro's and con's and show how to overcome them. I'd like to review the following topics (among others):

  • Lint-like functionality for PHP (pseudo static typing, exposing any mistakes as lint error)?
  • Runtime error reporting to logs/e-mail (with full stacktraces)
  • Built-in unit testing capability
  • Integration testing using HTTP

The final goal is to find whether PHP-based system is able to scale (in terms of source code/project complexity). I do expect some basic features from programming environment to grow without pain and will verify if/how they can be achieved in PHP.

allegro.pl connection problems - detailed analysis

I've just observed I cannot reach allegro.pl site. Let's check what has failed (this time).

First of all, let's check ICMP availability:

$ ping allegro.pl
ping: unknown host allegro.pl

Ops, looks like something wrong with DNS, confirmation below: $ dig allegro.pl

; <<>> DiG 9.8.1-P1 <<>> allegro.pl ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56412 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION: ;allegro.pl. IN A

;; Query time: 32 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Apr 4 22:18:10 2013 ;; MSG SIZE rcvd: 28

No DNS response from default servers.

Let's check WHOIS record for this domain:

$ whois allegro.pl | grep -A 1 nameservers:
nameservers:           gtmdc3.allegro.pl. [91.207.14.244]
gtmdc2.allegro.pl. [91.194.188.132]

We see we have two nameservers specified, lets check if they're available (ICMP): $ ping -c 1 91.207.14.244 PING 91.207.14.244 (91.207.14.244) 56(84) bytes of data. 64 bytes from 91.207.14.244: icmp_req=1 ttl=247 time=32.8 ms

--- 91.207.14.244 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 32.823/32.823/32.823/0.000 ms

$ ping -c 1 91.194.188.132 PING 91.194.188.132 (91.194.188.132) 56(84) bytes of data. 64 bytes from 91.194.188.132: icmp_req=1 ttl=248 time=57.0 ms

--- 91.194.188.132 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 57.038/57.038/57.038/0.000 ms

Seems to be up and running. Let's check DNS responses from those servers: $ dig allegro.pl @91.207.14.244

; <<>> DiG 9.8.1-P1 <<>> allegro.pl @91.207.14.244 ;; global options: +cmd ;; connection timed out; no servers could be reached

That's the problem: 91.207.14.244 is not responding for DNS queries. Let's check secondary server then: $ dig allegro.pl @91.194.188.132

; <<>> DiG 9.8.1-P1 <<>> allegro.pl @91.194.188.132 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55010 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; WARNING: recursion requested but not available

;; QUESTION SECTION: ;allegro.pl. IN A

;; ANSWER SECTION: allegro.pl. 30 IN A 72.52.5.208

;; Query time: 31 msec ;; SERVER: 91.194.188.132#53(91.194.188.132) ;; WHEN: Thu Apr 4 22:25:34 2013 ;; MSG SIZE rcvd: 44

Here we have a server located, let's see it's availability by HTTP protocol: $ telnet 72.52.5.208 80 Trying 72.52.5.208... Connected to 72.52.5.208. Escape character is '^]'. GET / HTTP/1.0

Connection closed by foreign host.

Great :-)

Reverse DNS:

$ host 72.52.5.208
208.5.52.72.in-addr.arpa domain name pointer unknown.prolexic.com.

Looks like DDOS again (guys at prolexic sells anti-DDOS software).