Dariusz on Software Quality

21/04/2013

Automated tests reorgnization

Filed under: en — Tags: , — dariusz.cieslak @

My customer develops software for embedded devices using Linux. In the very beginning of this project we faced low stability of builds produced due to complicated nature of platform (C++, manual memory management, new APIs to learn). Of course QA located such bugs, but it was triggered before release on release branches.

In order to support QA and track current state of software stability I added automated random tests feature:

Every build produced by the build system hits testing devices automatically and crash / failed asserts  / warnings reports are published to developers (daily, in aggregated form).

(more…)

08/06/2012

“Gray box” analysis example – FogBugz case

Filed under: en — Tags: , — dariusz.cieslak @

Recently I’ve hit the following exception at FogBugz site (hosted commercial bug tracker which one I’m a happy user):

System.ArgumentException: Invalid syntax: expected identifier, found ')'

Server stack trace:
   at FogCreek.FogBugz.Database.CSqlParser.ParseIdentifier(CSqlTokenList tokens)
   at FogCreek.FogBugz.Database.CSqlParser.ParseColumn(CSqlTokenList tokens, Nullable`1 fTableNameRequired)
   at FogCreek.FogBugz.Database.CSqlParser.ParseTerm(CSqlTokenList tokens, Nullable`1 fInsideSelect,
      Nullable`1 fInsideInsert, Nullable`1 fInsideOrderBy)
   at FogCreek.FogBugz.Database.CSqlParser.ParseExpression(CSqlTokenList tokens, Nullable`1 fInsideSelect,
      Nullable`1 fInsideInsert, Nullable`1 fInsideOrderBy, Nullable`1 fIsBoolean)
   at FogCreek.FogBugz.Database.CSqlParser.ParseBoolTerm(CSqlTokenList tokens)
   at FogCreek.FogBugz.Database.CSqlParser.ParseBoolExpression(CSqlTokenList tokens)
   at FogCreek.FogBugz.Database.CSqlValidator.AssertValid(String s, String sType)
   at FogCreek.FogBugz.Database.CSqlValidator.AssertValidWhereList(String s)
   at FogCreek.FogBugz.Database.CWhereQuery.AddWhere(String sSqlWhere)
   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args,
      Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg, Int32 methodPtr,
      Boolean fExecuteInContext)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at FogCreek.FogBugz.Database.CWhereQuery.AddWhere(String sSqlWhere)
   at AlbrektsenInnovasjon.FogBugzData.AddQueryFilter(CWhereQuery query)
   at AlbrektsenInnovasjon.FogBugzData.RenderTimesheetReport(StringBuilder page)
   at AlbrektsenInnovasjon.FogBugzData.RenderReport(StringBuilder page)
   at AlbrektsenInnovasjon.FogBugzTimeSheetReport.FogBugzTimeSheetReportPlugin.FogCreek.FogBugz.
     Plugins.Interfaces.IPluginPageDisplay.PageDisplay()
   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md,
     Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg, Int32 methodPtr,
     Boolean fExecuteInContext)

Exception rethrown at [1]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at FogCreek.FogBugz.Plugins.Interfaces.IPluginPageDisplay.PageDisplay()
   at FogCreek.FogBugz.__Global.PluginPage(Int32 ixPlugin) in c:\code\hosted\build\FB\8.8.28H\fogbugz\
     src-Website\pluginPages.was:line 35
   at FogCreek.FogBugz.__Global.RawpgPlugin() in c:\code\hosted\build\FB\8.8.28H\fogbugz\src-Website\
     default.was:line 3753
   at FogCreek.FogBugz.__Global.pgPlugin() in c:\Users\john\AppData\Local\Temp\dxcwqgiu.0.cs:line 0
   at FogCreek.FogBugz.__Global.RunPg() in c:\Users\john\AppData\Local\Temp\dxcwqgiu.0.cs:line 0
   at Wasabi.Runtime.Web.Response.PictureOf(Sub sub) in c:\code\hosted\build\FB\8.8.28H\wasabi\
     Wasabi.Runtime\ResponseGenerator.cs:line 24
   at FogCreek.FogBugz.__Global.RawRunFogBugz() in c:\code\hosted\build\FB\8.8.28H\fogbugz\
     src-Website\default.was:line 4661
   at FogCreek.FogBugz.__Global.RunFogBugz() in c:\Users\john\AppData\Local\Temp\dxcwqgiu.0.cs:line 0
   at FogCreek.FogBugz.__Global.StartDefault() in c:\code\hosted\build\FB\8.8.28H\fogbugz\
     src-Website\default.was:line 131
   at FogCreek.FogBugz.HttpHandler.ProcessRequest(HttpContext context) in c:\Users\john\
     AppData\Local\Temp\dxcwqgiu.0.cs:line 0

The fact that stacktrace is visible on public site (it may espose many internal details of implementation) is one major problem with that site, the information you can gather: a challenge. Let’s investigate then.

(more…)

22/04/2012

Classic Testing vs Design By Contract

Filed under: en — Tags: , , — dariusz.cieslak @

Automated unit tests are hard to write. Software architecture must be designed carefully to allow unit testing. You have to spend time to write tests as well and it’s not easy to write good tests. It’s easy to make big mess that is hard to maintain after few weeks.

On the other hand automated integration tests are hard to maintain and are fragile. You can “write” them pretty easy in a record-and-replay tool, but later they show their real cost during maintenance.

But there’s an answer for problems mentioned above. Do you know Eiffel language? The language has special built-in constructs that support executable contract specification. It’s called Design By Contract (DBC). DBC is easier to write and to maintain because no special test cases need to be specified, just conditions for method parameters, expected results and state invariant that must be preserved. How DBC can substitute old-fashioned tests? Read on!

(more…)

16/03/2012

What’s Wrong With Automated Integration Tests?

Filed under: en — Tags: , , — dariusz.cieslak @

A quite typical picture: software development company X, first delivery of project Y to testing team after few months of coding just failed because software is so buggy and crashes so often. Project Manager decides to invest some money in automated testing tool that will solve all stability problems with click-and-replay interface. Demos are very impressive. The tool was integrated and a bunch of tests were “clicked”.

After a month we have 10% of tests that are failing. 10% is not a big deal, we can live with them. After additional month 30% of tests fails because important screen design was changed and some tests cannot authorize them for some reason. Pressure for next delivery increases, chances to delegate some testers to fix failing test cases are smaller every week.

What are the final results of above tool?

  • unmaintained set of tests (and the tool itself) is abandoned
  • man-days lost for setting up test cases
  • $$ lost for the tool and training

Has the tool been introduced too late? Maybe wrong tool was selected?

In my opinion automation and integration tests don’t play well together. Let’s review then main enemies of automation in integration tests:

(more…)

16/01/2012

Static verificaiton tool for web2py templates

Filed under: en — Tags: , , , , — dariusz.cieslak @

Web2Py is a full-stack Python web framework that can be compared to Django, but is easier to learn due to convention-over-explicit-statement preference. In this article I’ll check how static verification techniques developed by me for many different environments (JSP, Django templates, TAL, …) can be applied for web2py environment.

Static verification means locating simple bugs without running application thus very high (>95%) testing coverage (and high related cost) is not required. Instead with trying to cover by tests every possible screen/workflow/code line/… we can scan all codebase and search for some constraints. Most of them (based on my experience) are static – do not depend on runtime data thus can be effectively checked without running an application.

(more…)

Older Posts »

Powered by WordPress