Software Testing

I recently met with an Australian Software Development company, PepperStack, and we got onto the subject of Software Testing. As someone who began their career as an Electronics Hardware Engineer, one of the things I learnt was that you have to test thoroughly to be sure everything is working as it should be. With Electronics, if you make a mistake with an Engineering Calculation you can easily destroy things. This is sometimes referred to as “letting the smoke out”. So it was good to meet with others who believe in the same level of rigorous software unit, module and system testing that we do.

Some Engineering Humour

Which reminds me of a joke I once heard:

There are 3 Engineers in a car going for a drive. The first is a Mechanical Engineer, the second an Electronics Engineer and the third is a Software Engineer. Fortunately the Mechanical Engineer is driving because the brakes fail and they are going downhill.  The Mechanical Engineer eventually brings the car safely to a halt and gets out to examine the hydraulic systems.  The Electronics Engineer gets out and checks and body computer, ABS system and the power train CAN bus.  The Software Engineer stays in the car and when queried about it says that they should all just get back in the car and see if it happens again!

Now don’t get me wrong, I’m not having a go at Software Engineers. The process of finding and eliminating faults is a very important part of the development cycle and is something that needs up front thinking and not just responding to symptoms.  And the more complex or sophisticated a system is, that harder it is to fully test every possible response to every possible stimuli and after a certain point it becomes impractical to have 100% Test Coverage (every line of code has been executed through all of the possible states).  The reason this is a bigger problem with Software Development is that the flexibility of software means that it is inherently complex and it takes skill and planning to manage that complexity so it is testable.

So here is the issue. More than any other discipline, faults can be experienced by an end user of a product under a situation or scenario you could not have proactively tested against before release.  There are many potential reasons for this including:

• change of hardware or operating system environment
• new standards or protocols
• the sheer number of potential combinations of drivers, peripherals, software and users
• the product being used for a purpose it wasn’t originally designed for
• gamma ray corruption of a memory location – I am getting esoteric now but in some areas like avionics and space this is a big threat

So how do you reduce the likelihood of these problems occurring?

Improving Software Quality

With many new products having Electronics and Embedded Software and the Software Development requiring 80% of the effort, it is important to delivery it as quickly and fault free as you can. The main weapons in your Software Quality arsenal have been known about for a long time but are, in our experience, just not used.  These are:

• Architectural Design – work out how the data and execution flow will happen and how you will manage the constraints
• Functional Decomposition – divide and conquer but with an emphasis on how each module fits into the system and how the interfaces work in detail
• Error handling - who will decide what to do with response codes – again this is data and execution flow and part of the architecture. In many cases exception management is at least 50% of the project.
• Have an Integration Test Plan – some thing that proves the data and execution flow matches the architectural design.  Too often “it builds” seems to be good enough here.
• Unit Test modules – so you remove all the issues before adding them to the integration
• Do the Integration Tests before you try system testing
• Design modules so you can integrate them as shells then add functionality down the track
• Have NVM and configuration data available at the beginning of the project and not as an after thought at the end
• Have a System Test Plan and use it
• Use some of the good practices of Test Driven Development – run the tests every time you change the code
• Have a rationale for what level of Code Coverage you can accept
• Have a rationale for what level of Churn you can accept – Churn is the percentage of the lines of code that have changed in the past time period.  Usually either a week or month depending on the size of the project.
• Use automated software quality tools. For instance we use both PC-Lint and RSM to automated many software quality metrics which saves a lot of time in Code Reviews
• Use Code Reviews, also known as Software Peer Review.  It really does save time.

Next I plan to look at what you can learn about software testing from a Squash Racquet.

Ray Keefe has been developing high quality and market leading electronics products in Australia for nearly 30 years.  For more information go to his LinkedIn profile. This post is Copyright © 2010  Successful Endeavours Pty Ltd.

Casey Business Awards

The City Of Casey are holding there inaugural Casey Business Awards and at the Casey Business Breakfast this morning Successful Endeavours were nominated as finalists in 2 categories.

Casey Business Awards

The 2 Casey Business Awards categories are:

• Manufacturer Of The Year
• Business and Professional Services

We fall into the Business And Professional Services category with our Electronics and Embedded Software development services where we design products for Australian Electronics Manufacturers  so they can achieve Low Cost Electronics Manufacture in Australia at a good profit margin.

The Manufacturer Of The Year award category recognises that for some of our clients, we also manufacture the product the product and delivered to them programmed, tested and calibrated; ready to sell.  This includes products like a DNP3 enabled power controller product for the US Smart Grid market which is made right here in Berwick as well as the Award Winning Borgtech CPL2 Corrosion Protection Data Logger with Wireless Data Logging.

It was an honour to be recognised by our city council together with other small business owners in the City Of Casey, a municipality in the outer south-eastern suburbs of Melbourne.  We will find out who the winners are on Friday 27th August at the Casey Business Awards gala dinner.

Cranbourne News 5th August 2010 Best in Business

 Ray Keefe has been developing high quality and market leading electronics products in Australia for nearly 30 years.  For more information go to his LinkedIn profile. This post is Copyright  Successful Endeavours Pty Ltd.

An Engineering Problem in Disguise

A funny thing happened to me the other day during the Christmas shopping rush at our local shopping centre in Endeavour Hills.  Our daughter had purchased some clothes for her nieces for Christmas and used the self serve checkout. When she got home she discovered she had not had one of the security tags removed so she asked for my help.

OK, I might be an Electronics Hardware and Embedded Software Engineer but I did do a year of Physics and Chemistry at Deakin University before switching to Engineering and I have had a role in the design of Multidisciplinary Systems with Electromechanical Actuators and Variable Frequency Motor Drives including Multi-Axis Robotic Handlers.  So I thought, “How hard can this be?”

The first step was to review the problem and identify the information.  Those familiar with Edward De Bono‘s Six Thinking Hats will recognise this as the White Hat stage.

I had:

• circular plastic sealed tag with an alignment  feature – a hole through it to accept a tapered pin
• a metal pin with a large head inserted into the centre of the plastic disk
• no other visible connection points

So assuming the tag was made at a minimum price, needed to be aligned correctly to be released and could be disconnected without an external power source; I concluded that the release mechanism was probably magnetic.  So I got a magnet and did some experiments and I could hear something click inside the security tag as I moved the external magnet around.  Now I am very confident that it is a Magnetic Latching Mechanism.  But no orientation of a single magnet released the pin.

I got 2 magnets and worked around the unit until the pin released and the problem was solved.

Having released the tag I gave the garment to my daughter to wrap in Christmas paper and put the tag with pin inserted back into it by the front door.  Since we were shopping the next day I thought I would return the tag.  At the very least it would get recycled.

What’s so good about being an Engineer?

At the shops, I went to the help desk and offered them the tag.  They were very confused.  I explained that it had been accidentally left on one of the items we purchased so I took it off and was returning it to them.  The stunned reply was, “You took it off yourself”?  “Yes” I said.  “I’ll have to call security” was the next reply.  So I said, “It’s all right, I’m an Engineer“.  “Oh, that’s fine then” was the reply and I wandered off to collect some final groceries for Christmas dinner.

So apparently there was a connection in the shop assistants mind that made being an Engineer something special.  They may not have know what that connection was.  And that got me wondering about Engineers and what is so special about us.  Here is a bit of a list of my initial thoughts if I ignore specific Engineering Disciplines:

• we create the future by designing and constructing the machines and systems that it requires
• we routinely solve complex problems that others do not know even exist
• we do all of this because we want a better world and are prepared to do our part to achieve it
• we have learned that covering up a symptom is not the same as solving the underlying problem

You might have some thoughts of your own so please leave a comment.

And of course, I hope you had a Merry Christmas in 2009 and that 2010 is a very good year for you all.  Happy New Year!

Ray Keefe has been developing high quality and market leading electronics products in Australia for nearly 30 years.  For more information go to his LinkedIn profile. This post is Copyright  Successful Endeavours Pty Ltd.

How Does Sleep Save Energy?

For this post, we will look specifically at Embedded Software techniques to save power and energy.  This is a well known Power Saving Strategy which doesn’t always get the recognition it deserves.  It is also something you have to design into the Power Management Plan from the beggining.

For this example, we will use the MSP430 from TI which has some of the best Power Saving and Power Consumption figures in the industry.  We have used them to create devices that run from a pair of AAA batteries for 2 years and which have time based control algorithms so that they can’tbe used in a purely event driven mode.  Here is how it works:

Low Power Sleep Mode

This shows the power consumption versus time.  In Low Power Sleep Mode the consumption is close to zero.  Almost no power consumed.  Then depending on what is happening it wakes up to varying degrees.

Get the best Electronic Sleep

So this is how you take advantage of this:

• make the time between wake ups as long as possible
• make the time awake as short as possible
• only turn on the peripherals needs for a particular wake period

Now if you system only has to wake once every minute then you can get low power operation from a lot of different processors.  If it wakes many times a second then you need a processor that gives you lots of ways to reduce power during wake, reduce the time awake, and increase the interval between wakes.

MSP430 Sleep

So back to the MSP430. It has Power Conservation features that allow it to do all three better than most.  Here is the list:

• Digitally Controlled Oscillator DCO allows it to wake and run quickly
• Can run a Timer from a 32KHz crystal making interval timing very low power
• Can use the DCO to set the run speed and so shorten the wake time
• Lot’s of Power Down Modes so you can always find one that suits your application
• Peripherals can be Shut Down when not in use
• Can run down to 1.8V – more on that later but it can also help here

Low Power System Architecture

To take advantage of all this, you have to develop the System Architecture so that  takes advantage of this.  An example from a very long life application we did runs like this:

• 32Hz Oscillator runs a timer that generates a 1 second wake
• User input keys set up to wake on change of state from high to low
• Use DCO at 1MHz to quickly wake, execute & sleep again
• Use State Machines to allow modules to execute predictably with eratic timing
• Have early exit tests to prevent unnecessary Code Execution

The result is an application that runs a process with User Interaction, LED Indicators, and a 2 week cycle where the average Power Consumption is 20uA at 2.7V or 54uW.  Of this, less than half is the processor executing the software and the single biggest energy use is the intermittently flashed LED Indicators.

To learn more, check out this more comprehensive article on “Low power MCU selection criteria and sleep mode implementation” from embedded.com which provides more examples.

Ray Keefe has been developing high quality and market leading electronics products in Australia for nearly 30 years.  For more information go to his LinkedIn profile. This post is Copyright © Successful Endeavours Pty Ltd.

Electronics Manufacturers are the people we serve

A common question we are asked is what sort of Electronics Manufacturers do we Develop Products for?

So I thought I would compile 3 lists:

• The first is a list of the Electronics and Embedded Software product types we have worked on
• The second list is a list of the industries we have Developed Products for
• And the third list is the Technologies we have worked with so far

I might have to regularly update this third list since knowledge and technology are constantly expanding.  Before I do the lists I’d like to present a video that specifically addresses this last point.  This is very much worth thinking about.  Enjoy.

Electronics and Embedded Software Products

Did you notice the section from 1:45 to 2:15?  We are being prepared for jobs that don’t yet exist, technologies that haven’t been invented, and problems we don’t even know we will have!

Here is the list of some of the Electronics and Embedded Software Products that do already exist and which we have helped to create:

(more…)

There are a lot of news feeds running this story.  Here are a few:

Timeline ABC News
IEEE Spectrum
National Geographic
Business Journal

And when the Internet first began, no-one knew what we would be doing with it today.

This has happened to a lot of other technologies.  Low Cost Electronics and Low Power Design that can be Battery Operated has made many thing possible such as mobile phones, portable computers including netbooks, notebooks and laptops; portable media players, MP3 players, PDAs and the list has just begun.

But where is it going?  Realistically, you need a few things to come together and the environment and carbon footprint considerations now sit alongside the more traditional requirements such as:

• Low Cost Electronics Manufacture
• Low Power Electronics Design
• Design Tool Productivity improvements
• Electronic Design Automation
• Increased Processing Power per milliwatt (mW)
• Embedded Software of immense complexity and flexibility
• Flexible circuits
• Transparent Electronics
• Compact component size
• Reduced Polluting and Increased Recyclability and reuse

Vernor Vinge looked into what might become of this in his book Rainbows End which I recommend as a good read and full of well thought out ideas about how augmented reality might operate including concepts such as  wearable computers, gesture recognition, graphic overlays, the equivalent of doing a Google search on any object in your field of view, and other ideas like that.  It is set 20 years from now.

The most interesting for me was the way work was conducted in the future and how much advantage there was in having 100,000 people as affiliates on a program.  Pay is based on royalties for contributions.  You choose what you join and contribute to.  Your income directly reflects the product of your contribution and your negotiated royalty rate.  A large company has 3 direct employees and everyone else is an affiliate on one or more programs of work.  This produces phenomenal synergies.

It will be very interesting indeed to see how much of his vision matches the future.

Ray Keefe has been developing high quality and market leading electronics products in Australia for nearly 30 years.  For more information go to his LinkedIn profile. This post is Copyright © Successful Endeavours Pty Ltd.

________________________________________________

EDN Innovation Awards Finalist

EDN Innovation Awards

This is a bit of a different post.  I’m pretty stoked that we are finalists in the 2009 EDN Innovation Awards in 2 separate categories.  The award categories are:

• Best Application Of Analogue Design
• Best Application Of Design Software

Here is a list of the EDN Innovation Awards Finalists and we are in the 2 categories at the bottom of the page dealing with Analogue Design and Design Software.

So I thought I might let you know a bit more about the project, and also give a public thanks to Pablo Varjabedian of Borgtech for allowing us to put the project forward. We design Electronics and Embedded Software products primarily for Australian Electronics Manufacturers.  Our focus is outstanding Electronics Design that will propel them into a world class competitive position while delivering improved profit margins.  Low Cost Electronics Manufacture but with outstanding performance and reliability.

We routinely use non-disclosure agreements, NDAs, with our clients and so don’t usually get the chance to put our design work forward for awards because we will never disclose a client’s Intellectual Property, IP, without their express permission.  In this case Borgtech gave us permission and so we were able.  As you can probably see, there is a real benefit to the client in allowing the award application because they also get recognition for the product.

This is also not an unusual project for us. We have done a lot of outstanding work over the 12 years we have been in operation.  So it is good to have some of it recognised by the Industry we are so passionate about.

Electronics Design Details

This project was an example of our Project Priorities Perspective in action.  In this case Performance was the primary concern with cost coming second and time coming last.  We spent the time to get the performance up and the cost down.  There was an earlier post on one aspect of this project where we looked at Analogue Electronics as a way to improve battery life in a Low Powered Electronics Data Logger.

The Electronics Design trade offs were:

• OH&S or Operational Health and Safety – must protect users from hazardous voltages
• Low Power Electronics – operates from 3 AA cells for up to 6 months
• Convenience – Analogue front end completely Software Controlled
• High Reading Accuracy – millivolt resolution over +/-10V range with 60dB Mains Rejection

There were many other Design Requirements but the above list are the core Electronics Design Requirements addressed as part of the award nomination.  Below I will look at each of these in turn.

Protection From Hazardous Voltages

Now lets look at the hazardous voltage issue in a little bit more detail.  The voltages in questions were:

• 5000V, 5KV, for 2 seconds
• 250VAC continuously

These come about due to the conduction of Lightning Strike Transients or Mains Leakage Voltages onto the Pipelines and Storage Tanks monitored for Corrosion Protection status.  The Analogue Electronics front end had to provide protection against these cases while meeting all the other Design Requirements.  And of course quickly settle so that only the readings during the disturbance were affected.

It also led to the use of an 802.15.4 RF Telemetry Link because this meant the monitoring PC could do Real Time Monitoring without hazard.  Many other products in this industry use RS232, RS485 or even I2C connections for monitoring, configuration and upload of the Data Logger Records.  In the case of the Borgtech CPL2 you can put it in place and then configure it and start the logging with no danger to the operator apart from the moment of electrical connection.  And the initial part of the run can be monitored to ensure everything is correctly set up.  Otherwise you could get a months worth of data that was useless.

And finally, because of the power budget and the possibility of the batteries going flat, the Analogue Electronics had to survive the above Abuse Voltages unpowered!

Low Power Electronics

The Borgtech CPL2 is a Battery Operated device.  There are several reasons for this but the 3 most relevant are that it is:

• IP68 sealed against water ingress – it is often installed in a pit that can flood
• Must operate remotely from a convenient power source
• Protects the operator and PC from Transient Voltages since there isn’t a direct electrical connection

But this is also part of the challenge.   For convenience it used off the shelf batteries you can buy at any service station.  But to get 6 months life required a strong Power Management approach including powering down anything not in use including the Analogue front end.  If you are taking a reading every minute over six months then most of the device is off most of the time.  In this mode the average Power Consumption is 37uA.

Analogue Electronics – Software Controlled

The Borgtech CPL2 handles both Current Shunt and voltage mode readings. The Analogue Electronics were designed to have a software selectable full scale range of +/-10VDC and +/-150mVDC so that is could do either mode of operation from the same input. The previous model required a different connection for each of these modes and most other models on the market are the same.

And all of this while maintaining accuracy, abuse voltage protection and low power operation.

High Reading Accuracy

By the standards of an Agilent (I still want to call them Hewlett Packard) 6.5 digit laboratory multimeter our millivolt, mV, resolution at +/-10VDC isn’t rocket science.  But for a device with the Voltage Abuse Protection and Low Power Electronics requirements we had to meet, it is pretty good. Another small twist you might not recognise is that it is +/-10VDC.  This means you can monitor it with the polarity inverted and fix it up later on by inverting all the readings. The previous model was unipolar and so you couldn’t do this meaning you could have just wasted a month.  And then there is the live monitoring so you can see what the readings look like before leavign the unit to log away in the background.

EDN Innovation Awards

On 17 September 2009 we know the final outcome but either way I am pretty happy to have the recognition this project has already received.

Ray Keefe has been developing high quality and market leading electronics products in Australia for nearly 30 years.  For more information go to his LinkedIn profile.This post is Copyright © Successful Endeavours Pty Ltd.

First some basic statistics that made me think about this issue a bit more:

•  Software development is responsible for 80% of the delays and complications associated with designing a new product.  Source Embedded Forcast
• 80% of embedded projects are delivered late.  Source Embedded.com
• Software typically consumes 80% of the development budget.  Digital Avionics Handbook and Embedded.com
• 80% of software projects are unsuccessful  IBM

That is a lot of 80% figures associated with the software component of product development.

So working from the Pareto Principle it is clear that product development success and cost can be most improved by addressing the Software Development component.  In my recent post on Reducing Electronics Manufacturing Parts Cost I argued that increasing the software component can reduce the hardware costs.  Which is a great idea as long as it doesn’t introduce an even more expensive problem. 

I agree with Jack Ganssle in his article looking at tools where he points out that software quality tools are often not budgetted for yet will find many classes of defect quickly and at a significantly lower cost than the test and debugging  effort required to find them after integration with the rest of the project.  Or put another way, the cheapest way to get rid of bugs is not to introduce them in the first place – Lean Coding.

Since we mainly develop in C and C++, this is what we do to ensure we minimise software development cost and overruns:

Static analysis and code reviews

We use static analysis and code quality tools such as PC-Lint and RSM and integrate them into our editors and IDEs so we can run the tests are part of our build or at the very least with a single click covering either the current file or the current project.  These tools find flaws you are hard pressed to identify by visual inpection and I believe they pay for themselves within a month of purchasing them.  They can also enforce coding standards.  Another great benefit is that when you do a code walkthrough and review, you are not looking for these classes of faults explicitly because you know the toolset will find them for you.  So the first thing you do is run the tests and focus on anything found there.

Code reviews save money.  Every issue identified in a code review is an issue you don’t have to debug later on. And another person is going to look at your code without the same assumptions you would so they will see the things you miss.  It just makes sense to do it.  Software debugging is more expensive than coding so not bugging in the first place is good budget management.

Smart Bear Software have an excellent whitepaper you can download for free that covers best practices of peer code review  and if this is a new idea to you, then I strongly recommend you get the whitepaper as they have distilled a lifetimes worth of learnign in this area into a concise and easily implementable strategy to improve code quality.

Unit testing

Next, we unit test.  A huge benefit of this is that you have to think about test and it makes you think about error handling in the design phase.  Many problems in implementing embedded systems come from not handling errors consistently.  Sometimes they aren’t handled at all!  In Failure is an option this gets explored a little.  Someone else once suggested that software developers were the most optimistic people on the market – you can tell this is true by looking at how they handle exceptions!  I’m not sure who said it so if you know then post a comment and I’ll credit them and provide a link too if you have one.

Integration testing

Integration testing itself does not have to be overly complex.  You want to know that things work and it is often easier to write a cut down system to manage the test process.  This way you are proving that each susbsystem is present and correct before doing the full scale system test.  This is an area that often gets overcomplicated.  Don;t try and do more here than you have to.

Oh, and by the way, just because something builds don’t mean it passes the integration test.  Some things to cover are:

• software manifest – do I have the right version of each module?
• data flow – do the higher level calls get at the right data lower down?
• exceptions – do error returns get passed back?
• exceptions again – if you raise exceptions, do they get acted on?
• communications – does it communicate? 
• IO – are they mapped to the right pins and peripherals?

Simulation

For some systems or subsystems we write fully fledged PC mocks around the code and ensure it handles all the parameter and error cases correctly and that all the functions are correctly implemented.  This is a form of integration testing that proves the software component of the system is doing what it is meant to but goes a lot further to fully excercise part of it.  And since 80% of the problems come from software this is a very effective way of reducing bugs and difficult to track down system defects that are expensive on time and resources to cover in real time operating tests.

To do this, you have to abstract the interface so the code can run in the embedded version or the PC version without any changes.  This is easy to do if you think about it in advance.

One word of caution; the PC has a lot more resources and clock speed available compared to a smaller embedded system so this is not a substitue for testing on the real hardware to ensure execution latency is acceptable.

And for the purposes of this post, the PC could just as easily be a Linux or Mac system.  The point is to use the higher level system to efficiently and fully test the embedded software module so you save time and money later on in the project.  And let’s face it, who like to be under unnecessary pressure at the back end of an embedded software project?

System testing

If you think in advance about how to most easily implement the system testing then you can save a lot here as well.  We put effort into deciding how the do the test process at the architecture design phase so that we have the data flow required to actually do the test.  This can be as simple as having some extra parameters or calls available to be able to inspect the state of the system and the communications facilities to get at this data.  Where possible 100% parameter range testing and 100% code coverage testing is very desirable.  One thing this means is that you had better think about how you will create each error condition that must be handled!

Low Cost Software Development

Low Cost Electronics Manufacture relies on Low Cost Software Development.  So make it a priority.  The Pareto Principle says that it is the most important thing to get right.

Ray Keefe has been developing high quality and market leading electronics products in Australia for nearly 30 years.  For more information go to his LinkedIn profile. This post is Copyright © Successful Endeavours Pty Ltd.