Software Production Management comes of age
There's no software engineering environment more demanding than the world of embedded systems. Embedded system developers are faced with building software that must run on a multitude of target hardware (with complex configurations for each processor or hardware platform targeted), with short time to market driven by rapid innovation and competition (as illustrated by the market for handheld devices), and with the challenge of incorporating third-party code.
Those challenges add up to a huge matrix of software configurations and target platforms, and result in a myriad of production management problems for build managers, SCM managers, and QA professionals. Just managing the number of possible configurations and generating builds for all of them on a regular basis is, in itself, a very complex task.
Add to that a general move to iterative development (as espoused by Agile or Extreme Programming methods) which place additional demands on software production. The “nightly build” is rapidly being replaced by developers requesting a build every time they commit code to gain frequent feedback and ultimately improve software quality. So as a result, build managers are faced with managing 10s or 100s of builds per day across a complex set of configurations.
All this is happening at a time when market forces are demanding short development cycles coupled with maintaining embedded software's traditional focus on reliability. And many companies are globalising, with teams working on the same code around the world and around the clock.
The response, from both open source and commercial software developers, is a new category of tools for Software Production Management that addresses the full range of build, package, test and deploy processes. Software Production Management contrasts with Software Creation (the process of creating new software) that for many years has seen rapid innovation in its toolset (i.e. source code control, editors, debuggers, etc.). Only recently have Software Production Management tools covering the full range of processes become available or even necessary. The combination of web-based interfaces (which can be trivially deployed throughout an organisation) and cost-effective rack-mounted servers means that advanced tools can leverage large clusters of cheap machines and make that power available through a familiar web-based interface hiding the underlying complexity of the system.
Most software organisations start out building their own Software Production Management tools from a collection of macros, Perl scripts, Makefiles and Cron jobs. When the number of builds is small, or the number of different configurations is small these home grown tools are manageable. More recently, teams have wrapped simple web-based interfaces around their existing infrastructure to make monitoring or management easier.
As the matrix of configurations grows and as pressure to build more frequently grows, home grown tools quickly become inadequate and hard to manage.
Software Production Management meets those challenges through three important components: management, acceleration and analysis.
Management
It's easy to imagine that once the number of software builds rises above single figures that it's very hard to answer simple questions like "what's the status of the system?", "when will this build be run?", "when will this build be finished?", or "where's the output of this build?".
Additionally, the large production systems required to run large numbers of software builds require improved access control and scheduling especially if developers are given access to the software production system and build cluster.
Day to day management of a large number of software production systems will inevitably involve hardware or software failure. Advanced software production management systems enable users to work around such failures (or even automatically heal themselves) to ensure that critical software production processes are not unduly affected.
Reporting also becomes a critical function as the status of software production on a daily basis becomes an indication of the overall health of a software project. Trend analysis will help spot recurrent problems (a particular software production process that fails more frequently than others) and important trends that affect future planning (the growth in the size or length of a build that will necessitate additional build resources at a later date).
Such management capabilities are typically not available in home grown or open source systems.
Acceleration
Ultimately, Agile techniques or simply the explosion in the software configuration matrix will drive the need for many builds per day.
Advanced embedded development organisations are currently driving to give developers full software builds eliminating the nightly build and improving quality by spotting problems quickly. In addition to being able to support 100s of builds per day, the builds must be fast enough that problems are discovered quickly. This means all software builds should be possible in less than one hour.
The one hour goal drives the need for acceleration techniques (such as parallel building on commodity hardware) to change hours-long builds to minutes-long builds. Traditional parallel build systems (such as GNU Make's -j feature) typically have a hard time scaling beyond a 4x speedup without extensive rework of existing build scripts, but getting under an hour often means speedups of 8x, 16x or higher.
Commercial acceleration tools are available to reliably drive build times down through parallelisation while guaranteeing that the build is correct (the same bits are built every time).
Analysis
Lastly, software production processes (and even the inner workings of an individual software build) are very complex, involving thousands of individual steps. Answering questions like "which step in this software build consumes the most resources?" or "how much would a faster processor speed up this build?" ”what was the last ‘good’ build?” or “how many unit tests were successfully completed in this process?” require visualisation and reporting tools tailored to the software production management process.
Software developers have long been used to advanced visualisation tools such as multi-way coloured diffs, graph visualisation of branching structures in source code control or entity-relationship diagrams. Software production processes require analogous visualisation and analysis tools to handle their increased complexity. At the same time, builds have historically been the domain of a few members of the organisations, such that managers and executives had little visibility into software production status. Commercial tools are starting to bridge that gap, with reporting that gives all stakeholders a view into the software production “critical path.”
Without investment in tools and process improvement, software production for embedded development projects can be unmanageable. Increased demands on the accuracy and speed of the software production process coupled with legislated requirements to keep audit trails of software production require a new class of software: Software Production Management.