Version 22.1 February 3, 2013
Capers Jones VP and CTO
Namcook Analytics LLC
Web: www.Namcook.com
Email: Capers.Jones3@gmail.com
INTRODUCTION
Number of benchmark sources currently: 23
Number of projects in all benchmarks sources: 91,000 (approximately)
Quantitative software benchmark data is valuable for measuring process improvement programs, for calibrating software estimating tools, and for improving software quality levels. It is also useful for studies of industry and company progress over time. This catalog of software benchmark data sources is produced as a public service by Namcook Analytics LLC for the software community. It is not copyrighted and can be freely distributed.
There are many different kinds of benchmarks including productivity and quality levels for specific projects; portfolio benchmarks for large numbers of projects, operational benchmarks for data center performance; security benchmarks, compensation and staffing benchmarks for human resource purposes; and software customer satisfaction benchmarks. SEI software assessment data is also included.
The information in this catalog is provided by the benchmark and assessment groups themselves and includes topics that the benchmark groups wish to be made available. In this version the benchmark groups are listed in alphabetical order.
TABLE OF CONTENTS
- Introduction to Software Benchmark Sources
- 4SUM Partners Inc. (Finland)
- AestimatGmbH
- Capers Jones & Associates LLC – Now Namcook Analytics LLC
- CAST Software
- COSMIC Consortium
- Cyber Security and Information Systems (CSIASSC)
- David Consulting Group (DCG)
- Galorath Incorporated
- German Computer Society Interest Group (GI)
- Industrial Information & Control Systems
- International Software Benchmark Standards Group (ISBSG)
- Jerry Luftman (Stevens Institute of Technology)
- Price Systems LLC
- Process Fusion
- Quantimetrics
- Quantitative Software Management (QSM)
- Q/P Management Group
- RCBS, Inc.
- Reifer Consultants LLC
- Software Benchmarking Organization
- Software Engineering Institute (SEI)
- Software Improvement Group (SIG)
- Test Maturity Model Integrated (TMMI) by Geoff Thompson
Appendix A: Books and web sites with quantitative data
Appendix B: Survey of Software Benchmark Usage and Interest
Appendix C: A New Form of Software Benchmark
INTRODUCTION TO SOFTWARE BENCHMARK SOURCES
The software industry does not have a good reputation for achieving acceptable levels of quality. Neither does the industry have a good reputation for schedule adherence, cost control, or achieving high quality levels.
One reason for these problems is a chronic shortage of solid empirical data about quality, productivity, schedules, costs, and how these results vary based on development methods, tools, and programming languages.
A number of companies, non-profit groups, and universities are attempting to collect quantitative benchmark data and make it available to clients or through publication. This catalog of benchmark sources has been created to alert software engineers, software managers, and executives to the kinds of benchmark data that is currently available.
The information in this catalog is provided by the benchmark groups themselves, and shows what they wish to make available to clients.
This catalog is not copyrighted and can be distributed or reproduced at will. If any organization that creates benchmark data would like to be included, please write a description of your benchmark data using a format similar to the formats already in the catalog. Please submit new benchmark information (or changes to current information) to Namcook Analytics LLC via email. The email address is Capers.Jones3@gmail.com. The catalog can also be downloaded from several web sites including www.Namcook.com which is the editor’s web site.
The catalog is expected to grow as new sources of benchmark data provide inputs. Benchmark organizations from every country and every industry are invited to provide information about their benchmark data and services.
4SUM Partners Inc
Web site URL: www.4sumpartners.com
Email: pekka.forselius@4sumpartners.com
Sources of data: The 4SUM Partners database contains high quality data from the completed projects of clients. Long-range client projects’ data has been collected mainly in project benchmarking studies and scope management assignments. The data have been validated and represent actual project completions. Anonymity of source organizations is protected.
Data metrics: Project data metrics is based on functional size of applications in FiSMA function points.
Analogy data consists of project delivery rate categorized by project classifiers. Project situation analysis data express project’s productivity factors.
Data usage: Data is used in making first estimate of project delivery rate (hours/fp) for project planning and especially estimating project cost and duration according to the northernSCOPE concept enhancing the scope management of software projects.
Data is also used in software project performance benchmarking studies. These comparison studies enable positioning productivity of project delivery in the terms of operational efficiency. They also help to identify and prioritize improvement opportunities in software project delivery and maintaining a culture of continuous improvement.
Data availability: Data is provided to clients of benchmark studies and users of Experience Pro size measurement software.
Kinds of data: Productivity data is collected from all kind of software projects. Several project and application classifiers are used to improve comparativeness of projects.
Functional size of projects ranges from 20 fp to several thousands of function points. Average size of local data is about 500 fp.
Project data represents several business areas and development tools.
Actual effort and cost data in Experience database is activity-based supporting costing of software development life-cycle from specifications to preparing to install.
Quality of new project data is categorized when storing data to the master repository.
Volume of data: Over 1000 projects. Majority of the data represents new development tailored projects. There is also data on enhancements and maintenance.
Industry data: Banking, insurance, public administration, manufacturing, telecom, wholesale & retail and other industries to some extent.
Methodology data: Project classifiers include case development tools, development model, project management tools and several various techniques and methods.
Language data: Data includes COBOL, PL/I, Java, C++, Visual Basic, COOL:Gen, Oracle, C, SQL and other languages to some extent.
Country data: Data is mainly from Finland.
Future data: Number of data is growing. Increase in 2011 was about 100 projects, and data volume is expected to increase by 300 projects in 2012.
Summary: High quality data in the 4SUM Partners database have been validated and represent actual project completions. Metrics of the data is expressed in terms of function point measured using standard measurement methods (IFPUG, FiSMA).
Data is highly useful in productivity analysis and software project performance benchmarking studies. Data supports software projects’ scope management in estimating cost and duration.
Volume of data is growing powerfully due to large-scale productivity studies expected to be carried out in 2012 and 2014.
aestimat GmbH
Web site URL: www.aestimat.de
Email: eva.schielein@aestimat.de
Sources of data: aestimat provides consulting services throughout the benchmarking process.
We help our clients to derive KPI’s from their company strategy and to collect the relevant project data.
As independent benchmarking advisors, we then help to find the most appropriate benchmarking repository for our clients’ needs. We also help our clients to understand benchmark assessment results and to find improvement strategies.
Data metrics: To size projects, we use function points (IFPUG standard).
Productivity = person hours (days) per fp
Time to market = hours (days) per fp
Quality = productive bugs per fp (classified)
Maintenance and Support is expressed in
Frequency of Changes = Sum of changes (classified) per fp of the application baseline (within one year)
Frequency of Bugfixing = Sum of bugs (classified) per fp of the application baseline (within one year)
All data collection activities are quality verified by aestimat measurement specialists. Sizing is performed by IFPUG Certified Function Point Specialists (CFPS).
Kinds of data: Data is collected and provided mostly for large German enterprises. Sizes range from 50 to 1,000 and more function points.
Data can be collected for individual software projects, but also for projects across companies.
In the last years, more and more data from agile projects have been assessed.
Volume of data: Depending on benchmark provider.
Industry data: Depending on benchmark provider.
Language data: Depending on benchmark provider.
Country data: Depending on benchmark provider.
Capers Jones & Associates LLC
(Data transferred to Namcook Analytics LLC in September 2012)
Namcook Analytics LLC
Web site URL: www.Namcook.com
Email: Capers.Jones3@gmail.com
Sources of data: Primarily on-site interviews of software projects. Much of the data is collected under non-disclosure agreements. Some self-reported data is included from Capers Jones studies while working at IBM and ITT corporations. Additional self-reported data from clients taught by Capers Jones and permitted to use assessment and benchmark questionnaires.
Data metrics: Productivity data is expressed in terms of function point metrics as defined by the International Function Point User’s Group (IFPUG). Quality data is expressed in terms of defects per function point.
Also collected is data on defect potentials, defect removal efficiency, delivered defects, and customer defect reports at 90 day and 12 month intervals.
Long-range data over a period of years is collected from a small group of clients to study total cost of ownership (TCO) and cost of quality (COQ). Internal data from IBM also used for long-range studies due to author’s 12 year period at IBM..
At the request of specific clients some data is converted into COSMIC function points, use-case points, story points, or other metrics.
Data usage: Data is used to create software estimating tools and predictive models for risk analysis. Data is also published in a number of books including The Economics of Software Quality, Software Engineering Best Practices, Applied Software Measurement, Estimating Software Costs and 12 others. Data has also been published in about 200 journal articles and monographs.
Data is provided to specific clients of assessment, baseline, and benchmark studies. These studies compare clients against similar companies in the same industry.
Data from Capers Jones is frequently cited in software litigation for breach of contract lawsuits or for suits alleging poor quality.
Some data is also used in tax litigation dealing with the value of software assets.
Data availability: Data is provided to clients of assessment and benchmark studies.
General data is published in books and journal articles.
Samples of data and some reports are available upon request.
Some data and reports are made available through the library,Webinars, and seminars offered by the Information Technology Metrics and Productivity Institute (ITMPI.org).
Kinds of data: Software productivity levels and software quality levels for projects ranging from 10 to 200,000 function points. Data is primarily for individual software projects, but some portfolio data is also collected. Data also supports activity-based costing down to the level of 40 activities for development and 25 activities for maintenance. Agile data is collected for individual sprints. Unlike most Agile data collections function points are used for both productivity and quality.
Some data comes from commissioned studies such as an Air Force contract to evaluate the effectiveness of the CMMI and from an AT&T study to identify occupations employed within large software labs and development groups.
Volume of data: About 15,600 projects from 1978 through today.
New data is added monthly. Old data is retained, which allows long-range studies at 5 and 10-year intervals. New data is received at between 5 and 10 projects per month from client interviews.
Industry data: Data from systems and embedded software, military software, commercial software, IT projects, civilian government projects, and outsourced projects.
Industries include banking, insurance, manufacturing, telecommunications, medical equipment, aerospace, defense, and government at both state and national levels.
Data is collected primarily from large organizations with more than 500 software personnel. Little data from small companies due to the fact that data collection is on-site and fee based.
Little or no data from the computer game industry or the entertainment industry. Little data from open-source organizations.
Methodology data: Data is collected for a variety of methodologies including Agile, waterfall, Rational Unified Process (RUP), Team Software Process, (TSP), Extreme Programming (XP), and hybrid methods that combine features of several methods.
Some data is collected on the impact of Six Sigma, Quality Function Deployment (QFD), formal inspections, Joint Application Design (JAD), static analysis, and 40 kinds of testing.
Data is also collected for the five levels of the Capability Maturity Model Integrated (CMMI™) of the Software Engineering Institute.
Language data: As is usual with large collections of data a variety of programming languages are included. The number of languages per application ranges from 1 to 15, with an Average of about 2.5. Most common combinations
Include COBOL and SQL and Java and HTML.
Specific languages include Ada. Algol, APL,ASP Net, BLISS,
C, C++, C#, CHILL. CORAL, Jovial, PL/I and many
Derivatives, Objective-C. Jovial, and Visual Basic..
More than 175 languages out of a world total of 2,500 are included.
Country data: About 90% of the data is from the U.S. Substantial data
From Japan, United Kingdom, Germany, France, Norway, Denmark, Belgium, and other major European countries.
Some data from Australia, South Korea, Thailand, Spain, and Malaysia.
Little or no data from Russia, South America, Central America,
China, India, South East Asia, or the Middle East.
Unique data: Due to special studies Capers Jones data includes information on more than 90 software occupation groups and more than 100 kinds of documents produced for large software projects. Also,
the data supports activity-based cost studies down to the levels of 40 development activities and 25 maintenance tasks. Also included are data on the defect removal efficiency levels of 65 kinds of inspection. static analysis, and test stages.
Some of the test data on unit testing and desk checking came from volunteers who agreed to record information that is normally invisible and unreported. When working as a programmer Capers Jones was such a volunteer.
From longitudinal studies during development and after release the Jones data also shows the rate at which software requirements grow and change during development and after release. Monthly change rates exceed 1% per calendar month during development and more than 8% per year after release.
From working as an expert witness in 15 lawsuits, some special data is available on litigation costs for plaintiffs and defendants. From on-site data collection and carrying out interviews With project teams and then comparing the results to Corporate resource tracking systems, it has been noted that “leakage” or missing data is endemic and approximates 50% of actual software effort. Unpaid overtime and the work of managers and part-time specialists are most common.
Quality data also leaks and omits more than 70% of internal defects. Most common omissions are those of desk checking, unit testing, static analysis, and all defect removal activities prior to release.
Leakage from both productivity and quality data bases inside Corporations makes it difficult to calibrate estimating tools and also causes alarm to higher executives when the gaps are revealed. The best solution for leakage is activity-based cost collection.
Future data: There are several critical areas which lack good sources of quantitative data. These include studies of data quality, studies of intangible value, and studies of multi-national projects with geographically distributed development locations.
Summary: Capers Jones has been collecting software data since working for IBM in 1978. In 1984 he founded Software Productivity Research and continued to collect data via SPR until 2000.
Capers Jones & Associates LLC was formed in 2001. He owns several proprietary data collection questionnaires that include both qualitative assessment information and quantitative data on productivity and quality. The majority of data comes from on-site interviews with software project teams but self-reported data is also included, especially from clients who have been trained and authorized to use the Jones questionnaires.
More recently remote data collection has been carried out via Skype and telephone conference calls using shorter forms of the data collection questionnaires.
Some self-reported or client-reported benchmark data is included from companies taught by Capers Jones and from consortium members.
Some self-reported data is also included from internal studies carried out while at IBM and ITT, and also from clients such as AT&T, Siemens, NASA, the Navy, and the like.
CAST Software
Web site URL: http://www.castsoftware.com/Product/Appmarq.aspx
Email: info@castsoftware.com
Sources of data: Appmarq is a repository of structural quality data for custom software applications in business IT. Data is collected via automated analyses with the CAST Application Intelligence Platform (AIP), which performs a thorough structural quality analysis at the code and whole-application level. Metrics from the application-level database are fed into the central Appmarq repository. All data is made anonymous and normalized before entering the central benchmarking database.
The AIP data are combined with application “demographics,” which are the qualitative application characteristics such as age, business function, industry, and sourcing paradigm. These demographics are collected directly from the customer via survey instrument and provide a means to identify peer applications when benchmarking.
Data metrics: The data represents software structural quality metrics, which, at their highest level, include:
- Business risk exposure (performance, security, robustness)
- Cost efficiency (transferability, changeability, maintainability)
- Methodology maturity (architecture, documentation, programming standards)
- Application Size (KLOC, backfired function points)
- Application Complexity (cyclomatic complexity, SQL complexity)
- Rule level details (specific rules being violated)
- Demographics (industry, functional domain, extent of in-house/outsource, extent of onshore/offshore, age of application, number of releases, methodology and certifications)
Data usage: The data collected is used for research on trends in the structural quality of business applications, as well as best practices and standards development. A detailed research paper with industry-relevant findings is published each year. Some initial discussions are starting with academia to use Appmarq data for scholarly research.
Data is also provided to specific clients in the form of customized reports, which from a measurement baseline, benchmarks the structural quality of their applications against those of their industry peers and of the same technology.
Data availability: Data is provided to clients of assessment and benchmark studies. General data is published in a yearly report. A summary of key findings is made available and distributed across a large number of organizations.
Volume of data: Data has been collected over a period of 4 years. The dataset currently stands at more than 800 distinct applications. Data is continually added to the dataset, as new benchmarks are conducted and/or data automatically extracted from the CAST AIP repository.
Industry data: Data is collected primarily from large IT-intensive companies in both private and public sectors. Industries include Finance, Insurance, Telecommunications, Manufacturing, Transportation, Retail, Utilities, Pharmaceuticals and Public Administration.
Methodology data: Data is collected from a variety of methodologies including agile/iterative, waterfall and hybrid methods that combine several methods. Data is also collected for the five levels of the Capability Maturity Model Integrated (CMMI™) of the Software Engineering Institute (SEI).
Technology data: Data is collected from a wide range of technologies, including Java EE, .NET, C/C++, SAP, Oracle, Mainframe technologies and other technologies commonly found in IT.
Country data: A quarter of the data comes from North America, the remainder is distributed across UK, France, Belgium, Italy, Germany, Spain, India and several other countries.
Summary: Starting in 2007, CAST has been collecting metrics and structural characteristics from custom applications deployed by large IT-focused enterprises across North America, Europe and India. Appmarq is a structural quality benchmarking repository with the purpose of analyzing the trends in the structural quality of business applications regarding characteristics such as robustness, performance, security, and changeability.
Unlike other benchmarking datasets, Appmarq is built on automated analysis conducted and normalized by machine; hence the data are an apples-to-apples comparison across different environments. Also unique is that Appmarq puts a singular focus on the characteristics of the software product, rather than the process, project and performance level metrics that feed into the actual product.
COSMIC (the Common Software Measurement International Consortium)
Web site URL: www.cosmicon.com
Email: harold.van.heeringen@sogeti.nl; luca.santillo@gmail.com;
Sources of data: Voluntary submission of software project data. All COSMIC-measured project data should be submitted to the ISBSG repository (see the ISBSG entry in this list of benchmark sources). – www.isbsg.org – which provides a service to all COSMIC method users. The data is collected under an agreement of anonymity.
Data metrics: Productivity, speed of delivery and quality data where COSMIC-measured software sizes are used as the measure of project work-output.
Data usage: Data has been used to develop COSMIC method benchmarks and has been analyzed in many ways to enable performance comparisons and for project estimating. A report entitled ‘The Performance of Real-Time, Business Application and Component Software Projects’ is available from www.isbsg.org. A version showing the report contents, but not the actual numbers, is available for free download from www.cosmicon.com.
Data availability: The ISBSG data is ‘open’ and can be licensed by anyone in its ‘raw’ form at a moderate cost.
Data is provided to academics for use in approved research work.
Kinds of data: Data on both Development and Enhancement projects, for business application, real-time and component software projects.
Projects range in size from 8 to 1,800 COSMIC Function Points.
Volume of data: Most data is for projects completed within the last five years. The number of projects is expected to pass 500 in the course of 2011.
Industry data: Business application software project data comes from Banking, Government and Public Administration, Insurance, Engineering, Medical & Healthcare, Retail & Wholesale industries.
Real-time software project data comes from Automotive, Telecommunications, Computer and Software industries.
Methodology data: See the entry for the ISBSG in this list of benchmarking services
Country data: See the entry for the ISBSG in this list of benchmarking services. The country of origin of any project data is not revealed.
Summary: The COSMIC Functional Size Measurement method is the only such method developed by an international group of software metrics experts, designed to conform to fundamental software engineering principles, and to be applicable to business application, real-time and infrastructure software. It can be used in any layer of a software architecture and at any level of decomposition. Use of the COSMIC method for performance measurement, benchmarking and estimating is therefore expected to enable much higher degrees of precision than when using older software size measures. For more, see www.cosmicon.com .
Software & Systems Cost and Performance Analysis Toolkit (S2CPAT)
Cyber Security and Information Systems Information Analysis Center (CSISIAC)
Web site URL:https://s2cpat.thecsiac.com/s2cpat/
Email: s2cpat@thecsiac.com
Sources of data: The initial release of S2CPAT contains Software Resources Data Report (SRDR) data provided by the Air Force Cost Analysis Agency. [http://dcarc.cape.osd.mil]. This data has been sanitized for public release by the US Department of Defense (DoD) and validated by a DoD-funded academic research team.
Data metrics:
CMMI Level
Operating Environment
Development Process/Life Cycle
Primary Language
Size vs. Effort
Size vs. Schedule
Effort vs. Schedule
Productivity (ESLOC/Hour)
Size (ESLOC)
Effort (Hours)
Duration (Months)
Peak Staff (Heads)
SW Effort Distribution:
Requirements Analysis (%)
Architecture and Detailed Design (%)
Coding and Unit Testing (%)
SW INT and System/Software INT (%)
Qualification Testing (%)
Developmental Test and Evaluation (%)
Program Management, SQA, SCM (%)
Staff Experience Distribution:
High Personel Experience (%)
Normal Personel Experience (%)
Entry Personel Experience (%)
Volatility (Rating 1-5)
Data usage: The goal of Software & Systems Cost and Performance Analysis Toolkit (S2CPAT) is to capture and analyze software and software engineering data from completed software projects that can be used to improve a) the quality of software –intensive systems and b) the ability to predict the development of software –intensive systems with respect to effort and schedule.
The toolkit allows users to search for similar software projects and use the data to support:
1. Rough order of magnitude estimates for software development effort and schedule
2. Project planning and management: life cycle model information, key risks, lessons learned, templates, estimation heuristics
3. Software engineering research.
Data availability: This data has been sanitized for public release by the US Department of Defense (DoD).
Kinds of data: Data may be queried by Size, Application Domain, Language, and/or Operating Environment to generate bar charts, pie charts, or scatter charts.
Volume of data:
513 testing data points
Industry data: U.S. Defense Department-funded contractors
Language data: Ada, BASIC, C, C++, C#, Java, Jovial, Pascal, PL/SQL, SQL, Visual Basic, and XML.
Country data: All from U.S. Defense Department projects.
Future data: Efforts are underway at the University of Southern California (USC) Center for Systems and Software Engineering (CSSE) to obtain permission from their affiliates to migrate the academic COCOMO II calibration data into S2CPAT. Efforts are also underway to include additional DoD SRDR data in the toolkit as it is approved for public release.
Summary: A User Manual is available at https://s2cpat.thecsiac.com/s2cpat/docs/S2CPAT%20User%20Manual.pdf
David Consulting Group (DCG)
Web site URL: www.davidconsultinggroup.com
Email: info@davidconsultinggroup.com
Sources of data: As part of many of DCG’s software measurement and software sizing assignments, we collect non-proprietary data for project size, effort, duration, resources, defects and cost. All data collection activities are quality verified by DCG Measurement Specialists. All DCG counts are performed by IFPUG Certified Function Point Specialists (CFPS).
Data metrics: All collected data is stored in an internal, proprietary, confidential database. All data is expressed (normalized) in terms of function point metrics as defined by the International Function Point User’s Group (IFPUG).
DCG Measurement Specialists extract selected Data Views of four key metrics:
- Size (Function Points)
- Quality (Defects)
- Duration (Project Months)
- Productivity (Development Hours per Function Point)
For each metric view, DCG provides an equation (formula) based upon a statistical analysis and the number of data points represented across the dimensions and attributes of the benchmark.
Data usage: The primary use of the DCG Industry Database is to support our benchmarking consulting engagements. The Database allows us to work with customers to analyze the data to give them realistic comparisons with their own internal performance.
Data availability: DCG data is not available for sale in its raw state. Aggregate DCG Data Views are available for sale as part of a client benchmarking consulting engagements.
Kinds of data: DCG collects size, effort, duration and quality data. At least two of these attributes are available for all projects in the database. Sizing data is expressed in IFPUG function points.
Volume of data: DCG’s repository contains several thousand projects. Most client benchmarks use a subset of all available projects because of the specificity of the comparison requested. The nature of the use of our data requires DCG to prioritize the quality of the data and sources of the data ahead of the volume of data.
Industry data: Data is collected from DCG records of software development or application maintenance project activity. Projects are typically performed for the software development or application maintenance divisions of major corporations and for software product companies of all sizes.
Industries include telecommunication; insurance; finance; manufacturing; technology; Government; and public sector.
Methodology data: Data is attributed along four key dimensions with sub-attributes within each dimension. The dimensions include Technology, Platform, SDLC and Application Development Type (e.g.,new development or enhancement).
Country data: A majority of the data comes from North American companies with the remainder, in order, from Europe, India and Asia. However, country origin data is not stored in the database.
Summary: The primary use of the DCG Industry Database is to support our benchmarking consulting engagements. The Database allows us to work with customers to analyze the data to give them realistic comparisons with their own internal performance.
The nature of the use of our data requires DCG to prioritize the quality of the data and sources of the data ahead of the volume of data.
Galorath Incorporated
Web site URL: www.galorath.com
Email: info@galorath.com
Sources of data: Repositories obtained through product customers, industry sources, and public domain and consulting. Galorath also maintains a partnership with ISBSG and offers the ISBSG data in SEER Historical Database (SEER-HD) format to subscribers.
Data metrics: Productivity expressed in function points and source lines of code. Most data has language, platform and application descriptors. Some of the data also contains, dates, defects delivered, time phased staffing, documentation pages, detailed SEER parameter settings.
Data usage: Data is used to calibrate SEER knowledge bases, determine language factors, generate trend lines, benchmark organizations, and validate model relationships.
Data availability: Data is delivered in the form of knowledge bases, language factors, trend lines and estimating relationships to SEER customers. Where permitted data is provided in source form so SEER users can see their estimates compared with individual data points. SEER customers who submit data to Galorath receive their data in the SEER Historical Database format that can be used directly by the application.
Kinds of data: Records of completed projects at the program and project level. Size ranges from extremely small (less than one month duration) to very large (hundreds of staff, many years’ duration.) Additionally metadata is available from some sources, showing items such as productivity ranges, size ranges, etc. where management is not willing to release raw data.
Volume of data: Over 20,000 records with nearly 13,000 total records containing at least size, in addition to other metrics of which approximately 8500 records contain completed effort and a lesser number, project duration. Volume increases regularly.
Industry data: Data from IT/business systems, industrial systems and embedded software, military, and commercial enterprises.
Methodology data: Project processes in collected data range from traditional waterfall through incremental and agile development.
Country data: Military and aerospace data is US and European. Commercial data is worldwide.
Summary: Galorath maintains an active data collection program and has done so for the last 20 years. It makes data available to users of SEER for Software in several forms such as its ProjectMiner data mining tool and Metrics and Benchmarking data visualization tool.
GI (German Computer Society Interest Group) Software Measurement
Web site URL: http://metrics.cs.uni-magdeburg.de/
Email: Christof Ebert (Vector), christof.ebert@vector.com
Cornelius Wille (FH Bingen), wille@fh-bingen.de
Reiner Dumke (University Magedeburg), dumke@ivs.cs.uni-magdeburg.de
Jens Heidrich (Fraunhofer IESE), jens.heidrich@iese.fraunhofer.de
Sources of data: Voluntary submission of software project data. Benchmarking data is collected by the board of the German Computer Society Software Measurement Interest Group. The data is collected under an agreement of anonymity.
Data metrics: Measurements refer to project an industry benchmarks, capturing size, complexity, productivity, effort, defect rates, duration and deviations from commitments
Data usage: Data has been used to develop a repository for scientific studies. A report entitled “Practical Software Measurement” is available from the board of the German Computer Society Software Measurement Interest Group.
Data availability: The data is closed but can be obtained in aggregated format through above mentioned source. Data is provided to academics for use in approved research work.
Kinds of data: Data on both Development and Service projects, for business application, real-time and component software projects.
Volume of data: Most data is for projects completed within the past ten years.
Industry data: Business application software project data comes from Automotive, Aviation, Finance, IT, Medical & Healthcare, and Transport industries.
Methodology data: We started with looking into our own project lessons learned and enriched it with experiences from books and conference proceedings, cost estimation tools. Today many projects are entered online (see below). Over time, this lead us to some simple rules of thumb (heuristics) that we could use even in situations where no historic information was accessible.
Country data: The country of origin of any project data is not revealed.
Summary: The GI (German Computer Society Interest Group) Software Measurement maintains the Software Measurement Laboratory (SML@b) at University of Magdeburg which is a prototype of a software measurement database in the Internet. It allows Java based interactive entry of measurement data of popular CAME tools such as Logiscope, Datrix or OOM and delivers respective reports. More details are at: http://metrics.cs.uni-magdeburg.de/
Industrial Information & Control Systems
Web site URL: www.ics.kth.se
Email: robertl@ics.kth.se
Sources of data: The data was obtained through case studies, surveys, and interviews. The case studies were carried through at major Nordic companies. The surveys with experts in the field were either done in workshops or online. All project manager interviews were done face-to-face.
Data metrics: Change project cost measured in man-hours, system coupling, system size (LOC), system complexity, change management process maturity (as defined in COBIT), as well as different subjective quality measures such as quality of tools or documentation.
Project cost, project productivity (no of function points per work hour), function points, project type, project priority, quality of delivery, as well as data on e.g. project participants, pre-studies, budgets, platforms, et cetera.
Project success in terms of time, budget, and quality. Project success factors such as goals and objectives, estimations, size and complexity, communication, technology maturity, risk analysis, system requirements, user involvement et cetera.
Data usage: Data is used in order to estimate project cost, productivity, and success. Most of the data is employed together with enterprise architecture models or using Bayesian networks.
Data availability: Data is available in our research papers, published in journals or conference proceedings. The data is often anonymous in order to protect the privacy of our partners.
Volume of data: Depending on research study, but in the size range of 70 projects, 100 survey respondents, and 30 interviews.
Industry data: Data has been collected in all types of industries, ranging from the power industry and banking, to consultancy and the public sector.
Country data: Mainly Swedish companies, but also some from other Nordic countries.
Summary: Industrial Information & Control Systems (ICS) is a department at the Royal Institute of Technology (KTH) in Stockholm, Sweden. ICS have been collecting software related data for many years and are currently focusing their research initiatives towards data for the prediction of non-functional qualities such as modifiability, interoperability, security, availability, performance, and cost. One main approach employed by ICS is the use of enterprise architecture models in the prediction of these qualities. These models need to be based on data from software projects.
References: Robert Lagerström, Pontus Johnson and David Höök, Architecture analysis of enterprise systems modifiability – Models, analysis, and validation, in the Journal of Systems and Software, vol. 83, no, 8, p. 1387-1403, 2010.
Liv Marcks von Würtemberg, Ulrik Franke, Robert Lagerström, Evelina Ericsson and Joakim Lilliesköld, IT project success factors – An experience report, in the Proc. of Portland International Center for Management of Engineering and Technology (PICMET) conference, 2011.
Robert Lagerström, Liv Marcks von Würtemberg, Hannes Holm and Oscar Luczak, Identifying factors affecting software development cost and productivity, in the Software Quality Journal, vol. 20, no. 2, p. 395-417, 2012.
ISBSG Limited
Web site URL: www.isbsg.org
Email: peter.r.hill@isbsg.org
NOTE: Visit the new ISBSG portal facility http://portal.isbsg.org This facility allow you to search and filter the ISBSG data to select specific kinds of benchmark information.
Sources of data: Voluntary submission of software project data. The data is collected under an agreement of anonymity. Data is submitted either directly, via ISBSG members, or by consultants with the permission of their customers.
Data metrics: Productivity data is expressed in terms of function point metrics as defined by the International Function Point User’s Group (IFPUG); NESMA; COSMIC; and FiSMA. Quality data is expressed in terms of defects per function point or functional unit.
LOC sizes are stored if provided but not validated and not used for any analysis. Likewise for other sizing methods, e.g. Feature Points, Use Case Points etc.
Also collected is data on the report of defect occurrence during the project and within the first 30 days of operation.
Data has come from more 25 countries and 20 major organization types
Data usage: Data is used with some basic ISBSG software estimating tools; used by commercial estimation tool vendors; and for benchmarking services. The data is also analyzed and published in a number of analysis reports and books including Practical Software Project Estimation.
Data is provided to academics for use in approved research work.
Data availability: The ISBSG data is ‘open’ and can be licensed by anyone in its ‘raw’ form at a moderate cost. Web portal access to the data will be available from January 2012.
General data and analysis results are published in books, analysis reports and journal articles. Samples of data and some reports are available upon request.
Kinds of data: Data on both Development and Enhancement projects.
Projects ranging from 10 to 10,000 function points in size, but with an average size around 300FP.
Data is primarily for individual software projects, but some portfolio data is also collected.
Volume of data: About 6,000 projects from 1989 through today. New data is added monthly. Old data is retained, allowing studies over time.
Industry data: Data from MIS commercial software; real-time systems; IT projects; government projects; outsourced projects; and Off-shore projects.
Industries include banking; Insurance; Legal; Manufacturing; Telecommunications; Accounting; Sales; Transport; Government; and Public Sector.
Data is submitted from a wide range of organizations of differing sizes.
Methodology data: Data is collected for a variety of methodologies including Agile, Waterfall, Joint Application Design (JAD), Rational Unified Process (RUP), Team Software Process, (TSP), and hybrid methods that combine features of several methods.
Some data is collected on the impact of being compliant with CMM and CMMI and relevant ISO standards.
Country data: About 30% of the data is from the U.S. 16% from Japan; 16% Australia; 10% Finland; 8% Netherlands; 6% India; 5% Canada; 2% China. Also data from 12 other countries.
Summary: The ISBSG’s formation in 1997 was built upon several years of previous cooperation by a group of national software metrics associations with the common aim of developing and promoting the use of IT industry history data to improve software processes and products, for the benefit of both businesses and governments worldwide.
Current members of the ISBSG represent IT and Metrics Associations based in the following countries: Australia, China, Finland, Germany, India, Italy, Japan, Korea, Netherlands, Spain, Switzerland, UK, USA.
Two independent repositories of IT industry data:
- Software Development and Enhancement – over 5,700 projects
- Software Maintenance and Support – ~500 applications
Jerry Luftman Ph.D. Stevens Institute of Technology; & Luftman LLC
Web site URL: http://howe.stevens.edu/index.php?id=14
Email: luftman@hotmail.com
Analytic: IT Business Alignment Maturity
Sources of data: The repository is comprised of data from over one-third of the Global 1,000 companies; albeit the assessment has been used by companies of all sizes. The data is captured primarily via on-site interviews of IT and business executives; in cases where a larger sample size is requested, online survey is available. The data is collected under non-disclosure agreements. While most of the data has been collected via Luftman, consulting companies and advanced graduate students from around the globe have participated; hence permission to use the assessment and benchmark is available.
A similar maturity assessment is also available to benchmark the alignment maturity of clients and service providers.
Data metrics: The maturity of the relationship of IT and business organizations is based on an assessment of 6 components: Communications, Value Analytics, Governance, Partnership, Infrastructure, and Human Resources. A 5 point Likert scale (much like CMMI) is applied to the different elements comprising the 6 components. While an overall alignment maturity score is derived, each of the components and elements also have scores that are benchmarked.
Data usage: Data is used to provide not just a description of where an organization is strong/weak/disagreement, but the model provides a prescriptive roadmap to identify specific initiatives to help improve the relationship among the participating organizations.
A report is provided that describes the identified strengths and opportunities for improvement, specific recommendations for addressing them, and a benchmark comparing the company to things like how they compare with other companies from the same industry and geography, as well as comparing how their C level executives compare to their peers.
Data from Luftman’s research is frequently cited in academic and business publications.
Data availability: Data is provided to clients of assessment and benchmark studies.
General data is published in books and journal articles.
Samples of data and some reports are available upon request.
Kinds of data: Data pertaining to the respective elements under each of the six components is extremely insightful. Identifying the relationship of alignment maturity and firm performance, as well as identifying appropriate IT organizational structure and IT reporting is fundamental.
Volume of data: With over one-third of the global 1,000 companies reported in the repository, the opportunities for analysis are endless.
Industry data: Data has been collected from all industries from around the globe. Hence, the opportunity to compare industries from different geographies becomes another value analytic.
Methodology data: The alignment assessment focuses on a “higher level” of relationship across an organization; hence, while methods applied for governance, value, and service levels are collected, the focus is on identifying opportunities for improving the relationship among IT and business organizations.
Language data: As indicated, the assessment is at a higher level than programming language; albeit the tool has been translated in to Spanish. Portuguese, Chinese…
Country data: The geographic spread of the data is: 50% – U.S.; 18% – Europe; 12% – Latin America; 12% – Asia; 8% – Australia.
Unique data: Of the over one-third of the Global 1,000 organizations that are included in the benchmarking repository, the participant spread is 1,200+ – IT executives; 1,330+ – Business executives.
Important insights include appropriate IT reporting structure, IT organizational structure, the need to assess ALL six of the components, and the strong correlation of IT business alignment and firm performance across industry.
Future data: Data continues to be collect and benchmarked from around the globe. Different industry benchmarks continue to provide important lessons. Additionally, performing subsequent assessments after several years has indicated continuous improvement. Additionally, over time, while maturity scores are improving, the overall average is still in the low 3’s; hence there are still opportunities for improvement.
Summary: Jerry Luftman has been collecting alignment maturity data since 2001. Jerry Luftman’s career includes strategic positions in management (Information Technology and consulting), management consulting, Information Systems, and education. Dr. Luftman’s experience combines the strengths of practitioner, consultant, and academic. His proficiency in business-IT alignment, eighteen books, published research, consulting, mentoring, and teaching/speaking engagements further exemplify Dr. Luftman’s expertise and leadership in his field.
After a notable twenty-two year career with IBM, he is currently a Distinguished Professor at Stevens Institute of Technology School of Management. Besides the traditional responsibilities of a Professor, teaching Masters and Ph.D. students, Dr. Luftman is the founder and Executive Director of the Stevens’ Information Systems programs; one of the largest in the world. In this capacity, Dr. Luftman manages all aspects of the program including course building, research, marketing, administrative processes, and executive programs. Dr. Luftman’s project experience ranges from senior management issues through tactical implementation. His current initiatives include creating business and information technology strategies and plans, facilitating IT governance processes, and developing and deploying applications leveraging emerging technologies. Dr. Luftman most recently pioneered the vehicle for assessing the maturity of IT-business alignment; where he has a repository of over one-third of the Global 1,000 companies. He also serves on the executive board of several companies, organizations, and publications.
Dr. Luftman’s last responsibility at IBM was a key speaker at their Customer Executive Conference Center in Palisades, N.Y. While responsible for management research, he played a significant role in defining and introducing the IT Strategy and Planning, and Business Process Reengineering practice methods for the IBM Management Consulting Group. His framework for applying the strategic alignment model is fundamental in helping clients understand, define, and scope the appropriate strategic planning approach to pursue. Dr. Luftman’s annual international IT trends survey sponsored by SIM, CIOnet, and other CIO associations is recognized internationally as an industry barometer.
PRICE Systems, LLC
www.pricesystems.com
Email: arlene.minkiewicz@pricesystems.com
Sources of data: Data comes from many sources. Proprietary data is collected from clients through consulting engagements, research engagements and training. Publically available data is purchased from external sources. Public data is also mined from sources published technical reports and articles as well as reports in the media.
Data metrics: Data collected include software size (in Source Lines of Code (SLOC), Function Points or Use Cases), project effort, project cost, various technical parameters about the nature of the product being developed, the development team, and the developing organization.
Schedule duration for projects is collected as well.
Data usage: Data is used to for creation, verification and validation of estimating relationships for cost, effort and schedule in the TruePlanning for Software product. Data is used to update knowledge bases which provide user guidance through various means within the product.
Data collected is also used to support research efforts, internally and externally funded, focused on specific areas of software development, technology, or process.
Data availability: Raw data is generally unavailable due to its proprietary nature, summary data is available upon request
Kinds of data: Software project data for projects (or project components) ranging from 1000 – 3,000,000 SLOC and projects (or project components) ranging from 25 – 1,000,000 Function Points. Data is primarily for individual software projects – some data is provided down to the component level, others are higher level project data.
Volume of data: Data collected from more than 3500 projects from 1977 through today. New data is added monthly. Old data is retained.
Industry data: Data from systems and embedded software, military software, real time software, commercial software, IT projects, MIS systems, civilian government projects, and outsourced projects.
Industries include banking, insurance, manufacturing, accounting, transportation, telecommunications, aerospace, defense, and government at both state and national levels.
Data is collected primarily from large organizations or agencies.
Methodology data: Data is collected for a variety of methodologies and development process models.
Language data: Data collected covers a wide range of programming languages.
Country data: About 85% of the data comes from sources in the US. Additional data is provided from mostly European sources.
Unique data: In addition to data that we have collected from our clients and other sources, PRICE also has been collecting data from our own software development efforts over the last 30+ years we have been developing software applications. Having been an agile shop for much of the last decade, this data includes many of the common productivity metrics associated with agile development
Summary: PRICE Systems has been collecting software data since the mid 70’s during the research and development of the PRICE Software model, which has evolved into the TruePlanning for Software model. Data collection continues to support regular upgrades to the software cost, schedule and effort estimating ability in this model.
PRICE Systems is a global leader of cost management solution in Aerospace, Defense, Space and Information Systems, serving over 250 customers worldwide.
Process-Fusion.net
Email: ggack@process-fusion.net
Sources of data: Approximately 75% of this data set comes from formal inspections training event workshops in which participants inspect actual work products prepared prior to the workshop. Approximately 25% of the data are from post-training inspections of work products.
Data metrics: This data set includes approximately 1,600 individual inspections conducted by team of 3 or 4 in most cases. All of these inspections are consistent with IEEE Std. 1028-2008. Data elements collected for each inspection include work product type (requirements, design, code), size in lines, orientation hours, preparation hours, team meeting hours, number of major and minor defects identified, and post-correction review hours.
Data usage: Summaries of these data may be used as a basis for estimating the number of defects likely to be found in the several work product types identified, the number of hours per defect (or defects per hour) likely to be required when using formal inspections consistent with the process defined by IEEE Std. 1028-2008 (a.k.a. “Fagan-style Inspections”). Standard deviation and distribution data are also available.
Data availability: Summary data are available from the author upon request.
Volume of data: Approximately 1,600 inspections.
Industry data: Data from systems and embedded software, military software, commercial software, IT projects and outsourced projects.
Data is collected primarily from large organizations with more than 500 software personnel.
Language data: A variety of programming languages are included.
Country data: About 80% of the data is from the U.S.
QuantiMetrics
Web site URL: www.quantimetrics.net
Email: bram@quantimetrics.net
Sources of data: Organisations (public and private) who subscribe to the software benchmarking services of QuantiMetrics. Data submitted is subject to rigorous consistency, validity and reasonableness checks; measurement practices of subscribers are pre-audited to avoid potential data problems. In many instances QuantiMetrics carry out the size measurements using certified staff; where size measures are provided by subscribers, these are audited by QuantiMetrics.
Data metrics: Project data: size measures; resource and time use by development phase, resource use by activity type; costs; quality measures by quality step; methods, languages/tools and platforms used; project team staffing and experience; qualitative factors deemed to have affected performance. Measures reflecting plans at significant mile-stones, are also collected.
Applications data: size measures; annual resource used by activity type; volumes of changes, change and support requests; costs; application support team experience and ratings on maintainability and other factors; languages/tools and platforms used; application age; application instances/versions; user population; support window
Data usage: Applications project and support performance benchmarking and estimating; at least a third of instances this data has been used in the context of outsourcing contracts
Data availability: Data is deemed to be proprietary – participants are provided with like-for-like benchmarks for all measures they provide for their projects and applications. Anonymity is preserved at all times.
Kinds of data: Project and Application support for business applications
Volume of data: Data accumulated goes back to mid 1980s; early data largely based on line-of-code measures of size, while use of IFPUG functional size measurement dominant from 1990s and onward. The number of projects accumulated in total is about 10,000, yet benchmarks are only based on the most recent 3,000 or so projects, of which a large proportion are post Y2K. Applications measured number about 1,000.
Industry data: Industry participants are largely from the private sector, with financial institutions the most dominant, then telecoms businesses although the applicability of the benchmarks is appropriate for all business type projects.
Methodology data: Project data includes information on approaches and methodology (proprietary or generic type). In the case of Agile-based methods ratings of the degree of adoption of practices also obtained.
Language data: No restriction is placed on the development languages (tools) used; where statistically reliable data is available benchmarks are language specific, otherwise they are based on type of language/tool. Early submissions were dominated by third generation languages, followed by a period when a variety 4th generation tools was the norm, while now object oriented languages and “visual” tools are more the norm. Projects also include those that configure and roll out business applications software.
Country data: Largest and dominant contributions are from UK, Germany, South Africa and India
Unique data: QuantiMetrics use statistical techniques to model and normalize performance, particularly for the effects of: size and time in the case of projects; size, age and user population for applications support data.
Future data: Expect to see more use of COSMIC as a size measure
Summary: QuantiMetrics have worked to ensure the database contains good-quality data and that the most statistically reliable like-for-like benchmarks are provided. Reports and presentations are aimed to identify systemic cause-effect relationships and, in particular, to suggest best practices and opportunities for improvement.
Quantitative Software Management, Inc. (QSM)
Web site URL: http://www.qsm.com
Email: info@qsm.com
Sources of data: Since 1978, QSM has collected completed project data from licensed SLIM-Suite® users and trained QSM consulting staff. Consulting data is also collected by permission during productivity assessment, benchmark, software estimation, project audit, and cost-to-complete engagements. Many projects in our database are subject to non-disclosure agreements but regardless of whether formal agreements are in place, it is our policy to guard the confidentiality and identity of clients who contribute project data. For this reason, QSM releases industry data only in summary form to preclude identification of individual projects/companies or disclosure of sensitive business information.
Data metrics: Our basic metric set focuses on size, time, effort, and defects (SEI Core Metrics) for the Feasibility, Requirements/Design, Code/Test, and Maintenance phases. These core measurements are supplemented by nearly 300 other quantitative and qualitative metrics. Approximately 98% of our projects have time and effort data for the Code and Test phase and 70% have time/effort data for both the R&D and C&T phases.
Productivity is captured via the following metrics:
- QSM Productivity Index (PI)
- Cost per SLOC or Function Point
- SLOC or Function Points per month
- SLOC or Function Points per Effort Unit (Months, Hours, Days, Weeks, Years)
Quality data is captured via the following metrics:
- Total Errors (System Integration test – Delivery)
- MTTD
- Errors/Month
- Errors/Effort Unit (various)
- Errors/KESLOC or Function Point
- Post release defects at 1, 3, 6, 9, 12, and 18 months
Data usage: Data from our database is used to:
- Keep our tools and algorithms in sync with the latest industry practices and technologies.
- Create the nearly 600 industry trend lines included with SLIM-Suite applications.
- Support our basic measurement, benchmarking, estimate validation and verification and other consulting services
- Conduct ongoing research available via books, papers, articles and blog posts available via our web site.
Data availability: To protect our clients, we release project data in summary form only. QSM consulting and tools clients may receive graphs and summaries that allow them to compare their projects against industry trend lines and actual projects that are similar in size, application type, and complexity. Summary statistics and information about the Putnam Model are also available via books like Five Core Metrics, Measures for Excellence and Industrial Strength Software, webinars, QSM IT Almanacs, blog posts, technical reports and whitepapers, conference presentations, and journal articles.
Kinds of data: Individual project duration, effort, size, productivity, and quality data is available for projects ranging from 2-57,000 function points and 100 – over 10 million SLOC. Our core metric set is supported by nearly 300 other quantitative and qualitative measures.
Volume of data: About 10,000 projects from 1978 through the present. To facilitate longitudinal studies of the industry, older project data is retained.
Industry data: QSM data is stratified into 9 major application domains (Avionics, IT, Command & Control, Microcode, Process Control, Real Time, Scientific, System Software, and Telecom) and 45 sub-domains. Software projects predominate, but we have a growing number of hardware and infrastructure (non-software call center) projects as well.
Data contributors include DoD; civilian commercial firms; and national, state and local government entities. In addition to domain complexity bins, our data is also broken out by major industry and industry sector. Major industries include the financial sector, banking, insurance, manufacturing, telecommunications, systems integration, medical, aerospace, utilities, defense, and government.
Methodology data: The QSM database includes a variety of lifecycle and development methodologies (Incremental, Agile, RUP, Spiral, Waterfall, Object Oriented) and standards (CMM/CMMI, DoD, ISO).
Language data: Approximately 600 languages are represented with most projects recording multiple languages. Common primary languages are JAVA, COBOL, C, C++, C#, VISUAL BASIC, .NET, IEF / COOLGEN, PL/1, ABAP, SQL, ORACLE, POWERBUILDER, SABRETALK, JAVA SCRIPT, DATASTAGE, HTML. Frequently used secondary languages include JCL, SQL, JAVA, COBOL, ASSEMBLER, C++, HTML, VISUAL BASIC, XML, ASP.NET, and JSP.
Country data: QSM has collected and analyzed software projects from North America, Europe, Asia, Australia, and Africa. About 50% of our data is from the U.S. Another 35-40% is from India, Japan, the Netherlands, the United Kingdom, Germany, France, and other major European countries.
Future data: We are currently expanding our data collection to include non-software and hardware projects as well as institutional or portfolio level performance measures.
Summary: As pioneers and innovators in the software estimation field since 1978, Quantitative Software Management is more than just a company name; it’s the philosophy that defines everything we do. Our goal is to help our clients become world-class software developers so they can deliver world-class software projects. For over 30 years we’ve provided superior estimation products and consulting services that enable our clients to estimate, track and benchmark projects with confidence. Our tools and methods can help you negotiate achievable goals, set realistic expectations and communicate more effectively with your colleagues and customers.
At the helm of QSM is Larry Putnam, who founded the company in 1978. Larry possesses more than 30 years of experience as a software management consultant and has worked on more than 400 client engagements. Considered to be among the top problem solvers in the software estimation and measurement field, Larry has authored four books and hundreds of articles. He is sought internationally as a speaker and management consultant.
Headquartered in Washington, DC, QSM also has regional distributors in Massachusetts, England, and the Netherlands. QSM has established itself as the leading total solution provider of choice for software developers in high-performance, mission-critical environments. For more information about QSM research, benchmarks, tools, and services please visit our website at http://www.qsm.com.
Q/P Management Group, Inc.
Web site URL: www.qpmg.com
Email: moreinfo@qpmg.com
Sources of data: Q/P Management Group’s benchmark database is comprised of over twenty thousand (20,000) projects and applications from major corporations, commercial developers, and government agencies. The database contains statistics on a broad range of tools and techniques utilized by these organizations. Data is collected through a number of vehicles including client benchmark studies, outsourcing baseline assessments, ongoing performance measurement engagements, and through helping companies establish measurement and estimating programs. The data is collected under confidential agreements with the clients contributing to the database. Project and application data are added to the database after rigorous analysis and verification to ensure the highest degree of data integrity in the industry.
Data metrics: Productivity for projects is expressed in function points developed per hour (FP/Hour) or hours per function point (Hours/FP). All function point data is manually counted, utilizing the IFPUG method. Project Cost is expressed in cost per function point (Cost/FP). Project Quality is expressed in defects reported per developed function point (Defect/FP). Project Schedule is expressed in project calendar days required to deliver the project from initiation to first implementation. Maintenance Productivity is expressed in application function points maintained per full time equivalent employee (FP/FTE) or function points maintained per hour (FP/Hour). Maintenance Quality is expressed in annual defects reported per application function point (Annual Defects/ FP). Maintenance Cost is expressed in annual maintenance cost per application function point (Annual Cost/FP). Other benchmark data is available including Percent Lifecycle Effort by Phase, Estimating Accuracy, Maintenance Activity Breakdown and Defects by Severity Level.
Data usage: Q/P and their clients utilize the data in various ways, including comparing performance of internal or vendor resources against industry benchmarks, as a means to identify and measure process improvements and to determine pricing for commercial software products. The data is also used for estimating software development projects’ productivity, cost, schedule, and staffing.
Data availability: Benchmarks are utilized by clients that engage Q/P consultants to conduct benchmark studies. Benchmark data can also be licensed through two primary mechanisms. The first allows an organization to license the data for use in estimating and reporting through Q/P’s Software Measurement, Reporting and Estimating (SMRe) product. Alternatively, organizations can license summarized segments of the database for internal use to evaluate performance. Samples of summary data tables are available from Q/P on request. Contact Q/P for pricing options.
Kinds of data: Data exists on both software development projects and application maintenance. Projects include both new development and enhancement and range in size from less than 5 to over 20,000 function points. Application age ranges from 1 to 40+ years and application size ranges from less than 100 to over 30,000 function points. The software classifications and functions in the database include:
| Data Warehouses | Middle-ware |
| Telemarketing | Customer Care |
| Cellular Services | Telco Billing |
| Equipment Procurement | Plant and Facilities |
| Network Management | Credit Card Processing |
| Funds Transfers | Securities Fraud |
| ATM Networks | P.O.S. Networks |
| Human Resources | Finance and Accounting |
| Telco Provisioning | Distribution |
| Stocks and Bonds | Account Management |
| Contract Management | Weather Forecasting |
Volume of data: Since 1993 there have been over 15,000 project data points and over 6,000 application data points collected. The database is updated annually utilizing data collected during the previous year. Older data is retained within the database for historical trend analysis purposes but is excluded from current performance reporting.
Industry data: Data is collected from a variety of industries including: financial services, government, healthcare, telecommunications, manufacturing, pharmaceuticals, public utilities, retail, transportation and software developers.
Methodology data: Data is collected on a variety of development methodologies including, but not limited to: Agile, Waterfall and Rational Unified Process (RUP), as well as hybrid techniques utilizing portions of several disparate methods. Data included in the data base are aligned with the CMMI level the organization had achieved at the time the data was collected.
Country data: Over 60 percent of the data is from North American sources. The remainder of the data is primarily obtained through European and South American sources. Offshore (India) development and maintenance is well represented in both project and application data.
Summary: Q/P Management Group (Q/P) was one of the first organizations to offer performance benchmark assessments using function point based benchmarks. The benchmarking methodologies used by Q/P are comprehensive and based on industry accepted standards. Q/P’s function point based benchmark data base is the largest and most wide-ranging available in the industry today. Numerous companies and government agencies utilize this database to benchmark software quality, productivity and cost.
RBCS, Inc.
Web site URL: www.rbcs-us.com
Email: info@rbcs-us.com
Sources of data: On-site interviews of software testing and quality assurance teams and groups within larger organizations that develop software and/or systems for internal use or for sale to customers, or which provide testing and quality assurance services. Discussions with clients in public and private training sessions. The data is collected under non-disclosure agreements, and thus only anonymized and collective data is available for comparative purposes.
Data metrics: The metrics are some two dozen testing and quality related process, product, and project metrics. Examples include: defect detection effectiveness, cost per defect found, percentage coverage of risks and/or requirements, percentage of invalid defect reports, percentage of re-opened defect reports, consistency of test case documentation, accuracy of test estimates, percentage overhead of test environment setup/maintenance, and various others.
Data usage: Data is used to assess the capability and process improvement potential of organizations involved in creation of software or systems for their own use or for use by clients/customers, and of organizations engaged in providing testing services. Data and metrics are mentioned in some of Rex Black’s book, such as Managing the Testing Process, Advanced Software Testing: Volume 2, and The Expert Test Manager (scheduled for publication in 2012). Data is also used in many of Rex Black’s presentations and articles.
Data is provided to specific clients of assessment, baseline, and benchmark studies. These studies compare clients against similar companies.
Data is discussed in various RBCS training courses, especially test management courses.
Data is also discussed in various RBCS free webinars on topics related to test management.
Data availability: Data is provided to clients of assessment and benchmark studies. General data is published in books and journal articles.
Kinds of data: Software and system quality and testing capability measurements focused on project, process, and product. People-focused metrics are discussed in consultancy but not provided for general use, due to the potential for problems with such metrics.
Data comes from work with clients on assessments of their projects, processes, and products, as well as work with training clients.
Volume of data: RBCS has over 200 clients and has been in the software testing and quality assurance training, consulting, and outsource business since 1994.
Industry data: Data from systems and software projects spanning almost all applications of software and hardware/software systems, with no particular emphasis in terms of industry. Both private and government clients are included.
Data is collected from organizations ranging in size from small companies with less than a dozen employees to some of the largest organizations in the world (e.g., Sony, CA, etc.)
Due to the fact that assessments are paid consulting services, open source data is not included.
Methodology data: Data is collected for a variety of methodologies including Agile, waterfall, Rational Unified Process (RUP), Scrum, Extreme Programming (XP), and hybrid methods that combine features of several methods.
Data covers all types of testing, including functional testing, performance testing, usability testing, internationalization testing, manual testing, automated testing, regression testing, and so forth.
Language data: Many programming languages are included, but some of the older languages such as COBOL and FORTRAN are under-represented.
Country data: RBCS clients are distributed widely around the world, and include many multinationals. Emerging software testing countries are under-represented in some cases, but RBCS is heavily involved in software testing and quality training and consulting with emerging economies such as India, Bangladesh, Kenya, Nigeria, South Africa, Malaysia, and China.
Unique data: RBCS data is unique in the range of test-related metrics and data available to clients. While other consultancies provide proprietary test assessment and/or “maturity” services, many of these are almost entirely questionnaire-based and/or interview based, and thus suffer from the Rashomon Effect. RBCS assessments, being evidence-focused and data-focused, provide deeper insights to clients. In addition, RBCS assessments are based on an open-source framework published in Rex Black’s book Critical Testing Processes.
Future data: RBCS intends to continue to provide quality and testing assessment services to clients around the world, and is working to expand the depth of the data through more assessments. RBCS is also considering developing an online assessment process using its proven e-learning platform.
Summary: RBCS is unique in test consultancies in being focused on an evidence-based and metrics-based approach to assessment, which provides clients with reliability benchmarks and assessment results. RBCS has been in the quality and testing consulting business since 1994, and founder Rex Black’s background in software engineering extends back to 1983.
Reifer Consultants LLC
Web site URL: www.reifer.com
Email: info@reifer.com
Sources of data: Data sharing agreements with firms, government agencies and universities. The data is collected and protected under a data sharing agreement through collaborators, clients and via public sources.
Data metrics: Productivity is expressed in equivalent source line of code (SLOC) per staff-month, cost in $/SLOC, duration in months and quality in defects/thousand (K) SLOC. Data are segmented into three sectors (industry, defense and government) and twenty-three applications domains. Information is provided on paradigm utilization (agile, waterfall, etc.), percentage work in-source/out-sourced and personnel turnover rates by sector.
Unique information about maintenance shops is presented. This is based on data collected during the past three years on hundreds of projects.
Data usage: Data is used to create benchmarks for use in competitive positioning, comparative analysis, improvement studies and should-cost studies. Benchmarks are often used to validate estimates submitted for software development and maintenance efforts. These data are used as the basis of benchmarking reports developed for clients.
Data availability: Benchmarks are provided clients on a subscription of contract basis. While analysis results can be shared, data are proprietary and are not sold.
Kinds of data: Records of completed projects at the program and project levels. Data employed comes from 49 commercial firms (699 projects), 27 defense firms (225 projects) and 12 government organizations (76 projects). Size average for commercial firms is 675 KSLOC (range is between 20 to 4,180 KSLOC), for defense is 487 KSLOC (range is between 18 to 5,200 KSLOC) and for government organizations is 591 KSLOC (range is between 25 to 1,825 KSLOC). The average duration for projects in the database averages between 12 to 18 months. Metadata is reported by sector and applications domains.
Volume of data: Data is taken from a database of over 10,000 records. Data used for benchmarks comes from 1,000 current projects selected to be representative of both development and maintenance efforts, none of which is over 7 years old. Demographics for data on size, types of firms and applications domain characteristics supplied for 23 applications types.
Industry data: Applications domains used to capture benchmarks include:
Industry Benchmarks (13): Automation, command and control, Info Technology – banking, Info Technology – general, Info Technology – insurance, medical, process control, scientific systems, software tools, telecommunications, test systems, training/simulation and web business.
Defense Benchmarks (7): airborne, ground, Info Technology, medical, missile, space and trainers.
Government Benchmarks (3): Info Technology, infrastructure systems and sensitive systems.
Methodology data: Project processes include from agile development to traditional waterfall by sector and applications domain.
Some data collected for CMM, CMMI and ISO utilization.
Country data: All data in database for United States. Separate Asian and European databases are maintained because data for the three are statistically not consistent. We believe it to be a mistake to mix such data because their attributes are not highly correlated.
Summary: Reifer Consultants LLC has primarily developed and used benchmarks in its own consulting practice for over 30 years. We trust them because the data in them is highly correlated and statistically sound. Our clients trust them because the benchmarks have proven to be accurate predictors of productivity, cost, duration and quality. Reifer Consultants LLC has offered these benchmarks to its clients who have used them beneficially to make selective improvements and validate bids and estimates.
Software Benchmarking Organization
Web site URL: www.sw-benchmarking.org
Email: hans.sassenburg@sw-benchmarking.org
Sources of data: On-site benchmark studies and assessments of software projects. Data is always collected under non-disclosure agreements.
Data metrics: Data is collected for the following metrics:
- Schedule (calendar months)
- Effort (person months)
- Productivity (function points per hour using IFPUG definition)
- Cost of Quality (effort distribution over 4 areas)
- Deferral rate (ratio of deferred baselined features)
- Feature size (function points)
- Technical size (KLOC)
- Re-use level (ratio)
- Complexity (using McCabe definition)
- Test coverage (unit, integration, system testing)
- Defect density (defects per function point at release time)
- Defect removal efficiency (ratio of defects removed before releasing)
Data usage: Data is used:
- to benchmark “engineering capability” against industry averages and best-in-class figures in same industry, using data from SBO assessments as well as published data from other sources (like Capers Jones)
- to assess the feasibility of initial and running projects
Data availability: Data is provided to clients of studies and assessments
Kinds of data: Data is primarily for individual software projects
Volume of data: About 150 projects from 2001 through today.
New data is added frequently.
Industry data: Data from systems and embedded software, military software, commercial software, IT projects, civilian government projects, and outsourced projects.
Industries include banking, insurance, manufacturing, telecommunications, medical equipment, aerospace, defense, and government at both state and national levels.
Data is collected primarily from projects with 10 – 75 software engineers
Methodology data: Data is collected for a variety of methodologies including
Agile, waterfall, Rational Unified Process (RUP), and hybrid methods that combine features of several methods.
Data is also collected for the five levels of the Capability Maturity Model Integrated (CMMI™) of the Software Engineering Institute.
Language data: As is usual with large collections of data a variety of programming languages are included. The number of languages per application ranges from 1 to 5, with an average of about 2. Most common combinations include Assembler, C, C++, C# and Java.
Country data: Most of the data is from Western Europe (including The Netherlands, Germany, France, Switzerland, United Kingdom) with a strong focus on the embedded software industry.
Limited data from other countries/regions.
Unique data: The collected data is used to compute 16 KPIs, arranged in four different categories. This enables an organization to identify root causes of underperformance and estimate the effect of corrective measures.
Future data: Currently, limited data is available regarding complexity and test coverage. This will receive more attention.
Summary: SBO has been collecting software data since working 2001 using benchmarking and assessment studies. SBO uses a network of accredited partners.
In addition, SBO delivers metrics workshops as well as supporting technology for metrics collection and analysis.
Software Engineering Institute (SEI)
Web site URL: www.sei.cmu.edu
Email: info@sei.cmu.edu
Process Appraisal Data
Sources of data: Data is provided by SCAMPI lead appraisers at the conclusion of the CMMI appraisal.
Data metrics: Data include the maturity level and capability profile of the appraised organization based on the scope of the processes covered by the appraisal. In addition, the date of the appraisal and country, size, and other characteristics of the appraised organization are also included allowing for segmentation of the data for analysis purposes.
Data usage: These data are used primarily to produce the semi-annual Community
Maturity Profile. An example of the profile is available at
http://www.sei.cmu.edu/cmmi/casestudies/profiles/pdfs/upload/2011SepMMI-2.pdf
More generally, see http://sas.sei.cmu.edu/pars/
Data availability: The maturity profiles are publicly available from the SEI website. Additionally, company-specific results can be obtained at: http://sas.sei.cmu.edu/pars/
Volume of data: Data is added continuously. The Sept 2011 report is based on more than 4600 organizations from over 70 countries. Different subsets of the data are used for different analyses.
Industry data: Data come from a wide variety of organization in terms of industrial sector, geography, and size of organization.
Country data: A graphic and table are included in the report showing the coverage and volume from of data various countries.
Unique data: Based on organizations reporting reappraisals, analyses of the time to move from one maturity level to the next are provided.
SEI Survey Data
- Performance Effects of Measurement and Analysis: Perspectives from CMMI High Maturity Organizations and Appraisers
- Use and Organizational Effects of Measurement and Analysis in High Maturity Organizations: Results from the 2008 SEI State of Measurement and Analysis Practice Surveys
Sources of data: Questionnaire responses from organizations that have attained CMMI maturity levels 4 and 5
Data metrics: Use of process performance baselines and models; issues faced in the adoption and use of measurement and analysis in high maturity organizations; breadth of statistical, experimental, and simulation methods used; attention paid to data quality and integrity; staffing and resources devoted to the work; pertinent training and coaching; and the alignment of the models with business and technical objectives
Data usage: Data is analyzed and reported by the SEI to increase understanding of the state of the practice in measurement and analysis
Data availability: Access via the Software Engineering Information Repository
Direct Link (password required):
To register for a free account to use the SEIR:
When you first login you can access this data by scrolling down the area under Measurement and clicking Research Data for Analysis
Volume of data: Responses from 156 appraisal sponsors and their delegates in CMMI maturity levels 4 or 5 organizations. Responses from 84 SEI-certified Standard CMMI Appraisal Method for Process Improvement High Maturity Lead Appraisers
Industry data: Data includes information about organization type (government, defense, commercial off-the-shelf, etc.)
Methodology data: N/A
Language data: N/A
Country data: Organizations surveyed were in multiple countries
Personal Software Process (PSP) Data
Sources of data: Results of programming exercises completed in PSP courses taught by the SEI and SEI partners.
Data metrics: Size (LOC), Effort (minutes), Defects (count, time, type)
Data usage: Currently, data is used by the people generating it to manage their own work and not analyzed for other purposes.
Data availability: One sanitized data set is available. It lacks defect data type information, but most other data is complete. Other data has been collected but is not yet in sharable form. Data is currently in the TSP Access Student tool or Process Dashboard.
We will work with collaborators to collect the individual data into a sanitized database suitable for distribution outside the SEI. Contact tsp-data@sei.cmu.edu for information.
Volume of data: Data from several thousand students
Team Software Process (TSP) Data
Sources of data: Similar to PSP data, with the addition of project-level metrics
Data metrics: Size (LOC), Effort (minutes), Defects (count, time, type)
Data usage: Currently, data is used by the organizations generating it to manage their own work and has not been analyzed extensively for other purposes. A related SEI report has been published: Using TSP Data to Evaluate Your Project Performance
Data availability: The data is not currently in a state ready to share publicly. All data is in detailed form rather than aggregated into summaries and must be kept confidential.
We welcome collaborations that could support analyzing, aggregating, and sanitizing this data. Contact tsp-data@sei.cmu.edu for information.
Country data: Data is from multiple countries
Summary: The Software Engineering Institute’s Software Engineering Process Management program has collected a variety of data over the years. Adoption and use of CMMI and its predecessor model, the SW-CMM, have been tracked and reported on most notably in the semi-annual Community Maturity Profile reports described above. Additionally, special studies in the form of surveys have also been undertaken. The SEI recently started to “scrub” those data so they could be made available to others to conduct their own analyses. Two sets of survey data have been made public to date and are listed above. Data from PSP courses and TSP projects have been collected but are not yet in a form that allows it to be shared publicly.
Software Improvement Group (SIG)
Web site: www.sig.eu
Email: info@sig.eu
Sources of data: SIG obtains source code and other data on software systems from its customers and from various open source projects. SIG is equipped with an ISO/IEC 17025 certified software evaluation laboratory, in which source code is processed and analyzed according to a standardized evaluation procedure. Analysis results are stored in the benchmark repository, called the Software Analysis Warehouse (SAW).
Data Metrics: The SAW stores diverse types of metrics calculated automatically from source code or other software system-related artifacts. For most systems, these metrics are available for multiple snapshots. Metrics stored in the SAW include:
- Traditional code metrics such as coupling, complexity, and size. These metrics are available at the most detailed level of methods and functions, as well as aggregated into quality profiles at the level of files, packages, modules, and entire systems.
- Churn metrics, i.e. added, deleted, and modified lines of code at the level of files.
- Maintainability ratings according to the SIG/TÜViT Evaluation Criteria at the level modules and systems.
- Programming languages used in projects and their respective volumes.
- Issue handling performance metrics, including resolution time, resolution speed ratings, and issue handling productivity.
Data Usage: The data collected is used for two main purposes. First, the data is used to support various consulting services. For example, the benchmark data set is used for annual re-calibration of the SIG maintainability model, which is applied in SIG’s services for software risk analysis, software quality and productivity monitoring, and software product certification. Yearly re-calibration ensures that the model remains a faithful reflection of the state of the art of software quality.
The second use of the data is for research. SIG is actively engaged in many research initiatives internationally in various research themes such as mining software repositories, software architecture, and software testing. The data is used for validating theories or beliefs in software engineering. Results of such validations have been published in international journals and conferences.
Data availability: Data obtained from customers is subject to non-disclosure agreements, and is therefore not available.
Data presented in public SIG Analysis Reports or in scientific publications is typically available (consult the respective publications), or otherwise may be provided upon request.
Kinds of data: Systems stored in the SAW have either one or multiple snapshots. Systems with only one snapshot are obtained from customers that requested a one-off quality assessment. Systems with multiple snapshots are those monitored by SIG throughout a certain period of time. Apart from proprietary system, the SAW contains data for a range of open source systems that were analyzed for research purposes.
Volume of data: At the time of writing, data in the SAW regards over 400 unique systems and around 7,500 snapshots. The total lines of code reach about 74 million lines of code. The size of systems ranges up to 3M source lines of code.
Customers upload new system snapshots on a regular basis. Depending on the agreement the upload frequency can be weekly, monthly, or quarterly. Over 100 new systems are added to the SAW yearly.
Industry data: Systems in the SAW come from various domains, which include transportation, insurance, banking, and telecommunication.
System owners are either medium to large size enterprises or governmental organizations.
Methodology data: Data is collected from systems developed using a variety of methodologies. Iterative/agile methodologies seem to be the most frequent methodology followed. Apart from systems that are under development or were recently developed, data is collected on systems that are under maintenance. These include legacy systems that were once developed using non-iterative, waterfall methodologies.
Technology data: The systems of which data is currently stored in the SAW are built using over 70 different technologies, which include Java, .NET, C/C++, PL/SQL, BPEL, various dialects of COBOL, and Pascal.
Country data: The majority of the data is obtained from customers in Europe, particularly from Western Europe and Scandinavian countries. Data is also increasingly coming from other parts of Europe and other regions such as UK, Greece, and Turkey. IT suppliers of the customers come from a more diverse region such as Eastern Europe and Asia.
Unique data: Multiple snapshots of system code give unique opportunities to perform various data analyses. With hundreds of systems, in which each of them has multiple snapshots, SIG has performed trend analysis for numerous metrics and has developed several prediction models. For example, SIG has developed an economic model to estimate renovation efforts based on churn analyses of multiple system snapshots.
Future data: SIG is constantly collecting more data from industrial systems. This ensures that new systems, especially those developed using new technologies, are represented in the benchmark data set. Furthermore, SIG has started to collect data on software specifications and models.
Summary: SIG is an Amsterdam-based consultancy firm that provides consulting services in the area of software quality, software risk, and software economics. These services are supported by a standardized software evaluation methodology, carried out by SIG’s certified software evaluation lab. In partnership with TÜV Informationstechnik (TÜViT), SIG provides certification of software maintainability.
The Software Analysis Warehouse (SAW) is the main benchmark database of SIG. The SAW contains detailed information on numerous snapshots of a steadily increasing number of software systems (currently more than 400). The benchmark is used in consultancy services as well as in software engineering research.
Well-established tooling and procedures allow SIG to generate a large volume of software engineering data from both commercial and open source systems in a consistent and controlled manner. Data collection is highly curated by technical and client-facing consultants, ensuring a reliable source of data with very strong traceability characteristics.
Test Maturity Model integrated (TMMi) survey
Survey Web site URL: www.experimentus.com
Email: geoff.thompson@experimentus.com
Sources of data: Submissions invited via multitude of social networking sites (LinkedIn, Twitter and Facebook etc.) and also from organisations who have been assessed using the TMMi model. The data is collected under an agreement of anonymity. Data is submitted directly via a weblink on the Experimentus website.
Data metrics: A questionnaire is used to collect responses aimed at identifying company and individual test process maturity, based upon the contents of the TMMi (Test Maturity Model – see www.tmmifoundation.org) model.
The data collected is pooled and is used to identify, for each of the 10 Key Process Areas in TMMi levels 2 and 3, what level maturity each companies test process is at, as well as an overall general perspective on the maturity of the test industry.
The results are published every 18 months, with the data being collected during the previous 18 months.
Data is collected from over 100 companies over 5 continents.
Data usage: The data is used to interpret the current maturity of the overall test processes in use across the world, as well as specifically within each company.
Data availability: The ‘raw’ data is not available for public viewing; however the report is freely downloadable from the Experimentus website.
Kinds of data: The data is in the form of responses to specific statements specific to the differing components of the TMMi test process, in four dimensions – well-established, exists but is not deployed successfully, embryonic stage and no process exists.
In addition to the data collected from these 16 questions, all data collected from all formal assessment carried out by Experimentus in the 18 months between reports is also summarised and included.
Volume of data: Each 18 months, approximately 100 different responses are collected to 16 specific statements.
Industry data: Industries responding include IT Support Services, Banking, Finance, Legal, Manufacturing, Retail, Local Government and Education.
Data is submitted from a wide range of organizations of differing sizes. There is no data from the military.
Methodology data: Data is collected for a variety of methodologies including Agile, SCRUM, Waterfall, Joint Application Design (JAD), Rational Unified Process (RUP), Team Software Process, (TSP), and hybrid methods that combine features of several methods.
Country data: About 30% of the data is from the UK, 20% from the U.S. 10% from Japan; 10% Australia; 10% Netherlands, 7% India. Also the data from 7 other countries is included.
Summary: Experimentus was established in 2004 to provide true Consultancy in the international IT Quality arena, helping companies to deliver better quality software quickly and cheaply. Its principal measurement of client maturity is to assess against the TMMi model. They were one of the first companies to have their TMMi assessment method accredited by the TMMi Foundation and were the first company in the world to have accredited assessors. They carry out around 14 formal TMMi assessments each year; in a variety of countries, most notably UK, Spain and Malaysia. The data from these assessments is included within the report.
The TMMi Foundation was formally launched in early 2007; it has over 600 members from all across the world. It owns and maintains the TMMi Model, as well as accreditation of assessment methods and assessors. More details can be found at www.tmmifoundation.com.
Appendix A: Books and Web Sites that have Quantitative Data
Amazon books on “Software Measurements” as of November 2011
Note: This section lists the titles and authors of books that appear based on a query in Amazon for “software measurement.” The Amazon results will change frequently. New books come out almost every month.
- Applied Software Measurement, Capers Jones
- The Economics of Software Quality, Capers Jones & Olivier Bonsignour
- Assessment and Control of Software Risks, Capers Jones
- Patterns of Software Systems Failure and Success, Capers Jones
- Critical Problems in Software Measurement, Capers Jones
- Software Engineering Economics; Barry Boehm
- Software Measurement, Christof Ebert and Reiner Dumke
- Measuring the Software Process, William A. Florac
- The Software IP Detective’s Handbook, William Zeidman
- Software Measurement and Estimation, Linda Laird and M. Carol Brennan
- How to Observe Software Systems, Gerald Weinberg
- Responding to Significant Software Results, Gerald Weinberg
- Best Practices in Software Measurement, Christof Ebert and Reiner Dumke
- Practical Software Measurement, John McGarry and David Card
- Controlling Software Projects, Tom DeMarco
- Value Based Software Engineering, Arum, Boehm, and Erdogmus
- Practical Software Metrics, Robert Grady
- Software Metrics, Ravindranath Pandian
- Function Point Analysis, David Garmus and David Herron
- Applied Statistics for Software Managers, K. Maxwell
- Software Metrics: Establishing a Company-Wide Program; R. Grady and D. Caswell
- The IT Measurement Compendium, Manfred Bundschuh and Carol Dekkers
- Making Software Measurement Work, William Hetzel
- Software Engineering Measurements, John C. Munson
- Object Oriented Test and Measurement, Lee Atchison
- Software Product and Process Measurement; Alain Abran et al
- A Framework of Software Measurement, Horst Zuse
- The Software Measurement Guidebook, John Gaffney
- Object-Oriented Software Metrics, Mark Lorenz and Jeff Kidd
- IT Measurement: Advice from the Experts; IFPUG authors
- Software Metrics for Product Assessment, Richard Bache and Gualtiero Bazzano
- Derivation and Validation of Software Metrics, Martin Shepard and Darrel Ince
- Software Metrics: A Practitioners Guide; K.H. Moller and Daniel Paulish
- Foundations of Empirical Software Engineering; Basili, Boehm, Rombach, Zelkowitz
- Practical Software Measurement, Bob Hughes
- Software Measurement: A Visualizaton Toolkit; Ellis, Fujiwara, Kuo, Simmons
- Measurement as a Powerful Software Management Tool, Nickolas Ashley
- Measurement for Software Control and Assurance, B.A. Kitchenham and B. Littlewood
- Software Metrics: Measurement for Software Process Improvement
- Investigations in Software Measurement, Alain Abran
- Practical Implementation of Software Metrics, Paul Goodman
- Software Metrics: A Rigorous and Practical Approach, N. Fenton and L. Pfleeger
- Managing Software Quality: A Measurement Framework; Richard Nance
- Software Process Improvement: Metrics, Measurement, and Modeling; Haug et al
- The Effectiveness of Software Project Management; Christopher Cullen
- The IFPUG Guide to IT and Software Measurement; IFPUG authors
- Quality Software Management; First-Order Measurement, Gerald Weinberg
- Metrics and Models in Software Quality Engineering, Dr. Steve Kan
- Software Performance and Scalability; Henry Liu
- Mastering Software Quality Assurance, Murali Chemuturi
- Six Sigma Software Quality Improvement; Vic Nanda, Jeffrey Robinson
- Software Verification and Validation; Steven R. Ratkin
- A Practical Guide for Measuring Usability, Jeff Sauro
- Joe Celko’s Data Measurement and Standards in SQL; Joe Celko
- Oracle Database Performance and Scalability, Henry H. Liu
- Agile Analysis, a Value-Driven Approach; Ken Collier
- Project Management Metrics, KPIs, and Dashboards, Harold Kerzner
- Entropy-Based Measurement for Software; Ozgur Aktunc
- ISO/IEC 14143-1:1998; Software Measurement, Functional Size Measurement
- IT Governance Metrics, Measurement, and Benchmarking, Blokdijk and Menken
- Object-Oriented Design Metrics; Scott Whitmire
- Quality Software Project Management, Donald Shafer, Isabell Shafer
- Handbook of Software Quality Assurance. G. Gordon Schulmeyer
- Software Engineering Reviews and Audits, Boyd L. Summers
- Managing the Development of Software-Intensive Systems; James McDonald
- Scaling Lean and Agile Development; Craig Larman and Bas Vodde
- Practical Software Measurement, Card, Layman, Clark, Dean, Hall, McGarry
- Antipatterns: Identification, Refactoring, and Management, Phillip Laplante
- Portfolio Performance Measurement and Benchmarking, Christopherson et al
- Software Reliability Engineering, John Musa
- Software Metrics and Software Methodology, Alain Abran
- Measuring the Software Process, David Garmus and David Herron
- ROI of Software Process Improvement, David Rico and Roger Pressman
- Agile Estimating and Planning, Mike Cohn
- Software Sizing, Estimation, and Risk Management, Dan Galorath
- Process Improvements and CMMI® for Systems and Software, Kenett and Baker
Amazon Books on “Software Cost Estimation” as of November 2011
- Estimating Software Costs, Capers Jones
- Practical Software Project Estimation, Peter Hill
- Managing the Black Hole, Gary Gack
- Software Sizing, Estimation, and Risk Management, Dan Galorath
- Software Engineering Economics, Barry Boehm
- Software Estimation: Demystifying the Black Art; Steve McConnell
- Estimating Software Intensive Systems, Richard D. Stutzke
- Software Project Cost and Schedule Estimating, W. Roetzheim and R. Beasley
- Cost Estimation with COCOMO II, Barry Boehm et al
- Software Estimation Best Practices, Murali Chemuturi
- Software Cost Estimation and Sizing Methods, Shari Lawrence Pfleeger
- Cost Estimation for Software Development, Bernard Londeix
- Agile Estimating and Planning, Mike Cohn
- IT Project Estimation, Paul Coombs
- Software Measurement and Estimation, Linda Laird and M. Carol Brennan
- Software Project Estimation by Analogy, Mohammad Azzeh
- Project Estimating and Cost Management, Parviz Read
- Cost Estimation in Software Product Line Engineering, Sebastian Rosensteiner
- An Approach to Software Cost Estimation, U.S. Government
- Cost Estimation Techniques for Web Projects, Emelia Mendez
- Software Metrics: A Guide to Planning, Analysis, and Application, C.R. Pandian
- Practices for Scaling Lean and Agile Development, Craig Larman and Bas Vodde
- Modern Tools to Support DoD Software…., Jo Anne Lane and Barry Boehm
- How to Save a Failing Project, R. Young, S. Brady, D. Nagle
- Value Based Software Engineering, Biffi, Arum, Boehm, and Erdogmus
- Software Project Dynamics, T. Abdel-Hamid and Stuart Madnick
- Reuse Based Software Engineering, Hafedh Mili et al
- Essentials of Software Project Management, Richard Bechtold
- Systems Management for Information Technology and Software Engineering, A. Sage
- New Trends in Software Process Modeling, Silvia T. Acuna
- Software Management, Donald Reifer
- Project Management the Agile Way, John C. Goodpasture
- Risk Management Processes for Software Engineering Models, Marian Myerson
- Trustworthy Systems Through Quantitative Software Engineering, L. Bernstein and C.M. Yunhas
Books with Quantitative Data
Boehm, Barry Dr.; Software Engineering Economics; Prentice Hall, Englewood Cliffs, NJ; 1981; 900 pages.
Booch Grady, Object Solutions: Managing the Object-Oriented Project; Addison Wesley, Reading, MA; 1995.
Capability Maturity Model Integration; Version 1.1; Software Engineering Institute; Carnegie-Mellon Univ.; Pittsburgh, PA; March 2003; http://www.sei.cmu.edu/cmmi/
Brooks, Fred: The Mythical Man-Month, Addison-Wesley, Reading, Mass., 1974, rev. 1995.
Charette, Bob; Software Engineering Risk Analysis and Management; McGraw Hill, New York, NY; 1989.
Charette, Bob; Application Strategies for Risk Management; McGraw Hill, New York, NY; 1990.
Cohn, Mike; Agile Estimating and Planning; Prentice Hall PTR, Englewood Cliffs, NJ; 2005; ISBN 0131479415.
DeMarco, Tom; Controlling Software Projects; Yourdon Press, New York; 1982; ISBN 0-917072-32-4; 284 pages.
Ewusi-Mensah, Kweku; Software Development Failures; MIT Press, Cambridge, MA; 2003; ISBN 0-26205072-2276 pages.
Gack, Gary; Managing the Black Hole – The Executives Guide to Project Risk; The Business Expert Publisher; Thomson, GA; 2010; ISBSG10: 1-935602-01-2.
Galorath, Dan; Software Sizing, Estimating, and Risk Management: When Performance is Measured Performance Improves; Auerbach Publishing, Philadelphia; 2006; ISBN 10: 0849335930; 576 pages.
Glass, R.L.; Software Runaways: Lessons Learned from Massive Software Project Failures; Prentice Hall, Englewood Cliffs; 1998.
Hill, Peter R. Practical Software Project Estimation; McGraw Hill, 2010
Harris, Michaael; Herron, David; and Iwanicki, Stacia; The Business Value of IT: Managing Risks, Optimizing Performance, and Measuring Results; CRC Press (Auerbach), Boca Raton, FL: ISBN 13: 978-1-4200-6474-2; 2008; 266 pages.
Humphrey, Watts; Managing the Software Process; Addison Wesley, Reading, MA; 1989.
Johnson, James et al; The Chaos Report; The Standish Group, West Yarmouth, MA; 2000.
Jones, Capers and Bonsignour, Olivier; The Economics of Software Quality; Addison Wesley Longman, Boston, MA; ISBN 10: 0-13-258220—1; 2011; 585 pages.
Jones, Capers; Software Engineering Best Practices; McGraw Hill, New York, NY; ISBN 978-0-07-162161-8; 2010; 660 pages.
Jones, Capers; Estimating Software Costs; McGraw Hill, New York; 2007; ISBN 13-978-0-07-148300-1.
Jones, Capers; Assessment and Control of Software Risks; Prentice Hall, 1994; ISBN 0-13-741406-4; 711 pages.
Jones, Capers; Patterns of Software System Failure and Success; International Thomson Computer Press, Boston, MA; December 1995; 250 pages; ISBN 1-850-32804-8; 292 pages.
Jones, Capers; Software Assessments, Benchmarks, and Best Practices; Addison Wesley Longman, Boston, MA; ISBN 0-201-48542-7; 2000; 657 pages.
Jones, Capers; Program Quality and Programmer Productivity; IBM Technical Report TR 02.764, IBM San Jose, CA; January 1977.
Jones, Capers; Programming Productivity; McGraw Hill, New York; ISBN 0-07-032811-0; 1986.
Jones, Capers; “Estimating and Measuring Object-Oriented Software”; American Programmer; 1994.
Jones, Capers; “Why Flawed Software Projects are not Cancelled in Time”; Cutter IT Journal; Vol. 10, No. 12; December 2003; pp. 12-17.
Jones, Capers; “Software Project Management Practices: Failure Versus Success”;
Crosstalk, Vol. 19, No. 6; June 2006; pp4-8.
Laird, Linda M and Brennan, Carol M; Software Measurement and Estimation: A Practical Approach; John Wiley & Sons, Hoboken, NJ; 2006; ISBN 0-471-67622-5; 255 pages.
Park, Robert E. et al; Software Cost and Schedule Estimating – A Process Improvement Initiative; Technical Report CMU/SEI 94-SR-03; Software Engineering Institute, Pittsburgh, PA; May 1994.
Park, Robert E. et al; Checklists and Criteria for Evaluating the Costs and Schedule Estimating Capabilities of Software Organizations; Technical Report CMU/SEI 95-SR-005; Software Engineering Institute, Pittsburgh, PA; January 1995.
McConnell; Software Estimating: Demystifying the Black Art; Microsoft Press, Redmund, WA; 2006.
Roetzheim, William H. and Beasley, Reyna A.; Best Practices in Software Cost and Schedule Estimation; Prentice Hall PTR, Saddle River, NJ; 1998.
Strassmann, Paul; Information Productivity; Information Economics Press, Stamford, Ct; 1999.
Strassmann, Paul; Information Payoff; Information Economics Press, Stamford, Ct; 1985.
Strassmann, Paul; Governance of Information Management: The Concept of an Information Constitution; 2nd edition; (eBook); Information Economics Press, Stamford, Ct; 2004.
Strassmann, Paul; The Squandered Computer; Information Economics Press, Stamford, CT; 1997.
Stukes, Sherry, Deshoretz, Jason, Apgar, Henry and Macias, Ilona; Air Force Cost Analysis Agency Software Estimating Model Analysis ; TR-9545/008-2; Contract F04701-95-D-0003, Task 008; Management Consulting & Research, Inc.; Thousand Oaks, CA 91362; September 30 1996.
Symons, Charles R.: Software Sizing and Estimating—Mk II FPA (Function Point Analysis), John Wiley & Sons, Chichester, U.K., ISBN 0-471-92985-9, 1991.
Wellman, Frank, Software Costing: An Objective Approach to Estimating and Controlling the Cost of Computer Software, Prentice Hall, Englewood Cliffs, NJ, ISBN 0-138184364, 1992.
Whitehead, Richard; Leading a Development Team; Addison Wesley, Boston, MA; 2001; ISBN 10: 0201675267; 368 pages.
Yourdon, Ed; Death March – The Complete Software Developer’s Guide to Surviving “Mission Impossible” Projects; Prentice Hall PTR, Upper Saddle River, NJ; ISBN 0-13-748310-4; 1997; 218 pages.
Yourdon, Ed; Outsource: Competing in the Global Productivity Race; Prentice Hall PTR, Upper Saddle River, NJ; ISBN 0-13-147571-1; 2005; 251 pages.
Books and Readings on Measurements and Metrics
Abran, Alain and Dumke, Reiner R; Innovations in Software Measurement; Shaker-Verlag, Aachen, DE; ISBN 3-8322-4405-0; 2005; 456 pages.
Abran, Alain; Bundschuh, Manfred; Dumke, Reiner; Ebert; Christof; and Zuse, Horst; Software Measurement News; Vol. 13, No. 2, Oct. 2008 (periodical).
Bundschuh, Manfred and Dekkers, Carol; The IT Measurement Compendium; Springer-Verlag, Berlin, DE; ISBN 978-3-540-68187-8; 2008; 642 pages.
Chidamber, S.R. & Kemerer, C.F.; “A Metrics Suite for Object-Oriented Design”; IEEE Trans. On Software Engineering; Vol. SE20, No. 6; June 1994; pp. 476-493.
Dumke, Reiner; Braungarten, Rene; Büren, Günter; Abran, Alain; Cuadrado-Gallego, Juan J; (editors); Software Process and Product Measurement; Springer-Verlag, Berlin; ISBN 10: 3-540-89402-0; 2008; 361 pages.
Ebert, Christof and Dumke, Reiner; Software Measurement: Establish, Extract, Evaluate, Execute; Springer-Verlag, Berlin, DE; ISBN 978-3-540-71648-8; 2007; 561 pages.
Garmus, David & Herron, David; Measuring the Software Process: A Practical Guide to Functional Measurement; Prentice Hall, Englewood Cliffs, NJ; 1995.
Garmus, David and Herron, David; Function Point Analysis – Measurement Practices for Successful Software Projects; Addison Wesley Longman, Boston, MA; 2001; ISBN 0-201-69944-3;363 pages.
IFPUG Counting Practices Manual, Release 4, International Function Point Users Group, Westerville, OH; April 1995; 83 pages.
International Function Point Users Group (IFPUG); IT Measurement – Practical Advice from the Experts; Addison Wesley Longman, Boston, MA; 2002; ISBN 0-201-74158-X; 759 pages.
Jones, Capers; Applied Software Measurement; McGraw Hill, 3rd edition, 2008; ISBN 978-0-07-150244-3; 662 pages.
Jones, Capers: “Sizing Up Software;” Scientific American Magazine, Volume 279, No. 6, December 1998; pages 104-111.
Jones; Capers; “A Short History of the Lines of Code Metric”; Version 4.0; May 2008; Capers Jones & Associates LLC; Narragansett, RI; 15 pages (monograph).
Kemerer, C.F.; “Reliability of Function Point Measurement – A Field Experiment”; Communications of the ACM; Vol. 36; pp 85-97; 1993.
Parthasarathy, M.A.; Practical Software Estimation – Function Point Metrics for Insourced and Outsourced Projects; Infosys Press, Addison Wesley, Upper Saddle River, NJ; 2007; ISBN 0-321-43910-4.
Putnam, Lawrence H.; Measures for Excellence — Reliable Software On Time, Within Budget; Yourdon Press – Prentice Hall, Englewood Cliffs, NJ; ISBN 0-13-567694-0; 1992; 336 pages.
Putnam, Lawrence H and Myers, Ware.; Industrial Strength Software – Effective Management Using Measurement; IEEE Press, Los Alamitos, CA; ISBN 0-8186-7532-2; 1997; 320 pages.
Stein, Timothy R; The Computer System Risk Management Book and Validation Life Cycle; Paton Press, Chico, CA; 2006; ISBN 10: 1-9328-09-5; 576 pages.
Stutzke, Richard D; Estimating Software-Intensive Systems; Addison Wesley, Upper Saddle River, NJ; 2005; ISBN 0-201-70312-2; 918 pages.
Books and Readings on Architecture, Requirements, and Design
Ambler, S.; Process Patterns – Building Large-Scale Systems Using Object Technology; Cambridge University Press; SIGS Books; 1998.
Artow, J. & Neustadt, I.; UML and the Unified Process; Addison Wesley, Boston, MA; 2000.
Bass, Len, Clements, Paul, and Kazman, Rick; Software Architecture in Practice; Addison Wesley, Boston, MA; 1997; ISBN 13: 978-0201199307; 452 pages.
Berger, Arnold S.; Embedded Systems Design: An Introduction to Processes, Tools, and Techniques; CMP Books; 2001; ISBN 10-1578200733.
Booch, Grady; Jacobsen, Ivar, and Rumbaugh, James; The Unified Modeling Language User Guide; Addison Wesley, Boston, MA; 2nd edition 2005.
Cohn, Mike; User Stories Applied: For Agile Software Development; Addison Wesley, Boston, Ma; 2004; ISBN 0-321-20568.
Fernandini, Patricial L; A Requirements Pattern; Succeeding in the Internet Economy; Addison Wesley, Boston, MA; 2002; ISBN 0-201-7386-0; 506 pages.
Gamma, Erich; Helm, Richard; Johnson, Ralph; Vlissides, John; Design Patterns: Elements of Reusable Object Oriented Design; Addison Wesley, Boston MA; 1995.
Inmon William H, Zachman, John, and Geiger, Jonathan G; Data Stores, Data Warehousing, and the Zachman Framework; McGraw Hill, New York; 1997; ISBN 10: 0070314292; 358 pages.
Marks, Eric and Bell, Michael; Service-Oriented Architecture (SOA): A Planning and Implementation Guide for Business and Technology; John Wiley & Sons, New York; 2006; ISBN 10: 0471768944; 384 pages.
Orr, Ken; Structured Requirements Definition; Ken Orr and Associates, Inc, Topeka, KS; 1981; ISBN 0-9605884-0-X; 235 pages.
Robertson, Suzanne and Robertson, James; Mastering the Requirements Process; 2nd edition; 2006; Addison Wesley, Boston, MA; ISBN 0-321-41949-9; 560 pages.
Martin, James & McClure, Carma; Diagramming Techniques for Analysts and Programmers; Prentice Hall, Englewood Cliffs, NJ; 1985; ISBN 0-13-208794-4; 396 pages.
Warnier, Jean-Dominique; Logical Construction of Systems; Van Nostrand Reinhold, London, UK; ISBN 0-4442-22556-3; 177 pages.
Wiegers, Karl E; Software Requirements; 2nd edition; 2003; Microsoft Press, Bellevue, WA; ISBN 10: 0735618798; 544 pages.
Books and Readings on Software Quality Control
Beck, Kent; Test-Driven Development; Addison Wesley, Boston, MA; 2002; ISBN 10: 0321146530; 240 pages.
Chelf, Ben and Jetley, Raoul; “Diagnosing Medical Device Software Defects Using Static Analysis”; Coverity Technical Report, San Francisco, CA; 2008.
Chess, Brian and West, Jacob; Secure Programming with Static Analysis; Addison Wesley, Boston, MA; 20007; ISBN 13: 978-0321424778; 624 pages.
Cohen, Lou; Quality Function Deployment – How to Make QFD Work for You; Prentice Hall, Upper Saddle River, NJ; 1995; ISBN 10: 0201633302; 368 pages.
Crosby, Philip B.; Quality is Free; New American Library, Mentor Books, New York, NY; 1979; 270 pages.
Everett, Gerald D. And McLeod, Raymond; Software Testing; John Wiley & Sons, Hoboken, NJ; 2007; ISBN 978-0-471-79371-7; 261 pages.
Gack, Gary; Applying Six Sigma to Software Implementation Projects; http://software.isixsigma.com/library/content/c040915b.asp.
Gilb, Tom and Graham, Dorothy; Software Inspections; Addison Wesley, Reading, MA; 1993; ISBN 10: 0201631814.
Hallowell, David L.; Six Sigma Software Metrics, Part 1.; http://software.isixsigma.com/library/content/03910a.asp.
International Organization for Standards; ISO 9000 / ISO 14000; http://www.iso.org/iso/en/iso9000-14000/index.html.
Jones, Capers; Software Quality – Analysis and Guidelines for Success; International Thomson Computer Press, Boston, MA; ISBN 1-85032-876-6; 1997; 492 pages.
Kan, Stephen H.; Metrics and Models in Software Quality Engineering, 2nd edition; Addison Wesley Longman, Boston, MA; ISBN 0-201-72915-6; 2003; 528 pages.
Land, Susan K; Smith, Douglas B; Walz, John Z; Practical Support for Lean Six Sigma Software Process Definition: Using IEEE Software Engineering Standards; WileyBlackwell; 2008; ISBN 10: 0470170808; 312 pages.
Mosley, Daniel J.; The Handbook of MIS Application Software Testing; Yourdon Press, Prentice Hall; Englewood Cliffs, NJ; 1993; ISBN 0-13-907007-9; 354 pages.
Myers, Glenford; The Art of Software Testing; John Wiley & Sons, New York; 1979; ISBN 0-471-04328-1; 177 pages.
Nandyal; Raghav; Making Sense of Software Quality Assurance; Tata McGraw Hill Publishing, New Delhi, India; 2007; ISBN 0-07-063378-9; 350 pages.
Radice, Ronald A.; High Qualitiy Low Cost Software Inspections; Paradoxicon Publishingl Andover, MA; ISBN 0-9645913-1-6; 2002; 479 pages.
Wiegers, Karl E.; Peer Reviews in Software – A Practical Guide; Addison Wesley Longman, Boston, MA; ISBN 0-201-73485-0; 2002; 232 pages.
Books and Readings on Software Security, Hacking, and Malware Prevention
Acohido, Byron and Swartz, John: Zero Day Threat: The Shocking Truth of How Banks and Credit Bureaus Help Cyber Crooks Steal Your Money and Identity; Union Square Press; ISBN 10: 140275695X; 2008; 304 pages.
Allen, Julia; Barnum, Sean; Ellison, Robert; McGraw, Gary; and Mead, Nancy; Software Security: A Guide for Project Managers (An SEI book sponsored by the Department of Homeland Security); Addison Wesley Professional, Boston, MA; ISBN 032150917X; 2008.
Anley, Chris, Heasman, John, Lindner, Felix, and Richarte, Gerardo; The Shellcoders Handbook: Discovering and Exploiting Security Holes; Wiley, New York; ISBN 10: 047008023X; 2007; 718 pages.
Chess, Brian; Secure Programming with Static Analysis; Addison Wesley Professional, Boston, MA; ISBN 10: 0321424778; 2007; 624 pages.
Dowd, Mark, McDonald, John, and Schuh, Justin; The Art of Software Security Assessment: Identifying and Preventing Software Vulnerabilities; Addison Wesley Professional, Boston, Ma; ISBN 10: 0321444426; 2006; 1200 pages.
Ericson, John; Hacking: The Art of Exploitation; 2nd edition; No Starch Press; ISBN 10: 1593271441; 2008; 488 pages.
Gallager, Tom; Landauer, Lawrence; and Jeffries, Brian; Hunting Security Bugs; Microsoft Press, Redmond WA; ISBN 10: 0735621879; 2006; 592 pages.
Hamer-Hodges, Ken; Authorization Oriented Architecture – Open Application Networking and Security in the 21st Century; Auerbach Publications, Philadelphia, PA; to be published in December 2009; ISBN 10: 1439800545; pages nnn. (To be published in 2009)
Hogland, Greg and McGraw, Gary; Exploiting Software: How to Break Code; Addison Wesley Professional, Boston, MA; ISBN 10: 0201786598; 2004; 512 pages.
Hogland, Greg and Butler, Jamie; Rootkits: Exploiting the Windows Kernal; Addison Wesley Professional, Boston, MA; ISBN 10: 0321291349; 2005; 352 pages.
Howard, Michael and Lippner, Steve; The Security Development Lifecycle; Microsoft Press, Redmond, WA; ISBN 10: 0735622140; 2006; 352 pages.
Howard, Michael and LeBlanc, David; Writing Secure Code; Microsoft Press, Redmond, WA; ISBN 10: 0735617228; 2003; 798 pages.
Jones, Andy and Ashenden, Debi; Risk Management for Computer Security: Protecting Your Network and Information Assets; Butterworth-Heinemann; ISBN 10: 0750677953; 2005; 296 pages.
Landoll, Douglas J.; The Security Risk Assessment Handbook: A Complete Guide for Performing Security Risk Assessments; CRC; ISBN 10: 0849339981; 2005; 504 pages.
McGraw, Gary; Software Security – Building Security In; Addison Wesley Professional, Boston, MA; ISBN 10-0321356705; 2006; 448 pages.
Rice, David: Geekonomics: The Real Cost of Insecure Software; Addison Wesley Professional, Boston, MA; ISBN 10: 0321477898; 2007; 384 pages.
Scambray, Joel: Hacking Exposed Windows: Microsoft Windows Security Secrets and Solutions; 3rd edition; McGraw Hill Osborne, New York, NY; ISBN 10: 007149426X; 2007; 451 pages.
Scambray, Joel: Hacking Exposed Web Applications; 2nd edition; McGraw Hill Osborne, New York, NY; ISBN 10: 0072262990; 2006; 520 pages.
Sherwood, John; Clark, Andrew; and Lynas, David; Enterprise Security Architecture: A Business-Driven Approach; CMP; ISBN 10: 157820318X; 2005; 608 pages.
Skudis, Edward and Liston, Tom; Counter Hack Reloaded: A Step-by-Step Guide to Computer Attacks and Effective Defenses; Prentice Hall PTR, Englewood Cliffs, NJ; ISBN 10: 0131481045; 2006; 784 pages.
Skudis, Edward and Zeltzer, Lenny; Malware: Fighting Malicious Code; Prentice Hall PTR, Englewood Cliffs, NJ; ISBN 10: 0131014056; 2003; 676 pages.
Shostack, Adam and Stewart, Andrews; The New School of Information Security; Addison Wesley Professional, Boston, MA; ISBN 10: 0321502787; 2008; 288 pages.
Stuttard, Dafydd and Pinto, Marcus; The Web Application Hackers Handbook: Discovering and Exploiting Security Flaws; Wiley, New York; ISBN 10: 0470170778; 2007; 768 pages.
Szor, Peter; The Art of Computer Virus Research and Defense; Addison Wesley Professional, Boston, Ma; ISBN 10: 0321304543; 2005; 744 pages.
Thompson, Herbert and Chase, Scott: The Software Vulnerability Guide; Charles River Media, Boston, MA; ISBN 10: 1584503580; 2005; 354 pages.
Viega, John and McGraw, Gary; Building Secure Software: How to Avoid Security Problems the Right Way; Addison Wesley Professional, Boston, MA; ISBN 10: 020172152X; 2001; 528 pages.
Whittaker, James A. and Thompson, Herbert H.; How to Break Software Security; Addison Wesley, Boston, MA; ISBN 10: 0321194330; 2003; 208 pages.
Wysopal, Chris; Nelson, Lucas; Zovi, Dino Dai; and Dustin, Elfriede; The Art of Software Security Testing: Identiyfing Software Security Flaws; Addison Wesley Professional, Boston, MA; ISBN 10: 0321304861; 2006; 321 pages.
Books and Readings on Software Engineering and Programming
Barr, Michael and Massa, Anthony; Programming Embedded Systems: With C and GNU Development Tools; O’Reilly Media; 2006; ISBN 10-0596009836.
Beck, K; Extreme Programming Explained: Embrace Change; Addison Wesley, Boston, MA; 1999.
Bott, Frank; Coleman, A; Eaton, J.; Roland, D; Professional Issues in Software Engineering; Taylor & Francis, 2000; ISBN 10: 0748409513; 384 pages.
Glass, Robert L; Facts and Fallacies of Software Engineering (Agile Software Development); Addison Wesley, Boston, MA; 2002; ISBN 10: 0321117425; 224 pages.
Hans, Professor van Vliet; Software Engineering Principles and Practices; 3rd edition; John Wiley & Sons, London, New York; 2008; ISBN 10: 0470031468; 740 pages.
Hunt, Andrew and Thomas, David: The Pragmatic Programmer; Addison Wesley, Boston, MA; 1999; ISBN 10: 020161622X; 352 pages.
Jeffries, R. et al; Extreme Programming Installed; Addison Wesley, Boston; 2001.
Marciniak, John J. (Editor); Encyclopedia of Software Engineering; John Wiley & Sons, New York; 1994; ISBN 0-471-54002; in two volumes.
McConnell, Code Complete; Microsoft Press, Redmond, WA; 1993; ISBN 13-978-1556154843; 886 pages.
Morrison, J. Paul; Flow-Based Programming; A New Approach to Application Development; Van Nostrand Reinhold, New York; 1994; ISBN 0-442-01771-5; 316 pages.
Pressman, Roger; Software Engineering – A Practitioner’s Approach; McGraw Hill, NY; 6th edition, 2005; ISBN 0-07-285318-2.
Sommerville, Ian; Software Engineering; 7th edition; Addison Wesley, Boston, MA; 2004; ISBN 10: 0321210263; 784 pages.
Stephens M. & Rosenberg, D.; Extreme Programming Refactored; The Case Against XP; APress L.P., Berkeley, CA; 2003.
Books and Readings on Software Development Methods
Boehm, Barry; “A Spiral Model of Software Development and Enhancement”; Proceedings of the Int. Workshop on Software Process and Software Environments; ACM Software Engineering Notes, Aug. 1986, pp. 22-42.
Cockburn, Alistair; Agile Software Development; Addison Wesley, Boston, MA; 2001.
Cohen, D. Lindvall M. & Costa, P. “An Introduction to agile methods”; Advances in Computers, pp. 1-66; 2004; Elsevier Science, New York.
Highsmith, Jim; Agile Software Development Ecosystems; Addison Wesley, Boston, MA; 2002.
Humphrey, Watts; TSP – Leading a Development Team; Addison Wesley, Boston, MA; 2006.
Humphrey, Watts; PSP: A Self-Improvement Process for Software Engineers; Addison Wesley, Upper Saddle River, NJ; 2005; ISBN 0-321-30549-3; 346 pages.
Krutchen, Phillippe; The Rational Unified Process – An Introduction; Addison Wesley, Boston, MA;; 2003; ISBN 10: 0321197704; 336 pages.
Larman, Craig & Basili, Victor; “Iterative and Incremental Development – A Brief History”; IEEE Computer Society; June 2003; pp 47-55.
Love, Tom; Object Lessons; SIGS Books, New York; 1993.
Martin, Robert; Agile Software Development: Principles, Patterns, and Practices; Prentice Hall, Upper Saddle River, NJ; 2002; ISBN 10: 0135974445; 529 pages.
Mills, H.; Dyer, M. & Linger, R.; “Cleanroom Software Engineering”; IEEE Software; 4, 5 (Sept. 1987); pp. 19-25.
Paulk Mark et al; The Capability Maturity Model; Guidelines for Improving the Software Process; Addison Wesley, Reading, MA; ISBN 0-201-54664-7; 1995; 439 pages.
Rapid Application Development; http://en.wikipedia.org/wiki/Rapid_application_development
Stapleton, J.; DSDM – Dynamic System Development Method in Practice; Addison Wesley; Boston, MA; 1997.
Books and Readings on Software Deployment, Customer Support, and Maintenance
Arnold, Robert S.; Software Reengineering; IEEE Computer Society Press, Los Alamitos, CA; 1993; ISBN 0-8186-3272-0; 600 pages.
Arthur, Lowell Jay; Software Evolution – The Software Maintenance Challenge; John Wiley & Sons, New York; 1988; ISBN 0-471-62871-9; 254 pages.
Gallagher, R.S.; Effective Customer Support; International Thomson Computer Press, Boston, MA; 1997; ISBN 1-85032-209-0; 480 pages.
Parikh, Girish; Handbook of Software Maintenance; John Wiley & Sons, New York; 1986; ISBN 0-471-82813-0; 421 pages.
Pigoski, Thomas M.; Practical Software Maintenance – Best Practices for Managing Your Software Investment; IEEE Computer Society Press, Los Alamitos, CA; 1997; ISBN 0-471-17001-1; 400 pages.
Sharon, David; Managing Systems in Transition – A Pragmatic View of Reengineering Methods; International Thomson Computer Press, Boston, MA; 1996; ISBN 1-85032-194-9; 300 pages.
Takang, Armstrong and Grubh, Penny; Software Maintenance Concepts and Practice; International Thomson Computer Press, Boston, MA; 1997; ISBN 1-85032-192-2; 256 pages.
Ulrich, William M.; Legacy Systems: Transformation Strategies; Prentice Hall, Upper Saddle River, NJ; 2002; ISBN 10: 013044927X; 448 pages.
Books and Readings on Social Issues in Software Engineering
Brooks, Fred; The Mythical Manmonth; 2nd edition; Addison Wesley, Boston, MA; 1995; ISBN 10: 0201835959; 336 pages.
DeMarco, Tom; Peopleware: Productive Projects and Teams; Dorset House, New York, NY; 1999; ISBN 10: 0932633439; 245 pages.
Glass, Robert L; Software Creativity; 2nd edition; developer.*books, Atlanta, GA; 2006; ISBN 978-0-9772133-1-3; 456 pages.
Humphrey, Watts; Winning with Software: An Executive Strategy; Addison Wesley, Boston, MA; 2002; ISBN 0-201-77639-1; 228 pages.
Johnson, James et al; The Chaos Report; The Standish Group, West Yarmouth, MA; 2007.
Jones, Capers; “How Software Personnel Learn New Skills”, 6th edition; July 2008; Capers Jones & Associates LLC; Narragansett, RI.; 25 pages (monograph).
Jones, Capers; Conflict and Litigation Between Software Clients and Developers; Software Productivity Research, Inc.; Narragansett, RI; 2008; 45 pages (monograph).
Jones, Capers; “Preventing Software Failure: Problems Noted in Breach of Contract Litigation”; Capers Jones & Associates, Narragansett, RI; 2008; 25 pages.
Kuhn, Thomas; The Structure of Scientific Revolutions; University of Chicago Press, Chicago, IL; 1996; ISBN 0-22645807-5; 212 pages.
Krasner, Herb; “Accumulating the Body of Evidence for the Payoff of Software Process Improvement – 1997;” Krasner Consulting, Austin, TX.
Starr, Paul; The Social Transformation of American Medicine; Basic Books; Perseus Group; 1982; ISBN 0-465-07834-2.
Weinberg, Gerald M.; The Psychology of Computer Programming; Van Nostrand Reinhold, New York; 1971; ISBN 0-442-29264-3; 288 pages.
Weinberg, Gerald M; Becoming a Technical Leader; Dorset House; New York; 1986; ISBN 0-932633-02-1; 284 pages.
Yourdon, Ed; Death March – The Complete Software Developer’s Guide to Surviving “Mission Impossible” Projects; Prentice Hall PTR, Upper Saddle River, NJ; ISBN 0-13-748310-4; 1997; 218 pages.
Zoellick; Bill; CyberRegs – A Business Guide to Web Property, Privacy, and Patents; Addison Wesley, Boston, MA; 2002; ISBN 0-201-72230-5; 307 pages.
WEB SITES
There are hundreds of software industry and professional associations. Most have a narrow focus. Most are more or less isolated and have no contact with similar associations. Exceptions to this rule include the various software process improvement network groups (SPIN) and the various software metrics associations.
This partial listing of software organizations and web sites is to facilitate communication and sharing of data across both organization and national boundaries. Software is a global industry. Problems occur from the first day of requirements to the last day of usage, and every day in between. Therefore mutual cooperation across industry and technical boundaries would benefit software and help it toward becoming a true profession rather than a craft of marginal competence.
What might be useful for the software industry would be reciprocal memberships among the major professional associations along the lines of the American Medical Association. There is a need for an umbrella organization that deals with all aspects of software as a profession, as does the AMA for medical practice.
American Electronics Association (AEA): www.aeanet.org
(May merge with ITAA)
American Society for Quality: www.ASQ.org
Anti-Phishing Working Group: www.antiphishing.org
Association of Computing Machinery: www.ACM.org
Association of Competitive Technologies (ACT): www.actonline.org
Association of Information Technology Professionals: www.aitp.org
Association for Software Testing: www.associationforsoftwaretesting.org
Brazilian Function Point Users Group: www.BFPUG.org
Business Application Software Developers Association: www.basda.org
Business Software Alliance (BSA): www.bsa.org
Center for Internet Security: www.cisecurity.org
China Software Industry Association: www.CSIA.org
Chinese Software Professional Association: www.CSPA.com
Center for Hybrid and Embedded Software Systems (CHESS): http//chess.eecs.berkley.edu
Computing Technology Industry Association (CTIA): www.comptia.org
Embedded Software Association (ESA): www.esofta.com
European Design and Automation Association (EDAA): www.edaa.com
Finnish Software Measurement Association: www.fisma.fi
IEEE Computer Society: www.computer.org
Independent Computer Consultants Association (ICCA): www.icca.org
Information Technology Association of America (ITAA): www.itaa.org
(May merge with AEA)
Information Technology Metrics and Productivity Institute (ITMPI): www.ITMPI.org
InfraGuard: www.InfraGuard.net
Institute of Electrical and Electronics Engineers (IEEE): www.IEEE.org
Institute for International Research (IIR): eee.irusa.com
International Association of Software Architects: www.IASAHOME.org
International Software Benchmarking Standards Group (ISBSG): www.ISBSG.org
International Function Point Users Group (IFPUG): www.IFPUG.org
International Institute of Business Analysis: www.IIBAorg
Japan Function Point Users Group: www.jfpug.org
Linux Professional Institute: www.lpi.org
National Association of Software and Service Companies (India): www.NASCOM.in
Netherlands Software Metrics Association: www.NESMA.org
Process Fusion: www.process-fusion.com
Programmers’ Guild: www.programmersguild.org
Project Management Institute: www.PMI.org
Russian Software Development Organization (RUSSOFT): www.russoft.org
Society of Information Management (SIM): www.simnet.org
Software and Information Industry Association: www.siia.net
Software Engineering Body of Knowledge www.swebok.org
Software Publishers Association (SPA): www.spa.org
Software Engineering Institute (SEI): www.sei.cmu.edu
Software Productivity Research (SPR): www.SPR.com
United Kingdom Software Metrics Association: www.UKSMA.org
U.S. Internet Industry Association (USIIA): www.usiia.org
Women in Technology International: www.witi.com
APPENDIX B: SURVEY OF BENCHMARK USAGE AND INTERESTS
| SOFTWARE BENCHMARK USAGE SURVEY | |||
| Version 2.0 3/7/2010 | |||
|
No |
Used |
Would |
|
|
Interest |
Today |
Use if |
|
|
|
Available |
||
|
“0” |
“1” |
“2” |
|
| 1. Portfolio benchmarks | |||
| 2. Industry benchmarks (banks, insurance, defense, etc.) | |||
| 3. International benchmarks (US, UK, Japan, China, etc.) | |||
| 4. Application class bench marks (embedded, systems, IT, etc.) | |||
| 5. Application size benchmarks (1, 10, 100, 1000, function points etc.) | |||
| 6. Requirements creep benchmarks (monthly rates of change) | |||
| 7. Data center and operations benchmarks (availability, MTTF, etc.) | |||
| 8. Data quality benchmarks | |||
| 9. Data base volume benchmarks | |||
| 10. Staffing and specialization benchmarks | |||
| 11. Staff turnover and attrition benchmarks | |||
| 12. Staff compensation benchmarks | |||
| 13. Organization structure benchmarks (matrix, small team, Agile, etc.) | |||
| 14. Development productivity benchmarks | |||
| 15. Software quality benchmarks | |||
| 16. Software security benchmarks (cost of prevention, recovery, etc.) | |||
| 17. Maintenance and support benchmarks | |||
| 18. Legacy renovation benchmarks | |||
| 19. Total cost of ownership (TCO) benchmarks | |||
| 20. Cost of quality (COQ) benchmarks | |||
| 21. Customer satisfaction benchmarks | |||
| 22. Methodology benchmarks (Agile, RUP, TSP, etc.) | |||
| 23. Tool usage benchmarks (Project management, static analysis, etc.) | |||
| 24. Reusability benchmarks (volumes of various reusable deliverables) | |||
| 25. Software usage benchmarks (by occupation, by function) | |||
| 26. Outsource results benchmarks (domestic) | |||
| 27. Outsource results benchmarks (international) | |||
| 28. Schedule slip benchmarks | |||
| 29. Cost overrun benchmarks | |||
| 30. Project failure benchmarks (from litigation records) | |||
| 31. Litigation cost benchmarks – breach of contract | |||
| 32. Litigation cost benchmarks – taxable value of software | |||
| 33. Litigation cost benchmarks – non competition violations | |||
| 34. Litigation cost benchmarks – damages due to poor quality | |||
| 35. Litigation cost benchmarks – intellectual property |
APPENDIX C: A NEW FORM OF SOFTWARE BENCHMARK
Introduction
Normally software assessments and benchmarks are provided to specific companies and compare selected applications against similar applications from other companies in the same industry. This is useful information, but it does not provide any context or any information about the industry itself.
What would be both useful and salable would be a new kind of benchmark that would consolidate information about specific industries, the major companies within the industries, the software used by those companies, and also productivity and quality ranges derived from assessment and benchmark studies.
Using “banking” as an industry example here are some 55 kinds of assessment and benchmark information that would be provided:
Table 1: Fifty Five Data Topics for Industry-Specific Software Benchmarks
- Number of large companies (source = Hoover’s Guides to business)
- Number of medium companies (source = Hoover’s Guides to business)
- Number of small companies (source = Hoover’s Guides to business)
- Regulatory agencies that control business sectors
- Standards and safety regulations for the industry
- Supply chains and related industries
- Current government investigations involving the industry
- Mergers and acquisitions within the industry
- Start-up companies within the industry
- Business failures, government takeovers, or bankruptcies within the industry
- Recent patents and intellectual property filed by industry members
- Ranges of industry profitability and economic health
- Current litigation involving the industry
- Domestic competitive situation of the major players within the industry
- Global competitive situation of the major players within the industry
- Professional associations that serve the industry
- Major kinds of hardware platforms used within the industry
- Major kinds of data utilized within the industry
- Major kinds of software applications utilized by the industry
- Major kinds of ERP applications utilized by the industry
- Major COTS vendors that provide packages to the industry
- Major open-source applications utilized by the industry
- Major outsource vendors that service the industry
- Major sources of reusable components serving the industry
- Ranges of portfolio and application sizes within the industry
- Ranges of data base and data warehouse sizes within the industry
- Numbers of software users within the industry
- Number of customer organizations served by software
- Number of actual clients or users served by software
- Numbers of software developers within the industry
- Numbers of maintenance personnel within the industry
- Numbers of technical specialists within the industry (quality, testing, etc.)
- Rates of change for software personnel (expanding, shrinking, stable)
- Software security issues within the industry
- Statistics on data theft, denial of service, and other security breaches
- Security policies, standards, and best practices for the industry
- Software development productivity benchmarks with function points
- Software maintenance/enhancement productivity benchmarks (function points)
- Software total cost of ownership (TCO) benchmarks with function points
- Software quality benchmarks with function points
- Software cancelled project benchmarks by size in function points and type
- Software costs and schedule overruns within the industry
- Legacy application replacement strategy within the industry
- Distribution of CMMI levels within the industry
- Distribution of TickIt scores or maintainability scores within the industry
- Major development methods used within the industry
- Major maintenance methods used within the industry
- Typical tool suites used within the industry
- Best practices utilized by the industry
- Average practices utilized by the industry
- Worst practices utilized by the industry
- Major quality control methods used within the industry
- Future industry technology trends (cloud computing, SOA, etc.)
- Future industry software trends
- Major sources of industry data (web sites; periodicals, etc.)
For a bank to assemble all of this information by itself it would be necessary to gather data from about a dozen industry and government sources plus probably commissioning benchmarks on a sample of 10 to more than 25 applications. Competitive information from other banks would not be accessible. Essentially this kind of information would not be gathered by individual banks because of a lack of organizational focus, plus the rather high costs involved.
Probable Clients for Software Benchmarks by Industry
As of 2011 the software benchmark business sector is divided into two subsets. One form of benchmark uses fairly simple questionnaires with the data being self-reported by clients.
Because self-reported benchmarks have no fees for providing information and consolidated benchmark reports are available for low cost, this form of benchmarking is widespread and popular. The International Software Benchmark Standards Group (ISBSG) is the major provider of self-reported benchmarks. The ISBSG clients consist mainly of project managers and some CIO’s.
The second form of software benchmark uses more complicated questionnaires and also includes on-site data collection in order to gather and validate quantitative and qualitative information from samples of 10 to more than 25 projects at the same time.
These on-site benchmarks usually include non-disclosure agreements for data collection and distribution so the data is delivered to specific companies.
Because the costs for collecting and analyzing the data ranges from $25,000 to more than $50,000 these benchmarks require approval and funding from the level of a CIO or a CTO. The reports back to clients are of course used by first-line and project managers, but the funding is usually provided by a higher level of management.
Software assessments are also on-site consulting studies. For assessments using the model of the Software Engineering Institute (SEI) certified assessors are used. For other forms of assessment such as the author’s, trained consultants are used. Here too the costs are fairly expensive and in the $25,000 to $75,000 range.
The data that would be assembled for the new kind of benchmarks discussed in this report would include a combination of self-reported data, on-site data collection, and subscriptions to a number of industry information providers such as Hoover Business Guides, Gartner Group, the Department of Commerce, plus access to other government statistics as well. At a nominal charge of $10,000 for such a benchmark report, funding approval would probably be at the CIO and CTO level.
While the costs of these benchmarks are less than the cost of today’s on-site benchmarks and assessments for individual clients, these new kinds of benchmarks could be offered to dozens or even hundreds of clients so they would generate much greater revenues and profits than conventional single-client benchmarks.
In order to be useful, the benchmark reports would consolidate data from at least 10 companies and 100 projects within an industry, and then use extrapolation to cover other companies within the same industry. Of course larger samples would be desirable. Remote data might ge gathered from 100 banks or more, while on-site data might be gathered from 20 banks or more.
The on-site data collection would probably be derived from conventional fee-based studies that provide information to specific clients. However once the data is sanitized and aggregated, it would be marketed to multiple clients.
Because of the richness of the data provided, these new benchmarks would attract a much wider and more diverse set of clients than normal self-reported or on-site software benchmarks. Obviously the information would be of use to CIO’s and CTO’s, but because of the in-depth industry coverage the information would also be of use to CEO’s and to client executives as well.
For example these new benchmarks would be of use to VP’s of marketing, sales, manufacturing, human resources, and research and development. In addition the information would no doubt be acquired by major consulting companies, by law firms that specialize in software litigation, by outsource vendors, and by other kinds of information providers such as journals and web sites.
In addition the new form of benchmark would also be useful to many related companies that provide services or products to banking clients: outsource vendors, software vendors, consulting companies, equipment manufacturers, personnel companies, major law firms, and government regulators. In fact sales of the new kind of benchmark to these ancillary companies would probably exceed sales to the banking community itself. For each benchmark study acquired by a bank, probably at least three studies would be acquired by banking service and product providers. This is a new and previously untapped market for benchmark studies.
The basic idea of the new form of benchmark is to elevate the value of benchmark information from data that is “useful but not essential” to the level of “we must have this information to stay competitive.” A second goal is to elevate the target audience of the benchmark information from project managers, CIO’s, and CTO’s up the level of CEO’s and senior operating unit executives.
Once this new form of benchmark is launched, it will probably lead to a significant increase in other forms of benchmarking.
It is obvious that the initial launch within an industry such as banking needs to attract a fairly significant number of clients. Therefore the new form of benchmark should start with and industry where such information is already perceived as valuable; i.e. banks, defense, insurance, health care, medical equipment, and the like.
Once launched in the United States, these new benchmarks would certainly lead to an increase in overseas benchmarks using the same formats and data collection methods. However to facilitate overseas benchmark data collection, local subcontractors would probably be a desirable method of proceeding.
In addition, some overseas data might be gathered via on-line methods such as Skype, remote surveys, and perhaps Wiki sites. In fact a virtual benchmark environment using the same technology as Second Life is technically possible. In such an environment avatars of consultants and clients might have conversations and discuss data gathering methods without actual travel.
It is apparent that once the expanded benchmarks start being created, continuous collection of data and continuous updates will become part of the benchmark and assessment process.
Expanding the Sources of Benchmark Data
Currently the actual personnel who provide data for both assessments and benchmarks are primarily software engineers and technical workers, project managers, some higher-level managers, and occasionally executives at the level of CIO or CTO.
Once the value of the expanded benchmarks becomes apparent, it can be anticipated that additional information might be collected from a much wider variety of stakeholders, executives, software personnel, and actual users of software:
Table 2: Executive Sources of Benchmark Information
- Corporate CEO’s
- Boards of directors or advisory boards
- Government executives (state and Federal CIO’s, agency chiefs, etc.)
- Operating unit VP’s (manufacturing, finance, etc.)
- Agile embedded stakeholders
- CIO’s for information systems
- CTO’s for systems software
- CTO’s for embedded applications and hybrid systems
- Customer executives who approve and acquire software
- Outsource executives
- Corporate attorneys
- User association executives
In addition, technical information might be acquired from a larger spectrum of software technical personnel than is normal for today’s assessments and benchmarks:
Table 3: Technical Sources of Benchmark Information
- Architects
- Business analysts
- Data base analysts
- Software quality assurance (SQA)
- Project Office personnel
- Test personnel
- Scrum Masters
- Integration and configuration control personnel
- Scrum masters
- Embedded stakeholders
- Software users and clients
- Customer support personnel
A combination of remote interviews using simple questionnaire, more detailed questionnaires for on-site use, and conference calls would be used to gather the expanded forms of information from an expanded set of software stakeholders and software production personnel.
The basic idea is to consolidate information on what stakeholders, top executives, development personnel, maintenance personnel, quality assurance personnel, and actual clients or users think about software applications and the methods used to create them.
The expansion of information sources will obviously cause some additional effort in gathering information. But after a few months of trials and tuning, hopefully the additional effort will not cause more than about a 25% increase in total data collection effort.
International Software Benchmarks by Industry
As of 2011 there are benchmark studies and assessments performed overseas, and there are a few reports that compare performance by both country and industry. Of the ones that attempt to do so, sample sizes are small and the results are somewhat marginal in terms of economic breadth and reliability.
There is a very strong demand for reliable international benchmarks that would show the comparative performance of the countries where software production is a major economic topic. The probable revenues for international benchmarks would be proportional to the size of the software industries within the countries. Due to the logistical issues of carrying out on-site international benchmarks, the global market for such benchmarks would probably be three to five years behind the U.S. market.
That being said, the revenues from international software benchmarks by industry would probably be sufficient to fund expansion throughout the world. The more countries that provide data, the more valuable the overall collection of data will become.
Following are a few hypothetical examples of potential annual benchmark revenues by about 2020, assuming U.S. benchmarks as described in this report begin in 2012:
Table 4: Probable Software Benchmark Revenues by Country Circa 2020
China $45,000,000
India $45,000,000
Japan $35,000,000
Russia $30,000,000
Brazil $25,000,000
U.K. $20,000,000
Germany $20,000,000
France $20,000,000
Italy $15,000,000
Ukraine $10,000,000
Spain $10,000,000
Scandinavia $10,000,000
Australia $10,000,000
Mexico $10,000,000
South Korea $10,000,000
Canada $7,000,000
Taiwan $7,000,000
Israel $5,000,000
Netherlands $3,000,000
Belgium $3,000,000
TOTAL $340,000,000
Of course all of these countries could not be studied at the same time, but eventually the value of expanded global and industry software benchmarks has the potential to create a significant body of knowledge about software in every industrialized country and every major industry.
Over and above the countries shown in table 2 benchmarks might also be provided for many other countries in Asia, Central and South America, Africa, and the Pacific regions such as Singapore and Malaysia.
Effort Needed to Create Benchmarks by Industry
Assembling the initial information needed to produce this example of an industry benchmark for banking would probably require about 90 days of full-time effort by at least one capable researcher. Two would be better in order to provide backup in case of illness.
The initial data collection would also require fee-based subscriptions to a variety of data sources such as Hoover, Gartner Group, the ISBSG, and other industrial information sources. However some of these sources of data provide short-term trials, so the initial expense would be fairly low for commercial data sources.
The probable initial cost of such an industry benchmark in a single industry such as banking would probably be in the range of $150,000. This includes data collection from clients, commercial information providers, analysis of the data, and production of an initial report.
Later expenses would include development of a web site, marketing materials, and other collateral materials that are not absolutely needed for the initial benchmark report.
Because the data in such a benchmark is dynamic and changes rapidly, continuous updates would be needed to keep the information current. Probably 36 days per year would be needed to refresh the information (i.e. 3 days per month per industry). Monthly or quarterly updates would be provided to clients.
The value of this kind of benchmark compendium would be high enough so that the benchmark report might be marketed at perhaps $10,000 for the initial report and annual subscription fees for updates of perhaps $2,500. Extracts and subsets from the report could be marketed individually for costs in the range of $500. These would appeal to smaller companies within the industry.
Information would also be made available via the web, with some samples of the data being provided for free, but extended data being fee-based. The fees could either be on a per-use basis or an annual subscription basis.
Another possible deliverable would be a daily web site entitled “Software Daily News” that resembles the very useful and popular current website entitled “Science Daily News.” The science web site covers a wide range of scientific disciplines ranging from archeology through astrophysics and includes both news summaries and full-length articles.
For a major industry such as banking, a benchmark report of this nature might attract about 250 domestic banks and 75 overseas banks. It would also go to government agencies and major software vendors.
Each industry benchmark report would probably generate about $2,500,000 in the U.S. and about $750,000 abroad, or $2,750,000. Recurring revenues would amount to perhaps $400,000 per year per industry per year. If there were 10 industries supported the revenues would ascend to more than $27,500,000 for initial subscriptions and more than $4,000,000 per year in recurring revenues.
Eventually on a global level these new benchmarks might have more than 5,000 major corporate clients, more than 1,000 government clients, and a large but unpredictable set of ancillary clients such as law firms, other consulting groups, universities, and the like.
Obviously such benchmarks would be most useful for industries that have a large amount of effort in the area of software and data processing. These industries include but are not limited to:
Table 5: Industry Candidates for Software Benchmarks
- Aerospace
- Agriculture
- Airlines
- Automotive
- Banking
- Chemicals
- Computers and peripheral equipment
- Cruise lines
- Defense
- Education – university
- Education – primary and secondary
- Entertainment
- Energy and oil
- Governments – state
- Governments – Federal
- Government – municipal
- Health care
- Hotels
- Insurance
- Manufacturing
- Open-source
- Process control
- Pharmaceuticals
- Public Utilities
- Publishing
- Retail trade
- Software
- Telecommunications
- Transportation
- Wholesale trade
A full suite of such benchmarks for major industries would probably generate in the range of $30,000,000 to $50,000,000 per year from U.S. clients. A team of perhaps 10 researchers and 5 logistical support personnel, plus licenses and subscriptions for external sources of data.
If the company producing the benchmark reports also collected benchmark and assessment data itself, probably another 10 consultants would be needed. The data collection would probably generate about $5,000,000 to $10,000,000 per year.
Data collection could also be subcontracted to existing benchmark groups such as ISBSG, Software Productivity Research (SPR), the Davids Consulting Group, and the like. Marketing and sales personnel plus a small executive contingent would be needed. The total size would probably be close to 35 total personnel. However subcontracts for collecting benchmark data might be issued to more than 25 companies in more than 25 countries.
It is conceivable that within 10 years of the initial launch, the new form of benchmark might involve more than 300 subcontract personnel in more than 25 countries.
The annual cost for operating such the core group would probably be in the range of $4,000,000 per year. However except for the $150,000 investment for the initial report, the organization should be self-sustaining and profitable because no heavy capital investments are needed.
The organization might generate revenues for benchmark subscriptions from eventually more than 500 domestic companies and perhaps 300 overseas companies and perhaps 100 government agencies. Assuming three to five subscriptions per company revenues from benchmark subscriptions might be in the range of $10,000,000 to $15,000,000 per year, plus the initial cost for each subscription.
The consulting work of collecting data on a fee basis would probably bring in revenues of perhaps $8,000,000 per year. Total revenues from all sources might total $30,000,000 to $50,000,000 per year.
By contrast as of 2011 software benchmarks are a niche industry with perhaps 20 U.S. companies collecting data, with combined annual revenues that probably are below $20,000,000. This is because benchmarks are perceived by clients as useful but not essential.
The business idea behind the new form of benchmark is to elevate the importance from being perceived as useful to being perceived as essential. To achieve this goal, more and better information needs to be provided to clients than is currently made available.
As useful as today’s benchmarks are for productivity and quality studies, the lack of context information limits the utility of benchmarks and restricts the potential audience.
Additional Tools and Services in Addition to Benchmarks and Assessments
In addition to marketing benchmarks and performing assessment and benchmark consulting studies, the organization would also be positioned to market and perhaps develop several kinds of tools and ancillary products and services:
- Benchmark tools for data collection that clients might use to facilitate their own data collection and analysis.
Predictive tools for estimating software schedules, efforts, costs, quality, reliability, maintenance, and enhancements.
Special studies that evaluate important topics. For example large-scale studies that compared a number of development methods such as waterfall, iterative, object-oriented, Agile, RUP, TSP, XP, and others would be both useful to the industry and highly salable.
Training in skills which the benchmarks demonstrate need improvement within an industry such as quality control, test case design, change control, legacy migration, and a number of others.
- Portfolio analyses are seldom performed because of the difficulty of sizing and analyzing as many as 3,000 applications that might total more than 7,500,000 function points. Further, between 25% and more than 50% of typical portfolios are in the form of COTS packages or open-source software which cannot be sized using standard methods because the vendors do not provide the needed inputs. It is possible to size a portfolio using one or more of the new high-speed function point sizing methods. Portfolio studies would no doubt be covered by non-disclosure agreements and hence marketed only to specific companies.
- Litigation support depends upon accurate data to determine industry averages for quality, productivity, and other benchmarks. The kinds of data discussed herein would probably be widely used as background information in breach of contract litigation between clients and outsource vendors. It would also be used in litigation against software companies for poor quality or damages.
- Outsource contracts frequently include clauses dealing with quality, schedule, productivity, reliability, change control, and the like. Without accurate benchmark data some contracts contain clauses that are probably impossible to achieve. Accurate benchmarks would be very useful for developing outsource contracts that are mutually agreeable.
- Mergers and acquisitions occur almost daily in many industries.. The data in this new form of benchmark would be of significant interest to business brokers and companies considering either acquisition or being acquired. It would also be of interest to companies that are merely seeking partnerships, distributors, subcontractors, or potential clients.
- Venture-backed start-up businesses have been in decline due to the recession, but are starting to ramp up again. The data contained in this new form of benchmark report should be of some interest to both entrepreneurs and venture capitalists considering starting new businesses.
Tool and special study revenues would be over and above the revenues already discussed. They are currently unpredictable because the suite of tools is not fully defined. However annual revenues in excess of $10,000,000 per tool would not be uncommon. Special studies could easily top $ 35,000,000 per year.
Benchmark Company Exit Strategies
Normally companies have four possible end games: 1) they go public; 2) they are acquired by larger companies; 3) they go out of business 4) they continue indefinitely with only marginal growth..
Options 1 and 2 are the most likely end games for such a benchmark organization as the one described here. While such benchmarks might be created by a non-profit group or perhaps by a university, it is more likely that a for-profit organization would be the best choice.
A for-profit company is most likely because if the idea of the new form of benchmark expands and becomes successful, the benchmark production group would be an attractive acquisition candidate for large data providers such as Gartner Group, Accenture, Google, or similar large corporations where information is a valuable commodity..
Software Benchmarks Circa 2011
As of early 2011 none of the normal benchmark sources or consulting companies provide this kind of information so there is no real competition. Gartner Group and the International Data Group (IDG) provide subsets of the kinds of information discussed, but not actual benchmarks.
- The International Software Benchmark Standards Group (ISBSG) provides useful benchmarks, but does not have any data on large applications > 10,000 function points in size.
- The Software Engineering Institute (SEI) provides assessments, but is sparse with benchmark data and provides little or no information about companies, industries, data, and other key topics.
- The Information Technology Metrics and Productivity Institute (ITMPI) provides many useful reports and webinars on specific topics, but does not provide assessment and benchmark information or any context information about various industry segments. Some of the CAI tools might well be useful for collecting benchmark data.
- The government-sponsored Data Analysis Center for Software (DACS) provides useful information in a government and defense context, but no benchmarks or assessments.
- The Standish Group publishes interesting statistics on software failures, but does not provide conventional benchmarks and assessments.
- A number of privately held consulting companies such as the Davids’ Consulting Group, Quantitative Software Management (QSM), Software Productivity Research (SPR), and the Software Improvement Group (SIG) in Amsterdam and several others provide benchmarks and assessments for individual clients. These groups occasionally publish studies using data from multiple clients, but sample sizes are fairly small.
- Universities tend to provide small-scale studies on specific topics but are not funded or equipped to produce large-scale industry-wide studies.
- The software metrics associations such as IFPUG and COSMIC provide the current rules for counting functional metrics, but seldom produce benchmarks and they don’t do assessments at all.
The bottom line is that the new kind of benchmark discussed in this report has little competition circa 2011. Gartner Group is the best positioned to compete, but to date has not attempted specific software benchmarks or assessments. Gartner Group aims at CEO’s and top executives, but does not get down to the level of assessments and benchmarks.
It is apparent that the overall benchmark that contains all forms of data shown in this report might be subset into special purpose reports that might be marketed separately or offered via subscription to specialized communities. Examples of such special reports might include, but are not limited to:
Table 6: Special Benchmark Reports
- Software Quality Assurance (SQA)
- Software Estimation Tool Analysis
- Software Development
- Software Maintenance and Enhancement
- Software Security and Safety
- Software Failure Rates by Industry
- Software Project Management
- Software Project Offices
- Data Quality by Industry
- Data Base Development
- Web Site Development
- Software Process Improvement
- Software Education and Training
- Analysis of Cloud Computing
- Best Practice Analysis Based on Empirical Results
These separate subsets would clearly generate additional revenues over and above those discussed for the entire assessment and benchmark report. However it is premature to attempt to quantify the numbers of subscribers and revenues for these subsets of information.
Summary and Conclusions about Software Benchmarks
The software industry circa 2011 is characterized by many project failures, by frequent cost and schedule overruns, by poor quality, and by lack of reliable information as to what the phrase “best practices” really means in terms of results.
The software industry is also subject to frequent fads and fallacies as new development methods surface, are touted as panaceas, and then gradually demonstrate only marginal improvements if any over alternate development methods. Poor measurement practices and inadequate benchmarks are what makes these fads and fallacies endemic problems for the software community.
Benchmarks are very useful methods for minimizing these common problems, but less than 30% of large U.S. software companies have either commissioned benchmarks or use benchmark data. For small and mid-sized U.S. companies, less than 10% utilize benchmark data. In fact many corporations not only fail to use benchmarks and assessments, they have never even heard of the SEI, ISBSG, ITMPI, and the other major sources of assessment and benchmark information.
The new form of benchmark discussed in this paper is aimed at expanding the information contained in software benchmarks from basic productivity and quality levels up to a complete condensation of critical industry topics where software is part of the equation. The additional kinds of data and information will hopefully elevate benchmarks and assessments from useful but optional studies to mandatory business practices that are demanded by a majority of CEO’s and top operational executives.
Once such information is published for a specific industry such as banking, it is expected that demands from other industries will drive the business growth for similar benchmarks in other industries.
The goal for the new form of benchmark is to reach close to 100% of major corporations, more than 50% of medium corporations, and perhaps 25% of smaller corporations. It is difficult to predict government penetration at all levels, but no doubt all 50 states would want to subscribe to this new form of benchmark if state government data were included. A number of Federal agencies would also want to have access to the kinds of data provided by the new benchmarks.
It is hoped that the kinds of information included in these new benchmarks will not only lead to a profitable business, but will assist the software industry in overcoming its traditional problems of cost and schedule overruns combined with poor quality.