CRA Reference

Selection criteria and evaluation protocol

Primary audience:Leadership

Published on

The weighted grid

Weight it once, in committee, before looking at products — otherwise the weighting quietly adjusts itself to the favourite.

Criterion Question Indicative weight
Ecosystem coverage Does it cover the languages and platforms you actually use? 20 %
Fidelity of the SBOMs produced Measured gap against a cross-check, on your artefacts 15 %
Correlation quality False positive rate measured on a known sample 15 %
VEX and CSAF support Can decisions be documented and published? 10 %
Long-term archiving History retained, full export possible 10 %
API and automation Is everything drivable without a graphical interface? 10 %
Sovereignty Where is the data hosted? What leaves the Union? 8 %
Reversibility Can you leave with your SBOMs, history and VEX statements? 5 %
Legal needs Attributions, licence reports, exception register 5 %
Total cost over five years Licence, operations, integration, training Disqualifying above budget

The two most commonly under-weighted criteria are reversibility and operating cost. Both are paid at the end.

The evaluation protocol

A credible evaluation runs on your artefacts, not on the vendor’s demonstration.

Duration: four to six weeks. Scope: three representative products — a simple one, a multi-language one, an embedded or containerised one.

Steps

  1. Build a reference set. For each product, establish by hand the list of expected components, at least for direct dependencies and a sample of transitive ones. It is the most thankless work, and it is what makes the evaluation objective.
  2. Inject known cases. Deliberately add a known vulnerable component, a copyleft-licensed component, a vendored component, a statically linked component.
  3. Measure, for each candidate:
    • components found / expected (recall);
    • components wrongly reported (precision);
    • real vulnerabilities detected and false positives;
    • licences correctly identified;
    • time added to the build;
    • integration effort in person-days.
  4. Test reversibility. Export all the data and check the export is usable without the tool.
  5. Test retrieval. Simulate an authority request about a version ingested three weeks earlier.
  6. Score against the grid, and document.

What is not an evaluation

  • A demonstration on the vendor’s demo repositories.
  • A comparison of feature lists.
  • A free trial on a single simple project.
  • An analyst opinion with no measurement on your artefacts.
  1. An open, standardised generator as the primary source, so SBOMs stay portable. An SBOM locked into a proprietary format is a liability.
  2. A self-hostable steering platform to start with, to validate the processes, measure real volumes and provide a factual baseline for comparison.
  3. An investment decision afterwards, based on measured gaps rather than promises: reachability, hash-based detection, depth of the licensing strand, integration.
  4. The evidence vault stays internal, whatever the tool. The platform is a working tool; the company’s evidential memory must not depend on a contract.

The three questions to ask any vendor

  1. What happens if you leave? Export format, exhaustiveness, history and VEX statements included, time to make it available.
  2. Where is the data, and who can access it? Location, subprocessors, support access conditions.
  3. How do you measure your false positives? An evasive answer to that question is itself information.