The NCCoE’s Automation of the CMVP

Earlier today at the ICMC24, we heard from a panel about the US National Cybersecurity Center of Excellence’s (NCCoE) work on the Automated Cryptographic Module Validation Program (ACMVP), which intends to tackle the troublingly long queue times we’ve seen for a while. Currently, the temporary solution has been to issue interim certificates for modules that would need to wait in queue for months, possibly years. These interim certifications are only valid for two years with reduced assurance resulting from the decreased rigor in reviewing the submitted modules, however, which doesn’t fully accomplish the goals of requiring certification. The ACMVP aims to improve the efficiency of the validation process via automation to address the growing queue length while still maintaining a high level of rigor, assurance, and the five years of certificate validity.

For the project, NCCoE pulled together experts from CMVP, testing laboratories, and vendors to tackle areas of the CMVP FIPS 140-3 validation process where automation can enhance efficiency, with a special focus on the test report. The ICMC panel discussions by the NCCoE’s ACMVP cover the completed work and future plans of all three workstreams (Test Evidence (TE), Protocol, and Research Infrastructure) and demonstrated the AMVP (Automated Module Validation Protocol) server’s capability of generating a Security Policy (SP). atsec co-leads the TE Workstream with the CMVP and we want to take this opportunity to elaborate on the three major accomplishments that have been completed by this workstream.

1. Classifying TEs

The TE Workstream classified test evidence into the following categories, depending on what needs to be checked, inspected, or tested, and how the vendor evidence (VE) is supposed to be provided:

• SP-TEs, whose assessments are based on reviewing the vendor provided SP
• OD-TEs, whose assessments are based on reviewing the vendor documentation other than the SP, such as design documents, user guidance, finite state module, etc.
• SC-TCs, whose assessments are based on inspecting the module’s source code
• FT-TEs, whose assessments are based on exercising/executing the module to cover functional testing

The above TE categories may be used in combination, and help ensure clear, consistent, and structured filing in lab-provided TE assessments.

2. Filtering non-applicable Assertions (ASs) and their related TEs and VEs

The TE Workstream provided TE filtering criteria based on the module specification, such as security level, module type, embodiment type. The filtering rule also takes into consideration supplemental module information that the CMVP currently asks for but is not yet incorporated in the report template generation by Web Cryptik. Being able to filter TEs based on the module characteristics results in the list of TEs for labs to fill in being shortened, leading to clearer and more concise reports.

3. Unifying the SP and the test report in JSON

The TE Workstream translated the CMVP’s current SP template from the hybrid combination of a Word file skeleton with JSON tables to JSON only. This new structure facilitates the JSON report directly referencing the needed content in the JSON SP, and this will be the first time the Security Policy is written entirely using JSON and the first time the AMVP server can generate a matching SP PDF from the JSON SP.

The TE Workstream extends the reference-based reporting from SP-TEs to all TEs. To achieve this goal, they are working on an evidence catalog file that is also in JSON to capture descriptions of evidence for OD-TEs, SC-TEs, and FT-TEs. It is the first time a test report can reference a well-structured evidence catalog, which contains the SP JSON for SP-TEs, as well as evidence descriptions for other categories of TE. These shifts will reduce redundancy and eliminate the root cause of inconsistency by using the single data entry principle, where information is entered and maintained in the evidence catalog file and that data is pulled by other documents. The new JSON format for everything contributing to a module submission enables automating the checks for existence and completeness of the evidence catalog in relation to the test report.

These major improvements also have short-term impacts to the current CMVP, as creating them generated suggested changes for the CMVP’s current guidance on TEs that rely on verifying vendor documentation instead of functional testing or source code review.

And things won’t stop there! The TE Workstream is still working diligently to improve TE filtering coverage, further develop test method recommendations for function testing TEs, and finalize the JSON structure for the test evidence catalog. The end goal is to allow for an evidence catalog that can be easily referenced by testers when the CMVP reviewers ask for specific TE evidence while also demonstrating the correctness of the evidence to the reviewers.

Coupled with today’s ICMC panel discussion, the NCCoE published documentation about the ACMVP on their website for public review.

About a year ago, atsec made a short animation video clip and played it at the opening of ICMC23, pointing to the direction that the NCCoE ACMVP was heading. It’s worth revisiting the lighthearted clip for a high-level understanding of the new structure – we also think you’ll get a good laugh out of it. Many things illustrated in the clip have already been implemented, and the project is planned for completion in 2025.

https://www.atsec.cn/downloads/media/shortening_the_fips_queue_through_automation%20(720p).mp4