I spend time digging into vulnerability data to understand trends, recurring patterns, and the growing volume of disclosures. That work supports a practical leadership responsibility: deciding where limited remediation capacity should go. There are never enough resources to fix everything, everywhere, at once.
I needed a dataset I could examine directly and return to as new questions emerged. Automating downloads and updates would leave more time for analysis. NVD was an important source for that work, though its records alone could not tell me which exposures mattered most to a particular organization.
I started building NVD Harvester in early 2025 to automate the data preparation this analysis required. The tool fetches records through the NVD 2.0 API, retains individual JSON files, and produces an aggregated CSV for further analysis. I made it open source because I expected others to need a local dataset too and wanted them to be able to use, inspect, and improve the tool.
Building it brought unexpected challenges: API outages, data corruption, and integrity problems. Data integrity should not depend on someone noticing a problem and stepping in to correct it. I wanted recovery and verification to be part of the automation, not another recurring task someone had to remember.
Working with an external service meant adapting to outages, inconsistent responses, and changes outside my control. Building the integration was only the beginning; keeping the dataset usable required ongoing monitoring and maintenance.
Keep the raw data alongside the analysis
NVD Harvester produces two complementary forms of the data.
The first is an individual JSON file for each CVE, organized by the year in its identifier. The file retains the CVE object returned by NVD, rather than only the fields selected for a report. That is useful when a normalized value needs explanation or an analysis needs information that was not originally selected.
The second is data/nvd-cve-kb.csv, a consolidated table built from those
records. The parser brings identifiers, status, publication and modification
dates, source information, and an English description into a consistent
layout. It also extracts CVSS metrics, weakness identifiers, references, and
vulnerable CPE criteria.
A CSV is convenient for spreadsheet review and downstream database loading. The tool writes the file; it does not provision a SQL database or manage how an organization loads the data into other systems.
Keeping both forms matters. The table makes routine analysis accessible. The JSON provides a way to examine what the source actually supplied before a transformation simplified it. For a team responsible for explaining its results, that is a more useful foundation than retaining only the final report.
There is a boundary here: the JSON files are overwritten when updated. They are a local working copy, not an immutable history of every version. Reproducing an earlier decision requires separately retaining the dataset and transformation version used at that time.
Refresh the dataset without starting over
In my experience, the NVD API was slow and unreliable, regularly becoming unavailable during peak hours. That made efficiency a practical requirement. I added concurrent downloads to retrieve multiple pages at once rather than wait for each request to finish before starting the next.
Incremental updates serve a different purpose: retrieving changed records instead of downloading the entire dataset again. Together, these approaches aim to shorten downloads and avoid repeated work, with consistency checks and recovery paths to handle detected gaps.
Recovery has to establish a usable result
My goal was a tool that could keep the dataset available and recover from known failures without someone repeatedly hunting for missing or damaged data. That goal requires more than restarting a failed request: the recovered state has to be checked before downstream analysis relies on it.
The tool handles several problems with downloading, updating, and maintaining the dataset:
- Temporary API request failures: retry failed requests with a configured delay and attempt limit.
- Missing local CVE files: when synchronization is needed, a local count below the saved count triggers a full download instead of relying only on recent changes.
- Incomplete incremental updates: retry the update, then fall back to a full synchronization if the counts still do not reconcile. A remaining mismatch after the full download stops that synchronization path.
- CSV and JSON getting out of step: rebuild the CSV after a full sync, or when no recently changed files are found but the counts disagree.
- Old CSV rows surviving an update: replace rows for updated CVE identifiers before merging them with the rest of the dataset.
- Missing reference or product-configuration fields: leave the corresponding output fields empty when those source columns are absent, instead of assuming every batch contains them.
These mechanisms reduce manual recovery work in specific failure cases. As the tool evolves, I would judge recovery by whether the dataset is ready for analysis—not simply whether a retry succeeds. That requires verification, limits on recovery attempts, and clear ownership of unresolved failures.
Matching counts did not mean a complete dataset
I encountered missing CVE records even though the total count appeared to match. The gap went undiscovered for some time because the total looked reassuring.
Finding missing records prompted me to add automated consistency checks and recovery paths. The tool checks counts across NVD, local files, and the analysis export, retries incomplete updates, and falls back to a full download or export rebuild when needed. Those checks reduced dependence on manual intervention, while remaining narrower than record-by-record verification.
The lesson was to build checks and recovery into the workflow while being clear about what they establish. Comparing counts can reveal some gaps, but cannot prove that every expected record is present and up to date.
Reliable data is the starting point, not the decision
NVD Harvester provides a dataset for investigating vulnerability trends and patterns. Turning that analysis into remediation priorities still requires context: which assets are exposed, evidence of exploitation, business consequences, and available capacity. A high or critical score alone cannot make that decision.
That distinction is central to Vulnerability Management Is a Decision System, Not a Patch Queue: a trustworthy dataset supports prioritization, but it cannot make the remediation decision by itself.
Lessons learned
Finding missing records behind an apparently matching total changed what I expected from the automation. A completed download and a reassuring count were insufficient. The workflow needed to establish which records were present, whether they were current, and whether the exported data preserved what the analysis required.
These are the operating standards I would apply, not a claim that the current tool implements every check or recovery path:
-
Make integrity monitoring part of the workflow. A matching total alone does not prove that all expected records are present. Define what complete, current, and usable data means for the work it supports, then automate checks for those exact conditions. Periodic human spot checks should test whether the controls work, rather than serve as the main way to discover problems.
-
Define recovery by what people can safely use afterward. Retrying a request addresses only one kind of failure; it does not establish that downstream data is ready. Set acceptance checks before processing resumes, limit recovery attempts, and give unresolved failures a named owner. Test whether failures are made visible to the people relying on the result.
-
Reduce repeated work without weakening verification. Process only what changed when the source supports reliable change tracking, while preserving checks that all expected records are present and up to date. Retain a full-rebuild path for cases that cannot be reconciled safely. Efficiency is useful only if the resulting data remains fit for use.
The outcome worth measuring is whether people can explain which data supported a decision, how current it was, and what uncertainty remained. Download time is useful operational information, but it cannot answer those questions alone.
What would change in your decisions if the data you trusted turned out to be incomplete or out of date—and how would you find out?
Sources
- NVD Harvester: source code, development history, documentation, and MIT license.
