Benchmark datasets for artificial intelligence and wildfire modeling

Abstract

Benchmark datasets shape both what artificial intelligence systems learn and what researchers can claim about their performance. This review examines the literature on their construction and maintenance, then traces its application to wildfire science. General AI research has progressed from shared labeled collections toward task suites, evaluations under distribution shift, data-centric experiments, and explicit benchmark governance. Empirical studies show that annotation errors and subtle sampling changes can materially affect measured performance. Wildfire research presents additional difficulties: many target variables are reconstructed from incomplete observations; fire events cross administrative and temporal boundaries; and response and loss records are generated by selective reporting systems. Public resources now cover occurrence, detection, active-fire forecasting, reconstructed progression, suppression, buildings, evacuation, ecological effects, and smoke. Their maturity and accessibility differ substantially. Task-specific benchmarks and emerging evaluation frameworks already exist, but the review identifies incomplete integration, uneven observations, and a lack of demonstrated common assessment across the full wildfire lifecycle. It distinguishes these gaps from a blanket absence of benchmarking. Dataset maintenance involves changes to historical labels, schemas, software, and access arrangements, as well as the addition of new observations.

1. Scope and review method

This is a critical narrative review of benchmark datasets and their supporting literature. Its questions are: How are datasets assembled and converted into standardized assessments? What evidence supports the validity of those assessments? How are datasets documented, governed, corrected, and extended? How do these practices appear across the wildfire lifecycle, including forest fire, the built environment, and insurance? Which capabilities are already available, which remain fragmented, and what is missing for standardized comparison of new models?

Sources were located through targeted web searches, publication and proceedings websites, agency repositories, and linked dataset and software documentation. Searches combined terms for benchmarking, dataset construction, annotation, evaluation, maintenance, versioning, and distribution shift with wildfire occurrence, ignition, detection, spread, suppression, evacuation, structure loss, insurance, burn severity, mortality, and smoke. Primary research papers anchor the synthesis; official documentation is used for release status, access, and maintenance. Foundational AI studies are included alongside recent wildfire data papers. The review separates publication year, observation period, and verified release information. Sources without a publication date are marked “n.d.”; their access date is 13 October 2026.

Selection was purposive: studies were included when they established a widely used benchmark, introduced a substantial wildfire dataset, empirically examined benchmark validity, or documented a distinct approach to stewardship. This is not an exhaustive database search or a PRISMA systematic review, and no completeness claim is made. English-language, publicly accessible documentation is overrepresented. U.S. and Canadian operational records receive substantial attention because detailed public data descriptions were available; global and Mediterranean datasets provide a broader geographical comparison. Proprietary catastrophe-model and insurance data are represented only to the extent that their construction and accessibility can be verified publicly. Descriptions of intended updates are distinguished from evidence of completed releases. A supplementary search specifically examined benchmark suites, evaluation frameworks, competitions, and leaderboards to test claims of missing standardization; recent repositories and preprints are identified as such rather than treated as established community standards.

2. What makes a dataset a benchmark?

A dataset supplies observations. A benchmark also defines an experiment: the task, admissible inputs, target, data partition, scoring procedure, and conditions under which results are comparable. OpenML makes this distinction explicit by representing a task separately from its underlying dataset. Its task specification includes the target, evaluation procedure, exact splits, and performance measure; experiment records are linked to that specification. The same data can therefore support several scientifically different tasks. This is particularly useful for interpreting wildfire studies that reuse a common archive but predict different outcomes or use different spatial and temporal partitions. (Bischl et al., 2021)

ImageNet and COCO illustrate how the benchmark task is built into data collection. The ImageNet challenge established shared image classification, localization, and detection tasks around an organized category system. COCO emphasized objects in complex scenes and created instance-level annotations through staged category identification, instance identification, and segmentation. Their importance lies partly in this coupling of an explicit visual task to an annotation procedure and evaluation convention. A collection of images without that coupling would leave substantial freedom in what each model was being asked to do. (Russakovsky et al., 2015) (Lin et al., 2014)

Later benchmarks broadened the unit of assessment. SuperGLUE combined multiple language-understanding tasks with task-specific measures and an aggregate score. WILDS instead organized evaluation around naturally occurring distribution shifts, such as changes in location or acquisition context, making performance outside the training distribution part of the experiment. These designs answer different questions: breadth across tasks and robustness across domains are related but not equivalent. A model can perform well on a heterogeneous suite while still failing when the same task moves to a new population. (Wang et al., 2019) (Koh et al., 2021)

DataComp changes what is held constant. It fixes a model-training framework and evaluates alternative training-data choices on a common collection of downstream tasks. The resulting experiment concerns data selection and curation rather than unrestricted competition among model architectures. Conversely, MLPerf rules establish constraints on submissions and reporting to make system performance comparable. Together, these examples show that standard assessment can concern the quality of a training corpus, the predictive capability of a model, or the efficiency of a computing system. “Benchmark” does not specify which of these is being measured. (Gadre et al., 2023) (MLCommons, n.d.)

3. Construction and the empirical validity of benchmark results

Benchmark construction begins with a sampling decision, even when that decision is obscured by scale. Image categories, selected institutions, sensor footprints, geographic boundaries, and inclusion thresholds determine the population represented. More observations can reduce sampling variability without correcting a systematically narrow population. This distinction becomes important when benchmark scores are interpreted as general measures of capability rather than estimates for the data-generating process actually represented.

Recht et al. (2019) tested this issue by building new CIFAR-10 and ImageNet test sets while closely following the original collection procedures. Accuracy fell across evaluated models, although gains on the original tests transferred to the new tests. The authors interpreted the results as sensitivity to somewhat harder sampled images rather than straightforward evidence of adaptive overfitting. Thus, replacing a test set with newly collected examples can change its difficulty even when the nominal task and collection method remain similar. Dataset renewal and measurement continuity are separate empirical questions. (Recht et al., 2019)

Northcutt et al. (2021) examined test-label quality across ten widely used vision, language, and audio datasets. They combined algorithmic identification of suspicious labels with human validation and estimated an average error prevalence of at least 3.3%. Their analysis showed that label errors can affect model comparisons, including the apparent advantage of higher-capacity models under some conditions. This supports a stronger interpretation of annotation work: quality control helps determine whether a leaderboard ordering is meaningful, not simply whether the training data are convenient to use. (Northcutt et al., 2021)

The practical organization of data work also matters. Sambasivan et al. (2021), drawing on qualitative research with AI practitioners, described how neglected data problems propagate through development. Their “data cascades” account emphasizes organizational incentives, disciplinary boundaries, and failures to incorporate domain knowledge. This complements statistical audits: an error can originate in institutional practice before it appears as a mislabeled row or an unstable result. In wildfire datasets, the comparable distinction is between errors introduced during processing and limitations inherited from operational reporting or observation. That comparison is an interpretation of the two literatures, not a finding that the practitioner study itself tested in wildfire systems. (Sambasivan et al., 2021)

BetterBench gives a broader assessment of current practice. Reuel et al. (2024) evaluated 24 benchmarks against 46 lifecycle criteria and found substantial variation in quality, including frequent weaknesses in statistical reporting and reproducibility. Their framework extends beyond the initial dataset to implementation, documentation, maintenance, and retirement. Taken together, these studies undermine the assumption that a popular dataset automatically supplies a valid standardized assessment. Popularity can facilitate comparison, but the validity of that comparison remains an empirical and procedural question. (Reuel et al., 2024)

4. Documentation, governance, and keeping benchmarks current

The stewardship literature addresses different layers of the problem. Datasheets for Datasets proposes structured documentation of motivation, composition, collection, preprocessing, uses, distribution, and maintenance. FAIR emphasizes discoverability, access conditions, interoperability, and reuse, including machine-actionable metadata and persistent identification. CARE adds collective benefit, authority to control, responsibility, and ethics for Indigenous data. These frameworks are complementary: a technically discoverable resource can still have poorly documented collection decisions, and technical openness does not resolve who has authority over data concerning a community. (Gebru et al., 2021) (Wilkinson et al., 2016) (Carroll et al., 2020)

Technical metadata standards address reproducibility at another level. Croissant describes machine-learning datasets and their structure in a form that software can interpret. STAC describes spatial and temporal assets and their organization. They help users discover and load resources consistently, but neither establishes whether a label is accurate or whether a train–test partition measures the intended generalization. Interoperable metadata and empirical validation solve different problems. (MLCommons, n.d.) (STAC community, n.d.)

The literature and release documentation reveal several distinct maintenance strategies. Fixed challenge datasets preserve a stable reference for comparison. New editions can revise the task or expand coverage while leaving previous editions available. Living platforms may continuously add experiments, tasks, or results. Dynabench introduces a different mechanism: humans generate examples against current models, and model weaknesses inform subsequent rounds of data collection. Its changing difficulty is part of the design. Consequently, scores across rounds need their collection context to remain interpretable. (Kiela et al., 2021)

MIMIC-IV provides a useful example of maintained research infrastructure that is not itself one prediction benchmark. The publication describes a deidentified relational clinical database, while PhysioNet distributes identifiable releases with access requirements and change notes. Version 3.1 documents corrections and can be distinguished from earlier releases. Credentialing and a data-use agreement are part of its access model. Patient-level date shifting also illustrates a substantive transformation: preserving longitudinal relationships within a patient does not preserve a common calendar across patients. Data management can therefore alter which temporal questions remain answerable. (Johnson et al., 2023) (Johnson et al., 2024)

Maintenance is not a single operation. Adding observations changes coverage; correcting labels changes the reference target; rebuilding historical data with a new algorithm can change past values; revising a schema changes how records are interpreted; and updating evaluation software can change scores without changing observations. This categorization is a synthesis of the reviewed examples. It explains why an active website, a recent repository commit, or an updated catalog entry alone does not establish that the underlying dataset has acquired new observations. The strongest evidence of updating is a release history that specifies what changed and preserves the identity of the earlier artifact.

5. Scientific and professional-domain precedents

WeatherBench 2 is particularly relevant to physical forecasting because it combines reference data, physical and machine-learning baselines, and open verification software. Rasp et al. (2024) discuss lead time, variables, initialization, and the status of ERA5 as a verification reference. ERA5 is a reanalysis constrained by observations, rather than a direct measurement of every atmospheric quantity; its limitations vary by variable. The paper also distinguishes retrospective initialization from information available to an operational forecast. This is a substantive qualification of measured skill, especially for surface variables and precipitation. The maintained project subsequently separates data access, evaluation software, and current scorecards. (Rasp et al., 2024) (WeatherBench maintainers, n.d.)

Hydrology offers a different precedent: harmonizing observations collected by many institutions. CAMELS coupled catchment attributes with meteorological and streamflow series to support large-sample analysis. Caravan extended this approach across regional collections using consistent forcing and attribute processing and a workflow through which additional catchments can be contributed. Both reduce repeated data-engineering work. However, a shared hydrological archive does not uniquely determine whether evaluation concerns previously observed catchments, unseen catchments, future climates, or altered management regimes. Harmonization enables those experiments without making them equivalent. (Addor et al., 2017) (Kratzert et al., 2023)

GEO-Bench applies common interfaces and evaluation procedures to multiple Earth-observation tasks. Its relevance lies in assessing models across imagery sources and downstream applications rather than treating one remote-sensing dataset as sufficient evidence of general capability. The successor project’s documentation further distinguishes public generation scripts from the official distributed datasets accepted for comparison. This distinction matters because regenerated source imagery, processing dependencies, or upstream corrections can produce different artifacts even when a workflow appears reproducible. (Lacoste et al., 2023) (GEO-Bench-2 maintainers, n.d.)

PDEBench exposes data-generation assumptions directly by organizing learning tasks around families of partial differential equations, parameter settings, and numerical solutions. Such datasets can provide dense, internally consistent reference trajectories and permit controlled experiments that would be difficult in the field. The limitation follows from the construction: performance is demonstrated for the represented numerical systems and parameter ranges. D4RL addresses sequential decision-making through logged trajectories for offline reinforcement learning. It highlights another distinction relevant to incident response: predicting a state from historical records and evaluating a policy from recorded decisions are different statistical tasks. (Takamoto et al., 2022) (Fu et al., 2020)

Actuarial datasets make the outcome definition especially explicit. The CASdatasets documentation includes linked motor-insurance frequency and severity data, with policy exposure and claims represented separately. The Casualty Actuarial Society’s reserving resources instead organize losses by origin and development periods. These are useful precedents for insurance benchmarking because the relevant denominator may be insured exposure, and recorded loss can develop after the original event. They are not wildfire benchmarks. Their role in this review is to establish why a physical damage label and an insurance outcome have different data structures. (Dutang and Charpentier, n.d.) (Casualty Actuarial Society, n.d.)

6. Wildfire data: observation systems and research datasets

Jain et al. (2020) reviewed 298 publications on machine learning in wildfire science and management, organizing them into fuels, detection and mapping, weather and climate, occurrence and risk, behavior, effects, and management applications. Their review establishes the breadth of the field and the uneven development of its subdomains. It also helps explain why “wildfire prediction” is an insufficient description of a benchmark task. Predicting whether an ignition occurs, locating an existing fire, forecasting active burning, estimating eventual damage, and evaluating suppression decisions involve different observations and targets. The 2020 review is a historical baseline, not an inventory of datasets available in 2026. (Jain et al., 2020)

The foundational distinction is between an observation and a reconstructed event. FIRMS distributes satellite active-fire detections, with acquisition times and sensor-dependent spatial characteristics. Such a detection identifies a thermal signal at an observation time. It does not directly delineate everything burned before that time. The MODIS burned-area product uses a different estimation procedure based on reflectance changes and active-fire information to infer burned pixels and burn dates. Clouds, revisit intervals, spatial resolution, and detection algorithms influence these products differently. An active-fire mask and a cumulative burned-area mask therefore cannot be treated as interchangeable labels. (NASA FIRMS, n.d.) (Giglio et al., 2018)

Event datasets add another interpretive layer. The Global Fire Atlas derives individual events and behavior summaries from satellite burned-area observations. GlobFire similarly constructs event histories from the spatiotemporal relationships among burned pixels. FIRED uses reproducible event-delineation procedures to produce country-level fire records. These resources enable analyses beyond pixel-level burning, but event boundaries and event identities partly depend on the algorithm that groups observations. Two products can share an upstream sensor while disagreeing about the number, duration, or extent of individual fires. Agreement between such products is not necessarily independent validation because their errors can share a common origin. (Andela et al., 2019) (Artés et al., 2019) (Mahood et al., 2022)

Administrative inventories organize fire differently. U.S. FPA-FOD integrates reported fire occurrences; the official current service identifies its seventh edition as covering 1992–2024, with approximately 2.66 million records. Canadian resources distinguish the National Fire Database, assembled from agency contributions, from the National Burned Area Composite. EFFIS supplies another regional system with its own mapping and reporting conventions. These infrastructures provide essential coverage and institutional continuity, but their differing definitions and collection procedures prevent automatic equivalence among an agency incident, a reported ignition, and a satellite-derived event. (USDA Forest Service, 2026) (Natural Resources Canada, n.d.) (European Commission, n.d.)

7. Ignition, occurrence, and early detection

Occurrence modeling is well served by extensive incident inventories, but an inventory of recorded fires does not by itself define the population at risk. FPA FOD-Attributes, introduced by Pourmohamad et al. (2024), enriches the sixth-edition U.S. occurrence database for 1992–2020 with approximately 270 physical, biological, social, and management attributes. Its contribution is harmonized explanatory context across many events. For occurrence prediction, the analyst still has to define the non-fire locations and times against which observed events are compared. For cause or final-size modeling among recorded fires, a different target population is involved. Those are consequences of the event-centered construction, rather than evidence that one particular modeling protocol is required. (Pourmohamad et al., 2024)

The longstanding UCI Forest Fires dataset demonstrates the difference between convenience and representativeness. Its 517 records from northeastern Portugal support a manageable tabular burned-area exercise with meteorological and related variables. That accessibility explains its instructional value, but its geographic extent, sample size, and target do not support interpreting performance as a general test of ignition prediction or fire spread. In particular, final burned area collapses an event history into one outcome; it cannot identify where or when spread occurred. (Cortez and Morais, 2007)

Early detection follows a separate literature. The FIgLib/SmokeyNet work uses camera image sequences around the appearance of smoke, making temporal visual evidence central to the task. FLAME provides aerial imagery from pile burning, including modalities useful for fire detection and segmentation. Sen2Fire uses Sentinel-2 imagery and Sentinel-5P aerosol information in 2,466 patches; its experiments examine spectral-band selection and report benefits from selected combinations and aerosol information. These studies demonstrate the value of purpose-built labels, but their collection settings differ: stationary cameras, controlled burns viewed from drones, and satellite scenes expose models to different backgrounds, scales, and observation schedules. (Dewangan et al., 2022) (Shamsoshoara et al., 2021) (Xu et al., 2024)

Across these studies, detection accuracy concerns the presence of observable fire or smoke under a particular acquisition system. It is not equivalent to forecasting ignition before any observable signal exists. Nor does frame-level performance alone establish operational detection delay or an acceptable false-alarm burden across many camera-hours. This is a comparison of task definitions rather than a claim that the cited studies all intended to measure operational performance. The distinction becomes especially important when nearby video frames or multiple views of the same event are treated as numerous nominally separate examples.

8. Progression and forecasting at different scales

Huot et al. (2022) introduced Next Day Wildfire Spread: 18,545 paired samples from 2012–2020 on a 1 km grid, combining current detections and environmental covariates to predict tomorrow’s active-fire mask. The original partition assigns weeks randomly to training, validation, and testing with one-day buffers. The authors select models using area under the precision–recall curve because burning pixels are relatively uncommon. Their neural-network baseline achieved 28.4% AUC-PR, compared with 11.5% for persistence under that protocol. These values illustrate a defined comparison, rather than a general ranking against other datasets. Uncertain observations are masked, and the target remains satellite-observed active burning rather than a surveyed perimeter or an arrival-time map. The paper also finds higher scores at coarser prediction scales, while recognizing that reduced localization diminishes operational usefulness. (Huot et al., 2022)

WildfireSpreadTS extends the temporal structure. Gerard et al. (2023) provide 13,607 daily images across 607 U.S. events from 2018–2021, with 23 channels and active-fire maps at 375 m resolution. Full sequences permit models to use more than a single previous observation. The paper explicitly identifies class imbalance and noisy observations due to clouds, smoke, and detection inaccuracies as challenges. Thus, its contribution is not merely a finer grid: it changes the information available to a temporal model. The effective number of independent events nevertheless remains far smaller than the number of pixels or image frames. (Gerard et al., 2023)

Mesogeos, introduced by Kondylatos et al. (2023), shifts emphasis to a common Mediterranean datacube covering 2006–2022 at daily, 1 km support. It harmonizes heterogeneous variables and demonstrates tasks including fire-danger and final-burned-area modeling. SeasFire operates at a still broader scale: its published cube spans 2001–2021 on a 0.25-degree grid with eight-day intervals, supporting seasonal fire analysis and forecasting. These are complementary resources. Their temporal aggregation and targets address regional or large-scale fire activity; they do not provide interchangeable tests of local fire-front propagation. (Kondylatos et al., 2023) (Karasante et al., 2025)

BCWildfire is a recent example of extending temporally rich prediction resources to boreal conditions. Xu et al.’s December 2025 preprint, reported as accepted at AAAI-26, describes 25 years of daily data over British Columbia and surrounding regions, with 38 covariates and comparisons among multiple time-series architectures. It expands the geographical and temporal settings in which forecasting can be studied. Because the reviewed record is the preprint, its bibliographic status is identified explicitly rather than silently treating it as a verified final proceedings version. (Xu et al., 2025)

The Canadian Fire Spread Dataset provides a different target construction. Barber et al. (2024) combined final fire boundaries with MODIS and VIIRS detections and used ordinary kriging to estimate day of burning on a 180 m grid. The initial 2002–2021 release contains 3,269 large fires and 50 associated covariates; selection includes a final-area threshold and sufficient hotspot observations. Validation against independent perimeter information is therefore particularly important. The fine output grid does not imply direct observation at that spacing, and the daily arrival estimates remain reconstructions. This is a reference for event progression and associated burning conditions, whereas the preceding active-fire datasets predict a sensor-observed state. (Barber et al., 2024)

Magstadt et al. (2026) assembled a complementary western U.S. progression dataset from National Infrared Operations observations. Its 2020–2024 release includes 12,705 perimeter observations for 737 incidents, with geometry and attribute harmonization. Airborne infrared interpretation supplies a different observation basis from satellite interpolation. The tradeoff is selective coverage: flights are requested for particular incidents, and observation times are not uniformly spaced across every fire. Operational WFIGS/IRWIN services likewise support incident and perimeter integration, but live service records require attention to observation time, update time, and incident identity before they become a historical forecasting dataset. (Magstadt et al., 2026) (National Interagency Fire Center, n.d.)

Fire Event Data Suite (FEDS) adds a useful operational comparison. Chen et al. (2022) describe object-based tracking from VIIRS detections; project documentation distinguishes near-real-time services from published retrospective datasets. The products track evolving events at 12-hour intervals. NASA’s documentation explicitly cautions that the inferred “perimeters” are not equivalent to operationally mapped perimeters. This reinforces the broader finding: a shared geometric representation does not ensure that two datasets measure the same physical boundary. (Chen et al., 2022) (Wildfire Tracking Lab, n.d.) (NASA, 2026)

Simulation and evaluation frameworks occupy still different positions. Wang et al.’s FireBench (2024) uses high-fidelity ensemble simulations to generate controlled fire-behavior data. The separately developed WIRC FireBench standardizes observational/model data and evaluation procedures. The shared name should not obscure their distinct roles: one supplies a simulation framework and synthetic trajectories, while the other supports structured evaluation against reference datasets. Simulation-based accuracy establishes fidelity to the represented numerical system; observational agreement additionally depends on the measurement and reconstruction processes discussed above. (Wang et al., 2024) (Wildfire Interdisciplinary Research Center, n.d.)

The spread literature consequently contains several legitimate but non-equivalent targets: tomorrow’s active-fire pixels, cumulative area burned, interpolated arrival day, observed perimeter geometry, eventual event size, and aggregated seasonal burning. These targets differ in physical meaning, temporal support, and uncertainty. Their published scores cannot be pooled into a defensible ranking of model quality without a common experiment. Even where papers report the same named metric, the prevalence of burning, event selection, masking, spatial support, and forecast horizon can make the resulting values incomparable.

9. Incident response, suppression, and evacuation

Incident response data are commonly generated to coordinate an unfolding emergency, rather than to record a complete experimental history. ICS-209-PLUS makes that history more usable by harmonizing successive reporting forms and linking incident information. St. Denis et al. (2023) assembled 187,160 reports for 35,170 incidents during 1999–2020, including 34,478 wildland fires. The data include repeated incident-level descriptions of resources, impacts, and status. Their major contribution is longitudinal linkage and cleaning across changing administrative forms. Coverage remains selective toward reportable incidents, and reported cumulative quantities can be revised. A report timestamp and the physical time of a fire outcome are therefore not automatically identical. (St. Denis et al., 2023)

Nguyen et al. (2024) converted 3,124 Incident Management Situation Reports from 2007–2021 into structured data and released processing software. This source describes the wider response environment, including regional preparedness and resource use. It complements incident reports because a fire’s management occurs within a larger system of simultaneous demands. However, regional totals do not resolve the exact timing and placement of every crew, aircraft, or tactical action on an individual fire. Open extraction code makes extension possible; it does not establish that later reports have already been processed and validated. (Nguyen et al., 2024)

Arkowitz et al. (2025) move closer to spatially explicit intervention records by quality-assuring containment firelines and engagement outcomes for large U.S. fires during 2017–2024. This connects the location of suppression infrastructure with mapped fire behavior. Interpretation still depends on how a line’s engagement or outcome is inferred from spatial relationships. A line coinciding with a final boundary does not, on its own, establish the counterfactual that the fire would have crossed that location without intervention. Weather changes, terrain, resource allocation, and the selection of places where lines were built can also contribute to observed outcomes. (Arkowitz et al., 2025)

Detailed incident reconstructions supply another kind of evidence. NIST’s Camp Fire progression study integrates multiple sources into a documented event timeline. Such work can reveal temporal relationships that are absent from national tables, but a well-reconstructed catastrophic case is not a representative sample of incidents. Canadian Wildfire Evacuation Data provide broader coverage for 1980–2019, assembled from news reporting and agency information and linked to likely associated fires. Its records describe community evacuations, dates, reasons, and estimated populations. They do not constitute individual trajectories through a road network, and completeness can depend on whether an evacuation was documented in the source record. (NIST, 2021) (Tepley et al., 2021)

A recent counterexample to a blanket claim that response benchmarks are absent is WILDFIREIA. Xu, Cheng, and Dong’s 2026 preprint defines an initial-attack-failure prediction task using naturally caused U.S. fires, public environmental inputs, a size-based outcome rule, chronological partitions, and explicit exclusions of post-discovery information from predictors. It standardizes prediction of an outcome proxy; it does not directly observe every suppression decision or identify the effectiveness of alternative tactics. Its publication status and restricted event population also qualify its interpretation. (Xu et al., 2026)

Across these resources, the literature is stronger on assembling response records and forecasting outcomes than on establishing shared benchmarks for the effectiveness of alternative decisions. Historical records can support prediction of resource demand or evacuation occurrence. Evaluating a decision policy additionally requires reasoning about actions that were not taken and the conditions that influenced the actions observed. The distinction mirrors the separation between predictive datasets and offline decision-learning datasets in the broader AI literature; it is a limitation of the inference, not simply a shortage of rows.

10. Built-environment exposure, physical damage, and insurance loss

Built-environment datasets expose the importance of the denominator. A list of destroyed structures describes losses; a vulnerability model also depends on which structures were exposed and survived. CAL FIRE DINS provides inspection-based damage information and structure attributes, but inspection practices and event coverage shape the observed population. Syphard and Keeley (2019) analyzed more than 40,000 exposed structures from California fires during 2013–2018 and examined associations between structure loss, building features, and defensible space. This is stronger evidence about vulnerability than a destroyed-buildings-only collection, while remaining observational and sensitive to recorded attributes and exposure selection. (CAL FIRE, n.d.) (Syphard and Keeley, 2019)

A 2025 USGS release broadens the geographic evidence substantially. “Wildfire building destruction and exposure in the conterminous United States, 2002–2022” compiles 362 fires with at least ten destroyed buildings. Construction combines earlier datasets with additional mapping, Microsoft building footprints, and pre- and post-fire imagery. Buildings are considered within fire perimeters and a surrounding buffer, and historical imagery helps remove buildings absent at the time of the fire. The resource includes destroyed and surviving buildings, but its selection of destructive events limits extrapolation to the full population of wildfires. Its mapped exposure definition is also broader than direct evidence that every included building experienced the same fire or ember conditions. (U.S. Geological Survey, 2025)

xBD approaches the problem through image interpretation across disaster types. It pairs pre- and post-event satellite imagery with building footprints and damage labels, supporting standardized building localization and damage classification. The wildfire subset is valuable for post-event assessment, but inference from overhead appearance differs from field inspection of construction features and from predicting vulnerability before an event. It also differs from insurance valuation. The National Structure Inventory supplies modeled structure and economic attributes that can support exposure analysis, but these should not be read as individual insurance contracts or observed claims. (Gupta et al., 2019) (U.S. Army Corps of Engineers, 2022)

The insurance literature illustrates what physical-damage datasets omit. IBHS’s Witch Creek/Poomacha study analyzes the claims experience of 21,768 residential exposures insured by six companies during the 2007 fires. The public summary reports both claim frequency and severity, including properties without claims. This is a relevant wildfire insurance study, but its underlying member claims and full report are not an openly downloadable row-level benchmark. The fire year is not treated here as the publication date. California Department of Insurance releases add public policy, market, and wildfire-related insurance statistics at their stated aggregation levels; they do not supply the same observations as a building-level claims file. (Insurance Institute for Business & Home Safety, n.d.) (California Department of Insurance, n.d.)

These studies distinguish hazard, exposure, vulnerability, and insured loss. A building may be physically damaged without generating a particular recorded claim, while a claim can include costs that are not visible in a roof-damage label. Contract terms, insured values, claim development, and the composition of the insured population affect what an insurance outcome means. The actuarial benchmark precedents discussed earlier make those differences explicit through policy exposure and loss-development records. Among the sources examined here, public wildfire physical-damage data are more accessible than standardized, property-level insurance loss data. This is a finding about the reviewed accessible literature, not proof that no such private or restricted datasets exist.

11. Forest, ecological, atmospheric, and community outcomes

MTBS provides a long-running U.S. record of burn extent and spectral burn severity from 1984 onward, subject to minimum fire-size thresholds and a scope that includes prescribed fires. Its consistency supports long-term comparison, but spectral change is not the same variable as tree death, ecological recovery, or financial damage. The HLS Burn Scars dataset adapts remote-sensing imagery and MTBS-derived masks into image patches for machine learning. Its construction and screening make a segmentation task more accessible; they do not create an independent field measurement of fire effects. This lineage matters when apparent agreement among datasets is used as evidence of label validity. (MTBS program, n.d.) (Phillips et al., 2023)

The Fire and Tree Mortality Database addresses outcomes through field observations. Cansler et al. (2020) harmonized records for 164,293 trees, 142 species, and 409 fires, with post-fire measurements extending beyond the immediate burn. The data retain variables relevant to injury and subsequent mortality. A companion evaluation found stronger support for simple mortality models for thick-barked conifers than for some other species groups. This provides an empirical example of how a large shared dataset can reveal the limits of a model’s biological transferability rather than merely produce a single overall score. Its observational histories also show why outcome timing matters: survival shortly after a fire and survival years later are distinct labels. (Cansler et al., 2020) (Cansler et al., 2020)

Vegetation and fuels layers such as LANDFIRE often appear as explanatory inputs rather than outcome labels. Their release vintage matters when used with historical fires, because disturbance updates can change the mapped vegetation or fuel state. A layer depicting the landscape after an event has a different interpretation from the pre-fire conditions a forecast would have available. The existence of updated landscape products therefore creates both an opportunity for improved representation and a temporal alignment problem for retrospective modeling. (LANDFIRE program, n.d.)

Atmospheric datasets further separate emissions, transport, and exposure. GFED combines burned area with modeled fuel consumption and emission factors to estimate fire emissions; van der Werf et al. (2017) document the assumptions behind one major generation of that product. NOAA HMS smoke polygons describe interpreted smoke extent, while EPA AQS provides measurements from air-quality monitors. These are complementary data sources rather than alternative measurements of an identical quantity. A smoke polygon does not directly specify a person’s inhaled dose, and total emissions do not uniquely determine surface particulate concentrations at a particular location. (van der Werf et al., 2017) (NOAA, n.d.) (U.S. Environmental Protection Agency, n.d.)

Community consequences are represented unevenly across the reviewed resources. Evacuation datasets capture displacement decisions and their reported scale; incident summaries capture selected impacts; structure datasets record physical damage; and air-quality systems record parts of the exposure pathway. They do not form a unified longitudinal record of recovery, health, housing, or livelihood outcomes. This limits how far performance on a single outcome dataset can be interpreted as performance across the social consequences of wildfire. It also makes record linkage and the meaning of missing outcomes central methodological issues.

12. How wildfire datasets are maintained and revised

Wildfire resources span operational services, periodic agency products, archived research releases, and software-maintained benchmarks. These arrangements leave different evidence of continuity. FPA-FOD’s current service identifies a numbered edition, but its service address is not a frozen historical artifact. FPA FOD-Attributes is based on an earlier occurrence edition; the subsequent release of the parent database does not automatically rebuild the enriched dataset. This illustrates a general dependency problem: an updated upstream record and a previously published derivative may no longer describe the same population or identifiers. (USDA Forest Service, 2026) (Pourmohamad et al., 2024)

The Global Fire Atlas offers unusually explicit evidence of historical reprocessing. Its 2026 archive updates input products to MODIS Collection 6.1 and extends coverage through February 2026. The release notes explain that parameters governing event persistence and fire seasons are recalibrated using the available time series. As a result, earlier years can change, and users are instructed to use a release as a complete dataset rather than append only its newest years to an older version. The documentation also distinguishes a locally defined fire season from a calendar year. An update is therefore a new reconstruction of fire history, not merely a longer table. (Global Fire Atlas maintainers, 2026)

FEDS supplies a parallel example with explicit software lineage. The October 2026 CONUS archive covers 2012–2025 and records both the input satellite-product version and FEDS algorithm version 1.4.1. Its description states that it supersedes earlier regional archives. This kind of release record makes it possible to identify which processing system produced the historical trajectories. It also prevents the publication date of a new archive from being confused with the date of its final observation. (Becker et al., 2026)

Some maintenance occurs in model-input code rather than observational files. WildfireSpreadTS’s repository documents corrections and a February 2026 issue concerning the encoding of angle features: the then-current code used a sine transform without the complementary cosine representation. The note described the issue as unresolved at that time. This is evidence of a documented limitation, not evidence that a corrected dataset or benchmark score had already been released. Conversely, the Canadian Fire Spread Dataset paper states an intention to expand annually; that statement alone is not a verified schedule of completed releases. (Gerard, n.d.) (Barber et al., 2024)

For response and damage data, continuity additionally depends on institutional reporting and interpretation. ICS-209-PLUS harmonizes historical form changes; IMSR publishes an extraction tool; DINS depends on field inspection; and building-loss compilations require imagery review and historical exposure checks. These activities differ in labor and automation. The reviewed publications document substantial curation, but generally provide less evidence about continuing staffing, long-term funding, service guarantees, or formal retirement criteria. The appropriate conclusion is that operational continuity is unevenly documented, rather than that every research dataset is either actively maintained or abandoned.

13. Gaps: what exists, what is missing, and what standardized evaluation requires

13.1 Is there an equivalent of an LLM benchmark suite?

Wildfire has standardized tasks and emerging evaluation frameworks, but this review did not identify a publicly documented, broadly adopted benchmark suite spanning ignition, progression, incident response, built-environment impacts, insurance losses, and ecological outcomes under a common evaluation and maintenance framework. The defensible finding is fragmented and incomplete standardization. Saying that no wildfire benchmark datasets or standardized comparisons exist would overlook the task-specific studies in Sections 7–9 and the frameworks considered below. The statement about the missing full-lifecycle suite is a bounded finding of this literature search, not proof of universal nonexistence.

The closest terminology for a “board of datasets” is a benchmark suite or battery of benchmarks. A leaderboard displays results; an evaluation harness runs the tests consistently. These functions can be separate. In language-model research, HELM combines scenarios, standardized adaptation procedures, multiple metrics, and published comparisons, while EleutherAI’s Language Model Evaluation Harness supplies reusable implementations across many tasks. These are precedents for shared assessment infrastructure, rather than evidence of one universal LLM standard or governing board. The important analogy is that multiple models can be evaluated on the same defined experiments. (Liang et al., 2023) (EleutherAI contributors, n.d.)

That analogy also has limits. Language models commonly expose a text interface across tasks; wildfire models can predict pixels, perimeters, event probabilities, resource demand, or monetary losses. A useful suite would therefore require separate task tracks and input/output specifications. A spread model need not produce insurance claims to participate. Combining all tracks into one number would additionally require explicit weighting and an explanation of which models and decisions that aggregate is intended to compare. A shared suite is compatible with reporting a profile of performance rather than a universal model ranking. This is a methodological inference from the task differences documented in the review.

13.2 Existing work that narrows the gap

Next Day Wildfire Spread, WildfireSpreadTS, and the Mesogeos task tracks already provide concrete bases for controlled comparisons. WIRC’s FireBench goes further toward shared infrastructure by standardizing observation/model formats, metrics, normalization, and scorecards. Its documented Caldor workflow includes versioned data and targets involving perimeters, building damage, and weather. It is therefore a substantive counterexample to claims that wildfire lacks evaluation software or multi-target benchmarks. Its documented case workflows do not, however, establish coverage of the entire lifecycle or field-wide adoption. (Wildfire Interdisciplinary Research Center, n.d.) (WIRC FireBench contributors, n.d.)

FireComp’s repository describes a geographically broader next-day spread benchmark, with nine regions on six continents, explicit temporal splits, common metrics, baselines, and cross-region evaluation routines. It addresses several elements of the standardization problem within one task. The reviewed evidence is project documentation; independent adoption and a peer-reviewed publication were not established. Ho et al. (2026) also report an actual model competition using the separate simulation-based FireBench dataset. This demonstrates comparative evaluation under shared conditions, while its simulated spread task leaves other wildfire processes outside the experiment. Neither effort should be dismissed merely because it is narrower than a lifecycle suite. (Karlsson, n.d.) (Ho et al., 2026)

WILDFIREIA similarly shows that response-related outcome prediction is beginning to acquire explicit benchmark protocols. The remaining distinction is between predicting a size-based escape proxy and evaluating suppression actions themselves. More generally, “data exist,” “a reproducible benchmark exists,” and “a benchmark is widely adopted and durably maintained” are different claims. A paper or repository can establish the first two without establishing the third. (Xu et al., 2026)

13.3 Specific gaps and the evidence needed to close them

Table 3 separates gaps in observations, integration, experimental protocols, and stewardship. “Not established” means that the reviewed sources do not demonstrate the stated capability. It does not mean that every individual project lacks it. The final column identifies methodological needs implied by the evidence; it is not a proposed project design.

The integration gap is particularly important. Several component datasets contain large amounts of information, yet combining them is not equivalent to concatenating tables. A satellite-defined event, an administrative incident, a fire complex, and a damage-inspection operation can have different identities and boundaries. Without documented relationships among those units, a model evaluated on linked data can inherit mistaken event assignments or selectively complete histories. This limits claims about learning relationships across ignition, spread, intervention, and outcome even when each component dataset is individually useful.

The observation gap also cannot be solved by an evaluation wrapper alone. Fine temporal progression, building-scale fire and ember exposure, actual response actions, developed insurance claims, and long-term community outcomes require measurements or records that are incomplete or restricted in the reviewed public sources. Synthetic data can support controlled experiments, but do not supply independent evidence that a learned relationship transfers to observed fires. Likewise, an inspection label or final-size threshold can be a useful proxy without becoming a direct measurement of vulnerability or suppression effectiveness.

The assessment gap is broader than a common metric name. Two studies reporting intersection-over-union or AUC-PR can still differ in event selection, spatial scale, forecast horizon, missing-pixel masks, class prevalence, and permitted information. Comparable evaluation needs these conditions to be specified together. Training-data rules also affect interpretation: a fixed-data track tests improvements under controlled data access, whereas an open-data track tests systems that may differ in both training information and modeling method. Both are legitimate, but their results support different claims.

The maintenance gap concerns the benchmark as a scientific instrument. New fire seasons make an evaluation more current, while revised observations can change earlier labels and scores. Stable archived tests and periodically refreshed tests serve different purposes. A maintained suite needs to distinguish them and document which results remain comparable after each change. The reviewed resources demonstrate many parts of this practice individually; the literature does not yet establish them as one shared standard across the full wildfire lifecycle.

The central unmet need is therefore a sustained, interoperable benchmark suite with agreed evaluation protocols and adequate observations for the tasks it claims to represent. This entails both data and assessment infrastructure. Some gaps can be reduced by standardizing and linking existing resources; others require new observations, better access, or stronger outcome definitions. Framing the gap this way recognizes existing benchmarks while identifying precisely what they do not collectively establish.

14. Synthesis of the literature

The general benchmark literature and wildfire data literature converge on three methodological issues. First, the observation and annotation process determines what the target means. The empirical label audits in general AI have a close analogue in wildfire’s sensor-derived masks, interpolated arrival dates, administrative outcomes, and modeled exposure attributes. Second, the sampling process determines the population for which measured performance is informative. New image samples, held-out hospitals, large-fire thresholds, inspected buildings, and selectively mapped incidents all produce different forms of sampling dependence. Third, a reproducible result depends on the complete experimental artifact, including data transformations and evaluation software, rather than on the dataset name alone.

They diverge in how well the experimental protocol is established. Mature AI benchmarks commonly package a specific task with fixed evaluation conventions. Much wildfire infrastructure is broader and more heterogeneous: it supports many scientific questions but leaves the analyst to construct the learning problem. This is an advantage for research flexibility and a limitation for direct model comparison. Next Day Wildfire Spread and WildfireSpreadTS move further toward task-specific comparison; occurrence inventories, incident records, and mortality databases are better understood as reusable evidence infrastructures unless an accompanying study defines a particular assessment.

Several recurring sources of ambiguity emerge from the comparison. Apparent negatives may be unobserved fires, uninspected buildings, or unreported outcomes. Closely related pixels and repeated reports are not independent events. A reconstructed target may use observations acquired after the forecast time. Multiple products may share an upstream algorithm and therefore share errors. Aggregate metrics can conceal differences among regions, species, incident sizes, or exposure groups. These are methodological deductions from the documented constructions, rather than measured error rates established for every dataset in this review.

The most substantial public coverage is in fire detection, burned area, environmental covariates, and increasingly event progression. Public response datasets have become more structured, but records of decisions and their causal consequences remain different things. Building-loss datasets now include important exposure denominators and broader geographic coverage, while open insurance outcomes remain more restricted or aggregated in the sources examined. Ecological and atmospheric datasets extend the meaning of outcomes beyond burned area, with targets that require their own observation and validation frameworks.

Finally, being current and remaining comparable can pull in different directions. New observations broaden coverage, but changes in sensors, reporting systems, and algorithms can alter the historical reference. Fixed archived releases preserve a scientific comparison; evolving services preserve relevance to an unfolding environment. The literature documents both approaches, as well as hybrid arrangements that publish successive frozen releases from maintained pipelines. The substantive finding is that wildfire benchmarking already rests on a diverse set of data infrastructures, each making particular phenomena observable. The limits of those observations remain central to interpreting what an AI assessment actually demonstrates.

Appendix. Comparative evidence table

This table summarizes the distinctions developed in the review. “Benchmark” refers to an experimental protocol; an observational reference or research database can support benchmarking without prescribing that protocol.

References

Research papers, data releases, and living documentation are identified separately. Long author lists are abbreviated. All web sources were consulted for this review on or before 13 October 2026; dates are publication or release dates unless otherwise stated.