Space Intelligence - How we make maps

Deforestation Maps produced for use in the ICE CoT platform.

The ICE CoT EUDR Deforestation Maps are available under licence from ICE

Benchmark Administration Limited.

Space Intelligence creates 10 m resolution land cover maps to help support commodity traders and operators with their EUDR compliance obligations..

The maps we produce set the baseline of where land was and was not forest as of 31st December 2020, and where deforestation has taken place and new forest has grown since then to the present day.

These maps follow the EUDR definition of forest:

‘Forest’ means land spanning more than 0.5 hectares with trees higher than 5 metres and a canopy cover of more than 10%, or trees able to reach those thresholds in situ, excluding land that is predominantly under agricultural or urban use.

And the EUDR definition of deforestation, defined as:

‘Deforestation’ means the conversion of forest to agricultural use, whether

human-induced or not [after the cut-off date of 31st December 2020]

Following these definitions involve the difficult process of determining where exactly there was agriculture in 2020 and in the present day, even though such agriculture is often located under tree cover or is itself trees (for example in agroforestry systems such as cacao or coffee). In fact areas of trees may not meet the forest definition for a variety of reasons:

We use the following satellite datasets, taking images throughout the year in order to gather information on how every patch of land changes in different seasons. To account for differing initial resolutions and map projections, all the datasets are aligned and terrain-corrected to the same 10 m x 10 m pixel size grid. This grid is in an equal area projection (ESPG: 6933) to ensure every pixel is the same size.

We use any field data we have from the countries being mapped, field trips we make specially, open geographically located photos, existing mapping datasets, and our experts looking at very high resolution (<1 m) satellite data, to create a set of polygons we are very confident are particular land cover classes.

This includes making a lot of effort to create polygons that are definitely tree plantations, tree crops, and agriculture under trees, as well as open and closed forest. These polygons are critical in separating these classes, which often superficially look similar to satellite data.

We also create an independent test dataset of points, not used to create the maps, but used to determine when they are sufficiently accurate for use.

The satellite datasets are combined into data cubes covering a six month period. This period covers the final six months of 2020 (to produce maps showing the situation up to 31st December 2020), and the most recent six months (to cover the period to present). The more recent period will be updated annually, using data up to the 30th September of that

calendar year.

Cloud and cloud shadow are removed from the optical satellite data, and all dataset are terrain and radiometrically corrected and aligned.

Our data scientists then train and run a machine learning algorithm combining the truth polygons and the satellite data cube, to predict maps for end 2020 and the most recent period. These are tested against the independent test dataset. Based on the results and visual inspection of the map by our experts, we improve the map by re-running it making changes to the truth polygons, the satellite datasets used, or the machine learning

algorithm parameters, until we have a final map for a country that meets our accuracy requirements.

The 10m resolution maps are post processed to make the following changes:

It’s important to understand the performance (‘accuracy’) of land cover products to know if the maps are fit for purpose for a certain use case.

We take this very seriously in Space Intelligence and assess the accuracy of our products and their uncertainty in a statistically rigorous way, following

good practice as described in the scientific literature1 and international standards (2,3). We set targets for performance based on the requirements of a specific contract.

Accuracy can be tested for in a number of ways but the only approach that gives a meaningful assessment of the accuracy of a random pixel in the output map is approach 4. This is the approach used by Verra in the new Consolidated REDD+ Methodology (VM0048) and recommended by good practice guidance (1-3), but is not always followed by companies keen to parade high accuracy statistics. Refer to page 4 for further assessment of each approach.

It’s important to state that using a single metric (such as overall accuracy) is not a reliable way of describing map accuracy.

A much more interpretable and transparent way is to make use of an error matrix that shows commission and omission accuracies for each class (essentially assessing if the map is over or under-predicting certain classes). For example, in an area that has a deforestation rate of 1%, reporting no deforestation would give an overall accuracy of 99% while not being at all fit for purpose. Better to report in this case that the map has a 100% accuracy for unchanged forest, and a 0% accuracy for deforestation: and is thus clearly unsuitable.

For the EUDR, the focus is on separating natural forest from all other land cover types (which must include classes with trees, such as cacao). This enables an assessment of the likely accuracy of an assessment that a particular polygon in, say, 2020, was at that point natural forest, or was already non-forest.

At Space Intelligence we believe the independent assessment of isolated points following the best practices is the best way to assess the accuracy of forest /non-forest maps (approach 4 previously noted). This is done by our highly skilled ecologists and data scientists assessing hundreds of points ‘blind’, following a statistically valid sampling scheme, using high resolution satellite or aerial imagery, including any other existing field data from local partners. In order to match to the way our maps are used for testing under ICE CoT, with ‘core forest’ pixels slightly separate from the edge being considered, points very near the edge of forest patches are excluded from consideration.

Our aim is to exceed commission and omission accuracy of 90% for both forest and non-forest classes, with an Overall Accuracy over 95%.

Often we will exceed this considerably, but it does depend on the vegetation

types and complexity of the landscape. For example, in a Brazilian landscape

featuring large blocks of tropical forest and large soybean fields, with an average clearance size spanning tens of hectares, we would expect overall accuracies over 99%. But in a complex landscape with smallholder farmers, small deforestation patches (<1 ha), and crops containing trees (e.g. cacao), such levels of accuracy are not obtainable and should be viewed with suspicion.

Using our advanced methods (combining multiple types of satellite data) and

local knowledge, commission and omission accuracies over 90% are possible even in these complex landscapes.

The tests that ICE CoT do on our maps are based on the accuracies of our maps, and the understanding that errors in maps are concentrated particularly at the edges of forest and deforestation areas, and the likely accuracy of field GPS units used to map field boundaries.

Cambodia is a landscape where deforestation rates are high, with most clearing being small-scale for a variety of crops including tree crops such as palm oil, rubber and betel nuts. The forest is a complicated mixture of dry, moist and wet, including swamp areas and mangroves.

As a case study, we summarise below results of a 2020 land cover map that Space Intelligence did as part of a set of maps studying the land cover dynamics of the country for over 13 years. The maps show high accuracy and very little confusion between forest and non-forest, with us successfully excluding timber plantations, tree crops and rubber plantations from the forest class, as would be required for an EUDR assessment.

Table 1: Thematic accuracies for Space Intelligence’s Cambodia forest baseline for 2023 as assessed independently using a probabilistic sampling design, showing high overall accuracy but also low amounts of confusion between the classes. All values include the 95th confidence interval.

Figure 1: A subset of space intelligence’s Cambodia forest baseline, showing forest in green, non-forest in black and water in blue. Red dots are some of the thousands of random accuracy assessment points used to assess the accuracy of the map.

Accuracy can be tested for in a number of ways but the only approach that gives a meaningful assessment of the accuracy of a random pixel in the output map is approach 4. This is the approach used by Verra in the new Consolidated REDD+ Methodology (VM0048) and recommended by good

practice guidance1-3, but is not always followed by companies keen to parade high accuracy statistics.

Approach 1: A comparison of input training data (normally polygons drawn by eye with classes such as ‘forest’ or ‘non-forest’) compared to the output map. Normally the unit used is the proportion of pixels correctly classified (‘overall accuracy’), with individual pixels being the unit of assessment, not the whole polygons.

Assessment: While common, this method is not scientifically appropriate as there is no independence between test and training datasets.

A further issue is the use of pixels not polygons as the basis of comparison. Neighbouring pixels (for example all pixels making up a field cleared in a forest) are treated as independent samples, whereas in fact they are neighbours and share many more characteristics than two random pixels in the output map.

For both these reasons, it will inevitably overstate accuracy.

Approach 2: A similar comparison of the output map with polygons, but using independently produced polygons not used to train the map. Normally the unit used is the proportion of pixels correctly classified (‘overall accuracy’), with individual pixels being the unit of assessment, not the whole polygons.

Assessment: This shares the issue above of the use of treating non-independent neighbouring pixels as independent, overstating accuracy

Approach 3: Using data collected in-situ (field data).

Field data is often proposed as the ‘gold standard’ of validation data, and assumed of higher accuracy than other sources of data.

Assessment: However, because of accessibility and its elevated cost as compared to, for instance, interpretation of high resolution remote sensing data, the small sample size and narrow proportion of the area available for sampling means its results cannot be extrapolated reliably to large regions.

It also tends to be spatially biassed, which can be a problem when assessing land cover products, since accessible areas are typically more disturbed.

Approach 4: The independent assessment of a set of isolated points, placed over the entire output map using a statistically valid sampling approach (usually a grid or a stratified random sample).

Assessment: this approach provides a reliable assessment of the accuracy of different classes of a map, and the confidence intervals of that assessment. More points can be added in a statistically valid way until confidence intervals on the accuracy assessment are sufficiently narrow to meet requirements.