This module contains classes to work with Spot 6/7 products.
In particular, it specializes scenes.core.Source and scenes.core.Scene
for Spot 6/7 products.
Overview
We distinguish 2 kinds of products:
- DRS products
- IGN products (without any metadata, cloud mask, etc)
classDiagram
Scene <|-- Spot67Scene
Spot67Scene <|-- Spot67DRSScene
Spot67Scene <|-- Spot67IGNScene
Instantiation
Generic
Spot67Scene are instantiated from the assets_paths dictionary.
import scenes
sc_drs = scenes.spot.Spot67SceneDRS(assets_dict=...)
sc_ign = scenes.spot.Spot67SceneIGN(assets_dict=...)
The Spot67DRSScene instantiates as its parent class.
However, to ease working with products stored on the local filesystem, the
get_local_spot67drs_scene can help:
sc = get_local_spot67drs_scene(
dimap_xs="/path/to/DIM_SPOT7_MS_..._1.XML",
dimap_pan="/path/to/DIM_SPOT7_P_..._1.XML"
)
From STAC
The following example show how to instantiate on the fly Spot67SceneDRS from
a STAC catalog of Dinamis (prototype from CROCc).
api = Client.open(
'https://stacapi-dinamis.apps.okd.crocc.meso.umontpellier.fr',
modifier=dinamis_sdk.sign_inplace,
)
res = api.search(
collections=["spot-6-7-drs"],
bbox=[4.09, 43.99, 5, 44.01],
datetime=['2020-01-01', '2021-01-01']
)
for item in res.items():
sc = from_stac(item)
assert isinstance(sc, Spot67DRSScene)
print(sc)
The scene metadata can be accessed with the metadata attribute, like any
scenes.core.Scene instance.
ms = sc.metadata
scenes.utils.pprint(md)
Sources
The following sources are delivered:
- xs
- pan
- pxs
Imagery sources delivered from Spot67DRSScene are instances of
Spot67DRSSource. Imagery sources delivered from Spot67IGNScene are
instances of CommonImagerySource.
The Spot67DRSSource have a few more useful methods, of which:
- reflectance(): basically a wrapper of OTB OpticalCalibration. You can use
the same parameters, e.g. reflectance(level="toa")
- cld_msk_drilled(): this drills down the image in clouds polygons. The
no-data value can be changed (default is 0).
classDiagram
Source <|-- CommonImagerySource
CommonImagerySource <|-- Spot67DRSSource
class Spot67DRSSource{
+cld_msk_drilled(nodata=0)
+reflectance(**kwargs)
}
The following example show how to derive a child source replacing the
pixels that are in the clouds with zero-valued pixels:
pxs_drilled = pxs.cld_msk_drilled()
The Spot67Source inherits from scenes.core.CommonImagerySource, hence
implemented sources transformations (e.g.
scenes.core.CommonImagerySource.masked(),
scenes.core.CommonImagerySource.clip_over_img(),
scenes.core.CommonImagerySource.resample_over(),
scenes.core.CommonImagerySource.reproject(), etc.)
clipped = pxs_drilled.clip_over_img(roi)
reprojected = clipped.reproject(epsg=4328)
Note that the resulting transformed Spot67DRSSource is still an instance of
Spot67DRSSource after generic operations implemented in
scenes.core.CommonImagerySource.
Usage with pyotb
As scenes.core.Source, it also can be used like any pyotb.core.OTBObject.
The following example show how to use an OTB application with a source at
input.
rgb_nice = pyotb.DynamicConvert(reprojected)
rgb_nice.write("image.tif", pixel_type="uint8")
Bases: Spot67Scene
Spot 6/7 class for ADS-DRS products.
The Spot67Scene class carries all metadata and images sources from the
scene. A Spot67Scene object can be instantiated from the XS and PAN DIMAPS
(.XML) file.
Source code in scenes/spot.py
| class Spot67DRSScene(Spot67Scene):
"""
Spot 6/7 class for ADS-DRS products.
The Spot67Scene class carries all metadata and images sources from the
scene. A Spot67Scene object can be instantiated from the XS and PAN DIMAPS
(.XML) file.
"""
@property
def req_assets(self) -> Dict[str, str]:
"""
Required assets (overrides Spot67Scene property)
Returns: required assets
"""
return {
"dimap": "XML DIMAP document",
"src": "imagery source",
"roi_msk_vec": "region of interest mask (vector data)",
"cld_msk_vec": "clouds mask (vector data)",
}
def __init__(self, assets_paths: Dict[str, str]):
"""
Args:
assets_paths: assets paths
"""
# Parse dimap
assert "dimap_xs" in assets_paths, "XS DIMAP XML document is missing"
additional_md = spot67_metadata_parser(assets_paths["dimap_xs"])
acquisition_date = datetime.strptime(
additional_md["acquisition_date"], "%Y-%m-%dT%H:%M:%S.%fZ"
)
# Call parent constructor, before accessing to self.assets_paths
super().__init__(
assets_paths=assets_paths,
acquisition_date=acquisition_date,
src_class=Spot67DRSSource,
additional_metadata=additional_md,
)
# Choice of the CLD and ROI masks based on the pxs overlap
dic = self.metadata
ref = "pan" if dic["pan_overlap"] > dic["xs_overlap"] else "xs"
pths = self.assets_paths
for key in ["cld", "roi"]:
pths[f"{key}_msk_vec"] = pths[f"{key}_msk_vec_{ref}"]
|
Required assets (overrides Spot67Scene property)
Returns: required assets
Parameters:
| Name |
Type |
Description |
Default |
assets_paths |
Dict[str, str]
|
|
required
|
Source code in scenes/spot.py
| def __init__(self, assets_paths: Dict[str, str]):
"""
Args:
assets_paths: assets paths
"""
# Parse dimap
assert "dimap_xs" in assets_paths, "XS DIMAP XML document is missing"
additional_md = spot67_metadata_parser(assets_paths["dimap_xs"])
acquisition_date = datetime.strptime(
additional_md["acquisition_date"], "%Y-%m-%dT%H:%M:%S.%fZ"
)
# Call parent constructor, before accessing to self.assets_paths
super().__init__(
assets_paths=assets_paths,
acquisition_date=acquisition_date,
src_class=Spot67DRSSource,
additional_metadata=additional_md,
)
# Choice of the CLD and ROI masks based on the pxs overlap
dic = self.metadata
ref = "pan" if dic["pan_overlap"] > dic["xs_overlap"] else "xs"
pths = self.assets_paths
for key in ["cld", "roi"]:
pths[f"{key}_msk_vec"] = pths[f"{key}_msk_vec_{ref}"]
|
Bases: CommonImagerySource
Source capabilities for Spot-6/7 ADS-DRS products
Source code in scenes/spot.py
| class Spot67DRSSource(CommonImagerySource):
"""
Source capabilities for Spot-6/7 ADS-DRS products
"""
def cld_msk_drilled(
self, nodata: Union[float, int] = 0
) -> Spot67DRSSource:
"""
Return the source drilled from the cloud mask
Args:
nodata: nodata value inside holes (Default value = 0)
Returns:
drilled source
"""
return self.drilled(
self.root_scene.assets_paths["cld_msk_vec"], nodata=nodata
)
def reflectance(self, factor: float = None, **kwargs) -> Spot67DRSSource:
"""
This function is used internally by `get_xs()`, `get_pxs()`, and
`get_pan()` to compute the reflectance (or not!) of the optical image.
Warning: there is a bug in OTB<=8.0.2
see https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2314
Args:
factor: factor to scale pixel values (e.g. 10000)
**kwargs: same options as BundleToPerfectSensor in OTB
Returns:
calibrated, or original image source
"""
# Radiometry correction
params = {"in": self}
params.update(kwargs)
out = pyotb.OpticalCalibration(params)
if factor:
out = out * factor
return self.new_source(out)
|
Return the source drilled from the cloud mask
Parameters:
| Name |
Type |
Description |
Default |
nodata |
Union[float, int]
|
nodata value inside holes (Default value = 0)
|
0
|
Returns:
Source code in scenes/spot.py
| def cld_msk_drilled(
self, nodata: Union[float, int] = 0
) -> Spot67DRSSource:
"""
Return the source drilled from the cloud mask
Args:
nodata: nodata value inside holes (Default value = 0)
Returns:
drilled source
"""
return self.drilled(
self.root_scene.assets_paths["cld_msk_vec"], nodata=nodata
)
|
This function is used internally by get_xs(), get_pxs(), and
get_pan() to compute the reflectance (or not!) of the optical image.
Warning: there is a bug in OTB<=8.0.2
see https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2314
Parameters:
| Name |
Type |
Description |
Default |
factor |
float
|
factor to scale pixel values (e.g. 10000)
|
None
|
**kwargs |
|
same options as BundleToPerfectSensor in OTB
|
{}
|
Returns:
Source code in scenes/spot.py
| def reflectance(self, factor: float = None, **kwargs) -> Spot67DRSSource:
"""
This function is used internally by `get_xs()`, `get_pxs()`, and
`get_pan()` to compute the reflectance (or not!) of the optical image.
Warning: there is a bug in OTB<=8.0.2
see https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2314
Args:
factor: factor to scale pixel values (e.g. 10000)
**kwargs: same options as BundleToPerfectSensor in OTB
Returns:
calibrated, or original image source
"""
# Radiometry correction
params = {"in": self}
params.update(kwargs)
out = pyotb.OpticalCalibration(params)
if factor:
out = out * factor
return self.new_source(out)
|
Spot67IGNScene
Bases: Spot67Scene
Spot 6/7 class for IGN products.
A Spot67IGNScene object can be instantiated from the XS and PAN rasters
paths.
Source code in scenes/spot.py
| class Spot67IGNScene(Spot67Scene):
"""
Spot 6/7 class for IGN products.
A Spot67IGNScene object can be instantiated from the XS and PAN rasters
paths.
"""
def __init__(self, assets_paths: Dict[str, str]):
"""
Args:
assets_paths: assets paths
"""
# Retrieve date
assert "src_xs" in assets_paths, "XS image is missing"
assert "src_pan" in assets_paths, "PAN image is missing"
xs_basepath = utils.basename(assets_paths["src_xs"])
# 8 first consecutive digits in the XS filename
result = re.search(r"\d{8}", xs_basepath)
datestr = result.group(0) if result else result
acquisition_date = dates.str2datetime(datestr)
# Call parent constructor
super().__init__(
assets_paths=assets_paths,
acquisition_date=acquisition_date
)
|
__init__(assets_paths)
Parameters:
| Name |
Type |
Description |
Default |
assets_paths |
Dict[str, str]
|
|
required
|
Source code in scenes/spot.py
| def __init__(self, assets_paths: Dict[str, str]):
"""
Args:
assets_paths: assets paths
"""
# Retrieve date
assert "src_xs" in assets_paths, "XS image is missing"
assert "src_pan" in assets_paths, "PAN image is missing"
xs_basepath = utils.basename(assets_paths["src_xs"])
# 8 first consecutive digits in the XS filename
result = re.search(r"\d{8}", xs_basepath)
datestr = result.group(0) if result else result
acquisition_date = dates.str2datetime(datestr)
# Call parent constructor
super().__init__(
assets_paths=assets_paths,
acquisition_date=acquisition_date
)
|
Spot67Scene
Bases: Scene
Generic Spot 6/7 Scene class.
The Spot67Scene class carries PAN, XS, and PXS images sources.
Source code in scenes/spot.py
| class Spot67Scene(Scene):
"""
Generic Spot 6/7 Scene class.
The Spot67Scene class carries PAN, XS, and PXS images sources.
"""
PXS_OVERLAP_THRESH = 0.995
def _check_assets(self, assets_paths: Dict[str, str]):
"""
Checks that all assets are here
"""
for src_key in ["xs", "pan"]:
for req_prefix, req_description in self.req_assets.items():
key = f"{req_prefix}_{src_key}"
assert key in assets_paths, \
f"key '{key}' for {req_description} " \
"is missing from assets paths"
@property
def req_assets(self) -> Dict[str, str]:
"""
Required assets
Returns: required assets
"""
return {
"src": "imagery source",
}
def __init__(
self,
acquisition_date: datetime,
assets_paths: Dict[str, str],
src_class: Type[Source] = CommonImagerySource,
additional_metadata: Dict[str, str] = None,
):
"""
Args:
acquisition_date: Acquisition date
assets_paths: assets paths
additional_metadata: additional metadata
"""
# Get EPSG and bounding box
epsg_xs, extent_wgs84_xs = get_epsg_extent_wgs84(
assets_paths["src_xs"]
)
epsg_pan, extent_wgs84_pan = get_epsg_extent_wgs84(
assets_paths["src_pan"]
)
# Check that EPSG for PAN and XS are the same
if epsg_pan != epsg_xs:
raise ValueError(
f"EPSG of XS and PAN sources are different: XS EPSG is "
f"{epsg_xs}, PAN EPSG is {epsg_pan}")
# Here we compute bounding boxes overlap, to choose the most
# appropriated CLD and ROI masks for the scene. Indeed, sometimes
# products are not generated as they should (i.e. bundle) and XS and
# PAN have different extents so CLD and ROI masks are not the same for
# XS and PAN. We keep the ROI+CLD masks of the PAN or XS lying
# completely inside the other one.
xs_overlap = extent_overlap(extent_wgs84_xs, extent_wgs84_pan)
pan_overlap = extent_overlap(extent_wgs84_pan, extent_wgs84_xs)
# Throw some warning or error, depending on the pxs overlap value
pxs_overlap = max(xs_overlap, pan_overlap)
if pxs_overlap == 0:
raise ValueError("No overlap between PAN and XS images")
if max(pan_overlap, xs_overlap) < self.PXS_OVERLAP_THRESH:
print(f"Warning: partial overlap of {100 * pxs_overlap:.2f}%")
# Final extent
extent = extent_wgs84_pan if pan_overlap > xs_overlap else \
extent_wgs84_xs
sources = {
"xs": partial(
src_class, root_scene=self, out=assets_paths["src_xs"]
),
"pan": partial(
src_class, root_scene=self, out=assets_paths["src_pan"]
),
"pxs": partial(self._get_pxs, src_class=src_class)
}
additional_md = additional_metadata.copy() if additional_metadata \
else {}
additional_md.update({
"xs_extent_wgs84": extent_wgs84_xs,
"pan_extent_wgs84": extent_wgs84_pan,
"xs_overlap": xs_overlap,
"pan_overlap": pan_overlap
})
# Call parent constructor, before accessing assets dicts
super().__init__(
acquisition_date=acquisition_date,
epsg=epsg_xs,
extent_wgs84=extent,
assets_paths=assets_paths,
sources=sources,
additional_metadata=additional_md,
)
def _get_pxs(
self,
src_class: Type[CommonImagerySource],
**kwargs
) -> CommonImagerySource:
"""
Create source for PXS (computed using OTB)
Args:
src_class: source class
**kwargs: OTB BundleToPerfectSensor options, e.g. method="bayes"
Returns:
Source for BundleToPerfectSensor output
"""
pan = self.get_pan()
params = {
"inxs": self.get_xs(),
"inp": pan
}
params.update(kwargs)
pansharp = src_class(self, pyotb.BundleToPerfectSensor(params))
binary_mask = pyotb.Superimpose({
"inr": pansharp,
"inm": pan,
"interpolator": "nn"
})
return pansharp.masked(binary_mask=binary_mask)
|
req_assets: Dict[str, str]
property
Required assets
Returns: required assets
__init__(acquisition_date, assets_paths, src_class=CommonImagerySource, additional_metadata=None)
Parameters:
| Name |
Type |
Description |
Default |
acquisition_date |
datetime
|
|
required
|
assets_paths |
Dict[str, str]
|
|
required
|
additional_metadata |
Dict[str, str]
|
|
None
|
Source code in scenes/spot.py
| def __init__(
self,
acquisition_date: datetime,
assets_paths: Dict[str, str],
src_class: Type[Source] = CommonImagerySource,
additional_metadata: Dict[str, str] = None,
):
"""
Args:
acquisition_date: Acquisition date
assets_paths: assets paths
additional_metadata: additional metadata
"""
# Get EPSG and bounding box
epsg_xs, extent_wgs84_xs = get_epsg_extent_wgs84(
assets_paths["src_xs"]
)
epsg_pan, extent_wgs84_pan = get_epsg_extent_wgs84(
assets_paths["src_pan"]
)
# Check that EPSG for PAN and XS are the same
if epsg_pan != epsg_xs:
raise ValueError(
f"EPSG of XS and PAN sources are different: XS EPSG is "
f"{epsg_xs}, PAN EPSG is {epsg_pan}")
# Here we compute bounding boxes overlap, to choose the most
# appropriated CLD and ROI masks for the scene. Indeed, sometimes
# products are not generated as they should (i.e. bundle) and XS and
# PAN have different extents so CLD and ROI masks are not the same for
# XS and PAN. We keep the ROI+CLD masks of the PAN or XS lying
# completely inside the other one.
xs_overlap = extent_overlap(extent_wgs84_xs, extent_wgs84_pan)
pan_overlap = extent_overlap(extent_wgs84_pan, extent_wgs84_xs)
# Throw some warning or error, depending on the pxs overlap value
pxs_overlap = max(xs_overlap, pan_overlap)
if pxs_overlap == 0:
raise ValueError("No overlap between PAN and XS images")
if max(pan_overlap, xs_overlap) < self.PXS_OVERLAP_THRESH:
print(f"Warning: partial overlap of {100 * pxs_overlap:.2f}%")
# Final extent
extent = extent_wgs84_pan if pan_overlap > xs_overlap else \
extent_wgs84_xs
sources = {
"xs": partial(
src_class, root_scene=self, out=assets_paths["src_xs"]
),
"pan": partial(
src_class, root_scene=self, out=assets_paths["src_pan"]
),
"pxs": partial(self._get_pxs, src_class=src_class)
}
additional_md = additional_metadata.copy() if additional_metadata \
else {}
additional_md.update({
"xs_extent_wgs84": extent_wgs84_xs,
"pan_extent_wgs84": extent_wgs84_pan,
"xs_overlap": xs_overlap,
"pan_overlap": pan_overlap
})
# Call parent constructor, before accessing assets dicts
super().__init__(
acquisition_date=acquisition_date,
epsg=epsg_xs,
extent_wgs84=extent,
assets_paths=assets_paths,
sources=sources,
additional_metadata=additional_md,
)
|
find_all_dimaps(pth)
Return the list of DIMAPS XML files that are inside all subdirectories of
the root directory.
Parameters:
| Name |
Type |
Description |
Default |
pth |
str
|
|
required
|
Returns:
| Type |
Description |
List[str]
|
|
Source code in scenes/spot.py
| def find_all_dimaps(pth: str) -> List[str]:
"""
Return the list of DIMAPS XML files that are inside all subdirectories of
the root directory.
Args:
pth: root directory
Returns:
list of DIMAPS XML files
"""
return utils.find_files_in_all_subdirs(pth=pth, pattern="DIM_*.XML")
|
get_local_spot67drs_scene(dimap_xs, dimap_pan)
Retrieve all required assets from the dimaps of the product.
Parameters:
| Name |
Type |
Description |
Default |
dimap_xs |
str
|
XML document for the XS image
|
required
|
dimap_pan |
str
|
XML document for the PAN image
|
required
|
Returns:
Source code in scenes/spot.py
| def get_local_spot67drs_scene(dimap_xs: str, dimap_pan: str) -> Spot67DRSScene:
"""
Retrieve all required assets from the dimaps of the product.
Args:
dimap_xs: XML document for the XS image
dimap_pan: XML document for the PAN image
Returns:
Instantiated scene
"""
# Get assets paths
def _check(dimap_file):
if not dimap_file.lower().endswith(".xml"):
raise FileNotFoundError(
f"An input XML file is needed (provided: {dimap_file})"
)
_check(dimap_xs)
_check(dimap_pan)
assets_paths = {
"dimap_xs": dimap_xs,
"dimap_pan": dimap_pan,
"src_xs": dimap_xs,
"src_pan": dimap_pan,
}
# Get ROI+CLD filenames in XS and PAN products, then set the final ROI+CLD
# filenames based on PAN/XS overlap
for key, pat in {"cld": "CLD", "roi": "ROI"}.items():
for src in ["xs", "pan"]:
pattern = f"{pat}*.GML"
cld_path = os.path.join(
utils.get_parent_directory(assets_paths[f"dimap_{src}"]),
"MASKS")
plist = utils.find_file_in_dir(cld_path, pattern=pattern)
if len(plist) != 1:
raise FileNotFoundError(
f"ERROR: unable to find a unique file in {cld_path} with "
f"pattern {pattern}"
)
assets_paths[f"{key}_msk_vec_{src}"] = plist[0]
return Spot67DRSScene(assets_paths=assets_paths)
|
get_spot67_scenes(root_dir)
Return the list of scenes that can be instantiated from a root directory
containing a "PAN" and a "MS" subdirectories.
Parameters:
| Name |
Type |
Description |
Default |
root_dir |
str
|
directory containing "MS" and "PAN" subdirectories
|
required
|
Returns:
| Type |
Description |
List[Spot67Scene]
|
list of Spot67Scenes instances
|
Source code in scenes/spot.py
| def get_spot67_scenes(root_dir: str) -> List[Spot67Scene]:
"""
Return the list of scenes that can be instantiated from a root directory
containing a "PAN" and a "MS" subdirectories.
Args:
root_dir: directory containing "MS" and "PAN" subdirectories
Returns:
list of Spot67Scenes instances
"""
# List files
look_dir = root_dir + "/MS"
print(f"Find files in {look_dir}")
dimap_xs_files = find_all_dimaps(look_dir)
print(f"Found {len(dimap_xs_files)} DIMAP files in MS folder", flush=True)
# Create scenes list
scenes = []
errors = {}
for dimap_file_xs in tqdm(dimap_xs_files):
try:
# Find pairs of XS/PAN DIMAPS
pan_path = dimap_file_xs[:dimap_file_xs.find("/PROD_SPOT")]
pan_path = pan_path.replace("/MS/", "/PAN/")
pan_path = pan_path.replace("_MS_", "_PAN_")
dimap_pan_files = find_all_dimaps(pan_path)
nb_files = len(dimap_pan_files)
if nb_files != 1:
raise ValueError(
f"{nb_files} DIMAPS candidates found in {pan_path} ")
dimap_file_pan = dimap_pan_files[0]
# Instantiate a new scene object
new_scene = get_local_spot67drs_scene(
dimap_xs=dimap_file_xs, dimap_pan=dimap_file_pan
)
scenes.append(new_scene)
except Exception as error:
if dimap_file_xs not in errors:
errors[dimap_file_xs] = []
errors[dimap_file_xs].append(error)
raise
print(f"{len(scenes)} scenes imported")
if errors:
print(f"{len(errors)} scenes could not have been imported.")
for dimap_file_xs, error_list in errors.items():
print(f"Errors for {dimap_file_xs}:")
for error in error_list:
print(f"\t{error}")
return scenes
|
Parse DIMAP XML file
Parameters:
| Name |
Type |
Description |
Default |
xml_path |
str
|
|
required
|
Returns:
| Type |
Description |
Dict[str, str]
|
|
Source code in scenes/spot.py
| def spot67_metadata_parser(xml_path: str) -> Dict[str, str]:
"""
Parse DIMAP XML file
Args:
xml_path: XML file
Returns:
a dict of metadata
"""
# Detect if document is remote or local
if any(prefix in xml_path for prefix in ("http://", "https://")):
root = ET.fromstring(requests.get(xml_path, timeout=10).text)
else:
root = ET.parse(xml_path).getroot()
scalars_mappings = {
"Geometric_Data/Use_Area/Located_Geometric_Values/Acquisition_Angles":
{
"AZIMUTH_ANGLE": "azimuth_angle",
"VIEWING_ANGLE_ACROSS_TRACK": "viewing_angle_across",
"VIEWING_ANGLE_ALONG_TRACK": "viewing_angle_along",
"VIEWING_ANGLE": "viewing_angle",
"INCIDENCE_ANGLE": "incidence_angle",
},
"Geometric_Data/Use_Area/Located_Geometric_Values":
{
"TIME": "acquisition_date"
},
"Geometric_Data/Use_Area/Located_Geometric_Values/Solar_Incidences":
{
"SUN_AZIMUTH": "sun_azimuth",
"SUN_ELEVATION": "sun_elevation"
}
}
metadata = {}
for section, scalars_mapping in scalars_mappings.items():
for node in root.find(section):
key = node.tag
if key in scalars_mapping:
new_key = scalars_mapping[key]
text = node.text
metadata[new_key] = float(text) if text.isdigit() else text
return metadata
|