gms_preprocessing.model package

Submodules

gms_preprocessing.model.gms_object module

class gms_preprocessing.model.gms_object.GMS_identifier(image_type, satellite, sensor, subsystem, proc_level, dataset_ID, logger=None)

Bases: object

to_odict(include_logger=True)

class gms_preprocessing.model.gms_object.GMS_object(pathImage='')

Bases: object

property GMS_identifier

static LBA2bandnames(LayerBandsAssignment)

Convert LayerbandsAssignment from format [‘1’,’2’,…] to bandnames like this: [B01, .., B8A,].

property LayerBandsAssignment

property MetaObj

property ac_errors

Returns an instance of GeoArray containing error information calculated by the atmospheric correction.

Returns

property ac_options

Returns the options dictionary needed as input for atmospheric correction. If an empty dictionary is returned, atmospheric correction is not yet available for the current sensor and will later be skipped.

Return type

dict

property accuracy_layers

property accuracy_layers_meta

apply_nodata_mask_to_ObjAttr(attrname, out_nodata_val=None)

Applies self.mask_nodata to the specified array attribute by setting all values where mask_nodata is 0 to the given nodata value.

Parameters

• attrname (str) – The attribute to apply the nodata mask to. Must be an array attribute or a string path to a previously saved ENVI-file.

• out_nodata_val (Optional[int]) – set the values of the given attribute to this value.

Return type

None

apply_nodata_mask_to_saved_ENVIfile(path_saved_ENVIhdr, custom_nodata_val=None, update_spec_vals=False)

Applies self.mask_nodata to a saved ENVI file with the same X/Y dimensions like self.mask_nodata by setting all values where mask_nodata is 0 to the given nodata value.

Parameters

• path_saved_ENVIhdr (str) – <str> The path of the ENVI file to apply the nodata mask to.

• custom_nodata_val (Optional[int]) – <int> set the values of the given attribute to this value.

• update_spec_vals (bool) – <bool> whether to update self.MetaObj.spec_vals[‘fill’]

Return type

None

property arr

Return type

GeoArray

property arr_meta

attributes2dict(remove_privates=False)

Returns a copy of the current object dictionary including the current values of all object properties.

Return type

dict

block_at_system_overload(max_usage=95, timeout=900)

build_combined_masks_array()

Generates self.masks attribute (unsigned integer 8bit) from by concatenating all masks included in GMS obj. The corresponding metadata is assigned to L1A_obj.masks_meta. Empty mask attributes are skipped.

Return type

dict

calc_mask_nodata(fromBand=None, overwrite=False)

Calculates a no data mask with (values: 0=nodata; 1=data)

Parameters

• fromBand – <int> index of the band to be used (if None, all bands are used)

• overwrite – <bool> whether to overwrite existing nodata mask that has already been calculated

Returns

close_loggers()

property cloud_masking_algorithm

combine_tiles_to_ObjAttr(tiles, target_attr)

Combines tiles, e.g. produced by L1A_P.L1A_object.DN2TOARadRefTemp() to a single attribute. If CFG.inmem_serialization is False, the produced attribute is additionally written to disk.

Parameters

• tiles (list) – <list> a list of dictionaries with the keys ‘desc’, ‘data’, ‘row_start’,’row_end’, ‘col_start’ and ‘col_end’

• target_attr (str) – <str> the name of the attribute to be produced

Return type

None

property coreg_info

property coreg_needed

delete_previous_proc_level_results()

Deletes results of the previous processing level if the respective flag CFG.exec_L**P[2]) is set to True. The function is skipped if the results of the current processing level have not yet been written.

delete_tempFiles()

Delete all temporary files that have been written during GMS object processing.

property dem

Returns an SRTM DEM in the exact dimension an pixel grid of self.arr as an instance of GeoArray.

property dict_LayerOptTherm

flush_array_data()

classmethod from_disk(tuple_GMS_subset)

Fills an already instanced GMS object with data from disk. Excludes array attributes in Python mode.

Parameters

tuple_GMS_subset – <tuple> e.g. (‘/path/gms_file.gms’, [‘cube’, None])

classmethod from_sensor_subsystems(list_GMS_objs)

Merge separate GMS objects belonging to the same scene-ID into ONE GMS object.

Parameters

list_GMS_objs (List[GMS_object]) – <list> of GMS objects covering the same geographic area but representing different sensor subsystems (e.g. 3 GMS_objects for Sentinel-2 10m/20m/60m bands)

Return type

GMS_object

classmethod from_tiles(list_GMS_tiles)

Merge separate GMS objects with different spatial coverage but belonging to one scene-ID to ONE GMS object.

Parameters

list_GMS_tiles (list) – <list> of GMS objects that have been created by cut_GMS_obj_into_blocks()

Return type

GMS_object

generate_accuracy_layers()

property georef

Returns True if the current dataset can serve as spatial reference.

get_copied_dict_and_props(remove_privates=False)

Returns a copy of the current object dictionary including the current values of all object properties.

Return type

dict

get_subset_obj(imBounds=None, mapBounds=None, mapBounds_prj=None, out_prj=None, logmsg=None, progress=False, v=False)

Return a subset of the given GMS object, based on the given bounds coordinates.

Array attributes are clipped and relevant metadata keys are updated according to new extent. In case the subset does not contain any data but only no-data values, None is returned.

Parameters

• imBounds (Optional[tuple]) – <tuple> tuple of image coordinates in the form (xmin,xmax,ymin,ymax)

• mapBounds (Optional[tuple]) – <tuple> tuple of map coordinates in the form (xmin,xmax,ymin,ymax)

• mapBounds_prj (Optional[str]) – <str> a WKT string containing projection of the given map bounds (can be different to projection of the GMS object; ignored if map bounds not given)

• out_prj (Optional[str]) – <str> a WKT string containing output projection. If not given, the projection of self.arr is used.

• logmsg (Optional[str]) – <str> a message to be logged when this method is called

• progress (bool) – <bool> whether to show progress bar (default: False)

• v (bool) – <bool> verbose mode (default: False)

Return type

Optional[GMS_object]

Returns

<GMS_object> the GMS object subset

get_tilepos(target_tileshape, target_tilesize)

property log

Returns a string of all logged messages until now.

log_for_fullArr_or_firstTile(log_msg, subset=None)

Send a message to the logger only if full array or the first tile is currently processed. This function can be called when processing any tile but log message will only be sent from first tile.

Parameters

• log_msg – the log message to be logged

• subset – subset argument as sent to e.g. DN2TOARadRefTemp that indicates which tile is to be processed. Not needed if self.arr_pos is not None.

property logger

property mask_clouds

property mask_clouds_confidence

property mask_nodata

property masks

property pathGen

Returns the path generator object for generating file pathes belonging to the GMS object.

Return type

path_generator

property proc_status

Get the processing status of the current GMS_object (subclass) instance for the current processing level.

Possible values: ‘initialized’, ‘running’, ‘finished’, ‘failed’

Return type

str

proc_status_all_GMSobjs = {}

record_mem_usage()

Record memory usage of current process ID for the current processing level (megabytes).

property resamp_needed

static rescale_array(inArray, outScaleFactor, inScaleFactor=1)

Adjust the scaling factor of an array to match the given output scale factor.

set_pathes()

property spat_homo_errors

property spec_homo_errors

Returns an instance of GeoArray containing error information calculated during spectral homogenization.

Returns

property subset

to_ENVI(write_masks_as_ENVI_classification=True, is_tempfile=False, compression=False)

Write GMS object to disk. Supports full cubes AND ‘block’ tiles.

Parameters

• self – <object> GMS object, e.g. L1A_P.L1A_object

• write_masks_as_ENVI_classification (object) – <bool> whether to write masks as ENVI classification file

• is_tempfile (bool) –

  <bool> whether output represents a temporary file -> suppresses logging and database updating - ATTENTION! This keyword asserts that the actual output file that

  is written later contains the final version of the array. The array is not overwritten or written once more later, but only renamed.

• compression (bool) – <bool> enable or disable compression

Return type

None

to_GMS_file(path_gms_file=None)

to_MGRS_tiles(pixbuffer=10, v=False)

Returns a generator object where items represent the MGRS tiles for the GMS object.

Parameters

• pixbuffer (int) – <int> a buffer in pixel values used to generate an overlap between the returned MGRS tiles

• v (bool) – <bool> verbose mode

Return type

Generator[GMS_object]

Returns

<list> of MGRS_tile objects

to_tiles(blocksize=(2048, 2048))

Returns a generator object where items represent tiles of the given block size for the GMS object.

# NOTE: it’s better to call get_subset_obj (also takes care of tile map infos)

Parameters

blocksize (tuple) – target dimensions of the generated block tile (rows, columns)

Return type

Generator[GMS_object]

Returns

<list> of GMS_object tiles

validate_pathes()

write_tiles_to_ENVIfile(tiles, overwrite=True)

Writes tiles, e.g. produced by L1A_P.L1A_object.DN2TOARadRefTemp() to a single output ENVI file.

Parameters

• tiles (list) – <list> a list of dictionaries with the keys ‘desc’, ‘data’, ‘row_start’,’row_end’, ‘col_start’ and ‘col_end’

• overwrite (bool) – whether to overwrite files that have been produced earlier

Return type

None

gms_preprocessing.model.gms_object.GMS_object_2_dataset_dict(GMS_obj)

Return type

OrderedDict

gms_preprocessing.model.gms_object.estimate_mem_usage(dataset_ID, satellite)

class gms_preprocessing.model.gms_object.failed_GMS_object(GMS_object_or_OrdDict, failedMapper, exc_type, exc_val, exc_tb)

Bases: gms_preprocessing.model.gms_object.GMS_object

delete_tempFiles()

Delete all temporary files that have been written during GMS object processing.

property pandasRecord

class gms_preprocessing.model.gms_object.finished_GMS_object(GMS_obj)

Bases: gms_preprocessing.model.gms_object.GMS_object

property pandasRecord

gms_preprocessing.model.gms_object.return_GMS_objs_without_arrays(GMS_pipeline)

Decorator function for flushing any array attributes within the return value of a GMS pipeline function.

Parameters

GMS_pipeline – A GMS mapper function that takes a GMS object, does some processing and returns it back.

gms_preprocessing.model.gms_object.return_proc_reports_only(GMS_pipeline)

Decorator function for flushing any array attributes within the return value of a GMS pipeline function.

Parameters

GMS_pipeline – A GMS mapper function that takes a GMS object, does some processing and returns it back.

gms_preprocessing.model.gms_object.update_proc_status(GMS_mapper)

Decorator function for updating the processing status of each GMS_object (subclass) instance.

Parameters

GMS_mapper – A GMS mapper function that takes a GMS object, does some processing and returns it back.

gms_preprocessing.model.metadata module

Module ‘metadata’ for handling any type of metadata of GeoMultiSens compatible sensor systems.

gms_preprocessing.model.metadata.LandsatID2dataset(ID_list)

class gms_preprocessing.model.metadata.METADATA(GMS_id)

Bases: object

property AcqDateTime

Returns a datetime.datetime object containing date, time and timezone (UTC time).

property LayerBandsAssignment_full

Return complete LayerBandsAssignment without excluding thermal or panchromatic bands.

NOTE: CFG.sort_bands_by_cwl is respected, so returned list may be sorted by central wavelength

Return type

list

Read_ALOS_LEADER()

Read metadata from ALOS leader file. binary.

For exact information content see: file:///misc/ro2/behling/Satelliten/ALOS/doc/ALOS Product Format description.pdf

Read_ALOS_summary()

—-METHOD_6———————————————————— read metadata from ALOS summary.txt

Read_ASTER_hdffile(subsystem)

#—-METHOD_5———————————————————— read metadata from ASTER hdf input:

hdffile: subsystem:

output: :param subsystem:

Read_LANDSAT_mtltxt(LayerBandsAssignment)

—-METHOD_3———————————————————— read metadata from LANDSAT metafile: <dataname>.MTL.txt. Metadatafile of LPGS processing chain :param LayerBandsAssignment:

Read_RE_metaxml()

—-METHOD_4———————————————————— read metadata from RapidEye metafile: <dataname>metadata.xml

Read_SPOT_dimap2()

—-METHOD_2———————————————————— # read metadata from spot dimap file

Read_Sentinel2_xmls()

Read metadata from Sentinel-2 generic xml and granule xml

add_rasObj_dims_projection_physUnit(rasObj, dict_LayerOptTherm, temp_logger=None)

property bandnames

calc_center_acquisition_time(fullSceneCornerLonLat, logger)

Calculates a missing center acquistion time using acquisition date, full scene corner coordinates and solar azimuth.

Parameters

• fullSceneCornerLonLat –

• logger –

calc_solar_irradiance_CWL_FWHM_per_band()

Return type

(dict, dict, dict)

filter_layerdependent_metadata()

from_odict(odict)

Return type

METADATA

get_EarthSunDistance(acqDate)

Get earth sun distance (requires file of pre calculated earth sun distance per day)

Parameters

acqDate –

get_overpassDuration_SceneLength(fullSceneCornerLonLat, fullSceneCornerPos, shape_fullArr, logger)

Calculates duration of image acquisition in seconds.

Parameters

• fullSceneCornerLonLat –

• fullSceneCornerPos –

• shape_fullArr –

• logger –

property overview

read_meta(scene_ID, stacked_image, data_folderOrArchive, LayerBandsAssignment=None)

Read metadata.

to_odict()

Creates an OrderedDict containing selected attribute of the METADATA object that will later be included in ENVI file headers in the same order.

Return type

OrderedDict

gms_preprocessing.model.metadata.get_FieldOfView(GMS_id)

gms_preprocessing.model.metadata.get_LayerBandsAssignment(GMS_id, nBands=None, sort_by_cwl=None, no_thermal=None, no_pan=None, return_fullLBA=False, proc_level='')

Returns LayerBandsAssignment corresponding to given satellite, sensor and subsystem and with respect to CFG.sort_bands_by_cwl, CFG.skip_thermal and CFG.skip_pan.

Parameters

• GMS_id (GMS_identifier) – <dict>, derived from self.get_GMS_identifier() NOTE: only if there is an additional key ‘proc_level’, the processing level will be respected. This is needed to get the correct LBA after atm. correction

• nBands (int) – should be specified if number of bands differs from standard (e.g. SPOT data containing no PAN)

• sort_by_cwl (bool) – whether to sort the returned bands list by central wavelength position (default: CFG.sort_bands_by_cwl)

• no_thermal (bool) – whether to exclude thermal bands from the returned bands list (default: CFG.skip_thermal)

• no_pan (bool) – whether to exclude panchromatic bands from the returned bands list (default: CFG.skip_pan)

• return_fullLBA (bool) – in case there is a subsystem: whether to return LayerBandsAssignment for all bands or for the current subsystem

• proc_level (str) – processing level for which the LayerBandsAssignment is returned (overrides the proc_level given with GMS_id)

Return type

list

gms_preprocessing.model.metadata.get_bands_after_AC(GMS_id)

Returns a list of bands that are not removed by atmospheric correction.

Parameters

GMS_id (GMS_identifier) – <dict>, derived from self.get_GMS_identifier()

Return type

List[str]

Returns

e.g. [‘1’, ‘2’, ‘3’, ‘4’, ‘5’, ‘6’, ‘7’, ‘9’] for Landsat-8

gms_preprocessing.model.metadata.get_center_wavelengths_by_LBA(satellite, sensor, LBA, subsystem=None)

Returns a list of center wavelengths of spectral bands for the given satellite/sensor/LayerBandsAss. combination.

Parameters

• satellite (str) – target satellite (e.g., ‘Sentinel-2A’)

• sensor (str) – target sensor (e.g., ‘MSI’)

• LBA (list) – LayerBandsAssignment

• subsystem (Optional[str]) – target sensor subsystem (e.g., ‘VNIR’)

Return type

List[float]

gms_preprocessing.model.metadata.get_dict_LayerOptTherm(GMS_id, LayerBandsAssignment)

gms_preprocessing.model.metadata.get_orbit_params(GMS_id)

gms_preprocessing.model.metadata.get_sensormode(dataset)

gms_preprocessing.model.metadata.get_special_values(GMS_id)

gms_preprocessing.model.metadata.isPAN(GMS_id, LayerNr)

gms_preprocessing.model.metadata.isTHERMAL(GMS_id, LayerNr)

gms_preprocessing.model.metadata.metaDict_to_metaODict(metaDict, logger=None)

Converts a GMS metadata dictionary to an ordered dictionary according to the sorting given in Output_writer.enviHdr_keyOrder.

Parameters

• metaDict – <dict> GMS metadata dictionary

• logger – <logging.logger> if given, warnings will be logged. Otherwise they are raised.

gms_preprocessing.model.mgrs_tile module

class gms_preprocessing.model.mgrs_tile.MGRS_tile(tile_ID='')

Bases: object

Parameters

tile_ID – <str> 5 digit tile ID, e.g. 32UUU

property EPSG

property UTMzone

clip_array_using_mgrsBounds(array, im_prj, im_gt, nodataVal=0, pixbuffer=0)

Parameters

• array –

• im_prj –

• im_gt –

• nodataVal –

• pixbuffer – <float> an optional buffer size (image pixel units)

property geom_wkb

get_bounds(prj=None)

property grid100k

property grid1mil

property poly_lonlat

poly_specPrj(prj)

Returns a shapely.Polygon in a specific projection.

Parameters

prj – <str> WKT string of the target projection

property poly_utm

property tile_ID

to_image_bounds(im_prj, im_gt, arr_shape, pixbuffer=0, ensure_valid_coords=True)

Module contents