gms_preprocessing.options package

Submodules

gms_preprocessing.options.config module

class gms_preprocessing.options.config.GMSValidator(*args, **kwargs)

Bases: cerberus.validator.Validator

Validator class. Normalizes and/or validates any mapping against a validation-schema which is provided as an argument at class instantiation or upon calling the validate(), validated() or normalized() method. An instance itself is callable and executes a validation.

All instantiation parameters are optional.

There are the introspective properties types, validators, coercers, default_setters, rules, normalization_rules and validation_rules.

The attributes reflecting the available rules are assembled considering constraints that are defined in the docstrings of rules’ methods and is effectively used as validation schema for schema.

Parameters

• schema (any mapping) – See schema. Defaults to None.

• ignore_none_values (bool) – See ignore_none_values. Defaults to False.

• allow_unknown (bool or any mapping) – See allow_unknown. Defaults to False.

• require_all (bool) – See require_all. Defaults to False.

• purge_unknown (bool) – See purge_unknown. Defaults to to False.

• purge_readonly (bool) – Removes all fields that are defined as readonly in the normalization phase.

• error_handler (class or instance based on BaseErrorHandler or tuple) – The error handler that formats the result of errors. When given as two-value tuple with an error-handler class and a dictionary, the latter is passed to the initialization of the error handler. Default: BasicErrorHandler.

• args – Arguments to be passed to cerberus.Validator

• kwargs – Keyword arguments to be passed to cerberus.Validator

checkers = ()

coercers = ()

default_setters = ()

normalization_rules = {'coerce': {'oneof': [{'type': 'callable'}, {'type': 'list', 'schema': {'oneof': [{'type': 'callable'}, {'type': 'string', 'allowed': ()}]}}, {'type': 'string', 'allowed': ()}]}, 'default': {'nullable': True}, 'default_setter': {'oneof': [{'type': 'callable'}, {'type': 'string', 'allowed': ()}]}, 'purge_unknown': {'type': 'boolean'}, 'rename': {'type': 'hashable'}, 'rename_handler': {'oneof': [{'type': 'callable'}, {'type': 'list', 'schema': {'oneof': [{'type': 'callable'}, {'type': 'string', 'allowed': ()}]}}, {'type': 'string', 'allowed': ()}]}}

rules = {'allof': {'logical': 'allof', 'type': 'list'}, 'allow_unknown': {'oneof': [{'type': 'boolean'}, {'type': ['dict', 'string'], 'check_with': 'bulk_schema'}]}, 'allowed': {'type': 'container'}, 'anyof': {'logical': 'anyof', 'type': 'list'}, 'check_with': {'oneof': [{'type': 'callable'}, {'type': 'list', 'schema': {'oneof': [{'type': 'callable'}, {'type': 'string', 'allowed': ()}]}}, {'type': 'string', 'allowed': ()}]}, 'coerce': {'oneof': [{'type': 'callable'}, {'type': 'list', 'schema': {'oneof': [{'type': 'callable'}, {'type': 'string', 'allowed': ()}]}}, {'type': 'string', 'allowed': ()}]}, 'contains': {'empty': False}, 'default': {'nullable': True}, 'default_setter': {'oneof': [{'type': 'callable'}, {'type': 'string', 'allowed': ()}]}, 'dependencies': {'check_with': 'dependencies', 'type': ('dict', 'hashable', 'list')}, 'empty': {'type': 'boolean'}, 'excludes': {'schema': {'type': 'hashable'}, 'type': ('hashable', 'list')}, 'forbidden': {'type': 'list'}, 'items': {'check_with': 'items', 'type': 'list'}, 'keysrules': {'check_with': 'bulk_schema', 'forbidden': ['rename', 'rename_handler'], 'type': ['dict', 'string']}, 'max': {'nullable': False}, 'maxlength': {'type': 'integer'}, 'meta': {}, 'min': {'nullable': False}, 'minlength': {'type': 'integer'}, 'noneof': {'logical': 'noneof', 'type': 'list'}, 'nullable': {'type': 'boolean'}, 'oneof': {'logical': 'oneof', 'type': 'list'}, 'purge_unknown': {'type': 'boolean'}, 'readonly': {'type': 'boolean'}, 'regex': {'type': 'string'}, 'rename': {'type': 'hashable'}, 'rename_handler': {'oneof': [{'type': 'callable'}, {'type': 'list', 'schema': {'oneof': [{'type': 'callable'}, {'type': 'string', 'allowed': ()}]}}, {'type': 'string', 'allowed': ()}]}, 'require_all': {'type': 'boolean'}, 'required': {'type': 'boolean'}, 'schema': {'anyof': [{'check_with': 'schema'}, {'check_with': 'bulk_schema'}], 'type': ['dict', 'string']}, 'type': {'check_with': 'type', 'type': ['string', 'list']}, 'valuesrules': {'check_with': 'bulk_schema', 'forbidden': ['rename', 'rename_handler'], 'type': ['dict', 'string']}}

validate(document2validate, **kwargs)

Normalizes and validates a mapping against a validation-schema of defined rules.

Parameters

• document (any mapping) – The document to normalize.

• schema (any mapping) – The validation schema. Defaults to None. If not provided here, the schema must have been provided at class instantiation.

• update (bool) – If True, required fields won’t be checked.

• normalize (bool) – If True, normalize the document before validation.

Returns

True if validation succeeds, otherwise False. Check the errors() property for a list of processing errors.

Return type

bool

validation_rules = {'allof': {'logical': 'allof', 'type': 'list'}, 'allow_unknown': {'oneof': [{'type': 'boolean'}, {'type': ['dict', 'string'], 'check_with': 'bulk_schema'}]}, 'allowed': {'type': 'container'}, 'anyof': {'logical': 'anyof', 'type': 'list'}, 'check_with': {'oneof': [{'type': 'callable'}, {'type': 'list', 'schema': {'oneof': [{'type': 'callable'}, {'type': 'string', 'allowed': ()}]}}, {'type': 'string', 'allowed': ()}]}, 'contains': {'empty': False}, 'dependencies': {'check_with': 'dependencies', 'type': ('dict', 'hashable', 'list')}, 'empty': {'type': 'boolean'}, 'excludes': {'schema': {'type': 'hashable'}, 'type': ('hashable', 'list')}, 'forbidden': {'type': 'list'}, 'items': {'check_with': 'items', 'type': 'list'}, 'keysrules': {'check_with': 'bulk_schema', 'forbidden': ['rename', 'rename_handler'], 'type': ['dict', 'string']}, 'max': {'nullable': False}, 'maxlength': {'type': 'integer'}, 'meta': {}, 'min': {'nullable': False}, 'minlength': {'type': 'integer'}, 'noneof': {'logical': 'noneof', 'type': 'list'}, 'nullable': {'type': 'boolean'}, 'oneof': {'logical': 'oneof', 'type': 'list'}, 'readonly': {'type': 'boolean'}, 'regex': {'type': 'string'}, 'require_all': {'type': 'boolean'}, 'required': {'type': 'boolean'}, 'schema': {'anyof': [{'check_with': 'schema'}, {'check_with': 'bulk_schema'}], 'type': ['dict', 'string']}, 'type': {'check_with': 'type', 'type': ['string', 'list']}, 'valuesrules': {'check_with': 'bulk_schema', 'forbidden': ['rename', 'rename_handler'], 'type': ['dict', 'string']}}

class gms_preprocessing.options.config.GMS_configuration

Bases: object

class gms_preprocessing.options.config.JobConfig(ID, json_config='', **user_opts)

Bases: object

Create a job configuration

Workflow: # 0. Environment # 1. 2 Wege, wo JSON herkommen kann: per console-command oder aus Datenbank # - bei console-command: GMS_JOB.from_… muss default-options in DB schreiben # => zuerst JobConfig auf Basis von JSON erstellen # 2. dann überschreiben mit user-defined parametern (entweder init-parameter oder db-settings per webapp)

Parameters

ID – job ID of the job to be executed, e.g. 123456 (must be present in database)

:param json_config path to JSON file containing configuration parameters or a string in JSON format :param user_opts keyword arguments as passed by gms_preprocessing.set_config()

property DB_config_table

Returns the content of the config table of the postgreSQL database as dictionary.

Return type

dict

property DB_job_record

Return type

GMS_JOB

static absP(relP)

check_no_read_write_limit_on_xtfs_mountpoint()

get_data_list_of_current_jobID()

Get a list of datasets to be processed from database and return it together with some metadata.

Returns

<list> of OrderedDicts, e.g. [OrderedDict([(‘proc_level’, None), (‘scene_ID’, 5895940), (‘datasetid’, 104), (‘image_type’, ‘RSD’), (‘satellite’, ‘Landsat-8’), (‘sensor’, ‘OLI_TIRS’), (‘subsystem’, ‘’), (‘acquisition_date’, datetime.datetime(2015, 2, 5, 10, 2, 52)), (‘entity_ID’, ‘LC81930242015036LGN00’), (‘filename’, ‘LC81930242015036LGN00.tar.gz’), (‘sensormode’, ‘M’), (‘logger’, None)]), …]

get_init_argskwargs(ignore=('logger'))

Return a tuple containing dictionary of calling function’s. named arguments and a list of calling function’s unnamed positional arguments.

get_json_opts(validate=True)

Get a dictionary of GMS config parameters according to the jobs table of the database.

NOTE: Reads the default options from options_default.json and updates the values with those from database.

get_parameter(key_user_opts, attr_db_job_record='', fallback=None)

get_virtual_sensor_specs()

Returns the content of the virtual_sensors table of the postgreSQL database as dictionary.

Return type

dict

static joinP(*items)

save(path_outfile)

Save the JobConfig instance to a JSON file in the same structure like the one in options_default.json.

Parameters

path_outfile – path of the output JSON file

to_dict()

Generate a dictionary in the same structure like the one in options_default.json from the current config.

to_jsonable_dict()

Return type

dict

validate_exec_configs()

gms_preprocessing.options.config.get_conn_database(hostname='localhost', timeout=3)

Return database connection string.

Parameters

• hostname (str) – the host that runs the GMS postgreSQL database

• timeout (int) – connection timeout in seconds

Return type

str

Returns

gms_preprocessing.options.config.get_options(target, validation=True)

Return dictionary with all options.

Parameters

• validation – True / False, whether to validate options read from files or not

• target – if path to file, then json is used to load, otherwise the default template is used

Returns

dictionary with options

gms_preprocessing.options.config.is_GMSConfig_available()

gms_preprocessing.options.config.json_to_python(value)

gms_preprocessing.options.config.python_to_json(value)

gms_preprocessing.options.config.set_config(job_ID, json_config='', reset_status=False, **kwargs)

Set up a configuration for a new gms_preprocessing job!

Parameters

• job_ID – job ID of the job to be executed, e.g. 123456 (must be present in database)

• json_config – path to JSON file containing configuration parameters or a string in JSON format

• reset_status – whether to reset the job status or not (default=False)

• kwargs –

  keyword arguments to be passed to JobConfig NOTE: All keyword arguments given here WILL OVERRIDE configurations that have been

  previously set via WebUI or via the json_config parameter!

Keyword Arguments

• inmem_serialization: False: write intermediate results to disk in order to save memory

  True: keep intermediate results in memory in order to save IO time

• parallelization_level: <str> choices: ‘scenes’ - parallelization on scene-level

  ‘tiles’ - parallelization on tile-level

• db_host: host name of the server that runs the postgreSQL database

• spatial_index_server_host: host name of the server that runs the SpatialIndexMediator

• spatial_index_server_port: port used for connecting to SpatialIndexMediator

• delete_old_output: <bool> whether to delete previously created output of the given job ID

  before running the job (default = False)

• exec_L1AP: list of 3 elements: [run processor, write output, delete output if not needed anymore]

• exec_L1BP: list of 3 elements: [run processor, write output, delete output if not needed anymore]

• exec_L1CP: list of 3 elements: [run processor, write output, delete output if not needed anymore]

• exec_L2AP: list of 3 elements: [run processor, write output, delete output if not needed anymore]

• exec_L2BP: list of 3 elements: [run processor, write output, delete output if not needed anymore]

• exec_L2CP: list of 3 elements: [run processor, write output, delete output if not needed anymore]

• CPUs: number of CPU cores to be used for processing (default: None -> use all available)

• allow_subMultiprocessing:

  allow multiprocessing within workers

• disable_exception_handler:

  enable/disable automatic handling of unexpected exceptions (default: True -> enabled)

• log_level: the logging level to be used (choices: ‘DEBUG’, ‘INFO’, ‘WARNING’, ‘ERROR’, ‘CRITICAL’;

  default: ‘INFO’)

• tiling_block_size_XY:

  X/Y block size to be used for any tiling process (default: (2048,2048)

• is_test: whether the current job represents a software test job (run by a test runner) or not

  (default=False)

• profiling: enable/disable code profiling (default: False)

• benchmark_global:

  enable/disable benchmark of the whole processing pipeline

• path_procdata_scenes:

  output path to store processed scenes

• path_procdata_MGRS:

  output path to store processed MGRS tiles

• path_archive: input path where downloaded data are stored

• virtual_sensor_id: 1: Landsat-8, 10: Sentinel-2A 10m

• datasetid_spatial_ref: 249 Sentinel-2A

Return type

JobConfig

gms_preprocessing.options.options_schema module

Definition of gms options schema (as used by cerberus library).

gms_preprocessing.options.options_schema.get_param_from_json_config(paramname, json_config)

gms_preprocessing.options.options_schema.get_updated_schema(source_schema, key2update, new_value)

Module contents