data_pipeline_api

finalise(token: str, handle: dict) → None

Renames files with their hash, updates data_product names and records metadata in the registry

Args:

token: registry token

config: Path to config file

script: Path to script file

Returns:

dict: a dictionary containing the following keys:

‘yaml’: config_yaml path

‘fdp_config_dir’: config dir path

‘model_config’: model config url

‘submission_script’: submission script object url

‘code_repo’: code repo object url

‘code_run’: coderun url

‘code_run_uuid’: coderun uuid

‘author’: author url

‘inputs’:

‘data_product’: data product

‘use_data_product’: data product

‘use_component’: component

‘use_version’: version

‘use_namespace’: namespace

‘path’: path

‘component_url’: component url

‘outputs’:

component_url: component url

data_product_url: data product url

get_handle_index_from_path(handle: dict, path: str) → Any | None

Get an input or output handle index from a path usually generated by link_read or link_write

Args:

handle: the handle containing the index

path: path as generated by link_read or link_write

initialise(token: str, config: str, script: str) → dict

Reads in token, config file and script, creates necessary registry items and creates new code run.

Args:

token: registry token

config: Path to config file

script: Path to script file

Returns:

dict: a dictionary containing the following keys:

‘yaml’: config_yaml path,

‘fdp_config_dir’: config dir path,

‘model_config’: model config url,

‘submission_script’: submission script object url,

‘code_repo’: code repo object url,

‘code_run’: coderun url,

‘code_run_uuid’: coderun uuid,

‘author’: author url

link_read(handle: dict, data_product: str) → str

Reads ‘read’ information in config file, updates handle with relevant metadata and returns path to write data product to.

Args:

data_product: Specified name of data product in config.

Returns:

path: Path to write data product to.

link_write(handle: dict, data_product: str) → str

Reads write information in config file, updates handle with relevant metadata and returns path to write data product to.

Args:

data_product: Specified name of data product in config.

Returns:

path: Path to write data product to.

raise_issue_by_data_product(handle: dict, data_product: str, version: str, namespace: str, issue: str, severity: int, group: bool = True) → None

Raises an issue for a given data_product in the run_metadata and writes it to the handle

Args:

data_product: the data_product name as a string

version: version of the data_product as a string

namespace: namespace of the date_product as a string

issue: what the issue is as a string

severity: How severe the issue is as an interger from 1-10

raise_issue_by_existing_data_product(handle: dict, data_product: str, version: str, namespace: str, issue: str, severity: int, group: bool = True) → None

Raises an issue for a given data_product and writes it to the handle

Args:

data_product: the data_product name as a string

version: version of the data_product as a string

namespace: namespace of the date_product as a string

issue: what the issue is as a string

severity: How severe the issue is as an interger from 1-10

raise_issue_by_index(handle: dict, index: bool | None, issue: str, severity: int, group: bool = True) → None

Raises an issue for a given input or output index and writes it to the handle

Args:

handle: the handle containing the index

index: a unique reference to an input or output in

the handle see get_handle_index_from_path()

issue: what the issue is as a string

severity: How severe the issue is as an interger from 1-10

raise_issue_by_type(handle: dict, type: str, issue: str, severity: int, group: bool = True) → None

Raise an issue by type of issue (with config, with submission_script, with github_repo)

raise_issue_with_config(handle: dict, issue: str, severity: int, group: bool = True) → None

Raise an issue with config: add an issue for the config to the handle.

Args:

handle

issue: what the issue is as a string

severity: How severe the issue is as an interger from 1-10

raise_issue_with_github_repo(handle: dict, issue: str, severity: int, group: bool = True) → None

Raise an issue with config add an issue for the github_repo to the handle.

Args:

handle

issue: what the issue is as a string

severity: How severe the issue is as an interger from 1-10

raise_issue_with_submission_script(handle: dict, issue: str, severity: int, group: bool = True) → None

Raise an issue with submission script: add an issue for the submission_script to the handle.

Args:

handle

issue: what the issue is as a string

severity: How severe the issue is as an interger from 1-10