FAIR data pipeline manual

initialise(config_file = $FDP_CONFIG_DIR/config.yaml,
           submission_script = its run_metadata.script_path;
           token = $FDP_LOCAL_TOKEN)

Read the working config that fair run wrote, register the config, the submission script and the code repository with the local registry, open a new code run, dated in UTC, and return the DataRegistryHandle the rest of the API works on. The registry is run_metadata.local_data_registry_url (default http://127.0.0.1:8000/api/) and the API version run_metadata.api_version (default 1.0.0). token is the registry's access token, which registering needs; fair run sets FDP_LOCAL_TOKEN to it.

With no arguments, the files are those in the folder fair run names in FDP_CONFIG_DIR when it launches a script: a call with that unset, or with a file that is not a working config, throws a ConfigFileException saying so. So does a working config that gives the run no description, which the registry requires of a code run; nothing is then registered. The submission script is the one the working config names as run_metadata.script_path - the copy fair run wrote in that folder, script.sh, or script.jl for a user config with shell: julia, and so on - else script.sh (script.bat on Windows) in the folder. After rehearse the same call opens a Rehearsal instead, from the job folder it prepared: nothing is registered, and the handle holds no token, whatever is given or set.

A session has one run, opened once, however the files are given: a second call while the run is open, or after finalise or discard has ended it, is a ReadWriteException - for a rehearsal, to rehearse again; for a recorded run, since a script making several runs is not supported.

The code repository is registered at its commit as one object, kept where the registry holds one already. A new one has the repository's own authors, read from the first of its metadata files that it has - codemeta.json, .zenodo.json, CITATION.cff, JuliaProject.toml, Project.toml - as it is at the commit in the clone run_metadata.local_repo, or in its working tree for a run from uncommitted changes; where that file lists none, the runner. The licence and keywords that file gives are added to the object either way. Nothing is taken from the files after it, even what it lacks, and what they say more or otherwise is told; that file being unreadable is an error.

A code repository with uncommitted changes, which fair run --dirty records as a <commit>-dirty commit, gets a warning and an issue saying that the code that ran cannot be recovered from its commit.

The clone run_metadata.local_repo is looked at as the run opens, and again as finalise ends it. A commit other than the run's checked out as it opens, another checked out as it ends, or a tracked file whose state or bytes differ between the two - untracked and ignored files apart, and changes the script made before the run opened allowed - gets a warning, and finalise raises one issue on the run's outputs saying so.

finalise(handle)

Complete the code run: name every output file by its hash and register it, register the issues raised, attach the inputs and outputs to the code run, and append the code run's uuid to coderuns.txt beside the working config, where the CLI's fair add and fair push find it. Return the data store's root, the folder under which the output files now sit, by namespace and data product. In a Rehearsal nothing is registered and the outputs are collected in its folder instead, which is returned.

Every output is registered as a new version: one written as a version that is registered already is an error, raised before anything is registered. An output whose file is that of a version registered before - its bytes, and where a link call named its source, the same source, authors and licence - is said to be identical to it and recorded in the handle's identical, its bytes stored once; where every output is, the run is said to need no adding and pushing. Where the run's checkout of its code repository was not what the run records, as initialise describes, one issue saying so is raised on the outputs.

finalise(handle::DataRegistryHandle{Rehearsal})

End a rehearsal, registering nothing. The outputs are checked as a recorded run checks them: that each source is one file, that none is written as a version registered already, and whether each is identical to a version registered before, which is reported, as is every one being so, and whether the checkout moved. They are then moved to the outputs folder as <namespace>/<data product>/<version>.<extension>, replacing what is there, with the working config and the script copied beside them. The issues raised are printed, the data store's empty directories pruned, the job folder deleted if rehearse had fair prepare it, and the session's rehearsal ended. Return the outputs folder. A handle whose rehearsal has ended cannot be finalised again.

DataRegistryHandle{M <: AbstractRunMode}

The state of one code run, created by initialise and completed by finalise. Every other function in the API takes one. M is the run's AbstractRunMode, which decides what the run registers.

Fields

• mode: the run's mode, an M
• config: the working config, as read from config.yaml
• config_dir: the directory the working config was read from, where coderuns.txt is appended to at finalise
• registry: the RegistryEndpoint every request goes to
• config_obj: registry URL of the object for the working config file, or nothing in a run that registers none
• script_obj: registry URL of the object for the submission script, or nothing
• repo_obj: registry URL of the object for the remote code repository, or nothing
• datastore_obj_url: registry URL of the storage root for the data store, or nothing
• code_run_obj: registry URL of the code run, or nothing
• code_run_uuid: the code run's uuid, as the CLI's staging refers to it
• inputs, outputs: metadata of what has been read and written so far, keyed by (data_product, component), with component nothing for a whole file
• identical: the outputs whose file is that of a version registered before the run - its bytes, source, authors and licence - each with the highest such version, keyed as outputs is; filled by finalise
• issues: issues raised so far, registered at finalise
• citations: the citations read by register_citations!, by key
• doi_lookups: the citations looked up from a DOI so far, by the DOI's URL
• authors: the identifiers that register_authors! has been given for each author's name
• checkout: the run's clone of its code repository as initialise found it, to compare with it at finalise, or nothing where there is none to look at

AbstractRunMode

The way a code run is being made, which decides what the API registers. A DataRegistryHandle carries its run's mode as its type parameter. A RecordedRun registers everything; a Rehearsal nothing.

RecordedRun()

The mode of a code run made under fair run: its working config, script, repository and code run are registered at initialise, and its outputs, issues, inputs and outputs at finalise.

Rehearsal(outputs)

The mode of a run that registers nothing, prepared by rehearse: the script runs as fair run would run it, reading what the registry holds and writing to the data store, and finalise collects what it wrote in the folder outputs, as <namespace>/<data product>/<version>.<extension>, beside the working config it ran under. The handle holds no registry token.

RegistryEndpoint(url, api_version = "1.0.0"; token = nothing)

A data registry to talk to: the base url of its REST API (with a trailing slash, e.g. "http://127.0.0.1:8000/api/"), the api_version sent in the Accept header of every request, and the access token sent with them, if there is one. Reading needs no token; registering anything does. initialise builds one from run_metadata.local_data_registry_url and run_metadata.api_version in the working config, with the token that fair run puts in FDP_LOCAL_TOKEN.

The token is never shown when the endpoint is printed.

ReadWriteException(msg)

Thrown when the registry or the data store does not hold what a read or write needs: a missing data product, a version already registered, a registry that is not running, a write with no registry token; or when a read or write asks for what is not yet implemented, such as a table.

ConfigFileException(msg)

Thrown when a configuration file - the working config, or the CLI's own - lacks the section or entry a call refers to.

rehearse(path; outputs = nothing, local_only = false, restart = false)
rehearse(f, [path]; outputs = nothing, local_only = false, restart = false)

Prepare this session to run a script through the pipeline without recording it. The script then runs unchanged: its initialise opens a Rehearsal rather than a recorded code run, its reads and writes go as they would, and its finalise collects the outputs in a folder and registers nothing. path is a user config, which is prepared as fair run prepares one, from the project the config is in and with --local if local_only is true: by fair run --rehearse, which writes the working config and the script into a new job folder, registers and stages nothing, and makes no git check, and which fair-cli has from 0.11.0. Or path is that job folder itself, or the working config in it. Return the job folder.

outputs is the folder the outputs are collected in, made if need be; by default a new folder under the system's temporary directory, which lasts as long as the system keeps it. The session keeps the rehearsal - the job folder, the outputs folder, and the handle once opened - and initialise() opens it once: a second call is an error until finalise or discard has ended it, and after that too, until rehearse is called again. A second rehearse before the rehearsal has ended is an error unless restart is true, when the rehearsal in progress is discarded first: its handle's files removed from the data store, and its job folder deleted if rehearse had fair prepare it. A job folder given as path is not deleted. A session that has recorded a code run cannot rehearse. A token in the environment is not used by a rehearsal.

With a function f first, as a do block, the rehearsal is a check: its handle is opened at once and passed to f, the rehearsal is discarded when f returns or throws - what f wrote removed from the data store - and f's value is returned. The session is then empty, so that a script's initialise() opens the run it was launched for. With no path, the rehearsal is of that run, the job folder fair run names in FDP_CONFIG_DIR, so that a script can find out whether it has anything to do before a code run exists:

nothing_to_do = DataPipeline.rehearse() do handle
    return isempty(missing_inputs(handle))    # the script's own check
end
nothing_to_do && exit()
handle = DataPipeline.initialise()

A block prints one line, the run after it saying again what a rehearsal says as it opens, and an outputs folder made for it is removed if f left nothing there. A script that fair run launched, which sets FDP_SESSION for it, rehearses in a block only: rehearse(path) there is an error, since its initialise() would open the rehearsal in place of the run fair run prepared, and record nothing.

The rehearsal's script reads the local registry, so the rehearsal keeps it running until finalise or discard ends it: through a session of the CLI's, which ends with it - and, run by fair run, through the run's own session. A registry that fair registry start started runs until fair registry stop either way.

The fair that prepares a user config is the one set_cli! recorded, else the fair on the path, else the copy install_cli made, and with none of those the call fails saying so. Julia started from an editor has no virtual environment on its path, which is what the last two are for. @fair_str runs the same fair with any arguments.

discard(handle)

End a rehearsal without collecting its outputs: remove the files it wrote to the data store, delete the job folder it ran from if rehearse had fair prepare it, and end the session's rehearsal. Return nothing. A handle whose rehearsal has ended only has its files removed. A recorded code run, being registered already at initialise, cannot be discarded: it ends with finalise.

fair"arguments"

Run the pipeline's command-line tool fair with its arguments written as on the command line, in the current directory, printing what it prints: fair"pull config.yaml" is fair pull config.yaml. The text is split as Julia's backticks split a command, so quotes group a word, and $name and $(expression) interpolate, as in fair"pull $config". An exit status other than 0 is a ReadWriteException. The fair run is found as rehearse finds it: the one set_cli! recorded, else the one on the path, else the copy install_cli made. A command that asks questions, as fair init does without --ci, needs a terminal. Interrupting the call, as with Ctrl-C, ends fair and every process it started, such as the job of a fair run, and so does Julia's exit.

set_cli!(path)
set_cli!(nothing)

Record path as the fair this package runs - by rehearse and @fair_str, before any fair on the path or the copy install_cli made - after checking that it runs. The record is a preference in the LocalPreferences.toml of the first environment on the load path that has DataPipeline: the active one where DataPipeline is among its dependencies, else the one it is loaded from, such as the default environment. It holds for every later session that takes DataPipeline from that environment. nothing removes it. Return the path recorded.

install_cli(; version = nothing, python = nothing)

Install the package's own copy of fair, or bring it up to date, and return the path of its executable. The copy lives in a scratch space of the package's, with a Python and a git of its own that MicroMamba fetches from conda-forge, so it needs no Python, no git and no virtual environment on the machine, and rehearse and @fair_str use it when no fair is on the path, running it with its own programs on the path. fair-cli is installed from PyPI at its newest version, or at version; run again, the call brings the copy up to date, printing its version before and after. The Python is 3.12 unless python names another minor version; a change rebuilds the environment, as does an environment whose Python no longer runs. A registry installed through this copy has an environment of its own built with the copy's interpreter, so after such a rebuild it must be installed again: fair registry stop, then fair registry install --force, which a CLI may refuse while the registry runs. Needs the network.

install_cli!(; version = nothing, python = nothing)

install_cli, passing it version and python, then set_cli! on the copy, whether it was installed by this call or before: so that the copy is the fair this package runs, whatever is on the path. Return the executable's path.

read_array!(handle, data_product, component)

Read component of an HDF5 read: data product of the working config as an array, recording it as an input.

read_table!(handle, data_product, component)

Read the table that is component of data_product. Not yet implemented: it throws a ReadWriteException saying so. A table kept as a file of its own, such as a CSV, is read from the path link_read! gives.

See also: write_table!, read_array!.

read_estimate!(handle, data_product, component)

Read the value of the point estimate component of a TOML read: data product of the working config, recording it as an input.

read_distribution!(handle, data_product, component)

Read the distribution component of a TOML read: data product of the working config as a Dict with distribution, parameters and type, recording it as an input.

link_read!(handle, data_product; path = nothing, force = false)
link_read!(handle, pattern)

Return the path of the file behind a read: data product of the working config, recording it as an input of the code run. A data product with no file, such as a deposit, which a register: entry with no_file: true or link_register! with no path registers, is recorded as an input in the same way, and nothing is returned.

The file is the data store's own, named by its hash, so it must not be written to; reading it copies nothing. For code that needs its input under a name or in a place of its own, path asks for a file of its own there, and is what is then returned:

• with nothing at path, a copy is put there, any missing directories being made - on macOS a clone where the file system makes one, which shares the data store's disk space until either file is written;
• a file already there with the same contents is left as it is, but for the data store's file under another name or a link to it, which is replaced by a copy;
• a file there with other contents is an error, and nothing is overwritten, unless force is true, when it is replaced by the registered file. A directory there is always an error.

The file at path may then be written to without changing the data store's. force does nothing without path. A data product with no file takes no path.

Given a pattern instead - a name with one or more *, each matching one segment of a name, so that era5/t2m/* matches era5/t2m/1940-1949 but not era5/t2m/a/b - return the path of a temporary directory holding the file of every read: data product that matches, each recorded as an input (the directory itself is not), a data product with no file being recorded and given none. Each is a symbolic link to the data store's file where one can be made, so it must not be written to, else a copy, as on a Windows machine that allows no symbolic links. Where the working config keeps the pattern's own entry, as a CLI may, a pattern that matches nothing gives an empty directory; where it does not, that is an error. Each is named after its data product with / replaced by _, plus the file's extension; the directory is named by the SHA-1 of the sorted list of those names and the files' hashes, so the same set of files gives the same name. The directory is removed when Julia exits. A pattern takes no path.

See also: canread

link_read_files!(handle, pattern)

Return the paths of the files behind every read: data product of the working config whose name matches pattern, in name order, recording each as an input of the code run. A pattern is a name with one or more *, each matching one segment of a name, and one that matches nothing gives no paths or an error, as for link_read!. A data product with no file is recorded and gives no path.

canread(handle, data_product)
canread(handle, pattern)

Return whether the working config has a read: entry for data_product, or for pattern: whether link_read! would succeed. A pattern the config holds is readable though it matches nothing, giving an empty result. Only the working config is consulted, and nothing is recorded, so a script can check every name it will read before it does any work.

See also: canwrite

write_array!(handle, data, data_product, component, description;
            supersede = false)

Write data as component of an HDF5 write: data product of the working config, recording it as an output, and return (data_product, component). The file records no times, so that the same arrays give the same bytes and are found identical to a version registered before. supersede is as for link_write!: the data product's new version supersedes the one before it.

See also: write_table!, read_array!, read_table!

write_table!(handle, data, data_product, component)

Write data as the table that is component of data_product. Not yet implemented: it throws a ReadWriteException saying so. A table is written as a file of its own, such as a CSV, to the path link_write! gives.

See also: write_array!, read_array!, read_table!

write_estimate!(handle, value, data_product, component, description;
               supersede = false)

Write value as the point estimate component of a TOML write: data product of the working config, recording it as an output, and return (data_product, component). supersede is as for link_write!: the data product's new version supersedes the one before it.

write_distribution!(handle, distribution, parameters, data_product,
                   component, description; supersede = false)

Write a distribution (its name, and a Dict of parameters) as component of a TOML write: data product of the working config, recording it as an output, and return (data_product, component). supersede is as for link_write!: the data product's new version supersedes the one before it.

link_write!(handle, data_product; path = nothing, supersede = false,
            doi = nothing, bibtex = nothing, title = nothing,
            primary = nothing, description = nothing,
            original_store = nothing, licence = nothing, license = nothing,
            release_date = nothing, release_version = nothing)

Return a path to write a write: data product of the working config to, in the data store, recording it as an output. finalise names the file by its hash and registers it, replacing ${{RUN_ID}} in the registered name with the code run's uuid. A file written there is never copied. It is registered public, or not, as the entry's public says, else the run's run_metadata.public.

path names a file that exists already, to be registered as the data product in place of one the caller writes. A copy of it is put in the data store - on macOS a clone where the file system makes one, which shares the file's disk space until either is written - and its path there is returned; the file at path is left as it was, and may be written to afterwards without changing the registered file. Its extension must be the entry's file type.

Each run registers its output as a new version, so the version the write: entry names must not be registered already, or the call is an error. A file whose bytes are those of a version registered before - and where the call names a source, with the same source, authors and licence - is registered as the new version all the same, and finalise says that it is identical.

A file that comes from somewhere else is registered with its source. The source is named by its doi, or by bibtex, the key of a citation that register_citations! has read - one or the other. A DOI with no citation registered for it is looked up: its formal metadata, DataCite's record of it or else its CSL record, gives what it has of the title, the description, the licence, the release date and version, and the authors with their ORCIDs or ROR ids. Each keyword below takes the place of what the citation says, and the title and the release date must come from one or the other. The authors of the registered file are the citation's, found when the call is made - by the identifier the citation gives one, which is registered and comes before any that register_authors! was given, else as register_authors! describes - and not the user. Of a source:

• primary must be given: true if the file is the cited item or one of its files, so that the same bytes can be fetched again; false if it was requested or derived from it, as an extract is.
• title is the file's own, where the citation's is that of a whole deposit: the files of one deposit are told apart by their titles.
• description says more of it; for an extract, the request that made it.
• original_store is the URL that the file was fetched from. It is recorded when it is one that other people can reach, and not for a file on this machine.
• licence, which may be written license, is the licence's identifier in the SPDX licence list, https://spdx.org/licenses/, as CC-BY-4.0, or its name there; or the URL of a licence that is not in the list. A licence in the list is looked for in the copy that the package keeps, in any case and with any dash, and is recorded by its identifier, or by that of the licence the list has replaced it by.
• release_date and release_version are the source's own, in place of what the citation says or lacks: a Date, a DateTime or text in ISO form as far as it goes; and a version, which is written in full, 2.1 as 2.1.0, and is 1.0.0 when there is none.

A source is one file: its identifier, title and release version may be held by any number of data products, all of the same bytes, and another file under the same three is an error, raised by this call when it has a path and otherwise by finalise before it registers anything. Such a file needs a title of its own, or the release_version it was taken from.

data_product may also be a name that a wildcard write: entry stands for - one whose name holds *s, each matching one segment - in which case that entry's file type, description and use: block are used and the name is registered as given. A name matching the patterns of several entries is an error, since nothing distinguishes them: the file type comes from the entry found, not from the caller.

supersede = true says that this version supersedes the one before it for everything made from that: once the new version is registered at finalise, an issue reading Superseded by namespace:name@vX.Y.Z is raised on the highest version before it, at severity 0, the lowest, so that identify of anything made from the old version reports it. Where there is nothing to supersede - the file is identical to that version, the name has no version before, or the version written is not above the highest - nothing is raised, and an info message says so.

See also: canwrite

canwrite(handle, data_product)

Return whether the working config has a write: entry that covers data_product - one naming it exactly, or one wildcard entry whose pattern it matches: whether link_write! would find an entry to write it under. A name that the patterns of several entries match is not covered. Only the working config is consulted, and nothing is recorded, so a script can check every name it will write before it does any work.

See also: canread

register_citations!(handle, citations...)

Read citations of the sources that files come from, and keep them in the handle for the rest of the code run, each under its own key. Return the keys of the entries read, in order.

Each of citations is BibTeX text if it starts with @, and otherwise the path of a BibTeX file; either may hold several entries. An entry of any type is accepted, @dataset among them. A value written as an abbreviation must be defined in the same text, by @string with its value in quotes; a month may be a number, a name or an abbreviation of one. An entry may be read more than once, but two different entries with one key are an error, and so is a citation from which no entry can be read; a call that fails keeps nothing. Nothing is sent to the registry.

From an entry are taken its doi, as a URL in lower case, or else its url, to identify the source; its title; its authors, a person as given names then family name and an organisation as it is written; its date, or its year, month and day as far as they go, as the source's release date; its version; its note, as a description; and its license or copyright, as its licence, when that is an identifier or a name in the SPDX licence list, or a URL - any other is warned of and passed over. Braces are dropped, and LaTeX commands are left as they are.

register_authors!(handle, authors...)

Register authors with the registry together with the identifiers that say who they are - an ORCID for a person, a ROR id for an organisation - and return their names, sorted. A citation in BibTeX gives its authors by name alone: a name registered here stands, for the rest of the run, for the author with its identifier, and any other name is an author by the name alone, whatever the registry holds under it. Where a DOI's metadata identifies one of its authors, that identifier is the one for that author.

Each of authors is a name => identifier pair, or the path of a TOML file with a line for each author:

"Josiah Carberry" = "https://orcid.org/0000-0002-1825-0097"
"An Example Institute" = "https://ror.org/000000000"

A name is written as a citation's author is read: a person's given names and then family name, an organisation as it stands. An identifier is a URL, and one written with http: is registered with https:. If any of authors cannot be read, nothing is registered.

An identifier may be given several names, the spellings that citations use. The registry keeps one name for it: the one it holds already, or else the first given here, taking the arguments in order and a file's authors in order of name. A message says when another name stands for that author.

Several authors may share a name, each with an identifier of its own. A citation of a name that the run has registered with several identifiers cannot say which of them is meant: its author is recorded by the name alone, and a warning says so.

Call this before a source that names these authors is cited: an author recorded by name alone is not given an identifier afterwards.

link_register!(handle, data_product; path = nothing, supersede = false,
               doi = nothing, bibtex = nothing, title = nothing,
               primary = nothing, description = nothing,
               original_store = nothing, licence = nothing,
               license = nothing, release_date = nothing,
               release_version = nothing)

Register something that comes from outside the pipeline, with its source, as the data product data_product, which the working config does not name, and record it as an input of the code run. No run made it: it is this run's input and no run's output. The data product is named as the pipeline names one, namespace:name@version, and may leave out the namespace, the version or both: name, name@1.2.3, namespace:name.

With no path it is a data product with no file - the deposit that a source's files belong to, which a run that fetches them reads, so that each file it registers as an output is derived from it - and nothing is returned. With a path it is that file, put in the data store as link_write! puts one, a copy - on macOS a clone where the file system makes one - and its path there is returned; the file at path may be written to afterwards without changing the registered file. It is registered public, or not, as the run's run_metadata.public says.

The namespace is by default the run's default input namespace. The version is the data product's in the pipeline - the source's own is release_version. With none given, a name not yet registered is version 0.0.1; a name whose highest version is registered with the same file, or with none, and the same source, authors and licence is that version, and nothing new is registered; any other is the highest version's next patch. A version given that is registered already must be registered so too, or it is an error.

The source is named by doi or bibtex, and must be: title, primary, description, original_store, licence or license, release_date and release_version say of it what they say for link_write!, and primary must be given here too. original_store, where a file was fetched from, needs a path. A source is one file, as there. supersede = true is as there too: a new version registered by the call supersedes the highest before it, with an issue on that version raised at finalise, and an info message where there is nothing to supersede.

isregistered(handle, data_product; doi = nothing, bibtex = nothing,
             title = nothing, primary = nothing, description = nothing,
             original_store = nothing, licence = nothing,
             license = nothing, release_date = nothing,
             release_version = nothing)

Return whether the registry holds data_product, registering and recording nothing. It is named as link_register! names one, namespace:name@version: the namespace by default the run's default input namespace, and with no version the highest. Or it is a ConfigDataProduct, the label a script gives link_write! or link_read!, resolved as the run resolves it: the write: entry naming it, or the one pattern covering it, under the name its use: block gives and in its namespace, else the run's default output namespace; else the read: entry naming it, or the kept entry whose pattern covers it. Its version is the entry's where that version is registered, else the highest. A label no entry covers, or several patterns cover, is a ConfigFileException, and a component of one an ArgumentError.

With no source keyword, whether anything is registered under the name, or at the version a name asks for. With the keywords link_register! or link_write! would be given, but no file, whether the version asked about records the same source - its identifier, title and release version, from the citation that doi or bibtex names and from title and release_version - the same authors, and the same licence, licence or license. primary, description, release_date and original_store are not compared, the registry keeping the last three as a first registration said them, so may be left out, though what is given is checked as the link calls check it; nor are the file's bytes compared. An author the registry has not got is not registered, and makes the answer false, as registering would add one. A doi with no citation is looked up at doi.org as the link calls look it up.

So a script can ask before it fetches anything whether there is anything to fetch, and why: false by the name, nothing is registered; true by the name and false with the source, it is registered with other facts.

See also: canread, identify

raise_issue!(handle, target, description; severity = 1,
             all_versions = false)
raise_issue!(handle, targets, description; severity = 1,
             all_versions = false)
raise_issue!(target, description; remote = nothing,
             severity = 1, all_versions = false)
raise_issue!(path, description; remote = nothing,
             severity = 1)
raise_issue!(registry, target, description;
             severity = 1, all_versions = false)

Raise an issue with one thing, or with several at once as one registry issue. A target is an AbstractIssueTarget - WorkingConfig, SubmissionScript, CodeRepository, ConfigDataProduct or ExistingDataProduct - or a String: the path of a file that exists, as identify takes one; a data product of this run by its config name, as ConfigDataProduct names one; or a registered data product as namespace:name@version (namespace:name for its newest version), as ExistingDataProduct names one; or an AbstractCommitSelection, which stands for the CodeRepository of the commits it selects.

An issue on a file is on every object the registry holds its bytes at - every version of every data product with them - as they are when the issue is registered, the file's bytes being taken when it is raised. The registry must hold the file already, and a path that is also the name of a data product in the working config is an ArgumentError, being both. severity is an integer, larger for worse, by default the registry's own default. Issues are queued in the handle and registered by finalise, so an output can be named before it is written.

A data product's bytes may be registered as other versions of the same data product too, which the issue would then not reach. finalise looks for them, and warns of any; with all_versions = true it raises the issue on every such version as well, as one issue over their components. There is no prompt, since a script may run where no one can answer. An issue on a file is on every such version already, so takes no all_versions = true.

Outside a run - given a registry, or none for the CLI's local registry or its remote registry named remote - the issue is registered at once, and its URL returned, which downstream takes as issue. Only a target that names what it is on without a run can be raised so: a file, an ExistingDataProduct, or a string of its form, and commits - an AbstractCommitSelection, or a CodeRepository given one; any other is an ArgumentError. So raise_issue!("results/table.csv", "Wrong units") raises an issue on every version registered with that file's bytes. So raise_issue!(AncestorCommits(of = "v1.2", including = false), "Fixed in v1.2") raises an issue on the code that came before v1.2. Registering needs the registry's token, which RegistryEndpoint reads from the CLI's configuration. While the session is rehearsing a run, the issue is printed as what would be raised, nothing is registered, and nothing is returned.

Arguments

• handle::DataRegistryHandle: the code run the issue is raised in.
• registry::RegistryEndpoint: the registry to raise it in, outside a run.
• target, targets, path: what the issue is on, as above.
• description::String: the issue.
• severity::Integer: larger for worse.
• all_versions::Bool: whether to raise it on the other versions of a data product holding the same bytes too; not with a file, whose issue is on them already.
• remote: with no registry given, the remote registry of that name in the current project's CLI configuration, else the CLI's local registry.

AbstractIssueTarget

The thing a registry issue is raised against. Concrete targets are WorkingConfig, SubmissionScript, CodeRepository, ConfigDataProduct and ExistingDataProduct. Each resolves, at finalise, to the object components the issue is attached to.

WorkingConfig()

Target for raise_issue!: this code run's working config file.

SubmissionScript()

Target for raise_issue!: this code run's submission script.

CodeRepository(commits = nothing)

Target for raise_issue!: commits of a code repository. With no commits, the commit this code run was made from. With an AbstractCommitSelection, every commit it selects that the registry holds a run of, as identify finds them, which needs no run: so CodeRepository(AncestorCommits(of = "v1.2", including = false)) is the code that came before v1.2. A selection given to raise_issue! by itself means this target.

ConfigDataProduct(data_product; component = nothing)

Target for raise_issue!: a data product of this code run, named as it is in the working config's read: or write: section (the name passed to link_read!, link_write! and the read_*/write_* functions), and optionally one named component of it. Resolved at finalise, so it may name an output that is not yet written. A bare String given to raise_issue! means this target. Given to isregistered, with no component, it asks about the data product the label stands for.

ExistingDataProduct(data_product; component = nothing)

Target for raise_issue!: a data product already in the registry, whether or not this code run uses it, named as the pipeline names one - namespace:name@version, or namespace:name for its newest version - and optionally one named component of it. A String of that form given to raise_issue! means this target.