Tracing provenance

registry_session(f; remote = nothing)

Call f with the RegistryEndpoint of the local registry, held up for as long as f runs through a session of the FAIR CLI's, which this process owns and closes when f returns or throws, and return what f returns. The endpoint is the registry the session names, with the token in the file it names. A session starts the registry where nothing holds it, and the last to close stops one a session started; a registry started by fair registry start runs until fair registry stop.

Each function that reads or writes a registry outside a run - identify, provenance, downstream, crate, register_release!, raise_issue! outside a run - holds a session for the call when given no endpoint, and takes this endpoint first to share one:

DataPipeline.registry_session() do registry
    DataPipeline.identify(registry, "figures/summary.pdf")
    DataPipeline.provenance(registry, "figures/summary.pdf")
end

With remote, the name of a remote in the project's CLI configuration, f is given that remote's endpoint instead, holding nothing. Under fair run, whose own session holds the registry, nothing is opened, and f is given the endpoint the CLI's configuration names.

identify(path; remote = nothing, superseded = false)
identify(registry, path; superseded = false)
identify(handle, path; superseded = false)
identify(commits; remote = nothing)
identify(registry, commits)
identify(handle, commits)

Ask a registry what it knows about a file on disk, or about commits of a git repository.

For a file, return a FileIdentification: the data products it is registered as, where the registry expects to find it, whether a newer version of any of those data products exists, the source of any that came from elsewhere with where the file was fetched from, the issues raised against it, and the issues raised on anything behind it - the inputs read on the way to it, at the versions read, and the code, config and script of the runs on the way - each with the run it reaches the file through. With superseded = true, the inputs read on the way that a newer version of has been registered since are reported too, as outdated rather than wrong. isempty(result.objects) means the registry has no record of a file with these contents. The file is identified by the SHA-1 of its contents, as everything in the pipeline is, so a copy under another name is recognised and an edited file is not.

For commits, an AbstractCommitSelection - such as AncestorCommits(), the checked-out commit of the repository holding the current directory and its ancestors - return a RepositoryIdentification: the code runs made from the selected commits, with their outputs, and any issues raised against those commits. The repository is known to the registry by the URL of its git remote - the one named by git.remote in its .fair/cli-config.yaml, else origin - or by the URL the selection was given as its repo. Runs made with uncommitted changes are included only when the selection is made with dirty = true. When the checkout itself has uncommitted changes, AncestorCommits of the checked-out commit and CheckedOutCommit warn and use its commit as committed.

Arguments

• path::String: the file to ask about.
• commits::AbstractCommitSelection: the commits to ask about: AncestorCommits, CheckedOutCommit, AllCommits or NamedCommit, each holding its repository.
• registry::RegistryEndpoint: the registry to ask, which needs no token, e.g. RegistryEndpoint("http://127.0.0.1:8000/api/"); a DataRegistryHandle may be given instead, and its registry is used.
• remote: with no registry given, the registry is the CLI's local registry, or the remote registry of that name in the current project's CLI configuration; see RegistryEndpoint.
• superseded: for a file, whether to report the inputs behind it that have been superseded.

FileIdentification

What a registry knows about one file on disk, as returned by identify. objects is empty when the registry has never been told about a file with these contents.

Fields

• path: the file asked about
• hash: its SHA-1, which is how the registry identifies it
• objects: the RegisteredObjects registered with that hash
• provenance: the Trace back from every data product the file is registered as, or nothing for a file the registry does not know
• upstream: every UpstreamIssue the trace finds, nearest the file first
• outdated: every OutdatedInput the trace finds, when asked for with superseded = true, else nothing

RegisteredObject

One registry object a file is registered as: its description, where the registry expects to find it, what data products it serves and what problems are recorded against it. An object with no data products is one of a code run's own files - a working config, a submission script or a repository.

Fields

• description: the object's description
• url: the object's URL in the registry
• root: the storage root the file is registered under
• local_root: whether that root is on this machine
• stored_path: the file's path within that root
• data_products: the DataProductRecords it serves, if any
• issues: the IssueRecords against it and its components

DataProductRecord

A data product a file is registered as.

Fields

• namespace: the namespace it is registered in
• name: its registered name
• version: the version this file is
• newest: the newest version of the same name in the same namespace, which is version itself unless the registry holds a later one
• url: its URL in the registry
• external_object: the URL of its external object record, if it came from outside the pipeline, else nothing
• source: that external object as a TraceSource - its identifier, title and release, and where the file was fetched from - else nothing

UpstreamIssue

An issue identify found behind a file: raised on something a run on the way to the file read, at the version it read, or on a run's own code repository, working config or submission script.

Fields

• issue: the TraceIssue
• on: the TraceTarget of the issue that lies upstream of the file
• through: the TraceRun it reaches the file through - the run that read the thing, or whose own file it is
• making: the TraceProduct that run made on the way to the file
• input: true when the thing is a data product the run read, false when it is the run's repository, config or script

OutdatedInput

A data product read on the way to a file, as identify finds with superseded = true, that a newer version of has been registered since.

Fields

• input: the TraceProduct read, with superseded_by the newer version
• through: the TraceRun that read it
• making: the TraceProduct that run made on the way to the file

IssueRecord

A problem recorded against something a file is registered as.

Fields

• severity: how bad it is, as the registry holds it (larger is worse)
• description: what the problem is
• component: the name of the object component it was raised against, or nothing when raised against the file as a whole
• url: its registry URL
• uuid: its uuid

An IssueRecord names the issue to downstream, as its issue.

RepositoryIdentification

What a registry knows about a git repository, as returned by identify. commits is empty when no run was made from any of the commits selected.

Fields

• path: the repository's top-level folder, or its URL when the selection named it by that alone
• remote: the URL of the git remote the registry knows it by
• head: the checked-out commit, or nothing with no clone
• dirty: whether the working tree has uncommitted changes to tracked files; false with no clone
• selection: the AbstractCommitSelection asked for
• commits: the RegisteredCommits selected, newest commit first; those the local clone does not have come last, newest first by their earliest run

RegisteredCommit

A commit of a git repository that the registry holds runs for.

Fields

• sha: the commit's hash
• dirty: whether its runs were made from a working tree with uncommitted changes, which the registry records as <sha>-dirty
• date: its commit date, in UTC, or nothing when the local clone does not have it
• objects: the URLs of the registry objects that stand for it
• runs: the CodeRunRecords made from it, newest first
• issues: the IssueRecords raised against it
• release: the TraceRelease it is, or nothing

CodeRunRecord

A code run the registry holds.

Fields

• uuid: its uuid, by which fair and coderuns.txt refer to it
• run_date: when it ran, in UTC, as the registry records it
• description: its description
• url: its URL in the registry
• outputs: the DataProductRecords it wrote

AbstractCommitSelection

Commits of a git repository: which commits, of which repository, and whether the runs made from them with uncommitted changes are included, which the registry records as <sha>-dirty. identify and downstream take one in place of a file, and raise_issue! takes one as a target. The selections are AncestorCommits, CheckedOutCommit, AllCommits and NamedCommit.

Each has a repo field and a dirty field, set by its constructor's keywords of the same names. repo is the repository: the top-level folder of a clone, as an absolute path, or the URL of its remote, which leaves the registry's record alone to go by. Given, a folder must be the top level of a repository; left out, it is the clone of the run open in this session, else the repository holding the current directory. A selection that needs a clone - the checked-out commit, or ancestors - cannot be made with a URL.

AncestorCommits(; of = nothing, including = true, repo = nothing,
                dirty = false)

Select a commit and every commit it descends from - the code it grew from - and, if dirty, the runs made from them with uncommitted changes too. Finding ancestors needs a clone holding the commit.

Arguments

• of: the commit to start from, named as NamedCommit names one: a tag, a branch, a commit hash or the version of a registered release of the code; or a RegisteredRelease, whose commit and clone it is, so that repo need not be given. nothing, the default, is the checked-out commit.
• including: whether that commit is selected itself; with false, only the commits before it, and so not the runs made from it with uncommitted changes either.
• repo: the clone, as for every AbstractCommitSelection.
• dirty: whether the runs made with uncommitted changes are selected too.

CheckedOutCommit(; repo = nothing, dirty = false)

Select only the checked-out commit of a clone and, if dirty, the runs made from it with uncommitted changes too. repo is the clone, as for every AbstractCommitSelection.

AllCommits(; repo = nothing, dirty = false)

Select every commit of a repository that the registry holds runs for, whether or not the local clone has it, and, if dirty, the runs made from them with uncommitted changes too. repo is a clone or the URL of its remote, as for every AbstractCommitSelection.

NamedCommit(name; repo = nothing, dirty = false)
NamedCommit(release; dirty = false)

Select one commit by name: a full commit hash, as it is; a tag, a branch or an abbreviated hash, resolved by the local clone; or the version of a release of the code that the registry holds, with or without a leading v, which resolves with no clone at all. If dirty, the runs made from the commit with uncommitted changes too. repo is a clone or the URL of its remote, as for every AbstractCommitSelection; by URL, a tag or a branch is resolved by asking the remote, which needs the network and a remote that asks for no credentials, and an abbreviated hash does not resolve. A RegisteredRelease names its commit, of the clone it was registered from, else of its remote.

check_store(; hashes = true, quiet = true)
check_store(registry; hashes = true, quiet = true)
check_store(handle; hashes = true, quiet = true)

Check the data stores on this machine against the local registry, and return a StoreCheck, shown as a report. A data store is a storage root that is a folder on this machine - a file:// root marked local, as fair and the APIs register each project's - and every one is checked. Found are the registered files that are missing, whose contents are not those registered, or that have another name - a hard link, through which writing to a file elsewhere changes the registered one; the files in a store that the registry does not know; the storage locations that nothing uses; and the outputs that a run left under their temporary names, dat-..., as one that fails before finalise does. The check changes nothing: it deletes, moves, fetches and registers nothing.

The calls that put a file in a data store, such as link_write! with a path, never link it. A copy of a shared file moved over it gives it a name of its own.

Arguments

• registry: the registry whose data stores are checked, a RegistryEndpoint, which needs no token; a DataRegistryHandle may be given instead, and its registry is used. With neither, the CLI's local registry, held through a session of the CLI's while the check runs.
• hashes: whether to hash each registered file, which reads every byte, to find those whose contents have changed; with false, only whether each is there is checked.
• quiet: whether the report leaves out the CLI's own files under jobs/ - each run's coderuns.txt, the working config of a job no run registered - and summarises the missing files rather than listing each, giving a store whose folder has gone in one line. The check holds every finding either way.

StoreCheck

What check_store found in the data stores on this machine, shown as a report.

Fields

• roots: each data store checked, as (root, folder, exists, locations): its storage root, the folder that is, whether the folder exists, and how many storage locations the registry holds under it
• missing: each registered file that is not there, as (path, location, hash): where it should be, the URL of its storage location and the SHA-1 registered
• changed: each registered file whose contents are not those registered, as (path, location, hash, found, links): also its SHA-1 now and how many names it has, more than one where it is hard-linked; empty when the check was made with hashes = false
• shared: each registered file that has more than one name - a hard link, through which writing to a file elsewhere changes it - and is not in changed, as (path, location, hash, links): also how many names it has
• unknown: each file in a data store that the registry has no storage location for, other than those in unfinished and cli_files
• unused: each storage location that nothing uses - no object, and no source as where it was fetched from - as (path, location, hash); its file is not checked further
• unfinished: each output that a run left under its temporary name, dat-..., as (path, modified), its modification time in UTC; a run in progress has them too
• cli_files: each file under a store's jobs/ folder that is not registered, which the CLI keeps there, such as a run's coderuns.txt
• hashes: whether the registered files were hashed, so that changed was checked
• quiet: whether the report leaves out the CLI's files and summarises the missing files rather than listing each

provenance(x; remote = nothing, depth = nothing)
provenance(registry, x; depth = nothing)
provenance(handle, x; depth = nothing)

Trace where a data product came from: the run that made it, what that run read, the runs that made those, and so on back to the data products that no run made, each with its source. Return a Trace{Upstream}, which prints as a tree, and holds every data product, run and issue reached, keyed by registry URL.

x names the data product: a file, whose contents identify it as identify does, every data product it is registered as being a root of the trace; or namespace:name@version, as the pipeline names one, with namespace:name meaning its newest version. The registry is the one the FAIR CLI is configured to use, or the project's remote named by remote, or registry, or the one handle was opened on; no token is needed.

depth is how many runs back along any path to go. By default the trace goes to the end of every path, and a data product that no run made is a root; with depth given, a data product at that depth that a run made is marked beyond, and the tree says so.

Every data product's line says whether a newer version of its name has been registered since, and the issues raised on it; an issue raised on several things is one issue, with each of them among its targets.

Trace{D <: AbstractTraceDirection}

What provenance and downstream return: the data products, runs and issues reached from one or more data products in one direction, D, each kept once by its registry URL. Printed as a tree from the roots.

Fields

• registry: the RegistryEndpoint it was read from
• roots: the registry URLs of the data products it starts from
• products: every TraceProduct reached, by URL
• runs: every TraceRun reached, by URL
• issues: every TraceIssue raised on any of them, by URL
• depth: the number of runs along any path it was cut at, or nothing for a trace that reaches the end of every path

TraceProduct

One version of a data product in a Trace.

Fields

• url: its registry URL, the key it has in the trace
• namespace, name, version: what names it
• hash: the SHA-1 of its file - for a code repository's release, its commit - or nothing for a data product with no file
• location: where the registry says the file is, or nothing
• description: its description, or nothing
• format: the extension of its file type, as txt, or nothing
• source: its TraceSource, or nothing when it has none
• superseded_by: the highest version of its name in its namespace, when that is a newer one, else nothing
• made_by: the registry URL of the run in the trace that made it, or nothing when the trace holds none
• beyond: true when the trace was cut at this data product by its depth: a run that made it, going upstream, or that read it, going downstream, is not held
• issues: the registry URLs of the issues raised on it or its components

TraceRun

A code run in a Trace.

Fields

• url: its registry URL, the key it has in the trace
• uuid: its uuid
• date: when it started, in UTC
• description: its description
• repository: the URL of the code repository it ran from, or nothing
• commit: the commit it ran at, <sha>-dirty for a working tree with uncommitted changes, or nothing
• release: the TraceRelease that commit is, or nothing
• config: where the registry says its working config is, or nothing
• script: where the registry says its submission script is, or nothing
• inputs: the registry URLs of the data products it read that the trace holds
• outputs: the registry URLs of the data products it wrote that the trace holds
• objects: the registry URLs of the objects that stand for its repository, config and script, those it has, which an issue on any of them names
• issues: the registry URLs of the issues raised on its repository, config or script

TraceIssue

An issue in a Trace: a problem recorded in the registry against one or more things, whether or not the trace holds them all.

Fields

• url: its registry URL, the key it has in the trace
• uuid: its uuid
• severity: how bad it is, larger for worse
• description: what the problem is
• targets: everything it is raised on, as TraceTargets

TraceSource

The source of a data product, in a Trace or a DataProductRecord: the external object it was registered from.

Fields

• url: the external object's registry URL
• identifier: its DOI or other identifier, or nothing
• title: its title, or nothing
• version: the source's own release version, or nothing
• date: its release date, or nothing
• location: where the file was fetched from, or nothing
• primary: true when the registered bytes are the identified item or one of its files, false for a supplement requested or derived from it

TraceRelease

The release of the code that a commit is, as a run in a Trace, or a commit that identify reports, was made from it; see register_release!.

Fields

• name: the release's name, the path of the repository's remote, as FAIRDataPipeline/DataPipeline.jl
• version: its version, the highest the registry holds for the commit
• website: its web page, or nothing
• doi: the DOI it is cited by, as a URL, or nothing

TraceTarget

One thing a TraceIssue is raised on.

Fields

• url: the registry URL of the data product, or of the object for a code repository, working config or submission script
• label: the data product as namespace:name@vX.Y.Z, else the object's description and where it is
• component: the named component the issue is on, or nothing for the whole thing

downstream(x; remote = nothing, depth = nothing)
downstream(registry, x; depth = nothing)
downstream(handle, x; depth = nothing)
downstream(; issue, remote = nothing, depth = nothing)
downstream(registry; issue, depth = nothing)
downstream(handle; issue, depth = nothing)

Trace what was made from a data product: the runs that read it, what they wrote, the runs that read those, and so on to the end of every path. Return a Trace{Downstream}, which prints as a tree, and holds every data product, run and issue reached, keyed by registry URL. What must be made again, now that a problem is known in the start, is every data product reached that no newer version has superseded.

x names the start: a file, whose contents identify it as identify does, every data product it is registered as being a root; or namespace:name@version, as the pipeline names one, with namespace:name meaning its newest version; or an AbstractCommitSelection, such as AncestorCommits(), every data product the runs of the commits it selects wrote being a root, the commits found as identify finds them. With issue, the start is everything the issue is raised on - each data product, at the version it is on, and everything the runs made whose code repository, working config or submission script it is on - so that what the problem affects is what the trace holds. The issue is a TraceIssue or an IssueRecord from a trace or identify, its uuid, its registry URL, or the number the registry's own pages show it by. The registry is the one the FAIR CLI is configured to use, or the project's remote named by remote, or registry, or the one handle was opened on; no token is needed.

depth is how many runs forward along any path to go. By default the trace goes to the end of every path; with depth given, a data product at that depth that a run read is marked beyond, and the tree says so. Every data product's line says whether a newer version of its name has been registered since, and the issues raised on it.

register_release!(commit = CheckedOutCommit(); website = nothing,
                  remote = nothing)
register_release!(registry, commit = CheckedOutCommit(); website = nothing)
register_release!(handle, commit = CheckedOutCommit(); website = nothing)

Register a commit of a code repository as a release of the code, and return the RegisteredRelease. The version is the commit's tag that reads as a semantic version - v1.2.0, 1.2.0-beta2 - written in full; a commit with none cannot be released.

Registered, each found rather than made where the registry holds it already: the repository's object at the commit, made as initialise makes one, with the authors, licence and keywords of the repository's metadata files; the registry's record that the commit is a release, named by the remote's path, as FAIRDataPipeline/DataPipeline.jl; and a data product of the release, <organisation>:<repository>@<version>, on the same object, so that a trace and a crate take the release as they take any data product. A second call on a release changes nothing.

The release's DOI, where it has one, is its data product's source: the version DOI given as doi, or, for a repository whose .zenodo.json says Zenodo archives its releases from GitHub, the DOI of the Zenodo version of its tag, looked up there. The DOI's metadata - its title, release date and description - describes the source, and is compared with the repository's metadata files: a difference in the authors, title, licence or version is warned of, an identifier one gives and the other lacks told of. A DOI given that cannot be looked up is an error; a Zenodo version not archived yet, or a lookup that cannot reach Zenodo or GitHub, is told of, and the release is registered without a source, which calling register_release! again attaches once it can.

A commit released already under another version - a beta, say, retagged as a release - has the new version's data product registered and its release record changed to the new version, where the registry allows it; where not, a warning says that the record keeps the old version.

While the session rehearses a run, what would be registered is printed, and nothing is.

Arguments

• commit: the commit to release: CheckedOutCommit, the default, which a checkout with uncommitted changes cannot be, or a NamedCommit - a tag, a branch or a hash - whatever the checkout holds. Either holds its repository: a clone, or for a NamedCommit the URL of its remote, whose tags are then listed over the network and, on GitHub, whose metadata files are read at the commit.
• website: the release's web page; by default, for a remote on GitHub, the page of the release of its tag.
• doi: the DOI of the version released, bare or as a URL - not a concept DOI standing for every version.
• registry::RegistryEndpoint: the registry to register it in, which needs its token; a DataRegistryHandle may be given instead, and its registry is used.
• remote: with no registry given, the remote registry of that name in the current project's CLI configuration, else the CLI's local registry.

RegisteredRelease

A release of the code that the registry holds, as register_release! returns it: the registry's record that a commit of a code repository is a release, and the data product registered with it, which carries its source.

Fields

• url: the registry URL of the release record, which holds version unless register_release! warned that it could not be changed
• name: the release's name, the remote's path, as FAIRDataPipeline/DataPipeline.jl
• version: the version released
• website: the release's web page, or nothing
• path: the top-level folder of the clone it was registered from, or nothing where it was registered by the remote's URL
• repository: the URL of the repository's remote
• commit: the commit released
• object: the registry URL of the repository's object at that commit, which the release record and the data product are on
• data_product: the DataProductRecord of the release's data product, <organisation>:<repository>@<version>
• source: the registry URL of the release's source, or nothing

crate(paths...; remote = nothing, depth = 100, level = nothing,
      pack = nothing, to = nothing, strict = true)
crate(registry, paths...; depth = 100, level = nothing, pack = nothing,
      to = nothing, strict = true)
crate(handle, paths...; depth = 100, level = nothing, pack = nothing,
      to = nothing, strict = true)

Fetch the RO-Crate the registry builds for data products: a package of their files with the provenance of each, as a crate's metadata describes it, in which every run with its code at its commit, everything it read, each data product with its source and its issues travel together.

Each of paths is a file, a folder, or a data product named namespace:name@version (namespace:name for its newest version). A file is identified by its contents, as identify identifies one, and every data product it is registered as is a root of the crate; a folder means every file under it, the paper's figures in one folder being the case in mind. Files and folders whose names begin with a dot are left out and reported. A file the registry does not know is an error; with strict = false such files are left out and reported, so that a folder may hold the paper's sources beside its figures.

depth is how many runs back the crate goes along any path, 100 by default. level and pack are the registry's: level 1 or 2, what travels with each file, and pack one of "none", "roots", "supplements" and "all", whose bytes a zip holds; left out, the registry's defaults apply.

Without to, return the crate's metadata, its ro-crate-metadata.json, parsed: a dictionary with "@context" and "@graph". With to, a path ending in .zip, save the crate there as that zip, the metadata and the files pack chooses, and return the path; a path ending in .json saves the metadata there. The registry is the one the FAIR CLI is configured to use, or the project's remote named by remote, or registry, or the one handle was opened on; no token is needed.