Building a paper

A LaTeX paper whose figures and tables come from pipeline runs can be built in a code run of its own. The pdf is then a data product like any other: its provenance runs through the files it uses to the runs that made them and the data behind those; its authors, licence and keywords are those of the paper's CITATION.cff, held by the registry and written into the pdf; and an issue raised on anything behind it reaches it. This page is the whole cycle: laying the paper out, setting it up once, and checking, updating and building it each time it changes.

Laying the paper out

A paper's files are of two kinds. Those that come from runs - figures, and tables written to be \input - are in its pipeline folder, pipeline/ beside the .tex file by default. Each must have the bytes of a registered data product, and the run that builds the paper records it as an input. Everything else the paper uses from its repository is its own, wherever it is, and recorded with the repository's commit: its .tex files and its bibliography, usually beside it, and whatever else it includes - drawings, photographs, tables of its own - by convention in include/. The pipeline folder is the only one the build treats otherwise.

\graphicspath{{pipeline/}{include/}}
...
\includegraphics{rainfall}    % pipeline/rainfall.pdf or include/rainfall.pdf
\input{pipeline/rates}        % \input does not look in \graphicspath
\bibliography{paper}          % paper.bib, beside paper.tex

The files from the pipeline are committed with the rest, so that a co-author can build the paper with latexmk alone. Building it needs latexmk and pdfLaTeX, LuaLaTeX or XeLaTeX. Each function below takes the path of the paper's .tex file, or a DataPipeline.LaTeXPaper for a paper with another pipeline folder or engine.

Setting it up

DataPipeline.setup_paper("paper.tex"), run once in Julia, creates each of these that is missing, leaving any that exists as it is:

CITATION.cff, drafted from the paper's \title, \author and \affil. The pdf takes its authors, licence and keywords from it, so add the authors' ORCIDs and a license: by hand. In a repository holding a package, whose citation the top level of the file is, the paper's is drafted as its preferred-citation.
paper.template.yaml, the template of the user config that builds the paper: the pdf's write: entry and a Julia script. It is yours to edit; the config is written from it.
pipeline/Project.toml and Manifest.toml, the script's environment, holding DataPipeline. With project = false none is created, and the script installs DataPipeline into a temporary environment each time it runs.

The template drafted for paper.tex at the top level of a repository called weed-study:

# The template of the config that builds the paper paper.tex in a run
# of the FAIR Data Pipeline. DataPipeline.update_paper writes
# paper.yaml from it, adding a read: block for the files the paper
# uses from the pipeline. Build the paper from the repository's top
# level with `fair run paper.yaml`.
run_metadata:
  description: "Build the paper Rainfall and the spread of an invented weed"
  shell: julia
  script: |
    import Pkg
    Pkg.activate("pipeline")
    Pkg.instantiate()
    using DataPipeline
    handle = DataPipeline.initialise()
    DataPipeline.build_paper!(handle, "paper.tex")
    DataPipeline.finalise(handle)
write:
- data_product: weed-study/paper
  description: "The paper Rainfall and the spread of an invented weed, built from paper.tex"
  file_type: pdf

setup_paper then writes the config, paper.yaml, as update_paper does, where every file the paper uses from its pipeline folder is registered; otherwise it says which are not, and the config waits for update_paper.

Each time the paper changes

In Julia:

DataPipeline.check_paper("paper.tex")
DataPipeline.update_paper("paper.tex", newest = true)

then, at the terminal, fair pull paper.yaml where the check says that the local registry lacks a file, a commit of what changed, and fair run paper.yaml.

DataPipeline.check_paper changes nothing. It reports what building the paper lacks, then everything that would stop the build or that is out of date, and the checks it could not make. For the paper above, after a newer version of one of its tables was registered:

Check of the paper /home/me/weed-study/paper.tex: 1 to do
  set up: nothing is missing
  uses 2 files from the pipeline, 1 of its own and 30 from elsewhere:
    pipeline/rates.tex
    pipeline/sites.tex
    paper.tex
  metadata: the top level of CITATION.cff
  to do:
    - pipeline/rates.tex is testing:weeds/rates@v0.0.1, and v0.0.2 is newer - `update_paper(newest = true)` brings it in
  notes:
    - the top level of CITATION.cff gives no ORCID for Tobias Quennell - add them by hand
    - pipeline/sites.tex is testing:weeds/sites@v0.0.1, its newest version

DataPipeline.update_paper brings CITATION.cff's title and authors into line with the LaTeX, and writes the config afresh: the template, then a read: block offering each file the paper uses from the pipeline at the version it is. With newest = true it offers the newest version of each instead, runs fair pull, and copies each newer file over the old one, so that the repository holds what the config offers. Here it replaced pipeline/rates.tex and wrote:

read:
# pipeline/rates.tex
- data_product: "weeds/rates"
  use:
    namespace: "testing"
    version: "0.0.2"
# pipeline/sites.tex
- data_product: "weeds/sites"
  use:
    namespace: "testing"
    version: "0.0.1"

The files it changed are then committed, config and all, so that each commit of the paper says which version of each file it used. Both functions identify the files in the local registry and the project's remote registry, origin, which co-authors share, taken together: the newest version and every issue either knows. A version the remote lacks is pushed by the run that made it, fair add and fair push, and update_paper refuses to pin one until it is; remote = nothing uses the local registry alone. fair pull paper.yaml brings the versions the config names into a local registry that lacks them, as on a new machine.

The build

fair run paper.yaml runs the template's script, in which build_paper! builds the paper with latexmk and registers the pdf under the template's write: entry. It refuses, before anything is registered, a paper whose files are not as the run's commit has them - unless the run is from uncommitted changes, fair run --dirty - whose title and authors are not those of CITATION.cff, or which uses a file from the pipeline folder that the config does not offer. The pdf is built in a folder of the package's own, kept between builds, so that an unchanged paper gives the same bytes, stored once and found identical to the version before; its title, authors, keywords, licence and the authors' ORCIDs are written into its document information and, but for XeLaTeX, read back. Once the run is registered, the pdf is copied beside the .tex file, where latexmk builds one, so that the file there is the registered one; a file git tracks there is left as it is, so ignore it in .gitignore. DataPipeline.rehearse followed by the same calls builds the paper without recording anything, and warns of what a recorded run would refuse.

Without the CLI, or in a fresh clone

Reading the paper and its CITATION.cff, and drafting the template, need TeX alone. Identifying the files needs a registry, found through the CLI's configuration of the project, which fair init writes in each clone and which is not committed. check_paper says what is missing, and each is supplied by one line, none of which the paper's functions run themselves:

DataPipeline.install_cli!()            # the package's own copy of fair
fair"registry install"                 # a local registry
fair"init"                             # in the repository: who you are, and your remote

Reference

DataPipeline.setup_paper — Function

setup_paper(paper; project = true, remote = "origin")
setup_paper(registry, paper; project = true, remote = "origin")

Set the LaTeX paper paper, a LaTeXPaper or the path of its .tex file, up to be built in a code run, creating the files that are then the user's own, each only where it is missing: CITATION.cff at the top level of its repository, drafted from the paper's title, authors and affiliations, the authors' ORCIDs being left to be added by hand; the template of its config, with the PDF's write: entry and a Julia script that builds it with build_paper!; and, with project, the script's environment in the pipeline folder, a Project.toml and Manifest.toml holding DataPipeline. Then write the config as update_paper does, where every file the paper uses from its pipeline folder is registered; otherwise say which are not, or why they could not be identified, and leave the config to a later update_paper. Nothing that exists is overwritten, and what is kept is said.

Return (created, kept, config, unregistered): the files created, and CITATION.cff where a preferred-citation was added to it; the files kept as they were; the config's path, or nothing where it was not written; and the files from the pipeline that are not registered, or nothing where they could not be identified.

In a repository holding a package - one with a codemeta.json, or a Julia project file naming a package - the top level of CITATION.cff describes the package, so the paper's citation is drafted as its preferred-citation, and added to an existing file that has none. With project = false no Julia file is created, and the script installs the newest DataPipeline into a temporary environment each time it runs. The files are identified as update_paper identifies them, in the local registry and the remote together; remote names the remote, and registry is the local registry, both as for check_paper. The paper is built with latexmk, in a folder of its own, to find the files it uses.

source

DataPipeline.check_paper — Function

check_paper(paper; remote = "origin")
check_paper(registry, paper; remote = "origin")

Check the LaTeX paper paper, a LaTeXPaper or the path of its .tex file, before it is built in a run, changing nothing, and return a PaperCheck, which shows as a report.

What building it lacks comes first: fair, the repository initialised by fair init, the remote's configuration and its token, and a local registry that answers. Then the files a build uses, found by building the paper with latexmk in a folder of its own, and of each file from the pipeline folder what the local registry and the remote know of it together: which version of what it is; whether a newer version exists; the issues raised on it and on what lies behind it, and the inputs behind it superseded since, each that one registry holds alone said to be; and a version one lacks, with how to bring it there - fair pull, or for the remote the fair add and fair push of the run that made it. Then whether the paper's title and authors, in order, are those of its metadata, and what update_paper would do where they are not; which file the metadata comes from; whether it gives a licence and the authors' ORCIDs; how the authors differ from those of the PDF's version registered last; the files the paper is built from that are not committed; whether the config is what update_paper would write; the files the paper uses from elsewhere in the repository that are registered, and so are likely files from the pipeline; and the files in the pipeline folder that are not registered, but for the environment there and any file git ignores; and whether git ignores the PDF beside the .tex file, where a recorded build copies the registered PDF. A check that needs a registry that cannot be reached, or that answers with an error, is said not to have been made, and why.

remote names a remote registry in the CLI configuration of the paper's project, "origin" being the one fair init writes, or is its RegistryEndpoint; with nothing, the files are checked in the local registry alone. Where one registry answers a lookup of a file with an error, the other's knowledge of the file is taken; a check that needs both is said not to have been made. registry is the local registry's endpoint, as registry_session gives it; without it, a session holds the local registry for the call.

source

DataPipeline.update_paper — Function

update_paper(paper; newest = false, remote = "origin")
update_paper(registry, paper; newest = false, remote = "origin")

Bring the files of the LaTeX paper paper, a LaTeXPaper or the path of its .tex file, up to date outside any run: its CITATION.cff, brought into line with the paper's title and authors, and its config, written afresh from its template with a read: block offering each file the paper uses from its pipeline folder. Return (config, files, citation): the config's path; the files from the pipeline replaced, each with the data product it now is; and the changes made to CITATION.cff.

In CITATION.cff only the paper's citation is changed - its preferred-citation where it has one - and in it only the title and the author entries: entries are moved into the LaTeX's order, one is drafted, with no ORCID, for an author added, and one is dropped for an author removed. Authors both removed and added may be a renaming, so they are left to be changed by hand.

Each file from the pipeline is identified by its bytes in the local registry and the remote together, and the config pins the version it is. With newest = true the config pins the newest version of each instead, fair pull is run with it, and each newer file is copied from the local data store over the old one under the same name, so that the repository holds what the config offers. Refused, with the config left as it was: a file that is not registered, bytes registered as several data products, with newest a newer version of another file type, and a version the config would pin that the remote does not hold, with how to push it, unless remote is nothing.

The paper must have been set up by setup_paper: the template and CITATION.cff are never created here. remote and registry are as for check_paper; fair pull pulls from the project's origin, so newest = true takes remote = "origin", or nothing, which uses the local registry alone and pulls nothing. The paper is built with latexmk, in a folder of its own, to find the files it uses. The environment in the pipeline folder is left as it is.

source

DataPipeline.build_paper! — Function

build_paper!(handle, paper; data_product = nothing)

Build the LaTeX paper paper, a LaTeXPaper or the path of its .tex file, in the code run handle: record each file it uses from its pipeline folder as an input of the run, register the PDF as its output, and return the PDF's path, as link_write! does with a path.

The paper must be in the run's repository, checked out at the run's commit, and every file the build uses from the repository must be as that commit has it, as must the metadata files read, unless the run is from uncommitted changes, as fair run --dirty records one, when they are taken as they are and an info message names them. Its title and authors, in order, must be those its metadata gives - the citation of the paper in CITATION.cff, its preferred-citation where it has one, else the first of .zenodo.json, codemeta.json and its project file that exists. Each file it uses from the pipeline folder must have the bytes of a data product the working config's read: block offers. The PDF is written under the working config's one write: entry of file type pdf, or under data_product, with the authors, licence and keywords of the paper's metadata, which are written into the PDF's document information as well and read back from it. Anything wrong is an error before anything is registered.

The paper is built with latexmk, from the folder holding its .tex file, into a folder of the package's own kept between builds, so that an unchanged paper gives the same bytes, is stored once and is found identical to the version before. A rehearsal builds in a folder of its own, registers nothing, and warns of what a recorded run would refuse rather than stopping. A change of authors from the version registered before is said.

In a recorded run, once finalise has registered it, the pdf is also copied to <stem>.pdf beside the .tex file, where latexmk builds one, so that the file there is the registered one - but not where git tracks that file, which is then left as it is, with a warning.

source

DataPipeline.PaperCheck — Type

PaperCheck

What check_paper found of a LaTeX paper, shown as a report.

Fields

paper: the LaTeXPaper checked
setup: what building the paper in a run lacks - fair, the repository initialised by fair init, the remote's configuration and token, a local registry that answers - each with what supplies it; empty when nothing is lacking
uses: the files a build of the paper uses, as (pipeline, own, outside): those from its pipeline folder and the repository's own, as paths from the repository's top level, and how many others, such as TeX's own; nothing where the paper was not built
metadata: the file the paper's metadata comes from, and for CITATION.cff where in it; nothing where there is none
actions: what is to be done before the paper is built, each with how
notes: what is worth knowing that asks nothing of this package
unchecked: the checks not made, each with why

source

DataPipeline.LaTeXPaper — Type

LaTeXPaper(tex; pipeline = "pipeline", template = "<stem>.template.yaml",
           config = "<stem>.yaml", engine = PdfLaTeXEngine())

A LaTeX paper the pipeline builds: its main .tex file, the folder holding the files it uses that come from the pipeline, the template of the user config that builds it and the config made from that, and the engine it is built with. Relative paths are taken from the folder holding tex, where latexmk runs, and the template and config are named by tex's stem, so paper/article.tex has paper/pipeline, paper/article.template.yaml and paper/article.yaml. Every function that takes a paper also takes the path of its .tex file, for a paper with these defaults.

Fields

tex: the main .tex file, as an absolute path
pipeline: the folder of the files from the pipeline, as an absolute path
template: the user config's template, as an absolute path
config: the user config made from the template, as an absolute path
engine: the engine, PdfLaTeXEngine, LuaLaTeXEngine or XeLaTeXEngine

source

DataPipeline.PdfLaTeXEngine — Type

PdfLaTeXEngine()

The pdfLaTeX engine, for building a paper with latexmk -pdf; the default.

source

DataPipeline.LuaLaTeXEngine — Type

LuaLaTeXEngine()

The LuaLaTeX engine, for building a paper with latexmk -pdflua.

source

DataPipeline.XeLaTeXEngine — Type

XeLaTeXEngine()

The XeLaTeX engine, for building a paper with latexmk -pdfxe.

source